calcifer - read the docs
TRANSCRIPT
Contents
1 Installation 3
2 Development 52.1 TL;DR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3 Release History 73.1 Next Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.2 Test-case Usage Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.3 License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.4 Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.5 Indices and tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
i
calcifer, Release 0.3.0
Calcifer is designed to provide interfaces for describing the evaluation and processing of nested higher-order datastructures by nested definitions of policy rules.
Policies may be used to evaluate some source object both for validation, and to generate template descriptions of a“complete” version of that object. This evaluation is done at runtime and can hook into arbitrary functions, e.g. forchoosing policies based on some current system state. (Hypermedia style)
Policies may be defined with implicit non-determinism, allowing the specification of multiple policy choices withminimal boilerplate for handling the aggregation of results. (Prolog style)
Calcifer also provides a system by which application-layer code can annotate specific policy rules, making the point-in-time context of a policy computation into a first-class value. This allows for rich error handling, by being aware ofspecific points of policy failure and allowing annotated policy rules to control the formatting of their own errors.
This library was written to facilitate the development of a hypermedia subscription management API. This library’sdesign is informed by that API’s goals of business logic cohesion and adaptability to changing policy rules. A majorgoal for that project has been to alleviate client integrations of their need to perform any policy determination locally;Calcifer has stemmed largely from this effort.
Contents 1
CHAPTER 2
Development
1. Create a new virtual environment
2. Install development requirements from dev-requirements.txt
3. Run tests nosetests
4. detox is installed and will run the test suite across all supported python platforms
5. python setup.py build_sphinx will generate documentation into build/sphinx/html
TL;DR
$ virtualenv env$ ./env/bin/pip install -qr dev-requirements.txt$ source env/bin/activate(env) $ nosetests(env) $ python setup.py build_sphinx(env) $ detox
5
CHAPTER 3
Release History
Next Release
• Implement greatness.
Test-case Usage Examples
# from tests/test_contexts.py
def test_apply_alchemy(self):# for our test today, we will be doing some basic alchemyinventory = [
"aqua fortis","glauber's salt","lunar caustic","mosaic gold","plumbago","salt","tin salt","butter of tin","stibnite","naples yellow",
]
# backstory:# ~~~~~~~~~~## falling asleep last night, you finally figured out how to complete# your life's work: discovering the elusive *elixir of life*!## and it's only two ingredients! and you have them on hand!#
7
calcifer, Release 0.3.0
# ...## unfortunately this morning you can't remember which two# ingredients it was.## you'll know it once you've gotten it, just have to try out# all possible mixtures. (should be safe enough, right?)
forgotten_elixir_of_life = set(random.sample(inventory, 2))
discoveries_today = set(["frantic worry", "breakfast"])
# ok time to go do some arbitrary alchemy!## game plan:alchemy_ctx = Context()
# you'll grab one ingredient,selected_first_ctx = alchemy_ctx.select("/inventory").each()first_substance = selected_first_ctx.value
# and another,selected_second_ctx = selected_first_ctx.select("/inventory").each()second_substance = selected_second_ctx.value
# take them to your advanced scientific mixing equipment,workstation_ctx = selected_second_ctx.select("/workstation")
# (btw this is your advanced scientific procedure that you are# 100% certain will tell you what some mixture is)def mix(first, second):
"""takes two ingredients and returns the resulting substance"""if set([first, second]) == forgotten_elixir_of_life:
return "elixir of life"return "some kind of brown goo"
# then you'll mix your ingredients...mixed_ctx = workstation_ctx.apply(
mix,first_substance, second_substance
)resulting_mixture = mixed_ctx.value
# ... and! in today's modern age, scientists now know to record their# results!mixed_ctx.select("/discoveries").append_value(resulting_mixture)
# got it? good!result = run_policy(
alchemy_ctx.finalize(),{"inventory": inventory, "discoveries": discoveries_today}
)
# in a flurry of excitement, i bet you didn't even stop to# look at your discoveries as you made them!## well, let's see...
8 Chapter 3. Release History
calcifer, Release 0.3.0
self.assertIn("elixir of life", result["discoveries"])
def test_apply_dangerous_alchemy(self):# nice job! and you even finished in time to go foraging for# more ingredients!inventory = [
"aqua fortis","glauber's salt","lunar caustic","mosaic gold","plumbago","salt","tin salt","butter of tin","stibnite","naples yellow",
# nice find"anti-plumbago"
]
# but unfortunately, it's the next day, and the same thing# has happened to you! except this time it was for your# other life's goal: discover the ~elixir of discord~!## well, since it was so easy...
whatever_concoction = set(['some ingredients'])
discoveries_today = set([])should_be_fine = 'overconfidence' not in discoveries_todayassert should_be_fine
# doing alchemy la la laalchemy_ctx = Context()
# grabbin' things off shelvesselected_first_ctx = alchemy_ctx.select("/inventory").each()first_substance = selected_first_ctx.value
selected_second_ctx = selected_first_ctx.select("/inventory").each()second_substance = selected_second_ctx.value
# got our ingredientsgot_ingredients_ctx = selected_second_ctx
workstation_ctx = got_ingredients_ctx.select("/workstation")
# mixin' - don't stop to thinkdef mix(first, second):
mixture = set([first, second])if mixture == whatever_concoction:
return 'missing elixir'if mixture == set(['plumbago', 'anti-plumbago']):
return 'concentrated danger'return 'more brown goo'
mixed_ctx = workstation_ctx.apply(
3.2. Test-case Usage Examples 9
calcifer, Release 0.3.0
mix,first_substance, second_substance
)resulting_mixture = mixed_ctx.value
mixed_ctx.select("/discoveries").append_value(resulting_mixture)
# wait wait wait!!def danger(mixture):
if mixture == 'concentrated danger':return True
return False
# we can't have that.danger_ctx = mixed_ctx.check(
danger,resulting_mixture
)danger_ctx.forbid()
# moral:## a strong understanding of policies and processes facilitates a# hazard-free lab environment.result = run_policy(
alchemy_ctx.finalize(),{"inventory": inventory, "discoveries": discoveries_today}
)
self.assertIn("errors", result)self.assertTrue(len(result['errors']))
License
The Calcifer library is distributed under the MIT License
Documentation
Concepts
Overview
Calcifer aims to provide a computing model for describing data processing procedures that closely match policies in asource domain.
Ultimately, Calcifer seeks to provide a means to build systems that not only validate input, but also “fill in the gaps”for incomplete or incorrect input. Calcifer offers systems the ability to generate template descriptions of valid input,as well as automatically indicate, with source domain semantics, what makes a given input invalid.
Calcifer hopes to offer this functionality with minimal impedance, to the end that code using the library can stillresemble traditional imperative and/or functional paradigms.
10 Chapter 3. Release History
calcifer, Release 0.3.0
Policy
The term policy is used to refer specifically to Calcifer “policies”, or computations in the Calcifer model. Policies aredesigned to allow analogous description of various kinds of real-world policies, e.g. software business logic, requestvalidation, and data pipeline operation.
Policy computation is stateful and matches imperative programming styles in that values are mutable and statementsare ordered. It is often more natural to describe procedures from a source domain in a non-pure1, state-driven fashion.
An example policy expression:
from calcifer import Policy
@Policydef allowed_favorites_policy(ctx):
# sorry greenctx.select("favorite_color").whitelist_values(["purple", "orange"])
This specifies that "purple" and "orange" are the only two valid choices for favorite_color.
Usage is as follows:
>>> allowed_favorites_policy.run({"favorite_color": "purple"})[{'favorite_color': 'purple'}]
>>> allowed_favorites_policy.run({"favorite_color": "red"}) # not valid[{'favorite_color': 'red',
'errors': [{u'code': 'INVALID_VALUE_SELECTION',u'context': [<policy 'allowed_favorites_policy'>,<policy 'select("favorite_color")'>,<policy 'whitelist_values'>],
u'scope': u'/favorite_color',u'value': 'red',u'values': ['purple', 'orange']}]}]
>>> allowed_favorites_policy.run({}) # anything goes![{'favorite_color': 'purple'}, {'favorite_color': 'orange'}]
Computing Model
Calcifer employs the following concepts to create its computing model:
Higher-Order Trees The data maintained in the computation of policies is structured as a tree where each node mayhave a value, and/or have a template description of values, or have neither of these.
Templates afford the ability to describe constraints on acceptable values without requiring producing actualinstances of values.
Nodes can have neither a value or a template and be completely unknown, the system attempts to make assump-tions as nodes get referenced through usage. This allows defining a node very precisely several levels deep,without committing to defining all the parent nodes upfront.
Scoping and Policy Partials In addition to the underlying tree, which can be thought of merely as a reference toits own root, the computation model also maintains a pointer to a given sub-tree or node. This is similar tojsonpointer or CSS selectors.
1 Future versions of Calcifer may attempt to model state less implicitly in order to encourage safer programming.
3.4. Documentation 11
calcifer, Release 0.3.0
This allows modular computations - individual sub-policies can be defined that operate on a scoped subset ofthe rest of the data.
At any given step in a particular computation, the whole of the computing model’s “data memory” is known asa partial. Partials are isomorphic to tuples of the form (𝑡𝑟𝑒𝑒, 𝑠𝑐𝑜𝑝𝑒),
Free Computation, First-class Context-awareness Calcifer operates in a free2 fashion, meaning that policies arebuilt as first-class values: operations for a given policy are specified and defined as data.
At a high level, contexts define nested collections of operating conditions for individual computations. Theparticular context at any step in a policy computation is analagous to the usual point-in-time instruction-pointerstack.
The difference is - the “code as data” structure gives certain affordances.
Namely, contexts themselves are first-class and can be passed around as values, providing certain “convenient”capabilities, such as customized error handling and the composition/nesting of contexts as “regular” Pythonvariables. Calcifer Context objects are individually strange little Turing toys, able to behave in different ways,depending how they are put together.
Non-Determinism Mimicking the common logic programming paradigm, Calcifer’s computing model allows for theforking and pruning of operations. Policy determination may return 0 results, 1 result exactly, or any number ofvalid results.
The over-arching mechanism is akin to the parallel computation of different policy alternatives, removing failingpolicies along the way, and producing some list of results.
Usage Examples of Features
Deferred Values - Rudimentary Role Permissions
Given some data access for permission lookup based on role:
def fetch_allowed_permissions(role='nobody'):"""Given some system role, return allowed permissions"""# ...# connect to db, fetch from an API, load from settings, whatever.# ...return list(permissions)
The policy for an incoming request can be expressed as follows, where "role" and "permission" are propertiesof the incoming request.
from calcifer import Policy
@Policydef allowed_permissions_policy(ctx):
role_ctx = ctx.select("role") # available when the policy is computedfetched_permissions_ctx = ctx.apply(
fetch_permitted_values, # apply this functionrole_ctx # over this value
)
ctx.select("permission").whitelist_values(
2 The concept is that of a Free Monad, giving access to the underlying AST of the computation at definition runtime as well as execution runtime.
12 Chapter 3. Release History
calcifer, Release 0.3.0
fetched_permissions_ctx)
The _ctx suffix is used to indicate that the variables do not hold actual values. role_ctx can be verbalized as “thecontext where the value of the role is known”, or fetched_permissions_ctx can similarly be thought of as“the context where the permissions have been fetched.”
(Sometimes these contexts are never reached, a topic for another section, but N.B. that these are first-class values andsubject to control flow)
In this example, Context values are used in three capacities:
1. As deferreds: role_ctx is used as a stand-in for the value of the role, whenever, say, a request comes in, andthe system is calculating what actions to allow.
2. As function applications over deferred values: ctx.apply() takes args function_or_function_ctx,*values_or_value_ctxes, and connects the plumbing to ensure that the function is called correctly whenvalues are available.
3. As stateful operators: ctx.select("permission").whitelist_values(...) indicates that thepolicy computation may have more than one valid result. This is the forking operation described above: oncethe policy knows the fetched permissions, the policy specifies some number of valid alternatives.
API
Context API
Overview and Purpose
Contexts are ordered containers of policy rules, functions that generate policy rules, and/or sub-contexts. Contextsprovide semantic grouping of policy rules and policy rule generation through means of deferred value resolution.
The primary goal for Context is to allow the semantic expression of request- processing policies, so that application-level policy code can be written with minimal regard for the common underlying non-determinism and error-propagation behaviors.
General Structure / Scope
Structurally, Context comprises the following parts:
• An ordered list of items contained in the context
• A wrapper, within which run the context’s contained items
• Additional semantic annotations such as a context name
A context object may be used for either or both of the following purposes:
• Contexts may be used to represent a distinct semantic grouping of policy rules, as might be expressedby business logic or security requirements. This may be granular or broad. This semantic grouping mayspecify some control flow for the contained items, or it may define the way in which errors are represented.
Typically, the semantic meaning of each context is along the lines of either: “with regard to some field orparameter”, or “with regard to some condition being the case”.
Using contexts in this fashion follows the builder pattern - applying chained method calls to the contextobject.
3.4. Documentation 13
calcifer, Release 0.3.0
• Or, the context may represent a value that will exist when policy is run on a given request. For instance,ctx.select(“merchant_type”) would represent the value for that node.
A context may be used as this deferred value by passing it as an argument to any exposed method on anyparent context object.
Contexts may be used as values in either a context free or a context specific manner. The former is moretypical, just passing the context to some method. Context-specific values can be accessed as a propertysomectx.value, and can only be used as arguments for methods in contexts descending from the valueprovider (somectx)
Context
class calcifer.contexts.Context(wrapper=None, *ctx_args, **kwargs)Context provides a high-level interface for building policies.
Policies are built by performing stateful operations on Context objects.
Method calls/property retrievals modify the Context to potentially include additional policy rules, as appropriate.
Implementation details can be found in BaseContext
add_error()Create a blank error
append_value(value)Appends value to the current node, assuming the node to be a list if not defined
children()Return the context with the list of scopes that are direct children of the current node
each(**kwargs)Create and return a context that operates on each child of the current node.
Parameters ref – An injectable reference object that has matching children nodes (same struc-ture dict or list)
err()Trigger error handling
error_ctx()Retrieves, possibly creating, a specialized error handler policy for the Context
fail_early()Returns a new context that checks node “/errors” and short-circuits if any errors exist.
forbid(*args)Opposite of require() - errors when value is defined
last_errorReturns the context selecting the most recently defined error
or_error()If context fails, inject error instead. Error has the following properties:
value Value found at node
scope The current scope at the time of error
context The contextual traceback
require(*args)Requires that a value is defined and truthy.
14 Chapter 3. Release History
calcifer, Release 0.3.0
Parameters value – if not provided, uses value for current node
set_value(value)Sets the value for the current node
whitelist_values(values)Forks computation, erring if value is provided already and does not match
Low-level Policy Operators
Premium Command Policy StateT Operators.
These are provided as building blocks for specifying Premium Command Policies for the purposes of template gener-ation and command validation.
Partial Operators
calcifer.operators.scope()Returns the current scope for the partial
Returns PolicyRule string json pointer
calcifer.operators.select(scope, set_path=False)Retrieves the policy node at a given selector and optionally sets the scope to that selector. Recursively definesUnknownPolicyNodes in the partial.
Parameters
• scope (json pointer string) – Scope to select
• set_path (bool) – Sets the scope
Returns PolicyRule (Node v)
calcifer.operators.get_node()Retrieves the node at the current scope
Returns PolicyRule (Node v)
calcifer.operators.define_as(node)Define the node at the current scope
Parameters node – Node v
Returns PolicyRule (Node v)
calcifer.operators.get_value()Retrieves the value for the node at the current pointer. Equivalent to get_node() >> unit_value
Returns PolicyRule v
calcifer.operators.set_value(value)Sets the value for the currently scoped policy node. Overwrites the node with a LeafPolicyNode
Parameters value (v) – new value
Returns PolicyRule v
calcifer.operators.append_value(value)Gets the value at the current node and appends value. The current node value should be either a set or a list, orundefined.
Parameters value – value to append
3.4. Documentation 15
calcifer, Release 0.3.0
calcifer.operators.children()For DictPolicyNodes or ListPolicyNodes, returns all scopes that are direct children.
Returns PolicyRule [scope]
Control-flow Operators
calcifer.operators.unit(value)Returns a value inside the monad
Parameters value – the value returned inside the PolicyRule monad
calcifer.operators.unit_value(node)Given a node (often returned as monadic result), return the value for the node.
Parameters node (Node _v_) – the node whose value is to be returned inside the PolicyRulemonad
Returns PolicyRule _v_
calcifer.operators.collect(*rule_funcs)Given a list of policy rule functions, returns a single policy rule func that accepts some value, provides that toeach function, resetting the scope each time.
calcifer.operators.policies()Given a list of policy rules, returns a single policy rule that applies each in turn, keeping scope constant for each.(By resetting the path each time)
calcifer.operators.regarding()Given a selector and a list of functions that generate policy rules, returns a single policy rule that, for each rulefunction:
1. sets the scope to the selector / retrieves the node there 3. passes the node to the rule_func to generate a policyrule 4. applies the policy rule at the new scope
In addition, regarding checks the current scope and restores it when it’s done.
calcifer.operators.check()Given a function that takes no arguments, returns a policy rule that runs the function and returns the result andan unchanged partial
calcifer.operators.each(*rule_funcs, **kwargs)each(rule_func) is a policy rule function that accepts a dictionary and calls rule_func(value) successively, withthe partial scope set to the key.
each optionally takes a named argument ref=dict() to provide a built-in lookup for some reference dictionary. Ifref is provided, rule_func(ref[key]) is called instead.
Non-Determinism
calcifer.operators.match()Given an expected value, selects the currently scoped node and ensures it matches expected. If the match resultsin a new node definition, the partial is updated accordingly.
For non-matches, returns a monadic zero (e.g. if we’re building a list of policies, this would collapse from[partial] to [])
calcifer.operators.require_value()Returns an mzero (empty list, e.g.) if the provided node is missing a value
16 Chapter 3. Release History
calcifer, Release 0.3.0
Examples
>>> select("/does/not/exist") >> require_value[]
calcifer.operators.forbid_value()Returns an mzero (empty list, e.g.) if the provided node is missing a value
For instance: select(“/does/not/exist”) >> forbid_value
returns []
calcifer.operators.permit_values()Given a list of allowed values, matches the current partial against each, forking the non-deterministic computa-tion.
calcifer.operators.fail()
Error-Handling
calcifer.operators.attempt(*rules)Keeping track of the value and partial it receives, if the result of *rules on the partial is mzero, then attemptreturns unit( (initial_value, initial_policy) ) otherwise, attempt returns the result of the rules.
calcifer.operators.trace()Collates the current scope, the current node’s value, and the current policy context and returns it as a dict
calcifer.operators.unless_errors()
Context Annotation
calcifer.operators.push_context()Add an additional context to the stack for the partial
calcifer.operators.pop_context()Pop the partial’s context stack, returning whatever value it was called with.
calcifer.operators.wrap_context()Run some operator inside some context
Release History
Next Release
• Implement greatness.
Contributing
If you want to help make this project better you are officially an awesome person. Any and all contributions, whetherit’s patches, documentation, or bug reports, are very much welcome and appreciated.
Pull requests or Github issues are always welcome. If you want to contribute a patch please do the following.
1. Fork this repo and create a new branch
2. Do work
3.4. Documentation 17
calcifer, Release 0.3.0
3. Add tests for your work (Mandatory)
4. Submit a pull request
5. Wait for Coveralls and Travis-CI to run through your PR
6. It’ll be code reviewed and merged
As a note, code without sufficient tests will not be merged.
Indices and tables
• genindex
• modindex
• search
18 Chapter 3. Release History
Index
Aadd_error() (calcifer.contexts.Context method), 14append_value() (calcifer.contexts.Context method), 14append_value() (in module calcifer.operators), 15attempt() (in module calcifer.operators), 17
Ccheck() (in module calcifer.operators), 16children() (calcifer.contexts.Context method), 14children() (in module calcifer.operators), 15collect() (in module calcifer.operators), 16Context (class in calcifer.contexts), 14
Ddefine_as() (in module calcifer.operators), 15
Eeach() (calcifer.contexts.Context method), 14each() (in module calcifer.operators), 16err() (calcifer.contexts.Context method), 14error_ctx() (calcifer.contexts.Context method), 14
Ffail() (in module calcifer.operators), 17fail_early() (calcifer.contexts.Context method), 14forbid() (calcifer.contexts.Context method), 14forbid_value() (in module calcifer.operators), 17
Gget_node() (in module calcifer.operators), 15get_value() (in module calcifer.operators), 15
Llast_error (calcifer.contexts.Context attribute), 14
Mmatch() (in module calcifer.operators), 16
Oor_error() (calcifer.contexts.Context method), 14
Ppermit_values() (in module calcifer.operators), 17policies() (in module calcifer.operators), 16pop_context() (in module calcifer.operators), 17push_context() (in module calcifer.operators), 17
Rregarding() (in module calcifer.operators), 16require() (calcifer.contexts.Context method), 14require_value() (in module calcifer.operators), 16
Sscope() (in module calcifer.operators), 15select() (in module calcifer.operators), 15set_value() (calcifer.contexts.Context method), 15set_value() (in module calcifer.operators), 15
Ttrace() (in module calcifer.operators), 17
Uunit() (in module calcifer.operators), 16unit_value() (in module calcifer.operators), 16unless_errors() (in module calcifer.operators), 17
Wwhitelist_values() (calcifer.contexts.Context method), 15wrap_context() (in module calcifer.operators), 17
19