haddock documentation - read the docs 1 going fishing, or - an introduction a look at haddock,...

Haddock DocumentationRelease 0.6.0

HawkOwl

December 28, 2013

Contents

1 Going Fishing, or - An Introduction 31.1 Introduction - Getting Off The Ground . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Introduction - Adding Global State with Service Classes . . . . . . . . . . . . . . . . . . . . . . . . 61.3 Introduction - Parameter Checking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.4 Introduction - Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2 Specifications 152.1 Haddock API Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

i

Haddock Documentation, Release 0.6.0

Haddock is a framework for easily creating APIs. It uses Python and Twisted, and supports both CPython 2.7 andPyPy.

Haddock revolves around versions - it is designed so that you can write code for new versions of your API withoutdisturbing old ones. You simply expand the scope of the unchanged methods, and copy a reference into your newversion.

You can get the MIT-licensed code on GitHub, or download it from PyPI.

Contents 1

https://github.com/hawkowl/haddock

https://pypi.python.org/pypi/haddock


2 Contents

CHAPTER 1

Going Fishing, or - An Introduction

A look at Haddock, starting from the ground up.

1.1 Introduction - Getting Off The Ground

Haddock was made to help you create simple APIs that divide cleanly over versions, with minimal fuss - Haddocktakes care of routing, assembly, parameter checking and (optionally) authentication for you. All you have to do isprovide your business logic.

1.1.1 Installing

To install, you simply need to run:

pip install haddock

This’ll install Haddock and all its dependencies.

1.1.2 A Very Simple Example

In this introduction, we will create a Planet Information API. We will create something that will allow us to query it,and will return some information about planets. So, first, let’s define our API.

Simple API Definition

Put the following in planets.json:

{"metadata": {

"name": "planetinfo","friendlyName": "Planet Information","versions": [1]

},"api": [

{"name": "yearlength",

3


"friendlyName": "Year Length of Planets","endpoint": "yearlength","getProcessors": [

{"versions": [1],"requiredParams": ["name"]

}]

}]

}

Now, we have made a metadata section that gives three things:

• The ‘name’ of our API.

• The ‘friendly name’ (human-readable) name of our API.

• A list of versions that our API has (right now, just 1).

We then define an api section, which is a list of our different APIs. We have defined only one here, and we have saidthat:

• It has a name of ‘yearlength’.

• It has a human-readable name of ‘Year Length of Planets’.

• It has an endpoint of ‘yearlength’. Haddock structures APIs as /<VERSION>/<ENDPOINT>, so this meansthat a v1 of it will be at /v1/yearlength.

There is also then getProcessors - a list of processors. A processor in Haddock is the code that actually does theheavy lifting. This one here only has two things - a list of versions that this processor applies to (in this case, just 1),and a list of required parameters (just the one, name).

Using this API description, we can figure out that our future API will be at /v1/yearlength and require a singleparameter - the name of the planet.

Now, lets make the processor behind it.

Simple Python Implementation

Put the following in planets.py:

import jsonfrom haddock.api import API

class PlanetAPI(object):class v1(object):

def yearlength_GET(self, request, params):pass

APIDescription = json.load(open("planets.json"))myAPI = API(PlanetAPI, APIDescription)myAPI.getApp().run("127.0.0.1", 8094)

This example can be this brief because Haddock takes care of nearly everything else.

So, let’s break it down.

1. First we import API from haddock.api - this is what takes care of creating your API from the config.

2. We then create a PlanetAPI class, and make a subclass called v1. This corresponds to version 1 of your API.

4 Chapter 1. Going Fishing, or - An Introduction


3. We then create a method called yearlength_GET. This is done in a form of <NAME>_<METHOD>. It hasthree parameters - self (this is special, we’ll get to it later), request (the Twisted.Web Request for the APIcall) and params (rather than have to parse them yourself, Haddock does this for you).

Currently, yearlength_GET does nothing, so lets fill in some basic functionality - for brevity, we’ll only supportEarth and Pluto.

def yearlength_GET(self, request, params):planetName = params["name"].lower()if planetName == "earth":

return {"seconds": 31536000}elif planetName == "pluto":

return {"seconds": 7816176000}

As you can see, we access params, which is a dict of all the things given to you in the API call. This is sorted out byHaddock, according to your API description - it makes sure that all required parameters are there, and throws an errorif it is not.

We then return a dict with our result. You can do this - Haddock will JSONise it automatically for you.

Running

Let’s try and run it!

python planets.py

This should print something out like this:

2013-12-27 11:46:21+0800 [-] Log opened.2013-12-27 11:46:21+0800 [-] Site starting on 80942013-12-27 11:46:21+0800 [-] Starting factory <twisted.web.server.Site instance at 0x192d998>

This says that the Twisted.Web server behind Haddock has started up, and is on the port we asked it to.

Now, go to http://localhost:8094/v1/yearlength?name=earth in your web browser. You should getthe following back:

{"status": "success", "data": {"seconds": 31536000}}

Now try http://localhost:8094/v1/yearlength - that is, without specifying the name.

{"status": "fail", "data": "Missing request parameters: ’name’"}

As you can see, it fails if we don’t pass in what we want.

API Documentation

Tired of having to document your APIs? Well, with Haddock, you can provide basic API documentation automatically.Simply go back to your planets.json and make your metadata look like this:

"metadata": {"name": "planetinfo","friendlyName": "Planet Information","versions": [1],"apiInfo": true

},

Then restart your planets.py and browse to http://localhost:8094/v1/apiInfo. You will get a listof what APIs you have, and some request and response params. It is a bit lacking right now - you’ll only have name

1.1. Introduction - Getting Off The Ground 5


in Request Arguments with no other documentation, but you’ll find out how to add descriptions and types to thisdocumentation in the more advanced articles.

1.1.3 Going Further

The next article is about adding global state to your Haddock API.

1.2 Introduction - Adding Global State with Service Classes

Most APIs are only useful with some form of global state - say, a database, or in-memory record of values. Haddockdoes this through Service Classes - a separate class where you put all of your global state, without affecting any ofyour API implementation.

1.2.1 A Basic Service Class

Using the same planets.json as in the Introduction, modify planets.py to look like this:

import jsonfrom haddock.api import API, DefaultServiceClass

class PlanetServiceClass(DefaultServiceClass):def __init__(self):

self.yearLength = {"earth": {"seconds": 31536000},"pluto": {"seconds": 7816176000}

}


def yearlength_GET(self, request, params):planetName = params["name"].lower()return self.yearLength.get(planetName)

APIDescription = json.load(open("planets.json"))myAPI = API(PlanetAPI, APIDescription, serviceClass=PlanetServiceClass())myAPI.getApp().run("127.0.0.1", 8094)

So, let’s have a look at what’s different here - we now have a service class.

• We import DefaultServiceClass from haddock.api and subclass it, adding things into it.

• We then pass in an instance (not a reference to the class, an instance) of our custom service class.

• In yearlength_GET, we then access the yearLength dict defined in PlanetServiceClass, usingself.

Additionally, the self of all of your processors is automatically set to your service class. This means that you can doglobal state fairly easily - but it does mean that every version of your API accesses the same service class. Haddockwon’t be able to help you too much with global state differences, such as database schemas or the like, but you canisolate business logic changes in different API versions.



1.2.2 Subclassing Is Optional

The subclassing is done to provide some boilerplate Klein routes which allow the automatic API documentation andCORS support to work. You can create your own blank service class, and everything except those two things will workperfectly. This is also test covered.

1.2.3 Going Further

Next, we’ll have a look at Haddock’s parameter checking.

1.3 Introduction - Parameter Checking

Haddock contains support for checking parameters given to your API. It supports checking for required and optionalparams, as well as restricting the content of each to certain values.

1.3.1 Required & Optional Params

This will show you how to use the required and optional params.

API Documentation

Using the planets.json example from before, let’s add a new API - this time, for getting the distance from thesun.

{"metadata": {

"name": "planetinfo","friendlyName": "Planet Information","versions": [1],"apiInfo": true

},"api": [

{"name": "sundistance","friendlyName": "Distance from the Sun for Planets","endpoint": "sundistance","getProcessors": [

{"versions": [1],"requiredParams": ["name"],"optionalParams": ["useImperial"],"returnParams": ["distance"],"optionalReturnParams": ["unit"]

}]

},{

"name": "yearlength","friendlyName": "Year Length of Planets","endpoint": "yearlength","getProcessors": [

{

1.3. Introduction - Parameter Checking 7


"versions": [1],"requiredParams": ["name"]

}]

}]

}

So now we’ve added a sundistance API, with a single processor. It has the following restrictions:

• For the request, name must be provided and useImperial may be.

• For the response, distance must be provided, and unit may be.

Python Implementation

So, lets add the code to do this into planets.py:

import jsonfrom haddock.api import API, DefaultServiceClass

class PlanetServiceClass(DefaultServiceClass):def __init__(self):

self.yearLength = {"earth": {"seconds": 31536000},"pluto": {"seconds": 7816176000}

}self.sunDistance = {

"earth": {"smoots": 87906922100, "miles": 92960000},"pluto": {"smoots": 3470664162652, "miles": 3670050000}

}


def yearlength_GET(self, request, params):planetName = params["name"].lower()return self.yearLength.get(planetName)

def sundistance_GET(self, request, params):planetName = params["name"].lower()sunDistance = self.sunDistance.get(planetName)if sunDistance and params.get("useImperial"):

return {"distance": sunDistance["miles"]}else:

return {"distance": sunDistance["smoots"], "unit": "smoots"}

APIDescription = json.load(open("planets.json"))myAPI = API(PlanetAPI, APIDescription, serviceClass=PlanetServiceClass())myAPI.getApp().run("127.0.0.1", 8094)

We now have an implementation that will return the distance if useImperial is some truthy value, and distance andunit otherwise. The API will not force you to specify useImperial as an API consumer, nor unit as a developer.Please note that not being specified will make it not appear in the dict, so using params.get() is a must!

Try it out at http://localhost:8094/v1/sundistance?name=earth.



1.3.2 Restricting Parameter Values

Haddock will also allow you to restrict the values of the parameters. Let’s change sundistance to look like thefollowing in planets.json:


{"versions": [1],"requiredParams": [

{"param": "name","paramOptions": ["earth", "pluto"]

}],"optionalParams": [

{"param": "useImperial","paramOptions": ["yes", "no"]

}],"returnParams": ["distance"],"optionalReturnParams": ["unit"]

}]

}

So, instead of giving requiredParams or optionalParams a list of strings, we are giving a list of dicts. Eachdict must have a param value, the rest are optional. We also specify a paramOptions, which is a list - it can takeeither dicts or strings, but dicts are only useful when documenting your API through Haddock. Using dicts with it willbe covered later, but we only need strings for now.

Since we have only implemented Earth and Pluto, we can now bracket our inputs to those values. Restartplanets.py and try going to http://localhost:8094/v1/sundistance?name=jupiter. You shouldget something like the following:

{"status": "fail", "data": "’jupiter’ isn’t part of [\"earth\", \"pluto\"] in name"}

If an API consumer tries to give an incorrect value, it will respond with an error message - saying that the value givenwas incorrect, what parameter was incorrect, and what the correct answers are.

paramOptions is valid for both request and return params, optional or otherwise.

1.3.3 List Return Format

Haddock also supports checking a list of ‘‘dict‘‘s as return values. It will go through each entry of the list and checkthat the dict contains all of the required values, just like it did above.

Here is a sundistance that will let us do that:


1.3. Introduction - Parameter Checking 9


{"versions": [1],"returnFormat": "list","requiredParams": ["name"],"returnParams": ["distance"],"optionalReturnParams": ["unit"]

}]

}

As you can see, we have added a returnFormat of list.

And now the sundistance_GET implementation:

def sundistance_GET(self, request, params):planetName = params["name"].lower()sunDistance = self.sunDistance.get(planetName)return [

{"distance": sunDistance["miles"], "unit": "miles"},{"distance": sunDistance["smoots"], "unit": "smoots"}

]

Then, if you restart planets.py and go to http://localhost:8094/v1/sundistance?name=earth,you will get the following:

{"status": "success", "data": [{"distance": 92960000, "unit": "miles"}, {"distance": 87906922100, "unit": "smoots"}]}

1.3.4 Going Further

Next, we’ll have a look at implementing authentication into your Haddock API.

1.4 Introduction - Authentication

Some APIs may need authentication before accessing - for example, if you are writing a service rather than just apublic data API. Haddock allows you to either do the authentication yourself, or hook in a “Shared Secret Source”which will request a user’s shared secret from your backend.

1.4.1 Using a Shared Secret Source

For this example, we will be using a new API Description.

API Description

Put this in authapi.json:

{"metadata": {

"name": "authapi","friendlyName": "An Authenticated API","versions": [1],"apiInfo": true

},"api": [



{"name": "supersecretdata","friendlyName": "Super secret data endpoint!!!!","endpoint": "supersecretdata","requiresAuthentication": true,"getProcessors": [

{"versions": [1]

}]

}]

}

The new part of this is requiresAuthentication in our single API, which is now set to true.

Python Implementation

Put this into authapi.py:

import jsonfrom haddock.api import API, DefaultServiceClassfrom haddock import auth

class AuthAPIServiceClass(DefaultServiceClass):def __init__(self):

users = [{"username": "squirrel","canonicalUsername": "[email protected]","password": "secret"

}]self.auth = auth.DefaultHaddockAuthenticator(

auth.InMemoryStringSharedSecretSource(users))

class AuthAPI(object):class v1(object):

def supersecretdata_GET(self, request, params):return "Logged in as %s" % (params.get("haddockAuth"),)

APIDescription = json.load(open("authapi.json"))myAPI = API(AuthAPI, APIDescription, serviceClass=AuthAPIServiceClass())myAPI.getApp().run("127.0.0.1", 8094)

In our implementation, we now import haddock.auth, and use two portions of it when creating our ser-vice class. We set self.auth to be a new instance of auth.DefaultHaddockAuthenticator, with aauth.InMemoryStringSharedSecretSource as its only argument, with that taking a list of users.

How Authentication in Haddock Works

Before your API method is called, Haddock checks the API description, looking for requiresAuthenticationon the endpoint. If it’s found, then it will look in the HTTP Authorized header for Basic. It will then callauth_usernameAndPassword on the Haddock authenticator, which will then check it and decide whether or notto allow the request.

Since this is boilerplate, Haddock abstracts it into the DefaultHaddockAuthenticator, which takes aSharedSecretSource. Currently, the source requires only one function - getUserDetails. This is called,

1.4. Introduction - Authentication 11


asking for the details of a user, which the authenticator will then check against the request. If it is successful, theauthenticator will return either the user’s canonical username or their username.

Canonical usernames are returned by the Haddock authenticator when possible, which are then placed in ahaddockAuth param. Your API method will get this, and know that this is the user which has been successfullyauthenticated.

The InMemoryStringSharedSecretSource Source

The InMemoryStringSharedSecretSource takes a list of users, which consists of a username, passwordand optionally a canonicalUsername.

Running It

Now, since we have got our authentication-capable API, let’s test it. Try running curlhttp://localhost:8094/v1/supersecretdata, you should get this back:

{"status": "fail", "data": "Authentication required."}

Haddock is now checking for authentication. Let’s try giving it a username and password, with curlhttp://localhost:8094/v1/supersecretdata -u squirrel:secret:

{"status": "success", "data": "Logged in as [email protected]"}

As you can see, we returned the canonical username in supersecretdata_GET, which [email protected].

1.4.2 Why Canonical Usernames?

Since this is an API, it may have sensitive data behind it, which you want to control access to. Controlling it viaauthentication is only solving part of the problem - you need to make sure that if the shared secret is lost, you canrescind access to it. Since changing passwords is a pain for users, a better solution is to have API specific credentials,and Haddock’s authentication is made to support that.

When giving out access to an API, you should create a set of API specific credentials - that is, a randomly generatedusername and password which is then used against your API, and can be revoked if required. Simply store the randomcreds, and a link to the user’s real (canonical) username, and give that to the authenticator.

1.4.3 Implementing Your Own Shared Secret Source

This is taken from Tomato Salad, a project using Haddock.

class tsSharedSecretSource(object):def __init__(self, db):

self.db = db

def getUserDetails(self, username):def _continue(result):

if result:res = {}res["username"] = result["APIKeyUsername"]res["canonicalUsername"] = result["userEmail"]res["password"] = result["APIKeyPassword"]return res



raise AuthenticationFailed("Incorrect API key.")

d = self.db.fetchAPIKey(username)d.addCallback(_continue)return d

class tsServiceClass(DefaultServiceClass):def __init__(self):

self.db = Database({"connectionString": "sqlite:///tomatosalad.db"})

self.auth = DefaultHaddockAuthenticator(tsSharedSecretSource(self.db))

1.4. Introduction - Authentication 13

CHAPTER 2

Specifications

Haddock has a lot more functionality - optional parameters, specifying specific return or request parameters, authen-tication, and even more to do with automatic API documentation. Browse through the other documentation articles tosee how to use these features.

2.1 Haddock API Description

The Haddock API Description is a standard structure that Haddock uses to build your API. It con-tains information about your project (metadata), your APIs (api), the processors behind those APIs(getProcessors/postProcessors) and parameters that your API takes and responds with (parametersand parameterOptions).

The API Description ends up having two top-level parts - the metadata and the api. They are laid out like this:

{"metadata": {

...},"api": {

...}

}

2.1.1 Metadata

The metadata contains three things:

• name: The computer-friendly name.

• friendlyName: The user-friendly name.

• versions: A list of applicable versions. They don’t have to be 1, 2, or whatever - they’re just used later on inapi. Note that there is one special version - “ROOT”, which moves all of the endpoints to the root (for example,/weather, instead of /v1/weather).

• apiInfo: Whether or not you want automatic API documentation generated.

15


2.1.2 API

The api contains a list of dicts, which are API endpoints. In each API method there is:

• name: The computer-friendly name. This is used in naming your functions later!

• friendlyName: The user-friendly name.

• description: The user-friendly description.

• endpoint: The URL endpoint. For example, it will make a processor for v1 be under “/v1/weather”.

• requiresAuthentication (optional): A boolean that defines whether this API needs authentication. De-fault is false.

• rateLimitNumber (optional): How many times per unit of time that this API may be called by the API key.

• rateLimitTimescale (optional): The timescale that the limit number works off, in seconds. For example,a rateLimitNumber of 10 and a rateLimitTimescale of 60 means that 10 requests can be made in asliding window of 60 seconds.

• getProcessors (optional): A list of processors (see below). These processors respond to a HTTP GET.

• postProcessors (optional): A list of processors (see below). These processors respond to a HTTP POST.

2.1.3 Processors

Processors are the bits of your API that do things. They are made up of dicts, and contain the following fields:

• versions: A list of versions (see metadata) which this endpoint applies to.

• paramsType (optional): Where the params will be - either url (in request.args) or jsonbody (forexample, the body of a HTTP POST). Defaults to url.

• returnFormat (optional): Either dict or list (which means a list of ‘‘dict‘‘s, conforming to theparams below)

• requiredParams (optional): Parameters that the API consumer has to give. This is a list, the contents ofwhich are explained below.

• optionalParams (optional): Parameters that the API consumer can give, if they want to. This is a list, thecontents of which are explained below.

• returnParams (optional): Parameters that your API has to return. This is a list, the contents of which areexplained below.

• optionalReturnParams (optional): Parameters that your API may return. This is a list, the contents ofwhich are explained below.

Please note that if you have set requiredParams, you MUST set every other key that may be given inoptionalParams! Same goes with returnParams.

2.1.4 Parameters

When defining the parameters your API can give/take, you can do it two ways. The first way is just giving a stringcontaining the param key, the second is giving it a more detailed dict. The dict fields are below.

• param: The parameter key.

• description (optional): The user-friendly description of what this parameter is for. This is shown in theAPI documentation.

16 Chapter 2. Specifications


• type (optional): A type that can be displayed in the API documentation. This isn’t enforced.

• example (optional): An example value, for the API documentation.

• paramOptions (optional): If this parameter can only accept a certain number of values, use this to restrict itto them automatically. It’s a list, containing either a string of an acceptable value, or a dict (details below).These are also shown in the API documentation.

2.1.5 Parameter Options

If you want to document your parameter options a bit better than simply giving it values, you can do so by makingparamOptions a list of ‘‘dict‘‘s with the following values:

• data: The acceptable value.

• meaning: The meaning of this value. Used in the API documentation.

2.1.6 Example

For a proper example, see betterAPI.json under haddock/test/. It has nearly every option defined in theresomewhere.

2.1. Haddock API Description 17

haddock documentation - read the docs 1 going fishing, or - an introduction a look at haddock,...

Documents