mongodb and rest apis a match made in heaven … · mongodb & rest apis a match made in heaven....

Good MorningPercona Live Amsterdam 2015

Who Am Ifighting back against impostor syndrome

Nicola IarocciCo-founder and CTO at CIR 2000

Nicola IarocciMongoDB Master

Nicola IarocciOpen Source junkie

Eve • Cerberus • Events • Flask-Sentinel • Eve.NET • Etc.

Nicola IarocciConsultant

Mongo • RESTful Services • Python • My Open Source Projects

Nicola IarocciCoderDojo

Coding Clubs for Kids

MongoDB & REST APIsA Match Made in Heaven

Agenda

Agenda1. Our use case for a RESTful API

Agenda2. What is a RESTful API and why we need it

Agenda3. Why MongoDB is a good match for RESTful Services

Agenda4. Build and run a MongoDB RESTful Service from scratch, live on stage.

Agenda5. Stories from the field (if there’s any time left, which I doubt)

The Casefor RESTful Web APIs

Amica 10invoicing & accounting for italian small businesses

http://gestionaleamica.com/

your old school desktop app

http://gestionaleamica.com/

what we started with

Client

LAN/SQL

Database

DesktopApplication

GoalA remote service that client apps can leverage to stay in sync withc each other

What we need #1Must be accessible by any kind of client technology

What we need #2Abstract the data access layer

so we can update/replace the engine at any time with no impact on clients

What we need #3An appropriate data storage engine

What we need #4Easily (re)deployable and scalable multi-micro-service architecure

Where we want to go

Clients

“Cloud”

Database

RESTful Web API

APIiOS

Android

Website

Desktop Client

? ?

Constraints• minimum viable product first

• add features over time

• frequent database schema updates

• easily scalable

• avoid downtime as much as possible

• cater upfront for a microservices architecture

RESTSo What Is REST All About?

REST is not a standard

REST is not a protocol

REST is an architectural stylefor networked applications

Defines a set of simple principlesloosely followed by most API implementations

“resource”the source of a specific information

A web page is not a resourcerather the representation of a resource

“global permanent identifier”every resource is uniquely identified. Think a HTTP URI.

#3 standard interface

used to exchange representations of resources (think the HTTP protocol)

“a set of constraints”separation of concerns, stateless, cacheability, layered system, uniform interface, etc.

Web is built on RESTand it is meant to be consumed by humans

RESTful APIs are built on RESTand are meant to be consumed by machines

Representational State Transfer (REST)by Roy Thomas Fielding

http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm

http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm

Goals #1 and #2 are metREST layer allows all kinds of client technologies and abstracts the data away

MongoDB and RESTOr why we picked MongoDB for our REST API

JSON transportMost REST services and clients produce and consume use JSON

JSON-style data storeMongoDB stores data as Binary JSON

JSON & RESTful API

JSONaccepted media type

Client

JSON(BSON)

Mongo

GET

maybe we can push directly to client?

JSON & RESTful API

JSONaccepted media type

Client

JSON(BSON)

Mongo

JSONsubset of python dict

(kinda)

API

GET

almost.

JSON & RESTful API

JSONobjects

Client

JSON(BSON)

Mongo

JSON/dictmaps to python dict

(validation layer)

API

POST

also works when sending data the database

Similarity with RDBMSmakes NoSQL easy to grasp (even for a sql head like me)

TerminologyRDBMS Mongo

Database Database

Table Collection

Rows(s) JSON Document

Index Index

Join Embedding & Linking

What about Queries?Queries in MongoDB are represented as JSON-style objects

db.things.find({x: 3, y: "foo”});

Filtering and Sorting

nativeMongoquery syntax

Client

JSON(BSON)

Mongo

(very) thin parsing

& validation layer

API

Expose the native MongoDB syntax?

?where={“x”: 3, “y”: “foo”}

JSON all along the pipelinemapping to and from the database feels more natural

No need for ORMNo need to map objects to JSON and vice-versa (win!)

schema-lessdynamic documents allow for painless evolution

REST is statelessMongoDB lacks transactions

Ideal API SurfaceMongo collection maps to API resource endpoint

api.example.com/contacts

Maps to a Mongo collection

Ideal API SurfaceMongo document maps to a API document endpoint

api.example.com/contacts/4f46445fc88e201858000000

Maps to a collection ObjectID

Goal #3 is metAn appropriate data storage engine: MongoDB

Eve REST API for Humans™

Free and Open Source Powered by MongoDB and Good Intentions

eve

http://python-eve.org

Philosopyeffortlessly build and deploy highly customizable, fully featured

RESTful Web Services

Quickstart

install$ pip install eve

run.pyfrom eve import Eveapp = Eve()

if __name__ == '__main__':app.run()

settings.py# just a couple API endpoints with no custom schema or rules.

DOMAIN = {‘people’: {}‘works’: {}

}

launch$ python run.py* Running on http://127.0.0.1:5000/

enjoy

HATEOAS AT WORK HERE

$ curl http://localhost:5000/people

{"_items": [],"_links": {"self": {"href": "people", "title": "people"},"parent": {“href": "/", "title": "home"},

},"_meta": {"max_results": 25,

"total": 0, "page": 1 }}

enjoy

CLIENTS CAN EXPLORE THE API PROGRAMMATICALLY


{"_items": [],"_links": {"self": {"href": "people", "title": "people"},"parent": {“href": "/", "title": “home”}


"total": 0, "page": 1}}




"total": 0, "page": 1 }}

enjoy

AND EVENTUALLY FILL THE UI

enjoy$ curl http://localhost:5000/people



"total": 0, "page": 1 }}

PAGINATION DATA

enjoy

EMTPY RESOURCE AS WE DID NOT

CONNECT A DATASOURCE




"total": 0, "page": 1 }}

settings.py# let’s connect to a mongo instance

MONGO_HOST = 'localhost'MONGO_PORT = 27017MONGO_USERNAME = 'user'MONGO_PASSWORD = 'user'MONGO_DBNAME = 'percona'

settings.py# let’s also add a few validation rulesDOMAIN['people']['schema'] = { 'name': {'type': 'string’,'maxlength': 50,'unique': True},

'email': {'type': 'string','regex': '^\S+@\S+$'},

'location': {'type': 'dict','schema': {'address': {'type': 'string'},'city': {'type': 'string’}}},

'born': {'type': 'datetime'}}

UNIQUE STRING, MAX LENGTH 50

settings.py# let’s also add a few validation rulesDOMAIN['people']['schema'] = { 'name': {'type': 'string','maxlength': 50,'unique': True},




ONLY ACCEPT VALID EMAILS





THIS REGEX SUCKS!DON’T USE

IN PRODUCTION

settings.py# let’s also add a few validation rulesDOMAIN['people']['schema'] = { 'name': {'type': 'string','maxlength': 50,'unique': True},




SUBDOCUMENT WITH 2 STRING FIELDS





ONLY ACCEPT DATETIME VALUES

settings.py# allow write access to API endpoints# (default is [‘GET’] for both settings)

# /peopleRESOURCE_METHODS = ['GET','POST']

# /people/<id>ITEM_METHODS = ['GET','PATCH','PUT','DELETE']

ADD/CREATE ONE OR MORE ITEMS




EDIT ITEM




REPLACE ITEM




YOU GUESSED IT

settings.py

CLIENT UI

# a few optional config options

DOMAIN['people'].update({'item_title': 'person','cache_control':'max-age=10,must-revalidate','cache_expires': 10,'additional_lookup': {'url’: 'regex("[\w]+")','field’: 'name'}

}}

)

settings.py

CLIENT CACHE OPTIONS



}}

)

settings.py

ALTERNATE ENDPOINT



}}

)

Features OverviewWe are going to focus on MongoDB power-ups

full range of CRUD operationsCreate/POST

Read/GET Update/PATCH and Replace/PUT

Delete/DELETE

filters, mongo style?where={“name”: “john”}

filters, the python way?where=name==john

sorting?sort=city,-name

SORT BY CITY, THEN NAME DESCENDING

projections?projection={"avatar": 0}

RETURN ALL FIELDS BUT ‘AVATAR’

projections?projection={"lastname": 1}

ONLY RETURN ‘LASTNAME’

pagination?max_results=20&page=2

MAX 20 RESULTS PER PAGE; PAGE 2

GeoJSONsupport and validation for all GeoJSON types

Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon, GeometricalCollection

document embeddingjoins

standard request$ curl example.com/works/<id>

{ "title": "Book Title", "description": "book description", "author": “52da465a5610320002660f94"

}RAW FOREIGN KEY

(DEFAULT)

request an embedded document$ curl example.com/works/<id>?embedded={“author”: 1}

{ "title": "Book Title", "description": "book description", "author": {

“firstname”: “Mark”, “lastname”: “Green”,

} }

REQUEST EMBEDDED AUTHOR

embedded document$ curl example.com/works/<id>?embedded={“author”: 1}

{ "title": "Book Title", "description": "book description", "author": {

“firstname”: “Mark”, “lastname”: “Green”,

} }

# embedding is configurable on per-field basis and# can be pre-set by API maintainer

EMBEDDED DOCUMENT

bulk insertsinsert multiple documents with a single request

request$ curl -d ‘

[{

"firstname": "barack", "lastname": “obama"

}, {

"firstname": "mitt", "lastname": “romney”

}]' -H 'Content-Type: application/json’ <url>

response[ { "_status": "OK", "_updated": "Thu, 22 Nov 2012 15:22:27 GMT", "_id": "50ae43339fa12500024def5b", "_etag": "749093d334ebd05cf7f2b7dbfb7868605578db2c" "_links": {"self": {"href": “<url>”, "title": "person"}} }, { "_status": "OK", "_updated": "Thu, 22 Nov 2012 15:22:27 GMT", "_id": "50ae43339fa12500024def5c", "_etag": "62d356f623c7d9dc864ffa5facc47dced4ba6907" "_links": {"self": {"href": “<url>", "title": "person"}} }] COHERENCE MODE OFF: ONLY META FIELDS ARE RETURNED

response[ { "_status": "OK", "_updated": "Thu, 22 Nov 2012 15:22:27 GMT", "_id": "50ae43339fa12500024def5b", "_etag": "749093d334ebd05cf7f2b7dbfb7868605578db2c" "_links": {"self": {"href": “<url>”, "title": “person”}}, "firstname": "barack", "lastname": "obama",

}, { "_status": "OK", "_updated": "Thu, 22 Nov 2012 15:22:27 GMT", "_id": "50ae43339fa12500024def5c", "_etag": "62d356f623c7d9dc864ffa5facc47dced4ba6907" "_links": {"self": {"href": “<url>", "title": "person"}} "firstname": "mitt", "lastname": "romney", }]

COHERENCE MODE ON: ALL FIELDS RETURNED

document versioning?version=3

?version=all?version=diffs

soft deletespreserve deleted documents and retrieve them with a simple

?show_deleted

file storagefiles are stored in GridFS by default; customizable for S3, file system, etc.

data validationpowerful and extensible data validation powered by Cerberus

rich validation grammarreferentrial integrity / unique values / defaults / regex / etc.

custom data typescreate your own data types to validate against

custom validation logicextended the validation system to cater for specific use cases

multi databaseserve endpoints and/or users from dedicated mongos

index maintenancedefine sets of resource indexes to be (re)created at launch

supports sparse, geo2d and background indexes

event hooksplug custom actions in the request/response cycle


def percona(resource, response):documents = response['_items'] for document in documents: document['percona'] = 'is so cool!'

if __name__ == '__main__':app.on_fetched_resource += percona_liveapp.run()

CALLBACK FUNCTION




LOOP ON ALL DOCUMENTS




INJIECT FIELD




ATTACH CALLBACK TO EVENT HOOK

rate limitingpowered

settings.py# Rate limit on GET requests:

RATE_LIMIT_GET = (1, 60)

ONE REQUEST PER MINUTE (CLIENT)

$ curl -i <url>

HTTP/1.1 200 OKX-RateLimit-Limit: 1X-RateLimit-Remaining: 0X-RateLimit-Reset: 1390486659

rate limited request

CURRENT LIMIT

$ curl -i <url>



REMAINING

$ curl -i <url>



TIME TO RESET

$ curl -i <url>

HTTP/1.1 429 TOO MANY REQUESTS


OUCH!

operations loglog all operations and eventually expose a dedicated endpoint

HATEOASHypermedia As Engine Of The Application State

XML support$ curl -H ”Accept: application/xml” -i http://example.com/people

<resource href="people" title="people" ><link rel="parent" href="/" title="home" /><resource href="people/55fc149138345b0880f07e3d" title="person" >

<_created>Fri, 18 Sep 2015 13:41:37 GMT</_created> <_etag>5d057712ce792ebb4100b96aa98bfe9b6693c07b</_etag> <_id>55fc149138345b0880f07e3d</_id> <_updated>Fri, 18 Sep 2015 13:41:37 GMT</_updated>

<email>[email protected]</email> <name>john</name></resource>

</resource>

conditional requestsallow clients to only request non-cached content

If-Modified-SinceIf-Modified-Since: Wed, 05 Dec 2012 09:53:07 GMT

ONLY RETURN DOCUMENT IF MODIFIED SINCE

If-None-MatchIf-None-Match:1234567890123456789012345678901234567890

>

ONLY RETURN DOCUMENT ETAG CHANGED

data integrity and concurrencyno overwriting documents with obsolete versions

missing ETag# fails, as there is no ETag included with request

$ curl \-X PATCH \-i http://example.com/people/521d6840c437dc0002d1203c \-H "Content-Type: application/json" \-d '{"name": “ronald"}'

HTTP/1.1 403 FORBIDDEN NO ETAG

REJECTED

ETag mismatch# fails, as ETag does not match with server

$ curl \-X PATCH \-i http://example.com/people/521d6840c437dc0002d1203c \-H "If-Match: 1234567890123456789012345678901234567890" \-H "Content-Type: application/json” \-d '{"firstname": "ronald"}'

HTTP/1.1 412 PRECONDITION FAILED

ETAG MISMATCH

REJECTED

valid ETag# success at last! ETag matches with server

$ curl \-X PATCH \-i http://example.com/people/50adfa4038345b1049c88a37 \-H "If-Match: 80b81f314712932a4d4ea75ab0b76a4eea613012" \-H "Content-Type: application/json" \-d '{"firstname": "ronald"}'

HTTP/1.1 200 OK

# Like most of features, ETags can be disabled.

ETAG MATCH

ACCEPTED

custom data layersbuild your own data layer

authentication and authorizationbasic / token / hmac / BYO / OAuth2 / you name it

and (a lot) moreCORS, cache control, API versioning, JOSNP, Etc.

vibrant community90+ contributors / 350+ forks / 2500+ github stargazers

Eve-Docsautomatic documentation for Eve APIs in both HTML and JSON

CHARLES FLYNN

Eve-Docs

Eve-ElasticElasticsearch data layer for your Eve-powered API

PETR JASEK

Eve-SQLAlchemySQL data layer for Eve-powered APIs

PETR JASEK

Eve-Mongoengineenables mongoengines data models to be used as Eve schema

STANISLAV HELLER

Eve.NETHTTP and REST client for Eve-powered APIs

PETR JASEK

Eve-OAuth2leverage Flask-Sentinel to protect your API endpoints with OAuth2

THOMAS SILEO

REST Layer“golang REST API framework heavily inspired by Python Eve”

THOMAS SILEO

Goal # 4 achievedeasy to setup, launch and scale up; also a good fit for microservices infrastracture

A look back to initial draft

Clients

“Cloud”

Database

RESTful Web API

APIiOS

Android

Website

Desktop Client

? ?

Clients

MultipleMongoDBs

Database

Adameve instances

APIiOS

Android

Website

Desktop Client

what we have in production

stories from the trenches#1. when too much (magic) is too much #2. sometimes you don’t want to debug

#3. so how do I login into this thing?

Take Aways

Enhance MongoDBwith powerful features on top of native engine

validation, document embedding (joins), referential integrity, document versioning, transformations, rate limiting, etc.

Consider the REST layeras an ideal data access layer

the story of pymongo 3.0 breaking changes mongo or sql or elastic or …

Consider the REST layeras an ideal data access layer

the story of Adam dashboards

Consider Microservicesleverage Eve features create a network of isolated yet standardized services

each service has a dedicated role

runs as an eve instance, with its own configuration has its own database(s)

callbacks route traffic between services

Clients

User-reservedMongoDBs

eve-multidb

Data

Autheve-oauth2

(flask-sentinel)

API

iOS

Android

Website

Desktop Client

Adam 1

Adameve instance

Redisauth tokens,rate limiting

Auth/UsersMongoDB

Clients

service-reserved

MongoDBs

Data

Autheve-oauth2,flask-sentinel

API

iOS

Android

Website

Desktop Client

Adam 2

Adameve instance

Redisauth tokens,rate limiting

Serviceseve instances

Auth/UsersMongoDB

very simplified

thanksnicolaiarocci

python-eve.orgeve

mongodb and rest apis a match made in heaven … · mongodb & rest apis a match made in heaven....

Documents