building node.js applications with database jones

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Database Jones! J.D. Duncan, [email protected]

! Craig Russell, [email protected]

1

1

mailto:[email protected]





J.D.! Senior Software Engineer, MySQL Cluster, at Oracle

! Former Unix Sysadmin & web developer

! MySQL AB (2004) - Sun (2008) - Oracle (2011)

! Projects

! Database Jones, NDB Memcache, mod_ndb

2

2


Craig! Architect, Oracle Corp.

! Java Data Objects JSR-243 Specification Lead

! Projects

! Database Jones connector for MySQL Cluster

! Cluster/J Java Connector for MySQL Cluster

! Secretary, Apache Software Foundation3

3

Why Database Jones?

•Node.JS

• Highly scalable Javascript platform for web services

•MySQL Cluster

• Highly scalable database engine

•Database Jones

• Highly scalable API for database access from Node.JS

4

MySQL Cluster Overview

5

MySQL Cluster Architecture

MySQL C

6

Database Jones

•Fast, easy database API for Node.js

•JavaScript for user API and common operations

•Two adapters currently shipping

• NDB: Native adapter to MySQL Cluster C++ API

• MySQL: Adapter to node-mysql (third party tool)

• Open architecture allows other adapters

•Most operations are asynchronous

7

Data Model

•Two models are supported:

• Operations using table names return plain objects

• {"id": 554, "first_name": "Roy", "last_name": "Raye"}

• Operations using JavaScript constructors return instances of JavaScript classes

8

Object Mapping

•Table mapping defines the relationship between

• Tables and columns

• Objects and fields

• Foreign keys and object relationships

•Table mapping is flexible

• Change field names

• Provide transformations of data

• Database format to JavaScript format

• e.g. Date types, True/False, custom formats

9

Mapped Table Formats

•Classic

• Fields are stored in individual columns

•Serialized

• Objects are serialized into JSON and stored in a JSON column

•Hybrid

• Some fields are stored in their own column

• All other fields are serialized

10

Classic Format

•Most appropriate for existing tables/schema

•Schema is defined outside the application

•Compose complex objects via database joins

11

Serialized

•Most appropriate for “schema-less” designs

•Limit query capability (we’re working on it)

•Limited join capability (we’re working on this too)

12

Hybrid

•Combines best features of serialized and classic

•Some fields are stored in their own column

• object composition via joins

•All other fields are stored serialized

• Object composition via serialization

•Hybrid tables can store arbitrarily complex objects efficiently

13

Database Jones Operations•Metadata (non-transactional)

• List tables

• Get metadata for a table

•CRUD (transactional)

• Insert (save)

• Find by key (includes joins)

• Delete by key

• Update by key

•Query (transactional)

• Complex query criteria

14


Install & Demo

15

Install from Github

✦ git clone http://github.com/mysql/mysql-js

✦ git config --global --add core.symlinks true

github.com/mysql/mysql-js

You may need to allow git to make symlinks:

16

http://github.com/mysql/mysql-js


Components

The user API

DB Service Providers

Network Configuration

A directory of symlinks

Sample Application code

Unified debugging library for C++ and JS

Benchmark

17

Architecture

Jones

Application Code

DB Service Provider

Jones API

Jones SPI

Database

Database Network Protocol

18

Using the DB Service Providers

•MySQL

• npm install mysql

•NDB

• cd jones-ndb

• node configure

19

Sample Twitter-like Application

•In samples/tweet

•5 tables

20

Contents of samples/tweet

4 simple API demos

2 SQL scripts

Main Application(Command Line & HTTP)

Demos of tweet.js

21

tweet.jsUsage: node tweet {options} {command} {command arguments} -a <adapter>: run using the named adapter (default: ndb) -h or --help: print this message -d or --debug: set the debug flag --detail: set the detail debug flag -df <file>: enable debug output from <file> -E or --deployment <name>: use deployment <name> (default: test)

COMMANDS: put user <user_name> << JSON Extra Fields >> get user <user_name> delete user <user_name> post tweet <author> << Message >> get tweet <tweet_id> delete tweet <tweet_id> put follow <user_follower> <user_followed> get followers <user_name> get following <user_name> get tweets-by <user_name> get tweets-at <user_name> get tweets-about <hashtag> get tweets-recent <count> start server <server_port_number> get help 22

Author table: Hybrid data model

CREATE TABLE author (

user_name varchar(20) CHARACTER SET UTF16LE not null,

full_name varchar(250),

tweet_count int unsigned not null default 0,

SPARSE_FIELDS varchar(4000) CHARACTER SET utf8,

PRIMARY KEY(user_name)

) ;

“Catch All” JSON Column

from create_tweet_tables.sql

23

Hybrid Insert into Author

node tweet put user caligula

'{ "full_name": "Gaius Julius Caesar Germanicus" ,

"profile_text": "I am your little boot!" }'

user_name full_name

SPARSE_FIELDS

from demo_populate_data.sh

24

API Essentials

•SessionFactory is a heavyweight master connection

•Session is a lightweight pooled connection for a user

• e.g. for an HTTP request

• a session has a single active transaction

•Session provides APIs for database operations

• Example: use session.find() to find a single row of data

25

API Sample application: find.js

/* This script shows an example find() operation using a table name and primary key, and working with promises.

For a similar example using callbacks rather than promises, see insert.js*/

"use strict";var jones = require("database-jones");

26

find.js: Configure a database connection

/* new ConnectionProperties(adapter, deployment)

The first argument names a database backend, e.g. "ndb", "mysql", etc.

The second argument names a "deployment" defined in a jones_deployments.js file. (A default file can be found two directories up from here). jones_deployments.js is the preferred place to customize the host, username, password, and other parameters of the database connection.*/

var connectionProperties = new jones.ConnectionProperties("mysql", "test");

27

find.js: Process the command line

/* node find.js table_name primary_key_value argv[0] argv[1] argv[2] argv[3] */

if (process.argv.length !== 4) { console.log("Usage: node find <table> <key>\n"); process.exit(1);}

var table_name = process.argv[2], find_key = process.argv[3];

28

find.js: Run the database operation

/* This version of openSession() takes one argument and returns a promise. The argument is the set of connection properties obtained above.

Once the session is open, use it to find an object. find() is a Jones API call that takes a primary key or unique key and, on success, returns *only one object*.*/

jones.openSession(connectionProperties). then(function(session) { return session.find(table_name, find_key); }). then(console.log, console.trace). // log the result or error then(jones.closeAllOpenSessionFactories); // disconnect

29

Many varieties of find()

The first argument can be:

• Table Name

• JavaScript constructor that has been mapped

• Projection describing a structure of related objects

30


The second argument can be:

• String or Number1-part primary key lookup

• { key_field_name : value } Primary key or any unique index

• { key_part_1 : value1, key_part_2: value2 }Multi-part primary key or unique index

31


• The optional third argument can be a callback

• Any extra arguments after the third will be supplied to the callback (for additional context)

• find() always returns a promise (even if you don’t use the promise)

32

insert.js: Callback Style/* This script shows an example persist() operation using a table name and primary key, and working with callbacks. */

function disconnectAndExit(status) { jones.closeAllOpenSessionFactories(function() { process.exit(status); });}

/* handleError() exits if "error" is set, or otherwise simply returns.*/function handleError(error) { if(error) { console.trace(error); disconnectAndExit(1); }}

33

insert.js: Run the operation

/* This version of openSession() takes three arguments: ConnectionProperties A table name, which will be validated upon connecting A callback which will receive (error, session)*/jones.openSession(connectionProperties, table_name, function(err, session) { handleError(err);

/* The callback for persist() only gets one argument */ session.persist(table_name, object, function(err) { handleError(err); console.log("Inserted: ", object); disconnectAndExit(0); });});

34

scan.js: Query

/* This script provides an example of the Jones Query API. In this example, we query the tweet table for posts by a particular author, and apply a sort and limit on the query.*/


if (process.argv.length < 3 ) { console.log("usage: node scan <author> [limit] [order]"); process.exit(1);}

35

scan.js: Query Parameters

// node scan.js <author> [limit] [order]

var connectionProperties = new jones.ConnectionProperties("ndb", "test"), queryTerm = process.argv[2], limit = Number(process.argv[3]) || 20, order = (process.argv[4] == "asc" ? "asc" : "desc");

36

scan.js: Query Operationjones.openSession(connectionProperties). then(function(session) { return session.createQuery("tweet"); }). then(function(query) { /* Here we can define query conditions. For more details see API-documentation/Query */ query.where(query.author_user_name.eq(queryTerm));

/* Then execute the query, using limit & order parameters. */ return query.execute({ "limit" : limit, "order" : order }); }). then(console.log, console.trace). // log the result or error then(jones.closeAllOpenSessionFactories); // disconnect

37

Session Methods

✦find()✦remove()✦persist()

✦update()✦save()✦load()

Metadata Operations

Key Operations

✦listTables()✦getTableMetadata()

Query Operations✦createQuery()

Others✦currentTransaction()✦close()

✦createBatch()✦setPartitionKey()

38

Batch: powerful grouping of “mixed” key operations

batch = session.createBatch();

tag = tags.hash.pop(); // # hashtags while(tag !== undefined) { tagEntry = new HashtagEntry(tag, tweet); batch.persist(tagEntry); tag = tags.hash.pop(); } return batch.execute();

✦batch.find()✦batch.remove()✦batch.persist()

✦batch.update()✦batch.save()✦batch.load()

39

Example from tweet.js/* Insert a tweet. - Start a transaction. - Persist the tweet & get its auto-increment id. - Create & persist #hashtag & @mention records (all in a single batch). - Increment the author's tweet count. - Then commit the transaction. */function InsertTweetOperation(params, data) { [ ... ] session.currentTransaction().begin(); session.persist(tweet). then(function() { return session.find(Author, authorName);}). then(incrementTweetCount). then(createTagEntries). then(commitOnSuccess, rollbackOnError). then(function() {return tweet;}). then(this.setResult). then(this.onComplete, this.onError);

40


Query & Projection APIs

41

Query API•Session is Query factory using asynchronous api

•Query has a filter based on the mapped object

•Filter has comparators• eq, ne, gt, ge, le, lt, between, isNull, isNotNull

•Filter has boolean operations• and, or, not, andNot, orNot

•Query execution is asynchronous

•Filter determines query strategy• Primary/unique key lookup; index scan; table scan

•Properties govern query execution

•Results are given in callback

42

Query Comparators

•Comparators compare properties to parameters

•Query Domain Type property names correspond to Constructor field names (properties)

•Parameters are created by name• qdt.param('date_low')

•Properties are referenced by field name in Constructor• qdt.date_created

•Comparators are properties of qdt properties• qdt.date_created.gt(qdt.param('date_low'));

•Comparators return predicates

43

Query Operators

•Predicates are used as query filters via where function

•Predicates are results of comparators or operators

•Operators combine predicates: and, or, andNot, orNot, not• var predicate1 = qdt.date_created.gt(qdt.param('date_low'));

• var predicate2 = qdt.date_created.lt(qdt.param('date_high'));

• var predicate = predicate1.and(predicate2);

• var predicate = predicate3.andNot(predicate4);

• qdt.where(predicate)

44

Query Execution

•var promise = query.execute(parameters,

callback);

•parameters: regular javascript object

• skip: number of results to skip over

• limit: number of results to return

• order: 'asc' or 'desc'

• user-specified parameters defined by q.param

•callback(err, results)

•results: result[ ], possibly no elements

45

Query Example

session.createQuery(Employee). then(function(query) { return query.where(query.age.gt(50).and(query.salary.lt(50000))). execute({limit: 20})}).

then(function(results) { results.forEach(function(result) { console.log('Name:', result.name, 'age:', result.age); }) }).

then(closeSession, reportError);

46

Projection

•Composition of complex objects by joins

• one to one, one to many, many to one, many to many

•Define relationships (bidirectional) in table mapping

•Choose fields to select in Projection

•Choose relationships to join in Projection

•find() can take a Projection as the first argument

47

Projection API

•Projection constructor defines the domain object

•Fields are added to Projection

•Relationships to other Projections are added to Projection

•Loops are not supported

•Arbitrary nesting depth

48

join.js


/* Constructors for application objects */

function Author() { }

function Tweet() { }

49

join.js: TableMapping for Author

/* TableMappings describe the structure of the data. */

var authorMapping = new jones.TableMapping("author");

authorMapping.applyToClass(Author);

authorMapping.mapSparseFields("SPARSE_FIELDS");

authorMapping.mapOneToMany( { fieldName: "tweets", // field in the Author object target: Tweet, // mapped constructor targetField: "author" // target join field });

50

join.js: TableMapping for Tweet

/* TableMappings describe the structure of the data. */

var tweetMapping = new jones.TableMapping("tweet");tweetMapping.applyToClass(Tweet);

tweetMapping.mapManyToOne( { fieldName: "author", // field in the Tweet object target: Author, // mapped constructor foreignKey: "author_fk" // SQL foreign key relationship });

51

join.js: Projection

/* Projections describe the structure to be returned from find().*/

var tweetProjection = new jones.Projection(Tweet);tweetProjection.addFields(["id","message","date_created"]);

var authorProjection = new jones.Projection(Author);authorProjection.addRelationship("tweets", tweetProjection);authorProjection.addFields(["user_name","full_name"]);

52

join.js: Run the database operation

/* The rest of this example looks like find.js, only using find() with a projection rather than a table name.*/

jones.openSession(new jones.ConnectionProperties("mysql", "test")). then(function(session) { return session.find(authorProjection, find_key); }). then(console.log, console.trace). // log the result or error then(jones.closeAllOpenSessionFactories). // disconnect then(process.exit, console.trace);

53

join.js: Result

$ node join.js nero

{ user_name: 'nero', full_name: 'Lucius Domitius Ahenobarus', tweets: [ { id: 3, message: 'I love to sing!', date_created: Thu Oct 01 2015 16:09:46 GMT-0700 (PDT) }, { id: 4, message: 'I am the best #poet and the best #gladiator!', date_created: Thu Oct 01 2015 16:09:46 GMT-0700 (PDT) } ] }

54


Thank you Code Camp!

✦ git clone http://github.com/mysql/mysql-js

55

