pycon 2010 sqlalchemy tutorial

Install

● Python 2.5 or 2.6 (or pysqlite for 2.4)● SQLAlchemy 0.5

1. easy_install, if needed● wget http://peak.telecommunity.com/dist/ez_setup.py● python ez_setup.py

2. easy_install sqlalchemy==0.5.8

● http://old.utahpython.org/sqla2010/

Michael Bayer

Michael Bayer is a software architect in New York City and is the creator of SQLAlchemy.

http://techspot.zzzeek.org/@zzzeek

ORM 101 – the bad old days

c = db.cursor()sql = "SELECT * FROM users WHERE name = %s"c.execute(sql, (name,))user = User(*c.fetchone())

user.last_login_at = time.time()sql = "UPDATE users SET last_login_at = %s WHERE name = %s"c.execute(sql, (user.last_login_at, name0))c.commit()

ORM 101 – and there was light

session = Session()user = session.query(User).filter(name=name).one()

user.last_login_at = time.time()

session.commit()

The devil is in the details

● Compound WHERE clauses, subqueries, outer joins, sql functions, ...

● Eager/lazy loading● Support for legacy schemas● Inheritance● Conceptual integrity● Setup overhead● Database support

What SQLAlchemy is not

[Demolition photo]

Tough love?

“Disproving the myth of 'the best database layer is the one that makes the database invisible' is a primary philosophy of SA. If you don't want to deal with SQL, then there's little point to using a [relational] database in the first place.”

Technical excellence

● PK: multi-column is fine; mutable is fine; any data type is fine; doesn't have to be named “id”http://blogs.ittoolbox.com/database/soup/archives/primary-keyvil-part-i-7327

● Recognizes all database defaults instead of allowing a few special cases like “created_at”

● Doesn't make up its own query language● No XML● Introspection or define-tables-in-Python● Session/unit-of-work based● Migrations

Supported databases

PostgreSQLMySQLSQLiteFirebirdOracleMSSQLSybaseDB2InformixSAPDBMSAccess

Database dependence: a feature

● Performance– Functions, partial indexes, bitmap indexes,

partitioning, replication, ...● Features

– Views, arrays, recursive joins, full-text searching, ...

See also: http://powerpostgresql.com/Downloads/database_depends_public.sxi

Caching

Beaker integration is an example w/ the 0.6 distribution (currently in beta)

Questions

Today's agenda: fundamentals

Data Mapper vs Active RecordSA FundamentalsMapping basicsQueriesSessions & the identity mapRelationship lifecycleBackrefs

Agenda 2: More ORM details

Multi-object queriesOne-to-oneMany-to-manyRelation queriesEager/lazy loadingTransactionsExtensions and related projects

Agenda 3: Extensions and related projects

MigrateFormAlchemySqlSoupElixirz3c.sqlalchemy

Two ORM patterns

Active RecordData Mapper

Active Record

Data Mapper

SQLAlchemy supports both

● Declarative plugin for common simple situations

● Full data mapper power when you need it

Tables for this tutorial

usersaddressesordersorderitemskeywordsitemkeywords

Tables

users = Table('users', metadata, Column('user_id', Integer, primary_key = True), Column('name', String(40)))

users = Table('users', metadata, autoload=True)

users = Table('users', metadata, autoload=True, Column('name', String(40), default='Jonathan'))

Legacy columns

ack = Table('ACK110030', metadata, Column('ITMNUMBER', Integer, primary_key=True, key='id'), Column('MNFCENTERLC_I', Integer, ForeignKey('MFC43222.id'), key='manufacturing_center_id'), ...)

Setup from scratch

engine = create_engine('sqlite:///:memory:', echo=True)metadata = MetaData()metadata.bind = engineSession = sessionmaker(bind=engine)

from sqlalchemy.ext.declarative import declarative_baseBase = declarative_base()

Quick setup

from tutorial_tables import *create()data()

http://old.utahpython.org/sqla2010/

Table + mapped class together

class Order(Base): __tablename__ = 'orders' order_id = Column(Integer, primary_key=True) user_id = Column(Integer, ForeignKey(users.c.user_id)) description = Column('description', String(50)) isopen = Column(Integer, ColumnDefault(1))

Full data mapper pattern

orders = Table('orders', metadata, Column('order_id', Integer, ...), Column('user_id', Integer, ForeignKey(users.c.user_id)), Column('description', String(50)), Column('isopen', Integer, ColumnDefault(1)),

class Order(object): pass

mapper(Order, orders)

The way we will do mapping

class Order(Base): __table__ = orders

Querying

session = Session()q = session.query(Order)print q

.all

.get

.first, .one

Query modification

q = session.query(Order)

.filter

.filter_by

.order_by [desc, asc]

.limit, .offset

Operators

== >= <= > <

~ | &not_ or_ and_

in_

between like startswith endswith

Some examples

q = session.query(Order)

q.filter_by(user_id=7).order_by(Order.isopen).first()q.filter(Order.description.like('order%')).all()q.filter((Order.user_id==7) | (Order.isopen==1)).all()q.filter(or_(Order.user_id==7, Order.isopen==1)).all()

Slicing: limit/offset sugar

q = session.query(Order)q.limit(1).offset(2)

q[2:3]

Questions

Exercise

● Map the orderitems table to an OrderItem class

● Get a list of all OrderItems– Where they belong to order #3– ... or the item name is “item 1”– ... ordered by item name

(Now would be a good time to look at tutorial_samples.py)

clear_mappers()

Creating, updatingo = Order()o.user_id = 7o.description = 'order 6'session.add(o)o.order_id is Nonesession.commit()o.order_id == 6o.description = 'order B'session.commit()

session.delete(o)session.commit()

Scoped (“smart”) sessions

Session = scoped_session( sessionmaker(autoflush=True, autocommit=False))

assert Session() == Session()

Scoped sessions 2

Base = declarative_base(metadata=Session.metadata)class Order(Base): ...

o = Order()o.user_id = 7o.description = 'order 6'session.commit()

Direct updates, deletes

orders.update(orders.c.order_id==2).execute(isopen=1)orders.delete(orders.c.order_id==2).execute()

SQL layer alert!

One-to-many relations

class User(Base): orders = relation(Order, order_by=[Order.order_id])u = session.query(User).first()print u.orders

Editing collectionso = Order(description='An order')u.orders.append(o)session.commit()

Why sessions are your friends

Some ORMs rely on explicit save

More convenient, less error-prone to let ORM track dirty objects

u = User.get(1)u.orders[0].description = 'An order'u.save() # not real SA code# doh! orders[0] was not saved!

Identity map

Rows with same PK get mapped to same object (per-session)Limited caching for get()Only for get()

Managing the identity map

sesion.query(cls).populate_existing()session.expire(obj)session.refresh(obj)session.expunge(obj)expunge_all, expire_all

Questions

Exercise

Load the user named 'jack' (lowercase)Remove his first orderSave changes to the db

Fun with collectionsu = session.query(User).filter_by(name='jack').one()u.orders = u.orders[1:]session.commit()

>>> session.query(Order).get(1)Order(order_id=1,user_id=None,...)

Two solutions

o = u.orders[0]>>> oOrder(order_id=1,user_id=7,description=u'order 1',isopen=0)session.delete(o)session.commit()

>>> u.orders[Order(order_id=3,...), Order(order_id=5,...)]

Why does it make sense for this to work differently than the previous example?

#2: delete-orphanclass User(Base): __table__ = users orders = relation(Order, cascade="all, delete-orphan", order_by=[Order.order_id])

u = session.query(User).get(7)u.orders = u.orders[1:]session.commit()

>>> session.query(Order).get(1) is NoneTrue

Questions

Exercise

def user_for_order(order): session = Session.object_session(order) return ?

Backrefs

class User(Base): __table__ = users orders = relation(Order, backref='user', order_by=[orders.c.order_id])

o = session.query(Order).first()o.user

That's it for fundamentals

Multi-object queriesOne-to-oneMany-to-manyRelation queriesEager/lazy loadingTransactionsExtensions and related projects

Exercise

List all users who have open orders (isopen is nonzero)

A taste of advanced querying

q = session.query(User).join(User.orders)q.filter(Order.isopen==1).all()

All orders with an open order:

Selecting multiple classes

q = session.query(User, Order) # cartesian join!q = q.join(User.orders)# or!q = q.filter(User.user_id==Order.user_id)

u, o = q.filter(Order.isopen==1).first()

What if we want the user and the order? (efficiently)

Dropping down to SQL

sql = """select u.*from users uwhere u.user_id in ( select user_id from orders where isopen = 1)"""session.query(User).from_statement(sql)

sql = """select u.*, o.*from users u join orders o on (u.user_id = o.user_id)where o.isopen = 1"""session.query(User, Order).from_statement(sql)

Dropping down a little less

q = session.query(User, Order)q = q.join(User.orders)q.filter("orders.isopen = 1").all()

Exercise

In a single query, select the users and orders where the order description is like 'order%'

One to one

class Address(Base): __table__ = addresses

class User(Base): orders = relation(Order, order_by=[orders.c.order_id]) address = relation(Address)

How does SQLA know to treat these differently?

Many to many

class Keyword(Base): __table__ = keywords

class Item(Base): __table__ = orderitems keywords = relation(Keyword, secondary=itemkeywords)

Relation queries

user = session.query(User).get(7)q = session.query(Order)q.filter(Order.user==user).all()# q.filter(Order.user_id==user.user_id).all()

Relation queries 2

q = session.query(Order)

q.filter(Order.user.has(name='jack')).all()q.filter(Order.user.has((User.name=='jack') | (User.user_id >= 9))).all()# q.filter(Order.user_id== select([users.c.user_id], users.c.name=='jack')).all()# q.filter(Order.user_id== select([users.c.user_id], (users.c.name=='jack') | (users.c.user_id >= 9))).all()

Relation queries 3

q = session.query(User)

q.filter(User.orders.any(Order.isopen > 0)).all()q.filter(User.orders.any( Order.description.like('order%'))).all()# q.filter(User.user_id.in_( select([orders.c.user_id], orders.c.isopen > 0))).all()# q.filter(User.user_id.in_( select([orders.c.user_id], orders.c.description.like('order%')))).all()

Relation queries 4

Just need a raw EXISTS clause? Use .any() or .has() without extra parameters.

Relation queries 5

Keyword = session.query(Keyword).filter_by(name='red').one()q = session.query(Item)

q.filter(Item.keywords.contains(keyword)).all()# q.filter(Item.item_id.in_( select([itemkeywords.c.item_id], itemkeywords.c.keyword_id ==keyword.keyword_id))).all()

Exercise

Retrieve all users that do not have any orders.

Breathe Easy

That was our last exercise!

Eager loading

class User(Base): orders = relation(Order, order_by=[orders.c.order_id], lazy=False)# orq = session.query(User).options(eagerload('orders'))

# also lazyload, noload

Transactions are simple

● session: autocommit=False– this is the default– commit() / rollback() manually

● autocommit=True– each flush() also commits

Other transaction features

● begin_nested● manual transaction management at the

Connection level

__init__

class Order(Base): __table__ = orders def __init__(self): self.foo = []

@reconstructor def loaded(self): self.foo = []

__init__ is for object creation, not loadinguse @reconstructor

Questions

Related projects

MigrateFormAlchemySqlSoupElixirz3c.sqlalchemy

Migrate

# one-time setup

migrate create path/to/upgradescripts "comment"

migrate manage dbmanage.py --repository=path/to/upgradescripts –url=db-connection-url

./dbmanage.py version_control

# repeat as necessary:

./dbmanage.py script_sql sqlite# edit script

./dbmanage.py upgrade

FormAlchemy

order1 = session.query(Order).first()

from formalchemy import FieldSet fs = FieldSet(order1) print fs.render()

FormAlchemy, cont.

from formalchemy import Grid orders = session.query(Order).all() g = Grid(Order, orders) print g.render()

SqlSoup

>>> from sqlalchemy.ext.sqlsoup import SqlSoup>>> db = SqlSoup(metadata)

>>> db.users.filter(db.users.user_id < 10).all()[MappedUsers(user_id=7,name='jack'), MappedUsers(user_id=8,name='ed'), MappedUsers(user_id=9,name='fred')]

>>> db.users.first()MappedUsers(user_id=7,name='jack')>>> _.ordersTraceback (most recent call last): File "<stdin>", line 1, in <module>AttributeError: 'MappedUsers' object has no attribute 'orders'

SqlSoup 2

>>> db.users.relate('orders', db.orders)>>> db.users.first().orders[MappedOrders(...)]

>>> db.users.filter(db.users.orders.any()).all()[MappedUsers(...)]

SqlSoup 3

s = select([func.count('*')], users.c.user_id==orders.c.user_id, from_obj=[orders], scalar=True)s2 = select([users, s.label('order_count')]).alias('users_with_count')db.users_with_count = db.map(s2)

Elixir

class Person(Entity): has_field('name', Unicode) acts_as_taggable() ... some_person_instance.add_tag('cool') ... cool_people = Person.get_by_tag('cool')

http://cleverdevil.org/computing/52/

Resources

irc://irc.freenode.net/#sqlalchemySQLA mailing listMigrate, FormAlchemy, Elixir

Final Questions