Download - Real World Django
-
8/14/2019 Real World Django
1/181
Django in the Real World
Jacob Kaplan-Moss
OSCON 2009
http://jacobian.org/TN
-
8/14/2019 Real World Django
2/181
Jacob Kaplan-Mosshttp://jacobian.org / [email protected] / @jacobian
Lead Developer, Django
Partner, Revolution Systems
2
-
8/14/2019 Real World Django
3/181
3
Shameless plug:
http://revsys.com/
-
8/14/2019 Real World Django
4/181
Hat tip:
4
James Bennett (http://b-list.org)
-
8/14/2019 Real World Django
5/181
So youve written aDjango site
5
-
8/14/2019 Real World Django
6/181
now what?
6
-
8/14/2019 Real World Django
7/181
API Metering
Backups & Snapshots
Counters
Cloud/Cluster Management Tools
Instrumentation/Monitoring
Failover
Node addition/removal and hashing
Auto-scaling for cloud resources
CSRF/XSS Protection
Data Retention/Archival
Deployment Tools
Multiple Devs, Staging, Prod
Data model upgrades
Rolling deployments
Multiple versions (selective beta)
Bucket Testing
Rollbacks
CDN Management
Distributed File Storage
Distributed Log storage, analysis
Graphing
HTTP Caching
Input/Output Filtering
Memory Caching
Non-relational Key Stores
Rate Limiting
Relational Storage
Queues
Rate Limiting
Real-time messaging (XMPP)
Search
Ranging
Geo
Sharding
Smart Caching
Dirty-table management
http://randomfoo.net/2009/01/28/infrastructure-for-modern-web-sites7
-
8/14/2019 Real World Django
8/181
The bare minimum:
Test.
Structure for deployment.
Use deployment tools.
Design a production environment.
Monitor.
Tune.
8
-
8/14/2019 Real World Django
9/181
Testing
9
-
8/14/2019 Real World Django
10/181
Tests are theProgrammers stone,
transmuting fear intoboredom.
Kent Beck
10
-
8/14/2019 Real World Django
11/181
Hardcore TDD
11
-
8/14/2019 Real World Django
12/181
I dont do test driven
development. I do stupidity
driven testing I wait until
I do something stupid, andthen write tests to avoid
doing it again.
Titus Brown
12
-
8/14/2019 Real World Django
13/181
Whatever happens, dont let
your test suite break thinking,Ill go back and fix this later.
13
-
8/14/2019 Real World Django
14/181
Unit testing
Functional/behaviortesting
Browser testing
unittest
doctest
django.test.Client, Twill
Windmill, Selenium
14
-
8/14/2019 Real World Django
15/181
You need them all.
15
-
8/14/2019 Real World Django
16/181
Testing Django
Unit tests (unittest)
Doctests (doctest)
Fixtures
Test client
Email capture
16
-
8/14/2019 Real World Django
17/181
Unit tests
Whitebox testing
Verify the small functional units of your
app
Very fine-grained
Familier to most programmers (JUnit,
NUnit, etc.)
Provided in Python by unittest
17
-
8/14/2019 Real World Django
18/181
django.test.TestCase
Fixtures.
Test client.
Email capture.
Database management.
Slower than unittest.TestCase.
18
-
8/14/2019 Real World Django
19/181
class StoryAddViewTests(TestCase):
fixtures = ['authtestdata', 'newsbudget_test_data']
urls = 'newsbudget.urls'
def test_story_add_get(self):
r = self.client.get('/budget/stories/add/')
self.assertEqual(r.status_code, 200)
def test_story_add_post(self):
data = {
'title': 'Hungry cat is hungry',
'date': '20090101',
}r = self.client.post('/budget/stories/add/', data)
self.assertEqual(r.status_code, 302)
19
-
8/14/2019 Real World Django
20/181
Easy to write & read.
Produces self-documenting code.
Great for cases that only use assertEquals.
Somewhere between unit tests andfunctional tests.
Difficult to debug.
Dont always provide useful test failures.
Doctests
20
-
8/14/2019 Real World Django
21/181
class Choices(object):
"""
Easy declarative "choices" tool::
>>> STATUSES = Choices("Live", "Draft")
# Acts like a choices list:
>>> list(STATUSES)
[(1, 'Live'), (2, 'Draft')]
# Easily convert from code to verbose:>>> STATUSES.verbose(1)
'Live'
# ... and vice versa:
>>> STATUSES.code("Draft")
2
"""
21
-
8/14/2019 Real World Django
22/181
****************************************************
File "utils.py", line 150, in __main__.Choices
Failed example:
STATUSES.verbose(1)
Expected:
'Live'
Got:
'Draft'
****************************************************
22
-
8/14/2019 Real World Django
23/181
a.k.a Behavior Driven Development.
Blackbox, holistic testing.
All the hardcore TDD folks look down on
functional tests.
But they keep your boss happy.
Easy to find problems; harder to find the
actual bug.
Functional tests
23
-
8/14/2019 Real World Django
24/181
Functional testing
tools
django.test.Client
webunit
Twill
...
24
-
8/14/2019 Real World Django
25/181
django.test.Client
Test the whole request path withoutrunning a web server.
Responses provide extra information
about templates and their contexts.
25
-
8/14/2019 Real World Django
26/181
class StoryAddViewTests(TestCase):
fixtures = ['authtestdata', 'newsbudget_test_data']urls = 'newsbudget.urls'
def test_story_add_get(self):
r = self.client.get('/budget/stories/add/')
self.assertEqual(r.status_code, 200)
def test_story_add_post(self):
data = {
'title': 'Hungry cat is hungry',
'date': '20090101',
}
r = self.client.post('/budget/stories/add/', data)
self.assertEqual(r.status_code, 302)
26
-
8/14/2019 Real World Django
27/181
The ultimate in functional testing for
web applications.
Run test in a web browser.
Can verify JavaScript, AJAX; even CSS.
Test your site across supported browsers.
Web browser testing
27
-
8/14/2019 Real World Django
28/181
Browser testing tools
Selenium
Windmill
28
-
8/14/2019 Real World Django
29/181
Exotic testing
Static source analysis.
Smoke testing (crawlers and spiders).
Monkey testing.
Load testing.
...
29
-
8/14/2019 Real World Django
30/181
30
-
8/14/2019 Real World Django
31/181
Further resources
Windmill talk here at OSCON
http://bit.ly/14tkrd
Django testing documentationhttp://bit.ly/django-testing
Python Testing Tools Taxonomyhttp://bit.ly/py-testing-tools
31
-
8/14/2019 Real World Django
32/181
Structuringapplications for reuse
32
-
8/14/2019 Real World Django
33/181
Designing for reuse
Do one thing, and do it well.
Dont be afraid of multiple apps.
Write for flexibility.
Build to distribute.
Extend carefully.
33
-
8/14/2019 Real World Django
34/181
1.
34
Do one thing, and do it well.
-
8/14/2019 Real World Django
35/181
Application == encapsulation
35
-
8/14/2019 Real World Django
36/181
Focus
Ask yourself: What does thisapplication do?
Answer should be one or two
shortsentences.
36
-
8/14/2019 Real World Django
37/181
Good focus
Handle storage of users and
authentication of their identities.
Allow content to be tagged, del.icio.us
style, with querying by tags.
Handle entries in a weblog.
37
-
8/14/2019 Real World Django
38/181
Bad focus
Handle entries in a weblog, and userswho post them, and their authentication,
and tagging and categorization, and some
flat pages for static content, and...
38
-
8/14/2019 Real World Django
39/181
Warning signs
Lots of files.
Lots of modules.
Lots of models.
Lots of code.
39
-
8/14/2019 Real World Django
40/181
Small is good
Many great Django apps are very small.
Even a lot of simple Django sitescommonly have a dozen or more
applications in INSTALLED_APPS.
If youve got a complex site and a shortapplication list, somethings probably wrong.
40
-
8/14/2019 Real World Django
41/181
-
8/14/2019 Real World Django
42/181
2.
42
Dont be afraid of many apps.
-
8/14/2019 Real World Django
43/181
The monolith anti-pattern
The application is the whole site.
Re-use? YAGNI.
Plugins that hook into the main application.
Heavy use of middleware-like concepts.
43
-
8/14/2019 Real World Django
44/181
(I blame Rails)
44
-
8/14/2019 Real World Django
45/181
The Django mindset
Application: some bit of functionality.
Site: several applications.
Spin off new apps liberally.
Develop a suite of apps ready for when
theyre needed.
45
-
8/14/2019 Real World Django
46/181
Django encourages this
INSTALLED_APPS
Applications are just Python packages,not some Django-specific app or
plugin.
Abstractions like django.contrib.sitesmake you think about this as you develop.
46
-
8/14/2019 Real World Django
47/181
Spin off a new app?
Is this feature unrelated to the apps focus?
Is it orthogonal to the rest of the app?
Will I need similar functionality again?
47
-
8/14/2019 Real World Django
48/181
The ideal:
48
-
8/14/2019 Real World Django
49/181
I need a contact form
49
-
8/14/2019 Real World Django
50/181
-
8/14/2019 Real World Django
51/181
Done.
51
(http://bitbucket.org/ubernostrum/django-contact-form/)
-
8/14/2019 Real World Django
52/181
But what about
Site A wants a contact form that just
collects a message.Site Bs marketing department wants a
bunch of info.
Site C wants to use Akismet to filterautomated spam.
52
-
8/14/2019 Real World Django
53/181
-
8/14/2019 Real World Django
54/181
-
8/14/2019 Real World Django
55/181
Common sense
Sane defaults.
Easy overrides.
Dont set anything in stone.
55
-
8/14/2019 Real World Django
56/181
Forms
Supply a form class.
Let users specify their own.
56
-
8/14/2019 Real World Django
57/181
Specify a default template.
Let users specify their own.
Templates
57
-
8/14/2019 Real World Django
58/181
You want to redirect after successful
submission.Supply a default URL.
(Preferably by using reverse resolution).
Let users override the default.
Form processing
58
-
8/14/2019 Real World Django
59/181
def edit_entry(request, entry_id):
form = EntryForm(request.POST or None)
if form.is_valid():form.save()
return redirect('entry_detail', entry_id)
return render_to_response('entry/form.html', {})
59
-
8/14/2019 Real World Django
60/181
def edit_entry(request, entry_id,form_class=EntryForm,
template_name='entry/form.html',
post_save_redirect=None):
form = form_class(request.POST or None)
if form.is_valid():form.save()if post_save_redirect:
return redirect(post_save_redirect)
else:
return redirect('entry_detail', entry_id)
return render_to_response([template_name, 'entry/form.html'], {})
60
-
8/14/2019 Real World Django
61/181
Provide a URLConf with all views.
Use named URL patterns.
Use reverse lookups (by name).
URLs
61
-
8/14/2019 Real World Django
62/181
4.
62
Build to distribute (even private code).
-
8/14/2019 Real World Django
63/181
myproject/
settings.py
urls.py
myapp/
models.py
mysecondapp/views.py
63
What the tutorial teaches
-
8/14/2019 Real World Django
64/181
from myproject.myapp.models import
from myproject. myapp.models import
myproject.settings
myproject.urls
64
-
8/14/2019 Real World Django
65/181
Project couplingkills re-use
65
-
8/14/2019 Real World Django
66/181
Projects in real life.
A settings module.
A root URLConf.
Maybe a manage.py (but)
And thats it.
66
-
8/14/2019 Real World Django
67/181
Advantages
No assumptions about where things live.
No PYTHONPATH magic.
Reminds you that projects are just a
Python module.
67
-
8/14/2019 Real World Django
68/181
You dont even need a project
68
-
8/14/2019 Real World Django
69/181
ljworld.com:
worldonline.settings.ljworldworldonline.urls.ljworld
And a whole bunch of apps.
69
-
8/14/2019 Real World Django
70/181
-
8/14/2019 Real World Django
71/181
Want to distribute?
Build a package with distutils/setuptools.
Put it on PyPI (or a private package
server).
Now it works with easy_install, pip,
buildout,
71
-
8/14/2019 Real World Django
72/181
General best practices
Establish dependancy rules.
Establish a minimum Python version
(suggestion: Python 2.5).
Establish a minimum Django version
(suggestion: Django 1.0).
Test frequently against new versions
of dependancies.
72
-
8/14/2019 Real World Django
73/181
Document obsessively.
73
-
8/14/2019 Real World Django
74/181
5.
74
Embrace and extend.
-
8/14/2019 Real World Django
75/181
Dont touch!
Good applications are extensible
without patching.
Take advantage of every extensibility point
an application gives you.
You may end up doing something thatdeserves a new application anyway.
75
-
8/14/2019 Real World Django
76/181
But this application
wasnt meant to beextended!
76
-
8/14/2019 Real World Django
77/181
Python Power!
77
-
8/14/2019 Real World Django
78/181
Extending a view
Wrap the view with your own code.
Doing it repetitively? Write a decorator.
78
-
8/14/2019 Real World Django
79/181
Extending a model
Relate other models to it.
Subclass it.
Proxy subclasses (Django 1.1).
79
-
8/14/2019 Real World Django
80/181
Extending a form
Subclass it.
There is no step 2.
80
-
8/14/2019 Real World Django
81/181
Other tricks
Signals lets you fire off customized
behavior when certain events happen.
Middleware offers full control over
request/response handling.
Context processors can make additionalinformation available if a view doesnt.
81
-
8/14/2019 Real World Django
82/181
If you must make
changes toexternal code
82
-
8/14/2019 Real World Django
83/181
Keep changes to a minimum
If possible, instead of adding a feature,add extensibility.
Keep as much changed code as you can
out of the original app.
83
-
8/14/2019 Real World Django
84/181
Stay up-to-date
Dont want to get out of sync with the
original version of the code!
You might miss bugfixes.
You might even miss the feature you
needed.
84
-
8/14/2019 Real World Django
85/181
-
8/14/2019 Real World Django
86/181
Be a good citizen
If you change someone elses code, letthem know.
Maybe theyll merge your changes in and
you wont have to fork anymore.
86
-
8/14/2019 Real World Django
87/181
Further reading
87
-
8/14/2019 Real World Django
88/181
Deployment
88
-
8/14/2019 Real World Django
89/181
Deployment should...
Be automated.
Automatically manage dependencies.
Be isolated.
Be repeatable.
Be identical in staging and in production.
Work the same for everyone.
89
-
8/14/2019 Real World Django
90/181
apt/yum/... virtualenv Capistrano
easy_install zc.buildout Fabric
pip Puppet/Chef/
zc.buildout
Dependency
managementIsolation Automation
90
-
8/14/2019 Real World Django
91/181
Dependancy management
The Python ecosystem rocks!
Python package management doesnt. Installing packages and dependancies
correctly is a lot harder than it should be;most defaults are wrong.
Here be dragons.
91
-
8/14/2019 Real World Django
92/181
Vendor packages
APT, Yum,
The good: familiar tools; stability; handlesdependancies not on PyPI.
The bad: small selection; not (very)portable; hard to supply user packages.
The ugly: installs packages system-wide.
92
-
8/14/2019 Real World Django
93/181
easy_install
The good: multi-version packages.
The bad: requires net connection; cantuninstall; cant handle non-PyPI packages;
multi-version packages barely work.
The ugly: stale; unsupported; defaults
almost totally wrong; installs system-wide.
93
pip
-
8/14/2019 Real World Django
94/181
piphttp://pip.openplans.org/
Pip Installs Packages
The good: Just Works; handles non-
PyPI packages (including direct fromSCM); repeatable dependancies;
integrates with virtualenv for isolation.
The bad: still young; not yet bundled.The ugly: havent found it yet.
94
zc buildout
-
8/14/2019 Real World Django
95/181
zc.buildouthttp://buildout.org/
The good: incredibly flexible; handles any
sort of dependancy; repeatable builds;
reusable recipes; good ecosystem;
handles isolation, too.
The bad: often cryptic, INI-style
configuration file; confusing duplication of
recipes; sometimes too flexible.
The ugly: nearly completely undocumented.
95
P k i l i
-
8/14/2019 Real World Django
96/181
Package isolation
Why?
Site A requires Foo v1.0; site B requires
Foo v2.0.
You need to develop against multiple
versions of dependancies.
96
P k i l i l
-
8/14/2019 Real World Django
97/181
Package isolation tools
Virtual machines (Xen, VMWare, EC2, )
Multiple Python installations.
Virtual Python installations.
virtualenvhttp://pypi.python.org/pypi/virtualenv
zc.buildouthttp://buildout.org/
97
Wh t t ?
-
8/14/2019 Real World Django
98/181
Why automate?
I cant push this fix to the servers until
Alex gets back from lunch.
Sorry, I cant fix that. Im new here.
Oops, I just made the wrong version of
our site live.
Its broken! Whatd you do!?
98
A t ti b i
-
8/14/2019 Real World Django
99/181
Automation basics
SSH is right out.
Dont futz with the server. Write a recipe.
Deploys should be idempotent.
99
Capistrano
-
8/14/2019 Real World Django
100/181
Capistranohttp://capify.org/
The good: lots of features; gooddocumentation; active community.
The bad: stale development; very
opinionated and Rails-oriented.
100
Fabric
-
8/14/2019 Real World Django
101/181
Fabrichttp://fabfile.org/
The good: very simple; flexible; activelydeveloped; Python.
The bad: no high-level commands; in flux.
101
C fi i
-
8/14/2019 Real World Django
102/181
Configuration management
CFEngine, Puppet, Chef,
Will handle alotmore than code
deployment!
I only know a little about these.
102
R d ti
-
8/14/2019 Real World Django
103/181
Recommendations
Pip, Virtualenv, and Fabric
Buildout and Fabric.
Buildout and Puppet/Chef/.
Utility computing and Puppet/Chef/.
103
-
8/14/2019 Real World Django
104/181
Productionenvironments
104
-
8/14/2019 Real World Django
105/181
-
8/14/2019 Real World Django
106/181
server
django
database
media
106
A li ti
-
8/14/2019 Real World Django
107/181
Application servers
Apache + mod_python
Apache + mod_wsgi
Apache/lighttpd + FastCGI
SCGI, AJP, nginx/mod_wsgi, ...
107
-
8/14/2019 Real World Django
108/181
-
8/14/2019 Real World Django
109/181
WSGIScriptAlias / /home/mysite/mysite.wsgi
109
-
8/14/2019 Real World Django
110/181
import os, sys
# Add to PYTHONPATH whatever you needsys.path.append('/usr/local/django')
# Set DJANGO_SETTINGS_MODULE
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'
# Create the application for mod_wsgi
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
110
-
8/14/2019 Real World Django
111/181
Does this scale?
-
8/14/2019 Real World Django
112/181
Does this scale?
server
django
database
media
Maybe!112
-
8/14/2019 Real World Django
113/181
113
Number of things
Things
persecong
Real world example
-
8/14/2019 Real World Django
114/181
Real-world example
Database A
175 req/s
Database B
75 req/s
114
Real world example
-
8/14/2019 Real World Django
115/181
http://tweakers.net/reviews/657/6115
Real-world example
-
8/14/2019 Real World Django
116/181
database server
web server
django
database
media
116
Why separate hardware?
-
8/14/2019 Real World Django
117/181
Why separate hardware?
Resource contentionSeparate performance concerns
0 1 is much harder than 1 N
117
-
8/14/2019 Real World Django
118/181
DATABASE_HOST = '10.0.0.100'
FAIL118
Connection middleware
-
8/14/2019 Real World Django
119/181
Proxy between web and database layers
Most implement hot fallover and
connection pooling
Some also provide replication, load
balancing, parallel queries, connection
limiting, &cDATABASE_HOST = '127.0.0.1'
Connection middleware
119
Connection middleware
-
8/14/2019 Real World Django
120/181
Connection middleware
PostgreSQL: pgpool
MySQL: MySQL Proxy
Database-agnostic: sqlrelay
Oracle: ?
120
-
8/14/2019 Real World Django
121/181
media server
database server
web server
django
database
media
121
Media server traits
-
8/14/2019 Real World Django
122/181
Media server traits
Fast
LightweightOptimized for high concurrency
Low memory overhead
Good HTTP citizen
122
Media servers
-
8/14/2019 Real World Django
123/181
Media servers
Apache?
lighttpd
nginx
S3
123
The absolute minimum
-
8/14/2019 Real World Django
124/181
media server
database server
web server
django
database
media
The absolute minimum
124
The absolute minimum
-
8/14/2019 Real World Django
125/181
web server
django
database
media
The absolute minimum
125
-
8/14/2019 Real World Django
126/181
-
8/14/2019 Real World Django
127/181
Load balancer traits
-
8/14/2019 Real World Django
128/181
Load balancer traits
Low memory overhead
High concurrency
Hot fallover
Other nifty features...
128
Load balancers
-
8/14/2019 Real World Django
129/181
Load balancers
Apache + mod_proxy
perlbalnginx
Varnish
Squid
129
-
8/14/2019 Real World Django
130/181
CREATE POOL mypool
POOL mypool ADD 10.0.0.100
POOL mypool ADD 10.0.0.101
CREATE SERVICE mysite
SET listen = my.public.ipSET role = reverse_proxy
SET pool = mypoolSET verify_backend = on
SET buffer_size = 120k
ENABLE mysite
130
-
8/14/2019 Real World Django
131/181
you@yourserver:~$ telnet localhost 60000
pool mysite add 10.0.0.102OK
nodes 10.0.0.101
10.0.0.101 lastresponse 1237987449
10.0.0.101 requests 97554563
10.0.0.101 connects 12924243510.0.0.101 lastconnect 1237987449
10.0.0.101 attempts 129244743
10.0.0.101 responsecodes 200 358
10.0.0.101 responsecodes 302 1410.0.0.101 responsecodes 207 99
10.0.0.101 responsecodes 301 1110.0.0.101 responsecodes 404 18
10.0.0.101 lastattempt 1237987449
131
-
8/14/2019 Real World Django
132/181
media server cluster
web server cluster
load balancing cluster
proxy
django djangodjango
database server cluster
database
mediamediaproxyproxy
database database
cache cluster
cachecache
132
-
8/14/2019 Real World Django
133/181
Shared nothing
133
-
8/14/2019 Real World Django
134/181
BALANCE = None
def balance_sheet(request):global BALANCE
if not BALANCE:bank = Bank.objects.get(...)
BALANCE = bank.total_balance()
...
FAIL134
-
8/14/2019 Real World Django
135/181
Global variables are
right out
135
-
8/14/2019 Real World Django
136/181
from django.cache import cache
def balance_sheet(request):balance = cache.get('bank_balance')
if not balance:
bank = Bank.objects.get(...)balance = bank.total_balance()
cache.set('bank_balance', balance)
...
WIN136
-
8/14/2019 Real World Django
137/181
def generate_report(request):
report = get_the_report()
open('/tmp/report.txt', 'w').write(report)
return redirect(view_report)
def view_report(request):
report = open('/tmp/report.txt').read()
return HttpResponse(report)
FAIL137
-
8/14/2019 Real World Django
138/181
Further reading
-
8/14/2019 Real World Django
139/181
Further reading
Cal Henderson, Building Scalable Web Sites
John Allspaw, The Art of Capacity Planning
http://kitchensoap.com/
http://highscalability.com/
139
-
8/14/2019 Real World Django
140/181
Monitoring
140
Goals
-
8/14/2019 Real World Django
141/181
Goals
When the site goes down, know it immediately.
Automatically handle common sources ofdowntime.
Ideally, handle downtime before it even happens.
Monitor hardware usage to identify hotspots andplan for future growth.
Aid in postmortem analysis.
Generate pretty graphs.
141
Availability monitoring
-
8/14/2019 Real World Django
142/181
principles Check services for availability. More then just ping yoursite.com.
Have some understanding of dependancies.
Notify the right people using the right
methods, and dont stop until its fixed.
Minimize false positives.
Automatically take action against common
sources of downtime.
142
Availability monitoring tools
-
8/14/2019 Real World Django
143/181
Availability monitoring tools
Internal tools
Nagios
Monit
Zenoss
...External monitoring tools
143
Usage monitoring
-
8/14/2019 Real World Django
144/181
Usage monitoring
Keep track of resource usage over time.
Spot and identify trends.
Aid in capacity planning and management.
Look good in reports to your boss.
144
Usage monitoring tools
-
8/14/2019 Real World Django
145/181
Usage monitoring tools
RRDTool
Munin
Cacti
Graphite
145
-
8/14/2019 Real World Django
146/181
146
-
8/14/2019 Real World Django
147/181
Logging
-
8/14/2019 Real World Django
148/181
Record information about whats
happening right now.Analyze historical data for trends.
Provide postmortem information after
failures.
Logging
148
Logging tools
-
8/14/2019 Real World Django
149/181
Logging tools
printPythons logging module
syslogd
149
-
8/14/2019 Real World Django
150/181
-
8/14/2019 Real World Django
151/181
-
8/14/2019 Real World Django
152/181
Performance
152
And when you should care.
Ignore performance
-
8/14/2019 Real World Django
153/181
g p
Step 1: write your app.
Step 2: make it work.
Step 3: get it live.
Step 4: get some users.
Step 94,211: tune.
153
Ignore performance
-
8/14/2019 Real World Django
154/181
g p
Code isnt fast or slow until its
deployed in production.
That said, often bad code is obvious.
So dont write it.
YAGNI doesnt mean you get to be
an idiot.
154
Low-hanging fruit
-
8/14/2019 Real World Django
155/181
g g
Lots of DB queries.
Rule of thumb: O(1) queries per view.
Very complex queries.
Read-heavy vs. write-heavy.
155
Anticipate bottlenecks
-
8/14/2019 Real World Django
156/181
p
Its probably going to be your DB.
If not, itll be I/O.
156
-
8/14/2019 Real World Django
157/181
Its slow!
157
-
8/14/2019 Real World Django
158/181
-
8/14/2019 Real World Django
159/181
159
-
8/14/2019 Real World Django
160/181
YSlowhttp://developer.yahoo.com/yslow/
160
http://developer.yahoo.com/yslow/http://developer.yahoo.com/yslow/ -
8/14/2019 Real World Django
161/181
Server-side
performance tuning
161
Tuning in a nutshell
-
8/14/2019 Real World Django
162/181
g
Cache.
Cache some more.
Improve your caching strategy.
Add more cache layers.
Then,maybe, tune your code.
162
Caching is magic
-
8/14/2019 Real World Django
163/181
g g
Turns less hardware into more!
Makes slow code fast!
Lowers hardware budgets!
Delays the need for new servers!
Cures scurvy!
163
-
8/14/2019 Real World Django
164/181
Caching is about
trade-offs
164
Caching questions
-
8/14/2019 Real World Django
165/181
g q
Cache for everybody? Only logged-in users?
Only non-paying users?
Long timeouts/stale data? Short timeouts/
worse performance?
Invalidation: time-based? Data based? Both?
Just cache everything? Or just some views?
Or just the expensive parts?
Djangos cache layer? Proxy caches?
165
Common caching strategies
-
8/14/2019 Real World Django
166/181
Are most of your users anonymous? UseCACHE_MIDDLEWARE_ANONYMOUS_ONLY
Are there just a couple of slow views? Use
@cache_page.
Need to cache everything? Use a site widecache.
Everything except a few views? Use@never_cache.
166
Site-wide caches
-
8/14/2019 Real World Django
167/181
Good: Djangos cache middleware.Better: A proper upstream cache. (Squid,
Varnish, ).
167
-
8/14/2019 Real World Django
168/181
-
8/14/2019 Real World Django
169/181
Conditional view
processing
169
GET / HTTP/1.1
Host: www2.ljworld.com/
HTTP/1.1 200 OK
-
8/14/2019 Real World Django
170/181
170
Server: Apache
Expires: Wed, 17 Jun 2009 18:17:18 GMTETag: "93431744c9097d4a3edd4580bf1204c4"
GET / HTTP/1.1
Host: www2.ljworld.com/
IfNoneMatch: "93431744c9097d4a3edd4580bf1204c4"
HTTP/1.1 304 NOT MODIFIED
GET / HTTP/1.1
Host: www2.ljworld.com/IfModifiedSince: Wed, 17 Jun 2009 18:00:00 GMT
HTTP/1.1 304 NOT MODIFIED
Etags
-
8/14/2019 Real World Django
171/181
171
Opaque identifiers for a resource.
Cheaper to compute than the resource itself.
Bad: 17, some title, etc.
Good:93431744c9097d4a3edd4580bf1204c4,
74c05a20-5b6f-11de-adc7-001b63944e73, etc.
-
8/14/2019 Real World Django
172/181
When caching fails
172
-
8/14/2019 Real World Django
173/181
I think I need a bigger box.
173
Where to spend money
-
8/14/2019 Real World Django
174/181
174
First, buy more RAM.
Then throw money at your DB.
Then buy more web servers.
-
8/14/2019 Real World Django
175/181
No money?
175
Web server
improvements
-
8/14/2019 Real World Django
176/181
improvements
Start with simple improvements: turn off
Keep-Alive, tweak MaxConnections; etc.
Use a better application server
(mod_wsgi).
Investigate light-weight web servers
(nginx, lighttpd).
176
Database tuning
-
8/14/2019 Real World Django
177/181
Whole books can be and many have
been written about DB tuning.
MySQL: High Performance MySQL
http://www.amazon.com/dp/0596101716/
PostgreSQL:http://www.revsys.com/writings/postgresql-performance.html
177
Build a toolkit
http://www.revsys.com/writings/postgresql-performance.htmlhttp://www.revsys.com/writings/postgresql-performance.html -
8/14/2019 Real World Django
178/181
profile, cProfile
strace, SystemTap, dtrace.
Django debug toolbar
http://bit.ly/django-debug-toolbar
178
-
8/14/2019 Real World Django
179/181
Morehttp://jacobian.org/r/django-cache
http://jacobian.org/r/django-conditional-views
179
-
8/14/2019 Real World Django
180/181
-
8/14/2019 Real World Django
181/181
Fin.Contact me: [email protected] / @jacobian
Hire me: http://revsys.com/