scaling django to the sky

SCALING DJANGO TOSCALING DJANGO TOTHE SKY THE SKY

HOW SMALL DECISIONS CAN MAKE A BIGHOW SMALL DECISIONS CAN MAKE A BIGDIFFERENCEDIFFERENCE

!# Created by = ; Social =Naren Arya @Narenarya3

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

1 of 22 Friday 12 February 2016 08:30 AM

WHOAMI WHOAMII am a

Python & JavaScript programmerLinux & Books loverDevOps Engineer by professionBlogging & Chess are favorite hobbies

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

2 of 22 Friday 12 February 2016 08:30 AM

PART-1PART-1 DESIGN OPTIMIZATION DESIGN OPTIMIZATION

++

STACK OPTIMIZATION STACK OPTIMIZATION====

SPEED SPEED

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

3 of 22 Friday 12 February 2016 08:30 AM

THUMB RULE OF MVCTHUMB RULE OF MVC

In most cases web application wastes it's time in interactingwith database. So Database is the slowest part of any web

framework.

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

4 of 22 Friday 12 February 2016 08:30 AM

EVIL SIDE OF DJANGO ORMEVIL SIDE OF DJANGO ORMMyth: Django ORM is dead easy to use in a web application

Truth: Django ORM can make a web application dead slow

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

5 of 22 Friday 12 February 2016 08:30 AM

TIPS FOR CODE OPTIMIZATIONTIPS FOR CODE OPTIMIZATIONThe techniques those can reduce the DB hits siginificantly:

Advanced ORM implementationsQuery cachingDjango template cachingNon-blocking codeAlternate data stores

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

6 of 22 Friday 12 February 2016 08:30 AM

ORM FACTORSORM FACTORSCACHED PROPERTIESCACHED PROPERTIES

Use cached proerty to avoid multiple db queries

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

7 of 22 Friday 12 February 2016 08:30 AM

In a single query we can fetch the related objects.Don'tmake multiple queries for every thing.

PREFETCH AND SELECT RELATEDPREFETCH AND SELECT RELATED

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

8 of 22 Friday 12 February 2016 08:30 AM

A novice developer's simple mistake at 20 places can costyou more servers

HOW EASILY ONE CAN ADD AHOW EASILY ONE CAN ADD ABOTTLENECK TO DJANGOBOTTLENECK TO DJANGO

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

9 of 22 Friday 12 February 2016 08:30 AM

like

Which can have greater performance bottlenecks

MISTAKES! MISTAKES!MISTAKES! MISTAKES!We all are doing basic mistakes

... bad design ...

... writing inefficient code

not analyzing latest documentation...

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

10 of 22 Friday 12 February 2016 08:30 AM

OTHER FACTORSOTHER FACTORS////

Non Blocking Django CodeUse Queue and Job systemsSpawn non-blocking IO from Django

Caching internal componentsQuery CachingTemplate Caching

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

11 of 22 Friday 12 February 2016 08:30 AM

OPTIONS FOR NON-BLOCKINGOPTIONS FOR NON-BLOCKINGCODECODE

RabbitMQ + Celery1. asyncio (Python 3) || future-requests (python 2.7+)2.

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

12 of 22 Friday 12 February 2016 08:30 AM

OPTIONS FOR QUERY CACHINGOPTIONS FOR QUERY CACHING

by Tobias McNulty by Jason Moiron

Cache-MachineJohnny-Cache

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

13 of 22 Friday 12 February 2016 08:30 AM

OPTIONS FOR TEMPLATEOPTIONS FOR TEMPLATECACHINGCACHING

Must use Django template cachingCache-Machine provide support for Jinja2 templating

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

14 of 22 Friday 12 February 2016 08:30 AM

PART-2PART-2STACK OPTIMIZATIONSTACK OPTIMIZATION

“If you are going to kick a tiger inthe ass, you’d better have a plan

to deal with the teeth” - Tomclancy

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

15 of 22 Friday 12 February 2016 08:30 AM

AN IDEAL STACKAN IDEAL STACKPyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

16 of 22 Friday 12 February 2016 08:30 AM

ESSENTIAL COMPONENTSESSENTIAL COMPONENTSLoad balancersWeb accelaratorsCache backendsDatabase SQL vs NoSQL

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

17 of 22 Friday 12 February 2016 08:30 AM

OPTIONS FOR STACKOPTIONS FOR STACKThese open source so�ware proved worthy in powering

huge web sites like Instagram, Disqus.

Load balancers - HAProxyWeb accelarators - VarnishCache backends - Redis, MemcachedDatabase SQL vs NoSQL - MongoDB, PostgreSQL JSONstore

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

18 of 22 Friday 12 February 2016 08:30 AM

With help of floating IP we can create a cluster of loadbalancers in front of Django application servers

IMPORTANCE OF LOADIMPORTANCE OF LOADBALANCINGBALANCING

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

19 of 22 Friday 12 February 2016 08:30 AM

Which is better web accelarator?

NGINX VS VARNISHNGINX VS VARNISH

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

20 of 22 Friday 12 February 2016 08:30 AM

APPLES ARE NOTAPPLES ARE NOTAIRPLANESAIRPLANES

Databases

MONGODB VS POSTGRE SQLMONGODB VS POSTGRE SQLJSON STOREJSON STOREDatastores

REDIS VS MEMCACHEDREDIS VS MEMCACHED

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

21 of 22 Friday 12 February 2016 08:30 AM

THANK YOU DELHI THANK YOU DELHI For giving me this wonderful opportunity

- -

self.python_blogself.personal_blog

PyDelhi 2016 http://narenaryan.github.io/PyDelhi-Infrastructure-proposal/?print-pdf#/

22 of 22 Friday 12 February 2016 08:30 AM