python on a plane
DESCRIPTION
Slides from ‘Python on a Plane’, my talk at PyConES 2014: http://davidarcos.net/blog/2015/04/09/python-on-a-plane-pycones-2014/ Video: https://www.youtube.com/watch?v=rZzrGhEgWWsTRANSCRIPT
![Page 1: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/1.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
![Page 2: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/2.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Abstract
● An entertainment system at 10,000 metres– Embedded computer in each plane– Synchronized after each flight– And a Platform with lots of integrations
● This is how we do it
![Page 3: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/3.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Hi!
● I'm David Arcos
● Python/Django developer since 2008
● Interested in distributed systems, databases, scalability, security
● Backend engineer at Immfly
![Page 4: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/4.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Immfly: In-Flight Entertainment
Private & Confidential
Immfly is a new Entertainment, Retail and Communication platform for the in-flight experience.
Focused on the European domestic flights market, Immfly offers wireless content to passengers via their Personal Electronic Devices.
![Page 5: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/5.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
The three challenges
1) On board the aircraft:● Provide a Backend with lots of features● Will work off-line
2) Synchronize the aircrafts● Keep the replicas consistent● Update the contents: I don't want yesterday's newspaper!
3) Integrate with 3rd parties● Content providers, payments, mailing, weather, flight info...
![Page 6: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/6.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
1) On board the aircraft
![Page 7: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/7.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Some requirements
● Users have different needs● Common authentication● It's offline● Show some flight info● Deliver static content
![Page 8: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/8.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Users have different needs
● Apps: web, android, iOS
● Devices: laptop, smartphone, tablet
● Return different results based on language, destination, airline, schedule...
![Page 9: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/9.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Users have different needs
● Apps: web, android, iOS– Same REST API for every frontend
● Devices: laptop, smartphone, tablet– Generate thumbnails with different sizes to allow
a responsive design● Return different results based on language,
destination, airline, schedule...– API calls allow filtering by many parameters
![Page 10: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/10.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Common authentication
● Same user, different apps– Example: web app opens mobile app– Example: mobile app embeds a webview
![Page 11: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/11.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Common authentication
● Same user, different apps– Example: web app opens mobile app– Example: mobile app embeds a webview
● Django Rest Framework has TokenAuthentication
– Just a HTTP header, example:
Authorization: Token 9944b09199c62bcf9418ad846dd0e4bbdfc6ee4b
![Page 12: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/12.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
It's off-line!
● Can't use any external service:– No analytics, online error logging, google maps– No fancy SaaS integrations– Can't just use CDN, email or DNS servers
![Page 13: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/13.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
It's off-line!
● Can't use any external service:– No analytics, online error logging, google maps– No fancy SaaS integrations– Can't just use CDN, email or DNS servers
● We deployed our own solutions– Very time-consuming– Can't hotfix bugs. Need to do lots of Q&A.
![Page 14: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/14.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Show some flight info
● Estimated Time of Arrival?
● Weather at destination
● Show a nice map
● Where is the plane?
![Page 15: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/15.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Show THIS flight info
![Page 16: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/16.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Lots of flight info
● Estimated Time of Arrival?– Pre-load Flightstats API
● Weather at destination– Pre-load OpenWeatherMap (pyowm)
● Show a nice map– OpenStreetMap
● Where is the plane?– The plane tells us :)
![Page 17: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/17.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Avionics data bus
● altitude
● flight_id
● ground_speed
● heading
● latitude
● longitude
● mach_speed
● outside_temperature
● pitch
● roll
● wind_speed
● yaw
We get data in real-time:
![Page 18: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/18.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Discrete-time signals
● DCFAILSIG
● ACFAILSIG
● OVERTEMPSIG
● GSM_POWER_STATUS
● ENB2SIG
● ENB1SIG
● ENB0SIG
● GSMSIG_STATUS
● CPLD_REV0
● CPLD_REV1
● SYSENSIG
● ENB3SIG
● ENB4SIG
● ENB5SIG
● ALERT
● CONFIGSIG0
● CONFIGSIG1
● CONFIGSIG2
● INTTEST_OUT
● INTTEST_IN
● ISO_OUT0
● ISO_OUT1
● ISO_OUT2
● ISO_OUT3
● GPIO_DCFAILSIG
● GPIO_ACFAILSIG
● GPIO_OVERTEMPSIG
● GPIO_SYSENSIG
![Page 19: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/19.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Deliver static content
● Provide the user with different contents:– TV Shows, Videos (format? size?)– Newspapers, Magazines– Images
![Page 20: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/20.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Deliver static content
● Provide the user with different contents:– TV Shows, Videos– Newspapers, Magazines– Images
● Just use nginx, but:– Transcode video to HLS (bitrate, chunks)– Pre-process PDFs– Load test to verify assumptions
![Page 21: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/21.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Other components
Our django apps:– common
– content
– flightinfo
– images
– stats
– surveys
– users
Some external libraries:– Django
– Django Rest Framework
– django-uuidfield
– django-redis
– django-celery
– django-extensions
– django-imagekit
![Page 22: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/22.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Aircraft Infrastructure
HTTP server:
+
+
Databases:
&
![Page 23: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/23.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Embedded computer (lab)
![Page 24: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/24.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Access point + dashboard (lab)
![Page 25: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/25.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Hardware schema
![Page 26: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/26.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
The three challenges
2) Synchronize the aircrafts● Keep the replicas consistent● Update the contents: I don't want yesterday's newspaper!
![Page 27: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/27.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
2) Synchronize the aircrafts
![Page 28: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/28.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Context
● An Aircraft needs to update contents and usage
● During the flight, the Aircraft is off-line● Once landed, it gets connectivity● Then it tries to synchronize
![Page 29: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/29.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Connectivity
● Internal 3G data card– Enabled when grounded– Disabled when about to flight
● VPN (“control tower”)– SSH to the Hangar– Hangar can SSH back
![Page 30: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/30.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
The Hangar
● Central point of the platform● Special instance, in the cloud
– Same software (db schema)– Different apps and settings
● Used as a CMS, to manage all resources– django.contrib.admin
● All other instances just replicate some data from the Hangar
![Page 31: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/31.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
The Heartbeat
● An Aircraft sends Heartbeats when it's online– It's just a “ping” to the Hangar– A simple POST, with some details
● The Hangar stores the heartbeats– Knows what planes are online, grounded, at a
given moment– Will send commands to those planes
● In example: synchronize
![Page 32: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/32.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Aircraft synchronization
● An Aircraft:– Sends usage data
● New/updated users, surveys, payments, stats, etc
– Gets usage data● New/updated users
– Gets updated contents● New/updated resources, contents, offers, destinations,
misc data...
![Page 33: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/33.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Trouble?
● For some models (like Users), the Primary Keys will collide
● Can get conflicts
● May need to merge resources, edited in several Aircrafts at the same time
![Page 34: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/34.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Prevent the trouble
● For some models (like Users), the Primary Keys will collide– Those PKs must be UUIDs
● Can get conflicts– Each model should be synchronized only in one
direction: A->H or H->A● May need to merge resources, edited in
several Aircrafts at the same time– Make sure our logic doesn't allow that
![Page 35: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/35.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
fabric
Python library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks
http://fabfile.org
![Page 36: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/36.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
fabtools.require
fabtools includes useful functions to help you write your Fabric files.
Using fabtools.require allows you to use a more declarative style, similar to Chef or Puppet.
http://fabtools.readthedocs.org
Example:
![Page 37: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/37.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Other tools
● Python– boto: upload/download from S3– requests: API calls– django.db.migrations
● Unix– openssh– rsync– bash script
![Page 38: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/38.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
The three challenges
3) Integrate with 3rd parties● Providers, payments, mailing, weather, flight info...
![Page 39: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/39.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
3) Integrate with 3rd parties
![Page 40: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/40.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Highlights
● Resource Ingest● API diversity● AWS● Celery● Other APIs
![Page 41: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/41.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Resource ingest
● Entertainment– Newspapers– Magazines– Videos– TV Shows
● Deals– Airport– Transport– Destination
● Lots of resources with lots of providers– Per language, country, etc
![Page 42: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/42.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
API diversity...
● Best case– Python API :)
● Good enough– REST API, lots of metadata, well documented
● Acceptable– S3 bucket or FTP, import all, but no metadata
● Worst case:– ?
![Page 43: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/43.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Beware of the Intern-API!
https://www.flickr.com/photos/reidrac/2387432357/
![Page 44: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/44.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Is this even a API?
● Word document– (why?)
● Excel document– (no format, of course)
![Page 45: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/45.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
AWS, using boto
● ElasticTranscoder– Media transcoding in the cloud
● Simple Notification Service (SNS)– A fast, flexible, fully managed push messaging
service
● Simple Storage Service (S3)– Secure, durable, highly-scalable object storage.
![Page 46: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/46.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Celery task queue
● Hangar tasks use Celery:– Celery is an asynchronous task queue/job
queue based on distributed message passing– Async– I/O intensive tasks: crawling APIs– CPU intensive tasks: thumbnails, billing pdfs
![Page 47: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/47.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Flower: Celery monitoring tool
● Flower is a web based tool for monitoring and administrating Celery clusters– http://flower.readthedocs.org
● Real time● Tip: a queue per task type
![Page 48: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/48.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Other APIs
● Bookings● Payments● Billing● Emails
● Weather● Flight info● Statistics● Slack
![Page 49: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/49.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Other libraries
● Images:
– Pillow
– django-imagekit
– pilkit
– PyPDF2
– Wand
● PDFs:
– django-easy-pdf
– xhtml2pdf
● Other:
– django-yubin
– libsaas
![Page 50: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/50.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Recap
● Solved the three challenges!
– In-flight API with tons of features
– When grounded, they get synchronized
– The Hangar manages the full platform
![Page 51: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/51.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Conclusions
● Python made it possible!
● Very versatile, covers all our use cases
● “We stand on the shoulders of giants”
● Developed in a short time
![Page 52: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/52.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Thanks for attending!
Get the slides at http://slideshare.net/DZPM
Any questions?
![Page 53: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/53.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Questions?
![Page 54: Python on a Plane](https://reader036.vdocuments.site/reader036/viewer/2022081419/557d6066d8b42aba3d8b4fcf/html5/thumbnails/54.jpg)
Python on a Plane – PyConES 2014 David Arcos - @DZPM
Thanks for attending!
Get the slides at http://slideshare.net/DZPM
Enjoy your flight! http://immfly.com