PBS on Amazon

Jon Brendsel

Vice President, Products

Iconic Brands

A little bit about PBS

• General Audience, PBS Kids, PBS Education• We are not NPR!• Member organization

– 165 dues paying stations

• Dues are used for…– License broadcast and digital content– Build distribution infrastructure

• Producers are not owned by PBS– Station sponsored/owned (e.g. Frontline)– Independent (Children’s Television Workshop)

#1 in Unique VisitorsAmong major networks

PBS sites 8555

ABC.com 5977

CBS.com 5338

NBC.com 5266

PBS.org (no Kids)4566

FOX.com 3073

CWTV.com 1120

0 2,000 4,000 6,000 8,000 10,000

PBS sites

ABC.com

CBS.com

NBC.com

PBS.org (no Kids)

FOX.com

CWTV.com

(comScore MediaMetrix, July 2011)

Fun With Numbers

• Facebook Fans: ~ 1M

• Unique visitors: 18M/mo

• Visits: 37M/mo

• Page views: 367M / mo

• Video streams: 93M/mo

• Mobile Apps

– 27% of hours watched, 11% of streams

Three Years AgoLife before Amazon

• Traditional Big Iron infrastructure– Ping Power and Pipe managed by PBS IT

– Applications (Apache and higher) managed by PBS Developers

– No DR

– Fixed bandwidth

• Shared Hosting Environment– Traditional three-tier architecture

– Little isolation between 1200+ applications / producers

– Limited freedom to innovate

• No real product development capability– Small team of “web technologists”

– Projects built by outside design build firms

– No product lifecycle management

• No shared services

Three Years AgoLife was Painful

• Routine outages, lots of downtime– Multiple single points of failure– Perpetual state of emergency

• No application/producer isolation– One mistake impacts everyone– Producers hated the restrictions

• Long lead times to get new hardware• Limited long term product planning

– Ready-Fire-Aim

• Poor morale– Lack of confidence in technology organization

TodayAll New Applications / Producers -> Ec2

• Hosted producer sites

• PBS.org National Portal

• Video distribution platform

• UUA (Registration and Profiles)

• Bento (NG templated publishing system)

• TV Schedules

• Other: tools, PBS Education, PBS Kids, dev, staging, etc.

A Birds Eye View

• ~270 EC2 machines

• 75 ELBs

• 260 Elastic IPs

• 5 AWS accounts

• 52 RDS databases

• 608 EBS volumes

• many S3 buckets

• 19 COVE Cloudfront distributions

Technology Stack

• Linux Centos 5.2 (32 and 64-bit)

• Python/Perl/PHP

• Webserver: Apache

• DB: Mysql (RDS and hosted)

• Caching: memcached, SOLR

• File storage: S3, EBS

• Job management: SQS, celery

• Transcoding: Zencoder service

• Configuration management: Rightscale

Rightscale

• Manage 5 AWS accounts

• Performance monitoring and alerting (collectd)

• Configuration management of 5 shared templates

• Easy provisioning of most AWS services

• Authentication and authorization for ~30 employees

Hosted Producer Sites

Hosted Producer Sites

• Dedicated VPS-style hosting

• 43 machines (mostly m1.small)

• LAMP templates

• Nightly backups (EBS or S3)

• Isolation from other producers / systems

• VPS style SLA

• Provides freedom to innovate

PBS Video for iPadLaunched Nov ‘10

PBSKids Video for iPadLaunched April ‘11

PBS Video for iPhoneLaunched Jan ‘11

Local/National Video Portals

video.pbs.orgNext Generation Video Platform

• CDN

– Limelight (static http)

– Cloudfront (rtmp, HTTP pseudo streaming (Apple HLS)

• Presentation Layer

– Manually scaled array (4 medium)

– ELB load balancing

– S3 application server template

• Caching Layer (memcached)

– Memcached

video.pbs.orgNext Generation Video Platform (cont.)

• Service Layer ( REST API)– Manually scaled array (m1.medium)

– ELB load balancing

– S3 application server template

• DB layer– RDS Master (m1.medium)

– RDS Slaves (4 x db.xlarge)

– HAProxy load balancing

PBS Video for MobileiPhone and iPad

• Compiled Apps– PBS General Audience iPad and iPhone Apps– PBS Kids iPad App (iPhone on the way)

• Video API• VTS (Video Transcoding Service)

– EC2 instance m1.small– API wrapper around Zencoder– Transcoding from MP4 to Apple HLS for iOS distribution– Application hosted on PBS App Server template (Ec2) with RDS

DB

• File storage in in S3• Distribution via Cloudfront CDN

Video Transcoding Service

• Transcoding for multiple distribution endpoints– Ingest: Mezzanine quality (5 mpbs)– Outputs:

• Current: Apple HLS for iOS distribution• Future: web output, + partners (iTunes, Amazon, etc.)

• API wrapper around Zencoder• Application:

– 1 App ServerEC2 instance with RDS DB

• File storage in in S3• Distribution via Cloudfront CDN

Zencoder

• Encoding needs– Today: Mobile only

– Soon: All web outputs

• Powerful API

• Fast encoding times

• Based on FFMPEG

• Strong support for Apple iOS HLS (HTTP Live Streaming)

• Reliable support for Amazon S3 storage

Development / QA / Staging

• Mirrored environment and topology

• Load testing

• Demos & training

• On demand

• Bamboo-based continuous integration

Takeaways & Lessons Learned

• Vastly improved uptime

• Infrastructure: from an obstacle to an asset

• DevOps

– Build applications around infrastructure

– Design for failure

• Easy can become expensive

– PBS has proliferating infrastructure

– Now need to invest in efficiency