The Future Is VideoJonathan Conway

vzaar.com

1

Who Am I?

2

3

4

5

My little blog at http://www.jaikoo.com. And yes I registered jaikoo.com in 2001, years before the jaiku guys came along.

jaikoo / noodlesinmysandals

6

Boring!

7

Why Video?

8

http://flickr.com/photos/oddsock/82535061/9

MTV

10

http://flickr.com/photos/hint-of-plum/1789639323/

11

12

YouTube came along and took the video industry by storm. User generated/submitted content exploded on the web.

13

Suddenly everyone could become a star! *does jazz hands*

14

Brother Bobby dramatically increased his sales by utilising video in his eBay auctions. Video strengthens trust and enriches the sales experience, especially for high priced goods.

15

Everyones now realised that adding video to a website can increase user retention and participation

http://flickr.com/photos/akahige/126473337/

16

YouPorn has become one of the most trafficked sites on the net. Their traffic stats have been growing exponentially. It seems porn and video really does sell especially when it’s user generated/submitted

Options?

17

Commercial

• Zencoder - Self hosted or Pay per encode

• Encoding.com

• Sesame Vault

• Hey! Watch

• vzaar

• YouTube

18

Avoid the pain!

• Time

• Cost

• Re-inventing the wheel

19

supporting video codecs is teh suck

20

I once wasted a couple of days trying to hunt down a video codec problem related to a codec that hadn’t been used since 2001 by a now defunct web cam manufacturer.

Roll Your Own

21

Merb / Ramaze

22

“Apart from thread safety, they’re

lightweight and perfect for resource intensive

apps”23

Yep, I know Rails 2.2 is thread safe but you really don’t need all the extra baggage for a light weight application that accepts uploads. You’re able to utilize limited resource far better by using Merb or Ramaze.

24

Previous setup:Expensive F5’s distributing sticky sessions to a group of Merbs. Costly and pushes more resource requirements onto the Uploader apps/Merb’s because of the high CPU requirements of the Merb upload progress plugin. Ruby uses a lot of CPU for parsing MIME boundaries.

FAIL

25

Nginx upload progress module

http://wiki.codemongers.com/NginxHttpUploadProgressModule

26

27

No sticky sessions required now. Cheaper to scale thanks to Nginx.

Messaging

28

Queues

• Build your own - Use a database?

• ActiveMQ

• Starling

• Beanstalk

• RabbitMQ

29

Build your own

• OK for small to medium traffic

• No special skills required

• Infrastructure is already there

30

Build your own

• Re-inventing the wheel

• Activerecord based queues suck. That is all.

• Scaling the database... Boring

• Golden pneumatic drill

31

ActiveMQ

• Tried and tested

• Rails/Ruby support

• STOMP adaptors

• Persistent

32

ActiveMQ

• Can be slow(er) compared to others

• Looses messages?

• Quite fat

33

I noticed ActiveMQ to ‘loose’ message under high load.

Beanstalk

• Ruby support and bindings for other languages

• Its fast! Bloody fast!

• Proven in high transaction environments

• 35 million transactions a month at Reevoo, 10 instances of Beanstalked, 100% uptime.

34

Beanstalk

• Not clusterable? But it’s fast enough.

• Non persistent queues.

35

Starling

• Nothing to learn, it’s Memcache API.

• It’s Ruby so nothing else to install.

• Persistent queues.

36

Starling

• Performance is average. Slows down after a while.

37

RabbitMQ

• Easy to install

• Persistent queues

• Clusterable

• It’s built on Erlang. It’s designed for messaging.

• Fast!

38

RabbitMQ

• It’s Erlang?

39

Although I love Erlang, some people don’t like having mixed language environments.

AMQPAman Gupta

git://github.com/tmm1/amqp.git

40

Kick ass Event Machine based AMQP client

Encoders

41

RVideohttp://rvideo.rubyforge.org

42

• Inspect video properties

• Execute encoding “recipes” via FFmpeg, Mencoder et al

• Encoding progress

• Letterboxing

43

Use a Supervisor daemon to herd the

encoders

44

Dedicated

• More bang per buck in the short term

• Hard to scale

• Expensive in the long term

45

Cloud

• Can be expensive in the short term

• Cheaper in the long term

• Dynamically resize your encoding nodes

46

47

It’s all in the API

• Joyent, EY, GoGrid, FlexiScale limited or no API.

• They’re ‘ticket’ based clouds

• AWS on the other hand allows you to dynamically grab and release resources as required.

48

FlexiScale aka FlexiFail has been experiencing a large number of stability problems. Engine Yard is apparently close to releasing an API, though I don’t know how comprehensive this will be.

• Right Scale

• Scalr

• Pool party

• Eucalyptus

Managing your cloud

49

Eucalyptus is interesting because it’s trying to provide an abstraction layer not just for EC2 but for any cloud that supports an API. One to look out for in the future.

Storage and Content Delivery

50

Pay Akamai / Limelight

51

The problem

• Storage is painful to manage (if you ignore ZFS for the minute)

• It’s expensive to continually buy/host more disks

• Need to get video files as geographically close to the client as possible

52

Solution

• Amazon S3 (again)

• S3 storage is cheap, bandwidth is expensive

• It’s had downtime though

• Limited geographic caching

53

Poor Mans CDN

• Based on an idea from Wordpress.com, but with edge caching.

• More resilient to S3 outages

• Can be cheap in the short term

• Joyent has a 10TB data party w00t!

54

55

I’m in yer Merbs checking out your

hawtness

56

http://jaikoo.com/2008/05/09/poor-mans-cdn

57

Open Source

58

Pandahttp://github.com/newbamboo/panda/tree/master

59

• Uses RVideo, S3, SimpleDB, EC2

• REST based

• SimpleDB as the queue/data store

• Used at Amnesty

60

http://www.protectthehuman.com/

61

Panda’s been successfully deployed on this site. Go check it out, it rocks!