Top 5 things I’ve F’d up (In Live
Streaming)
About me
• GotFrag.com
• Major League Gaming
• Product Manager for Streaming Media at
Fastly
• Head of Product at Fastly
What’s this all about?
• Live streaming (not live linear)
• The (comedic) errors:
5. Forest, Trees. The big picture.
4. Latency sucks.
3. Failover. Not Fall Over.
2. Redundancy is not redundant.
1. Death by a thousand cuts. Measure it!
Some assumptions
• What:
– Live streaming is hard
– It shouldn’t be.
– But it is.
– It’s real time.
– So it’s hard.
• Why:
– “Analog” broadcast to “Digital”
medium
– Every step of the signal path can
break it
– And then there’s the Internet.
#5: The whole picture is important
• Analog to Digital
• Wires to Code
• Complexity
• Troubleshooting and QA is therefore
hard
• Studio != Touring Show
Live Streaming (Analog Side)
Live Streaming (Digital side)
#5: What got F’d.
• Audio out of sync
• Root cause was on the switcher <>
audio board
• Keyframes >1s on Encoder
compounded problem
#4: Spoilers suck
• Time Behind Live (TBL) is critical
• Broadcast standard is <15s TBL
• In other words, latency sucks.
– Latency = Buffering, jitters
– Latency = Spoilers
– Latency = WTF just happened.
Latency Sources (Digital side)
Source Ideal Avg Latency
Encoding <100ms
Uplink IP = <100ms, Sat = <1500ms
Ingest <10ms
Transcoding 50 - 500ms
CDN 150 – 1500 ms
Last Mile: Player / Browser / User Internet
?????
Total: 3610ms (3.6sec) + Last mile
#4 What got F’d
• RTMP delivery sucks on a CDN
– 5min into broadcast = 5sec TBL
– 30min into broadcast = 30sec TBL
– 8hrs into broadcast = 2min TBL
• Trolls hate spoilers. So do legit fans.
• Separate stack from normal delivery, older
hardware
• HTTP streaming is the answer (HLS!!)
#3: Fail over. Not Fall over.
• Duh.
#3: What got F’d
• Just cause the vendor says so, doesn’t
mean it _IS_ so.
• Test it.
• No really, TEST IT.
• Can you fail back and forth?
• Does it cause disconnects at player?
#2: Redundancy is not redundant
• Analog
– Matrixed encoders
– Multiple uplinks
– Back up routers
• Digital
– Cloud transcoding
– Multi-CDN
#2: What got F’d
• Production Budget
– $10k/hr for sat truck
• Upstream ISP link
from venue died
• Fix: Microwave pt to
pt to the Luxor.
#1: Real time feedback. Flexible solutions
Digital feedback:
• Analytics.
– Disconnects
– Buffering
– Edge usage
– Load on systems
– Geo distro
– etc
Analog feedback
• Leverage the audience
• Watch the stream yourself
• Filter the noise
Flexible partners and solutions
#1: What got F’d
• Tendency is to have digital side isolated from the
analog side.
• Don’t. Analog thinks about the room. Digital thinks
about the “Internets” Put the two together.
Thank you
www.fastly.com
