aws re:invent 2016: turner's cloud native media supply chain for tnt, tbs, adult swim, cartoon...

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Michael Koetter | VP Digital Media System Turner

Usman Shakeel | WW Tech Lead (M&E) Amazon Web Services

11/28/2016

MAE302

Turner’s Cloud Native Content Supply Chain

for TNT, TBS, Adult Swim, Cartoon Network, CNN

Playout & Distribution

Visual Effects & Editing Analytics

DAM & Archive

Digital Supply Chain

Publishing

OTT

Acquisition

AWS Media Segments

Key Evolutions in our Industry

The Newcomers (Netflix, Amazon) effect

Direct B2C (OTT, Digital)

Know thy consumer and create the content they dig

Cost Crunch on production/post-production

Content has gravity and getting heavier Annual Expenditure on Programming ($B)

“Millennials spend nearly 50% of their time watching movies

and TV shows on devices other than a TV, primarily

laptops/desktops, followed by smartphones and tablets”

Source: Deloitte Digital Democracy Survey 2016

Millions Netflix Subscribers

The New Normal

Media – ALL IN – to the Cloud

Where do I begin?

Media – ALL IN – to the Cloud

De-Coupling

Micro-servicesBuzzword?

Specific scope

Interoperable services

Allow rapid innovation

An inside-out view

Playout & Distribution

Production/Post Production B2B/Cinematic Delivery

DAM & Archive

Digital Supply Chain

Publishing

OTT

Content Acquisition

Ingestion

Manual QC

Auto-QC

Metadata Extraction

Transcoding

Delivery

Packaging

Clips/Edits

Complex workflow/State management

Bursty sub-workflow tasks (transcoding/auto-QC, Packaging)

Elastic Licensing

Content movement between workflow tasks (a common repository)

Different delivery formats for each affiliate and downstream

Manual tasks – (content movement or interactivity)

CMS/DAM

Normalization

Normalization

All-In Cloud-Enabled Domino Effect

Playout & DistributionLive TV

Production/Post

ProductionEditorial

Dailies

Post

Social Media

Marketing/Promo

B2B/Cinematic DeliveryDPP, IMF, NABA

DAM & ArchiveActive Archive

Golden Copy

Preservation

Digital Supply Chain

Publishing

OTTLive-To-VOD

AVOD, SVOD, TVOD

Catchup TV

Cloud PVR

Content AcquisitionLive

File

Ingestion

Manual QC

Auto-QC

Metadata Extraction

Transcoding

Delivery

Packaging

Clips/Edits

CMS/DAM

ETC DI Workflow for Major Studios

An Industry View of Media Supply Chain

Complex workflow

State management

Bursty workflow

tasks (transcoding/auto-

QC, Packaging)

Elastic Licensing

Content movement

Delivery formats for

each affiliate and

downstream

Manual tasks(content movement or

interactivity)

Media Supply Chain (simplified)

Quality Check Create, edit, &

collaborate in the cloud

Distribute content to

playout/OTT solution

Import mastered

content, along with

metadata into S3

Macro Level Ideas/Highlights

• Amazon EC2 New Instances

• Cloud Storage

• Amazon S3, Amazon Glacier (new retrieval tiers)

• Amazon EFS

• Amazon EBS new Volume types and performance

• NLE in the Cloud (Remote Application Streaming)

• Graphics Amazon WorkSpaces

• Microservices/Serverless

• AWS Lambda

• Amazon API Gateway

Media Supply Chain at Turner

Michael KoetterVP Digital Media Systems @ Turner

Stuff for which I’m responsible:

• Content Creation Systems

• Content Supply Chain Systems

• Technology Infrastructure

Roll Video…

OVERALL OBJECTIVE.

We are leading the media industry’s transformation through our approach to technology, data science and innovation.

14

SUPPLY CHAIN MISSION

Connecting CONTENT

PRODUCERS with content monetization PLATFORMS to

drive Turner’s business across all brands, networks

and products.

ETC DI Workflow for Major Studios

An Industry View

D2C

ES

TS

VO

D

`

`

CO

NN

EC

TE

D

DE

VIC

ES

PA

Y T

VB

RO

AD

BA

ND

Linear Network

Origination

Downstream

Processing

Content

Suppliers

SchedulingRightsTitles Ordering

Turner Business Systems

Cloud Media Processing

Turner’s Content Supply Chain

Content Receiving26,000 Items Processed (Jan – Nov 2016)

Direct to Consumer~ 1400 movie initial load

SVOD Initial Load(deal dependent)

Promo Production~ 1000 items per program per agency

Flexibility Analytics Elasticity

SUPPLY CHAIN REQUIREMENTS

Emerging challenges are putting pressure on our traditional processing model.

Flexible Process Engine

Media Micro-Services

Supply Chain Components

Upload Auto QCInput

ProcessingMaster Version

Output

ProcessingDeliver

Cloud Library

Work Order Management (Process Engine)

Content tracking and

flow control.

Integration API for

ODT & Playout

Automation.

Final content

delivery to

distribution systems

Output sequence

conforming, trigger

insertion and

transcode.

Creation of golden library

master, preservation in

library and creation of

versions for specific

platform needsNormalization of

incoming

distribution masters

Automated quality

check of incoming

distribution masters

Content submission

to CSC

Ingest of material into

library’s normalized

media data model

Selection of material

from master asset and

creation of new and

unique sequences.

25

Order Core

Fle

xib

le P

roces

s E

ng

ine

• Fully abstracted

• Instance-less

Order Types

• Rapid Development

• Simplified Environment

• Helper Classes

Order Types

26

Order Core

State

MachineResource

AllocationRules Engine

Order

Scaffold

Generator

HTML

Template

Document

DB

Form Data

UI Shared

WidgetsFle

xib

le P

roces

s E

ng

ine

WHY USE AN INSTANCE-LESS

DESIGN?

to implementSimple

Agile

Available

Scalable

Zero effort to administer

on demand

without ”clustering”

coding practices

Instance-less

Fle

xib

le P

rocess E

ng

ine

Fle

xib

le P

rocess E

ng

ine

Fle

xib

le P

rocess E

ng

ine

AWS

S3

Rules Engine

incl.Lambda Cron

State Management

CSC Work Order Architecture Version 1.3Chris Hinton, 8th Nov 2016

SMTP

AWS

DynamoDB

Client / Chrome

Quark / Topbar

Work Orders App

List / Search / Filter

Work Order Item

CRUD &

Business Logic

JS Lambda

AW

S A

PI G

ate

wa

y

S3 H

TT

PS

Conversations

forms &

utils JS

Other AWS

Document Management

document templates

Audit Log

KPI & Cost

Turner Quark &

Content Apps

Client JS

Turner

Network

Public

Internet

SNS

SDVI

External

Orchestrator

PSP

Fle

xib

le P

rocess E

ng

ine

Media Micro-Services

Upload Auto QCInput

ProcessingMaster Version

Output

ProcessingDeliver

Cloud Library

Work Order Management

Dynamically created infrastructure

Usage based licensing

Media-aware analytics and

optimization

Fine grained media processing as a

micro-service

Media Micro-Services

Media Micro-Services

Content Receiving Workflow

Input files

in S3

S3 Multi-Part

Client

S3 Transfer

Acceleration

EC2 instances

S3 Notification

Trigger a Lambda Function to

update portal and job queue

File Transmission

complete

SDVI

Amazon SNS

Integration

Supplier 1

Supplier 3

S3 Life Cycle Policies for purging temp files

S3 as input/output to different tasks

Supplier 2

Order

fulfillment State

Web Portal

Upload Content

File to S3

Save Order

Amazon Dynamo DB

Order Portal

REST

Client API

Process Rules,

Update State

Turner’s Media Supply Chain in the AWS Cloud

AWS cloud – US East 1

Turner Campus - Atlanta

Workorders

Content

Amazon API

Gateway

AWS

Lambda

Turner VPC

SDVI VPC

ProxyAmazon

Elasticsearch

AWS SNS

Content

Distributors

Turner

Employee

Legacy

Infrastructure

DynamoDB

Macro Implications

Cloud Content Security Strategy

(Considerations)

39

• Traditional on prem content security has been perimeter

based

• Perimeters can be breached

• Risk domains are very large

• Logging is not always robust

• Cloud security affordances enabled Turner’s Information

Security Office to take a favorable position for change

Macro

Im

pli

cati

on

s

Cloud Content Security Strategy

40

• Separate AWS account for Content storage with limited

employee access (and no compute)

• S3 buckets with Bucket Policies limiting Get and Puts

based on Amazon VPC Endpoints (Turner and SDVI)

• Time-limited tokens for SDVI & other vendors to access

pre-air & master quality content

• Bucket Access logging into AWS CloudTrail S3 Bucket

• On-Prem Low-Res Proxy Streaming system limits access

to Turner's network for better cost & performance.

Macro

Im

pli

cati

on

s

Change Control

Continuous Delivery

41

• Media industry is traditionally averse to change.

• ”Set it and forget it” is held as an ideal

• Changes occur infrequently & at the most low impact

times… yet they are in reality always changing.

• Systems have been architected around these principles

• If the business wants agility and flexibility, we need to

change our delivery model

Macro

Im

pli

cati

on

s

Change Control

Continuous Delivery

42

Macro

Im

pli

cati

on

s

Automated

Daily Release

Process

Faster delivery

of value to the

business

Release in

smaller

increments,

control risk

Reduced “Bit

Rot” &

Improved

Developer

Productivity

Change Control

Continuous Delivery

43

Macro

Im

pli

cati

on

s

On Prem

(UI)

Stateful Infrastructure

(EC2, S3, AWS IAM, Dynamo DB)

Stateless Infrastructure

(Lambda, API gateway)

What? How? When?

Challenges Overcome

Ch

alle

ng

es

Ov

erc

om

eJavascript development and

realtime debugging

Out of Order Messages

Tagging Lambda for dynamic

rule-sets

15 second message read-filter

allows all arriving messages to

buffer.

Had to create an EC2 Instance to

proxy requests between lambda &

on-premises

Automated run-time selection

within code

• More Workflows

• Cloud Master Library

• Cloud Editing & Versioning

• Cloud Based Channel Origination

Playout & Distribution

Visual Effects & Editing Analytics

DAM & Archive

Digital Supply Chain

Publishing

OTT

Acquisition

Wh

at’

s N

ext

WANT TO WORK ON

THIS?

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

ChrisHintonVP, Product DevelopmentTurner BEST

ChrisFarrisEngineering Manager,Turner BEST

BrigitteMabryMgr, Content LogisticsTurner NATO

Thanks to:

Questions?

Remember to complete

your evaluations!

Related Sessions

Industry Pre-Day: Monday 11/28• MAE303 - Discovery Channel's Broadcast Workflows and Channel Origination on AWS

• MAE304 - High Performance Cinematic Production in the Cloud

• MAE301 - Accelerating the Transition to Broadcast and OTT Infrastructure in the Cloud

Regular Conference: Tuesday – Thursday 11/29 – 12/1• ARC406 - Encoding Artifacts to Emmy Awards: Taking on Terabyte-Scale, 1-Gbps, 4K Video

Processing in the Cloud

• CMP312 - Powering the Next Generation of Virtual Reality with Verizon

• CTD305 - Media Delivery from the Cloud: Integrated AWS Solutions for Premium Over the

Top (OTT) Content

• STG302 - Deep Dive on Amazon Glacier