moving your media supply chain to the aws cloud

MediaSupplyChainsinAWSTermsUsmanShakeel |AmazonWebServices

Playout&Distribution

VisualEffects&Editing Analytics

DAM&Archive

DigitalSupplyChain

Publishing

OTT

Acquisition

AWSMediaSegments

Playout&DistributionElemental,EC2,S3/Glacier

EFS,CloudFront

VisualEffects&EditingEC2,S3,EFS,EBS,GPU,DX

AnalyticsKinesis,DynamoDBAML,RedShift,EMR

DAM&ArchiveSnowball,S3,GlacierEC2,DX,LambdaDynamoDB,RDS

DigitalSupplyChainElemental,EC2,S3/GlacierCloudFront,EFS,Snowball

GPU,DX

PublishingEC2,S3,Glacier,ELB,CloudFront

OTTElementalEC2,S3

CloudFront

AcquisitionElemental,EC2,

S3/GlacierCloudFront,Snowball,DX

AWSMediaSegments

MediaSupplyChains

Playout&Distribution

Production/PostProduction B2B/CinematicDelivery

DAM&Archive

DigitalSupplyChain

Publishing

OTT

ContentAcquisition

Ingestion

ManualQC

Auto-QC

MetadataExtraction

Transcoding

DeliveryPackaging

Clips/Edits

Complexworkflow/StatemanagementBursty sub-workflowtasks(transcoding/auto-QC,Packaging)ElasticLicensingContentmovementbetweenworkflowtasks(acommonrepository)DifferentdeliveryformatsforeachaffiliateanddownstreamManualtasks– (contentmovementorinteractivity)

CMS/DAM

Normalization

Normalization

All-InCloud-EnabledDominoEffect

Playout&DistributionLiveTV

Production/PostProduction

EditorialDailiesPost

SocialMediaMarketing/Promo

B2B/CinematicDelivery

DPP,IMF,NABA

DAM&ArchiveActiveArchiveGoldenCopyPreservation

DigitalSupplyChain

Publishing

OTTLive-To-VOD

AVOD,SVOD,TVODCatchupTVCloudPVR

ContentAcquisitionLiveFile

Ingestion

ManualQC

Auto-QC

MetadataExtraction

Transcoding

DeliveryPackaging

Clips/Edits

CMS/DAM

Ingest Store Edit/ManualTasks Process

PUSHORPULLCONTENT&ARTIFACTS

ACCESSSOURCEFILESLOCATEDINS3

CREATE,EDIT,QC&COLLABORATEINTHECLOUD

SCALEOUTONELASTICCAPACITYFORPROCESSING

(Transcoding,QC,Packaginganddistribution)

MediaproductionanddeliverycompaniesareleveragingAWStoacceleratetheirmediasupplychains,ingestion,storage,editing/manualQCofcontentandprocessingwithhighlyscalablecloudcomputingandstorage.

UsingCloudforMediaSupplyChains

Ingest

PUSHORPULLCONTENT&ARTIFACTS

AWS Services

• Amazon S3 Transfer Acceleration: Fast data transfers• AWS Direct Connect: Dedicated network connectivity• AWS Snowball: Petabyte scale data transfer• Amazon Cloudfront: Global content delivery network• Amazon Storage Gateway: On-prem to cloud storage connectivity• Amazon Marketplace: One-click partner software deployments• Elemental: Content contribution from on-premises encoders

Partners

MediaSupplyChains

AWS Services

• Amazon S3/SIA: Highly scalable, durable object storage• AWS Storage Gateway: On-prem to cloud storage connectivity• Amazon Glacier: Secure, durable, petabyte-scale data archival• Amazon EFS: Simple, elastic, scalable file system• Amazon EBS: Persistent block storage

Partners

Store

ACCESSSOURCEFILESLOCATEDINS3

MediaSupplyChains

Edit/ManualTasks

CREATE,EDIT,Clips,ManualQC&COLLABORATEINTHECLOUD

AWS Services

• Amazon EC2: Scalable CPU/GPU computing capacity• Amazon Workspaces: Virtual desktops in the cloud• Amazon Appstream: Remote Application Streaming• Amazon EFS: Simple, elastic, scalable file system• Amazon EBS: Persistent block storage• Amazon S3: Highly scalable, durable object storage

Partners

MediaSupplyChains

Process

SCALEOUTONELASTICCAPACITYFORPROCESSING

AWS Services

• Amazon EC2: Scalable CPU/GPU/Spot computing capacity• Amazon EFS: Simple, elastic, scalable file system• Amazon EBS: Persistent block storage• Amazon S3: Highly scalable, durable object storage• Elemental Cloud: On-demand, scalable video processing

Partners

MediaSupplyChains

AWS Services

• Amazon S3/SIA: Highly scalable, durable object storage• AWS Storage Gateway: On-prem to cloud storage connectivity• Amazon Glacier: Secure, durable, petabyte-scale data archival• Amazon EFS: Simple, elastic, scalable file system• Amazon EBS: Persistent block storage

Partners

AssetManagement

MetadataforAssetsandsupplychainworkflow

MediaSupplyChains

INFRASTRUCTURE

TaskSpecificComputeInstancesComputeIntensiveIntelES-2666v3(Haswell)optimizedspecificallyforEC2

MemoryIntensiveLowestpricepointperGiB ofRAM

GPUs1,536CUDAcores4GBofvideomemory

EnhancedNetworkingHigherPPS,Lowernetworkjitter,lowlatency

IOIntensiveSSDStorage,EBSOptimized

HighStorage24x2000GiB perinstance

AmazonS3

SQSQueues(orchestratedbyAmazonSWF)

M4

M3

C4

C3 I3G2

R4

P2 D2R3X1

Reserved

Make a low, one-time payment and receive a significant discount on the hourly charge

For committed utilization

On-Demand

Pay for compute capacity by the hour

with no long-term commitments

For spiky workloads, or to define needs

Spot

Bid for unused capacity at a Spot Price -

fluctuates based on supply & demand

For time-insensitive or transient workloads

ConsumptionModels

AWS Storage Services Maturity

AmazonEFS

File

AmazonEBS AmazonEC2InstanceStore

Block

AmazonS3 AmazonGlacier

Object

DataTransfer(Ingest/Egress)

AWSDirectConnect AWSSnowball ISVConnectors AmazonKinesisFirehose S3VPCEndPoint

AWSStorageGateway

S3– InfrequentAccess

Events

S3EventNotifications

S3TransferAcceleration

PostProductiontasksNLE/Review&Approve/VFX(Hybrid,onpremisesorincloud)

RawAssetsSecondaryCopy

AmazonGlacierinanotherRegion

RawAssetsPrimaryCopyAmazonGlacier

Finished(GoldenCopy)ArchivePrimaryCopyAmazonGlacier

AssetM

anagem

ent(DA

M/M

AM)

(Hybrid

,onprem

iseso

rinclou

d)

ActiveArchiveAmazonS3(Standard)

ActiveArchiveAmazonS3

(InfrequentAcccess)

Secondary&TertiaryCopiesAmazonGlacierinanotherRegion

Asecondaryoffsitecopyoftheactivearchive– usuallycoldUsedforDRpurposes

Med

iaSup

plyCh

ain

(Hybrid

,onprem

iseso

rinclou

d)

B2BDe

liveryoverDXorInternet

AmazonGlacieringest

Affiliates/ThirdPartyVendors

ingest

AWSDirectConnect

AWSSnowball

S3TransferAcceleration

(PrimaryCopy)AWSRegion1

(Secondary&TertiaryCopies– optional)AWSRegion2oron-premises

Glacier

S3InfrequentAccess

S3StandardLifecycleM

gmt

ProductionArchive

Availability Zone A

Private subnet Private subnet

AWS region

VirtualPrivateGateway

ContentValue-addService

ContentValue-addService

Availability Zone B

Amazon S3

VPC

VPNconnection

VPCEndpointsNoIGWNoNATNopublicIPsFreeRobustaccesscontrol

Customernetwork

Access Control with S3 VPC Endpoint

corporate data center AWS cloud

users

Content

Servers

disk

tape storage

Processing Layer

Amazon S3

Amazon EBS

Amazon Glacier

KMS/HSMClient side

encryption

roleIAM

EncryptedContent

roleDirectConnect

S3VPC

End

point

HybridMediaNetwork

STORAGEPERFORMANCE

EBSVolumesSSDBased HDDBased

gp2GeneralPurposeSSD

io1ProvisionedIOPSSSD

st1ThroughputOptimizedHDD

sc1ColdHDD

Baseline: 3IOPSperGBupto10,000Burst:3000IOPS(forvolumesupto1TB)Throughput:160MB/sLatency:SingledigitmsCapacity:1GBto16TB

UsageBootVolumesBursty workloads(Rendering)Databases(Transactional)Lowlatencyoperations(Streaming)

Baseline: 100IOPSupto20,000Throughput:320MB/sLatency:SingledigitmsCapacity:4GBto16TB

UsageSustainedIOPSCriticalApplicationswithIOPSGuarantee(noburst)

Baseline:40MB/sperTBupto500MB/sBurst:250MB/sperTBupto500MB.sCapacity:500GBto16TB

UsageLargeBlock,highthroughputsequentialTranscoding,QC,PackagingRendering

Baseline:12MB/sperTBupto192MB/sBurst:80MB/sperTBupto250MB.sCapacity:500GBto16TB

UsageSequentialLogging,ActiveArchive/Backup

ObjectstoreasProcessingtier

Partition1

Partition2 Partitionn

AnS3BackendStorage

Auto-scaled/HAFront-end

TCP

DistributedKey-map

Optimizations

ReplicationacrossmultipleAvailabilityZones

NoFS– everythingGET/PUT/DELETE• Writeonce– Readmany• Globallyuniquenamespaces(objectnamesuniquewithinabucket)

RequestsPerSecond(>300)• Partitioning• RandomPrefixPutsvs.Gets• CDN– GETheavyworkloadsNetworkThroughput• TCPWindowScaling• TCPSelectiveAcknowledgements• S3SessionUsage• Multi-partUploads• BufferedWrites• RangeGETsConsistency/CommitLatency• ReadafterWrite• ObjectsreplicatedacrossmultipleAZs• Notsuitabletodayforworkloadswith<MSlatency/IOrequirement

Client(s)

Client-sidebandwidth

Elastic File System (EFS) • DesignedtosupportPeta-Bytescalefilesystems• Throughputscaleslinearlytostorage

– Burst:100MB/secperTBofstoragefor12hourseachday– Baseline:50MB/secperTBofstoragecontinuously

A10TBfilesystemcanburstupto1GB/sfor12hourseachdayanddrive500MB/secaggregateread/writethroughputcontinuously.

• SamelatencyspecacrosseachAZ• ThousandsofconcurrentNFSconnections• WorksgreatforLargeI/Osizes• Payforonlywhatyouusenotwhatyouprovision• Managedwithmulti-copydurability

RenderFarms

Bigdataanalytics

TranscodeFarms

Homedirectories

Contentmanagement

WebservingMetadata-intensive

jobs

HighthroughputandparallelI/O

LowlatencyandserialI/O

WorkloadSpecificStorageObject Store(AmazonS3)

BlockStore(Amazon EBS)

SharedFS(Amazon EFS)

Performance

Per OperationLatency Variable Lowest,Consistent LowConsistent

ThroughputScale Multiple GBspersec Single GBspersec MultipleGBsper sec

DataAvailability/Durability

AcrossMulti-AZ Redundantwithin anAZ AcrossMulti-AZ

Characteristics

Access 1-1000’sofClients AsingleEC2 instance 1-1000’sofClients

UseCases

Active ArchiveandlifecycletoColdArchiveMedia Processing(Transcoding,rendering,QC,Farms)dependingonApplicationconformity

DatabasesBoot VolumesMedia Processing(Transcoding,rendering,QC,Farms)SingletonStreaming

DataAnalyticsMedia Processing(Transcoding,rendering,QC,Farms)DistributedContentManagementWebServing/Publishing

SOLUTIONS

AWSImport/Export

S3multi-partAPIForSource,Renditions,MetadataSidecarFiles

Ingest

AWSElasticBeanstalk

DAMWebService

CloudFrontDistribution

AmazonDynamoDB

AmazonCloudSearch

DAMCatalog

Rendition WorkersEC2ASG

AmazonSNSTopic

AmazonSQSQueueRenditionJobs

AmazonSQSQueueMetadataProcessingJobs

S3Notification

ElasticTranscoderProxy/

Thumbnail/RenditionsGeneration

Processing

Metadata WorkersEC2ASG

SimpleMediaSupplyChain

AWS Import/ Export

S3 multi-part APIS3 as backend storage for Content Files

accessible to other processing tasks

Amazon Elastic Transcoder

S3 Notification

Trigger a Lambda Function to Start a transcoding job

Ingest

S3 Notification

Lambda function to generate a signed

URL to share the file

Update CMS or Metadata

Serverless MediaSupplyChain(concept)

§ Expand computecapacity of Manager node(s)

§ 100ms billing & short run-time

§ Ideal for chunk storage, pre- & post-processing

§ Don’t forget binary encapsulation!

https://github.com/binoculars/aws-lambda-ffmpeg

LambdaPreandPostProcessing

Ahigh-levelMediaSupplyChaininthecloud(concept)

EC2,ETScanrequestthedata-keyonbehalfofcustomerProvideCPKforS3encryptionatrest

AWSImport/Export

S3multi-partAPIAmazonS3ForSource,Renditions,Metadata,SidecarFiles

Ingest

AWSElasticBeanstalk

CloudFrontDistribution

DynamoDB CloudSearch

S3Notification

Content Processing

AmazonSWF,SQS,SNSandAWSLambdacanbeusedtoorchestratetheprocessingworkflowtasks

CloudFrontDistribution

UserLogs

AmazonDynamoDB

AmazonRedshift

Users Connected Devices

Analytics and Recommendations Engine

AmazonEMR

AmazonDataPipeline

ElasticTranscoder

EC2ASGRendition Workers

EC2ASGMetadata Workers

MetadataProc queueRenditionsQueue

Amazon Elastic Transcoder

Rights Management

EC2ASGELB

DRM WrapEncryptPackageWatermarkQC

EC2ASG

AmazonSNSTopic

DAM Web Service

PersonalizationRecommendations

EC2ASG RDS

Front End Web Application

DAM CatalogContentArchive

AmazonGlacier

Key Management Service Content owner provides the master key

BI & ReportingUsers Web Browser

Content Owner

IMFPACKAGINGStack:s3-ta/snowball/?Partners:teradek /nuke(vr/360)/?

Cloud Based Editing

• NVIDIAGPUbasedEC2instances• Teradici PCoIP• AmazonAppStream/NiceDCV• AmazonWorkspaces

Artistapplications

Artist workstation(Dumb Client)

Remote Application running on a G2 instance

G2

ThankYou!