enhancing media workflows with amazon machine learning · amazon ml : broadest set of capabiliies...

IABM Copyright 2019@THEIABM www.theiabm.org

Enhancing Media Workflows with Amazon Machine

LearningShweta Jain

APAC Head of Business Development, Media & Entertainment

Amazon Web Services

IABM Copyright 2019@THEIABM www.theiabm.org

The New Media and Entertainment Reality

Viewer

Expectation

Content discoverability, recommendations,

personalization

Business

Opportunity

Learn about viewers, use the data to

increase engagement

Customer

Obsession

Customer satisfaction increases brand

value and drives revenue

IABM Copyright 2019@THEIABM www.theiabm.org

Challenges that have so-far prevented organizations from adopting Machine Learning quickly?

ML expertise

is rareBuilding and scaling ML

technology is

hard

Deploying and operating

models in production is

time-consuming

and expensive

Lack of cost-

effective,

easy-to-use, and

scalable ML services

IABM Copyright 2019@THEIABM www.theiabm.org

We set out to build aMachine Learning platform that is

accessible to every Developer,,Data Scientist & IT Professional

IABM Copyright 2019@THEIABM www.theiabm.org

Amazon ML : Broadest set of capabilities

M L F R A M E W O R K S

I N F R A S T R U C T U R E

A I S E R V I C E S

R E K O G N I T I O N

I M A G E

A M A Z O N

P O L L Y

A M A Z O N

T R A N S C R I B E

A M A Z O N

T R A N S L A T E

A M A Z O N

C O M P R E H E N D

A M A Z O N

L E X

R E K O G N I T I O N

V I D E O

Vision Speech Chatbots

A M A Z O N S A G E M A K E R

B U I L D T R A I N

A M A Z O N

F O R E C A S T

A M A Z O N

T E X T R A C T

A M A Z O N

P E R S O N A L I Z E

D E P L O Y

Pre-built a lgorithms & notebooks

Data label ing (G R O U N D T R U T H )

One-cl ick model training & tuning

Opti mizati on (N E O )

One-cl ick deployment

& hosti ng

M L S E R V I C E S

F ra m e w o r ks I nt e r fa ce s I n f ra st r u c t u re

E C 2 P 3 E C 2 C 5 F P G A s G R E E N G R A S S E L A S T I C

I N F E R E N C E

Reinforcement learningAlgori thms & models

( A W S M A R K E T P L A C E

F O R M A C H I N E L E A R N I N G )

Language Forecasti ng Recommendati ons

IABM Copyright 2019@THEIABM www.theiabm.org

Sample ML Use Cases in M&EAutomate creation of rich metadata (object, scene, activity, faces etc.), extracted from

audio visual content and integrated into digital and media asset management systems

AMAZON REKOGNITION, AMAZON TRANSCRIBE, AMAZON COMPREHEND

Media Metadata

Tagging

Closed Captioning

Detect potentially inappropriate content to avoid compliance issues in global markets, and

to increase brand safety for advertisers

AMAZON REKOGNITION, AMAZON TRANSCRIBE, AMAZON SAGEMAKER

Automated

Compliance Marking

Identify objects and emotion in content that enables users to integrate personalized ads

into subscriber video streams

AMAZON REKOGNITION, AMAZON SAGEMAKER, AWS MEDIATAILOR

Ad Personalization

Automated captions, transcription and translation of audio content

AMAZON TRANSCRIBE, AMAZON TRANSLATE

IABM Copyright 2019@THEIABM www.theiabm.org

Sample ML Use Cases in M&EIdentify and path player activity in the field or identify the precise timecodes when actors

enter and leave a scene, both visually and in spoken dialog (off-screen voice)

AMAZON REKOGNITION, AMAZON TRANSCRIBE

Player/ Actor

identification &

activity pathing

Translate transcripts and metadata. Improve localization workflows and search experience

AMAZON TRANSLATE

Language Translation

Analyze disparate data and enable the consumer to make personalized content choices and

the business to predict behavior

AMAZON PERSONALIZE, AMAZON SAGEMAKER

Content

Recommendations

Detect and pixelate faces captured incidentally to preserve the privacy of non-persons of

interest in news feeds and security footage

AMAZON REKOGNITION, AMAZON SAGEMAKER

Automated

Redaction

IABM Copyright 2019@THEIABM www.theiabm.org

Media Enrichment and Analysis

IABM Copyright 2019@THEIABM www.theiabm.org

Automated Metadata GenerationVideo

Rekognition

Video

Search Engine:

Amazon ElasticsearchAsset Management System

1. Video is uploaded and

stored to the Data lake

2. Create metadata for

celebrities, emotions, scene

time, objects, voices in video

3. The output is sent to the

digital/media asset

management system and be

available in a search engine

Dynamic search indexing

Transcribe

Data Lake on AWSStorage | Archival Storage | Data Catalog

IABM Copyright 2019@THEIABM www.theiabm.org

AI based taggingBackground

• Over 200,000 hours of content

• Only half of content is manually tagged

• How can we enrich our metadata in AWS?

• How can we unleash the value of content we already

own once in AWS?

Challenge

• Large scale video library

• High accuracy required

• Limited budget

• Ability to extract from video

• Keep up with daily increase in content

Results

• Solution developed within three weeks

• Live video frame based analysis

• Established, searchable baseline archive

• All content is now tagged and indexed

• Over 99,000 faces indexed and searchabl

• Saved ~9,000 hours a year in manual ef

IABM Copyright 2019@THEIABM www.theiabm.org

Event Broadcasting Enrichment

Facial recognition

combined with

additional

metadata – Bios,

News, Fashion,

Emotions.

Matching trained

by humans

IABM Copyright 2019@THEIABM www.theiabm.org

Tag, filter, and redact inappropriate content

Person 99.2% Gun 84.6% Handgun 73.5%

Drink 96.4% Alcohol 80.1% Wine 69.9%

IABM Copyright 2019@THEIABM www.theiabm.org

Formula1: Fan EngagementBackground

• Formula 1 is the worlds premier auto racing championship.

• Since the first gran prix in 1950, Formula 1 engages with

500 million fans watching 21 global races annually.

Challenge

120 car telemetry sensors create 3Gb of data per second,

how can Formula 1 bring analytics and insights to car data that

drives advances in the sport and an deeper fan experience

delivered by their next generation OTT/VOD platform. https://aws.amazon.com/f1insights/

Use of AWS

• AWS Kinesis streams data into S3 in real-time

• Amazon Rekognition, Amazon SageMaker, & Amazon

Transcribe analyze race data

• AWS Media Services powers next generation video platform

Results

• Deeper fan experience

• Monetize data and direct to consumer video platform

IABM Copyright 2019@THEIABM www.theiabm.org

Sports media tagging

IABM Copyright 2019@THEIABM www.theiabm.org

Personalization

IABM Copyright 2019@THEIABM www.theiabm.org

Users expect every interaction to be personalized

Activity & Content

Recommendation

Search

Personalization

Personalized

NotificationsEmails

IABM Copyright 2019@THEIABM www.theiabm.org

Personalization: multiple hard problems

Popularity Trap Naïve models give recommendations similar to popular items

Cold StartsNew users should get relevant recommendations, new items should show in recommendations

ScaleRecommendations should scale across millions of users and items

Real-TimePersonalization must be responsive to the changing user intent

Custom modelsPersonalization models must accurately reflect business context and user behavior

IABM Copyright 2019@THEIABM www.theiabm.org

Introducing Amazon PersonalizeBased on the same technology used at

amazon.com

IABM Copyright 2019@THEIABM www.theiabm.org

Amazon Personalize - Workflow

Prepare your data,

then upload with the

Amazon Personalize

API

Choose one of our

algorithms or tell AutoML

to find the best fit

Modify your code Retrain continually to

improve the model

IABM Copyright 2019@THEIABM www.theiabm.org

Real-time data can be consumed by Amazon Personalize

Historical user

activity

User

attributes

Item

catalog

Real-time data

Mobile

SDKs

(coming soon)

JavaScript SDK

Amazon S3

bucket

Server-Side SDKs

Offline data

IABM Copyright 2019@THEIABM www.theiabm.org

30% of page views on Amazon are from recommendations

IABM Copyright 2019@THEIABM www.theiabm.org

How we can help

Discovery Workshops - ML Solutions Lab – ML ProServBrainstorming

Custom modeling

Training

Work side-by-side with Amazon experts

Partner Ecosystem

AWS Media Solutions

https://aws.amazon.com/media/solutions/

IABM Copyright 2019@THEIABM www.theiabm.org

Thank You

imshweta@amazon.com