methods and algorithms for automatic slicing of screen capture video using slide pdfs

.

YALMS: Review

.YALMS......

....stands forYetAnotherLectureManagementSystem

• basically, a class robot◦ robot: set of automation scripts and frameworks

• first presented inET研2013.07 04

• a long-term project with 4-5 main components -- see the roadmap at the end

04 M.Zhanikeev+1 "YALMS: Yet Another Lecture Management System..." IEICE ET研 (2013.09)

M.Zhanikeev/H.Koide -- [email protected] -- Methods and Algorithms for YALMS Video Annotation Robot 2/28...

2/28

.

YALMS: Overall Design

Step 1: Record and Stream

VA

SLecturer

Viewers

ContentServer

HTML5

HTML5Muxer

RecordScreenStreaming

Step 2: Annotate

Lecturer

S

Auto-annotate

C


3/28

.

YALMS: Annotation

VIDEO

V A S

Annotate

Pack/Encode Screen

VIDEO

V A S

V A Realtime Screen

VIDEO

S Mux

A V MISC. C

C

PNG

Replay Screen

VIDEO

A V MISC. C

Annotate

S

S

S The focus of this study YALMS Robot

S


4/28

.

Fuzzy Environments


5/28

.

Sources of Fuzziness

1. unusual screen sizes in presentation mode

◦ this PDF was created in16:9 mode to fit my notebook perfectly, will NOT fitothers

2. unpredictable or unavoidablemargins when saving your slides as PDF◦ PPTX! no way to avoid white margins in print◦ tex2pdf creates perfect PDFs! --- look the same in PDF and in presentation mode

3. slide animation◦ PPTX animation is really BAD! (confusing, difficult to print, ...)◦ latex/beamer creates sequences of pages for print -- same page number, many pages◦ the point is ... animation should look good both in print and on screen


6/28

.

Objectives and Assumptions


7/28

.

Objectives and Assumptions

.YALMS should be able.....

.... to retain automation capability in fuzzy environments

• ASSUMPTION 1: no rules for slide PDFs -- only the PDF formatitself

• ASSUMPTION 2: no rules for screen video -- black margins, wierdsizes -- all OK

• ASSUMPTION 3: no additional information is requested,

annotation should happen with only (1) and (2)


8/28

.

Tools and Actual Software

1. pdftk : a nice library for handing PDFs 07

2. ImageMagic : the best graphics library (basically free Photoshop) 08

3. VLC for screen capture, ffmpeg will also work

4. ffmpeg for video editing 06

5. all scripting is done in php -- suitable for both CLI and web server

07 "pdftk: The PDF toolkit" http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit (2013)

08 "ImageMagic Homepage" http://www.imagemagick.org/script/index.php (2013)

06 "Homepage of ffmpeg and ffserver tools" http://www.ffmpeg.org (2013)


9/28

.

Solutions


10/28

.

Solution 1: Simple Scene Detection

• the simplest way to detect scene detection is to extract all I-Frames 06

• createsmore images than actual scene changes, but redundancy is not aproblem

$ ffmpeg -i screen.mp4 -vf select="eq(pict_type¥,I)" -vsync 0 -an frames.%03d.png

$ stands for command prompt, all commands are single-line

06 "Homepage of ffmpeg and ffserver tools" http://www.ffmpeg.org (2013)


11/28

.

Solution 2: Trim the Black Margins

• deal with fuzzy margins and sizes using ImageMagic's

convert• OK to add a small border because trim does not cut everything anyway

• force frame resize

$ convert -colorspace gray -bordercolor black -border 2x2 -trim +repage -resize 1280x720! frames.004.png frames.004b.png



12/28

.

Solution 2: Trim Performance

Nothing to trim

Trim success!


13/28

.

Solution 3: Bring to Common Size• cannot compare without same size

$ convert -colorspace gray -bordercolor black -border 8x5 -resize 1280x720! -density 300x300 -quality 100 pages.002.pdf pages.002.png


$ convert -colorspace gray -bordercolor black -border 2x2 -trim +repage -resize 1280x720! frames.004.png frames.004b.png



14/28

.

Solution 3: Size Automation

• easy to find page size in command line, so, automation OK

platypus@platypus-PC /local/yalms.content$ identify -format "%w %h" frames.004.png1280 720

platypus@platypus-PC /local/yalms.content$ identify -format "%w %h" frames.004c.png1156 720



15/28

.

Solution 4: Compare using the AE Metric

• AE metric: simply the absolute pixel diff between two images

• alternatives: PNSR, MLSE ...

$ compare -colorspace gray -metric AE -fuzz 50% pages.002.png frames.004b.png diff.png

7021



16/28

.

Solutions: Example

Video frameVideo frame(trim, resize)

PDF page (border, resize)

Diff imageAE: 7019


17/28

.

Parameters and Performance


18/28

.

Performance: Experimental Setup

• a 1.5h class

• screen capture and slide PDF

• fuzzy: 16:10 screen size (black margin)

• fuzzy: many ALT-TAB switches between applications


19/28

.

Performance: Parameters and Metrics

• fuss: used by ffmpeg as comparison threshold

• pace : minimum frequency of page changes -- possible to remove jitter

• performance: annotation success


20/28

.

Performance: Frame Diffs

• generally, good

trend, becauseonly one framematches well

• not ideal performance

0 20 40 60 80 100Ordered list of frame diffs

1.2

2.4

3.6

4.8

6

log(

fram

e d

iff)


21/28

.

Performance: Results

• thickness:

fuss value(between 10% and 90%)

• changing fuss can helpsometimes

• some cases arepersistentfailures

0 2 4 6 8 10 12 14 16 18Page sequence

0

0.5

1

1.5

2

2.5

3

3.5

log(

tim

e d

iff)


22/28

.

Performance: Room for Improvement?

• multi-fuss comparison, pick the best

• when failing consistently, just setmarker between two neighboringsuccesses◦ mark it as failed and ask for human


23/28

.

YALMS.annotation: The Robot


24/28

.

YALMS.annotation : The Robot

screen video

SlidesPDF

LaTeX/BeamerKeynote

PowerPoint…

VLCffmpeg

I-Frames I-Frames

pages PNGs

ANNOTATION

slices

PNGstimestamps

End productscenes cleanup

splitpdf2png

03 "YALMS Annotation Robot (software behind this paper)" https://github.com/maratishe/yalms.annotation (2013)


25/28

.

YALMS.annotation : The Script

$ php robot.phpPURPOSE: to process two files: SLIDES PDF and

SCREEN CAPTURE MOVIEOUTCOME: ANNOTATIONS for the movie -- times for each

PDF page in the movie[pace] minimum time gap between page changes

-- will ignore fickers below that threshold[wdir] full path to working directory[screen] filename of the screen capture movie

-- should be in $wdir[slides] filename of the slides PDF file -- should be in $wdir$


03 "YALMS Annotation Robot (software behind this paper)" https://github.com/maratishe/yalms.annotation (2013)


26/28

.

YALMS: The Roadmap

YALMS: Intro

YALMS.annotation

2013.07・ET研

2013.09・ET研

YALMS.streaming2014.01・ET研

YALMS.hiperf2013.10・ITS-MMS研

ET研

その他YALMS.cloud

2013.11・IN研

YALMS.format

YALMS.done!

DASH Adaptive HTML5

PDF slices

VM sync groupdrive

SSDmulticore


27/28

.

That’s all, thank you ...


28/28

.

[01] (2013)The enPiT Projecthttp://www.enpit.jp

[02] (2013)YALMS Project Homehttps://github.com/maratishe/yalms

[03] (2013)YALMS Annotation Robot (software behind this paper)https://github.com/maratishe/yalms.annotation

[04] M.Zhanikeev+1 (2013.09)YALMS: Yet Another Lecture Management System...IEICE ET研

[05] (2013)Homepage of edubase Stream at NIIhttp://stream.edubase.jp


28/28

.

[06] (2013)Homepage of ffmpeg and ffserver toolshttp://www.ffmpeg.org

[07] (2013)pdftk: The PDF toolkithttp://www.pdflabs.com/tools/pdftk-the-pdf-toolkit

[08] (2013)ImageMagic Homepagehttp://www.imagemagick.org/script/index.php


28/28

methods and algorithms for automatic slicing of screen capture video using slide pdfs

Technology

yalms video annotation

annotation video vas

class robot robot

study yalms robot s

video editing

c c png replay screen

black margins

command line