data management for video analyticsdata management for video analytics brandon haynes, maureen daum,...
TRANSCRIPT
7/26/2019
1
1
Data Management for Video Analytics
Brandon Haynes, Maureen Daum, Amrita Mazumdar, Magdalena Balazinska, Luis Ceze, & Alvin Cheung
2
Video data is everywhere.
3
Existing systems treat video data like it’s the
20th century
4
Hei
ght
in p
ixel
sWidth in pixels
5
Many recent video applications
require jointly querying multiple cameras,
reasoning about position and orientation, or
querying complex metadata.
6
1 2
3 4
5 6
7/26/2019
2
7
SELECT Metadata.Location FROM Cameras
WHERE ‘James Bridle’ IN Metadata.Name
8
A Video Data Management Benchmark
Visual RoadA Database System for
Virtual & Augmented Reality Video Applications
Optimized storage and retrieval of video data
Video
File SystemMetastoreEfficient querying of rich
video content
(Maureen Daum)
Data Management for Video Analytics Modern
LightDBA Database System for
Virtual & Augmented Reality Video Applications
How would you write this application?12
#include <libavformat/avformat.h>
20th century Imperative Code LightDB Query
source = Scan(“kittens”)detection = source.Map(𝑑𝑒𝑡𝑒𝑐𝑡) result = Union(source, detection)result.Save(“output.mp4”)
7 8
9 10
11 12
7/26/2019
3
13
source = Scan(“kittens”)detection = source.Map(𝑑𝑒𝑡𝑒𝑐𝑡) result = Union(source, detection)result.Save(“output.mp4”)
LightDB Query:
14
Logical Plan
Scan
Map(Grayscale)
EncodeHEVC
Scan
Map
Union
Save
source = Scan(“kittens”)detection = source.Map(𝑑𝑒𝑡𝑒𝑐𝑡) result = Union(source, detection)result.Save(“output.mp4”)
15
GPUOverlay
GPUOverlay
ScanSingleFile
SaveSingleFile
GPUMap GPUMap GPUEncode GPUEncode
GPUTee GPUTee GPUToCPU
GPUToCPU
GPUQueuedDecode
Homomorphic Union
Physical Plan
0
15
30
45
Timelapse Venice Coaster Cats
FPS
LightDB Ffmpeg Scanner OpenCV
(Light Field)
Object Overlay AR Application
Brandon Haynes [email protected]
Key Features:• Data management system for VR & AR video applications• Unified data model for panoramic (360°) and light field video• Declarative queries with automatic optimization
Key Contributions:• ∼ Τ1 10 lines of code• Up to 4×performance for real-world workloads• Reduced client bandwidth & power requirements Scan(“LEGOS”)
.Map(GRAYSCALE)
.Store(“GRAYLEGOS”)
A Database System for Virtual & Augmented Reality
Video Applications
A Video Data Management Benchmark
Visual RoadOptimized storage and retrieval of video data
Video
File SystemMetastoreEfficient querying of rich
video content
(Maureen Daum)
Data Management for Video Analytics Modern
13 14
15 16
17 18
7/26/2019
4
19
The performance of video systems is evaluated
like it’s still the 20th century
20
Video System # Distinct Videos
LightDB (2018) 4
Chameleon (2018) 5
BlazeIt (2018) 6
NoScope (2017) 7
Focus (2018) 14
DeepLens (2019) ∼16
Scanner (2018) >100 (only 14 joined)
# Distinct videos performance tested by system
21
UA-DETRAC Dataset (2015)10 hours, 24 LocationsVideo by Wen et al.
Manually annotated with 1,210,000 hand-
drawn boxes!
22
Type Benchmarks Test Data
Video (Visual Road) Ad Hoc
OLTP (TPC-H) Synthetic
OLAP (SSB, DWEB) Synthetic
Streaming (Linear Road, DTDW) Synthetic
NoSQL (YCSB) Synthetic
Graph (LDBC) Synthetic
Privacy (SDV) Synthetic
Synthetic
23
Benchmark Queries & Verifier
Video Dataset Generator
Hyperparameters: scale, resolution,
duration, seed Configure
GenerateSynthetic Dataset
Commercial Gaming Engine
Autonomous Driving Simulator
Video Data Management System
Execute
Q1
Q2
Q2b
Q2c
Q2d Q
3
Q4
Q5
Q6a
Q6b Q
7
Q8
Q9
Q10
Verify
PregeneratedDatasets
24
19 20
21 22
23 24
7/26/2019
5
25 26
Automatic ground truth generation
27
Traffic Camera 𝑉𝑗 : Result for query instance 𝑖:
(4 × 𝑆𝑐𝑎𝑙𝑒 query instances)
⋮
Query 7: Object Detection
28
0
1
2
Q2(a)
0
8
16
Q2(d)
0
6
12
Q2(b)
0
8
16
Q3
0
1
2
Q1
0
36
72
Q2(c)
Scale Factor
0
1
2
0 2 4 6 8
Q5
0
2
4
0 2 4 6 8
Q6(a)
0
4
8
0 2 4 6 8
Q4
`
29
visualroad.uwdb.ioKey Features:
• Video data management benchmark• Synthetic dataset generation• Unlimited scale, resolution, duration, and overlap• Extensible suite of computer vision, VR, and microbenchmarks• Applicable to both general and specialized video DBMSs
A Database System for Virtual & Augmented Reality
Video Applications
A Video Data Management Benchmark
Visual RoadOptimized storage and retrieval of video data
Video
File SystemMetastoreEfficient querying of rich
video content
(Maureen Daum)
Data Management for Video Analytics Modern
25 26
27 28
29 30
7/26/2019
6
31
visualroad.uwdb.io
Key Features:• Video data management benchmark• Synthetic dataset generation• Unlimited scale, resolution, duration• Extensible suite of computer vision, VR, and
microbenchmark queries
Key Features:
• DBMS for VR & AR video applications• Unified data model• Declarative queries• Automatic optimization
lightdb.uwdb.io
A Database System for Virtual & Augmented Reality
Video Applications
A Video Data Management Benchmark
Visual RoadOptimized storage and retrieval of video data
Video
File SystemMetastoreEfficient querying of rich
video content
(Maureen Daum)
Data Management for Video Analytics Modern
Towards Efficient Querying of Rich Video Content
Maureen Daum
Motivation• We want to enable rich, content-based queries over video data
• Existing systems optimize running object detection over videos• As a result, they focus on simple queries only
• We want to use this metadata to enable more complex queries
Metadata
• Object labels
• Weather conditions
• Descriptions
• License plate numbers
• Aggregates
35
SELECT pixels FROM video WHERE dog
36
31 32
33 34
35 36
7/26/2019
7
Overlay the dog pixels from video 1 onto the background in video 2
37
Select sequences of frames that contain increasing numbers of cats
38
Metadata Model
t
y
x
Label Volume
Dog 𝑡0 , 𝑡1 , 𝑥0 , 𝑥1 , [𝑦0 , 𝑦1]
39
Executing Queries
40
• Videos are stored in a compressed format
• Encoding and decoding are expensive operators
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Select frames with bicycles
Select frames with buses
Select frames with people
Draw boxes over people
Percent of Query Time Spent in Encode or Decode
Encode and decode
SELECT frames FROM video WHERE dog
This Photo by Unknown Author is licensed under CC BY-SA 41
SELECT frames FROM video WHERE dog
This Photo by Unknown Author is licensed under CC BY-SA 42
37 38
39 40
41 42
7/26/2019
8
SELECT frames FROM video WHERE dog
This Photo by Unknown Author is licensed under CC BY-SA 43
SELECT frames FROM video WHERE dog
This Photo by Unknown Author is licensed under CC BY-SA 44
SELECT frames FROM video WHERE dog
This Photo by Unknown Author is licensed under CC BY-SA 45
SELECT frames FROM video WHERE dog
This Photo by Unknown Author is licensed under CC BY-SA 46
SELECT frames FROM video WHERE dog
This Photo by Unknown Author is licensed under CC BY-SA 47
SELECT frames FROM video WHERE dog
This Photo by Unknown Author is licensed under CC BY-SA 48
43 44
45 46
47 48
7/26/2019
9
SELECT frames FROM video WHERE dog
This Photo by Unknown Author is licensed under CC BY-SA 49
SELECT frames FROM video WHERE dog
This Photo by Unknown Author is licensed under CC BY-SA 50
This Photo by Unknown Author is licensed under CC BY-SA 51
SELECT frames FROM video WHERE dog
Random access point
Encoded as a delta
This Photo by Unknown Author is licensed under CC BY-SA 52
SELECT frames FROM video WHERE dog
Random access point
Encoded as a delta
This Photo by Unknown Author is licensed under CC BY-SA 53
SELECT frames FROM video WHERE dog
Random access point
Encoded as a delta
This Photo by Unknown Author is licensed under CC BY-SA 54
SELECT frames FROM video WHERE dog
Random access point
Encoded as a delta
49 50
51 52
53 54
7/26/2019
10
This Photo by Unknown Author is licensed under CC BY-SA 55
SELECT frames FROM video WHERE dog
Random access point
Encoded as a delta
This Photo by Unknown Author is licensed under CC BY-SA 56
SELECT frames FROM video WHERE dog
Random access point
Encoded as a delta
57
SELECT pixels FROM video WHERE dog
58
SELECT pixels FROM video WHERE dog
59
SELECT pixels FROM video WHERE dog
60
SELECT pixels FROM video WHERE dog
55 56
57 58
59 60
7/26/2019
11
61
SELECT pixels FROM video WHERE dog
62
SELECT pixels FROM video WHERE dog
Add Metadata
Putting Things Together
Similar to DB cracking, incrementally partition the video and add indices
Original Video
Execute Query
Update Layout
63
Ongoing Work
• Measure the effectiveness of selection optimization techniques
• Investigate optimization techniques for more compute-heavy queries
• Determine how to effectively layout videos with a lot of metadata• Possibly store multiple versions of a video with different layouts
64
Conclusion
• Deep learning opens the door to rich queries over video data
• Videos are large and slow to process
• Database techniques can accelerate such queries• Partitioning
• Indexing
• Incremental physical tuning
• Indexing must be balanced with maintaining reasonable storage sizes
61 62
63 64
65