Slide 1

LightSync: Unsynchronized Visual Communication over Screen-Camera Links Wenjun Hu Hao Gu Qifan Pu

Slide 2

2D barcodes everywhere

Slide 3

http://www.windowsphone.com/zh-cn (Linking to online information)

Slide 4

A video of barcodes instead Transmitting information (vs linking) E.g., information on Gibraltarian attractions E.g., movie show times for nearby theaters Receiver Transmitter

Slide 5

Visual communications on phones No need for additional connectivity Free, directional, lightweight links Optical software-defined radio Run as apps on commodity devices

Slide 6

Many challenges Frame synchronization Poor image quality Limited computation power on phones Aesthetics

Slide 7

In the rest of the talk Frame synchronization challenges LightSync design Implementation and evaluation

Slide 8

FRAME SYNCHRONIZATION CHALLENGES

Slide 9

Frame rate mismatch Screen side (TX) 30 fps video playback Fairly stable Camera side (RX) 15 30 fps in practice Variable Lighting conditions Device capability API/system factors High rate, steady Low rate, unsteady

Slide 10

Original frame Received frame patterns

Slide 11

Original frame Single frame 2-frame mix Mixing pattern varies by line Received frame patterns

Slide 12

2-frame mix Mixing pattern (& #components) varies by line Due to Rolling Shutter CMOS camera sensor scans one line (of pixels) at a time

Slide 13

Received frame patterns Original frame Single frame 2-frame mix 3-frame mix Consecutive frames indicating a missing frame Mixing pattern varies by line

Slide 14

Display frame rate C Frame capture rate C/2 C 2C Occasional missed frames 4-frame mixes 3-frame mix Occasional missed frames

Slide 15

Synchronized communication Display frame rate C Frame capture rate C/2 C 2C The only decodable pattern Frame synchronized Under-utilized and inefficient! Carries information But undecodable and dropped Cannot be used (of worst camera)

Slide 16

Unsynchronized communication Display frame rate (fps) 0 30 Works with any display and capture rates Higher camera capture rate, better performance Supported frame capture rate (fps)3015

Slide 17

Requirements Need to be able to decode mixed frames In-frame color tracking per line Need to guard against lost frames/lines Inter-frame erasure coding Essentially a temporal code across frames Works with any (monochrome) per-frame code

Slide 18

Our solution: LightSync Need to be able to decode mixed frames In-frame grayscale tracking per line Need to guard against lost frames/lines Inter-frame erasure coding Essentially a temporal code across frames Works with any (monochrome) per-frame code

Slide 19

LIGHTSYNC DEMO

Slide 20

LightSync Need to be able to decode mixed frames In-frame grayscale tracking per line Need to guard against lost frames/lines Inter-frame erasure coding

Slide 21

Per-line tracking: Mixing patterns 3 frames mixing 3 blocks mixing Bl or w per block 8 mixing possibilities ??? Frame 1 Frame 2 Frame 3 RXTX

Slide 22

Per-line tracking: Mixing patterns RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3

Slide 23

Per-line tracking: Mixing patterns Trivial cases RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3

Slide 24

Per-line tracking: Mixing patterns Easy cases RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3

Slide 25

Per-line tracking: Mixing patterns Tricky cases, to be tracked per line RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3

Slide 26

Code area Per-line tracking: Reference blocks Corner Scanning direction Tracking bar orientation Tracking bars

Slide 27

Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar 2 Tracking bar 3 Tracking bar 4 4 reference blocks per line

Slide 28

Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar 2 Tracking bar 3 Tracking bar 4

Slide 29

Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar 2 Tracking bar 3 Tracking bar 4 Decoding by comparison Reference colors of mixes

Slide 30

LightSync Need to be able to decode mixed frames In-frame grayscale tracking per line Need to guard against lost frames/lines Inter-frame erasure coding

Slide 31

Inter-frame coding Simple pairwise XOR Group of 3 original frames Coded frames

Slide 32

Inter-frame coding Looping display on screen Coded frames All original frames Camera stops frame capture after message decoded

Slide 33

PERFORMANCE

Slide 34

Implementation Current frame capture rate Inter-frame intervals

Slide 35

Performance Goal recap: Works with any display/capture rate combinations Metric: Effective frame capture rate Actual throughput scales with per-frame capacity

Slide 36

Schematic performance For a 20fps phone camera

Slide 37

Schematic performance For a 20fps phone camera Ideal Peaks at the camera rate Level at higher display rates Ideal

Slide 38

Schematic performance Peaks at ~ half the camera rate For a 20fps phone camera Conventional Drops to 0 at higher display rates Ideal

Slide 39

Schematic performance Conventional LightSync For a 20fps phone camera Peaks at the camera rate Slight drops at higher display rates Ideal

Slide 40

LightSync performance Nokia Lumia 920 (~30 fps)

Slide 41

LightSync performance Nokia Lumia 920 (~30 fps) HTC One X (~20 fps)

Slide 42

LightSync performance Nokia Lumia 920 (~30 fps) HTC One X (~20 fps) Huawei Ascend C8812E (~15 fps) Works with any display and capture rates Higher camera capture rate, better performance

Slide 43

Conclusion and outlook LightSync: Unsynchronized visual communication Primitive for similar scenarios Lots of potential in the space of visual communication

Slide 44

Thank you! wenjun@microsoft.com

Slide 45

Related work Temporal barcode design (inter-frame) Unsynchronized 4D Barcodes Basically slows down display, only for TX up to 10fps Spatial barcode design (per-frame layout) COBRA: Optimized for phone-to-phone PixNet: High-end camera, computationally intensive, offline processing Other work on visible light communications