Download - LightSync: Unsynchronized Visual Communication over Screen-Camera Links Wenjun Hu Hao Gu Qifan Pu
Transcript
- Slide 1
- LightSync: Unsynchronized Visual Communication over Screen-Camera Links Wenjun Hu Hao Gu Qifan Pu
- Slide 2
- 2D barcodes everywhere
- Slide 3
- http://www.windowsphone.com/zh-cn (Linking to online information)
- Slide 4
- A video of barcodes instead Transmitting information (vs linking) E.g., information on Gibraltarian attractions E.g., movie show times for nearby theaters Receiver Transmitter
- Slide 5
- Visual communications on phones No need for additional connectivity Free, directional, lightweight links Optical software-defined radio Run as apps on commodity devices
- Slide 6
- Many challenges Frame synchronization Poor image quality Limited computation power on phones Aesthetics
- Slide 7
- In the rest of the talk Frame synchronization challenges LightSync design Implementation and evaluation
- Slide 8
- FRAME SYNCHRONIZATION CHALLENGES
- Slide 9
- Frame rate mismatch Screen side (TX) 30 fps video playback Fairly stable Camera side (RX) 15 30 fps in practice Variable Lighting conditions Device capability API/system factors High rate, steady Low rate, unsteady
- Slide 10
- Original frame Received frame patterns
- Slide 11
- Original frame Single frame 2-frame mix Mixing pattern varies by line Received frame patterns
- Slide 12
- 2-frame mix Mixing pattern (& #components) varies by line Due to Rolling Shutter CMOS camera sensor scans one line (of pixels) at a time
- Slide 13
- Received frame patterns Original frame Single frame 2-frame mix 3-frame mix Consecutive frames indicating a missing frame Mixing pattern varies by line
- Slide 14
- Display frame rate C Frame capture rate C/2 C 2C Occasional missed frames 4-frame mixes 3-frame mix Occasional missed frames
- Slide 15
- Synchronized communication Display frame rate C Frame capture rate C/2 C 2C The only decodable pattern Frame synchronized Under-utilized and inefficient! Carries information But undecodable and dropped Cannot be used (of worst camera)
- Slide 16
- Unsynchronized communication Display frame rate (fps) 0 30 Works with any display and capture rates Higher camera capture rate, better performance Supported frame capture rate (fps)3015
- Slide 17
- Requirements Need to be able to decode mixed frames In-frame color tracking per line Need to guard against lost frames/lines Inter-frame erasure coding Essentially a temporal code across frames Works with any (monochrome) per-frame code
- Slide 18
- Our solution: LightSync Need to be able to decode mixed frames In-frame grayscale tracking per line Need to guard against lost frames/lines Inter-frame erasure coding Essentially a temporal code across frames Works with any (monochrome) per-frame code
- Slide 19
- LIGHTSYNC DEMO
- Slide 20
- LightSync Need to be able to decode mixed frames In-frame grayscale tracking per line Need to guard against lost frames/lines Inter-frame erasure coding
- Slide 21
- Per-line tracking: Mixing patterns 3 frames mixing 3 blocks mixing Bl or w per block 8 mixing possibilities ??? Frame 1 Frame 2 Frame 3 RXTX
- Slide 22
- Per-line tracking: Mixing patterns RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3
- Slide 23
- Per-line tracking: Mixing patterns Trivial cases RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3
- Slide 24
- Per-line tracking: Mixing patterns Easy cases RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3
- Slide 25
- Per-line tracking: Mixing patterns Tricky cases, to be tracked per line RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3
- Slide 26
- Code area Per-line tracking: Reference blocks Corner Scanning direction Tracking bar orientation Tracking bars
- Slide 27
- Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar 2 Tracking bar 3 Tracking bar 4 4 reference blocks per line
- Slide 28
- Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar 2 Tracking bar 3 Tracking bar 4
- Slide 29
- Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar 2 Tracking bar 3 Tracking bar 4 Decoding by comparison Reference colors of mixes
- Slide 30
- LightSync Need to be able to decode mixed frames In-frame grayscale tracking per line Need to guard against lost frames/lines Inter-frame erasure coding
- Slide 31
- Inter-frame coding Simple pairwise XOR Group of 3 original frames Coded frames
- Slide 32
- Inter-frame coding Looping display on screen Coded frames All original frames Camera stops frame capture after message decoded
- Slide 33
- PERFORMANCE
- Slide 34
- Implementation Current frame capture rate Inter-frame intervals
- Slide 35
- Performance Goal recap: Works with any display/capture rate combinations Metric: Effective frame capture rate Actual throughput scales with per-frame capacity
- Slide 36
- Schematic performance For a 20fps phone camera
- Slide 37
- Schematic performance For a 20fps phone camera Ideal Peaks at the camera rate Level at higher display rates Ideal
- Slide 38
- Schematic performance Peaks at ~ half the camera rate For a 20fps phone camera Conventional Drops to 0 at higher display rates Ideal
- Slide 39
- Schematic performance Conventional LightSync For a 20fps phone camera Peaks at the camera rate Slight drops at higher display rates Ideal
- Slide 40
- LightSync performance Nokia Lumia 920 (~30 fps)
- Slide 41
- LightSync performance Nokia Lumia 920 (~30 fps) HTC One X (~20 fps)
- Slide 42
- LightSync performance Nokia Lumia 920 (~30 fps) HTC One X (~20 fps) Huawei Ascend C8812E (~15 fps) Works with any display and capture rates Higher camera capture rate, better performance
- Slide 43
- Conclusion and outlook LightSync: Unsynchronized visual communication Primitive for similar scenarios Lots of potential in the space of visual communication
- Slide 44
- Thank you! [email protected]
- Slide 45
- Related work Temporal barcode design (inter-frame) Unsynchronized 4D Barcodes Basically slows down display, only for TX up to 10fps Spatial barcode design (per-frame layout) COBRA: Optimized for phone-to-phone PixNet: High-end camera, computationally intensive, offline processing Other work on visible light communications