Virtual Contact Sheets

Wolfgang RichterKiryong Ha, Alok Shankar, Ardalan Amiri Sani*, Jan

Harkes, Lin Zhong*, Mahadev Satyanarayanan

PARALLEL DATA LABORATORYCarnegie Mellon University

*Rice University

Crowdsourcing the News [1]

“When every cell phone can record video and take pictures, everyone is a potential news source.”

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 2

Who killedIan Tomlinson?

CNN: Caught Red Handed [2]

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 3

I’m a sneaky thief.I’m a sneaky thief.

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 4

Problem

CloudCloud

Recent,not uploaded

Older,uploaded

Fundamental Mobile Challenges

• Real-time sync not possible:

• Energy• Infinite battery life desired

• Bandwidth• Increasingly metered and charged

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 5

How far can we degrade the objects in a search before it loses meaning?

How far can we degrade the objects in a search before it loses meaning?

Just-in-Time Search1. Check virtual contact sheet

• Collection of fidelity-reduced images in the cloud

2. Pull full fidelity image on hit• Search full fidelity; present full fidelity results

to searcher

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 6

CloudCloud

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 7

Near Real-Time Sync(1) Search hit(2) Retrieve

Contact Sheet Analogy

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 8

Evaluating Feasibility [3]

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 9

Face Detection, RGB Histogram, Brightness Filter

Face Detection, RGB Histogram, Brightness Filter

Benchmark

UsingVarying Resolution and JPEG Quality Factor

Energy Savings

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 10

10 images =5 seconds call time (Droid)

10 images =5 seconds call time (Droid)

Bandwidth Savings

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 11

Orders of magnitude savings in bandwidth.Orders of magnitude

savings in bandwidth.

What about Search Accuracy?

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 12

No one-size-fits-all sweet spot.No one-size-fits-all sweet spot.

Face Detection

Conclusion & Future Work• Searches retain meaning

• No one-size-fits-all, but

• Orders of magnitude bandwidth savings

• Significant energy savings

• More search primitives?• Perceptual hashing• Face recognition• Texture Filters

• Better fidelity reduction?• Keep faces at high fidelity

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 13

Further Reading[1] TED. http://www.ted.com/talks/paul_lewis_crowdsourcing_the_news.html.

[2] CNN. http://www.cnn.com/2010/CRIME/08/24/new.jersey.theft.photo/index.html?hpt=C1.

[3] SimpleCV. http://simplecv.org/.

Jan Harkes, Mahadev Satyanarayanan, Ardalan Amiri Sani, and Benjamin Gilbert. A Contact Sheet Approach to Searching Untagged Images on Smartphones. CMU Technical Report CMU-CS-11-132.

Ardalan Amiri Sani, Wolfgang Richter, Xuan Bao, Trevor Narayan, Mahadev Satyanarayanan, Lin Zhong, and Romit Roy Choudhury. Opportunistic Content Search of Smartphones. Rice University Technical Report TR0627-2011.

Xuan Bao, Trevor Narayan, Ardalan Amiri Sani, Wolfgang Richter, Romit Roy Choudhury, Lin Zhong, and Mahadev Satyanarayanan. The Case for Context-Aware Compression. HotMobile 2011.

Jason Flinn and Mahadev Satyanarayanan. Energy-aware Adaptation for Mobile Applications. SOSP 1999.

Y. Nimmagadda, K. Kumar, and Y. H. Lu. Energy-efficient image compression in mobile devices for wireless transmission. ICME 2009.

[1] TED. http://www.ted.com/talks/paul_lewis_crowdsourcing_the_news.html.

[2] CNN. http://www.cnn.com/2010/CRIME/08/24/new.jersey.theft.photo/index.html?hpt=C1.

[3] SimpleCV. http://simplecv.org/.

Jan Harkes, Mahadev Satyanarayanan, Ardalan Amiri Sani, and Benjamin Gilbert. A Contact Sheet Approach to Searching Untagged Images on Smartphones. CMU Technical Report CMU-CS-11-132.

Ardalan Amiri Sani, Wolfgang Richter, Xuan Bao, Trevor Narayan, Mahadev Satyanarayanan, Lin Zhong, and Romit Roy Choudhury. Opportunistic Content Search of Smartphones. Rice University Technical Report TR0627-2011.

Xuan Bao, Trevor Narayan, Ardalan Amiri Sani, Wolfgang Richter, Romit Roy Choudhury, Lin Zhong, and Mahadev Satyanarayanan. The Case for Context-Aware Compression. HotMobile 2011.

Jason Flinn and Mahadev Satyanarayanan. Energy-aware Adaptation for Mobile Applications. SOSP 1999.

Y. Nimmagadda, K. Kumar, and Y. H. Lu. Energy-efficient image compression in mobile devices for wireless transmission. ICME 2009.

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 14

Full Fidelity

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 15

12,000+ Flickr dataset

Average photo size: 374 KBAverage photo size: 374 KB

Reduce Resolution

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 16

12,000+ Flickr dataset

Average photo size: 221 KBAverage photo size: 221 KB

JPEG Quality Factor

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 17

12,000+ Flickr dataset

Average photo size: 6 KBAverage photo size: 6 KB

Bandwidth Savings Table

Wolfgang Richter © Apr 20, 2023http://www.pdl.cmu.edu/ 18

Resolution 100 95 75 50 25 5

640x480 221 KB 114 KB 47 KB 31 KB 19 KB 6 KB

320x240 61 KB 33 KB 14 KB 10 KB 6 KB 2 KB

160x120 17 KB 10 KB 5 KB 3 KB 2 KB 826 B

80x60 5 KB 3 KB 2 KB 1 KB 892 B 460 B

40x30 2 KB 1 KB 725 B 588 B 484 B 345 B

20x20 1000 B 727 B 501 B 436 B 385 B 316 B