Redesigning the Wearable Camera Pipeline toDetect Key Body Parts at Low Power

Pengyu Zhang1, Bodhi Priyantha2, Jie Liu2, Matthai Philipose2

1University of Massachusetts Amherst, 2Microsoft ResearchApril 2015

ABSTRACTThis paper addresses the problem of designing wearable de-vices suited for continuous mobile vision. Given power bud-gets, it is infeasible either to stream all video off board foranalysis, or to perform all analysis on board. A compro-mise is to off-load only “interesting” frames for further anal-ysis. We identify windows containing people and parts oftheir bodies as being interesting. We show that using a care-ful combination of emerging thermal sensors and ultra-low-power coarse stereo enabled by modern low-power FPGAs,it is possible to detect the presence of these “key” body partsat well within wearable power budgets. We combine our(hardware) binary presence detector with a mobile CPU run-ning a classifier to identify which body part is detected. Us-ing this combination, we show how to detect faces, hands (ofthe video wearer) and bodies of those in the field of view atunder 30mW; running on just the phone draws over 1W.

1. INTRODUCTIONWe consider the design of a continuous mobile visionsystem based on a wearable device such as Google Glass.The potential benefits of a such a system have beendiscussed extensively recently [14, 9, 4, 29, 19]. In anutshell, the variety and detail provided by visual anal-ysis of the wearer’s surroundings enables a much largerset of applications than, for instance, those enabled byinertial sensors in popular step-counter-style wearablesavailable today. Given recent progress in visual recogni-tion algorithms, a primary bottleneck to realizing suchdevices is power consumption. Vision algorithms arecomputationally demanding, and video streams haveheavyweight communications requirements. Reconcil-ing the high resulting demands on energy with the mod-est battery capacity of wearable devices is challenging.In this paper, we examine techniques for attacking bothcomputational and communication bottlenecks for animportant subset of visual recognition tasks (those in-volving people and their parts) by re-designing the tra-ditional camera pipeline.

Figure 1 illustrates the setting we address. A wire-less wearable camera, positioned on the wearer’s lapel(or perhaps head) shares the wearer’s field of view. The

Figure 1: Our wearable camera usage (above) and thekey body parts we target (below): the wearer’s hands,their conversational partners’ faces and the bodies ofthose around them.

camera continually observes its wearer’s activities andtheir surroundings. Often, of greatest interest are ques-tions about what the user, their social partners andthose around them are doing. Much work over the pastdecade [30] has shown that the objects manipulated bya person and the nature of manipulation are a key tounderstanding the tasks they are performing. On theother hand, analyzing the faces of social partners witha view to identifying them, recognizing their affect andtracking their facial cues are key to decoding social in-teraction [37, 28]. Finally, a large body of work [18, 33]exists on tracking whole-body kinematics as a measureof coarse human activity. Accordingly, we designate thewearer’s hands, their conversation partners’ faces andthe bodies of those around them as “key” body parts.

Sensing and compressing the large, high-resolutionhigh-speed streams of image frames required for ana-lyzing entities of interest in the wearable’s field of viewrequires handling a large volume of data. The imager,codec and transmission components required to handlethis data exceed the power budget of a typical wearable.Recent pure-offloading-based architectures suited to therelatively large batteries of phones are therefore not ade-quate for wearables. One solution is to detect and trans-mit, for further analysis, a small subset of “interesting”windows from the video stream. Traditionally, detec-tion requires a dense windowed scan of the entire frameat several resolutions: a VGA frame can easily contain

1

tens of millions of windows to be checked. In somecases (e.g., face detection), the best algorithms are ex-tremely efficient, executing a few machine instructionson most windows [6]. In others (e.g., whole-body detec-tion), processing most windows may require hundreds ofthousands of instructions. In either case, software im-plementations of detection algorithms exceed wearablepower budgets. Hardware implementations promise ad-equate efficiency but are available only for front-on facedetection; efficient but general detection circuits are stillwell in the future.

In this paper, we present the design, implementa-tion and evaluation of a system for detecting key bodyparts at wearable power budgets. Intuitively, we ex-ploit the fact that the temperature of human skin is of-ten distinct from the background (our system degradesgracefully if this assumption is violated). Further, thehands of the camera wearer, the faces of their conver-sation partners and the bodies of people around themtend to be at distinct distances and positions relativeto the camera. We show how to exploit low-power, low-resolution imaging channels complementary to the tra-ditional RGB video pipeline to this end. In particu-lar, we show that a carefully designed combination ofspatially coarse thermal imaging and focused, coarse-resolution depth imaging optimized for ultra-low-powerprocessing on an FPGA can dramatically reduce boththe number of high-resolution RGB frames to be readfrom the imager, and the number of windows to bechecked in each frame. The end result is the first fullyrealized detection system we know of that, in commonuse cases of wearable cameras, can detect over 90% ofkey body parts using tens of mW of power.

2. MOTIVATION AND BACKGROUNDFigure 2 illustrates the structure and key performancecharacteristics of a wearable camera pipeline comprisedof state-of-the art components. A wearable camera musthave relatively wide field of view to cover the “front”of its wearers. When combined with the desire to re-solve, e.g., faces at 5m, an imager with 4k pixels perside is quite plausible. For such an imager running at15 frames/second or higher, today’s imager, compres-sion chips and wireless transmitters would together eas-ily require several hundred mW to off-load compressedvideo off-camera for analysis. However, assuming the

imager codec radio

90mW 200mW 500mW

videoA2-5Mbps

150-170oAFOVA25Apix/o

battery

70mW,A200mAh@3.5V,10hrs

Figure 2: Wearable camera pipeline.

0

0.2

0.4

0.6

0.8

1

0.1 1 10 100 1000

CD

F

Power (mW)

BaselineOracle

NormalizedNo Faces

(a) face

0

0.2

0.4

0.6

0.8

1

0.1 1 10 100 1000 10000

CD

F

Power (mW)

BaselineOracle

NormalizedNo Hands

(b) hand

0

0.2

0.4

0.6

0.8

1

0.1 1 10 100 1000 10000

CD

F

Power (mW)

BaselineOracle

NormalizedNo Body

(c) body

Figure 3: Power consumed for detecting key body partson a Galaxy Nexus phone.

device needs to run continuously for 10 hours (a “wholeday” for its wearers), size and weight constraints on thebattery yield a power budget well under 100mW. This“full offload”approach not only breaks the power budgetof the wearable device, it places the load of detectingobjects of interest on the devices offloaded to, typicallythe wearer’s mobile phone; alternately, the phone couldfurther offload the video to the cloud, requiring its radioto stay always on, an infeasible approach given trendsin wide-area wireless communication power.

We propose to push the detection of (some) interest-ing entities into the camera pipeline. Given that mostpixels detected by a wearable are uninteresting (theycontain no faces, hands, or bodies being monitored),this clearly has the potential to reduce significantly boththe amount of data streamed and the amount of data tobe analyzed. Doing detection on-camera poses severalchallenges, which we investigate in this section.

2.1 Detection in Software and Its LimitsPerhaps the simplest approach to incorporating detec-tors on the camera is to use a microcontroller to rundetection algorithms in the camera. To evaluate this ap-proach, we measured the overhead of wearer-hand, faceand person detection on a Galaxy Nexus phone with a750×480 imager running the corresponding OpenCV-based detectors [26]. We target detection at 1Hz. Weuse data from 14 people moving between 20 and 2 feetfrom the phone, or moving the hand from 6 inches to 2feet as a wearer’s hand would.

The pink and black lines of Figure 3a show the cumu-lative distribution of average power consumption whileprocessing a single frame, for face detection, over allframes of data collected; sub-figures (b) and (c) examine

2

wearer-hand and body detection. The pink line repre-sents power consumption when no body part was in theframe: the detector must still examine and reject eversub-window within the frame. The black “baseline” rep-resents the case when the relevant body part was withinthe frame. Power draw is somewhat higher, since thedetector tends to expend more cycles on windows whenbody parts are found. Note that each detector uses over1W of power for most frames: naively running detectorsin software is prohibitive.

The dotted orange line explores the hypothetical bestcase where an “oracle” is able to tell the detector wherein each frame to look. To generate this line, we markrectangles around body parts in our data by hand. Werun detectors only within these rectangles. As the or-ange lines show, if body parts tend to occupy relativelysmall fractions of the frame, oracles can reduce netpower consumption by orders of magnitude. Faces ofconversation partners only take up a relatively smallfraction of the frame even when the partners are closeby, so that oracles yield high gains (Figure(a)), gainingover 125× in efficiency at the 90th percentile. Wearers’hands, on the other hand, can dominate much of thefield of view, thus not gaining much (b). Peoples’ bod-ies dominate the frame even when they are only a fewfeet away, so serious gains in body detection efficiencyare limited to a fraction of the frames (c) (e.g., the 5thpercentile).

A fundamental reason that window-based detectorsare power hungry is that they search at several (rou-tinely, a dozen or more) scales within their search win-dow. Thus, if the above oracle could provide the scaleat which to search, search could be faster. For instance,if the oracle knew the distance to a pixel, and given thetypical size of a face, it could provide the detector witha very tight guess of the window sizes to consider. Theblue line in Figure 3 shows the effect of of knowing scaleduring detection. To obtain this curve, we re-scaled ev-ery oracle window to the smallest window-size exploredby the detection algorithm and ran detection on thiswindow. The impact is dramatic: we now detect allthree key parts at roughly 1mW at the 90th percentile.The gains are not at the expense of accuracy: detectionafter re-scaling found every previously detected face.

To summarize, if an “oracle” system were able to in-dicate roughly where a software-based detector shouldlook, and at what scale, even software-based detectorssuffice for on-camera detection at low power. Our workcan be considered an effort to realize such an oracle, atleast for the case of key body parts, and at substantiallyunder the 70mW we target for the total power budgetfor the wearable device. To cut to the chase, Figure 18summarizes our success in this direction.

2.2 Far Infrared (FIR) Thermal Imaging

20 22 24 26 28 30 32 34 36

0 5 10 15 20

Tem

pera

ture

(C)

Distance (ft)

North AmericaEast Asia

South AsiaBackground

(a) Temperature of Faces

15

20

25

30

35

40

45

0 0.5 1 1.5 2 2.5 3

Max

Tem

pera

ture

Distance (ft)

North AmericaEast Asia

South AsiaBackground

(b) Temperature of Hands

18 20 22 24 26 28 30 32 34 36

0 5 10 15 20

Max

Tem

pera

ture

Distance (ft)

T-shirtSweater

JacketBackground

(c) Temperature of Body (d) FIR Image of Hand

Figure 4: Temperature of key body parts measured witha FIR imager.

Far infrared (FIR), or thermal, imagers [7] report tem-perature, rather than traditional light intensity, at everypixel in the field of view. Figure 4d shows FIR viewsat multiple resolutions for the hand-holding-a-cup fromFigure 1. For instance, given that human facial skinpreserves a surface temperature of 30-35C, and ambienttemperatures are often substantial outside this range(e.g., temperature controlled buildings are kept in the20-23C range worldwide), thermal imaging is a promis-ing candidate for finding people in the field of view.Traditionally, FIR imagers have cost hundreds to thou-sands of dollars, consumed substantial power, and beenoptimized to high-cost high-resolutin settings. However,the past few years have seen the advent of MEMS-thermopile-based technologies developed for the auto-mobile industry, yielding 25mW, low-resolution parts atthe 50-dollar price point. We examine this technologyfrom the point of view of on-camera detection below.

We now describe some simple experiments aimed atunderstanding the intricacies of detecting key body partswith FIR. Figure 4 shows how FIR-measured tempera-tures of key body parts varies with distance from imagerand with possibly relevant auxiliary properties such asclothing and skin color. We generated these numbers byhaving three people with different skin colors (NorthernEuropean, East Asian and South Asian) pose at vary-ing distances from the imager (for face and body data),and move their hands at varying distances in front ofthe imager to simulate a camera wearer (for hand data).In each case, we report the temperature of the hottestpixel overlapping the body part.

Two related points are most relevant to our setting.First, when parts are close to the imager, bare skin tem-

3

peratures are close to 30C; temperatures are sometimeshigher, but very rarely lower in steady state. Cloth-ing results in a significant, but not catastrophic drop inreported temperature. In temperature controlled build-ings, commonly set to 20-23C, key parts are robustlyseparated by temperature thresholding. At higher tem-peratures, bare skin remains separable perhaps up to27C, clothed body parts only up to 24C if a jacket isworn. Skin color exhibits notable trends, but none withdirect relevance to thresholdability.

Second, as distance from imager increases, the ob-served temperature of faces drop rapidly, but hands andbodies remain relatively flat. The explanation lies inthe fact that each FIR pixel has a 15o field of view, andtherefore subtends increasingly larger areas at increas-ing distances from the imager. The reported tempera-ture is the average of body part and background tem-perature, weighted by the fraction of the field of viewsubtended by the part. For instance at 20ft, each pixelaggregates temperature measurements from a 1.8×1.8ftsquare: even if a face of temperature tf occupied a1×1ft square, the reported temperature would be 3

4 tb +tf4 , where tb is background temperature. Note, for in-

stance how as resolution decreases (i.e. the angle sub-tended by pixels increases), the pixels on the edges oofthe hand in Figure 4d become cooler

Finally, since wearers’ hands are close, they almost al-ways completely subtend the field of view of at least onepixel. Similarly, since torsos are large, unless they arerelatively far away, they too usually include such a“sub-tended” pixel. Since such a pixel is not “weighted down”by the background temperature, its reported tempera-ture will not change with distance and will tend to bemaximal among its neighbors. Figure 4, which showsthe variation of maximum temperature of pixels in eachbody part will thus show a relatively flat line.

One implication of this averaging effect over the fieldof view is that FIR imagers with narrower per-pixelfield of view will be more robust for temperature basedthresholding of relatively distant objects.

2.3 Stereo-Based Depth ImagingOnce a potential object of interest is identified by the

FIR imager, we use a depth sensor to determine if thatobject is located at a distance of interest. While dis-tance sensors such as proximity sensors and laser rangefinders can measure distance along a particular direc-tion object, we need a sensor that can produce a depthmap such that we can determine the depth of any givenobject in the scene.

Two most widely used vision-based depth map esti-mation techniques are stereo vision and auto focus. Ofthese two, auto focus requires a image sensor with adynamic focusing mechanism as well as a large image

L"

L’"

Y1" Y2"

D"β1"β2"

le#$lens$ right$lens$

image$plane$ image$plane$

Figure 5: Depth sensing with stereo imagers. L is thedistance to be estimated, L

′is the lens height, D is

the distance between two imager lens, ~Y1 and ~Y2 areprojected coordinates of an object on the left and rightimage planes.

Figure 6: Searching the corresponding object Y2 on theright image using correlation coefficients.

aperture. Because of these limitations, we use stereovision-based depth sensing.

Depth estimation from stereo vision is based on therelative shifting of the image captured on two image sen-sors that are placed some distance D apart. As shownin Figure 5, if a point source located at a distance Lfrom the image sensors are projected at coordinates ~Y1and ~Y2 of the two image sensors, then L can calculatedfrom: L = D×L′

~Y1− ~Y 2. Where L′ is the height of the image

sensor lens from the plain containing the image sensors.The relative shift, ~δY = ~Y1 − ~Y2, is called the disparityof the image of the point source.

Conventional stereo vision based depth estimationtechniques first compute the disparity map of the wholeimage by searching for corresponding pixels on the twoimages captured by the image sensors (Figure 6). Awidely cited technique in computer vision literature forfinding corresponding pixels computes cross correlationof the blocks of pixels from the two image sensors [24].

Figure 7 shows the power consumption of comput-ing the disparity map of 140 human face images on anARM Cortex M-4 microcontroller (STM32F417IGH6)at a frequency of 1 image/second. While the exactpower consumption varies with the size of the face, theaveraged power consumed is 78mW, which much higherthan our target power budget for continuous sensing.

4

0

0.2

0.4

0.6

0.8

1

50 100 150 200 250 300 350 400 450

CD

F

Power Consumption (mW)

Figure 7: Power consumption of searching Y2 by com-puting correlation coefficients.

Apart from the power consumption overhead, com-puting the cross correlation requires the two images tobe read in to some local storage medium. This willrequire additional memory overhead on a wearable de-vice that aims to capture and process only the relevantregions of the image.

While there exists a large body of work in visioncommunity on how to optimize the disparity map cal-culation using techniques such as dynamic program-ming and localized correlations, we observe that wehave a much restricted requirements on the disparitymap which enable us to use much simpler and energyefficient techniques to estimate the target distance.

First, we are only interested in well defined regionsof the image selected based on the FIR sensor readings.Hence, we only need to find correspondence for pixels ora subset of pixels in these regions. Second, we are onlyinterested in coarse grain depth estimation to differen-tiate among objects within arms length, people withinpersonal and group-wide interaction distances, and faraway objects. Such coarse grain estimates can easilytolerate 10% - 20% error in distance estimation. In sec-tion 3 we describe the design of an efficient depth esti-mation solution that take advantage of these restrictedrequirements.

3. DESIGNIn this section, we introduce the key components of

our low-power real-time key body part detection pipeline.

3.1 System OverviewFigure 8 shows the architecture of the body part de-

tection pipeline. The first stage is a FIR sensor thatuses the measured temperature of the scene to identifyimage frames with key human body parts, as well as thelocation of the body parts within the image frame. Thelocation information is then sent to the second stagewhich consists of a coarse-resolution depth sensor thatestimates the distance to these identified regions.

If the region temperature and distance to the regionmatches one of the criteria for detecting a body partsuch as detecting the hand to identify the object the

Figure 8: Architecture of the low-power key humanbody part detection pipeline.

Table 1: Comparison of Melexis and FLIR infrared sen-sors.

Power (mW) ResolutionMelexis (MLX90620) 23.4 16×4FLIR (LEPTON) 150 80×60

user is holding, the image pixels within the region ofinterest are captured for post processing such as feedingto a remote object recognizer to identify the particularobject.

The use of low-power low-resolution FIR sensor andthe low-power coarse-resolution depth sensor enables anend-to-end key human body part detector with a powerprofile suitable for continuous operation on a batterypowered wearable device.

3.2 Far Infrared SensingWhile our initial observations showed that a FIR sen-

sor can differentiate between human body parts and thebackground fairly well, one key design criteria is theresolution of the FIR sensor. For example, as Table 1shows, a state of the art LEPTON FIR sensor with80× 60 resolution from FLIR consumes 150mW, whichis too high for continuous sensing on a wearable device.

A low-resolution image sensor is attractive due to therelatively low power consumption. However such a sen-sor has two draw backs. First, it cannot accuratelydetect the edges of objects due the mixing of regionsof different temperatures in a single pixel. Second, forrelatively small objects at far away distances, the ob-ject ad a portion of the surrounding region maps in to asingle pixel, reducing the measured temperature of theobject, making it difficult to detect the object based onits temperature.

While a high resolution FIR sensor is better at iden-tifying the objects of interest and their boundary accu-rately, these benefits come at a higher power consump-tion.

We found the MLX90620 FLIR sensor from Melexiswith 16× 4 resolution is the right balance between theresolution and power consumption since it has enoughresolution to detect human parts at 12 feet from theuser while consuming only ' 23 mW.

From Figures 4a 4b 4c, we observe the measured tem-perature of human body parts depends on multiple fac-tors such as the skin color and the distance of the person

5

0 0.2 0.4 0.6 0.8

1

0 0.2 0.4 0.6 0.8 1

Fals

e Po

sitiv

es

False Negatives

FaceHandBody

Figure 9: Tradeoff between false positives and false neg-atives of FIR sensing when the threshold of human tem-perature changes.

from the sensor. Hence, we need to select a temperaturethreshold for separating human body features from theback ground.

As shown in Figure 9, selecting this threshold is atradeoff between the false positives and false negativesin FIR-based detection.

A high threshold results in close to zero false pos-itives, while causing high false negatives for far awayobjects whose measured temperature is lower. A lowthreshold results in a large number of false positives,while the false negatives become smaller.

While this threshold should be selected based on thespecific application needs, we selected 23.7oC as thethreshold since this minimizes the sum of false positivesand false negatives.

3.3 Depth SensingOnce the Far Infrared sensor detects potential re-

gions containing objects of interest based on the tem-perature profile, depth sensing module is triggered forcoarse-resolution distance measurement. This sectiondescribes design of our low-power depth sensing mod-ule.

3.3.1 Hardware ArchitectureThe main design goals of the depth sensing module

are to minimize the processing and storage overheadwhile providing coarse-granularity depth information ofspecific regions of the image.

To minimize the processing overhead, the depth sens-ing module only operates on the regions of interest thatare identified by the FIR sensor. To further reduce com-putational complexity, we use the bit-wise XOR opera-tion of pixel values as a measure of similarity betweenthem (smaller results corresponds to high similarity).

After capturing an image, a typical image sensor out-puts the pixels of the image row-by-row as a serial bitstream. To minimize storage requirements, instead ofreading the two images from the image sensors to some

Figure 10: Micro hardware architecture for low-powerdepth sensing.

storage medium, our depth sensor directly operates onthe two pixel streams coming from the image sensors.

Here we use the key observation that finding the cor-respondence between two rows of spatially separatedpixels in the two image sensors translates to finding thecorrespondence between the two pixel streams delayedwith respect to each other.

Figure 10 shows the hardware architecture of the ofdepth sensor design. This is composed of three HWbuilding blocks: delay, XOR, and accumulator.

Delay: Delay modules are used to delay the pixelstream coming from the left image sensor because theright picture is a delayed version of the left image dueto spatial disparity. The delayed pixels from the leftimage are XORed with the pixels from right image tocompute disparity.

In our design, delays are implemented with D-flip-flops, that temporarily store the pixels. For example, ifeach pixel is represented by an 8-bit value, each delaymodule is an 8-bit D-flip-flop.

XOR: We use bit-wise XOR to compare the differ-ence between two pixels instead of computing the cor-relation coefficients to reduce power consumption. Onepotential drawback using XOR is the lower accuracy.However, our evaluations show that XOR is accurateenough for coarse-resolution depth sensing for humandetection.

Accumulator: Accumulator aggregates the resultsof the bit-wise XOR operation and outputs the similar-ity of the two image blocks. Our design contain multipleXOR-Accumulator blocks that are connected to multi-ple tap points of the delayed pixel stream of the leftimage sensor.

Each of these XOR-Accumulator blocks measures thesimilarity of pixels delayed by a number of pixels cor-responding to the length of the delay line up to thatblock. Hence, these multiple accumulators correspondto different distance estimates of the target object.

6

For example, in our implementation we use 4 accumu-lators that correspond to {4, 8, 12, ∞} feet. Once theimage blocks pass through all the accumulators, the ac-cumulator values are compared and the object distanceis associated with the accumulator holding the smallestsum (bit-wise XOR of two identical pixels is zero).

Hardware complexity: Our hardware depth sens-ing architecture with its data-flow style processing onmultiple simple arithmetic blocks is particularly suit-able for direct implementation on an FPGA. We candirectly compute the FPGA resource requirements asfollows.

Given N delays and M accumulators, the number ofD flip-flops (single bit storage elements) needed for im-plementing this logic is (8N +24M) because each delayelement is an 8-bit register which buffers an 8-bit pixeland a 24-bit register is needed for each accumulator toprevent overflow.

For example, in section 4.3, we show that our designfor detecting if an object of interest is at X feet, whereX ∈ {4, 8, 12,∞}, can easily fit inside a tiny FPGA orCPLD such as Igloo Nano which only consumes severalmW [23]. In contrary, traditional depth sensing, suchas Capella [1], needs to be operated on a powerful CPUand consumes several Watts, orders of magnitude higherthan our micro hardware architecture.

4. IMPLEMENTATIONWe implemented our low-power key body part detec-

tor design described in section 3. While, the currentprototype measures 8.7cm by 11.5cm, we believe thatthis can be shrunk to fit a small wearable device.

4.1 Hardware PlatformFigure 11 shows our hardware prototype. This pro-

totype is implemented as two modules: an image sensordaughter board and a processor mother board.

The image sensor daughter board is equipped withthree Aptina image sensors (MT9V034). As shown inFigure 11a, this enables us to evaluate the accuracy ofdepth sensing under image sensor separations of 3cm,6cm, and 9cm.

The motherboard contains FIR sensor (MLX90620)for measuring the scene heat map. The motherboardalso contains a low-power FPGA (Igloo Nano AGLN250)and a ARM Cortex M-4 microcontroller (STM32F417IGH6).The low power depth sensing module is implemented inthe FPGA while microcontroller coordinates and man-ages various sensing subsystems.

The microcontroller can also capture and store fullimages from the image sensors for offline analysis andsystem debugging.

4.2 Processing Pipelines

(a) Imager daughter board (b) Processor mother board

Figure 11: Prototype of our hardware platform.

Figure 12 shows how different processing pipelines aremapped to various processing elements on the mother-board. The mother board runs three processing pipelines:FIR and image processing pipelines on the MCU andthe depth sensing pipeline on the FPGA.

FIR sensor pipeline: The FIR sensor pipe line ismore involved than simply reading the temperature val-ues directly from the sensor. The Melexis MLX90620FIR sensor does not directly output the object temper-ature. Instead, it compares the amount of radiationsensed against the temperature of the sensor die andreturns this relative value.

To determine the object temperature, these readingsneed to be compensated according to the die temper-ature. This compensation needs to be done for eachpixel because each pixel has different compensation co-efficients. Each pixel compensation involves significantamount of floating points calculations with some coeffi-cients of magnitude 10−8.

We managed to reduce duration of this processingpipeline to 3.68ms for a 16×4 pixel IR sensor when theSTM32F417IGH6 micro controller runs at 168MHz. Toachieve this, we precompute most of the compensationcoefficients and store them in RAM. With these opti-mizations, the MCU only needs to be on for 4% of thetime even when the FIR sensor is operated at 10Hz.

FPGA depth sensing pipeline: Figure 12 showsthe hardware architecture of depth sensing implementedon an FPGA. The implementation includes three com-ponents. First, a window control module is implemented

Figure 12: Architecture of our hardware platform.

7

(a) Horizontal errors (b) Vertical errors

Figure 13: Horizontal and vertical manufacture errorswhere the observed picture Y ∗ deviates from its idealposition Y.

for interfacing Aptina image sensors. The window con-trol only pass pixels in a specific region where depthneeds to be estimated to the following modules.

Next, delays modules, which are composed by D-flip-flops, are used for achieving synchronization betweenleft and right images. Delay module uses 8-bit D-flip-flops since each pixel is represented as an 8-bit value.Last, accumulators with XOR and summation logic areused for comparing similarity of blocks on left and rightimages. Depth is estimated based on the content of theaccumulator at the end of the image.

MCU image processing pipeline: The image pro-cessing pipeline built on an MCU is used to captureand store raw images for system debugging and evalua-tion. DCMI interface, which is a standard image sensorinterface, is used in combination with processor DMAmodule to capture and transfer images directly to MCUmemory.

Once a complete image is captured into the RAM,it is written into a SD card with a FatFS file system.Because SDIO is used for interfacing the SD card, theimages can be written in a short time. We also im-plemented correlation-based depth computation on theMCU for comparing with our low-power FPGA imple-mentation.

4.3 Handling Manufacturing ErrorsOne challenge in using stereo image sensors for depth

sensing is handling manufacture errors. These errors areintroduced by stereo image sensor mis-alignment, lensmounts that deviates from the center of image sensors,and other factors. Figure 13a and Figure 13b show twotypes of errors: horizontal and vertical manufacture er-rors. The observed object position Y ∗ deviates from itsideal position Y in horizontal and vertical directions.

4.3.1 Horizontal manufacturing errorsHorizontal manufacturing errors shown in Figure 13a

are introduced by the horizontal lens deviation from thecenter of an image sensor plane. Figure 14 shows thetheoretical and empirical curves of our hardware plat-

0 5

10 15 20 25 30 35

0 10 20 30 40 50 60

Dis

tanc

e (fe

et)

Pixel Offset

6cm - Theoretical6cm - Empirical

6cm - Model

(a) 6cm lens separation

0 5

10 15 20 25 30 35

0 10 20 30 40 50 60 70 80 90 100 110

Dis

tanc

e (fe

et)

Pixel Offset

9cm - Theoretical9cm - Empirical

9cm - Model

(b) 9cm lens separation

Figure 14: Theoretical, empirical, and modeled curvesfor depth estimation using coordinates offset Y2 − Y1.

form where stereo image sensor lens are separated by6cm and 9cm. The theoretical curves are calculated us-ing L = DL′

Y2−Y1with parameters calibrated. For both

6cm and 9cm lens separation, a significant gap betweenthe theoretical and empirical curves is observed. The6cm lens separation has slightly larger gap than the9cm case because smaller lens separation is more sen-sitive to manufacturing errors. This is a result of thesmaller pixel offset range (2∼57) under 6cm separationcompared to the larger range (1∼102) of 9cm lens sep-aration. We come up a model to compensate these hor-izontal manufacturing errors.

The model shown in equation 1 is used for capturingthe horizontal manufacturing errors, where parameterb deals with errors introduced by lens mount that hor-izontally deviates from the center of an image sensor,and parameter c deals with the rest of errors within thevision system.

D =a

Y2 − Y1 − b− c (1)

By calibrating parameters a, b, and c as shown inequation 2, both modeled 6cm and 9cm curves fit theempirical curves well as shown in Figure 14, and there-fore can be used for practical and accurate depth esti-mation.

D =160.1026

Y2 − Y1 − 8.5671− 0.5894 (2)

D =298.7770

Y2 − Y1 − 5.7169− 0.7962 (3)

4.3.2 Vertical manufacturing errorsAnother manufacturing error is the vertical error shown

in Figure 13b which is usually introduced by verticallens deviation from the center of an image sensor plane.This error is not a significant issue for conventionalstereo image systems because left and right images arestored in RAM and post-processed, where these verticalerrors can be compensated in software. However, ver-tical manufacture errors are detrimental to our depthestimation solution.

8

As shown in Figure 13b, the targeted block Y ∗ willbe outputted much later than it is expected because ofthe vertical offset. While this error can be compensatedby adding extra delay elements, the power consumptionand HW complexity of the solution increases dramat-ically due to the large number of storage elements re-quired.

Instead, we use the windowing capability of the im-age sensor to define the vertical boundary of the imagewindows to compensate the vertical manufacturing er-rors. The built-in windowing capability of the imagesensor can horizontally or vertically trim the capturedimage and outputs only a window of the image. Thiscapability allows us to compensate the vertical manu-facture errors at the image sensor itself without extrahardware resources.

For example, when the vertical error of the right im-age sensor is X rows, which implies that the targetedobject will be outputted X rows earlier than expected,we instruct the left image sensor to skip its first X rowsand starts its output from the X+1 row. As a result,the images of the left and right image sensors becomevertically aligned.

4.4 Mapping between FIR and depth sensorsOne particular problem that we need to address is

the mapping between FIR and depth sensors. Once theFIR sensor detects the presence of “hot” objects, depthsensor will be triggered to measure the distance to thatobject. To enable this pipeline, we need to map theobject observed by the FIR sensor to the filed of viewof the depth sensor. Establishing the mapping is non-trivial due to two reasons. First, mapping needs to bedone for sensors with different resolution. The resolu-tion of a FIR sensor is 16×4 pixels while each imagerof the depth sensor has 752×480 pixels. Second, map-ping needs to be done for sensors with different angleof view. The horizontal and vertical angle of view of aFIR sensor is 20o and 5o while depth sensor has 26o and18o.

The mapping between FIR and depth sensors is builtfollowing two principles. First, depth is computed onlyfor pixels that are horizontally and vertically covered byboth FIR and depth sensors. As a result, some pixelsof the depth sensor are not included in the mappingbecause of the narrow field of view of the FIR sensor.Second, a single FIR pixel is mapped to 35×32 pixels ofthe depth sensor. Such linear transformation is feasiblebecause both FIR and depth sensors use linear opticalsystems for imaging.

5. EVALUATIONWe evaluate three aspects of our system. How accurateare FIR and depth sensing, both separately and jointly?How much power does our system consume? When cou-

pled with a software body-part detector to confirm partsdetected by our front-end, how much power does thesystem save compared to the pure software approach?In the terminology of Section 2.1, how well does oursystem serve as an oracle?

5.1 Accuracy of the Hardware Detector

5.1.1 Accuracy of FIR sensingTo evaluate the accuracy with which the FIR sensor de-tects key body parts, we tested it in a static and a mo-bile configuration at home and in a university (“school”)setting. The settings were empty except for at most asingle experimental subject. In the static home case weplaced the sensor overlooking the living room; in thestatic university case we placed it facing a corridor. Inthe mobile case, we moved around with the sensor aswith a wearable, traversing the same living room andcorridor. We collected footage for 10 minutes in eachconfiguration, sampling at 8Hz. To test detection ac-curacy, we had one subject behave “naturally” in eachsetting, while the experimenter stayed behind the cam-era (and moved it, in the mobile case). The footage waslabeled by hand as to whether a face, wearer-hand orbody was showing.

We define the recall of our system with respect to apart as the fraction of frames labeled as containing thepart where the system correctly detects a part (noteFIR-thresholding does not report which part). We de-fine the false positive with respect to a part as the frac-tion of all the frames where the FIR detector claims apart is visible where that part was not labeled as visible.

Table 2 shows the results. Perhaps unsurprising, re-call rates are quite high, since the background temper-ature in these indoor settings is at the conventional 20o

level. Encouragingly, however, false positives are rela-tively low as well. They are higher in the mobile casethan fixed case, because when walking around, morelighting fixtures, computer monitors and heating ventscome into the field of view. They are also somewhathigher in faces than for hands and significantly higherfor bodies. This is because (Figure 4) while wearer-hands stay close to the imager and maintain a relativelyhigh temperature, faces tend to report lower tempera-tures further away due to the background-averaging ef-fect reported in Section 2.2 and bodies, being coveredclothes report even lower temperatures. The lower thereported temperature of a body part, the more objectsin the surroundings it may be confused with. It is worthnoting that even a relatively high false positive rate doesnot cause incorrect detection of body parts (since laterstages of our system will verify the pixels flagged), butwill “only” diminish the power saved by our system.

5.1.2 Accuracy of depth sensing

9

Table 2: Accuracy of detecting key body parts using FIR.

Detecting human faces Detecting human hands Detecting human bodiesStatic sensor Mobile sensor Static sensor Mobile sensor Static sensor Mobile sensorRecall FP Recall FP Recall FP Recall FP Recall FP Recall FP

School 91.8% 2.4% 99.7% 13.9% 83.9% 0.9% 99.6% 6.4% 98.8% 36% 99.8% 58.7%Home 96.8% 0.2% 94.1% 6.3% 95.2% 0.2% 92.9% 6.2% 99.6% 1.1% 98.9% 22.3%

Fine Grained Depth Estimation: We wish to mea-sure how accurately we can estimate depth, and theeffect of lens separation and distance measure (correla-tion versus XOR (Section 3.3.1)) on this accuracy. Tothis end, we had 14 persons each stand at distancesof 2, 4, 6 . . . 20ft from the imager, and for each configu-ration, compared the system-estimated depth with theground truth. We estimate depth using both 3- and 6-cm lens separation (recall that our board has both sep-arations built in), and using both correlation and XOR-based matching. Figure 15 shows accuracy in both (a)relative (depth error÷ground truth distance) and (b)absolute terms.

Accuracy of estimation is quite good, given the rel-atively coarse needs for finding key body parts: e.g.,for 80% of data points, the error of depth estimationis smaller than 20% and 2 feet. Large errors occurfor points a the 2- and 4-ft ranges, as per the lim-itations of stereo. Further, 9cm lens separation hashigher accuracy than the 6cm lens separation: as ex-pected, higher depth resolution is obtained with largerlens separation. Finally, for both 9cm and 6cm lens, ourXOR-based matching is less accurate than the conven-tional correlation-coefficient-based approach. However,the degradation in accuracy is less than 5%, while thegains in reduced power consumption are considerable,as shown later.

Coarse Grained Depth Estimation: We use the14-person dataset just discussed to estimate coarse dis-tance, rather than fine distance. We group the distancesinto bins as shown in the first column of Table 3. Thebins are intended to represent distances correspondingto “touched objects”, “conversation partners” , “peoplepossibly performing relevant actions” and “everythingelse”. We classify the dataset into one of these fourcategories by running the micro hardware architecture.Recall is the fraction of true instances of the class cor-

0

0.2

0.4

0.6

0.8

1

0 20 40 60 80 100

CD

F

Estimated Distance Error (%)

6cm+Corr6cm+XOR9cm+Corr9cm+XOR

(a) Error in percentage

0

0.2

0.4

0.6

0.8

1

0 0.5 1 1.5 2 2.5 3 3.5 4

CD

F

Estimated Distance Error (ft)

6cm+Corr6cm+XOR9cm+Corr9cm+XOR

(b) Error in feet

Figure 15: Accuracy of fine-grained depth estimation.

Table 3: Accuracy of coarse depth estimation.

9cm lens 6cm lensDepth (ft) Recall FP Recall FP

(2,4) 93% 0% 83% 0%(6,8,10) 83% 0% 88% 2%

(12,14,16) 91% 5% 83% 4%(18, 20, ∞) 100% 5% 91% 10%

rectly inferred. The false positive fraction (FP) is thefraction of claimed instances of a class that were incor-rect. As the table indicates, our depth sensor excels atcoarse estimation, with both 6- and 9-cm baselines.

Impact of Correcting Manufacturing Errors: Asmentioned in section 4.3, manufacturing errors can bedivided into horizontal and vertical errors. We addressthese by using imager hardware windowing and a cor-rection model. Figure 16 illustrates the impact of errorcorrection on the 14-person dataset above. When errorsare corrected, only 10% of depth estimates have errorsabove 2ft absolute, or 20% relative to ground truth dis-tance. Uncorrected, over 70% of estimates have errorsabove 10ft absolute/60% relative. Correction is there-fore essential. It is worth noting here that our errorcorrection has low power overhead: neither the hard-ware windowing nor the horizontal correction model re-quires additional computation, since they require just aone-time configuration.

5.2 Power Draw of the Hardware Detector

5.2.1 Power draw for platform componentsTable 4 shows the static and active current consumptionof each component in our hardware platform. The staticcurrent draw of the whole hardware platform is 4mA,which is measured when the IR sensor is not initial-

0

0.2

0.4

0.6

0.8

1

0 20 40 60 80 100 120 140

CD

F

Estimated Distance Error (%)

CorrectedDefects

(a) Relative error.

0

0.2

0.4

0.6

0.8

1

0 5 10 15 20 25

CD

F

Estimated Distance Error (ft)

CorrectedDefects

(b) Absolute error.

Figure 16: Impact of correcting manufacture errors.

10

Table 4: Component-wise power consumption. × indi-cates that the corresponding state does not exist.

Power (mA) Component active asleepStatic Current Draw × 4

16×4 IR sensor 9 ×752×480 Stereo Imagers 73 11∗

MCU 55 0.002FPGA 2.58 0.01

ized, stereo imagers are not mounted, the MCU staysin STANDBY mode and the FPGA clock is disabled.The IR sensor consumes 9mA after initialization andunfortunately, cannot be turned into sleep mode due toits lack of duty cycling mechanism. The stereo imagers,which are used for depth estimation, consume 11mA inSTANDBY mode and 73mA in active mode. The MCUconsumes 55mA and 2.3mA for the FPGA, which followthe corresponding data sheet specification.

Table 5 shows the timing of tasks executed on thehardware platform. Some tasks need to be done onlyonce, such as stereo imager initialization and SD cardinitialization. Other tasks, such as FIR and depth sens-ing, need to be done frequently. FIR sensing takes3.68ms and depth estimation, including exposure andsensing, takes 16ms. Although operating the MicroSDis time-consuming, it is designed for debugging and notused in runtime system.

5.2.2 Power draw for estimating depthThe power consumption of depth sensing depends onthe physical distance of targeted objects because closerobjects occupy more pixels. Figure 17 compares the dis-tribution of average power drawn for processing framescomprising our 14-person dataset when our FPGA-basedarchitecture is used (red line labeled “Micro Arch”) tousing a low-power microcontroller (“MCU”). Critically,even when frames are filled with body pixels (e.g. at100% of the CDF), power draw tops off at 5.1mW (me-dian 3.1mW). Thus, although this evaluation was ondetecting bodies, we expect wearer-hands, which oftendominate the field of view, to draw acceptable power.

Our efforts in adapting our algorithms to the low-power FPGA seem worthwhile. Power consumed bythe FPGA-based architecture (the black line in Fig-ure 17) is roughly 18× lower than when an MCU (theSTM32F417IGH6) is used. There are three reasons for

Table 5: Timing of tasks executed on the platform.

Time (ms)64 pixels FIR sensing 3.68

depth sensing 15Stereo imager initialization 1.2Stereo imager exposure 1

MicroSD Card initialization 2400MicroSD Card logging 125

0

0.2

0.4

0.6

0.8

1

1 10 100 1000

CD

F

Power Consumption (mW)

MCUMicro Arch

Figure 17: Power consumption for estimating depth.

0

0.2

0.4

0.6

0.8

1

10 100 1000

CD

F

Power (mW)

BaselineFarIR+Depth

+Face Position+Normalization

(a) Detecting Faces

0

0.2

0.4

0.6

0.8

1

10 100 1000 10000

CD

F

Power (mW)

BaselineFarIR+Depth

+Hand Position+Normalization

(b) Detecting Hands

0

0.2

0.4

0.6

0.8

1

10 100 1000 10000C

DF

Power (mW)

BaselineFarIR+Depth

+Body Position+Normalization

(c) Detecting Bodies

Figure 18: Power consumed for detecting key bodyparts using our system (compare Figure 3).

this reduction. First, interfacing the imager with anFPGA avoids the software driven DCMI interface onan MCU, and is thus significantly cheaper. Second, theFPGA processes pixel streams in real time and as anatural result, avoids pixel storage overhead, such asSRAM. Third, some key operations, such as XOR, aresimply cheaper on an FPGA.

5.3 Overall System EvaluationIn Section 2.1, we pointed out that the overall goal ofour system is to realize an oracle that indicates whereand at what scale in the imager frame a software parts-detector should search for key body parts. Figure 3(cyan line) pointed out that such an oracle, if it ranat zero power, could then detect body parts at 1mW insoftware. We now examine to what extent our FIR/depth-based architecture achieves this goal. Similar to Sec-tion 2.1, we collected footage of subjects from 3 to15 feet away in a corridor, and a hand moving from6 inches to 2 feet. We then used our system to de-tect parts, bound their pixel windows and scales, andexecuted OpenCV-based body-part detectors on these

11

windows on a Galaxy Nexus. We added the measuredenergy consumed on our detector to that of detectingparts within those windows on the phone and dividedby total execution time to get average power draw.

The cyan lines on Figures 18a-c illustrate the perfor-mance of our oracle. The entire system runs at roughly24mW when no body part is detected (this correspondsto running the 9mA FIR sensor at 2.6V), and roughly28mW when a body part is detected (an additional mWfor the on-phone calculation and 3mW to run the depthunit). Given our 70mW budget, we believe thesenumbers provide compelling evidence that anarchitecture cascading FIR and depth sensingvia today’s low-power off-the-shelf components(such as our FPGA) could enable key body partdetection on wearable devices, at least in areaswhere background temperature is outside the 30-35Crange, and in temperature controlled buildings.

6. RELATED WORKBody parts detector: There is an extensive bodyof work on algorithms for identifying key human bodyparts, including faces [36, 31, 27], torsos [38, 22], andhands [15, 16]. These algorithms have been used in var-ious applications, including human computer interfaces[39], smart rooms [43], and intelligent robots [40]. Ourwork does not target specific algorithmic improvements.Instead, we investigate how multiple sensing modalitiescan be used as pre-filters to limit the invocation of ex-pensive algorithms.

Low-power imagers: Ultra low-power imagers, suchas [13, 5], consume less than 1mA of current when ac-tive. In addition, various mechanisms have been pro-posed by [17, 21, 12] for reducing the power consump-tion of imagers. Therefore, imagers will not likely bethe bottleneck for achieving low power depth estima-tion. Instead, reducing the processing overhead is goingto be the key to achieving low power depth estimation.

Low-power depth sensing: Computer vision com-munity has investigated accurate disparity map compu-tation using stereo web cameras [35], reducing the com-putational overhead of disparity map calculation [11],3D image construction based on disparity map [20], etc.These approaches focus on improving computer visionalgorithms while our focus is to improve the hardwarearchitectures. We see these as complementary efforts.

Some previous work use FPGAs for efficient depthcomputation. [34] presents a hardware architecture forcomputing depth map which avoids modifications ofdepth algorithms. [32] designed an optimized hardwarearchitecture for computing depth map in 1D. [25, 3] dis-cuss low cost FPGA implementation of specific depthalgorithms. These approaches focus on computing thecomplete disparity map while our focus is on estimatingthe depth of specific objects within the scene.

[41, 42, 35] propose several algorithms for stereo vi-sion calibration when manufacture defects are present.These algorithms are computationally expensive to runcontinuously on a battery power device. We use a sim-plified correction model for achieving low power con-sumption at the cost of small degradation of depth es-timation performance.

Stereo imager platform: Stereo camera hardwareplatforms have been on the market for a long time. [2]is a commercial stereo camera that supports captur-ing of two images simultaneous at different sensitivity,magnification, and color tonality. Unfortunately, thiscamera is not a programmable device which preventsits use in low-level investigations. [1] is a stereo cameradevelopment board that allows low-level firmware con-figuration of imagers. However, detailed power bench-marking is hard on this platform because of its deepcoupling between OS and stereo image processing, andadditional on board hardware components (Ethernet,USB, HDMI). In contrast, our hardware platform al-lows detailed power benchmarking of individual hard-ware modules.

Far Infrared sensing: [8] uses FarIR sensors to de-termine the occupancy of zones within a building. [10]exploits low power 8×8 FarIR sensor array to moni-tor human activities in home and office environments.While both these work provide valuable insight into us-ing FarIR for human detection, none of them investigatethe possibility of combining depth sensor for detectingobjects of interest.

7. CONCLUSIONIn this paper, we present a low-power human body

part detector that exploits Far Infrared (FIR) and depthsensors to gate the traditional vision pipeline. We maketwo main contributions. First, we design and imple-ment a low-power hardware architecture which is ableto obtain depth at low power. Second, we show thatthe combination of FIR and depth sensors serves as ameans to detect (parts of) people their in the field ofview. When combined with software classifiers that canidentify the body parts, our system can identify “key”body parts at identical accuracy to the pure software so-lution but at much lower power. We believe our paperonly scratches the surface of developing gating circuitryfor vision, and fully expect both more general and lowerpower gating systems to emerge in the near future.

8. REFERENCES[1] Capella - stereo vision camera reference design.

http://www.e-consystems.com/

stereo-vision-camera.asp.[2] Fuji finepix real 3d w3. http://www.fujifilm.

com/products/3d/camera/finepix_real3dw3/.

12

[3] Stereographic depth mapping on an fpga.https://courses.cit.cornell.edu/ece576/

FinalProjects/f2010/pfk5_jk459/pfk5_

jk459/index.html.[4] Sharad Agarwal, Matthai Philipose, and Paramvir

Bahl. Vision: The case for cellular small cells forcloudlets. In International Workshop on MobileCloud Computing and Services, 2014.

[5] Suat U Ay. A 1.32 pw/framee pixel 1.2 v cmosenergy-harvesting and imaging (ehi) aps imager.In Solid-State Circuits Conference Digest ofTechnical Papers (ISSCC), 2011 IEEEInternational, pages 116–118. IEEE, 2011.

[6] Junguk Cho, Shahnam Mirzaei, Jason Oberg, andRyan Kastner. Fpga-based face detection systemusing haar classifiers. In Proceedings of theACM/SIGDA international symposium on Fieldprogrammable gate arrays, pages 103–112. ACM,2009.

[7] E. L. Dereniak and G. D. Boreman. InfaredDetectors and Systems. Wiley, second edition,1996.

[8] Varick L Erickson, Alex Beltran, Daniel AWinkler, Niloufar P Esfahani, John R Lusby, andAlberto E Cerpa. Toss: Thermal occupancysensing system. In Proceedings of the 5th ACMWorkshop on Embedded Systems ForEnergy-Efficient Buildings, pages 1–2. ACM, 2013.

[9] Kiryong Ha, Zhuo Chen, Wenlu Hu, WolfgangRichter, Padmanabhan Pillai, and MahadevSatyanarayanan. Towards wearable cognitiveassistance. In MobiSys, 2014.

[10] Peter Hevesi, Sebastian Wille, Gerald Pirkl,Norbert Wehn, and Paul Lukowicz. Monitoringhousehold activities and user location with acheap, unobtrusive thermal sensor array. InProceedings of the 2014 ACM International JointConference on Pervasive and UbiquitousComputing, pages 141–145. ACM, 2014.

[11] Heiko Hirschmuller. Accurate and efficient stereoprocessing by semi-global matching and mutualinformation. In Computer Vision and PatternRecognition, 2005. CVPR 2005. IEEE ComputerSociety Conference on, volume 2, pages 807–814.IEEE, 2005.

[12] Steve Hodges, Lyndsay Williams, Emma Berry,Shahram Izadi, James Srinivasan, Alex Butler,Gavin Smyth, Narinder Kapur, and Ken Wood.Sensecam: A retrospective memory aid. InUbiComp 2006: Ubiquitous Computing, pages177–193. Springer, 2006.

[13] K Kagawau, Sanshiro Shishido, MasahiroNunoshita, and Jun Ohta. A 3.6 pw/framea· pixel1.35 v pwm cmos imager with dynamic pixelreadout and no static bias current. In Solid-State

Circuits Conference, 2008. ISSCC 2008. Digest ofTechnical Papers. IEEE International, pages54–595. IEEE, 2008.

[14] Takeo Kanade and Martial Hebert. First-personvision. Proceedings of the IEEE,100(8):2442–2453, 2012.

[15] Mathias Kolsch. Vision based hand gestureinterfaces for wearable computing and virtualenvironments. University of California, SantaBarbara, 2004.

[16] Mathias Kolsch and Matthew Turk. Robust handdetection. In FGR, pages 614–619, 2004.

[17] Robert LiKamWa, Zhen Wang, Aaron Carroll,Felix Xiaozhu Lin, and Lin Zhong. Draining ourglass: An energy and heat characterization ofgoogle glass. arXiv preprint arXiv:1404.1320,2014.

[18] Shai Litvak, Tomer Yanir, and Eran Guendelman.Tracking body parts by combined color image anddepth processing, July 22 2014. US Patent8,787,663.

[19] Haowei Liu, Matthai Philipose, MartinPettersson, and Ming-Ting Sun. Recognizingobject manipulation activities using depth andvisual cues. Journal of Visual Communicationand Image Representation, 25(4):719–726, 2014.

[20] Bruce D Lucas, Takeo Kanade, et al. An iterativeimage registration technique with an applicationto stereo vision. In IJCAI, volume 81, pages674–679, 1981.

[21] WW Mayol, B Tordoff, and DW Murray. Towardswearable active vision platforms. In Systems,Man, and Cybernetics, 2000 IEEE InternationalConference on, volume 3, pages 1627–1632. IEEE,2000.

[22] Antonio S Micilotta, Eng-Jon Ong, and RichardBowden. Real-time upper body detection and 3dpose estimation in monoscopic images. InComputer Vision–ECCV 2006, pages 139–150.Springer, 2006.

[23] Microsemi. Igloo nano fpgas.[24] Karsten Muhlmann, Dennis Maier, Jurgen Hesser,

and Reinhard Manner. Calculating densedisparity maps from color stereo images, anefficient implementation. International Journal ofComputer Vision, 47(1-3):79–88, 2002.

[25] Chris Murphy, Daniel Lindquist, Ann MarieRynning, Thomas Cecil, Sarah Leavitt, andMark L Chang. Low-cost stereo vision on an fpga.In Field-Programmable Custom ComputingMachines, 2007. FCCM 2007. 15th Annual IEEESymposium on, pages 333–334. IEEE, 2007.

[26] OpenCV. Haar feature-based cascade classifier forobject detection.

13

[27] Edgar Osuna, Robert Freund, and FedericoGirosi. Training support vector machines: anapplication to face detection. In Computer Visionand Pattern Recognition, 1997. Proceedings., 1997IEEE Computer Society Conference on, pages130–136. IEEE, 1997.

[28] Hamed Pirsiavash and Deva Ramanan. Detectingactivities of daily living in first-person cameraviews. In Computer Vision and PatternRecognition (CVPR), 2012 IEEE Conference on,pages 2847–2854. IEEE, 2012.

[29] Moo-Ryong Ra, A. Sheth, L. Mummert, P. Pillai,D. Wetherall, and R. Govindan. Odessa: Enablinginteractive perception applications on mobiledevices. In Mobisys, 2011.

[30] Swati Rallapalli, Aishwarya Ganesan, KrishnaChintalapudi, Venkat N Padmanabhan, and LiliQiu. Enabling physical analytics in retail storesusing smart glasses. In Proceedings of the 20thannual international conference on Mobilecomputing and networking, pages 115–126. ACM,2014.

[31] Henry A Rowley, Shumeet Baluja, and TakeoKanade. Neural network-based face detection.Pattern Analysis and Machine Intelligence, IEEETransactions on, 20(1):23–38, 1998.

[32] Yuki Sanada, Katsuki Ohata, Tetsuro Ogaki,Kento Matsuyama, Takanori Ohira, SatoshiChikuda, Masaki Igarashi, Tadahiro Kuroda,Masayuki Ikebe, Tetsuya Asai, et al. Fpgaimplementation of a memory-efficient stereo visionalgorithm based on 1-d guided filtering.

[33] Yale Song, David Demirdjian, and Randall Davis.Tracking body and hands for gesture recognition:Natops aircraft handling signals database. InAutomatic Face & Gesture Recognition andWorkshops (FG 2011), 2011 IEEE InternationalConference on, pages 500–506. IEEE, 2011.

[34] Ngo Huy Tan, Nor Hisham Hamid, PatrickSebastian, and Yap Vooi Voon. Resourceminimization in a real-time depth-map processingsystem on fpga. In TENCON 2011-2011 IEEERegion 10 Conference, pages 706–710. IEEE, 2011.

[35] Roger Y Tsai. A versatile camera calibrationtechnique for high-accuracy 3d machine visionmetrology using off-the-shelf tv cameras andlenses. Robotics and Automation, IEEE Journalof, 3(4):323–344, 1987.

[36] Paul Viola and Michael J Jones. Robust real-timeface detection. International journal of computervision, 57(2):137–154, 2004.

[37] He Wang, Xuan Bao, Romit Roy Choudhury, andSrihari Nelakuditi. Insight: recognizing humanswithout face recognition. In Proceedings of the

14th Workshop on Mobile Computing Systems andApplications, page 7. ACM, 2013.

[38] Fengliang Xu, Xia Liu, and Kikuo Fujimura.Pedestrian detection and tracking with nightvision. Intelligent Transportation Systems, IEEETransactions on, 6(1):63–71, 2005.

[39] Ming-Hsuan Yang and Narendra Ahuja. Facedetection and gesture recognition forhuman-computer interaction, volume 1. SpringerScience & Business Media, 2001.

[40] Woo-han Yun, DoHyung Kim, and Ho-Sub Yoon.Fast group verification system for intelligent robotservice. Consumer Electronics, IEEETransactions on, 53(4):1731–1735, 2007.

[41] Zhengyou Zhang. A flexible new technique forcamera calibration. Pattern Analysis and MachineIntelligence, IEEE Transactions on,22(11):1330–1334, 2000.

[42] Zhengyou Zhang. Camera calibration withone-dimensional objects. Pattern Analysis andMachine Intelligence, IEEE Transactions on,26(7):892–899, 2004.

[43] Zhenqiu Zhang, Gerasimos Potamianos, Ming Liu,and Thomas Huang. Robust multi-viewmulti-camera face detection inside smart roomsusing spatio-temporal dynamic programming. InAutomatic Face and Gesture Recognition, 2006.FGR 2006. 7th International Conference on,pages 407–412. IEEE, 2006.

14