multi-person multi-camera tracking for easyliving
DESCRIPTION
Multi-Person Multi-Camera Tracking for EasyLiving. Brian Meyers. Steve Harris. John Krumm. Vision Technology Research Group Microsoft Research Redmond, WA USA. Barry Brumitt. Steve Shafer. Michael Hale. What Is EasyLiving?. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/1.jpg)
Multi-Person Multi-Camera Tracking for EasyLiving
John Krumm Steve Harris Brian Meyers
Barry Brumitt Michael Hale Steve Shafer
Vision Technology Research GroupMicrosoft ResearchRedmond, WA USA
![Page 2: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/2.jpg)
What Is EasyLiving?EasyLiving is a prototype architecture and technologies for building intelligent environments that facilitate the unencumbered interaction of people with other people, with computers, and with devices.
![Page 3: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/3.jpg)
Example Behaviors
• Adjust lights as you move around a space
• Route video to best display
• Move your Windows session as you move
• Deliver e-messages to where you are
• Monitor a young child or old person
![Page 4: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/4.jpg)
EasyLiving Demo (7 min.)
![Page 5: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/5.jpg)
Self-Aware SpaceEasyLiving must know about people, computers, software, devices, and geometry to work right.
![Page 6: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/6.jpg)
Who’s Where?
![Page 7: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/7.jpg)
Person-TrackingSystem
5 Triclops stereo cameras
5 PCs running “Stereo Module”(and Microsoft Windows 2000)
1 PC running “Person Tracker”
(only U.S. $319)
(includes Internet Explorer)
(as part of the OS)
TriclopsColor Stereo
Cameras
Stereo Processingand Person Detection
PersonTracking
(for a limited time only?)
![Page 8: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/8.jpg)
Triclops Cameras
Now superceded by “Digiclops” digital IEEE-1394 version
![Page 9: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/9.jpg)
Typical Images
Color image from Triclops Disparity image from Triclops
![Page 10: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/10.jpg)
RequirementsTo work in a real-life intelligent environment, our tracking system must …
1. Maintain location & identity of people
2. Run at reasonable speeds (we get 3.5 Hz)
3. Work with multiple people (we handle up to three)
4. Create and delete people instances
5. Work with multiple cameras (we’re up to five)
6. Use cameras in the room
7. Work for extended periods
8. Tolerate partial occlusions and variable postures
![Page 11: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/11.jpg)
Other SystemsNon-Vision• Olivetti Research (’92) & Xerox PARC (’93) – IR badges• AT&T Laboratories (Cambridge) (’97) – Ultrasonic badges• PinPoint, Ascension, Polhemus – commercial RF badges
Vision (for multiple people)• Haritaoglu & Davis (’98-’99)• Darrell et al. (’98)• Orwell et al. (’99)• Collins et al. (’99)• Rosales & Sclaroff (’99)• Kettnaker & Zabih (’99)• Intille et al. (’95, ’97)• Rehg et al. (’97)• Boult et al. (‘99)• Stiefelhagen et al. (’99)• MacCormick & Blake (’99)• Cai & Aggarwal (’98)• Halevi & Weinshall (’97)• Gavrila & Davis (’96)
“I see by the current issue of ‘Lab News,’ Ridgeway, that you’ve been working for the last twenty years on the same problem I’ve been working on for the last twenty years.”
![Page 12: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/12.jpg)
Why Use Vision?Alternative sensors:• Active badges• Pressure-sensitive floors• Motion sensors• Localized sensors, e.g. on door, chair
But …• Cameras are getting cheap• Cameras are easy to install• Cameras give location and identity• Cameras can find other objects, e.g. video screens• Cameras can be use to model room geometry
(active badge)
![Page 13: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/13.jpg)
Person Detection Steps
1. Background subtraction2. Blob clustering3. Histogram identification
Camera calibrationBackground modeling
![Page 14: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/14.jpg)
Camera Calibration
0
0.5
1
1.5
2
2.5
3
3.5
4
-2 -1 0 1 2
0
0.5
1
1.5
2
2.5
3
3.5
4
-2 -1 0 1 2
• All tracking done in ground plane• Record path of single person walking around room• Compute (x,y,) that best aligns paths• Requires robust alignment to deal with outliers
Paths before calibration Paths after calibration
![Page 15: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/15.jpg)
Background Modeling
View of space Combined color & disparity background image
![Page 16: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/16.jpg)
Background SubtractionForeground if:
• valid depth over invalid depth- OR -
• depth difference > Td
- OR -• any (R,G,B) difference > Tc
• Color takes over when person sinks into couch cushions• Potential problem when person walks in front of moving video
(thus turn on moving video when acquiring background)
![Page 17: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/17.jpg)
Person Detection
Region-growing on foreground pixels gives fragmented blobs
Group blobs into people-shaped clusters
![Page 18: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/18.jpg)
Blob Clustering• Minimum spanning tree• Break really long links• Find five remaining longest links• Break all combinations of these five:
1 2 3 4 5
1 0 0 0 0 0
2 0 0 0 0 1
3 0 0 0 1 0
30 1 1 1 0 1
31 1 1 1 1 0
32 1 1 1 1 1
• Covariance matrices of 3D coordinates of linked blobs• Eigenvalues of covariance matrices• Compare eigenvalues to person model
![Page 19: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/19.jpg)
Color Histograms• Identify people with RGB color histograms, 16x16x16• Each camera PC maintains its own histograms• Space-variant histograms built as person moves around room• Person tracker uses histogram to resolve ambiguities
windowwindowBluish tint
Regular color
![Page 20: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/20.jpg)
So FarTriclops
Color StereoCameras
Stereo Processingand Person Detection
PersonTracking
calibration, background
• background subtraction (color & depth)• blob clustering• histogram maintenance
![Page 21: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/21.jpg)
Person Tracking• Takes reports from stereo modules• Transforms to common coordinate frame
(common coordinate frame)
![Page 22: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/22.jpg)
Person Tracking – Steady State
One “track” for each person
Predicted location
Resolve with color histograms
Feed back results to stereo modules for histogram updating
![Page 23: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/23.jpg)
Person Tracking – Bad Data
Measurement Noise:• Computed position based on predicted position from many reports
Occlusions:• Multiple cameras• Long timeout on unsupported tracks
![Page 24: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/24.jpg)
Person Creation Zone
• Tracks begin and end here• Initial tracks are provisional• Makes remainder of room more robust
![Page 25: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/25.jpg)
Summary• Live demos, 20 minutes long• Person tracker runs at 3.5 Hz• Up to three people in room• People can:
• enter• leave• walk around• stop moving• sit• collide
![Page 26: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/26.jpg)
Recent Efforts• Stop breaking the vision system!
• Moved chairs & changing lights bad background model• Special behavior, e.g. slow through person creation zone• Lots of people, e.g. around conference table
• Find other objects to enable interesting behaviors, e.g. “Where’s that book?”
• Easier method to model room geometry
![Page 27: Multi-Person Multi-Camera Tracking for EasyLiving](https://reader035.vdocuments.site/reader035/viewer/2022062809/56815735550346895dc4d6cc/html5/thumbnails/27.jpg)
Workshop on Multi-Object Tracking