kitchen occupation project presentation
TRANSCRIPT
![Page 1: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/1.jpg)
Kitchen OccupationTSBB11
2013
![Page 2: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/2.jpg)
Problem
• Students need student kitchens to heat food.• Students want to be able to find the least occupied
kitchen.• The university wants to make informed decisions on
where and if more kitchens are needed.
![Page 3: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/3.jpg)
The Group (main contributions)• Mattias Tiger – Project Leader, Software Architecture• Gustav Häger – System Integration, embedded• Erik Fall – Tracking, counting• Malin Rudin – Evaluation• Alexander Sjöholm – User and sensor interfaces• Nikolaus West – Quality control, queue detection• Martin Svensson – Documentation, system I/O
![Page 4: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/4.jpg)
System overview
• Counts entries and exits to room• Estimates severity of potential queues• Communicates to server via HTTP
![Page 5: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/5.jpg)
System overview
Kinect Kinect
Room Entrance
Computer
![Page 6: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/6.jpg)
System Overview
• Highly configurable • Simple to Calibrate• Runs with and without GUI• Easily extendable– We ended up creating a great, reusable,
architecture for doing computer vision.
![Page 7: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/7.jpg)
Initial solution
• Initial plan to use network cameras and process on a server
• Used a background model with basic tracker• Occlusion was a large problem
![Page 8: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/8.jpg)
Changing circumstances
• Could not send data over the network• Needed to process on the spot (on a
cheap/slow computer)• Background model needs to process each
pixel, a bit slow• A new approach was needed.
![Page 9: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/9.jpg)
New approach
• Needed some way to segment humans• Idea to detect only heads, or only movement• Optical flow for movment• Hough circles for heads
![Page 10: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/10.jpg)
Tested approaches – hough circles
• Use hough circles to detect heads in the image
![Page 11: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/11.jpg)
Tested approaches – hough circles
• Hough circle method is not robust enough
![Page 12: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/12.jpg)
Tested approaches
• Segmentation based on optical flow• To computationally heavy• Not fast enough for a cheap computer
![Page 13: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/13.jpg)
New approach again – depth data
• Tried using depth data• Stereo is hard to calibrate, and slow to
calculate• Very easy and cheap to aquire using a kinect
![Page 14: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/14.jpg)
Image processing
• Human segmentation• Depth image from Microsoft Kinect camera• Simple assumptions
![Page 15: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/15.jpg)
Image processing
![Page 16: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/16.jpg)
Image processing
• Occlusion handling
![Page 17: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/17.jpg)
Tracking
• Segmentation not enough for solving our problem, need some approach for counting
• How to track appearing objects in a robust way
• Follow objects from one frame to another – Pairing closest objects
![Page 18: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/18.jpg)
Main error sources
• Objects randomly appearing, outliers– Maximum pairing distance limits
• Occlusion– Keep lost objects alive
• Noise, flickering objects – Minimum lifetime before considered human
counted
![Page 19: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/19.jpg)
![Page 20: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/20.jpg)
Counting
• Registrating entries & exits– Door area• New object Entries + 1• Object lost Exits + 1• Not enough
– Checkpoints • Circle lines
– Removes some false entries & exits
![Page 21: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/21.jpg)
![Page 22: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/22.jpg)
Queue Severity Estimation
• What is a queue?• People traveling slowly along the same path• Position, speed and movement factor in.
![Page 23: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/23.jpg)
Queue Severity Estimation
• Cubic Beziér Splines completely defined by:– Two endpoint positions (person positions)– Velocities at endpoint (person velocities)– Curve parametrization (relative weight between
velocities and positions)
![Page 24: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/24.jpg)
Queue Severity Estimation• People connected by short spline in same
queue• Proportion of frames with visible queues
determines queue severity
![Page 25: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/25.jpg)
System Setup
• Two steps– Calibration– Configuration
• Target user computer vision skills– None
![Page 26: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/26.jpg)
Calibration
• What is the current height of the sensor?– Calibrate!
![Page 27: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/27.jpg)
Configuration
• Just listen tothe manual• Specify – Doors– Checkpoints– Exclusions
![Page 28: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/28.jpg)
Configuration
![Page 29: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/29.jpg)
Dense Debugger
• Where did it go wrong?– Intuitive debugging
• Speed is crucial– Built-in profiling
• Talk to me– Verbose logger
![Page 30: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/30.jpg)
Dense Debugger
![Page 31: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/31.jpg)
The Grid
![Page 32: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/32.jpg)
System Input
• Initially only from pre-recorded data• Need to run from network cameras– Control multiple sensors at once
• Desire to manage different sensor types– Enable easy integration of other sensor types
![Page 33: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/33.jpg)
System Input• Final supported video input– Any OpenCV-compatible video files– Any OpenCV-compatible cameras– Microsoft Kinect• RGB• Depth
• Developed with modularity and extendability taken into account
![Page 34: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/34.jpg)
System Output
• Transmit desired information using HTTP– Make it easy to change output information– Possibility to easily switch target servers
![Page 35: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/35.jpg)
System settings
• Loads of different parameters in the code– Some algorithms have more than 10 parameters– Some parameters are calibrated using the GUI• Height threshold• Queue sensitivity
– Camera types and network camera locations
![Page 36: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/36.jpg)
System settings
• One configuration file– Easy to tune different parameters– Good portability– Tune using GUI and then switch to the embedded
version• Automatically writes all settings to file after
program termination.
![Page 37: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/37.jpg)
System settings
Example file with required parameters.
![Page 38: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/38.jpg)
Software architecture
• Cross platform• Modular– Extendable– Reusable
• Configurable• Ideal for rapid prototyping & testing
![Page 39: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/39.jpg)
PerformanceData sets
![Page 40: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/40.jpg)
PerformanceData sets
![Page 41: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/41.jpg)
PerformanceData sets
![Page 42: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/42.jpg)
Performance
Evaluation metric
![Page 43: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/43.jpg)
Performance
Queue severity
![Page 44: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/44.jpg)
Results
Sequence name Total number of entered (GT)
Ain Total number of exited (GT)
Aout Length
R-Kitchen 108 (108) 100% 101 (104) 97% 32 min
U-Kitchen 123 (122) 99% 134 (135) 99% 31 min
B25-Kitchen 131 (141) 93% 82 (91) 90% 30 min
![Page 45: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/45.jpg)
Results
R-Kitchen: Number of entries (esitmated and ground truth).
![Page 46: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/46.jpg)
Results
U-Kitchen: Accuracy of entries.
![Page 47: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/47.jpg)
Results
B25-Kitchen: Number of exits (estimated and ground truth).
![Page 48: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/48.jpg)
The final product• A cheap and accurate system for monitoring room
usage• A very general purpose image processing pipeline• Total configuration without re-compiling• Great debug and installation user interfaces• Support for built-in, automatic evaluation of
algorithms
![Page 49: Kitchen Occupation Project Presentation](https://reader031.vdocuments.site/reader031/viewer/2022032619/55c0e987bb61eb77678b46d3/html5/thumbnails/49.jpg)
Questions?