![Page 1: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/1.jpg)
Advanced Practical Course: Sensor-enabled Intelligent Environments
Barcode-based Object Recognition
Final Presentation
Presented by:Nacer KHALIL
Supervised by:Dejan PANGERCIC
1
![Page 2: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/2.jpg)
Table of content
I- Overall project goal
II- Autofocus
III- Bacode decoding
IV- information retrieval
V- Barcode localization
VI- Conclusion
2
![Page 3: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/3.jpg)
II-AutofocusHow autofocus works
Active vs passive autofocus
Courtesy of howstuffworks.com 3
![Page 4: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/4.jpg)
II-Autofocus(continued)
4
![Page 5: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/5.jpg)
II- AutofocusImplementation in the project
Used camera: Logitech QC PRO 9000Driver used: ROS::uvc_cameraProblem: Autofocus is not supported by the driverSolution:
Autofocus was added to uvc_camera driverAutofocus algorithm was taken from GUVCVIEW
software and integrated within uvc_camera driver
5
![Page 6: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/6.jpg)
II- Autofocus result
6
![Page 7: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/7.jpg)
III-Barcode decodingHow Zbar works
Row 1 Row 2 Row 3 Row 40
2
4
6
8
10
12
Column 1
Column 2
Column 3
Courtesy of Jeff Brown7
![Page 8: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/8.jpg)
IV-Information retrieval
Barcoo is a product information store that has a database composed of 7 million commercial objects.
Access to this database was granted to us.Communication to the database is done through
HTTP protocol.Request: an http link containing the barcodeResponse: XML file containing all information about
the object8http://www.barcoo.com
![Page 9: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/9.jpg)
IV- Information retrievalBarcoo request response example
Request: http://www.barcoo.com/api/get_product_complete? Pi=73705207908
&pins=ean& ;format=xml&source=ias-tum
Response: We are parsing for:- Image- product name- category- producer
9
![Page 10: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/10.jpg)
V- Barcode localizationTechniques used
Techniques used to find the barcode region of interest– Blob-based barcode localization– Parallel line-based localization– Adjacent line-based localization
10
![Page 11: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/11.jpg)
V- Barcode localizationBlob-based localization(working example)
11
![Page 12: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/12.jpg)
V- Barcode localizationBlob-based localization (not working example)
12
![Page 13: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/13.jpg)
V- Barcode localizationAdjacent line-based localization
13
![Page 14: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/14.jpg)
V-Barcode localizationHow adjacent line-based localization works
14
![Page 15: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/15.jpg)
V-Barcode localizationAdjacent line-based approach explanation
- Take picture-Convert to grayscale-Parameters: interval size, min/max # of transitions, max Jeffrie’s value, min # of rows per ROI
255 15 56 54 84 165 75 0
250 20 60 84 120 0 240 97
248 18 61 0 13 51 15 85
246 17 55 70 55 52 0 200
1 0 2 2 2 2
1 0 1 2 2 2
1 0 2 1 2 2
1 0 2 1 2 2
Image matrix
Transitions matrix
1 0 -1 -1 -1 -1
1 0 1 -1 -1 -1
1 0 -1 1 -1 -1
1 0 -1 1 -1 -1
Eliminated intervals
15
![Page 16: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/16.jpg)
0,2 1 5,2 8,4 5,3 1,3
1,2 2 2,4 2,4 6,7 1
0,5 1 3,2 0,1 8,4 2,4
Jeffrie ’s distance matrix
1 0 -1 -1 -1 -1
1 0 1 -1 -1 -1
1 0 -1 1 -1 -1
1 0 -1 1 -1 -1
0,2 1 -1 -1 -1 -1
1,2 2 0 -1 -1 -1
0,5 1 -1 0,1 -1 -1
Eliminated intervalsmatrix
Final matrix
V-Barcode localizationAdjacent line-based approach explanation (continued)
16
![Page 17: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/17.jpg)
IV- Barcode localizationAdjacent line-based localization - results
17
![Page 18: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/18.jpg)
18
Open Source Code
Packages list:-zbar_barcode_reader_node-zbar_qt_ros-uvc_camera-barcode_detection
Repositories:-http://code.cs.tum.edu/indefero/index.php//p/seie2011fall/source/tree/HEAD/khalil-http://code.cs.tum.edu/indefero/index.php//p/ias-perception/source/tree/master/
![Page 19: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/19.jpg)
ConclusionProject is composed of three parts:
Barcode localizationImplementation of autofocusInformation retrieval of objects
Future work:Creation of the barcoo ontology and storage on
KnowRobIntegration and testing on PR2Integration with object modeling center
19
![Page 20: Advanced Practical Course: Sensor-enabled Intelligent Environments Barcode-based Object Recognition Final Presentation Presented by: Nacer KHALIL Supervised](https://reader035.vdocuments.site/reader035/viewer/2022070407/56649e195503460f94b05e6f/html5/thumbnails/20.jpg)
20
Demonstrations of the project in the kitchen lab after the presentations end