writing apps that can see: getting data from coreimage to computer vision - 360idev 2013...
DESCRIPTION
Apps that Can See: Getting Data from CoreImage to Computer Vision You hear about the app that can solve a Rubik's Cube after you take pictures of each side? How about the one that can import Sudoku puzzles by letting you take a picture of a puzzle? Wouldn't it be cool if your apps could get information that way? Come and learn how to integrate image processing in your apps. We'll start with Apple's built-in CoreImage libraries, and then move on to open-source C++ libraries that let you detect colors, shapes, letters and numbers. We'll go through quite a bit of sample code that you can take with you to use in your apps. We'll also talk about practical tips and experiences, like how to debug and troubleshoot code when things aren't working the way you expected.TRANSCRIPT
#360iDev@CarlBrwn
Apps That Can SeeCarl Brown
Missing Last Vowel -I blame Flickr/Tumblr
1Monday, September 9, 13
@CarlBrwn #360iDev
Computer Vision?Turning Photos Into Useable Data
Image Analysis
Colors
Shapes
Numbers & Letters
2Monday, September 9, 13
@CarlBrwn #360iDev
Word Lens
http://www.youtube.com/watch?v=h2OfQdYrHRs
3Monday, September 9, 13
@CarlBrwn #360iDev
Word Lens
http://www.youtube.com/watch?v=h2OfQdYrHRs
3Monday, September 9, 13
@CarlBrwn #360iDev
Today's TalkHow quick and easy it is to get actionable info from photos
4Monday, September 9, 13
@CarlBrwn #360iDev
Today's TalkHow quick and easy it is to get actionable info from photos
Demo Code with Source
4Monday, September 9, 13
@CarlBrwn #360iDev
Today's TalkHow quick and easy it is to get actionable info from photos
Demo Code with Source
Not about HOW OpenCV works
4Monday, September 9, 13
@CarlBrwn #360iDev
Today's TalkHow quick and easy it is to get actionable info from photos
Demo Code with Source
Not about HOW OpenCV works
"OpenCV by Example"
4Monday, September 9, 13
@CarlBrwn #360iDev
CubeCheater &Sudoku Grab
http://www.youtube.com/watch?v=HNwx0nbgm7M http://www.youtube.com/watch?v=wt96OomJY9A
5Monday, September 9, 13
@CarlBrwn #360iDev
Both have had Legal Problems
http://cubecheater.efaller.com http://sudokugrab.blogspot.com/2010/12/sudoku-grab-removed-from-sale-in-japan.html
6Monday, September 9, 13
#360iDev@CarlBrwn
Based onCode From:github.com/carlbrown/OpenCVDemo360iDev2013
7Monday, September 9, 13
@CarlBrwn #360iDev
About the CodeExplicit and Redundant to make it easier to follow/explain
8Monday, September 9, 13
@CarlBrwn #360iDev
About the CodeExplicit and Redundant to make it easier to follow/explain
Not production ready - for teaching purposes
8Monday, September 9, 13
@CarlBrwn #360iDev
About the CodeExplicit and Redundant to make it easier to follow/explain
Not production ready - for teaching purposes
Minimum Viable Interface
8Monday, September 9, 13
@CarlBrwn #360iDev
About the CodeExplicit and Redundant to make it easier to follow/explain
Not production ready - for teaching purposes
Minimum Viable Interface
Don't hate, okay?
8Monday, September 9, 13
#360iDev@CarlBrwn
Face CounterDemo 1/3
9Monday, September 9, 13
#360iDev@CarlBrwn
Core ImageOption 1
10Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- coreImageButtonPressed:
[CIImage imageWithCGImage:]Detect
11Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- coreImageButtonPressed:
[CIImage imageWithCGImage:][CIDetector detectorOfType:! ! ! ! CIDetectorTypeFace]
Detect
11Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- coreImageButtonPressed:
[CIImage imageWithCGImage:][CIDetector detectorOfType:! ! ! ! CIDetectorTypeFace]NSArray *features= [detector ! ! ! ! ! ! featuresInImage:]
Detect
11Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- coreImageButtonPressed:
[CIImage imageWithCGImage:][CIDetector detectorOfType:! ! ! ! CIDetectorTypeFace]NSArray *features= [detector ! ! ! ! ! ! featuresInImage:]for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)
Detect
11Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- coreImageButtonPressed:
for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)
Debugging
12Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- coreImageButtonPressed:
for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.bounds
Debugging
12Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- coreImageButtonPressed:
for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.boundsframe *= contentScaleFactor
Debugging
12Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- coreImageButtonPressed:
for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.boundsframe *= contentScaleFactor//flip to CIImage Coords
Debugging
12Monday, September 9, 13
@CarlBrwn #360iDev
Coordinate Systems
UIView has (0,0) at Upper Left
Core Image has (0,0) Lower Left
(Like OS X)
13Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- coreImageButtonPressed:
for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.boundsframe *= contentScaleFactor//flip to CIImage Coords
Debugging
14Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- coreImageButtonPressed:
for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.boundsframe *= contentScaleFactor//flip to CIImage CoordsfaceView.layer.borderWidth = 4;
Debugging
14Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- coreImageButtonPressed:
for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.boundsframe *= contentScaleFactor//flip to CIImage CoordsfaceView.layer.borderWidth = 4;addSubview:faceView
Debugging
14Monday, September 9, 13
#360iDev@CarlBrwn
Core ImageOption 1
15Monday, September 9, 13
#360iDev@CarlBrwn
OpenCVOption 2
16Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- openCVButtonPressed:
convert to cv::MatDetect
17Monday, September 9, 13
@CarlBrwn #360iDev
Convert to cv::MatCGImageGetColorSpace(image.CGImage)
cv::Mat cvMat()
CGBitmapContextCreate(cvMat.data,...)
CGContextDrawImage(ctx,r,image.CGImage)
CGContextRelease()
18Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- openCVButtonPressed:
convert to cv::MatDetect
19Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- openCVButtonPressed:
convert to cv::Matcv::cvtColor( CV_BGR2GRAY )
Detect
19Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- openCVButtonPressed:
convert to cv::Matcv::cvtColor( CV_BGR2GRAY )cv::equalizeHist()
Detect
19Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- openCVButtonPressed:
convert to cv::Matcv::cvtColor( CV_BGR2GRAY )cv::equalizeHist()cv::face_cascade.\! ! ! ! ! ! detectMultiScale()
Detect
19Monday, September 9, 13
@CarlBrwn #360iDev
Cascade Classifierhttp://docs.opencv.org/doc/tutorials/objdetect/cascade_classifier/cascade_classifier.html
20Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- openCVButtonPressed:
convert to cv::Matcv::cvtColor( CV_BGR2GRAY )cv::equalizeHist()face_cascade.detectMultiScale()
Detect
21Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- openCVButtonPressed:
convert to cv::Matcv::cvtColor( CV_BGR2GRAY )cv::equalizeHist()face_cascade.detectMultiScale()for( int i = 0; ! ! ! ! i < faces.size(); i++ )
Detect
21Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- openCVButtonPressed:
for( int i = 0; ! ! ! ! i < faces.size(); i++ )
Debugging
22Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- openCVButtonPressed:
for( int i = 0; ! ! ! ! i < faces.size(); i++ )CGRectMake(faces[i].frame)
Debugging
22Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- openCVButtonPressed:
for( int i = 0; ! ! ! ! i < faces.size(); i++ )CGRectMake(faces[i].frame)frame *= contentScaleFactor
Debugging
22Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- openCVButtonPressed:
for( int i = 0; ! ! ! ! i < faces.size(); i++ )CGRectMake(faces[i].frame)frame *= contentScaleFactorfaceView.layer.borderWidth = 4;
Debugging
22Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- openCVButtonPressed:
for( int i = 0; ! ! ! ! i < faces.size(); i++ )CGRectMake(faces[i].frame)frame *= contentScaleFactorfaceView.layer.borderWidth = 4;addSubview:faceView
Debugging
22Monday, September 9, 13
#360iDev@CarlBrwn
OpenCVOption 2
23Monday, September 9, 13
#360iDev@CarlBrwn
CubeFaceGrabberDemo 2/3
24Monday, September 9, 13
#360iDev@CarlBrwn
DetectEdgesdetectFacesPressed:
25Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- detectFacesPressed:
convert to cv::MatDetect
26Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- detectFacesPressed:
convert to cv::Mat
cv::Canny()
Detect
26Monday, September 9, 13
@CarlBrwn #360iDev
Canny27Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- detectFacesPressed:
convert to cv::Mat
cv::Canny()
Detect
28Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- detectFacesPressed:
convert to cv::Mat
cv::Canny()cv::HoughLinesP()
Detect
28Monday, September 9, 13
@CarlBrwn #360iDev
HoughLinesP29Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- detectFacesPressed:
convert to cv::Mat
cv::Canny()cv::HoughLinesP()
Detect
30Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- detectFacesPressed:
convert to cv::Mat
cv::Canny()cv::HoughLinesP()for( size_t i = 0; ! ! ! i < lines.size(); i++ )
Detect
30Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- detectFacesPressed:
convert to cv::Mat
cv::Canny()cv::HoughLinesP()for( size_t i = 0; ! ! ! i < lines.size(); i++ )
if (LineIntersect(l, l2))
Detect
30Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- detectFacesPressed:
convert to cv::Mat
cv::Canny()cv::HoughLinesP()for( size_t i = 0; ! ! ! i < lines.size(); i++ )
if (LineIntersect(l, l2))//Measure Distance to Image corners
Detect
30Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- detectFacesPressed:
for( size_t i = 0; ! ! ! i < lines.size(); i++ )
Debugging
31Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- detectFacesPressed:
for( size_t i = 0; ! ! ! i < lines.size(); i++ )cv::line() //Draw
Debugging
31Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- detectFacesPressed:
for( size_t i = 0; ! ! ! i < lines.size(); i++ )cv::line() //Draw//Check for bounding corners
Debugging
31Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- detectFacesPressed:
for( size_t i = 0; ! ! ! i < lines.size(); i++ )cv::line() //Draw//Check for bounding cornerscv::circle() //Draw
Debugging
31Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- detectFacesPressed:
for( size_t i = 0; ! ! ! i < lines.size(); i++ )cv::line() //Draw//Check for bounding cornerscv::circle() //Drawconvert to UIImage //Display
Debugging
31Monday, September 9, 13
@CarlBrwn #360iDev
Convert to UIImage
[NSData dataWithBytes:cvMat.data length:]
CGDataProviderCreateWithCFData()
CGImageCreate()
[[UIImage alloc] initWithCGImage:]
CGXXXRelease()
32Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- detectFacesPressed:
for( size_t i = 0; ! ! ! i < lines.size(); i++ )cv::line() //Draw//Check for bounding cornerscv::circle() //Drawconvert to UIImage //Display
Debugging
33Monday, September 9, 13
#360iDev@CarlBrwn
DetectEdgesdetectFacesPressed:
34Monday, September 9, 13
#360iDev@CarlBrwn
UnskewgetAndApplyTransformPressed:
35Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- getAndApplyTransformPressed:
convert to cv::Mat
36Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- getAndApplyTransformPressed:
convert to cv::MatgetPerspectiveTransform()
36Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- getAndApplyTransformPressed:
convert to cv::MatgetPerspectiveTransform()warpPerspective(transform)
36Monday, September 9, 13
@CarlBrwn #360iDev
Map the Corners circled on the Left
To the Corners of the Images on the Right
37Monday, September 9, 13
@CarlBrwn #360iDev
Map the Corners circled on the Left
To the Corners of the Images on the Right
37Monday, September 9, 13
@CarlBrwn #360iDev
Map the Corners circled on the Left
To the Corners of the Images on the Right
37Monday, September 9, 13
@CarlBrwn #360iDev
Map the Corners circled on the Left
To the Corners of the Images on the Right
37Monday, September 9, 13
@CarlBrwn #360iDev
Map the Corners circled on the Left
To the Corners of the Images on the Right
37Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- getAndApplyTransformPressed:
convert to cv::MatgetPerspectiveTransform()warpPerspective(transform)
38Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- getAndApplyTransformPressed:
convert to cv::MatgetPerspectiveTransform()warpPerspective(transform)convert to UIImage //Display
38Monday, September 9, 13
#360iDev@CarlBrwn
UnskewgetAndApplyTransformPressed:
39Monday, September 9, 13
#360iDev@CarlBrwn
ExtractColorsextractColorsPressed:
40Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractColorsPressed:
convert to cv::Matcv::cvtColor(CV_RGB2HSV)
Detect
41Monday, September 9, 13
@CarlBrwn #360iDev
HSV Space
Hue/Saturation/Value
Easier to get Color out
http://en.wikipedia.org/wiki/HSV_color_space42Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractColorsPressed:
convert to cv::Matcv::cvtColor(CV_RGB2HSV)
Detect
43Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractColorsPressed:
convert to cv::Matcv::cvtColor(CV_RGB2HSV)for (int hSlice;;) { ! ! for (int vSlice;;) {
Detect
43Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractColorsPressed:
convert to cv::Matcv::cvtColor(CV_RGB2HSV)for (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()
Detect
43Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractColorsPressed:
convert to cv::Matcv::cvtColor(CV_RGB2HSV)for (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)
Detect
43Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractColorsPressed:
convert to cv::Matcv::cvtColor(CV_RGB2HSV)for (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)avgColor=cv::mean(roi)
Detect
43Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractColorsPressed:
convert to cv::Matcv::cvtColor(CV_RGB2HSV)for (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)avgColor=cv::mean(roi)//Check Saturation and Hue
Detect
43Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractColorsPressed:
for (int hSlice;;) { ! ! for (int vSlice;;) {
Debugging
44Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractColorsPressed:
for (int hSlice;;) { ! ! for (int vSlice;;) { roi = avgColor //Fill
Debugging
44Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractColorsPressed:
for (int hSlice;;) { ! ! for (int vSlice;;) { roi = avgColor //FillputText(roi, [color cString]
Debugging
44Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractColorsPressed:
for (int hSlice;;) { ! ! for (int vSlice;;) { roi = avgColor //FillputText(roi, [color cString]putText(roi, stringWithFormat:@"%.0f/%.0f/%.0f")
Debugging
44Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractColorsPressed:
for (int hSlice;;) { ! ! for (int vSlice;;) { roi = avgColor //FillputText(roi, [color cString]putText(roi, stringWithFormat:@"%.0f/%.0f/%.0f")
convert to UIImage //Display
Debugging
44Monday, September 9, 13
#360iDev@CarlBrwn
ExtractColorsextractColorsPressed:
45Monday, September 9, 13
#360iDev@CarlBrwn
SudokuGrabberDemo 3/3
46Monday, September 9, 13
#360iDev@CarlBrwn
FindEdgesfindPuzzlePressed:
47Monday, September 9, 13
@CarlBrwn #360iDev
Just like Cube
48Monday, September 9, 13
#360iDev@CarlBrwn
UnSkewPuzzlesquarePressed:
49Monday, September 9, 13
@CarlBrwn #360iDev
Just like Cube
50Monday, September 9, 13
#360iDev@CarlBrwn
ExtractTextextractPressed:
51Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractPressed:
Tesseract* tesseract =Setup
52Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractPressed:
Tesseract* tesseract =initWithDataPath:@"tessdata" language:@"eng"
Setup
52Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractPressed:
Tesseract* tesseract =initWithDataPath:@"tessdata" language:@"eng"tesseract setVariableValue:@"0123456789"forKey:@"tessedit_char_whitelist"
Setup
52Monday, September 9, 13
@CarlBrwn #360iDev
tesseract-ocrhttp://code.google.com/p/tesseract-ocr/
53Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractPressed:
convert to cv::MatSlice
54Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractPressed:
convert to cv::Matfor (int hSlice;;) { ! ! for (int vSlice;;) {
Slice
54Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractPressed:
convert to cv::Matfor (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()
Slice
54Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractPressed:
convert to cv::Matfor (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)
Slice
54Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractPressed:
convert to cv::Matfor (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)cv::cvtColor(CV_RGB2GRAY);
Slice
54Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractPressed:
convert to cv::Matfor (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)cv::cvtColor(CV_RGB2GRAY);cv::threshold(roi_gray)
Slice
54Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractPressed:
convert slice to UIImage Detect
55Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractPressed:
convert slice to UIImage tesseract setImage:imageToParse
Detect
55Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractPressed:
convert slice to UIImage tesseract setImage:imageToParsetesseract recognize
Detect
55Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractPressed:
[[UILabel alloc]initWithFrame:CGRectMake()]
Debugging
56Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractPressed:
[[UILabel alloc]initWithFrame:CGRectMake()] labelsetText:recognizedString
Debugging
56Monday, September 9, 13
@CarlBrwn #360iDev
ViewController.mm- extractPressed:
[[UILabel alloc]initWithFrame:CGRectMake()] labelsetText:recognizedStringself.viewaddSubview:recognizedLabel
Debugging
56Monday, September 9, 13
#360iDev@CarlBrwn
ExtractTextextractPressed:
57Monday, September 9, 13
#360iDev@CarlBrwn
Based onCode From:github.com/carlbrown/OpenCVDemo360iDev2013
Questions?
58Monday, September 9, 13