real-time camera-based character recognition free from layout constraints
DESCRIPTION
Real-Time Camera-Based Character Recognition Free from Layout Constraints. M. Iwamura , T. Tsuji, A. Horimatsu , and K. Kise. Real-Time Camera-Based Character Recognition System. Recognizes ~200 characters/sec. Recognizes characters immediately!. Web camera. IMP. Capture. Document. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/1.jpg)
Real-Time Camera-Based Character Recognition Free from Layout Constraints
M. Iwamura, T. Tsuji, A. Horimatsu, and K. Kise
![Page 2: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/2.jpg)
IMPWeb camera
Document
Recognizes characters immediately!
Capture
Real-Time Camera-Based Character Recognition SystemRecognizes ~200 characters/sec
![Page 3: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/3.jpg)
DEMO
![Page 4: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/4.jpg)
Applications
Car-free mall
Voice navigation forvisually disabled people
Translation service forforeign travelers
Recognizes all characters in a scene and provide useful information only
♪♪
“Push button” is on your right side
![Page 5: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/5.jpg)
Recognizes designed characters and pictograms
1: Real-time・ Recognizes ~200 characters/sec2: Robust to perspective distortion・ Recognition accuracy is >80% in 45 deg.
3: Layout free
3 Advantages of theProposed Method
First method that realizes three requirements
![Page 6: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/6.jpg)
Existing Methods and Problems1. Real-time recognition capable only for
characters in a straight text line
2. Can recognize each character in a complex layout with much computational time
Recognizable
Not recognizable
![Page 7: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/7.jpg)
Existing Methods vs Proposed Method
Kusachi 2004Li 2008
Myers 2004
Proposed methodRecognition of
Individual Characters
2: Perspective distortion 3: Layout free
Real-time Processing
1: Real-time
![Page 8: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/8.jpg)
Contents1. Background2. Overview of the Proposed Method3. Contour Version of Geometric Hashing4. Proposed Method
1. Real-Time Processing2. Recognition of Separated Characters3. Pose Estimation
5. Experiment6. Conclusion
![Page 9: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/9.jpg)
Overview of theProposed Method 1
Recognizes individual connected components
Assumptions Black characters are
written on a flat white paper
All connected components are easily segmented
S c h o o l
3: Layout freeRealizes
iHandled by post processing
How to quickly match segmented connected
components
![Page 10: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/10.jpg)
A
Overview of the Proposed Method 2 Affine invariant
recognition Three corresponding
points help matching
Input Image
Reference Image
Normalization
Normalization
Realizes robust recognition to 2: Perspective
distortion
Match
![Page 11: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/11.jpg)
Overview of the Proposed Method 2: Contour Version of Geometric Hashing
ANo. of Points :P
Matching of point arrangement Matching of Shape
Existing method : Geometric Hashing (GH)
Contour Version of GHStart point of the proposed method
Applied GH to recognition of CCs
![Page 12: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/12.jpg)
Overview of the Proposed Method 3:Three-Point Arrangements of CVGH CVGH examines all three points out of P
points
P
1st 2nd 3rd
(P-2)
(P-1)
× × = O(P3)
Database
No. of Patterns
![Page 13: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/13.jpg)
Overview of the Proposed Method 3:Three-Point Arrangements of Prop. Method
Proposed method snips useless three-point arrangements
1 1P× × = O(P)
1st 2nd 3rd Database
No. of Patterns
O(P3)
In case of P=100CVGH
Proposed Method970,200
100 Realizes 1: Real-time
![Page 14: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/14.jpg)
Contents1. Background2. Overview of the Proposed Method3. Contour Version of Geometric Hashing4. Proposed Method
1. Real-Time Processing2. Recognition of Separated Characters3. Pose Estimation
5. Experiment6. Conclusion
![Page 15: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/15.jpg)
A
Contour Version of GH:Matching by Feature Vectors Calculation of feature vector
1. Normalize2. Divide into subregions3. Create a histogram of black pixel4. Quantize
0 1 2 11 2 ...
Feature Vector
4x4 Mesh Feature
![Page 16: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/16.jpg)
Contour Version of GH:Storage Feature vectors are stored in the hash table
AAA
Hash table
0123456
…
Hash ID : 1
Hash ID : 5
Hash ID : 2
![Page 17: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/17.jpg)
Contour Version of GH:Recognition1. Calculate feature vectors2. Cast votes
A B ... R ...
0123456
…Result
A
ID : 1 ID : 5 ID : 2
Hash table
![Page 18: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/18.jpg)
Contents1. Background2. Overview of the Proposed Method3. Contour Version of Geometric Hashing4. Proposed Method
1. Real-Time Processing2. Recognition of Separated Characters3. Pose Estimation
5. Experiment6. Conclusion
![Page 19: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/19.jpg)
A
Proposed Method 1:Real-Time Processing by Affine Invariant Area ratio
Three-point arrangement Area ratio
S1 S’1
=S1
S0
S’1
S’0
S0 S’0
Usual usage
Area Ratio
Affine Invariant
![Page 20: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/20.jpg)
Proposed Method 1:Real-Time Processing by Affine Invariant Area ratio
Two-point arrangement + Area ratio Third point
Unusual usage
AS1 S’1
=S1
S0
S’1
S’0
S0 S’0
Area Ratio
Affine Invariant
![Page 21: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/21.jpg)
Proposed Method 1:How to Select Three Points
1st point: Centroid (Affine Invariant) 2nd point: Arbitrary point out of P points 3rd point: Determined by the area ratio
ANo. of Points :P
Uniquely Determined
![Page 22: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/22.jpg)
Contents1. Background2. Overview of the Proposed Method3. Contour Version of Geometric Hashing4. Proposed Method
1. Real-Time Processing2. Recognition of Separated Characters3. Pose Estimation
5. Experiment6. Conclusion
![Page 23: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/23.jpg)
CC Char. Relative Position
Area of CC
Area of corresponding CC
i 5 25j 5 40i 25 5j 40 5
Proposed Method 2:Recognition of Separated Characters Create a separated character table for post
processing
Area: 5
Area: 40
Stored
![Page 24: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/24.jpg)
Contents1. Background2. Overview of the Proposed Method3. Contour Version of Geometric Hashing4. Proposed Method
1. Real-Time Processing of CVGH2. Recognition of Separated Characters3. Pose Estimation
5. Experiment6. Conclusion
![Page 25: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/25.jpg)
Proposed Method 3:Pose Estimation Estimates affine parameters from
correspondences of three points
AAffine Transformatio
nParameters
Independent Scaling Shear Rotation Scaling
Pose of Paper Pose of Characters
![Page 26: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/26.jpg)
Contents1. Background2. Overview of the Proposed Method3. Contour Version of Geometric Hashing4. Proposed Method
1. Real-Time Processing2. Recognition of Separated Characters3. Pose Estimation
5. Experiment6. Conclusion
![Page 27: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/27.jpg)
Experiment:Recognition Target 236 Chars3
Fonts
![Page 28: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/28.jpg)
Experiment:Recognition Target Captured from three different angles A server was used
CPU: AMD Opteron 2.6GHz
Angle : 45 deg.Angle : 0 deg. Angle : 30 deg.
![Page 29: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/29.jpg)
Experiment:Conditions Some characters are difficult to distinguish
under affine distortions Characters in a cell were treated as the same class
0 O o6 9C cI lS su n
W wX xN Z zp dq b7 L V v
![Page 30: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/30.jpg)
Settings High recognition rates High speed
Angle (deg.) 0 30 45 0 30 45Time (ms) 7990 7990 7020 1300 1260 1140
Recog. Rate (%) 94.9 90.7 86.4 86.9 81.8 76.3Reject. Rate (%) 0.4 3.0 6.4 6.4 9.3 16.5Error Rate (%) 4.7 6.4 7.2 6.8 8.9 7.2
Experiment:Recognition Result Achieved high recognition rates and high
speed by changing a control parameter180-210
characters/sec
![Page 31: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/31.jpg)
Contents1. Background2. Overview of the Proposed Method3. Contour Version of Geometric Hashing4. Proposed Method
1. Real-Time Processing2. Recognition of Separated Characters3. Pose Estimation
5. Experiment6. Conclusion
![Page 32: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/32.jpg)
IMPWeb camera
Document
Recognizes characters immediately!
Capture
Real-Time Camera-Based Character Recognition SystemRecognizes ~200 characters/sec
![Page 33: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/33.jpg)
Future Work Recognition of Chinese characters Improvement of segmentation for
Broken connected components Colored characters
![Page 34: Real-Time Camera-Based Character Recognition Free from Layout Constraints](https://reader034.vdocuments.site/reader034/viewer/2022051317/568166ec550346895ddb3698/html5/thumbnails/34.jpg)
Real-Time Camera-Based Recognition of Characters and Pictograms
M. Iwamura, T. Tsuji, A. Horimatsu, and K. Kise