—temporal operations, color and binary imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent...
TRANSCRIPT
![Page 1: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/1.jpg)
Intelligent Control Systems
Image Processing (3) —Temporal Operations, Color and Binary Images —
Shingo KagamiGraduate School of Information Sciences,
Tohoku Universityswk(at)ic.is.tohoku.ac.jp
http://www.ic.is.tohoku.ac.jp/ja/swk/
![Page 2: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/2.jpg)
2Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
Image Processing Classification
input output
image
image sequence
image image to image processing(2-D data) Fourier trans., label image
1-D data projection, histogram
scalar values position, recognition
image motion image processingsequenceimage1-D datascalar
example
![Page 3: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/3.jpg)
3Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
Outline
• Temporal Image Processing
• Color Image Processing
• Binary Image Processing
![Page 4: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/4.jpg)
4Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
Simple Example: Frame Difference
input
output
t
– + – + – + – + – + – +
![Page 5: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/5.jpg)
5Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
Outline
• Temporal Image Processing
• Color Image Processing
• Binary Image Processing
![Page 6: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/6.jpg)
6Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
8-bit Grayscale Image (CV_8U)
F0,0 F1,0 F2,0 FM-1,0
F0,1 F1,1 F2,1 FM-1,1
F0,N-1F1,N-1F2,N-1 FM-1,N-1
x axis
y axis
single pixel = 1 byte
![Page 7: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/7.jpg)
7Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
24-bit Color Image (CV_8UC3)
B0,0 B1,0
B0,1 B1,1
B0,N-1 B1,N-1
G0,0
G0,1
G0,N-1
R0,0
R0,1
R0,N-1
G1,0
G1,1
G1,N-1
R1,0
R1,1
R1,N-1
x axis
y axis
single pixel = consecutive 3 bytes
![Page 8: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/8.jpg)
8Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
Accessing a Color Pixel Value
cv::Mat input = cv::imread("lena.jpg", cv::IMREAD_COLOR);…
for (j = 0; j < height; j++) {for (i = 0; i < width; i++) {
cv::Vec3b pixel = input.at<cv::Vec3b>(j, i);uchar blue = pixel[0];uchar green = pixel[1];uchar red = pixel[2];…
}}…
Class of consecutive 3 bytes
operator[] is overloaded so that each component is accessed
![Page 9: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/9.jpg)
9Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
RGB Color Space
Why R, G, and B?• Our eyes have three types of wavelength-sensitive
cells (cone cells)• So, the color space we perceive is three-dimensional
http://commons.wikimedia.org/wiki/File:Cone-response.png
![Page 10: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/10.jpg)
10Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
Other Color Spaces
XYZ, L*a*b, L*u*vdefined by CIE (Commission Internationale de l‘Eclairage)
YIQ, YUV, YCbCrused in video standards (NTSC, PAL, …)
HSV (HSI, HSL)based on Munsell color system
cf. CMY, CMYK (for printing; subtractive color mixture)
cv::cvtColor(input, output, CV_BGR2HSV);
![Page 11: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/11.jpg)
11Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
HSV Color Space
red
yellowgreen
cyan
blue magenta
SaturationHue
Value
value = 1
value = 2/3
value = 1/3
![Page 12: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/12.jpg)
12Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
Notes for cv::Mat data
cv::Mat image1(480, 640, CV_8U);cv::Mat image2, image3;
cv::Laplacian(image1, image2, CV_8U);image2 = image1;image1.copyTo(image3);cv::Laplacian(image1, image3, CV_8U);
Some parts of the sample codes may seem strange if you are not familiar with OpenCV. It is important to understand:
• In C++, operators can be overloaded and thus even operator= can be overloaded
• OpenCV defines operator= of cv::Mat so that the image data are not copied but shared
• When an image data region becomes shared by no cv::Mat, OpenCV automatically removes the region
![Page 13: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/13.jpg)
13Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
image1 image2 image3
image1 image2 image3
image1.data
cv::Mat image1(480, 640, CV_8U); cv::Mat image2, image3;
cv::Laplacian(image1, image2, CV_8U);
automaticallyallocated
![Page 14: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/14.jpg)
14Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
image1 image2 image3
image1 image2 image3
image2 = image1;
image1.copyTo(image3);
automaticallyremoved
not copiedbut shared
automaticallyallocatedcopied
![Page 15: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/15.jpg)
15Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
image1 image2 image3
cv::Laplacian(image1, image3, CV_8U);
automaticallyallocated
automaticallyremoved
![Page 16: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/16.jpg)
16Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
Outline
• Temporal Image Processing
• Color Image Processing
• Binary Image Processing
![Page 17: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/17.jpg)
17Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
Binary Image Processing
Processing of binary images is highly developed in a distinctive way other than grayscale/color image processing, because
• It found important specific applications (e.g. document processing)
• geometrically rigorous discussion is possible
We have to introduce Digital Geometry:• Because an image is discretized into pixels,
conventional concepts of geometry for continuous shapes (e.g. connectivity, distance) may not be used just as they are.
![Page 18: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/18.jpg)
18Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
Connectivityneighbor: set of pixels that are near the pixel of interest. Many definitions are possible, e.g.
4-neighbor 8-neighbor
n-adjacent: If two pixels are in n-neighbor of each other, we say they are n-adjacent
n-neighbor connected: If there exists a sequence of pixels p0 , p1 , p2 , , pn-1 , pn where all the pi have the same pixel value and pi and pi+1 is n-adjacent, we say the pixels in this sequence are n-neighbor connected
![Page 19: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/19.jpg)
19Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
Morphological Operations
dilation: “becomes 1 if any of neighbor pixels are 1”
erosion: “becomes 0 if any of neighbor pixels are 0”
opening: erosion, then dilation
closing: dilation, then erosion
Gi,j = Fi,j | Fi-1,j | Fi+1,j | Fi,j-1 | Fi,j+1
Gi,j = Fi,j & Fi-1,j & Fi+1,j & Fi,j-1 & Fi,j+1
(4-neighbor)
(4-neighbor)
Note: These are nonlinear local operatoins
![Page 20: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/20.jpg)
20Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
Connected Component Labeling• or simply “Labeling” for short• Segments an image into connected components and give each
component a unique number (label)• An image whose pixel values are labels is called a label image
Example Algorithm for 4-neighbor connectivity• Scan the image from top-left to bottom-right to process “1” pixels
• If a label has been assigned to either of the upper or left pixels, assign the same label to the current pixel
• If the upper and the left pixels have different labels, one of them (say, the smaller one) is assigned to the current pixel, and record that both labels refer to the same component to a connectivity table.
• If none of the upper and the left pixels have labels, a new label is assigned to the current pixel.
• Scan the image again to update the labels referring to the table
![Page 21: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/21.jpg)
21Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
1 1 12 2
2 2 2 2
2 2 2 22 2
3 3 3
3 34 4 4 33
3 4
![Page 22: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/22.jpg)
22Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
Shape Features
Moment features
0th order moment: i.e. area(for binary images)
1st order moment in x direction
1st order moment in y direction
Center of Gravity is computed from 0th and 1st order moments(gx , gy ) = (m1,0 /m0,0 , m0,1 /m0,0 )
Higher order moments convey more complicated shape information
![Page 23: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/23.jpg)
23Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
Notes for std::vector
• std::vector is part of C++ Standard Template Library (STL)• can be used as a better array
• Many other convenient containers are also available:• list, map, queue, stack, deque, …
std::vector<int> array; array.push_back(100);array.push_back(120);array.push_back(130);
array[2] = array[2] + 50;
printf("array[2] = %d¥n", array[2]);printf("size of array: %d¥n", array.size());
extends automatically
can be used almost like an array
prints 180prints 3
![Page 24: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/24.jpg)
24Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
What’s the common way in OpenCV?
• So far, connected component labeling seems not provided as standard functions of OpenCV
• Contour retrieval using a border following algorithm can be used for similar purpose. See:•cv::findContours()• You can analyze the shape of a retrieved contour by e.g.
•cv::arcLength()•cv::boundingRect()•cv::contourArea()•…
![Page 25: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/25.jpg)
25Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
Summary
• Simple example of video processing• frame difference
• Color Image Processing• cv::Mat for color images• color spaces
• RGB, HSV, …
• Binary Image Processing• connectivity• morphological operations
• dilation, erosion, opening, closing• connected component labeling• moment features
![Page 26: —Temporal Operations, Color and Binary Imagesswk/lecture/ic2014/kagami_ic20140624.pdf · gent Control Systems 2014.06.24 25 Summary • Simple example of video processing • frame](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0af1587e708231d42e19d5/html5/thumbnails/26.jpg)
26Shingo Kagami (Tohoku Univ.) Intelligent Control Systems 2014.06.24
References
• R. Szeliski: Computer Vision: Algorithms and Applications, Springer, 2010. • A. Hornberg eds.: Handbook of Machine Vision, Wiley-VCH, 2006. • G. Bradski and A. Kaebler: Learning OpenCV, O‘Reilly, 2008. • OpenCV Documentation: http://docs.opencv.org/index.html
(in Japanese)• ディジタル画像処理編集委員会, ディジタル画像処理, CG-ARTS協会, 2004. • 田村: コンピュータ画像処理, オーム社, 2002.
Sample codes are in sample20140624.zip available athttp://www.ic.is.tohoku.ac.jp/~swk/lecture/