Practical Descattering of Transmissive InspectionUsing Slanted Linear Image Sensors

Takahiro Kushida1 Kenichiro Tanaka1,2 Takuya Funatomi1 Komei Tahara3

Yukihiro Kagawa3 Yasuhiro Mukaigawa1

1Nara Institute of Science and Technology8916–5 Takayama-cho, Ikoma, Nara 630–0192, JAPAN

{kushida.takahiro.kh3, ktanaka, funatomi, mukaigawa}@is.naist.jp2Ritsumeikan University

1–1–1 Noji-higashi, Kusatsu, Shiga 525–8577, JAPANken-t@fc.ritsumei.ac.jp

3Vienex Corporation262 Yoshioka-cho, Kanonji, Kagawa 768–0021, JAPAN{komei tahara, yukihiro kagawa}@vienex.co.jp

Abstract

This paper presents an industry-ready descatteringmethod that is easily applied to a food production line.The system consists of multiple sets comprising a linearimage sensor and linear light source, which are slantedat different angles. The images captured by these sen-sors, which are partially clear along the perpendiculardirection to the sensor, are computationally integratedinto a single clear image over the frequency domain.We assess the effectiveness of the proposed method bysimulation and by our prototype system, which demon-strates the feasibility of the proposed method on an ac-tual production line.

1 Introduction

Foreign-object inspection for food production linesis important for preserving the safety of products.Inspection is required to be both fast and non-destructive, to increase the productivity of a food fac-tory. A typical approach is to put a light source anda detector on the opposite sides of an object to seethrough the object; however, the captured images areblurred because of the scattering of the light. In thispaper, we aim to obtain a clear image of the transmis-sive observation that is suitable for an existing produc-tion line.

One of the traditional approaches for image restora-tion is a deconvolution approach, whereby some priorson the scene and the kernel are assumed [1, 2, 3, 4].Unfortunately, the quality of the recovered image isgenerally low because of the loss of high-frequency com-ponents during the observation process. Another ap-proach is to use a projector–camera system [5, 6, 7, 8]to probe the light transport. Although the image qual-ity is very high with this approach, it requires multiple

observations of the same scene; hence, it is not straight-forward to apply it to a production line. We take ahybrid approach that is suitable for a production lineand conserves high-frequency detail.

In our approach, we simply use multiple sets com-prising a linear image sensor and linear light source,which are already used on the production lines. Thekey point is that the sets of sensors and light sourcesare slanted relative to the direction of the conveyor.For each sensor, it is possible to obtain an image of themoving subject by stacking the observations. This im-age is blurred along the sensor direction but clear alongthe perpendicular direction. Using different anisotrop-ically clear images, we can computationally recover atotally clear image.

The chief contribution of this paper is that itpresents the first industry-ready descattering method.Our method can easily be applied to an productionline without any new components, simply by placingtwo traditional linear sensors, each rotated by ±45 de-grees.

2 Line-scan imaging on a conveyor system

Line-scan imaging is common on production linesbecause of its efficient mechanism. The imaging deviceconsists of a linear image sensor and linear light sourceand outputs a single row of pixels per exposure. Asthe objects on the conveyor pass in front of the sen-sor, a two-dimensional(2D) image can be obtained bystacking the captured rows, one by one, as shown inFig. 1.

A foreign object inspection usually observes thelight transmitted through the scanned subject. Thelight that straightly passes straight through the sub-ject forms a clear shape of the foreign object; however,there is also scattered light, which blurs the image and

17th International Conference on Machine Vision Applications (MVA)Fully Online, July 25–27, 2021.

© 2021 MVA Organization

P2-9

Linearimage sensorLinearimage sensor

Linearlight sourceLinearlight source

Scanning axis Sensing axis

Scan

ning

axi

s

Sensing axis

Figure 1. Line scan imaging . A set, compris-ing a linear light source and linear image sensor,is installed on each side of a conveyor. Subjectsmove on the conveyor, and the linear image sen-sor measures the light transmitted through thesubjects. A 2D image is obtained by stacking thelinear images.

Sensing axis Scanning axisLinearlight sourceLinearlight source

Linearimage sensorLinearimage sensor

Figure 2. Anisotropic descattering effect on line-scan imaging. The scattered light is capturedalong the sensing axis but not along with thescanning axis.

makes it difficult to recognize the presence of the for-eign object.

Because the scattering is diffusive, the blur isisotropic in common 2D area imaging, whereas itappears differently in line-scan imaging because ofits anisotropic illumination and sensing, as shownin Fig. 2. The blur is similar to that of area imagingalong the sensing axis but less scattered light existsalong the scanning axis.

3 Descattering with multiple slanted linearimage sensors

In a single observation with the linear light sourceand linear image sensor, the scattering blur still existsalong the sensing axis. We aim to develop a method ofisotropic descattering, from multiple partially descat-tered observations, that is suitable for production lines.To obtain the multiple images, the observations ofeach object must be conducted at different orientations.However, physically rotating the subject may degradeproductivity. Our key idea is a co-design of hardwareand algorithm. We use multiple sets comprising a lin-ear light source and linear image sensor slanted at dif-ferent angles (Fig. 3) and to computationally integratethe multiple observations into one image (Fig. 4).

K sl

ante

d lin

ear i

mag

e se

nsor

s AlignmentSkew correction

Figure 3. Multiple slanted linear image sensorsfor line-scan imaging. The observed images arecorrected and aligned.

3.1 Multiple slanted linear image sensors

We use multiple sensors to obtain differently blurredimages without rotating the subjects, to maintain pro-ductivity. To enable subjects to keep moving along theproduction line, we install the linear light sources andlinear image sensors slanted from the scanning axis,as shown in Fig. 3. Suppose that we wish to sampleat K angles evenly. In line scanning, there is a re-striction that the sensing axis must not be identical tothe scanning axis, or a 2D image cannot be obtained.Therefore, we set the angle between the sensing andscanning axes to be

θk =2k + 1

2Kπ, (1)

where k = 0, · · · ,K − 1.Because the sensing and scanning axes are not or-

thogonal, the 2D images obtained may be skewed. Wecorrect the skew in the 2D images according to themoving speed of the subject and the sensor bandwidth.Alignment of the observations is necessary for the in-tegration. This is possible by adjusting the synchro-nization of the sensors according to the moving speedof the subject. However, in this study, we placed somemarkers at known positions to achieve the alignmentwithout the need for engineering work.

3.2 Image integration based on maximumFourier amplitude spectrum

We propose a method to recover a clear image fromthe K observations with anisotropic blur, as shown inFig. 4. We formulate the blurring in the Fourier domainto cope with this anisotropicity.

We first consider the isotropic case. Suppose thatthe isotropic blur is caused by diffusive scattering and isspatially invariant. The blurring process is formulatedas a convolution of a clear image f(x, y) and Point

K im

ages

Fourier Transform

Inverse Fourier Transform

SelectMaximumAmplitude

Figure 4. Integration in the Fourier domain. Ap-plying Fourier transform to the observed images,K images are obtained. For each frequency in theFourier-transformed image, the component withthe maximum amplitude among the K images isselected and integrated into a single image. Theintegrated clear image is recovered by applyinginverse Fourier transform.

Spread Function (PSF) h(x, y) [1].

g(x, y) = f(x, y) ∗ h(x, y), (2)

where g(x, y) is the blurred image and ∗ is a convo-lution operator. The PSF depends on the propertiesof the scatterer, such as its material and thickness. Italso depends on the imaging process. When the scat-tering is diffusive, h(x, y) is isotropic in standard areaimaging.

However, it becomes anisotropic when a linear imagesensor and light source are used. The PSF is formu-lated as a function of the angle θ of the sensing direc-tion. The k-th observation, with angle θk, is formulatedas,

gk(x, y) = f(x, y) ∗ hk(x, y). (3)

In the Fourier domain, the blurring process is in-terpreted as a decay of the high-frequency compo-nents. In isotropic blurring, the high-frequency com-ponents of the F [h(x, y)] are close to zero, where Fdenotes the Fourier transform operator. In addition,the high-frequency components of F [hk(x, y)] are smallalong the sensing direction. Therefore, each of theanisotropic images maintains the high-frequency com-ponents along a different direction. If they can be com-plementary to each other, the blur can be mitigatedby aggregating the remaining high-frequency compo-nents. As the aggregation operator, we adopt max inthe Fourier domain.

Applying the Fourier transform to Eq. (3), the equa-tion becomes

F [gk(x, y)] = F [f(x, y) ∗ hk(x, y)]

Gk(u, v) = F (u, v)Hk(u, v), (4)

Bolt Mosquito Orange Tomato(a) Subjects for simulation

Obs

erva

tions

Res

ult

K 1 2 4 8

(b) Simulated results for ‘Mosquito’

AreaImaging

1 2 4 8 12

K

0.85

0.90

0.95

1.00

ZNC

C BoltMosquitoOrangeTomato

(c) ZNCC vs K

Figure 5. Performance analysis by simulation.

where G, F , and H are the Fourier transforms of g,f and h, respectively. For each pixel (u0, v0) of the

integrated image in frequency domain G, we assign thecomponent with the maximum amplitude among theK images.

G(u0, v0) = Gk′(u0, v0)

k′ = arg maxk

|Gk(u0, v0)|

∀(u0, v0) ∈ (u, v). (5)

Finally, the integrated clear image g is obtained byapplying the inverse Fourier transform:

g(x, y) = F−1[G(u, v)]. (6)

4 Simulation

The performance of the descattering depends on thenumber of observations. We clarified the relationshipbetween them by simulation. We simulated line-scanimaging by convolving a blur kernel onto a clear imageof size 512 × 512, using Eq. (2). To simulate the blur

Scanning axisScanning axisLinear light sources(back side)Linear light sources(back side)

Linear image sensorsLinear image sensors

Acrylic stageAcrylic stage

SubjectSubject

Figure 6. Prototype system.

in the k-th observation, we applied a 1 × 63 rectan-gular window mask, slanted at an angle of θk, for anisotropic 2D Gaussian blur kernel with σ = 21 in thespatial domain. We also simulated the blurred imagewith the kernel without masking, which corresponds toobservation using area imaging, for reference. We eval-uated the descattering performance by calculating thezero-normalized cross-correlation (ZNCC) between theclear and recovered images.

The results are shown in Fig. 5. We used the four im-ages shown in Fig. 5(a). Figure 5(b) shows the observa-tions from simulated line-scan imaging and the recov-ered image for various numbers of observations: K =1, 2, 4, and 8. Figure 5(c) shows the ZNCC for areaimaging and for line-scan imaging with K = 1, · · · , 12.

The results show that the ZNCC for line-scan imag-ing, even with K = 1, is significantly better than thatfor area imaging. They also show that the ZNCC de-pends on the subject. The proposed method is more ef-fective for complex-shaped and high-contrast subjects,such as ‘Mosquito’ and ‘Orange.’ Generally, increasingthe number of observations K improves the quality ofthe recovered image. The benefit becomes smaller asK increases, whereas the cost increases linearly withK. Therefore, K = 2, 3 would be ideal for practicaluse.

5 Prototype system

We developed a prototype system with two pairscomprising a linear image sensor and near infrared lightsource, as shown in Fig. 6. The sensors were installedin the system rotated by ±45 degrees.

Figure 7 shows the observations and preprocessingfor a white acrylic plate with a black plastic piece.The left column shows that the captured images areskewed and misaligned. The preprocessing describedin Sec. 3.1 was executed to correct the skew and alignthe images as shown in the middle column. The rightcolumn shows the result of the proposed method.

We measured foreign objects in real foods as shownin Fig. 8. Three black plastic pieces were used as for-eign objects, and a cracker and a slice of bread were

Captured Skew corrected,aligned and clipped Integrated Result

+45°

- 45°

Figure 7. Images from two linear image sensorsand the result of the proposed method. Left: Thesubject is skewed and misaligned in the capturedimages. Middle: Skew correction and alignmenthave been applied. Right: Result of proposedmethod.

+45° - 45° ResultC

rack

erB

readForeign

objects

Figure 8. Results with real food, showing thatour proposed method worked successfully in theprototype system and high-frequency detail wasobtained.

placed on these objects. The results show that the for-eign objects were significantly blurred along the sens-ing axis in each of the captured images. However, theproposed method worked successfully to clarify the sil-houettes of the foreign objects in the real foods.

6 Conclusion

In this paper, we propose a co-design of hardwareand algorithm for foreign object inspection. The designcan easily be installed in a production line with line-scan imaging. It enables a clear image to be obtainedby computationally integrating multiple line-scan im-ages slanted at different angles. We assessed the effec-tiveness of the proposed method by simulation. Wealso constructed a prototype system, which demon-strated the feasibility of the proposed method on anactual production line.

Acknowledgment

This work is partly supported by JST CREST JP-MJCR1764.

References

[1] S. G. Narasimhan and S. K. Nayar, “Shedding lighton the weather,” in 2003 IEEE Computer Society Con-ference on Computer Vision and Pattern Recognition,2003. Proceedings., vol. 1, pp. I–I, 2003.

[2] S. Metari and F. Deschenes, “A New Convolution Ker-nel for Atmospheric Point Spread Function Applied toComputer Vision,” in Proc. International Conferenceon Computer Vision (ICCV), (Rio de Janeiro, Brazil),pp. 1–8, IEEE, 2007.

[3] C. Fuchs, M. Heinz, M. Levoy, H.-P. Seidel, and H. P. A.Lensch, “Combining Confocal Imaging and Descatter-ing,” Computer Graphics Forum, vol. 27, pp. 1245–1253, June 2008.

[4] R. Wang and G. Wang, “Single image recovery in scat-tering medium by propagating deconvolution,” OpticsExpress, vol. 22, p. 8114, Apr. 2014.

[5] S. Seitz, Y. Matsushita, and K. Kutulakos, “A theory ofinverse light transport,” in Proc. International Confer-ence on Computer Vision (ICCV), pp. 1440–1447 Vol.2, 2005.

[6] S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar,“Fast separation of direct and global components of ascene using high frequency illumination,” ACM Tran.on Graphics (ToG), vol. 25, pp. 935–944, July 2006.

[7] K. Tanaka, Y. Mukaigawa, Y. Matsushita, and Y. Yagi,“Descattering of transmissive observation using Paral-lel High-Frequency Illumination,” in Proc. InternationalConference on Computational Photography (ICCP), pp. 1–8, Apr. 2013.

[8] M. O’Toole, S. Achar, S. G. Narasimhan, and K. N. Ku-tulakos, “Homogeneous codes for energy-efficient illumi-nation and imaging,” ACM Tran. on Graphics (ToG),vol. 34, pp. 1–13, July 2015.