IMAGE MOSAICING AND REAL-TIME IMAGING WITH MEMS BASED HANDHELD CONFOCAL MICROSCOPE

Wibool Piyawattanametha

Faculty of Engineering, King Mongkuts Institute of Technology Ladkrabang, Thailand

Advanced Imaging Research Center, Faculty of Medicine, Chulalongkorn University Thailand

ABSTRACT In this paper, we demonstrated a handheld confocal fluorescence microscope using dual-axis confocal architecture with a microelectromechanical systems (MEMS) scanner. The laser sources for the microscope are both in visible (660 nm) and near-infrared (778 nm) wavelengths. The microscope will be used for in vivo cervical cancer screening in real-time with human patients. The maximum imaging depth is over 300 m into the tissue for epithelial cancer screening. The maximum field of view (FOV) is 550 m 500 m with 14 frames/second.

1. INTRODUCTION

Over the past several decades, medical imaging technologies such as PET, MRI, and CT have played a critical role in the early detection and treatment of cancer. These imaging modalities have the ability to detect cancer at the anatomic scale (e.g. tumors) but lack the resolution needed to see cancer at the cellular scale (e.g. colon or skin cancers). As a result, the diagnosis of cancer at the cellular scale is usually performed through tissue biopsy and subsequent pathology. Recently, a new class of miniature confocal microscopes has emerged that can detect cancerous and pre-cancerous tissues at the cellular scale [1-2]. The endoscope has the potential to revolutionize health care by replacing tissue biopsies with in vivo pathology for fast, cheap, and non or minimally-invasive screening. Cervical cancer is the first and second most common cause of cancer death in women in Asia and developing countries, respectively [3]. At present, popular diagnostic techniques are Papanicolaou (Pap) and Human Papilloma Virus (HPV) Deoxyribonucleic acid (DNA) tests [4]. However, the aforementioned techniques require sample preparations and medical expertise to correctly diagnose results [5]. The purpose of our work is to develop a miniature cervical cancer screening tool based on confocal microscopy technique for in vivo real time diagnosis. Confocal microscopy technique offers subcellular to cellular resolution with optical section property enabling three-dimensional (3D) imaging [6].

2. CONFOCAL SYSTEM SETUP

Fig. 1a shows a photograph of handheld dual-axis confocal microscope. It is a rigid type microscope with 10-mm diameter at its imaging tip. Furthermore, the length of the microscope is designed to be 200 mm compatible with the actual length of human cervix (60-140 mm) [5]. Fig. 1b shows optical setup of the microscope. Excitation wavelength is 778 nm. The laser light is coupled into an input single-mode optical fiber (numerical aperture 0.12) achromat collimator (2.8 mm in diameter) as shown in Fig. 2. The light exiting collimator has its full width at half maximum (FWHM) diameter of 950 m. The collimated beam is focused by a parabolic mirror (focal length of 4.7 mm) before reflecting of from one side of the MEMS mirror [3]. MEMS scanner die size is 3.23.1 mm2 with the mirror shape of a barbell to accommodate input illumination and output collection.

Figure 1: (a) Photograph of handheld dual-axis confocal microscope (b) Schematic drawing of the microscope showing internal components. Scale bar is 5 cm.

The focused beam exits the microscope housing via a solid immersion lens (SIL) and illuminating on either tissues soaking in indocyanine green (ICG) dye or

978-1-4799-4963-2/14/$31.00 2014 IEEE

2014 International Computer Science and Engineering Conference (ICSEC)

978-1-4799-4963-2/14/$31.00 2014 IEEE 140

specimens. A MEMS scanner will perform a raster scanning with point by point illumination over the entire FOV. Fluorescent signal from the sample is defocused into the output path of the microscope via the collection side of the microscope consisting of a MEMS mirror, a parabolic mirror, and an output collimator. The maximum imaging frame rate is 14 frames/second. Then, the fluorescence signal is travelling through an emission long pass filter (790 nm) to block the excitation wavelength before entering a photomultiplier tube (PMT). A low noise amplifier is used to convert light into an electrical signal, which then can be acquired by a data acquisition (DAQ) system via LabView. The acquired image is displayed on personal computer in real-time. Imaging depth, FOV, contrast, gain adjustment can be executed through the custom software in LabView. Image mosaicing has been implanted to help enlarge FOV into a millimeter-scale for ease of tissue interpretation during real-time diagnosis.

Figure 2: Schematic drawing of a 2.8 mm diameter achromat collimator assembly. The FWHM beam size is approximately 1.8 mm.

3. IMAGE MOSAICING Image mosaicing is a technique to enlarge the FOV for image display based on multi-picture alignment method and commonly used in the field of vision-based robot navigation systems and virtual reality [7]. Moreover, this method can be applied to microscopes or endoscopes in displaying the whole interested areas, which cannot be shown within only one video frame shot. Individual small frame shots used in image mosaicing technique requires the overlapping display areas for the utility of image alignment to be as smooth as possible. Image mosaicing application in biomedical imaging solves the problem in FOV limitation. Microscope and microendoscope integrated with image mosaicing technique are able to perform the high-resolution panorama image in subcellular level for ex vivo and in vivo abnormal tissue detection and diagnostics in real time [8]. Therefore, rapid image processing for feature frame alignment used in such equipment is necessary to instantly display the interested field. Image mosaicing method is generally

divided into two main processes consisting of image registration and image blending process. Image registration is a method to identify the coordinate relationship between two aligned frames, where the overlapping areas in each picture are similar to each other as much as possible, and place the feature frames in proper positions [8]. Image blending process provides the method for adjusting light intensity of the images at the edge of frames alignment to improve the image display concord. In this work, we have presented the image blending process based on computer vision technique. The limited FOV of tissue section sequence frames taken from optical microscope were enlarged and displayed by image mosaicing technique. Computer vision technique allows us to estimate the moving length between the present and the previous feature frame for image alignment process. This motion tracking technique provides the rapid moving length calculation method to determine the frame positions in the area of interest [9].

4. RESULTS

4.1 Microscope resolution We use a multi-wavelength generator (Qioptic, Inc.) to couple more than one wavelength into the handheld microscope. Each wavelength has the maximum output power of around 40 mW. However, after coupling the laser power into the handheld imaging system, typical power output measured at the tip of the microscope is in the range of 2-4 mW. The first imaging test is to measure the performance of the handheld microscope with a laser beam profiler with a 100 objective lens as shown in Fig. 3. An optical fiber with 50:50 coupling ratio is used to split the laser power into half and feeding both ends into both input and output collimators. The intersection of both input and out beams defines the microscope resolution. The center wavelength used in this experiment is 660 nm. The axial response is measured at the output of collection collimator with a power meter while translating a perfect mirror in axial direction through the focal imaging plane. Resolution results are shown in Fig 4. The full-width-at-half-maximum (FWHM) of transverse (X and Y directions) resolutions are 4.1 and 3.6 m, respectively (Fig. 3a) while FWHM of axial (Z direction or depth) resolution is 7.2 m (Fig. 3b).

4.2 Resolution and biological samples The reflection image acquired from the handheld confocal microscope shows group 7, element 6 of the United States Air Force (USAF) standard resolution target (Fig. 5). The porcine colon images are shown in Fig. 6. The tissue sections are rinsed for a few times with phosphate buffer saline (PBS) solution pH 7.2, and subsequently immersed in 5 mg/ml of ICG in PBS pH

2014 International Computer Science and Engineering Conference (ICSEC)

141

7.4 for 1 hour. After that, the tissue sections are rinsed for several times to remove excess dye, and are captured by both the standard microscope (Fig. 6a) and the handheld dual-axis confocal microscope (Fig. 6b) for comparison. The maximum FOV is 550 m 500 m. Image gain level can be control through the software to balance the image contrast over the entire FOV.

4.3 Image stitching

To demonstrate concept of our image stitching

algorithms, we used high-resolution images obtained from a microscope with cellular definitions, we captured a high-resolution image (1228 pixel 802 pixel) from a high-resolution camera (Olympus, Inc., Model: DP72) attached to a microscope as shown in Fig. 7(a). A sequence of image (120 frames) will be randomly

cropped and saved along x- and y-axes from this high-resolution image into many synthetic small images to be used with our optical flow mosaicing technique. Each cropped image will be exported as a video file with frame rate of 10 Hz. After we have processed the video data, the result is shown in Fig. 7(b). The mosaicing alignment accuracy is calculated to be at over 87 percent with 20% to 40% overlapping features between each frame. The maximum input video frame rate to be processed with our algorithms is estimated to be 30 Hz.

(a)

Figure 3: Microscope x-y resolution measurement setup, scale bar is 4 cm.

Figure 4: Measured resolutions of handheld dual-axis confocal microscope. (a) FWHM transverse (X-Y direction) resolutions are 4.1 and 3.6 m. (b) FWHM axial (Z direction) resolution is 7.2

Figure. 5: The reflection image shows group 7 of USAF standard resolution target, scale bar is 20 m.

Figure 6: Porcine colon images. (a) a 40 optical image from the Olympus microscope model BX63. (b) an image from the handheld dual-axis confocal microscope, scale bar are 20 m.

2014 International Computer Science and Engineering Conference (ICSEC)

142

(b) Figure 7: (a) A high-resolution image of hypo-plastic tissue sample acquired with a high-resolution camera attached to a microscope. (b) An implemented result of mosaicing algorithms.

1. SUMMARY

We have demonstrated a MEMS scanner based cervical confocal imaging probe and successfully image ex vivo tissues of porcine colon with cellular resolution. Performance of the handheld microscope has been characterized. Real-time cancer diagnosis is capable with high-speed imaging (upto 14 Hz). Image real-time mosaicing program will be implemented in the experiments to enable FOV enlarge for ease of use. We anticipate broad sets of in vivo and ex vivo experiments with this imaging tool in the near future.

5. ACKNOWLEDGEMENT

The work is support in part by the Fraunhofer-Bessel Research Award from the Alexander von Humboldt Foundation, Germany.

6. REFERENCES [1] W. Piyawattanametha and T. D. Wang, MEMS-

Based Dual Axes Confocal Microendoscopy (Invited Paper), The IEEE Journal of Selected Topics in Quantum Electronics (JSTQE), July-August 2010, Vol. 16, Issue 4, pp.804-814.

[2] N. Khemthongcharoen, S. Rattanavarin, R. Jolivot, and W. Piyawattanametha, Advanced in imaging probes and optical microendoscopic imaging techniques for early in vivo cancer assessment (invited paper), Journal of Advanced Drug Delivery Reviews, October 9, 2013, 10.1016/j.addr.2013.09.012.

[3] World Health Organization, Cervical cancer screening in developing countries: report of a WHO consultation, WHO Library Cataloguing-in-Publication Data, 2002.

[4] J. S. Mandelblatt, W. F. Lawrence, S. M. Womack, D. Jacobson, B. Yi, Y. T. Hwang, K. Gold, J. Barter, and K. Shah, "Benefits and costs of using HPV testing to screen for cervical cancer", Journal

of the American Medical Association, Vol. 287, pp.2372-2381, 2002.

[5] W. Piyawattanametha, H. Ra, Z. Qiu, S. Friedland, J. T. C. Liu, K. Loewke, G. S. Kino, O. Solgaard, T. D. Wang, M. J. Mandella, and C. H. Contag, In Vivo Near-infrared Dual-Axis Confocal Microendoscopy in the Human Lower Gastrointestinal Tract, Journal of Biomedical Optics 17(2), February 2012, 021102:1-4.

[6] J. S. Tan, E. S. Lukacz, S. A. Menefee, K. M. Luber, M. E. Albo, and C. W. Nager, "Determinants of vaginal length", American journal of obstetrics and gynecology, Vol. 195, pp.1846-50, 2006.

[7] Cobzas, D., Jagersand, M. and Zhang, H., A Panoramic Model for Remote Robot Environment Mapping and Predictive Display, International Journal of Robotics and Automation, 20(1):25-34, 2005.

[8] W. Piyawattanametha, M. Mandella, H. Ra, J. Liu, E. Garai, G. Kino, O. Solgaard, and C. Contag, MEMS base dual-axes confocal clinical endoscope for real time in vivo imaging, Proceedings of Optical MEMS and Nanophotonics, pp. 42-43, 2008.

[9] R. Szeliski, Video mosaics for virtual environments, IEEE Computer Graphics and Applications, vol. 16, no. 2, pp. 22-30, 1996.

2014 International Computer Science and Engineering Conference (ICSEC)

143

/ColorImageDict > /JPEG2000ColorACSImageDict > /JPEG2000ColorImageDict > /AntiAliasGrayImages false /CropGrayImages true /GrayImageMinResolution 200 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages false /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict > /GrayImageDict > /JPEG2000GrayACSImageDict > /JPEG2000GrayImageDict > /AntiAliasMonoImages false /CropMonoImages true /MonoImageMinResolution 400 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 600 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict > /AllowPSXObjects false /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile (None) /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped /False

/CreateJDFFile false /Description >>> setdistillerparams> setpagedevice