Analysis of Fused Ophthalmologic Image Data

J. Jan, L. Kube�ka, R.Kolá� and R.Chrastek1

Department of Biomedical Engineering Brno University of Technology

Kolejni 4, 612 00 Brno, Czech Republic phone: +420 541 149 540, e-mail: jan@feec.vutbr.cz

1Erlangen University (now Siemens Erlangen), Germany

Keywords: Image equalization, image registration, image segmentation, glaucoma, ophtalmology

Abstract - The contribution summarises the results of a long-term project concerning processing and analysis of multimodal retinal image data, run in cooperation between Brno University of Technology – Dept. of Biomedical Engineering and Erlangen University – Clinic of Ophthalmology. From the medical application point of view, the main stimulus is the improvement of diagnostics (primarily of glaucoma but other diseases as well) by making the image segmentation and following analysis reproducible and possibly independent on the evaluator. Concerning the methodology, different image processing approaches had to be combined and modified in order to achieve reliable clinically applicable procedures.

1. INTRODUCTION

Retinal images carry important diagnostic information. This well known statement has been confirmed recently by several medical studies and analysis of retinal images thus became a recognised tool in the diagnostic process of some diseases. Namely, a method established particularly for glaucoma diagnosis is the morphological analysis of the optic nerve head by scanning laser tomography. So far, the analysis is mostly based on manual outlining important objects in the retinal images, e.g. the optical nerve terminal or autofluorescence areas, and consequently analysing their shape, dimensions, areas etc. The main problem of such manually driven diagnosis is that it is very demanding as the previous training of the operator (mostly the diagnosing ophthalmologist) concerns and also in terms of time requirements and the consequential fatigue. Last but not least, there are substantial differences between results of outlining objects by different operators; this unevenness should be avoided by making these operations fully or at least partly automatically based on established and proven algorithms.

Developing suitable algorithms enabling the automatic image analysis in this particular area is the main purpose of the long term project to be briefly summarised here. From the image processing point, the involved tasks, on the first look appearing simple, turned out quite challenging when they should be reliable enough to serve a serious clinical, possibly even routine, use.

There are basically two types of imaging systems used in this area: scanning-laser-tomography (SLT), based on the principle of confocal microscope, using a laser as the (visible or infrared) light source and providing basically 3D image data; the other source of image data is the color fundus photography using basically standard optical camera with flash illumination. Each of these modalities carries partly different information and it can be expected

that fusing images from these two sources would provide new information unavailable in each of them separately.

The organisation of the paper is as follows: None of the image sources is perfect, so that the obtained images must be first enhanced, primarily by compensating for very uneven illumination of the image field and perhaps also enhancing the contrast or sharpness. A recently developed approach to this part of processing will be described in Section 2. The second stage of the diagnostic analysis, which turned out very difficult, is registration of images of the same scene provided by the different sources, with different imaging geometry including scale, different contrast or colour scale, and without clear landmarks visible in both modalities. The registration phase including a brief description of the used similarity criteria, flexible transforms and stochastic optimisers will be subject of the Section 2. The Section 3 will be devoted to the analysis phase automatically detecting and outlining the optical nerve terminal (s.c. optical disc) and specific structures nearby, based on the visual features determined via collaboration with the ophtalmologists. Particular attention will be placed on how to utilise the information in the fused bimodal images. The final Section 4 deals with analysing autofluorescence areas indicating the degree of glaucoma development that are visible in the retinal images imaged by the laser scanner under special conditions. Here, the first task was to improve the originally unfavourable signal to noise ratio without loosing on image sharpness, and then to identify and quantify the autofluoresce area count, shape and size.

2. PRE-PROCESSING OF RETINAL DATA

The original image data suffer from uneven illumination which prevents reliable registration as well as segmentation of the images. Several methods have been used to equalize the brightness level without loosing any important details. Finally, a formalized method [3] to determine the correction “inverse bias” field retrospectively – relying only on the image data without explicit identification of the distortion – was designed that provided rather good results. The B-spline based model of the correction field has the form

� � � �1

1

ˆ ,K

i ii

b s�

�

��x � x

where �i are the parameters of the transform defining the contribution of each basis. In order to find the parameters of the model, an entropy based criterion is defined, with respect to which the parameters could be optimized. The idea (Likar 2000 [5] ) is, that the illumination is an

33

additional (though undesirable) information added to the image information; because the illumination bias is to be removed, the information content of the corrected image should be lowered, ie. the Shannon’s entropy H of the image minimized. This leads to the determining the optimum parameter vector � by optimisation,

� � � �� �� �1opt arg min H v b��

�� x x

. Thanks to the found continuous-like formulation of the inherent formulae, the very efficient L-BFGS algorithm for high dimensional optimization can be applied providing reliably a fast solution.

The method has been tested so far on a series of 20 retinal images obtained by means of HRT II confocal scanning laser ophthalmoscope and colour fundus camera Kowa. An example of such correction of a real retinal image is illustrated on Fig. 1; the intensity profiles clearly show an improvement.

Fig. 1.: A retinal image heavily distorted by non-homogenous illumination (upper left), normalized correction field controlled by 3x3 parameters automatically obtained using the proposed

method (upper right). The image after multiplicative correction by the derived correction field (below left), and intensity profiles

along the indicated row (below right)

3. FUSING BIMODAL IMAGES

The main idea behind a substantial improvement in accuracy of retinal object segmentation achieved in the project during last three years is in fusing the information obtained via both used imaging modalitites, the HRT II confocal scanning laser ophthalmoscope and the optical colour fundus camera Kowa. The achieved results proved the idea of each of the modalities carrying partly different information right; in this sense the images are partly complementary. The extra information provided via fusion approach [1] thus supports obtaining more reliable results.

However, the fusion requires highly precise registration of the images from both modalities that differ substantially, as it can be seen from Fig. 2 – the imaging geometry, scale and namely quite different grey scale, differently representing the same objects are complicating the registration process.

Fig.2 Kova optical fundus image (left) and HRT II compound (intensity) 2D image of retina (right)

The registration of two images (labelled A, B) means to find a geometric transform T,

AB xTx �� such that the points belonging to the same structures in both images coincide. Unfortunately, due to different character of the modalities, any well-defined markers (“landmarks”) cannot be reliably found, which excludes direct registration procedures. The only way remains determining the proper parameters �0 of the transform via optimisation,

� �� � ATB� ��

,minarg0 C�

based on a suitable criterion C, which evaluates the quality of the registration. Finding a suitable criterion of similarity in multimodal registration under incompatible contrast scale situation is a problem of itself. Finally, the mutual information (MI) criterion

� � � � � �BAHBHAHI ,),( ����A

� � � �� � � �ba

bam

a

n

bba fpfp

ffpffp ,log,1 1

��� �

�

based on estimates of joint probabilities estimated via the joint histograms proved again to be basically the best solution. However, it has been found that a modified criterion, taking into account also a smoothed versions of both images,

� �� � � �� � � � � �ATBATBATBC ��� GoG,GoGMI.,MI, �

gives better results for this type of images when suitable parameters of the GoG (Gradient of Gaussian) operator were found experimentally.

Unfortunately, even this criterion shows many local extremes, as seen in example on Fig. 3, so that the optimisation may easily be trapped into a suboptimum solution. To prevent this, certain optimisation strategy had to be found based on experimenting – a kind of multiscale procedure gradually refining the resolution in a specified way. No less important was the choice of the optimisation algorithm. After extensive testing, the best turned out the Controlled Random Search (CRS) algorithm, gradually contracting the set of estimates in the � space (Price 1979). Example of such a development can be seen on Fig. 4.

34

Fig. 3 An example of the shape of the similarity criterion for a particular pair of bimodal images

-250 -200 -150 -100 -50 0 50 100 150 200 250-250

-200

-150

-100

-50

0

50

100

150

200

250Level 1, iteration number: 1 : 389

translation x

tran

slat

ion

y

Fig. 4 Gradual contraction of estimates in two-parameter

subspace of the registering transform parameter space

As for the choice of the spatial transform, it turned out soon that although a rigid type (i.e. shift, rotation and scaling) is insufficient, the affine transform sufficesfor the given purpose. This is also demonstrated on Fig. 5 where both this way registered images are presented together with the fundus image edge representation overlay superimposed on the HRT image.

Fig. 5 Registered images prepared for fusion. Left: transformed fundus photo, right: HRT image with the edge representation of the transformed fundus photo overlaid – a rather good registration can be seen.

4. SEGMENTING RETINAL OBJECTS Initially, the segmentation of the optical disk (the optical nerve terminal) was only based on a single modality of HRT images. It was however found that,

although the HRT image carries much information, the fundus photographs may contribute to proper segmentation by adding some information that is not visible in the HRT images (and vice versa). The intention is to fuse both the images literally thus providing a vector-valued image carrying all the available information in compact form and to find segmentation procedures processing the fused vector images. So far, it remains a goal for future, as no satisfactory approach of this kind has been designed so far. However, a substantial improvement in the segmen-tation reliability and precision has been achieved [2] utilizing fusion at the level of results of analysis, i.e. using both precisely registered images in a mutually supporting mode, when each of them provides an important part of the segmentation supporting information. This follows from the fact that each modality describes some objects of interest better while others worse than the othe modality; the information from both sources is fortunately at least partly complementary.

Fig. 6 Segmentation contribution from a colour fundus

photograph (green channel): ROI filtered by weak membrane method, edge image with overlaid preliminary contour of the

optical disc.

Fig. 7 Above: Processed HRT image and the initial contour

of the neuroretinal rim derived via Hough transform. Below: two examples of final contours derived via anchored flexible contour

method. Dashed-dotted line: mono-modal segmentation, solid line: bimodal segmentation, dashed line: manual expert

segmentation

The rather complex procedure leading to automatic segmentation of the optical disc and objects in its neighbourhood is in detail described in [3]; commented Figs. 6 and 7 illustrate it.

35

5. ANALYSING AUTOFLUORESCENCE AREAS

A relatively new direction in the described project aims

to semiautomatic detection and evaluation of the autofluorescent (AF) zones in retinal images, indicating higher distribution of lipofuscin in retinal pigment epithelium which is supposed important diagnostic value. The detection is based on the AF property of lipofuscin, when illuminated by blue laser light in the Heidelberg Retina Angiograph (HRA2) working in the autofluorescent mode. Besides this modality, infrared retinal images provided in parallel are used to ease segmentation of the optical disc, in the defined neighborhood of which the AF isles are to be sought.

From the methodological view, the fusion of these bimodal images is again needed, together with specially tuned segmentation procedures.

An important part of this subproject was an effort to design a method improving the originally rather poor signal to noise ratio in the AF images (SNRsingle_frame=22.3dB). The AF images are usually taken as a time sequence of snaps of basically identical scene thus in principle enabling SNR improvement via posterior temporal averaging. However, fusing the partial images turned out a challenging task as the eye is moving during the scanning process and even during scanning of a single frame, this way causing serious frame-specific geometrical distortion of individual frames. A complex flexible B-spline transform is thus needed to reconstruct the individual frames with respect to a chosen reference frame. Although it is a monomodal case, the mutual information criterion turned out to be the only useful similarity test in the registering optimization. An example of the registered image is depicted on Fig. 8 overlaid with a originally rectangular grid distorted by the registering transform.

Fig. 8 Autofluorecence images overlaid with a grid visualizing the distortion correction by the registering transform: reference

image (left) and one of the flexibly registered images (right)

The method [4] proved to provide a substantial improvement in SNR (by over 4dB) so that many further details are revealed (Fig. 9 below) visible neither in the original frames (Fig.9 above) nor in the unregistered average (Fig.9 centre).

The improved AF images are then used in the AF zone analysis. The results of the running project concerning the AF quantitative evaluation in comparison with expert analysis are to be published later.

Fig. 9 From above to bottom: an individual autofluorescence

image, the average of non-registered images, the obtained average of flexibly registered images

ACKNOWLEDGEMENT The project has been supported by the Ministry of

Education of the Czech Republic via the research frame MS 0021630513, further by the research project 1M6798555601 – DAR centre, and also by the Czech- German project D20-CZ8/07-08.

6. CONCLUSION

Results of a long-term project on processing and analyzing different ophthalmologic image data are presented in brief. Details of the individual methods and more formal description of the approaches can be found in the referenced papers.

REFERENCES

[1] Kubecka L., Jan J., „Registration of Bimodal Retinal Images - improving modifications”, in Proc. 26th Internat. Conf. IEEE-EMBS, San Francisco (USA) 2004, pp. 1695-1698

[2] H.Niemann, R.Chrastek, B.Lausen, L.Kube�ka, J.Jan, C.Y.Mardin, G.Michelson, „Towards Automated Diagnostic of Retina Images“, J. Pattern Recognition and Image Analysis, ISSN 1054-6618, 2006, no.4, pp. 671-676

[3] Kubecka,L., Jan,J., Kolar,R., Jirik,R., „Retrospective Illumination Correction of Retinal Images“, Proc. EURASIP BIOSIGNAL 2006 conf., Brno (Czech R.) 2006, pp. 257-260

[4] Kubecka, L., Jan, J., Kolar,R., Jirik, R., „Improving Quality of Autofluorescence Images Using Non-Rigid Image Registration“, Proc. 14th EUSIPCO conf. Florence (Italy), 2006

[5] Likar B., Maintz J. B., Viergever M., Pernus F., “Retrospective shading correction based on entropy minimization.” Journal of Microscopy, Vol. 197, No. 3, 2000, pp. 285-295.

36