the validation of optscore, a software to evaluate cornea photographs christine m. toutain-kidd,...

The validation of Optscore, The validation of Optscore, a software to evaluate cornea photographsa software to evaluate cornea photographs

Christine M. Toutain-Kidd, Travis Porco, Eric M. Kidd, Thomas M. Lietman, Michael E. Zegans

Dartmouth Medical SchoolHanover, NH - USA

The authors have no financial interests.

Purpose

Digital photo-grading systems for corneal photographs are underdeveloped compared to systems for retinal photographs. Nevertheless these systems could be an important help to clinicians. The Optscore software was created to attempt to fill that void. Its validation study is presented here.

Validation of Optscore, a software to evaluate cornea pictures

Methods

Optscore is based on the Ruby on Rails web technology. One hundred cornea pictures from the Mycotic Ulcer Treatment Trial (MUTT) pilot study (Prajna et al, in press), were looked at by 6 graders (5 medical students plus an ophthalmologist used as Gold Standard). After going through a training module, they evaluated, twice at least 2 weeks apart, several variables commonly used in the diagnosis of a corneal infection: size, location, opacity, neovascularization, conjunctival inflammation and presence of a hypopyon (see list of questions on following slide). The Interclass Correlation Coefficient (ICC) was calculated to assess the intra and inter-grader variability.


Question 1:Is the ulcer/scar visible on this picture?1. Yes2. No

Question 2 (if yes in question 1):Circle the lesion on the photo

Question 3 (if yes in question 1):Location of the lesion.1. Entirely in periphery2. Overlaps central 4mm circle and

periphery without filling center3. Entirely in central circle4. Completely fills 4mm circle and

extends into periphery

Question 4 (if yes in question 1):Opacity of the lesion (worst 50% of the

lesion).1. Mild, iris details visible2. Moderate, some iris visible3. Complete, no iris visible

Question 5:Corneal neovascularization (in clock hours).0. 01. 12. 23. 34. 45. 56. 67. 78. 89. 910. 1011. 1112. 12

Question 6:Conjunctival inflammation (in clock hours)0. 01. 12. 23. 34. 45. 56. 67. 78. 89. 910. 1011. 1112. 12

Question 7:Hypopyon.0. No hypopyon1. Hypopyon but non-hemorrhagic2. Hemorrhagic

Question 8: Quality of the picture.1. Good2. Poor but gradable3. Ungradable

Questions asked during the evaluation of all pictures in this study



Snapshot of the software screen presenting a picture ready to be evaluated (with the 4 mm central circle and the 12 mm outside circle overlapping the limbus) as well as the first question asked to the graders.Once the grader has answered the question, the software automatically goes to the next question.

Snapshot of the screen summarizing the questions and answers provided for a picture. Until the grader clicks on “next image” (not visible on this snapshot), all answers can be revised and changed.


The Steroids for Corneal Ulcers Trial (SCUT) and Mycotic Ulcer Treatment Trials (MUTT) I and II are collaborative studies involving the Aravind Eye Hospital in India, the F.I. Proctor Foundation at UCSF and Dartmouth Medical School. We used some of the pictures obtained during the MUTT pilot study in this work.

The following tables present the Interclass Correlation Coefficient (ICC) calculated for all variables, which allows to compare the consistency of each grader and between the graders and the gold standard (GS).


Results and Conclusions

Variable Area Location Opacity Neovascularization

Conj. Inflam.

Hypopyon

Overall ICC 0.8942 0.7974 0.6322 0.6313 0.8632 0.8425

Interclass Correlation Coefficient (ICC) calculated for all variables evaluated in this study.

Table A shows the overall ICC for each variable.

A.


- The measure of the area and the evaluation of the presence of a hypopyon are the two variables with the highest ICC.- More subjective variables like level of opacity or estimation of neovascularization have a low ICC.

Intergrader variability

Grader A/

GS1

Grader B/

GS1

Grader C/

GS1

Grader D/

GS1

Grader E/

GS1

Area 0.9228 0.9461 0.8966 0.8722 0.9365

Location 0.7877 0.8996 0.8097 0.8496 0.7794

Opacity 0.6509 0.5338 0.6275 0.5885 0.6877

Neovascularization

0.6354 0.6415 0.5161 0.5757 0.6454

Conj. Inflam

0.8722 0.8342 0.8826 0.8287 0.7903

Hypopyon 0.8241 0.8329 0.9416 0.7542 0.8390

B.

Table B presents the intergrader variability calculated between each grader and the Gold Standard’s first grading.


- When comparing the evaluations of each grader with the gold standard (GS), the variables with the highest ICC here are also the ones with the highest overall ICC, such as area, location and hypopyon.

- The evaluation of the level conjunctival inflammation also shows in both cases, tables A and B, a high ICC. This variable is continuous but also quite subjective like neovascularization, the subjectivity has to be taken into account. There were many cases where the number of clock hours of inflammation estimated by the graders was different by one or two clock hours (which is not much on the calculation of the ICC but still leads to a lack of consistency).


Table C shows the intragrader variability calculated using the two evaluations done by each grader at least 2 weeks apart.

Intragrader variability

GS1/GS2 Grader A Grader B Grader C Grader D Grader E

Area 0.9823 0.9772 0.9836 0.9762 0.9086 0.9771

Location 0.8533 0.7655 0.9481 0.6736 0.7419 0.8132

Opacity 0.6918 0.3915 0.8534 0.6194 0.6402 0.8164

Neovascularization

0.7568 0.8836 0.9026 0.9339 0.8895 0.9019

Conj. Inflam

0.9510 0.9754 0.9793 0.9469 0.9529 0.9364

Hypopyon 0.8879 0.9353 0.8531 0.9416 0.7772 0.8883

C.

- It should be noted that all graders went through the same training module composed of two parts, a power point presentation teaching them about the cornea and the variables that they would have to evaluate, and a training set of pictures to teach them how to use the software. The time required by each grader to perform the evaluations was recorded as well.- Grader D, who overall has a relatively low intragrader ICC, including for non subjective variables such as area size and presence or absence of hypopyon, did the evaluations the fastest.

- This study demonstrates that Optscore can be reliably used to evaluate several variables required for the assessment of corneal infections. - The training of the graders is essential and the importance of spending enough time per picture should be emphasized. - The least subjective variables show the highest correlation amongst graders.

- Optscore is a very flexible software that can be modified to fit any requirements.

- We believe that Optscore could be developed as a diagnostic help and an educational tool.


the validation of optscore, a software to evaluate cornea photographs christine m. toutain-kidd,...

Documents

periphery question

photo question

hemorrhagic question

optscore software

iris visible question

cornea pictures

study validation of

software screen