uog journal club: july 2014

UOG Journal Club: July 2014Interobserver agreement in describing adnexal masses using the International Ovarian Tumor

Analysis simple rules in a real-time setting and using three-dimensional ultrasound volumes and digital clips

B. RUIZ DE GAUNA, P. SANCHEZ, L. PINEDA, J. UTRILLA-LAYNA, L. JUEZ, J. L. ALCÁZAR

Volume 44, Issue 1, Date: July 2014, pages 95-99

Intra- and interobserver agreement with regard to describing adnexal masses using International Ovarian Tumor Analysis terminology: reproducibility study involving seven observers

L. ZANNONI, L. SAVELLI, L. JOKUBKIENE, A. DI LEGGE, G. CONDOUS, A. C. TESTA, P. SLADKEVICIUS, L. VALENTIN

Volume 44, Issue 1, Date: July 2014, pages 100-108

Journal Club slides prepared by Dr Tommaso Bignardi(UOG Editor for Trainees)

Background

• In 2008, the International Ovarian Tumor Analysis (IOTA) group proposed the so-called ‘simple rules’ for ultrasound classification of adnexal masses.

• Simple rules can be applied in about 75–80% of all adnexal masses.

• When the mass is classified as benign or malignant using simple rules, the diagnostic performance is good, even when applied by examiners with differing levels of expertise.

Interobserver agreement in describing adnexal masses using the International Ovarian Tumor Analysis simple rules in a real-time setting and using three-dimensional

ultrasound volumes and digital clipsRuis de Gauna et al., UOG 2014

Objective: To estimate agreement between expert and non-expert

examiners using the IOTA simple rules for classifying adnexal masses on real-time ultrasound and using three-dimensional

(3D) ultrasound volumes and digital clips





N=42 women with adnexal mass were evaluated by transvaginal power Doppler ultrasound



Examination by non-expert (trainee)

Expert examination

Videoclips &Static 3D volumes acquired by expert

Assessed by four trainees with different levels of training

Methods

• Examiners were blinded to each other.

• Examiners had to classify the mass as benign, malignant or inconclusive using real-time ultrasound evaluation, short video clips (about 15–20s) and by manipulating 3D volumes.

• Agreement in classifying the mass as benign, inconclusive or malignant was estimated by weighted kappa index and percentage of agreement.

• Agreement for each ultrasound feature was estimated by the standard kappa index.



Methods

Features for malignant tumor Features for benign tumor

M1 Irregular solid tumor B1 Unilocular tumor

M2 Presence of ascites B2 Presence of solid components where solid component largest

diameter is <7mm

M3 At least four papillary projections B3 Presence of acoustic shadows

M4 Irregular multilocular solid tumor with largest diameter ≥100mm

B4 Smooth multilocular tumor with largest diameter <100mm

M5 Very strong blood flow (color score 4) B5 No blood flow (color score 1)

Timmerman et al, UOG 2008

Simple rules for classifying adnexal masses proposed by the International Ovarian Tumor Analysis (IOTA) group



Trainee

Benign Inconclusive Malignant Total

Expert examiner

Benign 17 0 0 17

Inconclusive 4 2 2 8

Malignant 1 2 14 17

Total 22 4 16 42

Weighted kappa = 0.76 (95% CI, 0.61–0.90); percentage of agreement=78.6% (33 of 42)

Results: Agreement between expert examiner and trainee on real-time ultrasound examination with regard to classifying

adnexal masses as benign, malignant or inconclusive



FeaturePrevalence %

(n/n)*Kappa index

(95% CI)Percentage agreement

(n/n)

B1: Unilocular tumor 17.8 (15/84) 0.89 (0.75 to 1.0) 95.2 (40/42)

B2: Presence of solid components where solid component largest diameter is <7mm

4.8 (4/84) 0.64 (0.19 to 1.0) 95.2 (40/42)

B3: Presence of acoustic shadows 4.8 (4/84) 0.36 (−0.20 to 0.92)

92.9 (39/42)

B4: Smooth multilocular tumor with largest diameter <100mm 8.3 (7/84) 0.91 (0.73 to 1.0) 97.6 (41/42)

B5: No blood flow (color score 1) 19.0 (16/84) 0.53 (0.25 to 0.82) 80.9 (34/42)

M1: Irregular solid tumor 14.3 (12/84) 0.58 (0.27 to 0.88) 85.7 (36/42)

M2: Presence of ascites 10.7 (9/84) 0.42 (0.03 to 0.80) 85.7 (36/42)

M3: At least four papillary projections 4.8 (4/84) 0.64 (0.19 to 1.0) 95.2 (40/42)

M4: Irregular multilocular solid tumor with largest diameter ≥100mm 8.3 (7/84) 0.81 (0.55 to 1.0) 95.2 (40/42)

M5: Very strong blood flow (color score 4) 28.6 (24/84) 0.63 (0.40 to 0.85) 80.9 (34/42)

*Number of times the feature is observed by any observer/total number of observations.

Results: Agreement between expert examiner and trainee on real-time ultrasound for each ultrasound feature

3rd year trainee 2nd year trainee 1st year trainee

4th year trainee WK=0.56 (0.33–0.80) [76.2%]

WK=0.58 (0.35–0.81) [76.2%]

WK=0.51 (0.27–0.73) [71.4%]

3rd year trainee - WK=0.51 (0.26–0.74) [72.2%]

WK=0.46 (0.21–0.71) [71.4%]

2nd year trainee - - WK=0.45 (0.20-0.71) [71.4%]

Data are given as weighted kappa (WK), (95% CI), [percentage of agreement]

Results: Agreement between trainees with regard to classifying adnexal masses as benign, malignant or unclassifiable when analyzing video clips

and three-dimensional volumes



• Good agreement with regard to classifying adnexal masses as benign, malignant or inconclusive using the IOTA simple rules between an expert and a less experienced examiner.

• Moderate agreement using a video clip plus a 3D volume for trainees with different degrees of training.

• ‘Presence of acoustic shadows’ and ‘Presence of ascites’ showed the worst agreement between expert and trainee on real-time imaging.

Results



• First study assessing interobserver agreement with regard to describing adnexal masses using the IOTA simple rules during real-time ultrasound.

Strengths of the study

Weaknesses

• The series is small and comprises of a selected population.

• No sample size estimation with wide CIs for the kappa index.

• Analysis does not include inherent sources of variability between different acquisitions.



Background

• Even when using standardized terms and definitions, ultrasound examiners may evaluate the features of an adnexal mass differently.

• Consequently, the risk of malignancy calculated by mathematical models may vary both within and between examiners.

• Problems that ultrasound examiners may encounter when using gray-scale IOTA terminology for adnexal masses have not been evaluated in any published study.

Intra- and interobserver agreement with regard to describing adnexal masses using International Ovarian Tumor Analysis terminology: reproducibility study involving seven

observersZannoni et al., UOG 2014

Objective: To estimate intraobserver repeatability and interobserver agreement in assessing the presence of papillary projections in adnexal masses

and in classifying adnexal masses using the IOTA terminology for ultrasound examiners with different levels of experience



• Digital clips (2-8 clips per adnexal mass) with gray-scale and color/power Doppler information of 83 adnexal masses were acquired.

• Evaluated independently by 4 experienced and 3 less experienced ultrasound observers.

• Four evaluations: 2 before and 2 after a consensus meeting.

• Variables analyzed were tumor type and presence of papillary projections.

Methods



Parameter Agreement (%) Fleiss kappa Cohen’s kappa

First evaluation before meeting

All observers (n=7) 76.0 (60.2–85.5) 0.695 (0.687–0.703) 0.690 (0.500–0.810)

Experienced observers (n=4) 78.7 (72.3–81.9) 0.735 (0.726–0.744) 0.727 (0.653–0.762)

Less experienced observers (n=3) 70.6 (67.5–74.7) 0.621 (0.611–0.632) 0.617 (0.576–0.672)

First evaluation after meeting

All observers (n=7) 75.4 (66.3–85.5) 0.682 (0.675–0.689) 0.682 (0.554–0.812)



Data are given as mean (range) for pairwise agreement and pairwise Cohen’s kappa, and point estimate (95% CI) for Fleiss kappa

Results: Interobserver agreement with regard to tumor type among seven observers before and after a consensus meeting



Parameter Agreement (%) Fleiss kappa Cohen’s kappa

First evaluation before meeting

All observers (n=7) 86.6 (75.9–92.3) 0.536 (0.522–0.550) 0.520 (0.148–0.747)


Less experienced observers (n=3) 87.1 (84.3–89.2) 0.441(0.419–0.463) 0.434 (0.348–0.577)

First evaluation after meeting

All observers (n=7) 82.7 (68.7–92.8) 0.487 (0.474–0.500) 0.480 (0.194–0.787)



Data are given as mean (range) for pairwise agreement and pairwise Cohen’s kappa, and point estimate (95% CI) for Fleiss kappa

Results: Interobserver agreement with regard to papillary projections among seven observers before and after a consensus meeting



Before meeting After meeting

Observer Agreement (%)

Kappa (95%CI) Agreement (%)

Kappa (95%CI)

Most experienced observers

A 87.9 0.84 (0.75–0.93) 91.6 0.89 (0.81–0.97)

B 86.7 0.83 (0.74–0.92) 90.3 0.86 (0.78–0.94)

C 89.1 0.86 (0.78–0.94) 86.7 0.83 (0.73–0.92)

D 86.7 0.83 (0.74–0.92) 86.8 0.83 (0.74–0.92)

Less experienced observers

E 92.7 0.90 (0.82–0.98) 92.7 0.90 (0.86–0.94)

F 95.1 0.94 (0.88–1.0) 90.3 0.87 (0.79–0.95)

G 95.2 0.94 (0.88–1.0) 86.8 0.83 (0.74–0.92)

Results: Intraobserver repeatability in 83 cases of adnexal mass before and after a consensus meeting when classifying tumors

Before meeting After meeting

Observer Agreement (%)

Kappa (95%CI) Agreement (%)

Kappa (95%CI)

Most experienced observers

A 91.6 0.72 (0.52–0.92) 96.4 0.89 (0.77–1.0)

B 94.0 0.83 (0.69–0.97) 95.1 0.90 (0.80–1.0)

C 92.8 0.77 (0.59–0.95) 88.0 0.67 (0.48–0.86)

D 94.0 0.80 (0.63–0.97) 92.8 0.81 (0.67–0.95)

Less experienced observers

E 98.8 0.88 (0.66–1.0) 96.4 0.75 (0.47–1.0)

F 91.6 0.76 (0.59–0.93) 90.4 0.76 (0.60–0.92)

G 100.0 1.0 (1.0–1.0) 90.3 0.64 (0.41–0.87)

Results: Intraobserver repeatability in 83 cases of adnexal mass with regard to the presence of papillary projections before and after a consensus meeting

Results:

• There was uncertainty about how to define a solid component and a papillary projection, but consensus was reached at the consensus meeting.

• Interobserver agreement was good for tumor type.

• Interobserver agreement for papillary projection was moderate but very variable between observer pairs.



• First study assessing problems that ultrasound examiners may encounter in describing gray-scale ultrasound images of adnexal masses using IOTA terminology.

Strengths of the study

Weaknesses• No sample size estimation with wide CIs for the kappa index.

• Results might not be applicable to live ultrasound examinations.

• Results are generalizable only to adnexal masses similar to those in the study, and to observers with similar experience.



• The IOTA terms ‘papillary projection’ and ‘solid component’ might need a more precise definition.

• Practical advice is needed on how to discriminate between a solid component and ovarian tissue, a solid component and densely packed septa or small cysts, a solid component and a tangentially cut septum, etc.

• The consensus meeting did not change inter- or intraobserver agreement.

Discussion



Discussion points

• What is the clinical evidence for the utility of ultrasound classification of tumors to dictate subsequent clinical management?

• How are ovarian tumor types classified by ultrasound in your unit?

• Are videos/3D volume acquisition a source of significant variability in the evaluation of ovarian masses?

• How should we define a papillary projection or a solid component of an adnexal mass using IOTA terminology?

• How does the known reproducibility of the test affect estimation of risk of malignancy using IOTA or simple rules

uog journal club: july 2014

Health & Medicine