Syphilis POC Tests:ypWhat is the verdict?

A Systematic Reviewy

Yalda JafariMSc Candidate

Department of Epidemiology. Biostatistics and Occupational HealthMcGill University

yalda.jafari@mail.mcgill.ca

We have no conflictWe have no conflict of interest withof interest with 

industry!industry!

CONTEXTCONTEXT•12 million new cases occur every year! (WHO, 2006)

WHO/TDR, 2006

year! (WHO, 2006)

•3‐15% of women  of child bearing age infected with syphilis(WHO, 2006)g yp

•30% chance of still birth (WHO, 2006)

•30% of birth to baby with congenital syphilis (WHO, 2006)

•63% of primary & secondary syphilis cases in US were in MSM (CDC, 2008)

http://www.myhimalayas.com/mustang/3.htm

CONTEXTCONTEXT•In developing countries, there is lack of proper screening due to limited laboratory 

WHO/TDR, 2006

services and long distances

•serologic testing at 28 weeks’ gestation g g gand at delivery (CDC, 2006)

•Curable with Penicillin: single•Curable with Penicillin: singleintramuscular injection will cure a person with infection duration of  less than a year (CDC, 2010)

•Most cost effective intervention—antenatal screening and treatment (WHO, 2006) http://www.myhimalayas.com/mustang/3.htm

CONTEXTCONTEXTDiagnosis: 

•Dark field microscopy

•Serology•Serology

Non‐Treponemal Tests Treponemal Tests spirochaete Treponema pallidumhttp://www.epsomandewellhistoryexplorer.org.uk/Mott.html

RPRVDRL

TPPATPHAFTA‐ABSEIAEIA

•Point‐of‐care tests : TP Specific

METHODSMETHODSTime: Jan 1, 1980‐Sept 24, 2010

Databases: MEDLINE,  EMBASE, GLOBAL HEALTH, CINAHL, Web of Science and SCOPUS.

Keywords: syphilis treponema pallidum Point of Care rapid test and rapid assaysKeywords: syphilis, treponema pallidum, Point‐of‐Care, rapid test and rapid assays

Inclusion: research articles reporting on any aspect of point‐of‐care syphilis tests

Exclusion: Abstracts, letters and brief reports, editorials, perspectives, opinion pieces and manufacturer reports

Quality Critique: QUADAS (Quality Assessment of Diagnostic Accuracy Studies) andQuality Critique: QUADAS (Quality Assessment of Diagnostic Accuracy Studies) andSTARD (Standards for Reporting of Diagnostic Accuracy) checklists.

Review: PRISMA (Preferred Reporting Items for Systematic Reviews and Meta‐Analyses) was followed for reporting the review.y ) p g

METHODSMETHODSStatistical analysis:

• meta‐analysis using Hierarchical Summary Receiver Operation Characteristic (HSROC), as an add on package p p gto R. 

h h ld d•HSROC  assumptions: positivity threshold and accuracy are independent test characteristics that together impose correlation between a test’s sensitivity and specificitycorrelation between a test s sensitivity and specificity

•HSROC accounts for within and between study variationy

Rutter CM, Gatsonis CA. A hierarchical regression approach to meta‐analysis of diagnostic test accuracy evaluations. Stat Med. 2001 Oct 15;20(19):2865‐84.

METHODSMETHODSStratification:

POC Diagnostic TestTest

Serum Whole BloodSample

TP specific Non‐TP specific

TP specific Non‐TP specificre

nce 

ndard

TP & non‐TP specific

p

TP & non‐TP specific

p

Refe

Stan

Minimum of 4 entries/strata are needed for pooling. 

METHODSMETHODS

•Within study variation:

•Between study variation:

•Adjusted for imperfect reference standard using a range of 90‐100% for sensitivity and specificity. 

Rutter CM, Gatsonis CA. A hierarchical regression approach to meta‐analysis of diagnostic test accuracy evaluations. Stat Med. 2001 Oct 15;20(19):2865‐84.

RESULTSRESULTSPRISMA Flow Chart

Records identified through d b hion Additional records identified 

h h hdatabase searching (n = 741)

Iden

tificatio

Records after duplicates removed (n = 647)

through other sources(n = 1)

Screen

ing

Records screened (n =647 )

Records excluded(n = 583 )

Eligibility

( 583 )

Full‐text articles assessed for eligibility 

(n = 64)Full‐text articles excluded, with reasons (n =  18 )                                                                                              not POC, n=9                             Bulletin, n=1                                  

lude

d

Letters, n=3                                Reviews, n=2                                           Meetingbv abstract, n=1 Incomplete cell values, n=1

Studies included in qualitative synthesis (n = 18)

Studies included in quantitative synthesis 

(meta‐analysis) (n = 30)

Inc

ACCURACY STUDIESIncluded studies:

• Languages: 40 English4  Spanish2  Japanese1 P t1  Portuguese1  Russian1  Chinese

•Setting:  Low‐income or Lower‐middle income:   73%  (91/124)Upper‐middle income or High income:   27% (33/124)

WHO/TDR, 2006

•POC tests:  16 unique tests. Top 4 tests: Determine (Abbott Diagnostics, UK)

Bioline Syphilis (Standard, South Korea)Syphicheck test (Qualpro, India)Visitect test (Omega Diagnostics)

Immunochromatographic strip: 15/16

ACCURACY STUDIESIncluded studies:•Samples:  Whole Blood: 56% (69/124),   Prev: 20%  (min: 0% , max:50.4%)Serum: 44% (55/124),              Prev: 17% (min: 0.3% , max: 53.5%)

•Population : Whole  Blood, 53%(16/30) Serum, 67% (20/30)

•44% (7/16) tested STI clinic attendees • 25% (5/20) tested STI clinic attendees( / )

•19% (3/16) tested FSWs

•38% (6/16) tested pregnant women

( / )

•15% (3/20) tested FSWS

•25% (5/20) tested pregnant/women of•38% (6/16) tested pregnant women

•6% (1/16) used blood supplied by hospital

•25% (5/20) tested  pregnant/women of child‐bearing age

•35%(7/20) tested  reference labs/hospitals

•10% (2/20) source unclear

ACCURACY STUDIESIncluded studies:

Quality: STARD: 16.5/25,  (9‐25)QUADAS: 10.0/15,  (7‐14)39% d i i t f t d d39% used inappropriate reference standard

•Conflict of Interest: 37% (11/30) reported conflict of interest9% (1/11) had conflict of interest10% (3/30) evaluated tests they had manufactured10% (3/30) evaluated tests they had manufactured 

•Blinding:33% (10/30) reported on blinding33% (10/30) reported on blinding100% (10/10) blinded between index and reference results

ACCURACY STUDIES

Determine test: S lSerum sampleTP specific reference standard:(n=10)

Assuming perfect gold standardPooled Parameters Estimate (95% CI)

Sensitivity 98.89% (96.47 , 99.99)

Assuming imperfect gold standard

Specificity 97.91% (96.56, 99.05)

Pooled Parameters Estimate

Sensitivity 99.04% (96.91, 100)

Specificity 99.81% (98.71, 100)p y ( , )

RESULTSPOC Test Pooled 

ParametersAssuming Perfect Reference 

StandardAssuming Imperfect Reference Standard

Reported by ManufacturerRESULTS

D t i

Serum

TP Specific (n=10)

Sensitivity 98.89% (96.47 , 99.99) 99.04% (96.91, 100) 100%

Specificity 97.91% (96.56, 99.05) 99.81% (98.71, 100) 100%TP & non‐TP Specific (n=4)

Sensitivity 97.26% (88.36, 100) 98.34% (89.43, 100)Specificity 98.71% (92.77, 100) 99.76% (98.03, 100)

Determine

Whole Blood

TP Specific (n=14)

Sensitivity 85.55% (76.15, 94.49) 88.74%(79.46, 96.50) 92.30%

Specificity 99.50% (98.95, 99.93) 99.98% (99.57, 1) 100%

TP & non‐TP Specific (n=8)

Sensitivity 83.60 (64.85, 97.81) 90.65% (68.68, 99.99)Specificity 98 59% (94 91 100) 98 93% (96 49 99 97)Specific (n 8) Specificity 98.59% (94.91, 100) 98.93% (96.49, 99.97)

Bioline

Serum TP Specific (n=8)

Sensitivity 93.99 (88.79, 98.45)  95.32% (90.24 , 100) 99.3%

Specificity 98.58 (97.54,  99.31)  98.76 %(97.80 , 100) 99.5%

Wh l Bl d TP Specific  Sensitivity 87.70% (84.78, 90.58) 89.36%(85.02,  96.14)Whole Blood p

(n=13) Specificity 99.07% (98.50 , 99.59) 99.29% (98.66, 100)

Syphicheck

Serum TP Specific (n=7)

Sensitivity 81.91% (69.10, 93.19) 85.17% (71.84, 100) 89.20%

Specificity 99.05% (98.35, 99.62) 99.47% (98.59, 100) 96.20%

Whole Blood TP Specific (n=12)

Sensitivity 76.78 %(69.36, 84.03) 80.62% (70.53, 97.52)

Specificity 99.44%(99.06, 99.76) 99.57% (99.17, 100)

Serum TP Specific (n=6)

Sensitivity 93.96% (89.24,  98.01) 99.34% (95.84, 1.00)

Specificity 98 51% (97 70 99 19) 99 67% (98 78 100)Visitect

Specificity 98.51% (97.70, 99.19) 99.67% (98.78, 100)

Whole Blood TP Specific (n=12)

Sensitivity 78.05% (70.34, 85.02) 85.90% (75.19, 97.61)

Specificity 99.52% (99.16, 99.82) 1 00(99.64, 100)

ACCURACY STUDIES

•Increased accuracy in all four tests in serum in comparison to accuracy in whole y p yblood sample but result is only statistically significant  in Determine and Visitect, since the credible intervals do not overlap.

•Determine in serum has highest accuracy but not statistically significant compared•Determine in serum has highest accuracy, but not statistically significant compared to second highest accurate test in serum, Bioline.

•Bioline in whole blood has higher accuracy, but not statistically significant compared to second highest accurate test in whole blood, Determine.

•Determine in serum is the only scenario where specificity is lower than sensitivity.

•Adjusting for imperfect reference standard improved accuracy estimates.

BEYOND ACCUACYBEYOND ACCUACY18 articles identified

Patient Centered Operational Cost and health services delivery

Acceptability(n=8)

Acceptability(n=8)

Feasibility (n=2)

Feasibility (n=2)

Prevalence(n=8)

Prevalence(n=8)

Preference(n=3)

Preference(n=3)

( )( )

Impact( 6)Impact( 6)

(n=8)(n=8)

Economic EvaluationEconomic Evaluation(n=6)(n=6) Evaluation

(n=7)Evaluation

(n=7)

Barriers and ChallengesBarriers and ChallengesChallenges, 

(n=3)Challenges, 

(n=3)

BEYOND ACCURACY• Acceptability (N=8):

– acceptability by the patient and providers remained high across settings (e.g, 100% in South Africa )• Preference (N=3):

– Preference by patients and providers remained high across settings (eg. 79% in MSM to 100% inMongolian ANC)

• Feasibility (N=2):y ( )– Point‐of‐care testing as one‐stop service was operationalizable as demonstrated by Munkhuu et al in

Mongolian ANC• Impact (N= 6):

Increase in number of people who know their sero status and thus received treatment– Increase in number of people who know their sero‐status and thus received treatment– Increased number of people who received treatment (eg. 93.2% in Bolivian ANC to 100% in Mongolian

ANC)– Decrease in incidence of congenital syphilis with testing (93.5% reduction in Mongolian ANC)

d h ll ( )• Barriers and Challenges (N=3):– uncertainty about the accuracy of the test under investigation (60% of clinicians in Brazilian STD clinic)– geographic isolation of study site– inadequate supply of kitsq pp y– lack of political will

IMPLICATIONSIMPLICATIONS•Syphilis POC tests detect TP‐specific antibodies and thus, cannot differentiate between new and old infections!•Hence, their usefulness is dependent on prevalence!•Low prevalence, useful!•High prevalence, may over‐treat!

•Higher accuracy in serum than in whole blood •Based on this limitation,  it is advisable that available resources  be focused towards adequate methods for obtaining serum ultimately allowing fortowards adequate methods for obtaining serum, ultimately allowing for widespread serum testing.

•New POC tests that have TP and non‐TP component will be greatlyNew POC tests that have TP and non TP component will be greatly beneficial!•Currently, one such POC test exists but can be used only with serum.•Biotechnological advances are a major obstacle in syphilis diagnostics.g j yp g

IMPLICATIONS FOR POLICYIMPLICATIONS FOR POLICY•Improve standard for reporting of studies and manufacturers

•Best algorithm with current diagnostic tools?

• If possible serum should be used rather than whole blood for more• If possible, serum should be used rather than whole blood for more accurate results

C t i j d t i t•Cost is a major determinant.

•POC tests should be adopted as first line of screening in remote areas h l b f ili i i li i d d h diwhere access to laboratory facilities is limited and there are dire 

consequences such as congenital syphilis

ACKNOWLEDGENTSThis work was supported by a Knowledge Syntheses grant from 

the Canadian Institute for Health Research CIHR KRS 102067the Canadian Institute for Health Research. CIHR KRS 102067Thank you!Dr.Nitika Pant PaiDr.Nitika Pant PaiDr. Rosanna PeelingDr. Lawrence JosephDr. Jorge M CajasDr. Gilles LambertDr. Christiane ClaessensDr. Marina B KleinDr Madh kar PaiDr. Madhukar Pai