ruben j. echemendia fda workshop assuring the safety and effectiveness of seizure, cognitive...
TRANSCRIPT
![Page 1: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/1.jpg)
NEUROPSYCHOLOGICAL EVALUATION OF SPORTS CONCUSSION/TBI:
HISTORICAL AND PSYCHOMETRIC CONSIDERATIONS
Ruben J. Echemendia
FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive
Function, and TBI/Sports Concussion Diagnostic Devices
Washington, DC: June, 2011
![Page 2: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/2.jpg)
Disclosure
Dr. Echemendia is a paid consultant to: The National Hockey League Major League Soccer US Soccer Federation
![Page 3: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/3.jpg)
Historical Background
Neuropsychological assessment has been a key component in evaluating/managing TBI for decades.
Neuropsychological evaluations have become a key tool in the evaluation of MTBI since mid 70’s.
Focus on functional vs. structural assessment.
NP assessment of sports concussions began in mid 80’s.
![Page 4: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/4.jpg)
Background
Sports Laboratory Assessment Model Pre-post injury evaluations Paper & Pencil Assessment Batteries Computerized Batteries
![Page 5: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/5.jpg)
Psychometric Considerations Validity vs. Reliability Validity
Does the test measure what it is supposed to measure? Construct Validity –Operationalization of the construct,
measurement of the construct Cognitive tests DO NOT measure concussion! Assess cognitive domains that are believed to be affected by
concussion Convergent/Divergent Validity
Does the test correlate with other known measures of the construct?
Does the test not correlate with domains that it is not supposed to measure Face Validity
Face Validity Does the test appear to be measuring what it is supposed to
measure
![Page 6: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/6.jpg)
Psychometric Considerations
Sensitivity vs. SpecificitySensitivity: Is the measure “sensitive” in
detecting the construct being measured?○ Do concussed individuals score lower than
non-concussed individuals?○ What proportion of concussed individuals are
identified by the test? Positive Predictive Power
○ What proportion of Non-concussed individuals are identified by the test? Negative Predictive Power
![Page 7: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/7.jpg)
HYPOTHETICAL 2X2 VALIDITY MATRIX
Abnormal NP Scores Normal NP Scores
0
0 150150
300
Concussed
Normal
150150
![Page 8: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/8.jpg)
HYPOTHETICAL 2X2 VALIDITY MATRIX
Abnormal NP Scores Normal NP Scores
30/ 20%
150
300
Concussed
Normal
150
120/ 80%
90/ 60%60/ 40%
True Positive False Negative
False Positive True Negative
![Page 9: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/9.jpg)
Psychometric Considerations
Sensitivity vs. SpecificitySpecificity: Is the measure “specifically”
measuring the identified construct and not others?
Concussion vs ADHD, Depression, Anxiety, Fatigue, etc.
Ideal: 100% Sensitivity – 100% SpecificityReal: NOT!Sensitivity vs Specificity trade-offTypically as sensitivity increases, specificity
decreasesThe user must decide on what side to err.
![Page 10: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/10.jpg)
ROC Curve
![Page 11: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/11.jpg)
Psychometric Considerations
Reliability – Estimating ErrorCan we expect to get the same results when
the test is given….○ To the same person an infinite number of
times? (Confidence Intervals)○ To the same person under different
circumstances? (Sideline vs. Office)○ To the same person at different points in time?○ To the same person by different test
administrators?○ In different/alternate forms?
![Page 12: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/12.jpg)
Tests Used in Sports TBI….
![Page 13: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/13.jpg)
Paper & Pencil Tests Selected from tests already in use and validated
for assessing MTBI Test Requirements:
ShortEasily administered by non-NPsEasily scoredSensitive to changes in specified domainAcceptable psychometricsAlternate forms.
Assess several cognitive domains. Typical Tests: HVLT. BVMT. Trails, Digit Span,
SDMT, Verbal Fluency, LNS, Cancellation tests.
![Page 14: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/14.jpg)
Paper & Pencil Tests Cont. Advantages:
Direct observation of player behaviorDirect monitoring of effort/motivation & task
performanceSuperior assessment of memoryBroad normative db with multiple samples.
Disadvantages:Labor intensive (Expensive)Variability in standardized adminLess reliable assessment of info proc speed.Tests must be hand scored
![Page 15: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/15.jpg)
Computerized Batteries Two approaches:
Adapt existing P&P measures into computer platform
Measure multiple cognitive functions (memory, proc speed, reaction time, etc.)
Develop tests de novo for computers Limited assessment of domains (largely
RT, Info Proc Speed) New Instruments are being developed rapidly Existing Instruments: ImPACT, ANAM Sports,
HeadMinders CRI, CNS Vital Signs, CogSport.
![Page 16: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/16.jpg)
Computerized Tests Cont. Advantages:
Can be used in groups (Less costly)Standardized admininistrationImmediate scoring/data captureFiner assessment of RT/Proc Speed
(altho other problems)Multiple languagesTheoretically infinite alternate
forms
![Page 17: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/17.jpg)
Computerized Tests Cont. Disadvantages:
Less complete assessment of memoryLoss of observational dataLess control of effort/motivationDiminished ability to assess
understanding/track task performanceComputer-specific measurement issuesData loss
![Page 18: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/18.jpg)
Hybrid Approaches
Combine strengths of both P&P and Computerized tests
Once adequate normative data have been established….
Computer for baseline testingComputer plus P&P post-injury
![Page 19: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/19.jpg)
Sports MTBI specific issues
Phase of Evaluation
Baseline
Acute – On Field Management
Sub acute – Office/Training Room
Return to Play/Work/School
Long Term Follow-up
![Page 20: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/20.jpg)
Sports MTBI specific issues Baseline Testing – Is it necessary/helpful? Theoretically designed to minimize error due
to intra-individual comparison Does it do this? Does it increase sensitivity/specificity? Does it introduce additional sources of error? What about multiple post-injury tests? How often should it be repeated? Age dependent?
![Page 21: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/21.jpg)
Sports MTBI specific issues Test – Retest Reliability Significant variability across studies What test-retest intervals should be
used? Reliability estimates for multiple tests
![Page 22: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/22.jpg)
Sports MTBI specific issues Practice Effects Content Practice vs. Procedural Use of Alternate Forms Reliability/Validity of Alternate Forms Are practice effects bad? How to control for / appropriately use
practice effects?RCI, regression based estimates
![Page 23: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/23.jpg)
Sports MTBI specific issues Motivation/Effort Unique aspect of sports was the
assumption that players are motivated to perform well.
Different motivation as different phases of assessment
Baseline vs. Post-injury Individual vs. Group Identification of poor effort
Test specific indicesOutlier identification
![Page 24: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/24.jpg)
User Qualifications
Although not the responsibility of the FDA, clear delineation of user qualifications for administering and interpreting tests is critical.
Failure to do so may lead to scope of practice challenges, medico-legal quandaries and poor practice.
![Page 25: Ruben J. Echemendia FDA Workshop Assuring the Safety and Effectiveness of Seizure, Cognitive Function, and TBI/Sports Concussion Diagnostic Devices Washington,](https://reader035.vdocuments.site/reader035/viewer/2022081602/5517e973550346d5568b4884/html5/thumbnails/25.jpg)
Thank you!