1. Evaluation of Multisite E-learningTraining for VA Mental HealthProviders within the CAMS Study1

2. PresentationJan York, PhD, APRN, FAAN (CO-I)1,2,3Nurse Researcher, Research ProfessorUNM Visiting FacultyElizabeth Marshall, MD, MBA1,3Research Coordinator, DesignerKathryn Magruder, PhD, MPH (PI)1,3Research Health Scientist, ProfessorMark De Santis, PsyD, (CO-I) 1Suicide Prevention Coordinator &LeadDerik Yeager, MBS 1, 3Research AssociateRalph H. Johnson VAMC, CharlestonAffiliation1. Ralph H. Johnson VAMC2. UNM College of Nursing, Albuquerque, NM3. Medical University of South Carolina2 3. Colleagues & e-Learning Co-authors3Affiliations1. The Catholic University ofAmerica, Washington, DC2. Ralph H. Johnson Veterans AdministrationMedical Center, Charleston, SC3. Medical University of SouthCarolina, Charleston, SCDavid A. Jobes, PhD, ABPP 1Professor, Co-Director of Clinical Training, Co-IRebecca Knapp, PhD 3Statistician, ProfessorLouisa Burriss, PhD 2Research CoordinatorMary Mauldin, EdD 3ProfessorStan Sulkowski, BS3CARC AssistantCharlene Pope, PhD, MPH, 2 ,3Co-I, Associate Professor, Assoc NurseJonathan Coultas, BA3CARC Assistant 4. Presentation Outline Grant information Background Targeted intervention On line training Presentation objectives Methodology Approval Implementation-development, CEUs, sites, recruitment,delivery Evaluation-measures, initial findings Preliminary conclusions Next steps and off shoots4 5. Patient and Provider Outcomes ofe-Learning Training in CAMSObjective:to develop and test the effectivenessof an electronic learning alternativeto the Collaborative Assessment andManagement of Suicidality (CAMS)in-person approach.VA HSR&D EDU 08-424 fundedhealth education research3 year (2009-2012), multisitestudy, SE VISN (VA region)5 6. Suicide in the U.S.(2010 CDC data)Suicide is the Tenth leading cause of death: >38,000 suicides that year in theU.S. (rate 12.4) 91 suicides occur each day A suicide every 13.7 minutes NM ranks 5th rate 20.1 (n-413) 2 victims of homicide/3 suicides Twice as many deaths due tosuicide than due to HIV/AIDS Highest rates in elderly, midlifemen & young, males, firearms Almost 1 mil attempts per year, 25attempts/ 1 completion (youth100-200, elderly 4/1)6 7. Background:Veterans are at high risk for suicideThe VA has identified suicide in Veterans asa priority.The risk for suicide in Veterans is:- higher than for non-Veterans- higher for some subgroups- higher for rural than urban Veterans- highest in subgroups of Veterans withdepression, psychiatric treatmentThe risk in military populations is highest inthe Army and the Marines.7 8. Background: Consider aVA- specific study of suicideRetrospective review,887,859 Veteransreceiving depression intervention in VAmedical centers, found:Significantly elevated rates of suicide:- 48 weeks after hospitalization- 12 weeks after hospitalization for61-80 year olds (highest suiciderate group)- 12 weeks after medicationchanges(Valenstein et al. )8 9. Targeted Intervention:CAMSThe Collaborative Assessment and Managementof Suicidality (CAMS) is an overall process ofclinical assessment, treatment planning, andmanagement of suicidal risk.The CAMS core multipurpose risk assessmenttool is the Suicide Status Form (SSF).The SSF serves as a roadmap for guiding theclinician and patient, providing crucial andcomprehensive documentation.9 10. Suicide Status FormThe Suicide Status Form(SSF) document is used for:1. Assessment2. Treatment Planning3. Tracking4. OutcomesVA purchased rights touse in CPRS template in process10 11. CAMS is ConsistentwithVA Suicide Prevention Strategy(2009)VISN7 & 2 CoE prioritiesDOD, VA, NIMH systematic reviewsBlue ribbon panelNational and VA RecoveryInitiatives2012 National Strategy for SuicidePrevention (DHHS)Objective 7 training providers inassessment and management11 12. Empirical Support forCAMSCAMS is used in multiple settingsCore SSF assessment aspects &quantitative propertiesestablished, qualitative support6 published correlational studiesover 25 years supporting feasibility& clinical use of CAMS & SSF withsuicidal outpatients & 1 inpatientpsychiatric studyLevel of evidence-Best Practice12 13. Why is training important?A patients ambivalence about dying is anopportunity for a provider to save a life.A systematic method of managing suicidalitycan assuage the fear of losing a patient.Training can help increase confidence andcompetence and dispel common myths.13 14. Why should I use CAMS?David Koerner, MSW,VA provider/ champion/ early adopter ofCAMSI have always considered it a privilege to beallowed into the life of an individual in crisis.For me, one of the most challenging clients isthe person who can no longer find a reasonto live. Personal experience has shown thatthis is a life threatening situation. I havefound the CAMS approach, (and specificallythe SSF tool), to be effective at engagingsuicidal persons and eliciting importantinformation that might help in theirrecovery.14 15. Background:Health Education ResearchU.S. Department of Education(2009) meta-analysis:The effectiveness of eLearningwith blended learning comparedfavorably, and generally led tomore learning than traditionalface-to-face interaction.Mixed studies but little researchevidence for changes in practice15 16. Background:Systematic Reviews of OnlineEducationCook et al. 2008 in JAMAExemplary systematic review51 trials (30 RCTs) 1990-2008compared web-based with othereducational activity or nointerventionFindingsLarge heterogeneity-configuration, blending, presentation, methodsMixed, none or insignificantdifference favoring web-based16 17. Background: Systematic Review ofempirically-supported instructionalmethods in online educationCook et al. 2010a, 10bInstrument measures in 266studiesInteractivity, practice exercises,feedback, repetitionassociated with improvedlearning outcomesInteractivity, online discussion& audio associated withimproved satisfaction17 18. Presentation ObjectivesDescribe the process and outcomesrelated to aims:1) Develop CAMS e-learning includingthe same material & objectives ofIn-person training2) Testing effectiveness of the e-learning compared to in-person &non-intervention control in terms ofprovider evaluation of trainingDescribe offshoots of project18 19. Methodology Multicenter, randomized, clusterthree group design, noninferiority Multivariable modeling strategy toanalyze change inconfidence, beliefs, and practice Pilot delivery to assess providerevaluation and improve training Formative evaluation of facilitatingand inhibiting factors of the process19 20. Approval IRB Medical University ofSouth Carolina/VA VA Office of Research Site specific VA IRBsIn hindsight - We WISH weused VA Central IRB20 21. Benefits ofParticipation ARMs 1-2: 2 Training Deliveries CAMS Training 6.5 hours of CEU credit biweekly telephone coaching calls CAMS manual ARM 3: Control Emergencies in Mental HealthPractice book21 22. Risks of ParticipationMay experience: discomfort due tocontent increased anxietydue to performingnew interventionsand review of patientrecordsConfidentiality risk inall studies22 23. Participant EligibilityOutpatient mental healthproviders-psychiatrist, psychologist,APRN, socialworker, case managers(included RNs)No previous CAMS trainingInformed consent 23 24. Implementation Time-Line24 25. Empirically-Supported Instructional Methods(Cook et al., 2010; Kok et al., 2004; Means et al, 2009; Williams et al., 20011) Evidence-based intervention strategies Interactivity and engagement (video clips) Practice exercises (patient cases) and repetition Blending Behavioral journalism Computer tailoring Simplicity, ease of use Organization by modules 24/7 accessibility and platform-independence Anonymous and self-paced Visual attractiveness and appeal Individuation Resources for help and feedback Instructor-direction Auditory information modeling25 26. eLearning DevelopmentIterative process with multiple pathsand revisionsEarly stages In-person CAMS and Moodle (platform)trainings for study staff Balancing CAMS research & How to doCAMS Transcripts of In-person training Use of Jobes (2006) manual to informcurriculum Guidance of education and technologyexperts (development of modules, Moodlecapacity, use of web site)26 27. e-learning DevelopmentVideo segments27 28. 28Example Veteran-specific Vignette 29. eLearning DevelopmentProduction stages Development of scripts formain video & 2 vignettesreflecting diversity & shortintroductions One day filming of DaveJobes and Keith JenningsBarrierDelivery in first siteunderscored problems andlimitations29 30. eLearning DevelopmentLate stagesMajor revision ofeLearning curriculumEnsuring simplicityand adding artisticappeal30 31. Barriers in Development Microphone problemsduring filming Subtitles developed Technology issues withbandwidth Multiple compressionattempts in order for videosto download Consultation with VISNtechnology group31 32. Barriers in DevelopmentLimits of file sharing Large amount of filegraphics & security issues(burning of DVDs, thumbdrives)Development of dedicatedshare drive32 33. Barriers in DevelopmentRemember:Great Minds DontAlways Think Alike!Multisite-culture, IRBMultidiscipline-unionsTravel to sites for training33 34. In-Person vs. e-LearningBoth: 6.5 CEUsthe Suicide Status Form (SSF)The CAMS Approach to Suicide Risk AssessmentCAMS Intervention (Problem-Focused Treatment)in-Person:AM & PM sessionsCAMS research studiesCAMS in college population and militaryEthics/Malpractice and Next StepsE-Learning:Veteran specific4 modulesCAMS video segments with VeteransVideos illustrating Veteran diversityVA Suicide Prevention Strategy module34 35. Tick-TockGross underestimate oftime for eLearningdevelopment : Projected- 6-12 months Actual- 15 monthsReality of chartabstraction-permissions,complexity, timeNo cost extension35 36. DisseminationBarriers- CEUsVA approved In-PersonCAMS brochureNew & unclear processfor e-learningGuidelines changed inprocessChange in personnel atTMS36 37. DisseminationBarriers- WebsitesCAMS eLearning training Process for VA platformdelivery lengthy Website independent of VAE-Learning CEU accreditationon TMS website VA VA Training Evaluationsatisfaction eLearning Quiz (SocialWorkers have strictestrequirements)37 38. Provider Recruitment& RandomizationGet Their Attention!Goal 268 providers309 eligible230 (77%) consentedCompleted presurvey to berandomized (220)75-e-leaning, 71-in-person,76-controlIRB requirement in one siteverbal consent38 39. Mother NatureApril 27th 2011 tornado hitTuscaloosa County,Alabama43 died & >1000injuredTuscaloosa VAMC servedas a morguePeople living in hotelsCAMS in person cancelled39 40. Delivery of TrainingClinic blocking 6-8 weeks inadvance4 In-person trainings Tuscaloosa attended anothersite CHS staff attended each trainingE-Learning delivery Available same day as in-person 3 week accessibility extended40 41. Delivery:Coaching ComponentThe Purpose:Determine CAMS implementation& increase disseminationThe Format: VANTS call with Dr.Jobes 6 Bi-monthly hour sessions(lunch & learn) Multiple email reminders78 % had NO attendees41 42. Learning Measures CAMS Training Surveys Pre-training Post-training 3 month Follow-up Measures 10-15 minutes(Adapted from Jobes, Knox & VISN2 CoE)42 43. CAMS Survey ItemsEleven Items Competence Reactions Beliefs Motivations Practice & CAMS Delivery mode-satisfaction &preference Demographics43 44. Adoption factors:Focus GroupsProvider experience Impression of trainingexperience Experience in delivery Organizational incentives &rewards for implementation Communication of relatedorganizational goals Organizational & facilitatingfactors or barriers Implementation success Compatibility with professionalbeliefs, values and practices Fit with workflow and program44 45. Providers were primarily midlevel, females, 40-49 yearsRates of completion of training similar betweentraining conditions and sites (one lower)75% rate of completion of training (3modules/sign out) by disciplinePsychologists highest rateProviders career experience with suicidalpatients32% lost > 1 patient due to suicide75% treated > 100 suicidalpatients8% NEVER treated a suicidal patient.Findings: Provider and SiteProfile45 46. Lessons Learned Creative recruitment-walking the halls Identify people/site early for productreview Build in a formal pilot site andparticipants Know VA technology Plan for unexpected-weather barriersand site withdrawal Leadership support-ACOS and SPC Early birds more likely to complete Low cost-benefit ratio of coaching46 47. Interesting Find472 Separate Focus Groups: E-learning & inpersonParticipants experience -liking bothtrainings, using parts of CAMS, CAMSsimilarity to VA Safety PlanBarriers-time constraints, other requiredclinical protocols/processes, few patientsqualifying for CAMS, experience of beingrusty due to infrequent useRecommendations-use a dedicatedclinician, use in younger Veterans,integrate in other protocols and units(inpatient), and use a SharePoint resourcefor training 48. Finding: Satisfaction48VA Evaluation of Training TMS changed twice during deliveryperiod TMS provided results by condition &site This limited us to descriptive statistics Collapsed into 7 themes by consensusof 2 raters Overall Content Objectives Job impact Enablers & barriers Logistics Environment 49. 7.9% 7.7%3.2% 1.3%88.9% 91.0%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral Agree9.1% 9.7%3.0% 0.0%87.9% 90.3%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral AgreeContentThe scope of the material was appropriate to myneeds.I found the material in this learning activity to berelevant and up-to-date.The content was relevant to my job-relatedneeds.ObjectivesOverall, I fully accomplished the learning activitysobjectives. 50. 8.3% 8.1%13.3%21.0%78.3%71.0%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral Agree7.4% 4.5%10.1%7.7%82.4%87.7%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral AgreeLogisticsI obtained information on the learning activityslogistics (i.e. date, location, time) in a timelymanner.If you required any accommodations for adisability, your request was addressed respectfullyand in a timely manner.EnvironmentThe appropriate technology was utilized to facilitatemy learning.The training environment was conducive to mylearningI found that the technology in this learning activitywas easy to use.Overall, I was satisfied with the use of technology inthis learning activity. 51. 6.9%2.1%3.4%4.3%89.7%93.6%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral Agree6.8% 3.9%5.7%3.9%87.5%92.2%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral AgreeOverallOverall, I was satisfied with this learning activity.I would recommend this learning activity toothers.LearningThe learning activities and/or materials wereeffective in helping me learn the content.I learned new knowledge and skills from this learningactivity. 52. 10.0%4.3%15.0%12.8%75.0%83.0%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral Agree6.7%14.3%23.3% 11.1%70.0%74.6%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral AgreeJob ImpactI will be able to apply the knowledge and skillslearned to my job.This learning activity will help improve my jobperformance.Enablers & BarriersMy manager and I set expectations for this learningprior to attending this learning activity.I feel competent to apply the skills/knowledge Ideveloped during the learning activity.This learning activity aligns with the businesspriorities and goals identified by my organization. 53. VA Evaluation of TrainingWhat we know: Faculty Rating & Participant Satisfactionpositive for both (mostly agree orstrongly agree) Trend for In-person to be rated slightlymore positive & slightly less negative53 54. 1. I dont have anxiety about working with suicidal patients.2. I am confident in my ability to successfully assess suicidalpatients.3. I am confident in my ability to determine suicidal risk level inpatients.4. I am confident in my ability to form a strong therapeuticalliance with a suicidal patient.5. I am confident that I can help motivate a patient to live.6. I can develop an adequate safety/coping plan with patientswho are at-risk for suicide.Strongly Disagree Strongly AgreeCAMS Post-Survey Adjusted Means by Training ConditionPost-Survey MeansSurvey Item 55. 7. I am not hesitant to ask a patient if s/he is suicidal.8. I dont believe that hospitalization is alwaysthe best response for suicidal patients.9. I believe that suicidal patients should take an active rolein all aspects of their own treatment.10. I believe my current practices are sufficient to protectme from liability in the event one of my patients shouldcomplete suicide.11. I am motivated to use what are considered the "bestpractices" in suicide prevention even if it requires me to dosomething different in my clinical practice.* p-values from comparison of least squarespost-survey means from MEMCAMS Post-Survey Adjusted Means by Training ConditionPost-Survey Means*Survey Itemp = 0.040p = 0.029p = 0.003Strongly Disagree Strongly Agree 56. Conclusions-Breaking New Ice The complexity of integratingproduct development, trainingdissemination, and evaluation ofhealth education- bumpy, unpredictable road The gift was our multitalentedteam and collaboration Little known about healtheducation research that includesassessing patient outcomes56 57. Conclusions-Breaking New IceCAMS eLearning appears tobe as effective as CAMSin-person learning57 58. Next StepsPatient Level Analyses 3 Month Survey Followup analysis Assessing high risk flagpatient (>300)outcomesand provider adherencefor one year post training(>10 page abstraction form)58 59. Next StepsDissemination National VA CyberSeminar Feb 2012 TMS & DOD invitations tooffer training nationally DOD Suicide PreventionWorkshop June 2012 VA Nursing Research Day59 60. Off Shoots & NextSteps Systems Improvement project Inpatient CAMS groupTraining nursing staff in group interventions Manuscript on suicide-specific inpatientsafety in review Cost analysis (VA QUERI) Mentoring team to be first author onmanuscripts60 61. Next StepsManuscript, OnlineDevelopment and EvaluationRecommendations Use intervention mapping fordevelopment Develop competencies for CAMS Use Gorrindo Measure of SystemUsability Allow time for iterative processand barriers61 62. Articles 63. Articles 64. ReferencesBagley S, Munjas B, Shekelle P. A systematic review of suicide preventionprograms for military or Veterans. Suicide and Life-Threatening Behavior2010; 40:257-265.Bossarte R, Claassen C, Knox K. Veteran suicide prevention: emergingpriorities and opportunities for intervention. Military Medicine 2010;175:461462.Brenner L, Department of Veterans Affairs, Centers for Disease Control andPrevention, Department of Defense. Self-directed Violence (SDV)Classification System. 2010.Department of Veterans Affairs, Health Services Research and DevelopmentServices. Strategies for Suicide Prevention in Veterans. Washington DC:Department of Veterans Affairs; January 2009.Department of Veterans Affairs. Office of inspector general implementingVHAs mental health strategic plan initiatives for suicide prevention. 2009.http://www.va.gov/oig/publications/reports-list.asp. Accessed July 29,2009.Hawks S, Smith T. Thomas H, et al. The forgotten dimensions in healtheducation research. Health Education Research 2008; 23:319-324.Jobes D. Managing Suicidal Risk: A Collaborative Approach. New York, NY:Guilford Press; 2006.Jobes D, Comtois K, Brenner L, Gutierrez P. Clinical Trial Feasibility Studies ofthe Collaborative Assessment and Management of Suicidality (CAMS). InR OConnor, S Platt, J Gordon (eds), International Handbook of SuicidePrevention: Research, Policy & Practice. Chichester, UK, Wiley Blackwell:2011.64 65. ReferencesMagruder K, York J, Jobes D, et al. Patient and provider outcomes of e-learning trainingin CAMS. EDU 08-424.Health Services R &D, Department of Veterans Affairs.8/1/09-7/31/12.Means B, Toyama Y, Murphy R, Bakia M, Jones K. Evaluation of evidence-basedpractices in online learning: A meta-analysis and review of online studies. U.S.Center for Technology in Learning, Office of Planning, Evaluation, and PolicyDevelopment, U.S. Department of Education 2009. Available at:http://www.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf.Accessed on January 12, 2012.Oordt M, Jobes D, Fonseca V, et al. Training mental health professionals to assess andmanage suicidal behavior: Can provider confidence and practice behaviors bealtered. Suicide and Life-Threatening Behavior 2009; 39:21-32.Report of the Blue Ribbon Work Group on Suicide Prevention in the VeteranPopulation. www.mentalhealth.va.gov/suicide_prevention/Blue_Ribbon_Report_FINAL_June30_08.pdf.Seal K, Bertenthal D, Miner C et al. Bringing the war back home: Mental healthdisorders among 103,788 US Veterans returning from Iraq and Afghanistan seen atDepartment of Veterans Affairs Facilities. Archives of Internal Medicine 2007;167:476-82.Sundararaman R, Panangala S, Lister S. Among Veterans- CRS Report to Congress Reportfor Congress. Washington, DC: Congressional Research Services, Domestic SocialPolicy Division; 2008.Valenstein M, Kim H, Ganoczy D et al. Higher-risk periods for suicide among VA patientsreceiving depression treatment: Prioritizing suicide prevention efforts. Journal ofAffective Disorders 2009; 112:50-58.Williams R, Gatien G, Haggerty B. Design element alternatives for stress-managementintervention websites. Nursing Outook 2011: 59: 286-291. 65 66. Contact InformationJan York, PhD, APRN, FAAN (CO-I)Nursing Researcher, UNM Visiting FacultyMUSC Research Professorjanet.york@va.govElizabeth Marshall, MD, MBAResearch Coordinator, Designerelizabeth.marshall3@va.govKathryn Magruder, PhD, MPH (PI)Research Health ScientistMUSC Professormagrudkm@musc.edu66