LookingBackonClinicalDecisionSupportandDataWarehousing

ClinicalDecisionSupportandDataWarehousingCulturally-DrivenProcessImprovementEnabledbyTechnologyGuestLectureforHealthInformationScience-HINF551UniversityofVictoriaMay2008[00:01]

[DaleSanders]ThanksTyler.Thankseveryone.Thanksforjoiningus.I'lldomybesttomakethisagooduseofyourtimetoday.Thisiskindofanexperimentofsort.Itjustoccurredtomethatitmightbeinterestingtoreflectbackonwhatwethought,oratleastwhatIthought,wasimportant,andIhavetopicalinformationatthetime,andreflectonhowaccurateitwasorinaccurateitwasatthat timeandshouldwebe learning fromhistoryandapplying thatgoing forward. The firstversion of this presentation and lecture that I gavewas at NorthwesternUniversityMedicalInformaticsProgramin2006,andthenIrepeateditagainin2007,andthenthiswasthefinalversionattheUniversityofVictoriaandDenisProttiupinD.C.in2008.

TherearealotofslidesinherewhenIcommittedtodoingthisandthislookback,Iforgothowlongitwas.Thesewereactuallytwo-hourlectures.So,we'llhavetobreezethroughsomeoftheseslides.Ithinkthere'sanorderof80or90slides.ButithasbeenfunformeandIreallylookforwardintheQ&Asessiontohearyourthoughtsandreactionstotheseslides.Sopleasestartthinkinginthoseterms.

MyBackground[01:31]

Okay.Now,thisisnotaslideobviouslythatIhadbackin2008.Justforthoseofyouwhodonotknowme,Iwantedtogiveaquicksummaryofmybackground.IstartedoutasanAirForceCommandandControlandIntelligenceOfficerbackin1983.Totheleftofthatdiagram,Ihavea Chemistry and Biology undergraduate degrees. And you can say that the first half ofmycareer was really spent around National Defense, National Intelligence, specifically in thenuclearworld, nuclearweaponsworld. Imade the jump to healthcare because I could seesomeparallels indecision-making,as I indicated in that secondbulletat the topof theslide.Decision-making in these environments kind of all boils down to false positives and falsenegatives,and thenoptimizingan interventionand response toagivensituation. And thoseparallelsbetweenthemilitaryandhealthcarepre-direct. SoI'vebeeninhealthcareandbeenblessed to be associated with some great organizations, especially that relates to IT and

decisionsupport,andnowI'mwithHealthCatalyst®.Andhopefullybuildingproductswillmakeallofthiseasierforallofyou.

Acknowledgements&Thanks[02:48]

Okay.Nowbacktotheslides.Bytheway,I'veinsertedacoupleofslidesthatwerenotintheoriginal presentation just to contrast and kind of give an update on a particular topic thatreflectscurrentthinking.Soyou'llseeacoupleofmoreslides,likethepreviousonethere.

Atthistime,backin2008,IwasacknowledgingRobertJendersandMatthewSailorsforsomeofthework that theyhad contributed to this fieldofwork, especially aroundClinicalDecisionsSupport& theArdenSyntax. And frankly, I've lost trackwithbothof those colleaguesand Iwouldbeinterestedtoknowiftheyarestilloutthereornot.

Overview[03:30]So,we'llgooversortofthetrendsthatIobservedinpatientinformationsystemsatthattime.We'lltalkaboutsomebasictermsandconceptsindatawarehousing.Thatwasaveryhottopicin 2008. Not many organizations were doing that. I'll go through a couple of case studyexamplesbasedonourexperienceat IntermountainandNorthwestern. And thenwe'll diveintokindofthecurrentstateofthinkingaroundclinicaldecisionsupportatthattime.

InformationSystems:TheThreePerspectives[04:03]So,Ibrokeinformationsystemsintothreelargecategories–Analyticsystemsbeingkindofgoalmeasurement, query and reporting tools, enterprise data warehouses, benchmarking data,aggregating, exposing data. Knowledge systems, which was about organizing, sharing, andlinking information right. So these were tools like document imaging, videoconferencing,collaboratetools.Andthenthetransactionsystemsthatwereallaboutcollectingdata–EMRs,billingsystems,GL.Idonotthinkmuchofthishaschange.IthinkIwouldstillkindofbreaktheworldofinformationsystemsupintothesecategories.Butthebigdifferencenowisofcoursethatwearestartingtoblendalloftheseintoasingleuserinterface.Soyougobackto2008,thesewereallprettyseparate.Therewasatransactionsystem,therewasadatawarehouse,there might be, if you are lucky, a video conferencing system or a collaborative system ordocumentimaging,buttheywereallprettyseparate.Wellnowwhatweareseeingandwhatwearedoingandrightly so isweareblendingallofthesetogether intoasingleuser interface,sothatyoucan interactwiththebenefitsandthefunctionsofeachoftheseformerlyseparateentitiesinIT.

PatientInformationSystemsTrends[05:30]At that time, the trends that I thought were noteworthy, we are moving towardstransportabilityand interoperability. It'samazing thatwearenotany furtheralong. Imeanjust fromexperiment, I tried to get a copyofmymedical record fromahealthcareproviderrecentlyandthatwasoverthreemonthsago.Stillnotveryaffordable.Idonotthink.Notatall. Real-timealertsand reminders,wewerestarting to seemoreof that. I thinkwewouldprobablysaytodaytheyhaven'tbeenallthathelpful.Alotoforganizationshaveturnedthoseoff. ButIthinktheyaregettingbetter. WeweremakingprogressandIthinkwehavemadegoodprogressondata-driventreatmentplanning.Diseasemanagementatthepoint-of-care,Iwouldsaythatwascertainlyatrendthatwewerepursuingverymuchsoat Intermountain. Wearegettingbetteratthatnow. Westillhavealong way to go. Payer-driven data collection and Pay for Performance was just emerging.Qualityofcarereportingirrelativetopayerswasjuststartingtocomeonthesceneandbecomea realhot topic. And I thought transparencyof costwas coming sooner thanwhatwehaveseen.Certainlynotalotofprogressinthatregard.

PatientInformationSystemTrends[06:56]Healthconsumerismwasinitsinfancy.Ithinkwehaveseensomeverysignificantprogressinthis regard and that is going to continue to increase. More demands for transparentinformationaccess.Securityandprivacyatthattime,thisisallpre-HIPAA,oratleastrealtotopre-HIPAA.Wehadcomputerizedpatientrecords.Theywerestartingtoemerge.Allsortsoffederalinvestmentatthattimewasjustbeginningtohappen.Ithinkalotofusfeelthatthattook off under the high tech act, but the reality is incentives and money for EHRs actuallystarted out in the Bush Administration. And then the regional health information networkswerestartingtoreceivefundingatthattimeaswell.Prettyinteresting.Again,thisis2006to2008.Soweare10yearsintosomeofthesetrends.

PatientCareData"Customers"[08:04]I have put patient care data at the center of this diagram then, indicating thatwe had fourgeneral categories of what I considered customers of that data. So the financial customerswereallaboutbillingforthemostpartandcostaccounting.Clinicalcustomersarephysiciansand clinicians. Our third-party payers, insurance companies relative to the financial side ofthings.Andthentheexternalreportingandaccreditation/regulatorykindsofthings.

FunctionalFramework:ElectronicHealthRecord[08:45]ItookashotatdiagramingwhatIthoughtwassortofthefunctionalframeworkofanelectronichealthrecord,andrememberthisis10yearsago,tryingtofigureoutstructurallyhowshouldwethinkabouttheprogressionandtheevolutioninanelectronichealthrecord.Sooverontheleftiskindofthecorefunctions–registration,scheduling,accountsreceivable,allthosesortofthings.Youareinanadvantageouspositionasanorganizationifyoucouldtrackbenefitplans,co-pays,referrals,coordinationofbenefits,riskmanagement,patienteducation.Atthattime,youwouldhavebeenasignificantdifferentiator ifyouhadencounterdocumentation,chargecapture,diagnosticcoding,e-prescribing,allergyalerts,drug-druginteractions,medicalhistory.And then you were really a leading edge if you had messaging and real time collaborationbetweenthecareteammembersofthepatientandpatientfamily,ifyouhadapatientportal,self-scheduling,self-registration,accountmanagement,resultsandhistory,pharmacyrefills,e-prescribing, credit card payment. Those were leading-edge things. Kind of interesting, weprogressed in some of those pretty well. Not so much in others, especially around self-schedulingandself-registration.Andofcourse,thereisstillalottodebateaboutwhetherweshould givepatients access to their own resultswithout first being filteredby their clinician.Andthenyouareofftheedgeintermsofcapabilityifyoucouldoffermeaningful,maintainablepoint-of-caredecisionsupport.

Underneath all of this was analytics, I thought at that time, business intelligence, pay forperformancemetrics,nowworkflow,and then regionalandexternalentities thatwe,at thattime,hadtostarttryingtofigureouthowweweregoingtosharethisinformationwiththeseexternalentities.SoIthink,youknow,reasonablyaccuratedepictionhowthingshaveevolvedlookingbackonit.Idonotknowthatwehavemadeasmuchprogressasfastasweshouldhavebutnottoofaroffbasewiththesethoughtsabout10yearsago.

TheFutureEHRUserInterface[11:22]I proposed at that time what I thought a future EHR user interface would look like, notnecessarilygraphicallybutfunctionally.Sotherewouldbepatient-specificdata,muchlikeourcurrentEHRs,youknow,tellmeaboutthispatientinthisencounter.Wewouldhaveadiseasemanagementdatadisplayedinthatuserinterface. Tellmeaboutmanagingpatientslikethis.HowamIdoing,whatshouldIbedoing.Finallythereweretreatmentoptionsaboutpatientslikethat–wherearemyoptionsfortreatment ispatientthat isbasedondata,whatarethemost common tests inmedicationsordered forpatients like this. Iwouldalso, at that time,believe that we should be exposing folks to cost of care. Sowe had those discussions and

raised thatawareness. And thenat that time I thought it is important thatweshareclinicaloutcomesdata in thatsameuser interface just togetan ideaonhowsatisfiedpatientswerewiththesetreatmentsandinthiscontext.Andatthattime,Icategorizedoutcomesaspatientsatisfaction.Ithinkwenowhaveadifferentviewofthat.Butforthemostpart,Ithinkthisisstillwhereweneedtoheadwiththisblendofanalytics,algorithms,andtransaction-baseddataallsupportingclinicalqualityandcostcontrolandmakingiteasierforphysicianstodotherightthinginformedbydata.Ithinkthisstillholdsup.

ClosedLoopAnalytics[13:02]ThisisthefirsttimeIactuallystartedscribblinglikeIdomorecommonlynowadays.IscribbledauserinterfaceherethatwaslargelybasedaroundtheworkthatwedidatIntermountain.Andbytheway,Iwouldliketogobacktothatpreviousslide.

TheFutureEHRUserInterface[13:21]This isnotanythingthatuniquelycameoutofmybrain. I justpatternedthis functionaluserinterface around the success that I saw at Intermountain with some of our clinical decisionsupportmodules. Andthemostadvancedmoduleswehadatthattimeinthehealthsystemsatisfied almost all of the criteria that I am describing here. So I was just borrowing frompatterns thatprecededmeand the (13:51) that Iwroteonat Intermountainandsaying,youknow what, in general, for commercial EMRs, we need to follow these patterns that areequationsthatIntermountainclearlypreferredandvaluedinthehealthsystem.

ClosedLoopAnalytics[14:07]So takingall of that, then I sketched thisout and I said, youknow, the futureneeds to looksomething like this, with patient-specific information on the left about this encounter, thencomparativeanalyticsaboutpatientslikemewithacostofcarecomponentaswell.Andthatwaswhen I started calling this closed loop analytics. So thatwe are taking that transactioninformation about patients on the left and we are matching it against patterns of similarpatients inthebackgroundandclosingthatallbacktothepointofcare. I still thinkthis isavalidwaytodescribeingeneralwhatwearetryingtodo.ImightalsonotethatthislookslikewhatamountstotheTripleAim.IdidnotcallittheTripleAim. CredittoDonBerwickfor labelingitassuch. Butthis isessentiallytheTripleAim. It ispatient-specificinformation,diseaseandpopulationmanagement,andeconomicsofcareallinthesameuserinterface.

'ClosingtheLoops'onClinicalOutcomestoOptimizeQuality[15:14]ThisisaslidethatIinsertedtoshowkindofcurrentthinking.IamalongwithmycolleaguesupinCanada,CorinneEggert,KenMoselle, andDenisProtti,wehaveproduced thismodel thatlooksalittlemorecomplicatedthanitis,butwearenowdescribingclinicaldecisionsupportinthree simple loops. The decision support that you need to support population health, thedecisionsupportthatyouneedtooptimizeandunderstandandapplyprotocolsforsubsetsofthat population, and then the decision support that you need to improve a very specificpatient'slifeinloopA.Andoneof the things that Iwant tomentionhere is everyonce in awhile, Iworry thatwebelievethatsomehowpopulationhealthisgoingtotrickledown.Itislikegoingbacktotrickle-downeconomics. That somehowour focusonpopulationhealth is going to trickledown topatients. But I would argue the complete opposite. The strength of Intermountain, forexample, has always been the application of really personal care in loopA. And populationhealthstartsonepatientatatimeanditrollsup.Populationhealthdoesnotrolldown.Itrollsuponepatientatatime.Andeverynow,again,IthinkthatwearealittlebitlostonpopulationhealthandweareoverlookingtheimportanceofverypersonalizedcareatloopAandwhatwecandowithtechnologyforbothpatientsandclinicianstomakethatloopmoredata-drivenandmoreeffective.

EnterpriseDataWarehousing[17:06]So,atthispoint,Iwentintodatewarehousing.Again,thiswasaveryearlytopicatthattime.Notmanypeopleintheindustryweredoingit.

Multiple,CollaborativeOrganizations[17:17]This was the diagram that I used to describe what we were trying to achieve with datawarehouses. At Intermountain, we were not so much about multiple, collaborativeorganizations, sharing data. We were pretty much hospital X but having kind of a singleorganization. But we actually had over 20 hospitals and some in the neighborhood of 120clinicsatthattime.Andwewerehavingachallengeorganizingandputtingallthatdataintoasingleperspectiveonpatientcareandlookingacrosstheentiresystemtoidentifyhowweweretreatingpatientsofparticulartypes.So, we have this history in healthcare of having very disparate information systems allsupporting these different functions – billing and accounts receivable, claims processing,patient perception of outcomes, results and outcomes, encounters, orders, procedures,diagnoses. AllofthoseinthepastwereverydisparateinformationsystemssometimesgluedtogetherwithHL7. With theadventof themonolithicsystems likeEpicandCerner, that thedisparate vendors are not quite the sameproblem. The data integration challenges are notquiteasbadastheywereatIntermountainandthenagainintheearlydaysatNorthwestern.Buttheyarestillchallengingtoday,especiallyinthemodelthatweseenow,whichIthinkthisdiagramaccuratelypredictedwhatwasgoingtohappen,andthatismorecollaborationacrossorganizationsandtheinescapableneedwhenyouaretryingtocollaboratetoconsolidatedata

acrossthosemultipleorganizations.Imeanthisisifyoukindofstepbackandlookatit,thisisessentiallyanACO.Differentgovernancestructures,differentITsystems,samepatientstryingtounderstandhowyouaretreatingthosepatients. There isreallynootherwaytogoaboutthatunderstandingthantoimplementadatawarehouse.Now, the concepts around data warehousing and our practice of that, the patterns of datawarehousingcertainlyprovedandchangedbutthegeneralconceptisstillthere.

Sanders'HierarchyofAnalyticMaturity[19:36]I published at this time what I call the Hierarchy of Analytic Maturity from basic businessreportingtowhatIcalledreal-timeanalyticfusion,blendingpatient-specificdatawithgeneralpatienttypedata.Otherphysicianswhosawpatientslikethis,orderedthesemedicationsandtests,andwhatweretheoutcomes.And one of the reasons I felt it is important to put this slide together is I saw a lot oforganizationsgrabbing for thebrass ringof thatbottombulletwhentheystilldidnothaveagoodhandleonbasicbusinessreporting,compliancereporting,accreditation.Andinfact,thiswasareflectionofourearlyjourneyatIntermountainwhenwebuiltthedatawarehouse.We

were a little bit enamored with things in these last two bullets and we were spending aninordinate amount of time in a patchwork sort of way addressing the needs of JointCommissionreporting,forexample,orSTSreporting.Sowewerespendingtoomuchtimeininefficiencyattheseotherlevelsbecauseitwasmoreinterestingtoworkatthesehigherlevelsofmaturity.SoweactuallypausedtheiranalyticstrategyatIntermountainandsaid,look,wehave got to clean up and make it more efficient to get these mundane sort of utilitarianreportingoutofintheway,sothatwecanfreeupthoseresourcesandthoseskillstoworkonthishighervalue,moreinterestinganalyticsatthehigherendofthis.SothatwaskindofthebackgroundthinkingIhadwiththishierarchy.

HealthcareAnalyticsAdoptionModel[21:15]Thateventuallymergedacoupleofyearsagointothismodel,whichDenisProtti,DavidBurton,and I published in, I believe,2011or2012. And it isourattempt to create kindof a coursecurriculum,aprogressiontowardsanalyticmaturityandawaytomeasureyourmaturityonthisscale. We borrowed purposely from the EMR Adoption Model, the HIMSS EMR AdoptionModel. Andagain, themessagehere is that thesearekindof thesteps thatyouhave togothroughandyoushouldgothroughandyoushouldgothroughinordertoachievelevel8.And

ifyoutrytojumpuptolevel8,it'slikeafreshmanPhysicsstudenttakingbarmechanicsinthefirstsemester.Itisgoingtobearealstruggle,andinfactyouareprobablygoingtoeventuallyhavetogoback,startoverandgetthefoundationandthefundamentalsoutoftheway.Soifyouprogressthroughthisinaveryrecipe-likefashion.Thisisanevidence-basedwaytoachievelevel8ifyoudeliberatelygothroughthisAnalyticAdoptionModelthatweproducedafewyearsago.

VerticalandHorizontalStrategy[22:40]At that time, I proposed a vertical and horizontal strategy around decision support andanalytics.Now,Idividedituphereintostepone,focusingonclinicalexcellenceprogram.So,prettymuchalongservicelineandthisis,again,areflectionofwhatwedidatIntermountain.Andthensteptwo,focusingontheoperationalexcellenceneededfortheseservicesacrossallof these clinical programs. So,what canwedo tomake labmore efficient, pharmacymoreefficient, radiologymoreefficient. And thenhowdowe reflectbetteranalyticsanddecisionsupport in each of these clinical programs areas, taking kind of a process improvementperspective. My thoughtson thishaveevolved. I amnot so sure it is a good idea tobreak

theseclinicalexcellenceprogramsdownlikethis. ImeanIthinkyoustillcanbutthere isthedangerinkindofsiloedoptimizationifyoudonotthinkaboutthesenowintermsofpopulationhealthfromstarttofinish.Thesetendtobekindofveryfocusedonasnapshotofcaredeliverytoapatient,asopposedtoanentirepatient'slifeandepisodesofcare.SoIthinkthereissomedanger inbreaking theworldup into this. There is still somevalidity to itbut it isnotquitegoodenoughtoaddresspopulationhealth,Idonotbelieve.

ExamplesofClinicalGoals[24:23]So thesewere some of the examples of clinical goals thatwe established. Again, very datadriven at Intermountain. And then we took some of these and incorporated the same atNorthwestern. So, decreasing the number of elective inductions by 50 percent. These areboardlevelgoals.Keepthevariablecostofdeliverieswithoutcomplicationsto5.73percent.Imean incredibly specific, right? Diabetes, LDLmanagement. Diabetes, glucosemanagementratherinICUpatients.Post-surgeryradiationtherapyprotocolis100percentforbreastcancerpatientswithpositivenodesandtumorsizesindicatedthere.So these are examples of kind of clinical goals that we have raised and supported from ananalytics and a clinical decision support perspective at Intermountain. And Iwould say that

Intermountainwas incrediblyvisionary inthisregard. Istilldonotseethis levelofspecificityand commitment at the board level to clinical process improvement and clinical outcomesacrossthecountry.

DOQ-IT/PQRIExamples[25:48]

At that time,DOQ-IT,which I thinkemergedaround2005,as I recall.PQRIwascomingout.Thoseweretheearlydaysof federal incentives forcomputerizinghealthcareandthoseweretheearlydaysofdevelopingwhatweseenow,whichiskindofrunawayprocessmetrics,andIsaythat inadiscouragingwaybecauseIthinkwehavetakentheseprocessmetrics,dictatingthephysicianhowtheyshouldpracticecare insteadoffocusingonoutcomestoawholenewlevel of chaos and I would lobby a bit now as Macro evolves that we put emphasis onmeasuringoutcomes insteadofsomanyprocessmeasures. It iskindof interesting inHealthCatalyst® howmuch timewe spendwith clients on this process of caremeasures thatmayormaynothaveanythingtodowithoutcomesandtheyaresomewhereintheneighborhoodof 2000 now quality measures in the US Inventory and less than 7 percent of those haveanythingtodowithclinicaloutcomes.Therestismeasuringwhatcliniciansdo,worryingmoreaboutthemeansthantheendresult.So,alltheseshouldunionizeinoppositiontothattrend.

RequiringMoreThanaMetric[27:17]Iborrowed this slide fromtheAdvisoryBoardCompany. Ihighly respect theAdvisoryBoardCompanyat that timeand still do. Theyhada goodwayofdepicting that it ismore thanametric.Andso,theytookDVTrateasanexampletodepicthere.Patientsatrisk,whatareyougoingtodo,prophylaxesandthingslikethat,whathaveyounotdone.Walkingthroughsortofthethinking,theclinicaldecisionmakingassociatedwiththatandthemeasurementthoughts,butthenhowyouoverlaythatwiththedatathatyouhave.Imeanjustveryinsightfulontheirpartbecausewhat Iseea lotof times isunfortunatelywepursuemetricsandthingswithouttying tometrics back to these kinds of clinical thinking, number one. Number two, a lot oftimeswewillpursueaprocessimprovementinitiativeandnotappreciatethefactthatwedonothavedataorthedatawehaveismushyandinaccurate.So,youhavetothinkaboutthesetwoenvironmentsinthesamecontext.AndIwouldsaythatthis stilldefinitelyapplies to today'sworld. I amparticularly concerned in thepatient safetyarena.Iseegrowingemphasisonpatientsafety,whichIthinkisgreat.Buttheproblemwiththat in terms of trying tomeasure it is the data associatedwith the root cause events thatcauseapatientsafetyeventisstilllargelymissing.WereallydonothavewhatIcallthatdark

matter data aroundpatient safety events. We just donot have themeans and the tools tocollectthatdatayet.So, Iworry a little bit thatwe are attempting tomeasure somethingwithour patient safetyemphasis that is rampingupat the federal levelwithoutagoodstrategy toaddress thedatathatisneededforit.

RollingOutWhat'sEasilyAccessible[29:21]TheAdvisoryBoardCompanyagainhadgreatinsighthere,whichwasletusfocusonthedatathatwedohaveandwhatcanwemeasurethatwillgetuspartiallydownthewayofprocessimprovementandtrytodosomethinginthefirst90daystodoso.Thenovertimewewilltryto figure out how to expand the data content that we need to improve the accuracy ofmeasuringthatDVTprocess,whatdoweneedtodoaroundinstrumentationofpatient,whatdoweneedtodowithintheEHR,whatisthemissingdata,andhaveaformaldataacquisitionstrategytoroundouttheprecisionofunderstandingasoutlineintheclinicalenvironment.So,very insightfulon theirpart. Still completelyapplicable,but I seea lotof times this conceptbeingoverlookedandunderappreciatedintheindustryrightnow.

TacklingOneProblemataTimeFocusingonaFewKeyMetrics[30:23]Theyalsohadgreatadviceontacklingoneproblematatime.AndIlikethisdiagrambecauseitiskindof interesting. Itreflectswhat Icallconferenceroomanalytics,which isokay,butnotquite good enough. Conference room analytics is that in those three loops that I describedearlier–populations,protocols, andpatients. Conference roomanalytics tends todominatetheenvironmentrightnowandwearekindofstuckintheprotocolsandthepopulationloopsrightnow.Butwehavetomoveanalyticsanddecisionsupportdowntothatpatientloop.Atthistime,therewasreallynohopefordoingthatrightbackin2008to2006.Soconferenceroomanalytics,asindicatedinthisdiagram,wastotallyappropriate.Noproblem.Butthekeymessagehereandisstillapplicabletodayisthatfocusonafewkeymetricsatatimebecauseitisverytoughtobiteoffmorethantwoorthreeofthesemetricswhenyoustartplottingoutaprocess or other clinical improvement initiative. So, it is really hard to move from theconferenceroomtotheimprovementlevelifyoutrytotakeontoomuch.Overtime,asyoubecomeaccustomedtothisculturallyandhowtospinuptheseteamsandhowtoreviewthedataandgovernthedata,improveitsquality,youwillbeabletomovethroughtheseprocessimprovementinitiativesatafasterclipbutinitiallydonottrytodotoomuch.

FixingProblemsattheSourceDataQualityFeedbackLoops[32:04]Another slide that I thought was particularly useful at that time, and that was fixing dataproblemsattheirsource.Idonotthinkthereismucharguingagainstthisnow.Butinthosedays there was a belief that you had to cleanse data before you loaded it in the datawarehouse. AndthankfullyIhadmadelotsofmistakesinmycareerpriortocomingintothehealthcarearena.IwasontothispatternandprincipleyearsaheadbutittookalongtimeforustoappreciatethisinhealthcarebutIthinkforthemostpartweareoveritnow.Youactuallydonotwanttotrytocleanupanydatainthedatawarehouse.Youhavetoidentifyproblemsthroughthedatawarehousewhenyouaggregatedata,youhaveto identifyqualityproblemsusing thedata tools in theEDWasa tool for that,butultimatelyyouhavetogoback to thesource systems and fix these data quality problemswhere they reside. Otherwise, theywillchaseyourtailalldaylonginthedatawarehouse.

ChoosingthePerfectPilot[33:10]AndthenIthinkthisistheslidefromtheAdvisoryBoard,andthatischoosingtheperfectpilot.Andby theway, these slideswereall influencedbymeetings thatwehadwith theAdvisoryBoard aboutwhatwehad learnedat Intermountain. So this is kindof a combinedAdvisoryBoard,aswellasIntermountainperspectivethatyouareseeingintheseslides.So,itisimportanttohavedatastewards.Ifthisisadatamartinthedatawarehouse,youhavetohavephysicianreferrals,someonehastoownthat,supplychain,materialsmanager,isyourdata steward there. So assigning data stewardship to these sources of data is criticallyimportant.Then choosing clear ROI, looking for high variability, processes, is this environment evenmeasurable,isitaknownbusinessproblem,dowehavethetechnicalcapabilityandfeasibilitytocollect thedata,and is therea responsivemotivatedpersonthat iscomfortablewithdatathat can take all of this and drive improvement into the organization. This recipe is stillcompletelyappropriate.Andinfact,Iwastalkingtoaclassyesterday,AdamDillard'sclass,andheasked,"Whatareyouseeingintheindustryrightnowthatisholdingorganizationsback?" AndIwouldsayitmight

be able to summarize it in this slide. Imean there is a couple of others things but you canaggregatedataandyoucanpulldatatogether.Butifyoudonotputtogetheraframeworklikethis,adatagovernance,datachampions,beingverydeliberateaboutwhatyouchoosetofocusonforimprovement,youaregoingtomissthemarkwithanalyticsanddecisionsupport.SothisslideingeneralrepresentswhatIthinkiskindofstillamissingbehaviorintheindustryrightnowthatisholdingthisback.

Structurevs.UnstructuredData[35:02]Okay.Sospeakingofdataandcontentandworkingwithwhatyouhaveandwhatyoudonot,thisismyattemptatthattimetoportraykindofthecomputableanalyticvalueofdatathroughtherepresentationofhumanexperienceandknowledge.Sofromacomputerstandpoint,itiseasiesttounderstandstructuredanddiscretedata.Buttherealityisface-to-faceinaudioandvideoarethebestwayforustounderstandandrepresenthumanknowledgeandexperienceinaclinicalsetting.Sowehavethistension,inwhichwekeepforcingmoreandmorestructureddatacollectiononourclinician.Theyarebecomingourdatasamplersinsteadofourclinicianswhentherealvaluetounderstandinghumanknowledgeandexperienceresidesintheseformsofdatadownhereinthelowerright.

We have not made a whole lot of progress in this yet. I mean you are starting to seetelemedicineandthatkindofthingsthatwearemakingitalittleeasierforthis.Wedonotdonotdoawholelot.Wedonotrecordintermsofaudio.Wedonotrecordaphysician-clientinteraction.Soitisonethingtorepresentapatient-physicianinteractioninanotebutIhavealways thought it might be handed to record that interaction, assuming that all parties,especiallythepatients,arecomfortablewiththat,sothatyoucouldgobackandfill inwherethetextnotemissedthenuancesoftheinteraction.IhavealwaysthoughtthatEMRsareoughttobehaveavideoandaudiorecordingcapabilitythatyoucanreferbacktowhennecessary.Someday,usingnatural languageprocessing,wewouldbeabletoprocessthatnarrative,thatdialogue,thatconversationthattakesplacebetweenthepatientandthephysicianandturnitintosomethingthatlooksmorelikecomputableinformationintheupperleft.

CaseStudyExampleIntermountainHealthcare[37:16]SoletusrunthroughacoupleofcasestudiesatIntermountain.

CaseStudy[37:20]Andagain,Ihavegottobecautiouswithtime.Ithinkweareonlyone-thirdofthewaythroughhere. So we will have to breeze through some of these. One of the early case studies ofanalytic success and decision support success at Intermountain was around our diabetesprogram, and it was driven by this recognition that most diabetic patients were receivinginadequatecare.So,weintegrateddatafromfivedifferentsources,lab,problemlist,insuranceclaims,pharmacy,hospital,coding,andweendedupwinningtheNationalExemplaryPracticeAwardin2002becauseofthat.Theresultsofthis.

DiabetesCPM:KeyIndicators[38:00]These are the key indicators thatwe focused on. A1c, to get that below 7, blood pressuremanagement,LDLmanagement,triglycerides,atleastannualfootexams,microalbumintestatleastannually,andeyechecksatleastannually.Sothosewerealltheclinicalgoalsthatwesetouttoachieve. Forthemostpart, theyareprocessgoals. IshockedsomeonetheotherdaywhenIsaiditdoesnotmatterifyouhavediabetes.Itmattersifyouhavetheconsequencesofdiabetes,thecomorbiditiesandallthethingsdownstreameffectsofdiabetes. Nottosayweshouldtrytomanagediabetesbutwhatreallymattersiswhethertheyareprogressingtowardsthedownstreameffectsofdiabetes.Sothesearestillprocessmeasurestosomedegreebutweallknowthattheydohelpstopthatprogressiontothedeleteriouseffectsofdiabetes.

CaseStudy:DiabetesManagement[39:05]So,whenwestartedthisprograminJuneof1999–again,thisisascreenshotfromtheactualIntermountain,atthattimedifferentlogoandeverything,IntermountainHealthcare'sdiabetesprogram. You can see the trend aroundHbA1c is greater than 9. So this is the number ofpatientsthathadA1cabove9andyoucanseejustadramatic improvementinfiveyearsthenumberofpatientsthatwereoutofcontrol.

CaseStudy:diabetesManagementLikewise,youcanseeadramaticincreaseinpatientsthatareverymuchwithincontrolandweweredoingeverythingwecaninthiscontexttolowertheirriskevenfurtherbygettingthoseA1cunder7.

DiabetesManagementPeerComparisonChart[39:49]Weputall this together inadiabetesmanagementpeercomparisonchart so thatphysicianscould seehow theyweredoingaccording to themselves, the regionwehad. At that time, IthinkwehadsixregionsinIntermountain.Andthensystem-wideandhowtheyweredoingineachofthesedifferentmeasures.Soaphysiciancouldseehowtheywereperformingagainstthesystemandtheregion.

CaseStudy[40:21]Andweenabledthis.Wewouldpushthoseoutindifferentformatsandthingslikethatbutaphysiciancouldcallthatuprelativelyeasilyevenatthepointofcareiftheywantedtoseeit.Soabigsuccesstherearounddiabetesatthattime,andIstillthinkthatmodelprettywellholdsup.Thatisstillsomethingthatweshouldallbepursuingintheindustrytoday.Another success story was around CV dischargemeds and it was just basically the protocoladherence.ItisaroundappropriatedischargemedicinesfollowingaCVevent,ischemicheartdiseaseandMIinparticular.Andso,in1994,whenwerealizedthatthiswasaproblem,andthenweestimatedbecausewehad no data, but we estimated about 15 percent of the time patients were receivingappropriatedischargemedicines at discharge. But after putting a data collection strategy inplaceandorganizingaclinicalprocessimprovementteamandadatagovernanceteam,justliketheAdvisoryBoardslidesuggested.Wewentto98percent.Wehadaharddatatoprovethat.

CaseStudy:CVDischargeMeds[41:28]Andthisisanexample,againascreenshotfromthoseolddashboards.Thisshowsbeta-blockerusage,oneoftheimportantmedsfordischargeofaCADpatient.Ourgoalis90percentofthattimeandthenyoucanseehowouractualperformancecameabout,farachievingthatrightupabove95percentabovethegoal.

CaseStudy:CVDischargeMeds[41:59]Same thing for Coumadin usage. And for themost part wewould present this data in theconferenceroomstophysiciansandwewouldalsopushthisouttothemintheirownpersonaldashboards.Sowewerekindofstuckinconferenceroomanalytics,notquiteatthepointofcareyet,butstillhavingagreatimpactonpatientcare.

TheTangibleBenefits[42:21]Andthisslidesummarizesthetangiblebenefitsofadheringtothosestandardprotocols.Beforeand after implementing this, the adherence to these discharge medications compared thatnationallytobenchmarkin2000,wewerefarexceedingthose.Andthenthisistheimportantpart and we backed into these numbers and you can attribute better adherence to thesethrough actual lives they have reached through these clinical categories. So, that's how ittranslatesintopatientlivessaved.

CaseStudy[43:03]Onemorecasestudy,Ithink,herefromIntermountain.Thiswasaroundourlaboranddelivery–electiveinductions.Again,wesetouttheseveryspecificgoalsatthattime,aroundelectiveinductionsin39-weekgestation.

ElectiveInductions[43:17]And you can see that it drops like a cliff when we start measuring. A lot of this is theHawthorneeffect.Butthatisonimportantpart.Thedataisnotdoinganythinginandofitself.Thedata isrevealingcurrentstateofaffairs,makingpeoplemoreawareofthesituation,andyoucansee thedramaticdrop-off that ithadwhenwe implemented thedatacollectionanddataanalysisserviceswiththis.

ElectiveInductions[43:48]And then you can seehereon this slide, the cumulative savings. Now, the cool thingaboutIntermountain isexecutivesat Intermountainwere incredibly committed tomakingdecisionsthatimpactedthebottomlineinanegativeway.Theywerecommittedtoclinicalqualityandpatient improvementof lifenomatterwhat itmeanttothebottomline. Sotheywere livingaccountablecareyearsbeforeitwaspopular.Andofcourse,theyhadavestedinterestinthatbecausetheyhadaninsuranceplanatthetime,butthisappliedtoeveryinsurancepayerthatweworkedwith then. Weonly insure about 40 percent of our patients at that time. Sixtypercentofourpatientscamefromotherpayersthatbenefitedfromthistoo.Andso,youcouldarguethatthatfee-for-servicemodelthatwewereunderatthattimewerethosethird-partypayers.WetookaheapfinanciallyforthisandthecreditgoestoIntermountainleadershipfordoingthat.

Sofar,sogood…Northwestern'sEDW[44:56]Okay. So, at this time, then I also started talking about Northwestern's EDW. I moved toNorthwestern in around 2005, as I recall, and we wanted to replicate at Northwestern andimproveuponwhatwedidatIntermountain.It'salittleshockyformebecauseNorthwesternis an academicmedical center that operated a lot differently and had differentmotives andgoalsthandidIntermountain.Andso,Ihadanadjustmenttogothroughatthattime.Thereare just differentdatamotives anddifferent things goingon as an academicmedical center,especiallyatthattimethatwerenotacross-overtoIntermountain.So,Iwasalittleoff-basethefirstyearorsothatIwasatNorthwestern,tryingtofigureoutwhatIwasgoingtodowithdatawarehousinganddecision support in this contextbecause itwasquiteabitdifferent inIntermountain.

DataLoadedtoDate[45:47]SowehavedealtwithdatawarehousethereatNorthwesternandthis is justanindicationofthenumberof, youknow, theheappointswehad. Thiswasagainabout2007, Iwouldsay,thesenumbers.Wehadthree,whatisthat,billionrecords.Threetrillionrecords.Terabytes.2.2Terabytes.We jokedabout thenumberof truckloadsofdataand the completeworksofShakespearethatthatequatedto.

EarlyAdoptersandValueoftheEDW[46:17]WehadsomeearlyadoptersandvalueoftheEDWatthattime–Nugene,whichwas,andthisisagainkindof–youwouldnotseemedoinganythingwithgenomicsatIntermountain,butinNorthwestern it was very important. Imean the theme therewas really research – clinicaltrials, research, publications, and that kind of thing. That was a part of Intermountain butcertainlynotthedominantthemeandtheculture.So genomics was an important part of this. We had some really interesting success withphenotyping very early on. Neurosurgery outcomes, especially aroundmovement disorders,created perinatal patient registry for clinical quality outcomes and BMI relationships tocomplications inWomen and Newborns area, deliveries. So you can see some of the earlythingsweweredoinginthisdiagramatNorthwesternandIwouldsaythatallthiskindoftookplacewithinthefirstyearafterwestarteddevelopingtheEDWthere.Sowewerestartingtoseeresultsprettyquickly.

SpecificResearchExample[47:27]There were some specific research examples. Rapid turnaround. In the past the poorresearchersmightnoteverhaveaccesstothiskindofdatabutnumberone,wearegivingthemaccess to data they never had before; and number two, we were doing that in a very fastturnaround.So this is anexampleof kindof a complicatedqueryanddata set thatwe turnedaround tosupport a grant submission. So one of the cool and early values for the datawarehouse atNorthwesternwasthevaluethat ithadtoobtain ingrant funding. Weestimated inthefirstthree years the data warehouse at Northwestern that it directly contributed to about $15million in grant money that we would have otherwise not seen. So again, a very differentmotivethanIntermountain.Nolessimportantthough.

OtherExamples[48:26]Herearesomeotherexamplesforresearchstudies,howmanypatientswithanNSAIDwhohadlowrenalfunction.Thereistheanswertothat.Whatpercentageofpatientsdiagnosedwithmultiplemyelomainremissionoverage18whowereprescribedbisphosphonatesinthepast12months–18percent.Howmanypatientshada1ormorelowejectionfractionwhohavereceivedalowejectionfractionmeasurementwithinthelast180dayswhohavenotbeenseenbyaclinicianandthereisalistofcommissions.Andsoyouarelookingatsituationsthererightwithgapsincare,whereinthiscaseitisgoodforbusinessinafee-for-servicemodeltobringthosepatientsinforfollow-upwiththosekindofhighriskindicators.Itisalsogoodforpatientsandtheir lives. Sothosearesomeexamples. Stillverycurrent. This isthesortofthingthatshouldgoonintheEDWdatawarehouseeveryday,multipletimesadayintoday'sworld.

Examples[49:35]This isareflectionofsomeworkthatDavidBakerdid. IamveryblessedtoworkwithDavidwhenIwasthereasaGIMdocandIlearnedalotfromDavidatthattime.ThesearejustsomequalitymeasuresthatwewerepursuingatthetimeandDavidwasleading,showingadherencetothesemeasuresprimarilyaroundcardiovasculardisease,andtheprogressionwemadeandimprovementswemadefrom2007to2008.Somearedramatic,somearenotsodramatic,andthatwasactuallysomethingthatwerealized–isthattheremaybeanasymptotebeyondwhichyoujustcannotkeepimproving.Theremaybefactorsthatwejustcannotaddressanditcouldbethatifyouarealreadyatthehighlevelofperformanceclinically,youmaynotbeabletodomuchmorethansomeasymptote.Andwherethatasymptoteresides,Ithink,dependsonthepatienttype,theircondition,andespeciallytheirsocioeconomicstatusandsocialdeterminantsofhealth.Youcaninvesttoallsortsofmoneytryingtosqueezemoreandmoreimprovementoutofthatasymptoteandyouaregoingtofinallyreachadecliningreturnoninvestment.AndIthinkthatisgoingtobeoneofthemostimportantthingsgoingforwardinpopulation–iswhat I call return on engagement. And knowingwhen you probably hit the asymptote andfurtheremphasisontryingtoimproveaclinicaloutcomeorconditionbeyondthatasymptotemakes no sense. I think culturally and societally speaking, I think we are going to have achallengethatisinevitablygoingtofaceusinthatregard.

Changesinqualitymeasuresduringthefirst3monthsofthestudy[51:30]This is another study that David supported showing screeningmammography rates, cervicalcancerscreening.Youwillnotseedramaticincreaseshere,andagaintherewasthatawarenessof,wow,itcostsa lottoachievelessthanimpressiveresults. Andthat iswhenIfirststartedthinkingaboutthisnotionofreturnonengagementwiththeNorthwesternasaconsequenceofthisanditissomethingwearegoingtoface.ImeanIamabsolutelyconvincedthisnotiononreturnonengagementwearegoingtofaceinthissociety.

PhysicianPerformance(mostrecent3months)AspirinforPrimaryPreventioninDiabetes[52:11]This is kind of interesting. Wemeasured the adherence to prophylactic aspirin for diabetespatients over time. And this is one we actually embedded the analytics and the decisionsupportbackintoEpic.Soweclosedtheloop.Wemeasuredthisonthebackendandthenweclosedthe loopbycreatingbestpracticesalertsandotherchanges to theuser interfacethatwould remind physicians when they are treating patients with diabetes to prescribe thatprophylactic aspirin. Those of youwhomight recall, during the same timeframe, it becameclearthatifthatdiabeticpatientdidnothaveanyotherindicationsforcardiovascularrisk,thattheriskofprophylacticaspirinwasprobablynotworththebenefit.Andso,therewasachangetoprotocol thatwe implementedandreflected thatback intoEpicandanalyticsveryquicklybecausewehadthedataandthewehadthepathwaytodothatbacktoEpic.

AnticoagulationforHeartFailurewithAtrialFibrillation[53:16]Thesamekindofthingherewithanticoagulationtherapyforheartfailurepatientswithatrialfibrillation.Youcanseedramaticimprovementsherejustbyproducingaresult.

CervicalCancerScreening[53:31]Andagain,kindofreflectingback,thisallstillappliestowhatwedotoday.Thereisstillvalueinthese kinds of things today. There is our cervical cancer screening. You can see dramaticimprovementsthere.

WhyDidn'tthePatientFollowtheProtocol?[53:42]One of the things we started to recognize at Northwestern was this notion of the patient'sengagement in their care. And itgetsback to thatasymptoteandhowmuchyou invest ifapatient is either notwillingor incapableof participating in theprotocol. So therewere 167reasonsinthisparticularcasefornotfollowingtheadviceofpreventiveservice.Wewentback.Nineresulted–actuallythedatawasbad,theyresultedinhavingaservice.Wejustdidnotseeit. Twopatients couldnotafford themedicationand14patients just refused, andof those,zerostarted.So,thisagaingetsbacktowhatcanaphysicianreallybeheldaccountablefor?Ifapatientisunwillingorincapableofparticipatinginaprotocolforsomereason,Ithinktherehastoberiskadjusted. I justdonotseehowwecanholdphysiciansaccountableforthatsocialfactor. Sothe future here I think is risk stratification and you could call it maybe severity adjustingpatientsaccording to their socioeconomicstatusandtheirwillingness toengage in theirowncare. I think that is an inevitable part of where we are headed and I think we areunderappreciatingtheimportanceofthatrightnow,andinfactIthinkitisoneofthereasonsphysiciansaregettingsoburnedoutbecausetheyrealizedtheycanonlydosomuchandyettheyarebeingheldaccountableasiftheycandosomuchmore.

WhyDidn'tthePhysicianFollowtheProtocol?[55:19]So,wealsostudiedwhythephysiciandidnotfollowprotocols.Youfolkshaveallprobablyseenthiskindofthingsbefore. Iwillnotgo intothedetails. Wewillposttheseslidessoyoucanlookatthose. Butitall involvessomereviewwithpeersabout,youknow,letusdiscusswhyyoudidnotfollowtheprotocol,whatarewedoinganddoweneedtoadjusttheprotocolforsomereasonthatwedidnotpreviouslyunderstand.Onlysixofthose147indicatedachangeinmanagementwas required. Some significant resistanton thepartof thephysician to followwhatmosteveryoneelsethoughtwasagoodprotocol.So tome, this, youknow,everyone likes to say,ohphysiciansarekindofa toughculture tochangeandall thatkindofthing. Idonotbelieve it. Physiciansareverydata-driven. Ifyougivethemthedatatodobetterwork,theywilland6outof147whowereabitobstinateisnotindicativeofanentiregroupofpeople.

ClinicalDecisionSupportSystems[56:26]Okay.Clinicaldecisionsupportsystems.Now,wetransition.Weweretalkingaboutanalyticsanddatawarehousing.Now,thiswaskindofwhatisthestatusofallthisgettingtothepointofcare,wherearewe.

ClinicalDSSStructure[56:39]Soatthattime,mythinkingwasthatwehadkindofthreetypesofdecisionsupport.Wehavepoint-of-care,whichatthattimewasamountingtoalertsandreminders;wehadkindofofflineretrospective, what happened? You have seen that in those previous slides; and we haveprospectiveandthatwaskindofthepredictiveworld–whatisgoingtohappen?ThatishowIsaw decision support at the time. I think this still holds up a little bit. I think alerts andreminders,wehavefoundthatthatisnotnecessarilyveryeffectiveoratleastthewaythatweareimplementingrightnowarenoteffective.Thereistoomanyfalsepositivesanditisdrivingphysicianscrazy.

WhereDoesitAppear?[57:25]Andwheredoesitappear?Wheredoesclinicaldecisionsupportappearatthattime?Itwasorganization of data and kind of the "checklist effect", howdo you implement checklist andordersetsandthingslikethat.Therewerestand-aloneexpertsystemsatthattime.Thismight(57:42)caredoctorasanexampleofthat.Wehadtheemergencydatarepositories,likedatawarehouses and mining data to support the retrospectives. And then we were trying tointegrate it intoworkflow through theEMRandCPOE. So thiswaskindofhow I saw thingsappearingatthattime.

TheRevolutioninCDSS[58:02]I also saw it and I was curious to see that I call this a revolution. Phase 1 really was notrevolutionarybut Ithought itwasatthetime. Phase1wasfocusingonqualityandsafetyofcare.Wearebarelyenteringthatphasenowbackthen.Phase2,IpredictedwouldbewhatIcalledeconomicsofcareandareweprovidingcost-effectivecare,notjustgoodcareandis itcost-effective and could we be more cost-effective. And then at that time what I calledgenomicsofcareandIthinktodaythiswouldequatetopersonalizedmedicineandhowarewemaking all of thismore tailored to the individual patient. So that is how I thought decisionsupportevolving.Wehavenotdonemuchofanything,Iwouldargue,inanyofthesephasestoa significant degree in the current EHR environment. Not to say that we cannot. I thinkeveryonehasbeenconsumedwith thepullingEHRs. Nowwehavegot to figureouthowtoimplement some version of these three phases of decision support to help physicians andpatients.

KeyArchitecturalElements[59:13]ThekeyarchitecturalelementsatthattimefordecisionsupportwereanEMRandthecentraldata repository, thedatawarehouse at that time. You could also argue that all theseotherdatasystems,likelab,radiology,etc.,theyarealsoanimportantpartofthatdatacapture.Youhadtohaveacontrolled,structuredvocabularysothatyoucouldseeconsistencyacrossthesesystems. At that timewewere struggling. Wewere trying to figure out how to representknowledge and I think a big lesson learned here is that those knowledge representationvocabularies have essentially fallen by the wayside because they are just too difficult. Andinstead,whatwenowhavethecapabilitytodoisinferknowledgefromdatathatweneverhadinthepastandfromalgorithmsthatweneverhadinthepast.SoratherthantryingtoimpartaknowledgereferenceinpatientliketheArdensyntaxontheworld,thedatacanessentiallytelluswhatthatknowledgelookslike,andIthinkthatisoneofthecoolestthingsaboutwherewearerightnow.

FoundationandRationaleforDecisionSupportModels[60:20]Sothefoundationandrationaleatthattime,itwasaboutmath,mathmodelsandhowdoyoubuildamathematicalmodelaroundthesedecisionsupportenvironments.WewereapplyingalotofBayesianstatisticstothat.Thatstillhappensnow.Wewerealsoapplyingalotofrule-baseddecision-making,IFTHEN.Andwhatwefound,andthisisactuallysomethingIlearnedinthemilitary,thatthoseIFTHENsystemsareveryfragileandtheylargelyhavenotproventobesuccessfulinhealthcareorforthemostpartinanycomplicateddecision-makingenvironment.Andagain,goingbacktowhereweheaded,machinelearningisgoingtogiveustheflexibilitythatIFTHENlittlesystemscouldnotanddonot,andalotofthosemachinelearningalgorithmshave,ofcourse,Bayesianmodelsembeddedinitnow.SoIFTHENisnotagoodwaytohandlerepresentationofhumanknowledge. It is tooconcrete, it is toobinary. Bayesianandotherfuzzytechniquesaremuchbetter.

JustificationforCDSS:MedicalErrors[61:38]ThiswasaslidethatIusedtohelpjustifyclinicaldecisionsupport. Iwasfocusingonmedicalerrorsatthetimeandpatientsafetyevents.Thistopicnowhasre-emerged.IquotedtheIOMreport.ThemostrecentstudythatIthinkJohnHopkinssponsoredandIrecallBMJpublished,those numbers are probably more like 400,000 deaths per year, not 98,000. So we areprobablykillingabout400,000deathsperyear.IfyoureadthatBMJarticle,healthcareisthethirdleadingcauseofdeath.So,Iexpectofnewandgreateremphasisonpatientsafetygoingforwardinthecountryandthedecisionsupportthatweneedtoimprovepatientsafety.Now,thechallengetherewillbe,asImentionedearlier,howdoweactuallygetthedata,thatdarkmatterdata,intotheenvironment.Itdoesnotexistrightnow.Idonotknowexactlyhowwearegoingtodothatyetbecauseitisinherentlydifficulttomeasuresomeoftheeventsandprocedures that lead to a patient's safety event. But this is going to be a big new area fordecisionsupportforusandwehavenotmade,youknow,notmuchsuccessinthisregardoverthelastfewyears. Althoughsomesuccess. Iwillnotcriticizethistoomuch. Wehavemadesomesuccessinthisarea.

Definitions:Whatisanerror?[63:03]Thisiskindofacommondefinitionofaclinicalerroratthattime,orerrorsingeneral,notjustclinical. But error of execution, failure of an action to be completed as planned. Error ofplanning,itwasthewrongplantobeginwith.Therewasanadverseeventcausedbyamedicalmanagementornottheresultofthepatient'scondition.Wedidsomething.Preventablewasanadverseeventattributabletoanerror.Andthennegligentwasexactlywhatitdescribes.Sothatwaskindofmyattempttoputadefinitionaroundwhatisanerrorinaclinicalsense.

ErrorsinMedicine[63:51]Andthenmorenumbersatthattimeabouttheerrorratesinhospitals,justjustifyingthatweneedtospendsometimeonthis.

ErrorsinMedicine[63:59]Again,moredataatthattime.Ithinkthisisprettyoutdated.Itwouldbeinterestingtobouncethisagainstcurrentnumbers.

ClinicalDSS:TheImpact[64:13]Wehadseenuptothattimesome impactwhereclinicaldecisionsupportcouldhelp. InthestudyproducedandpublishedinJAMA,CDSSimprovedpractitionerperformancein64percentofthe97studies.ThatisnottoobadandIwouldliketothinkwecandobetterthanthatnowgiventhedataandthetoolsthatwehave,andIamsurewecan.Butwhatwehavetodoisgetitoutofacademicmedicine.Thatistheproblemwithalotofthesedecisionsupporttools,isthattheyhaveneverreallymadeitoutofthelargeorganizations.ThatwascertainlythecaseatIntermountain.

CaseStudies:ExamplesofCDSSEffectiveness[64:56]And speaking of Intermountain, at LDS Hospital, I was blessed to serve as the Director ofMedicalInformaticsthere,followinginthefootstepsofpeoplelikeHomerWarnerandAlPryorandReidGardnerandPeterHaug.Imeanjustincrediblyaluckythingformetobeabletodothat.AndtheantibioticassistantthereIthinkisaframeworkfordecisionsupportthatweneedtorinseandrepeatacrosseverypatienttype.Ifyouhavenotreadaboutit,Iencourageyoutodoso,butitisIthinkthecasestudyexampleofawesomedecisionsupportinaclinicalsettingthathitsthatTripleAim.Itisdatadriven.Itretainstheabilityforthephysiciantodecideontheir ownwhether they are going to follow the decision support advice or not, butwe hadgreatresultsasaconsequenceofthesenumbersindicatedhere.Sothatwasabigdeal.Iwastalkingaboutit10yearsago.Istilltalkaboutittoday.Istillthinkitisthebellwetherofwhatweneedtodoinotherconditiontypes.

Examples(continued):PreventableADEs[66:03]CPOE implementationat that timewas still very complicatedand very controversial and youcanseethenumbersassociatedwithwhatwesaw.Allsortsofreductionsinerrorsandotherpatientsafetyevents.

Examples(continued)[66:25]Acoupleofotherexamplesofreducing.Wehavedoneatestordering,resultsinsavingmoney.Again,atthattime,alittlecontroversialbecausewewerestillinafee-for-servicemedicineandnotmanyorganizationswerereadytocuttheirchargesby13percentaroundlabs.AndthenapreventivehealthremindersatthattimewithHIVandhowtoscreeningmoreeffectivelythere.

Examples(continued)[66:57]This was a systematic review. The 68 studies, 66 percent of those 65 showed benefit tophysicianperformance,9outof15fordrugdosing,1outof5fordiagnosticaids,14outof19forpreventivecare,mostlycaregaps.So,plentyofexamplesofwhereitcanhelp.

OtherCDSSSuccessStories[67:18]Other success stories that I have been associated with, especially at Intermountain, wasbilirubin management in neonates, ventilator management around ARDS, Coumadinmanagement,andtheseareallveryembeddedintheEMR,bytheway.Thesearepoint-of-caredecision support. High glucose management in the ICU, antibiotic assistant, I mentionedearlier,andthenjustinfectiousdiseasemonitoring.Sothiswasnotconferenceroomanalytics.Thisispoint-of-caredecisionsupport.

MedicalArtificialIntelligenceJustanothertermfordecisionsupport[67:48]Okay. Now,intothereallyrealstuff,atthattime,Iwastalkingtoyouaboutandwecalleditmedicalartificialintelligence.Itisatermthatfelloutoffavorandnowitiskindofcomingbackintofavoragain.I,atthattime,wouldtellpeopleitisjustanothertermfordecisionsupport.

GoalsofAI[68:09]So thegoalsofAI that Iwouldsay,andagain this iskindofanearlydiscussionon the topic,createcomputerassistancewhichachieveshuman levelsof reasoning. Thatwas thebottomline.

KnowledgeRepresentationFormalisms:TheirRole[68:20]Wespentalotoftimeduringthoseyears,bothwhileIwasinhealthcareandthenprevioustohealthcare, trying tounderstandandexpresshumanknowledgeandhowdoes that relate tohowwecancomputerizeit.So,humanknowledgekindofcameinexpresspolicies, institutional,national, local,formulateinterventionsinmedicalpractice,makelocalvariationsinguidelines,andthenprovideallthat"intelligence"toclinicalexpertsystems.AndyouwillnotethatIusedtheterm"expertsystem"herebecauseat that timeexpert systemwas typically associatedwith theArdenSyntax andwith IF THEN rules. And so, I was still kind of stuck in the IF THENmindset, which I havecompletelyjettisonedbynow.Butwearemovingfromnationalsortofguidelinesallthewaydownto,okay,howdoweimplementthisinanexpertsystemordecisionsupportmodule.

FormsofKnowledgeRepresentation[69:20]At that time, these were the forms of knowledge representation that were emerging andexistedinhealthcare.Bayesian/probabilistic,GLIF,theGuidelineInterchangeFormat,thatIdonoteventhinkisaroundanymore.Iamnotsure.Ihavenotheardanybodytalkaboutinalongtime. Case-based reasoning. Weweredoinga lotofworkwithontologies. Decision tables,neuralnetworks,Bayesianbeliefnetworks,theproceduralstuff,againgoingbacktotheArdenSyntax,andtheproductionrules.Thosewerealldifferentformsofknowledgerepresentation.Now, I think the interesting thing is we have seen that these attempts at knowledgerepresentation are very challenging and I will show you some examples of why they arechallenging.Now,weareswitching.Andratherthantryingtoimpartknowledgeandputaboxaroundwhatwe think theworld looks like,we are now collecting enough data and I thinkwe now havealgorithmsandsortofensemblesofalgorithmsthatwedidnothavebeforethatwillallowthealgorithms to tell us how knowledge is being represented, rather than us imparting theseframeworks.So,bigshiftinthatregardandthatisoneofthereasonsIammoreencouragedandoptimisticthanIhaveeverbeenbeforeaboutmachinelearningandAIinhealthcare.

RootsofMedicalAI[70:39]

Sohereisaslidethatdepictskindoftheprogressionofit. Note,Ipurposelyleftthatareainthere. Thiswas inthe late1070s. ThatwouldhavebeenTheodoricofYorktimeframe. Thiswas 1970s, MYCIN at Stanford, focusing on the rules-based decision support for infectiousdiseaseandantibiotictherapies.PUFF,whichwasbasedonMYCIN,whichwaspulmonarydatainterpretation.Again,veryrules-basedandIthinkwehavealllearnedthatrules-basedAIjustdoesnotworkverywell.

RootsofMedicalAI[71:13]APACHEwasoneoftheearlythingsandcredittofolkslikeViShafferthatwereinvolvedwithAPACHEanditwasagreatexample. Pointofcare,realtimedecisionsupport intheICUsforriskmonitoring.

ComputersAreGoodAt…[71:31]

Ialsowouldtalkabout,youknow,computersaregoodatwhat?Theyaregoodcomputationalfunctions–add,subtract,andthatkindofthing.Symbolicreasoningandpatternrecognition.And in particular, whatwe are seeing now, andwe knew for a long time, that nowwe arestartingtocollectingupdataandhealthcaretoappreciatethis.Patternrecognitionisgoingtobe the basis for everything that we do both at Health Catalyst® but also I think in theindustryand it is this ability to recognize patterns in the data that frankly you could neverseewithadeclarative programming language, with the programmer siting down, trying tosee thesepatternsthemselves,youwouldneverbeabletoseethesepatterns.Now,wehaveensemblesof algorithmswe can piece together that show us patterns in data thatwe havenever hadbefore.Sothatisthebasisofthefuture–isthispatternrecognition(72:23)base.

TheArdenSyntax[72:25]IwillnotgointoArdenSyntaxtoomuch.Ifyouwanttostudythat,youcan.Butitwas,atthetime,kindof the leadingwaytorepresentknowledge inaclinicalsetting. ItwasadoptedbyHL7.Idonotthinkthereismuchgoingonwithitanymore.Ithinkithaskindofbecomeabitstale. Again,becauseit issohardtoimparttheseknowledgeframeworksonknowledge, it isjusthardtodo.

ArdenSyntax:Assessment[72:49]Therewereafewvendorsatthattime.IbelieveSiemens,McKesson,Eclipsys.IthinkwearealltakingashotatusingtheArdenSyntaxtoimprovetheirEHRs.Itdidnotgoveryfar.

SupportforArdenSyntax[73:05]Andherearesomeofthose.AtthattimeatCedars-Sinai,Iwasdoingalotofworkthere.AlotofworkwiththisatIntermountainaswell.Ithasnotgoneanywhere.

ArdenSyntax–History[73:17]OhandRegenstrief.IforgotaboutRegenstriefwasinvolvedandthatwasallpublishedaround.Itactuallycameout,Ithink,before1989.SoIthinkthatmightbe–Imightbewrong.

ArdenSyntax–Rationale[73:35]Butagain,itwasanattempttomakemedicalknowledgeavailableatthedecisionmakingandmakingthatknowledgetransportableandverifiable.AndIwillnotgothroughtheseanymore,buttherealityisArdendidnotworkoutandforallthosereasonsImentioned.

PatternRecognition[73:52]Italkedaboutthebenefitsofpatternrecognition. This isgoingtobethefutureforbetterAIand better decision support in healthcare. So if you have not studied pattern recognition, Iwouldbecomefamiliarwiththebasicconcepts.Weareallgoingtobeaffectedbyit.

Wikipedia[74:06]And this is kind of interesting. I mean this still applies up to 10 years later, but patternrecognition basically means, you know, based on either a priori knowledge or statisticalinformationextractedfrompatterns.Soyoualwaysarecombiningatrainingsetofdataandarealdata.Yourunthatthroughasensor,youlookforfeaturestoextractandthenyouclassifythosefeaturesinthepatterns.OneofthethingsthatWatsonhasstruggledwith,forexample,IBMandWatson,isnothavingan adequate training set in healthcare, especially around clinical outcomes to train thealgorithms. So,weallhadtobeawareofthisandwehavetobeawareofwhatvendorsaretryingtosellus.Wehavetoask,dowehavethetrainingsettobounceagainsttherealdata.Andquiteoftenwedonothavethattrainingdatasetyetinhealthcare.Itisgoingtotakeusanumberofyearstobuildupthedatavolumesweneedtoreallygetthehighvalueoutofthesealgorithms.Butwewillgetthere.Iamconvincedtothat.

OtherAIMethods[75:15]At that time, there were genetic algorithms, search algorithms, constraint-based problemsolving.Frame-basedreasoning.Thesewereallthethingsthatwereaffectingourperceptionof clinical decision support. Some of these still hold up. Some have change a lot. I thinkgenetic algorithms conceptually has kind of died in favor of simpler to understand models,althoughtheconceptsarestillinthewayinthealgorithms.Alotoftheseotherskindoffallonother(75:47).Notalotgoingonwithconstraintorframe-basedreasoninganymoreeither.

FrameExample[75:53]Letmegiveyouanexampleoftheseframe-basedreasoningenvironments,andthisisthekindoftaggingthatyouhavetoputaroundknowledge,andatthattime,therewasnootherwaytotagknowledgeandtrytocategorizeknowledgethantodoitmanually.Andsoyoucanseethatitisjustnotscalable.So the framehere isof courseatauniversity. The slot is apatientenrollment. Theclass isstudent.Thecardinalityallowedminimumis2,themaximumwouldbe30.Soyoucanjustseehow difficult it would be to scale this across a large body of knowledge. It is just almostimpossible to do, which is why these machine learning algorithms and pattern recognitionalgorithms are going to tell us a lot of this without having to go through these reallycomplicatedframes.Now, there is some value intellectually in just going through this process, so that youunderstand the problem that you are trying to address. But other than a problem solvingtechnique, it is not a scalable technique for actually implementing decision support in anysetting.

ArdenExample[77:02]Hereisanexample.ThisisanewslidethatIaddedtothisdeck.Bytheway,friends,thiscamefromaJAMIAarticle, inpartpublishedbyaformercolleague,PeterHaug,showingtheArdenSyntax and how you have to tag knowledge and data with this knowledge representationtaggingthatArdenrequires.Itisjustaveryhardthingtoscale.

InSummary[77:27]SoIfinallymadeitattheendoftheseslides.Wow.84slides.Thebigsummaryhere,friends,atthattime,wasthatEnterpriseDataWarehousesandElectronicMedicalRecordsworkhand-in-handtoaddressClinicalDecisionSupport. AndIthinkthatstillholdsuptoday. IstillthinkwehavealottodoatthepatientlevelatthatloopthatImentionedearlier.Butwearegettingbetterandbetteratthisasanindustry.Iamveryencouraged.At that time, I opined that Artificial Intelligence had yet to prove itself scalable beyondinformaticsresearchprojectsandIthinkthatisstilllargelytruetoday.ButIthinkweareonthevergeofseeingarenaissance inAIandmachine learning inhealthcarebecausewenowhavemoredata thanwehad in thepast. Wearestillmissing really importantdata like thesocialdeterminants of health that are so important outcomes andwe are not collecting outcomesdatafrompatientsyet.WithoutthosetwodaysthatitisgoingtobehardforustoachievetherealpotentialofAIandmachinelearning.SoIwouldjustclosetodaybysayingwehavetostartcollectingpatientoutcomesdataandwehave to start collecting social determinants of health data, according to that IOM study thatcameoutacoupleofyearsago,ifwearereallygoingtoderivebenefitfrommachinelearningandAI.

ThankYou![79:00]Thatisit.Wehavegotquestionsanddiscussion?

HealthcareAnalyticsSummit™16[79:02]

Again,thisisaslidethatIhadmanyyearsago.Tyleraskedmetopopthisslideupwhichwasitsadvertising,Ithink,ourHealthAnalyticsSummitinthefall,September6ththroughthe8thinSaltLakeCity.Wehavegotsomereallyinterestingspeakers.DonBerwickwillbethere.EricSiegelwillbethere.Heisagreatpredictiveanalyticsguy.DavidTorchianafromPartners,animportantcolleagueofours,willbethere.Youcanseeallthenameshere.

ImightcallattentiontoonethatIthinkisgoingtobeparticularlyinteresting,theyallwillbe,butthatisAnneMilgramisgoingtotalktousabouttheuseofpredictiveanalyticsandmachinelearning in the criminal justice system. And a lot of people do not know thatwe are usingpredictiveanalyticsandmachinelearningtoinformsentencingandalsoinformwhetherweputcriminalsonparoleandprobationornot.Soitisgoingtobeveryinteresting.

Okay.Thatisit.

Tyler,doyouhaveanythingtosay,friend?Orshouldwegotothequestion?Wedonothaveawholelotoftimeleft.

[TylerMorgan]Well, we do have a couple of poll questions. We have our giveaways for the HealthcareAnalytics Summit™. We would like to be able to do that right now before we get tothequestions,ifwecan.Thiswilltakejustamoment.

SothisisjustacouplepollsthatIwilllaunchuphere.

Are you interested in attending the Healthcare Analytics Summit™ in Salt Lake City?(SingleRegistration) [80:29]

Wehavegot–thefirstisgoingtobethesingleregistrationforthesummit.Pleaserespondifyouliketoattendonthe6ththroughthe8th.Now,becauseofhighdemandandlimitedspace,theseregistrationsmustberedeemedbyregisteringforthesummitbyAugust15thwhentheywillexpire.So Iwill leave thisup for justa fewmomentsmoreand thenwewillputup theteamof3.

AndDale,Ibelieveyouhavegotaccesstolookatthequestionsnow,soyoucanstartreviewingsomeofthose,sowecanjumpintotheQ&A?

[DaleSanders]Yes.

[TylerMorgan]Thattime.Wonderful.

[DaleSanders]Yes,thereisnotmanyquestions.Weshouldprobablygiveitonlyfiveminutes.

QUESTIONS ANSWERSThis is all material and ancient history. What aboutnowandintothefuture?

WellIdonotknowifyoustuckaroundtotheendbutItriedtotouchonthatalittlebit.Andagain,thewholepremise of this presentation today was a look atancienthistory andbounce that againstwhatweareseeing today. So that was kind of the whole intenthere. IguessinthatregardIhitthemarkbutmaybenottoyoursatisfaction.

Requestforslides Yes,wewillposttheslides.Questionaboutthecostfor400,000patientdeaths. Yes,thatisacomplicatedquestiontoaskbecauseyou

wouldhavetoaskthelawyershowtheywouldequatethattosortoftotaleconomiclosstothecountryfrom400,000unnecessarydeaths.

Whereispatientengagementinthisadherence? Wellyes, Ithinkthat istheproblem. Ithinkwehaveto start measuring a patient's willingness andcapability whenwe take them into our care deliveryenvironment. Part of the care management updateprocesshastoincludeprofilingpatientstothedegreethat we can, their ability to participate in their owncare,and theirwillingness toparticipate in theirowncare. Wehavepatientactivationmeasures. Wekindofknowhowtodothat.Wehavethe15verysimplesocialdeterminantsofhealthdataelementsthatIOM,BillSteadmanwasonthatstudy,producedacoupleofyearsago.Soweknowthebasicdatathatweneedtostartcollecting.Wejustdonotcollectityet. Butwehavetostartdoingthat.Thenwehavetostartpullingthatintoourriskstratificationalgorithmsandalsoourcaremanagementstrategies.

You seem optimistic about machine learning andpattern recognition in other areas. Has it reallydemonstrated its (83:27) to curated or structuredorganizationofdataorhavewejustdecideditisgoodenough?

Well we are seeing things that make me moreoptimisticabout this than in thepast. Goingback totheworkwehadat Intermountain, ifyou lookat thethreepatternsthatIthinkweconsistentlyfollowedatIntermountain, it was patients like this who weretreated like this had outcomes like this. And so,wewere developing algorithms right now that help usidentify without us imparting the definition of a

patient type. Wecan identifypatternsofpatients inthe data thatwedonot have to define a priori. So,patients like this pattern, we are nailing that rightnow.Patients that were treated like this, right? How arepatientslikethistreated?Thatisamorecomplicated,lots more variability in that pattern, but we arestartingtomakeprogressonthattoo.And then the last thing who had outcomes like this,thatisthepartthatisfranklymissingbecausewearenot measuring outcomes. But we are seeing veryencouraging results in those first two patterns,patientslikethiswhoweretreatedlikethis,andthenthenextstepwillbehadoutcomeslikethis.

Forexample,canGoogletypesavoidaddingnon-fruitstoitsfruitcategory?

It is complicated, as healthcare is. I think we havesimplerpatternrecognitionproblemstoaddressthanwhatGoogleistryingtoaddress.Iwasreadingabooktomylittlegirltheothernight,'FrostytheSnowman'.Shewantedtoread'FrostytheSnowman'inJuly,andIwasall infavorofthat.Andinthebook,thekidsarebringingtwigstothesnowman,andIsaid,"Whatarethose twigs going to be, Swift? What are the sticksgoingtobe?"Andshesaid,"Thosearegoingtobehisarms."AndIthoughtforasecondandthenIthoughtthere is not a pattern recognition program in theworld that would ever recognize that those twigs inthatcontextaregoingtobearms.Butthatisnotthekindofproblemthatwehavetoaddressinhealthcare.Ithinkourproblemsaresurprisinglyeasiertoidentifytheyhadsignificantimpact.Now, over time, our challenge is we will probablyapproachGoogle.ButrightnowIthinktheyarewithinreachandaddressable.

[TylerMorgan]Okay.WellDale,wedohaveonelastpollquestionwewouldliketoaskeveryone.Wehavegotacoupleminutesleftforthat.

How interested are you in someone from Health Catalyst® contacting you aboutademonstrationofoursolutions?[86:30]

And that is thatwe have hadmany requests in the past formore information aboutHealthCatalyst®, who we are or what we do. Well, our webinars are intended to be educational.Wewould like to be able to let those know, if you are interested in having someone fromHealthCatalyst® reachout to you, schedule ademonstrationofour solutions, pleaseanswerthispollquestion.Iwillleavethatupthereforyou.

It looks likewehavegotanotherquestioncame in. Iwill leave thisup,Dale, if youwant toaddressthatquestion.

[DaleSanders]Ohyeah.ThisisfromVinceVitali.HiVince,goodtohearfromyou.Vinceisalwaysgoodaboutretrospectives too,by theway. "Amazinghowmuch is still relevant. Whyaren'twemakingmuchprogress insomanyareas?(Ormaybeyouare justthatsmart." Idonotthink it isthelatter,butthanks,Vince.Idonotknow.Tobehonestwithyou,Vince,Ireallydonotknow.ImeanIthinktheeconomicmodelofhealthcarehasnotnecessarilydemandedthatwedothis.Ithinktherealityisithasbeenokaytobemediocreinalotoftheseareas.Theeconomicsarechanging.Ithinkwedidnotreallyhavemuchdatabackthen.EHRswerestillprobablyinthe

25to30percentadoptionrate.Wedidnothavethedatasavvyconsumerthatwehavetodaythatisgoingtodemandthatwebecomemoredatadriveninhealthcare.

AndthenIthinkanotherreasonis,youknow,franklyitisveryhardtomodifyourcurrentEHRstosupportthiskindofdecisionsupport.Theyaremadetosupportgeneralpatientcare.Wehadchallenges.WhenIcamefromIntermountainatNorthwestern,whereIwentfromHealthtoCernerandEpic.AtHealth,weownedtheentireEHR,everythingabouttheAPI.Wecoulddowhateverwewantedtowithit.WhenIcametoNorthwesternandtriedtoreplicatethat,itwas pretty hard because Cerner and Epic were developing tools that had to be generallyapplicable across all patient types in the entiremarket. And that kind ofworks against theapproachwehadanIntermountain,whichwasverytargeted,veryspecificpatienttypes,anddata tosupportveryspecificdecisionsupport initiatives.Wewereveryspecific.CernerandEpicareverygeneralandunderstandably.SoIthinkwearegoingtoseeashiftactually,whereIthinkFHIRwillhelpusbecomemorespecific,andIthinkyouaregoingtostartseeingdecisionsupportmodulesthataremoretargetedaroundpatienttypes.

Okay.Iguesswebetter…

[TylerMorgan]Yes. Weareat time. Dale, thankyou somuch for thispresentation. I thankeverybody forhangingonwithus.

[DaleSanders]Yes.

[TylerMorgan]Iwouldliketoremindeverybodythatshortlyafterthiswebinar,youwillreceiveanemailwithlinks to the recording of the webinar and the presentation slides, as well as the Summitgiveawaywinners. Also, please look forward to the transcript notificationwewill send youoncethatisready.

OnbehalfofDaleSanders,aswellastherestofushereatHealthCatalyst®,thankyouforjoiningustoday.Thiswebinarisnowconcluded.

[DaleSanders]Thankseveryone.Haveagreatday.

[ENDOFTRANSCRIPT]