the why and how of machine learning and ai: an implementation guide for healthcare leaders

© 2017 Health Catalyst

Proprietary and Confidential

An Implementation Guide for

Healthcare Leaders

The Why and How of Machine Learning and AI



Eric Just

• SVP at Health Catalyst in Product Development

• Started career in healthcare at Northwestern University

• Genomics

• Clinical Data Warehouse

• Health Catalyst

• Early platform vision

• Sales

• Client Operations

• Product



Ken Kleinberg

• VP of Research at Chilmark Research, covering

• Analytics, AI, machine learning, mobile/wireless, IoT, EHRs, medical

devices…

• Healthcare transformation, PHM, consumerism, payer-provider

convergence…

• 38 years in IT – last 20 in healthcare, including

• Managing Director, Advisory Board

• VP of Bus Dev, Health Language/Wolters Kluwer

• VP and Hospital Strategist, Allscripts

• Senior Director of Global Health, Symbol/Motorola/Zebra

• VP and Editor-in-Chief of Healthcare, Gartner



Learning Objectives and Agenda

1) What Are Machine Learning (ML) and

Artificial Intelligence (AI) and Why

Should I Care

• Getting past the hype to real definitions

• Understanding use cases and benefits

2) How Do I Implement ML and AI in My

Organization

• Practical steps, resources, costs,

timeframes, skills, and vendors

• The importance of data management

and an analytics platform

• Real-time data and moving targets

3) Dealing with Challenges

• Change management, regulation,

liability, human touch

4) What Does the Future Hold for ML

and AI (and for My Strategy)

• Job augmentation and replacement,

taking on greater challenges, scary (yet

bright) futures

What You Need to Know and Do



Executive Summary

• Machine learning and the broader area of AI, inspired by biology and evolution to learn and

adapt, have long histories.

• Use cases in healthcare include improved predictions, patient clustering and classification, early

detection/diagnosis, scheduling, medical image recognition, and natural language processing

(NLP)/text mining.

• Practical steps for getting started with ML and AI include: goal setting, allocating resources,

establishing realistic timelines, costs, and benefits; managing complex data sources; running models;

measuring results and taking action/operationalizing.

• Existing investments in data management and analytics, as well as processes to identify needs and

create operational decision models, give organizations an advantage.

• Typical challenges include change management, regulation, liability, and concerns about human

“touch.”

• The future of ML and AI include increased job augmentation and replacement, tackling more complex

problems (e.g., genomics), and the need to ensure these technologies are applied with the proper rigor

and ethical oversight.

5



What Are Machine

Learning and AI and

Why Should I Care?



Understanding the Hype in Machine Learning and AI• World-leading companies have invested billions into voice assistants, facial recognition, shopping

assistants, and semi-autonomous vehicles.

• There has been a media frenzy over AI and job displacements, disruptions, and dangers – the rise of

the intelligent machines!

• It is difficult to discern the degree of healthcare vendor investments in these technologies and the

sophistication of their solutions.

• To more fully appreciate the opportunities and challenges requires some understanding of what

machine learning and AI are.

7



Machine Learning and AI Definitions

* ML and AI draw and

advance approaches and

principles from traditional

statistics and mathematics,

rule-based systems, and

biological systems to create

solutions that can learn and

adapt.

* They reduce the effort

required by humans to

manually choose algorithms,

data sources, model

parameters, and steps in

order to test hypotheses,

discern knowledge, and

achieve goals.

8

Rules,

inference

engines,

symbolic

reasoning,

programming

Biology, parallel

processing,

genetics, evolution

Machine Learning

and AI

Math,

statistics,

probability

theory,

analytics



Clarification of Definitions

Artificial intelligence refers to systems or machines that can adapt and learn from data and

their environments to exhibit the intelligent behavior found in living things.

Machine learning is a subset of AI with its origins in math, statistics, computer science, and

biology (e.g., neural networks, which simulate networks of neurons in the brain). Its models

adjust themselves when presented with training data. This can be:

• Unsupervised (meet an algorithmic goal, such as clustering data into discernible groups)

• Supervised (the system is given labeled training data and adjusts itself to provide the right

answer)

Deep learning is a more sophisticated form of ML that incorporates multiple layers (e.g., of

neurons) capable of pre-processing and filtering, to model more complex input and

relationships.

Cognitive computing attempts to model human thought processes, including generating

ranked hypothesis.

9



Some Additional Definitions

Expert systems refer to rule-based reasoning systems that either attempt to determine

which possible answers are supported by data (backward chaining) or start with known

information and work towards reaching a viable answer (forward chaining) – rules can be

used to apply known knowledge and context to ML and AI.

Descriptive, predictive, and prescriptive analytics are increasingly valuable approaches

to determine what has happened (descriptive), what will or might happen (predictive), and

what to do about it (prescriptive).

Natural language processing uses a variety of ML and AI techniques to understand

human text or speech.

10



Tackling (Big) Data and Problem Complexity

11

The volume, variety, and velocity of information can far exceed a care professional’s abilities

Social

determinants

of health

Microbiome

Environmental

data

Genomic data

Medical and

family history

Patient-

generated

health data

Population

health data

Resource

availability (care

facilities, clinicians,

medications)

Policy and

regulation

Insurance

coverage

Medical

knowledge

Patient Journey

Digital images,

lab tests



What to Apply Where and WhyApproach Best For Positives Risks

Statistics

• Good assumptions to start with

• Standard models sufficient

• Clean (structured) data

• Skills and mature tools often

already exist in the org

• Acceptance by org

• Techniques may not be powerful enough

for difficult problems

• Models may not transfer well to real and

changing world

• Lots of manual effort required

Machine

Learning

• A hypothesis to test or pattern to

recognize

• Some assumptions to start with

• Many models to try

• May be able to utilize some

existing organizational skills

• Removes some of the manual

burden of model development

• Can identify most impactful

input features

• Models may appear to do well but be

simply “memorizing” training data

• Models may not be sophisticated enough

for more difficult problems

Deep

Learning

• A difficult problem to solve

• Complex interactions

• Messy and/or huge amounts of

data

• A “black-box” answer will do

• May be able to provide answers

or predictions difficult to

discern by any other method

• Skills and tools may not exist in the org

• Users may require more specific

explanation for model results

• Even large datasets may not carry

enough info for the most sophisticated

systems to find insight

Expert

System

• Extensive domain knowledge

available

• Processing order not predefined

• Reasons for answers must be

apparent

• Answers can be traced to

which rules have “fired”

• Organizations can agree on

rules or customize locally

• Too many rules (over a few thousand) can

be difficult to manage/curate

• Has challenges with conflicting data

12



ML and AI Use Cases

• Pattern recognition

• Clustering

• Prediction

• Diagnosis

• Scheduling

• NLP

• Smart network

• Robotics

• Facial recognition

• Consumer buying patterns

• Financial market trends

• Car problems

• Train schedules

• Virtual assistant

• Internet of things (home)

• Assembly line

• Tumor type

• Disease variations

• Length of stay, readmissions

• Cancer care

• Surgery and rehab

• Chart abstraction

• Internet of things (hospital room)

• Medication delivery

13

Type of Goal Cross-Industry Example Healthcare Example



Setting Expectations While Accelerating Exponentially

14

Useful

Too good?

Promising

Essential

No way Geeky/Niche

Advances have a way of creeping up

on us – one day a joke, another day

useful, and all of a sudden, Wow!

We are here

with AISmart

Phones

Self-driving cars

Virtual Assistants

GPS

“Doc in a box”

Human brain

interface

Internet



How Do I Implement

Machine Learning and

AI in My Organization?



Practical Steps

Major Considerations include:

16

Strategy – Ensuring that ML and AI are applied to the rights kinds of problems for the right organizational goals

Resources & Infrastructure –Determining and implementing analytics platforms, data warehouse capabilities, tools to access data, and security required to succeed

Funding –Distributing reasonable costs for ML and AI infrastructure and initiatives across relevant stakeholders

Timeframes and Benefits –Determining how long ML and AI initiatives will take and what are the expected benefits

Skills – Ensuring the rights kinds of skills for project scoping, sourcing decisions, model development, and operational changes

Vendors, Partners, and Tools –Choosing which vendors and partners will be able to provide the tools and skills to help



Approach to Strategy for ML and AI

• A balanced approach supported and visible to executives

and the board, desired or driven by (or they are at least

willing to give it a chance) the departmental level (clinical,

financial), and enabled by IT

• Incorporated into the overall organizational strategy

and initiatives (PHM, ACO, Quality, Consumerism)

• Pursued within the process of operational

governance to ensure priority, funding, ownership,

results, P&L

• Nurtured with a staged and natural progression

growth that considers start-up costs, sourcing

of skills and resources, knowledge transfer,

risk, and scale

17

• Driven totally from the top or

totally bottom up

Good Worrisome

• Considered an IT project with no specific

links to business or clinical initiatives

• Politics, lack of transparency, lack of

accountability

• Big bang, shoot for the moon, high expense



Flexible and Scalable Infrastructure

Matching to an Organization’s Needs and Capabilities

18

Turnkey application

(simpler use, more

reproducible)

Configurable application

(varied needs, controlled

by business user)

Customizable application

(specific needs, pilots)

Typical Adoption Progression

these can be

incrementally addedmultiple approaches

– mix and match

full leveraged

environment to support

many needs

web-based decision

support

ALGORITHM TOOLKIT BI PLATFORMWORKFLOW

INTEGRATION



Real-Time Data and Moving Targets• One of the challenges with any prediction, decision support or analytics solution is its applicability/accuracy as

input sources and situations change.

• Should the system see something it was not trained on, depending on the influence/combinations of the inputs, the

system might be able to come up with an accurate answer, semi-accurate, or it could be completely wrong.

• ML and AI systems, by definition, should be readily adaptive to new training data and new situations.

• It may be possible to monitor/identify different “alert levels” – that is, operating within a data stream thoroughly

trained on, working with a different yet similar set of data, or working with data outside the training range.

• Use of real-time data platforms can help ensure that systems are continually retrained on relevant data for

successful operational deployment.

19

Retrospective

training data

Good for initial

model development

Real-time

training dataNeeded for operational

model deployment



The Cost Realities of Machine Learning and AI

20

Organizations

should consider

change

management and

cultural costs

associated with

implementing ML

and AI

Costs are mostly

incremental for

organizations that

have already

invested heavily in

BI/Analytics and are

using that data to

affect improvements

Organizations that

have not gone this

route will have much

higher hills to climb

The most important

first step is to use

data effectively, even

at a basic level, to

begin making

decisions



Timeframes and Benefits

21

What will you do if benefits are not (completely or substantially) met??

Are you making

interim

measures at key

points?

Do you have

buy-in from

stakeholders

about the

benefits?

Are you

providing a

range of

(compelling)

expected

benefits?

Do you have

baseline

measurements

to compare

results?

Are you setting

short, medium,

and long-term

goals?

Are you

measuring

benefits in terms

of costs, times,

quality, or other

measures

(precision vs

recall)?



Skillset Sourcing for Implementing ML and AI

Job Function Description Internal External

Clinical Identifying uses cases, discerning valuable outcomes, dismissing wrong/dangerous results, rallying support **** *

Financial Recognizing opportunity costs, spreading funding across relevant stakeholders, evaluating ROI **** *

IT Integrating data sources and data warehouse systems, monitoring performance, sourcing/hosting, upgrades *****

Security Ensuring that data sources are properly secured *****

Project Mgmt. Setting realistic timeframes, meeting resource requirements, and conveying progress to stakeholders **** *

Bus Dev Identifying and contracting with suitable partners, data sharing, consortiums, co-development *** **

Marketing Ensuring visibility to achievements for purposes of organizational reputation and patient recruitment ** ***

ML/AI Choosing the right algorithms and modeling approaches, running the models * ****

BI/Data Science Ensuring value add, integration with existing techniques, dashboard design ** ***

User Support Education, training, mentoring, answering questions, forwarding on more difficult problems, tracking concerns *** **

Legal Minimizing regulatory risks, meeting FDA requirements, ensuring HIPAA security, vetting business partners **** *

Operations Minimizing disruption, scheduling, meeting reporting period requirements *****

22



Considerations – Does Your Team Know the Answers?

AI and ML Experience

• Which ML algorithms and AI approaches (or combinations) are you going to pick?

• Do you have experience with descriptive, predictive, and prescriptive analytics?

• Has your team or your vendor any experience (and success) with competitions such as Kaggle?

• Does your team have access to online courses to increase their expertise?

Hardware Resources

• Do you have the processing power (GPUs) and memory to run the models with the amount of data necessary?

• Have you considered cloud and virtualization?

Data Sets

• Are you able to identify and have access to relevant data sources?

• Do you have access to adequate sample datasets?

• To what degree will you be attempting to transform/analyze unstructured data?

Vendors and Products

• Will you be using a turnkey solution, a configurable application system, or an application you have to customize or develop

(and later support)?

• Will you be using a proprietary commercial solution, or open source? If using open source, what support is available from who?

• To what degree do you expect to use or need consulting services?

23



Questions to Ask Vendors of ML and AI

Issue Description Concern

ML and AI

ExperienceHow long has the vendor been using ML and AI?

Do they know how to use it? Can they explain

how it works?

BI Platform Does the vendor have a platform for BI and analytics?Is it a modern architecture built with software

engineering principles?

Healthcare

Experience

Does the vendor have relevant/dedicated healthcare experience (with similar

organizations)?

Will you be teaching them about healthcare?

ML and AI SourceAre the ML and AI capabilities developed by the vendor, commercially available, or

open source?

Proprietary lock-in?

Compelling

DifferenceCan the vendor demonstrate a compelling difference ML and AI make to its solution?

Is it worth it?

FlexibilityTo what degree is the vendor able to switch in new ML and AI capabilities or

customize their solution?

In love with their hammer? – what does their

roadmap look like?

Integration with

EHRs

What approach and success does the vendor have with integrating (or competing)

with major EHR vendors?

Will there be cooperation?

ScalabilityAre there any scalability issues regarding the amount of data, processing times,

database structure (e.g., Hadoop), hardware requirements?

Will you hit these limits?

Business ModelHow does the vendor charge for their software, hosting, and services capabilities

(over time)?

Do they price the way you want?

Self-Sufficiency Does the vendor offer tools or education to help make you self-sufficient?What skill levels will be required to use this

solution?

24



Dealing with Challenges



Typical ChallengesIt would seem that after identifying the use cases and technologies, and then sourcing, developing

and deploying the models, an organization would be home free. As hard as it might be to succeed on

those fronts, there are additional challenges that are likely to be faced that could be more difficult

to address and risk success of the system and worse.

These include dealing with:

26

Change

Management

AI and ML systems may

be discovering results,

exposing weakness,

taking away people’s jobs

and more – often at a

pace faster expected – as

in any transformative

project, a focus on change

management is key.

Regulation

As ML and AI take on

more complex situations,

the may come under

increasing regulatory

scrutiny.

Liability

As ML and AI become

more autonomous, a

question arises as to who

is liable when things go

wrong.

Human Touch

As ML and AI systems and

results touch more of our

lives, how can we ensure

a respectful and

compassionate interaction

while also balancing the

needs of systems and

society?



Change (Disruption/Transformative) Management

• People generally don’t like change – particularly if their job, career, or ego is on the line.

• The bar is not always high enough to overcome inertia – good enough my prevail.

• Intelligent automation often unveils inefficiency, waste, fraud, abuse, bureaucracy.

• Once projects are underway, results may come much faster than previously experienced.

• Fear over job loss can lead to lack of cooperation, feet-dragging, passive aggressive behavior, sabotage, or open hate (Johnson and Johnson’s Sedasys for anesthesiology).

• AI can be risky since the mechanisms are often not well understood, the skills and tools may be unfamiliar, and the costs and benefits can be uncertain.

27

How To Succeed

• Successful change management requires open and transparent communication, a clear vision of the future, instilling

belief that people will succeed under the new system, that there is no path backwards, and that everyone will benefit.

• Successful projects require good planning, adequate resources, leadership, governance, accountability, realistic

milestones, and the right tools and knowledge of how to use them (example, the right partners).

What to Expect



Regulation

• Privacy/Security - Because AI, BI, and analytics

systems are increasingly being used to deal with

complex data with many additional input sources,

security and privacy concerns may be higher

than with other systems.

• Safety – While it is clear that medical devices are

subject to a risk-based framework of regulation by

the FDA, clinical software applications (e.g.,

EHRs), analytic models, and clinical decision

support remain in a gray area.

• Oversight - As systems move from being

reference systems, to providing advice, to taking

(autonomous) action, the reasons for them to

come under regulatory control increase.

28

How To Succeed

What to Expect

• Ensure data operating systems are especially secure and able to manage multiple data sources.

• Understand who will be responsible for regulatory approvals (if any) from the start – recognize that some vendors

may tell you a system does not require approvals when it may.



How To Succeed

What to Expect

Liability

• Crossing Boundaries – It’s one thing for a machine to

carry out a mundane repetitive task, and another for it to

start making decisions that affect health and lives.

• With rule-based systems, the decisions can be traced

to specific rules that “fire,” and the responsible parties

for developing those rules are accountable.

• With ML and AI, it’s possible for predictions, answers,

and decisions to be made without a complete (or even

substantial) understanding of how the result was

reached.

• As ML and AI take on more complex tasks, its only

natural for people to be concerned about “who’s calling

the shots” and to what degree ethics or compassion have

been incorporated in the solution.

• Media attention to such issues as which obstruction

should a self-driving car hit if a crash is inevitable (school

bus or wheelchair).

29

• Health systems should ensure, to the degree possible, AI systems can be transparent to the decisions they make,

traceable to training sets, and accountable to human experts who deploy them – black boxes are a last resort.

• Consider initially picking use cases that provide advisory results or are non-clinical – they may be less “sexy,” but will

present fewer barriers and risks.



Human Touch – Moving Beyond Informing Decisions

One final (future) challenge concern about AI systems as they start to play an increasing role in people’s lives is to what

degree they can exhibit human touch vs. being (perceived as) cold and inhuman

Fortunately, its possible to have them exhibit compassion and understanding, perhaps with more consistency

than some humans, and to be attuned (via facial recognition, voice processing, historical data, etc.) to the emotional/behavioral

state of the user.

Given the nuance and complexity of human behavior, it’s more than easy to get this wrong, especially

if the subject is intentionally trying to derail or game the solution, or exhibits conflicting or new behaviors outside the

experience of the system.

On the scale of population health, care rationing, costs, etc., these become more organizational, cultural/societal issues. We’re

still some time away from machines being on equal footing with humans in deciding courses of action and the future.

30

How To Succeed

• Recognize that how users interact with a system and to what degree that system design shows a concern for

its results will become increasingly important as humans rely on these systems to a greater degree.

What to Expect



What does the Future Hold for Machine Learning and AI (and for My Strategy)?



What Role Will AI Play in the Future?

32

Will depend on your type of job (tech, care manager, nurse, PA, physician) and your training

3-5 yearsToday Future

Minimal

experience

or skills

Most

experience

or skills

Medium

experience

or skills

Assist

Suggest

Annoy?

Super-charge

Assist

AssistSuggest

Super-charge

Replace



Dangers of AI

33

I think someone (or

something) is trying

to target our

genetics

I don’t know how to

do that anymore

There’s no future in

that career

anymore

Which of these two

systems should I

trust?

Over-reliance and

trust

Fewer new

professionals

enter the field

Competing

medical

knowledge

Loss of privacy,

exposure of

vulnerabilities



Advice

• Include machine learning and AI in short and longer-term strategic planning – look to apply these

approaches to gain tactical (labor saving), strategic (new markets), and transformational

(breakthrough) effects.

• Evaluate strategic options for introducing and expanding these technologies into the organization

• Determine to what degree and pace to build internal teams and skills (especially quants who

can identify and translate technology advances to clinical and business use cases), and to

what degree to enlist vendors and external experts as partners.

• Place increasing emphasis on first ensuring a solid BI platform to deal with more extensive and

complex data sourcing and management.

• Address change management – expect to deal with it at an increasing pace – especially in terms

of changing job functions and staffing requirements across the continuum.

• Continue to look for signposts in other industries and healthcare where advances are making a

difference and disrupting existing business models.

34



The Bright Future

35

Awesome collaboration of human and machine

Engineered health – genetics (remove disease)

Significantly reduced healthcare costs

More preventive care and treatment (nanobot repair)

Cyborg-capabilities -– implants (senses, motor function, organs)

Limitless life – brain scans

Greater time for leisure, exploration



Thank YouQ&A with Ken and Eric



Healthcare Analytics Summit 17

Summit highlights

Industry Leading Keynote SpeakersWe’ll hear from well-known healthcare visionaries. We’ll also

hear from two C-level executives leading large healthcare

organizations.

CME Accreditation For CliniciansHAS 17 will again qualify as a continuing medical education

(CME) activity.

30 Educational, Case Study, and Technical

SessionsWe have the most comprehensive set of breakout sessions of

any analytics summit. Our primary breakout session focus is

giving you detailed, practical “how to” learning examples

combined with question and opportunities.

The Analytics WalkaboutBack by popular demand, the Analytics Walkabout will feature

24 new projects highlighting a variety of additional clinical,

financial, operational, and workflow analytics and outcomes

improvement successes.

Analytics-driven, Hands-on Engagement for

Teams and IndividualsAnalytics will continue to flow through the three-day summit

touching every aspect of the agenda.

Networking and FunWe’ll provide some new innovative analytics-driven

opportunities to network while keeping our popular fun run and

walk opportunities and dinner on the down.

Sept. 12-14, 2017

Grand America Hotel

Salt Lake City, UT

ERIC J. TOPOLAuthor, The Patient Will

See You Now and The

Creative Destruction of

Medicine. Director,

Scripps Translational

Science Institute

DAVID B. NASH,

MD. MBADean, Jefferson

School of

Population

Health

JOHN MOOREFounder and Managing

Partner, Chilmark Research

ROBERT A. DEMICHIEIExecutive Vice President and

Chief Financial Officer, University

of Pittsburgh Medical Center

THOMAS D.

BURTONCo-Founder, Chief

Improvement Officer,

and Chief Fun Officer,

Health Catalyst

DALE SANDERSExecutive Vice

President, Product

Development,

Health Catalyst

THOMAS DAVENPORTAuthor , Consultant

Competing on Analytics*, ,

Analyitcs at Work, Big Data at

Work, Only Humans Need

Apply:Winners and Losers in the

Age of Smart Machines.

*Recognized by Harvard

Business Review editors as one

the most important management

ideas of the past decade, one of

HBR’s ten must-read articles in

that magazine’s 90-year history.