i

University of Michigan Health System Adult and Children’s Emergency Departments

Efficiency and Effectiveness of Electronic Health Record (EHR) Use in the

Emergency Department Final Report

To: Jeffrey Desmond, M.D., Client - Associate Chair for Clinical Operations, UMHS Emergency Department

Mary Duck, Coordinator - Senior Management Consultant, UMH Program and Operations Analysis Alex Lai, Coordinator - IOE 481 Graduate Student Instructor

From: IOE 481 Project Team 9 Sarah Bach Samuel Pettinato Megan Taylor Date: April 23, 2013

ii

Table of Contents

Executive Summary ………………………………………………………………………………1 Goals and Objectives ……………………………………………………………………..1 Methods …………………………………………………………………………………...1 Checklists …………………………………………………………………………1 Surveys ……………………………………………………………………………1 Focus Groups ……………………………………………………………………..2 Findings …………………………………………………………………………………..2 Checklists …………………………………………………………………………2 Surveys ……………………………………………………………………………2 Focus Groups ……………………………………………………………………..2 Conclusions ……………………………………………………………………………….3 Determining a Methodology ……………………………………………………...3 Assessing MiChart Usage ………………………………………………………...3 Improving MiChart Usage ………………………………………………………..3 Recommendations ………………………………………………………………………...3 Introduction ……………………………………………………………………………………….4 Background ……………………………………………………………………………………….4 Key Issues ………………………………………………………………………………...4 Goals and Objectives ……………………………………………………………………..5 Project Scope ……………………………………………………………………………..5 Methods …………………………………………………………………………………………...5 Shortcomings of Task Timing ……………………………………………………………5

iii

Checklists ……………………………………………………………………….………6

Surveys ………………………………………………………………………….………6

MiChart Self-Assessments……………………………………………….……..7 Focus Groups ………………………………………………………………………...…8

Findings …………………………………………………………………………………….……8 Shortcomings of Task Timing…………………………………………………….……..8 Checklists ………………………………………………………………………………..8 Surveys …………………………………………………………………………………..10 Nurse: MiChart User Levels …………………………………………………….10 Nurse: Most and Least Used Features …………………………………………..11 Nurse: Correlations ……………………………………………………………...12 Nurse: Training Preferences …………………………………………………….13 Physician Assistant: MiChart User Levels ……………………………………...14 Physician Assistant: Most and Least Used Features …………………………….14 Physician Assistant: Correlations ……………………………………………….15 Physician Assistant: Training Preferences ………………………………………16 Physician Resident: MiChart User Levels ………………………………………16 Physician Resident: Most and Least Used Features …………………………….17 Physician Resident: Correlations ………………………………………………..18 Physician Resident: Training Preferences ………………………………………19 Physician Faculty: MiChart User Levels ………………………………………..19 Physician Faculty: Most and Least Used Features ……………………………...20 Physician Faculty: Correlations …………………………………………………21

iv

Physician Faculty: Training Preferences ………………………………………..22 Comparison between Roles ……………………………………………………...22 Focus Groups ……………………………………………………………………………23 Conclusions ……………………………………………………………………………………...24 Determining a Methodology …………………………………………………………….24 Assessing MiChart Usage ……………………………………………………………….24 Top 3 Defining MiChart Features ……………………………………………….24 Improving MiChart Usage ………………………………………………………………24 Recommendations ……………………………………………………………………………….25 Expected Impact …………………………………………………………………………………25 Appendix A: Epic’s Post-Live Checklist for Nurses ……………………………………………26 Appendix B: Epic’s Post-Live Checklist for Providers …………………………………………29 Appendix C: MiChart Usage Survey for Nurses ………………………………………………..33 Appendix D: MiChart Usage Survey for Providers ……………………………………………..39 Appendix E: Nurse Focus Group Summary and Transcript …………………………………….46 Appendix F: Provider Focus Group Summary and Transcript ………………………………….51 Appendix G: Nurse MiChart Self-Assessment Feature Usage Rankings ……………………….56 Appendix H: Physician Assistant MiChart Self-Assessment Feature Usage Rankings ………...60 Appendix I: Resident MiChart Self-Assessment Feature Usage Rankings ……………………..64 Appendix J: Faculty MiChart Self-Assessment Feature Usage Rankings ………………………68 Appendix K: Recommended Training Focuses …………………………………………………73 Appendix L: Top 3 Defining Features for Each Role……………………………………….…...76

v

List of Tables and Figures

Figure 1. Most Used Advanced MiChart Usage Features among Nurses ………………………9

Figure 2. Most Used Advanced MiChart Usage Features among Providers ……………………9

Figure 3. Overall ED MiChart User Level Breakdown …………………………………………10

Figure 4. Histogram of Nurse MiChart Self-Assessment Scores ……………………………….11

Figure 5. Nurse MiChart User Level Breakdown ……………………………………………….11

Figure 6. Nurse Training Method Preferences by Age Group …………………………………..13

Figure 7. Histogram of Physician Assistant MiChart Self-Assessment Scores …………………14

Figure 8. Physician Assistant MiChart User Level Breakdown ………………………………...14

Figure 9. Histogram of Physician Resident MiChart Self-Assessment Scores …………………16

Figure 10. Physician Resident MiChart User Level Breakdown ………………………………..17

Figure 11. Resident Training Method Preferences by Age Group ……………………………...19

Figure 12. Histogram of Faculty MiChart Self-Assessment Scores …………………………….19

Figure 13. Faculty MiChart User Level Breakdown ……………………………………………20 Figure 14. Faculty Training Method Preferences by Age Group ……………………………….22

Figure 15. ED MiChart User Level Breakdown by Role ………………………………………..22

Figure 16. ED MiChart User Level Breakdown by Role ………………………………………..23

Table 1. MiChart User Classification Scale ………………………………………………………7

Table 2. Survey and MiChart Self-Assessment Response Rates for all Roles ………………….10

Table 3. Most and Least Used MiChart Features by Nurses ……………………………………12 Table 4. Nurse User Levels with Previous EHR Experience ……………………………………12

Table 5. Nurse Actual User Level vs. Perceived User Level ……..……………………………..13

Table 6. Nurse Perceived User Level vs. Actual User Level …………………..………………..13

vi

Table 7. Most and Least Used MiChart Features by Physicians Assistant ……………………...15

Table 8. Physician Assistant User Levels with Previous EHR Experience ……………………..15

Table 9. Physician Assistant Actual User Level vs. Perceived User Level……………………...16

Table 10. Physician Assistant Perceived User Level vs. Actual User Level ……………………16

Table 11. Most and Least Used MiChart Features by Residents ………………………………..17

Table 12. Resident User Levels with Previous EHR Experience ……………………………….18

Table 13. Resident Actual User Level vs. Perceived User Level ………………………….……18 Table 14. Resident Perceived User Level vs. Actual User Level …………………………….…18 Table 15. Most and Least Used MiChart Features by Faculty ………………………………….20

Table 16. Faculty User Levels with Previous EHR Experience ………………………………...21

Table 17. Faculty Actual User Level vs. Perceived User Level ………………………………...21

Table 18. Faculty Perceived User Level vs. Actual User Level ………………………………...21

1

EXECUTIVE SUMMARY The University of Michigan Health System (UMHS) Emergency Department (ED), both Adult and Children’s divisions, uses an electronic health record (EHR) system to record patient information. MiChart-ASAP (MiChart), UMHS’s customized version of the healthcare software provider Epic’s EHR, was implemented within the ED on June 15, 2012. The Associate Chair for Clinical Operations, project client, in the UMHS ED was interested in determining a methodology to assess and improve MiChart usage efficiency and effectiveness. Thus, this study focused on assessing the efficiency and effectiveness of MiChart use in the UMHS Adult and Children’s ED. Yet, this study recommends a methodology for MiChart usage assessment that can be generalized to other departments throughout UMHS as MiChart continues to be implemented. Lastly, this study recommends additional training measures that can be implemented to improve MiChart usage.

Goals and Objectives The primary goal of this project was to determine a methodology to assess and improve efficiency and effectiveness of MiChart usage in the UMHS ED. Secondary project goals were:

• Improve efficiency and effectiveness of MiChart use within the ED • Standardize methodology used to measure MiChart efficiency and effectiveness • Develop findings that can be applied to improve MiChart training

Methods Various methods were tested to determine a methodology to assess and improve MiChart usage effectiveness and efficiency. Task timing was one of the methods tested, but it resulted in insignificant findings. A detailed report on the method of task timing can be found in the full report. All other methods used are summarized below. Checklists The team used Epic’s Post-Live Checklist for Nurses and Post-Live Checklist for Providers to determine which advanced features of MiChart were utilized by nurses and providers. These checklists each list 70 to 80 advanced tasks that MiChart users should be able to implement and can be found in Appendix A and B, respectively. Using the checklists, the team shadowed 10 providers in the Main area of the ED for 2 to 3 hour time periods, resulting in a total of 28 hours of provider data collection. The team shadowed 18 different nurses in the Main area of the ED for 3 to 4 hour time periods, resulting in a total of 46 hours of nurse data collection.. The team used the checklist results to determine which advanced MiChart features were most used. The 15 to 20 most used MiChart features were included in the MiChart Self-Assessments created by the team and included in the surveys. Surveys The MiChart Self-Assessments were distributed to nurses, physician assistants, residents, and faculty throughout both the Adult and Children’s ED in an online survey created through Qualtrics. The surveys contained 30 to 35 questions including questions focused on the user’s

2

background, MiChart training preferences, general MiChart opinions, and the MiChart Self-Assessments. The main goal of the surveys was to quickly and objectively identify MiChart user levels. User levels were determined by the following scale: scores above 75% were identified as high functioning users, scores 50 to 75% were identified as medium functioning users, and scores less than 50% were identified as low functioning users. The survey results also were used to identify advanced MiChart features that are most and least commonly used. Least used advanced MiChart features were recommended as focuses in additional training. Additional training methods were recommended based on results from the training portion of the survey. Lastly, the survey results identified any correlations present between user level and perceived user level, age, or previous EHR experience. Focus Groups The team led two focus groups, one for nurses and one for providers, to discuss MiChart usage strategies. These sessions were used to identify usage root causes that affect efficiency and effectiveness and to understand why users use MiChart the way they do. The focus group discussion questions were created by the team with input from the Associate Chair for Clinical Operations. Findings Using the methods and analysis techniques described above, the following results were found. Checklists The most used advanced features observed from Epic’s Post-Live Checklists for Nurses and Providers were calculated using Microsoft Excel. The 15 to 20 most used advanced features determined from this checklist observation were included in the MiChart Self-Assessments. Figures containing specific advanced feature observation data are included in the full report. Surveys Through online distribution of the surveys the following response rates were achieved: 31% of Nurses, 50% of Physician Assistants, 84% of Physician Residents, and 72% of Physician Faculty. The overall user level breakdown of the ED survey respondents was found to be: 14% low users, 74% medium users, and 12% high users. Each role within MiChart had a user level breakdown similar to the overall user level breakdown; however, findings by role can be seen in the full report. Further data regarding each role’s most and least used features, correlations between user level and perceived user level, age, and previous EHR experience, and training preferences are also presented in the full report. Focus Groups The nurse and provider focus groups had 9 nurses and 8 providers in attendance, respectively. Generally, the team found that nurses were not familiar with customization and advanced

3

features, were eager for additional training opportunities, and still had difficulty with intake and output and blood documentation. Providers were quite familiar with customization and advanced features, were interested in additional optional training materials, and felt they could improve efficiency through real time dictation. Complete summaries and transcripts of the nurse and provider focus groups are provided in Appendix E and F, respectively. Conclusions The primary project goal was to determine a methodology to assess and improve MiChart usage efficiency and effectiveness. Thus, conclusions were developed for the following 3 main topics of determining a methodology, assessing MiChart usage, and improving MiChart usage. Determining a Methodology The team concluded that the method of task timing, which involved shadowing users and timing common tasks performed in MiChart, is not an appropriate method for this study. The MiChart Self-Assessments created, with influence from the Epic Post Live Checklists, did accurately identify MiChart user levels. Assessing MiChart Usage Across all MiChart user roles it can be concluded that a majority of MiChart users are medium functioning users. Additionally, all other users are nearly evenly distributed between high functioning and low functioning users. It was found that user level is not strongly correlated with perceived user level, age, or previous EHR experience, both Epic or otherwise. Improving MiChart Usage To improve MiChart usage different training methods and areas of focus are preferred and needed between different roles. However, overall it was concluded that customization features and MiChart Tip Sheets and Videos are not utilized to their fullest potential. Recommendations The team recommends that this methodology to assess MiChart usage be continued in order to document progress made with MiChart usage improvements. The team also recommends that further MiChart training be implemented for each role as outlined below:

• Nurses: Classroom sessions with instructors every 6 months to 1 year • Physician Assistants and Residents: Presentations during scheduled meetings every 6

months to 1 year • Physician Faculty: Mentoring session with colleagues every 6 months

Features to be addressed during training should be determined by the combined metric of less than 50% observed usage and an average score of less than 3 on the MiChart Self-Assessment. Features that fit this metric can be found in Appendix K.

4

Introduction Use of Electronic Health Records (EHR) is becoming widespread throughout hospitals today. The University of Michigan Health System (UMHS) uses software developed by the healthcare software provider Epic as their form of an EHR. Epic currently holds about 40% of the EHR market share and has created an EHR software package that is highly customizable to the specific institution in which it is implemented. The UMHS Emergency Department (ED) implemented MiChart-ASAP (MiChart), a customized version of Epic’s EHR, in June 2012. However, the MiChart roll out has not been completed for all of UMHS. Nurses and providers received MiChart training, but it was unknown whether they are using MiChart as trained and how effectively they are using MiChart. Therefore, the Associate Chair for Clinical Operations in the UMHS ED asked the IOE 481 student project team from the University of Michigan to determine a methodology to assess and improve MiChart usage efficiency and effectiveness in the UMHS ED. The developed methodology was created with the goal of being general enough to be implemented throughout UMHS once the MiChart roll out is completed. Additionally, the team was asked to identify current weaknesses of MiChart usage and to recommend improvements to training. The purpose of this final report is to present the student team’s methods, findings, conclusions, and recommendations.

Background

The UMHS ED, both Adult and Children’s divisions, at the University of Michigan uses an EHR system to record patient information. The UMHS chose to standardize their EHR system and selected Epic healthcare software in 2011. This comprehensive Epic system is used by both nurses and providers and collects demographic information and clinical information; it also sends and receives orders, tests, and results. MiChart, UMHS’s customized version of Epic, was implemented within the ED on June 15, 2012; however, has yet to be implemented throughout all of UMHS. All staff received 8-14 hours of training on the MiChart system prior to its implementation; however, a learning curve still exists. While the team was informed that Deloitte consultants had partnered with UMHS to show the workload and process flow benefits of EHR implementation, no studies had specifically focused on effectiveness and efficiency of MiChart usage itself. Thus, this study sought to assess and improve MiChart usage efficiency and effectiveness in the UMHS ED, both Adult and Children’s divisions. This study also recommends a methodology for MiChart usage assessment that can be generalized to other departments throughout UMHS as MiChart continues to be implemented. Lastly, this study recommends additional training that can be implemented to improve MiChart usage.

Key Issues

The issues that created the need for this project were as follows:

• Unknown how MiChart is being used within the ED, thus standardized work may or may not exist

• No methodology exists to measure the efficiency and effectiveness of MiChart usage • Overall efficiency could be improved if the gaps between MiChart training and actual

usage were measured, understood, and improved upon

5

Goals and Objectives

The primary goal of this project was to determine a methodology to assess and improve efficiency and effectiveness of MiChart usage in the UMHS ED. Secondary project goals were:

• Improve efficiency and effectiveness of MiChart use within the ED • Standardize methodology used to measure MiChart usage efficiency and effectiveness

that can be applied to other departments as MiChart is implemented • Develop findings that can be applied to improve MiChart training

Project Scope

This study focused solely on use of MiChart within the UMHS ED. The study consisted of observations and checklists completed in the Adult ED and surveys and focus groups regarding MiChart use performed in both the Adult and Children’s ED.

Any other department of the UMHS was not analyzed in this study; however, the methodology developed from this study is able to be generalized and implemented throughout all of UMHS. While the Children’s ED staff was invited to participate in the survey and focus groups, no observations of the Children’s ED were performed. Lastly, no analysis of how MiChart usage relates to quality of care was performed.

Methods The methods used to complete the project goal of assessing and improving MiChart usage efficiency and effectiveness are as follows. The methodology used to complete this study affected the follow departments:

• UMHS Adult Emergency Department • UMHS Children’s Emergency Department

Shortcomings of Task Timing The team shadowed 18 nurses in the Main area of the ED for 3 to 4 hour time periods, resulting in a total of 46 hours of data collection. These hours covered daytime shifts on Monday through Friday from February 13, 2013 - March 1, 2013. The team observed all MiChart use by the shadowed nurse both at his or her desk and at the patient’s bedside. The team observed all tasks in the MiChart system and recorded the time taken to complete each task. The main task categories observed included:

• Chart review for patient information • Triage assessment and documentation • Order placement • Care documentation • Medication administration • Procedure documentation • Results review

6

Originally, the team planned to compare these task times across MiChart users to classify users into high functioning, medium functioning, and low functioning user groups. These user groups would be analyzed to determine which functions and usage strategies led to increased effectiveness and efficiency between user groups. However, through task timing the team did not observe enough tasks to have statistically relevant data. To increase data collection, the team extended nurse shadowing and task timing from one to two weeks; however, the quantity of data needed was still not reached. At least 10 data points for each task per user would be needed to statistically compare differences across users. The team collected on average 19 data points per observed user when at least 80 data points per observed user would be needed for comparison. Therefore, the results of this data were not the main focus of the project, but did eliminate task timing as a potential method to measure MiChart usage efficiency and effectiveness. Thus, as the team’s goal was to determine a methodology to assess and improve MiChart usage efficiency and effectiveness, different methods were implemented. Checklists The team used Epic’s Post-Live Checklist for Nurses and Post-Live Checklist for Providers to determine which advanced features of MiChart are utilized. These checklists were created by Epic and therefore include advanced features of the Epic EHR system as determined by Epic. These checklists each list 70-80 advanced features that MiChart users should be able to implement. Copies of both the Nurse and Provider Post-Live Checklists are provided in Appendix A and B, respectively. The appropriate checklist, nurse or provider, was completed by the team for each MiChart user observed. Using the checklists, the team shadowed 10 providers in the Main area of the ED for 2 to 3 hour time periods, resulting in a total of 28 hours of data collection from March 11-15, 2013. The team used data collected from the previous two weeks of nurse task timing observations to complete the checklist for nurses observed. As needed, the team followed up with previously observed nurses to complete missing checklist information. Using the checklist results, the team computed the percentage of observed users who implemented each advanced MiChart feature with Microsoft Excel to determine which features were most used. The 15 to 20 most used MiChart features found from Epic’s Post-Live Checklists were included in the MiChart Self-Assessments created by the team. These MiChart Self-Assessments were distributed to all MiChart users to quickly and objectively identify high functioning, medium functioning, and low functioning user groups. Surveys The MiChart Self-Assessments were distributed to nurses and providers throughout the ED in an online survey created through Qualtrics. The surveys were electronically distributed to all nurses, physician assistants, residents, and faculty in the ED and were available from March 25, 2013 – April 8, 2013. The surveys contained 30 to 35 questions including the MiChart Self-Assessments. Questions not related to the MiChart Self-Assessment were divided into the following categories:

• Background: Addressed user’s role, age, and previous electronic health record usage

7

• Training : Addressed both previous training received and preferences regarding future training methods and topics

• General MiChart Opinions: Allowed the user to express any likes, dislikes, or additional comments and questions regarding MiChart

MiChart Self-Assessments MiChart Self- Assessments included the most used features as identified by the team’s results, unused features which may provide opportunity for efficiency improvement, and unobserved features deemed essential by the Associate Chair for Clinical Operations. The MiChart Self-Assessments included 20 to 30 advanced usage questions. The survey responders reported their use of each feature on the following 0-5 scale: 0- I do not know how to use this feature, 1- Never, 2-Rarely, 3-Sometimes, 4-Frequently, 5-Always. All survey questions were selected through discussion between the team and the client. The entire survey was also reviewed by Barry DeCicco, Statistician for UMHS, who has extensive experience with survey creation and analysis. The Nurse and Provider Surveys are provided in Appendix C and D, respectively. The main goal of the surveys was to quickly and objectively identify MiChart user levels. Therefore, the MiChart Self-Assessment results were analyzed to determine the user level classification of survey respondents. Each survey respondent’s total points on the MiChart Self-Assessment were calculated with Microsoft Excel. Using the total number of available points, based on the number of questions in the MiChart Self-Assessment, each survey respondent’s percentage was calculated with Microsoft Excel. User group classification was determined by the scale shown in Table 1.

Table 1. MiChart User Classification Scale

MiChart User Classification

Score on MiChart Self-Assessment (%)

High Functioning User Over 75%

Medium Functioning User 50-75%

Low Functioning User Under 50%

In addition to determining MiChart user group classifications, the most and least commonly used advanced MiChart features were identified from the survey results. Using Microsoft Excel, the mean response value for each advanced MiChart feature question was calculated. The most and least used advanced MiChart features were calculated for all MiChart users, each role within MiChart, and each MiChart user level within each role. Least used advanced MiChart features were recommended as focuses in additional training. Additional training methods were

8

recommended based on results from the training portion of the survey. For each role, the team determined the most popular training method and frequency of training option. Survey results were also analyzed with Microsoft Excel to determine any correlations present between MiChart user classification and perceived user level, age, or previous electronic health record experience. Any frequently expressed sentiment in the general opinions section was summarized and reported. Focus Groups The team led two focus groups, one for nurses and one for providers, to discuss MiChart usage strategies. These sessions were used to identify usage root causes that affect efficiency and effectiveness and to understand why users use MiChart the way they do. The focus group discussion questions were created by the team with input from the Associate Chair for Clinical Operations. The Provider Focus Group took place on Monday, March 25, 2013 from 3:30PM-4:30PM and the Nurse Focus Group took place on Wednesday, March 27, 2013 from 9AM-10AM. The team summarized the opinions found from the focus groups. Findings The following results were found after using the methods and analysis techniques described above to assess and improve MiChart usage efficiency and effectiveness. Shortcomings of Task Timing Task timing, as described above, did not result in enough data to statistically analyze. The times found were also found to be highly variable. Due to the many different ways to perform the same function within MiChart task times were not representative of MiChart functionality level. For example, when nurses conduct the initial assessment they can either perform the assessment and then return to their computer to enter the data into MiChart or enter the data into MiChart at the patient’s bedside while performing the assessment. The times varied from 2.2 minutes to 9.6 minutes. In addition, these times were not indicative of MiChart efficiency because charting at the patient’s bedside can lead to higher overall process efficiency as the nurse does not have to walk back to their desk to complete the chart. Therefore, task timing was found not to be an appropriate method for assessing MiChart usage efficiency and effectiveness. Checklists The team calculated the most used advanced features from Epic’s Post-Live Checklists by determining what percentage of observed users implemented each feature. The most used advanced features as obtained from data of 18 nurses and 10 providers are shown in Figure 1 and 2, respectively.

9

Figure 1. Most Used Advanced MiChart Usage Features among Nurses (Source: IOE 481 Team Observations, Sample Size: 18 Nurses, Dates: Feb. 13- March 1, 2013)

Figure 2. Most Used Advanced MiChart Usage Features among Providers (Source: IOE 481 Team Observations, Sample Size: 10, Dates: March 11-15, 2013)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Adds comments to findings

Sorts by clicking column headers

Reviews labs using flowsheets or graphs.

Presses ENTER instead of clicking ACCEPT

Searches for medications by first few letters

Presses TAB instead of using the mouse.

Validates vitals

Can access her dashboard to view reports.

Searches for things by completion matching

Uses time and date shortcuts

Uses OrderSets

Can search for reports in SnapShot

Uses SmartPhrases

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

Presses ENTER instead of clicking ACCEPT

Sorts by clicking column headers

Searches by completion matching on partial words.

Reviews results using flowsheets or graphs.

Adds comments to findings

Uses SmartPhrases and SmartLinks in charting

Can view patient images on the Media tab

Can order a facility-administered medication.

Finds diagnoses by completion matching

Can search for reports in SnapShot

Knows naming conventions for SmartTools.

Understands standing and future orders.

Uses the Facility List and Database Lookup tabs.

Percentage of Nurses Observed

Percentage of Providers Observed

10

Surveys Through distribution of the MiChart Usage Surveys developed by the team through Qualtrics the following response numbers and rates were obtained are shown in Table 2.

Table 2. Survey and MiChart Self-Assessment Response Rates for all Roles (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 147, Dates: March 25-April 8, 2013)

Role Number of Responses Response Rate % of Total Nurse 75 31% 51%

Physician Assistant 9 50% 6% Physician Resident 27 84% 18% Physician Faculty 36 72% 25%

Total 147 43% All of the above response rates met or exceeded the Associate Chair for Clinical Operations’ expectations for response rates. Therefore, these response rates are high enough to ensure that the data provides an accurate representation of each group. Overall, the MiChart user level breakdown across all ED survey respondents is shown in Figure 3.

Figure 3. Overall ED MiChart User Level Breakdown

(Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 147, Dates: March 25-April 8, 2013) The team analyzed the data further by each role (Nurse, Physician Assistant, Physician Resident, and Physician Faculty) as each role uses MiChart differently. Nurse: MiChart User Levels Findings from the Nurse MiChart Self-Assessment scores resulted in a mean of 62% with a standard deviation of 11%. The maximum and minimum scores were 85% and 35%, respectively. A histogram of the Nurse MiChart Self-Assessment scores, shown in Figure 4, displays the concentration of scores in the range of 50-70%.

Low 14%

Medium 74%

High 12%

11

Figure 4. Histogram of Nurse MiChart Self-Assessment Scores

(Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 75, Dates: March 25-April 8, 2013) The Nurse MiChart user level breakdown was very similar to the overall ED MiChart User Level Breakdown and is shown in Figure 5.

Figure 5. Nurse MiChart User Level Breakdown

(Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 75, Dates: March 25-April 8, 2013) Nurse: Most and Least Used Features The average score received by each feature was calculated in Microsoft Excel to understand what features of MiChart are most and least used. The average score was calculated across all Nurse survey respondents with the following 0 to 5 scale: 0- I do not know how to use this feature, 1- Never, 2-Rarely, 3-Sometimes, 4-Frequently, 5-Always. The five most and least used features by Nurses, along with their average score, are shown in Table 3.

0

5

10

15

20

25

30

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Freq

uenc

y

MiChart Self-Assessment Score (%)

Low: 12%

Medium: 75%

High: 13%

12

Table 3. Most and Least Used MiChart Features by Nurses (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 75, Dates: March 25-April 8, 2013)

In addition, the team ranked the entire feature checklist from the MiChart Self-Assessment both overall and by user level within the role of Nurses. These lists of most and least used features can be found in Appendix G. Nurse: Correlations The team investigated possible correlations within the survey data. Specifically, it was hypothesized that a correlation may exist between MiChart user level and age or previous EHR experience, both with Epic or other software. The correlation coefficient between age and MiChart user level was found to be -0.15. The percentage of Nurses of each user level with previous EHR experience is shown in Table 4.

Table 4. Nurse User Levels with Previous EHR Experience (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 75, Dates: March 25-April 8, 2013)

User Level EHR Experience (Epic) EHR Experience (Other)

Low (9) 11% (1) 89% (8) Medium (56) 5% (3) 91% (51)

High (10) 30% (3) 100% (10)

The team also investigated whether a correlation existed between a users’ actual user level and what they perceived their user level to be. Survey responders were asked if they believed themselves to be a low, medium, or high functioning user. The resulting percentages of each user

Most Used: Average

Score

I add medications to the patient’s home medication list when appropriate 4.5

I use OrderSets for NIPCO's. I select only the appropriate items and can add orders on the fly. 4

I edit/correct my documentation when needed before completing a patient’s chart. 3.9

I delete medications from the patient’s home medication list when appropriate. 3.8

I review my documentation before completing a patient’s chart. 3.7

Least Used: I search for things like orders, reports, and recipients by completion matching on partial words. 1.6

I use the Trauma and Code Narrator solely to document trauma cases without taking notes on paper. 1.5

I use SmartPhrases and SmartLinks in charting. 1.3

I create personal SmartPhrases. (Also known as dot phrases.) 1.1

I access MiChart-ASAP Tip Sheets and Videos if I have questions. 0.9

13

type and what they perceived their user level to be are shown in Table 5. The percentages of each perceived user level group’s actual user level is shown in Table 6.

Table 5. Nurse Actual User Level vs. Perceived User Level (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 75, Dates: March 25-April 8, 2013)

User Level: Low (Perceived) Medium (Perceived) High (Perceived) Low (Actual) (9) 22% (2) 78% (7) 0%

Medium (Actual) (56) 5% (3) 82% (46) 13% (7) High (Actual) (10) 0% 40% (4) 60% (6)

Table 6. Nurse Perceived User Level vs. Actual User Level

(Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 75, Dates: March 25-April 8, 2013)

User Level: Low (Actual) Medium (Actual) High (Actual) Low (Perceived) (5) 40% (2) 60% (3) 0%

Medium (Perceived) (57) 12% (7) 81% (46) 7% (4) High (Perceived) (13) 0% 54% (7) 46% (6)

Nurse: Training Preferences In response to the training frequency questions, the team found that of Nurses surveyed 39% believed training should be offered annually and 31% believed training should occur every 6 months. It was also found that 27% of Nurses preferred the training method of a classroom session with a training instructor and 23% preferred elective, self-directed e-learning online resources. Training preferences did vary with age as seen in Figure 6.

Figure 6. Nurse Training Method Preferences by Age Group (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 75, Dates: March 25-April 8, 2013)

0%5%

10%15%20%25%30%35%40%45%50%

Classroom session with traininginstructorPresentations during scheduledmeetingsMentoring session withcolleaguesElective, self-directed e-learningonline resources

20-29 Years Old 30-39 Years Old 40-49 Years Old

14

Physician Assistant: MiChart User Levels The Physician Assistant MiChart Self-Assessments had a mean score of 60% and a standard deviation of 12%. The maximum score was 85% and the minimum score was 46%. A histogram of the Physician Assistant MiChart Self-Assessments scores is shown in Figure 7.

Figure 7. Histogram of Physician Assistant MiChart Self-Assessment Scores (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 9, Dates: March 25-April 8, 2013)

The Physician Assistant MiChart user level breakdown, shown in Figure 8, was very similar to the overall ED MiChart User Level Breakdown.

Figure 8. Physician Assistant MiChart User Level Breakdown

(Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 9, Dates: March 25-April 8, 2013) Physician Assistant: Most and Least Used Features The average score for each MiChart feature was calculated across all Physician Assistant survey respondents, using the following scale 0 to 5 scale: 0- I do not know how to use this feature, 1- Never, 2-Rarely, 3-Sometimes, 4-Frequently, 5-Always. The five most and least used features by Physician Assistants, along with their average score, are shown in Table 7.

Low: 11%

Medium: 78%

High: 11%

0

1

2

3

4

5

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Freq

uenc

y

MiChart Self-Assessment Score (%)

15

Table 7. Most and Least Used MiChart Features by Physicians Assistant (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 9, Dates: March 25-April 8, 2013)

In addition, the team ranked the entire feature checklist from the Physician Assistant MiChart Self-Assessment both overall and by user level. These lists of most and least used features can be found in Appendix H. Physician Assistant: Correlations The team investigated possible correlations within the survey data. Specifically, it was hypothesized that a correlation may exist between MiChart user level and age or previous EHR experience, both with Epic or other software. The correlation coefficient between age and MiChart user level was found to be -0.51. The percentage of Physician Assistants of each user level with previous EHR experience is shown in Table 8.

Table 8. Physician Assistant User Levels with Previous EHR Experience (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 9, Dates: March 25-April 8, 2013)

User Level EHR Experience (Epic) EHR Experience (Other)

Low (1) 0% 0% Medium (7) 0% 71% (5)

High (1) 100% (1) 0%

The team also investigated whether a correlation existed between a users’ actual user level and what they perceived their user level to be. Survey responders were asked if they believed themselves to be a low, medium, or high functioning user. The resulting percentages of each user type and what they perceived their user level to be are shown in Table 9. The percentages of each perceived user level group’s actual user level is shown in Table 10.

Most Used: Average

Score

I find and view patient images and scans on the Media tab in Chart Review. 4

When using NoteWriter, I use the Review of Systems function. 3.9

When using NoteWriter, I use the Physical Exam tab. 3.9

I use the Facility List and Database Lookup tabs to find orders that aren’t on my preference list. 3.5

I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 3.4

Least Used: I use the history tab to enter and/or edit patients’ past family history. 1.7

I delete medications from the patient’s home medication list when appropriate. 1.7

I use time and date shortcuts, such as “n”, “t-1”, “m+2”, and “y+1”. 1.6

I add orders to my preference list. I can edit orders display names so that they’re easier to find. 1.6

I access MiChart-ASAP Tip Sheets and videos if I have questions. 1.2

16

Table 9. Physician Assistant Actual User Level vs. Perceived User Level (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 9, Dates: March 25-April 8, 2013)

User Level: Low (Perceived) Medium (Perceived) High (Perceived)

Low (Actual) (1) 0% 100% (1) 0% Medium (Actual) (7) 14% (1) 86% (6) 0%

High (Actual) (1) 0% 0% 100% (1)

Table 10. Physician Assistant Perceived User Level vs. Actual User Level (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 9, Dates: March 25-April 8, 2013)

User Level: Low (Actual) Medium (Actual) High (Actual)

Low (Perceived) (1) 0% 100% (1) 0% Medium (Perceived) (7) 14% (1) 86% (6) 0%

High (Perceived) (1) 0% 0% 100% (1) Physician Assistant: Training Preferences In response to the training frequency questions, the team found that of 9 Physician Assistants surveyed 56% believed training should occur every year and 33% believed training should occur every 6 months. It was also found that a majority of 67% believed training should be offered during scheduled meetings. This majority training preference of preferring training during scheduled meetings was consistent among all age groups surveyed. Therefore, training preferences did not vary with age. Physician Residents: MiChart User Levels The Resident Assistant Self-Assessment had a mean score of 60% and a standard deviation of 9%. The maximum score was 79% and the minimum score was 45%. A histogram of the Physician Residents MiChart Self-Assessments scores is shown in Figure 9.

Figure 9. Histogram of Physician Resident MiChart Self-Assessment Scores

(Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 27, Dates: March 25-April 8, 2013)

0

5

10

15

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Freq

uenc

y

MiChart Self-Assessment Score (%)

17

The Resident MiChart user level breakdown, shown in Figure 10, was very similar to the overall ED MiChart User Level Breakdown.

Figure 10. Physician Resident MiChart User Level Breakdown

(Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 27, Dates: March 25-April 8, 2013) Physician Resident: Most and Least Used Features The average score for each MiChart feature was calculated across all Physician Assistant survey respondents, using the following 0 to 5 scale: 0- I do not know how to use this feature, 1- Never, 2-Rarely, 3-Sometimes, 4-Frequently, 5-Always. The five most and least used features by Physician Residents along with their average score, are shown in Table 11.

Table 11. Most and Least Used MiChart Features by Residents (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 27, Dates: March 25-April 8, 2013)

Most Used: Average

Score

When using NoteWriter, I add comments to specific findings or entire sections as needed. 3.7

I press ENTER after typing a search term instead of clicking Accept or Search. 3.7

I review patient information using CareWeb. (Excluding access through MiChart-ASAP) 3.6

I create personal SmartPhrases (Also known as dot phrases) 3.6

I create and share macros for typical exams. 3.5

Least Used: I add orders to my preference list. I can edit orders’ display names so that they’re easier to find. 1.6

I use the history tab to enter and/or edit patients’ past family history. 1.5

I edit (add and/or delete) medications from the patient’s home medication list when appropriate. 1.5

I use time and date shortcuts, such as “n”, “t-1”, “m+2”, and “y+1”. 1.0

I access MiChart-ASAP Tip Sheets and videos if I have questions. 0.7

Low: 8%

Medium: 85%

High: 7%

18

The feature above in Table 11 in italics corresponds to a low MiChart user level. Therefore, the fact that it is a most used feature should be viewed negatively. In addition, the team ranked the entire feature checklist from the Residents MiChart Self-Assessment both overall and by user level. These lists of most and least used features can be found in Appendix I. Physician Resident: Correlations The team investigated correlations within the survey data. Specifically, it was hypothesized that a correlation may exist between MiChart user level and age or previous EHR experience. The correlation coefficient between age and MiChart user level was found to be -0.08. The percentage of Residents of each user level with previous EHR experience is shown in Table 12.

Table 12. Resident User Levels with Previous EHR Experience (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 27, Dates: March 25-April 8, 2013)

User Level: EHR Experience (Epic) EHR Experience (Other) Low (2) 100% (2) 100% (2)

Medium (23) 59% (14) 95% (22) High (2) 50% (1) 100% (2)

The team also investigated whether a correlation existed between a users’ actual user level and what they perceived their user level to be. Survey responders were asked if they believed themselves to be a low, medium, or high functioning user. The resulting percentages of each user type and what they perceived their user level to be are shown in Table 13. The percentages of each perceived user level group’s actual user level is shown in Table 14.

Table 13. Resident Actual User Level vs. Perceived User Level

(Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 27, Dates: March 25-April 8, 2013)

User Level: Low (Perceived) Medium (Perceived) High (Perceived) Low (Actual) (2) 50% (1) 50% (1) 0%

Medium (Actual) (23) 26% (6) 52% (12) 22% (5) High (Actual) (2) 0% 0% 100% (2)

Table 14. Resident Perceived User Level vs. Actual User Level

(Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 27, Dates: March 25-April 8, 2013)

User Level: Low (Actual) Medium (Actual) High (Actual) Low (Perceived) (7) 14% (1) 86% (6) 0%

Medium (Perceived) (13) 8% (1) 92% (12) 0% High (Perceived) (7) 0% 71% (5) 29% (2)

19

Physician Resident: Training Preferences In response to the training frequency questions, the team found that of 27 Physician Residents surveyed 52% believed training should occur every year and 26% believed training should occur every 6 months. It was also found that 37% believed training should be elective, self-directed e-learning online resources and 33% preferred presentations during scheduled meetings. Training preferences did vary slightly with age as seen in Figure 11.

Figure 11. Resident Training Method Preferences by Age Group (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 9, Dates: March 25-April 8, 2013)

Physician Faculty: MiChart User Levels The mean score on the Faculty MiChart Self-Assessment was 60%, with a standard deviation of 13%. The maximum and minimum scores were 87% and 36%, respectively. The histogram shown in Figure 12 displays the range of scores.

Figure 12. Histogram of Faculty MiChart Self-Assessment Scores

(Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 36, Dates: March 25-April 8, 2013)

0%

5%

10%

15%

20%

25%

30%

35%

40%

45% Presentations duringscheduled meetings

Elective, self-directed e-learning online resources

Other (One on One, Cometo resident conference, MIchart person available toask questions during shift,none)

02468

1012

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Freq

uenc

y

MiChart Self Assessment Score (%)

20-29 Years Old 30-39 Years Old

20

The Faculty MiChart user level breakdown, shown in Figure 13, was very similar to the overall ED MiChart User Level Breakdown.

Figure 13. Faculty MiChart User Level Breakdown

(Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 36, Dates: March 25-April 8, 2013) Physician Faculty: Most and Least Used Features The average score for each MiChart feature was calculated across all Faculty survey respondents, using the following 0-5 scale: 0- I do not know how to perform this function, 1- Never, 2- Rarely, 3- Sometimes, 4- Frequently, 5- Always. The five most and least used features by Faculty, along with their average score, are shown in Table 15.

Table 15. Most and Least Used MiChart Features by Faculty (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 36, Dates: March 25-April 8, 2013)

Most Used: Average

Score

While using the In-Basket function, I know which attestation tab to use for different circumstances. 4.5

When using NoteWriter, I use the Physical Exam tab. 4

When using NoteWriter, I add comments to specific findings or entire sections as needed. 3.9

While using the In-Basket function, I use protocol notes for patients on observation protocols. 3.8

I use the Facility List and Database Lookup tabs to find orders that aren’t on my preference list. 3.7

Least Used: I use the history tab to enter and/or edit patient’s surgical history. 1.6

I use the history tab to enter and/or edit patient’s family history. 1.6

I use time and date / shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” 1.5

I add orders to my preference list. I can edit orders’ display names so that they’re easier to find. 1.4

While using the In-Basket function, I know what to do with deficiencies that are not mine. 0.6

Low: 22%

Medium: 64%

High: 14%

21

In addition, the team ranked the entire feature checklist from the Faculty MiChart Self-Assessment both overall and by user level. These lists of most and least used features can be found in Appendix J. Physician Faculty: Correlations The correlation coefficient between age and MiChart user level was found to be 0.02. The percentage of Faculty of each user level with previous EHR experience is shown in Table 16.

Table 16. Faculty User Levels with Previous EHR Experience (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 36, Dates: March 25-April 8, 2013)

User Level: EHR Experience (Epic) EHR Experience (Other) Low (8) 0% 100% (8)

Medium (23) 26% (6) 87% (20) High (5) 20% (1) 60% (3)

Faculty were asked to rate themselves as a low, medium, or high functioning user. This perceived user level was compared to their actual user level based on their MiChart Self-Assessment score. The relationship between Faculty actual user level and Faculty perceived user level is shown in Table 17. The percentages of the each perceived user level group’s actual user level is shown in Table 18.

Table 17. Faculty Actual User Level vs. Perceived User Level (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 36, Dates: March 25-April 8, 2013)

User Level: Low (Perceived) Medium (Perceived) High (Perceived)

Low (Actual) (8) 25% (2) 63% (5) 12% (1) Medium (Actual) (23) 13% (3) 70% (16) 17% (4)

High (Actual) (5) 0% 20% (1) 80% (4)

Table 18. Faculty Perceived User Level vs. Actual User Level (Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 36, Dates: March 25-April 8, 2013)

User Level: Low (Actual) Medium (Actual) High (Actual)

Low (Perceived) (5) 40% (2) 60% (3) 0%

Medium (Perceived) (22) 23% (5) 73% (16) 4% (1) High (Perceived) (9) 12% (1) 44% (4) 44% (4)

22

Physician Faculty: Training Preferences The team found that out of 36 Faculty surveyed, 36% believe training should occur every 6 months, 19% believe training should occur every year, and 17% believe that additional training should never be offered. The survey showed that 31% of Faculty prefers the training method of a mentoring session with colleagues, 19% prefer elective, self-directed e-learning online resources, and 17% prefer a classroom session with a training instructor. Faculty training preferences were found to somewhat vary by age group as shown in Figure 14.

Figure 14. Faculty Training Method Preferences by Age Group

(Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 36, Dates: March 25-April 8, 2013) Comparison between Roles Overall the team found that across all roles the user level breakdown was consistent with a majority being medium functioning users. The user level breakdowns within each role were similar and are shown below for comparison in Figure 15.

Figure 15. ED MiChart User Level Breakdown by Role

(Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 147, Dates: March 25-April 8, 2013)

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

30 - 39 Years Old 40 - 49 Years Old 50 - 59 Years Old

Classroom session with traininginstructorPresentations during scheduledmeetingsMentoring session withcolleaguesElective, self-directed e-learningonline resourcesOther

0%20%40%60%80%

100%

Nurse PA Resident Faculty

LowMediumHigh

23

Across roles the team found that training preferences varied greatly. The training preferences within each role are shown below for comparison in Figure 16.

Figure 16. Training Preferences by Role

(Source: MiChart Usage Survey Developed by IOE 481 Team, Sample Size: 147, Dates: March 25-April 8, 2013) Focus Groups The nurse and provider focus groups had 9 nurses and 8 providers in attendance, respectively. The participants in the nurse focus group covered a wide range of ages and user levels, while the participants in the provider focus group skewed toward advanced MiChart users. Generally, the team found the following trends among nurses:

• Not familiar with customization and advanced features • Eager for additional training opportunities • Still had difficulty with intake/outputs and blood documentation

Generally, the team found the following trends among providers: • Quite familiar with customization and advanced features • Interested in additional optional training materials • Believed they could improve efficiency through real time dictation.

Complete summaries and transcripts of the nurse and provider focus groups are provided in Appendix E and F, respectively.

0% 15% 30% 45% 60% 75%

Nurse

PA

Resident

FacultyWritten instructionsand referencematerials

Elective, self-directede-learning onlineresources

Mentoring sessionwith colleagues

Presentations duringscheduled meetings

Classroom sessionwith traininginstructor

24

Conclusions The primary project goal was to determine a methodology to assess and improve MiChart usage efficiency and effectiveness. This involved 3 main parts of determining a methodology, assessing MiChart usage, and improving MiChart usage. Therefore, the team’s conclusions address these 3 topics. Determining a Methodology The team concluded that the method of task timing, which involved shadowing users and timing common tasks performed in MiChart, is not an appropriate method for this study. Task timing did not result in enough data to have statistically relevant conclusions. The data that was collected was also highly variable. In addition, the data collected did not accurately reflect MiChart usage efficiency and effectiveness due to this variability. However, the team did conclude that the MiChart Self-Assessments accurately identified MiChart user levels and were an appropriate method for this study. Assessing MiChart Usage Across all MiChart user roles it can be concluded that a majority of MiChart users are medium functioning users. Additionally, all other users are nearly evenly distributed between high functioning and low functioning users. It was found that user level is not strongly correlated with perceived user level, age or previous EHR experience, both Epic or otherwise. Top 3 Defining MiChart Features The team determined a list of top 3 defining features for each role within MiChart to even more quickly assess MiChart usage and identify MiChart user levels in the future. These top 3 defining features were created to establish what to look for in a user to quickly distinguish their user level. These top 3 defining features were determined by the team through reflection on observation experience, calculation of the range of each feature’s MiChart Self-Assessment average score across user levels, and discussion with the Associate Chair for Clinical Operations. The top 3 defining features for Nurses, Physician Assistants, Physician Residents, and Physician Faculty are provided in Appendix L. Improving MiChart Usage In order to improve MiChart usage different training methods and areas of focus are preferred and needed between different roles. However, overall it was concluded that customization features and MiChart Tip Sheets and Videos are not utilized to their fullest potential.

25

Recommendations To continue to assess MiChart usage efficiency and effectiveness the team recommends continuing the methodology used in this study. After training improvements are made, this methodology should be repeated to assess whether MiChart usage has changed due to the additional training. As the MiChart rollout is continued throughout UMHS the team recommends assessing MiChart usage at an interval earlier than 8 months post go-live to have more initial usage data. While the team’s findings show that many users would prefer elective, self-directed e-learning modules, it was also found that current MiChart Tip Sheets and Videos are rarely accessed by MiChart users. Therefore, the team recommends educating MiChart users about current MiChart Tip Sheets and Videos before implementing additional self-directed online learning methods. Therefore, specific recommendations regarding training for each role are as follows:

• Nurses: Classroom sessions with instructors every 6 months to 1 year • Physician Assistants and Residents: Presentations during scheduled meetings every 6

months to 1 year • Physician Faculty: Mentoring session with colleagues every 6 months

The team recommends that features to be addressed during training should be determined by the combined metric of less than 50% observed usage and an average score of less than 3 on the MiChart Self-Assessment. Complete lists of MiChart usage features, both overall and by role, found to currently fit this metric are included in Appendix K. The team recommends that additional MiChart training focus on the features found in these lists. Expected Impact The results from this project are expected to improve MiChart usage in the ED by highlighting the least used MiChart features. MiChart training is expected to improve as upcoming MiChart training sessions can follow the preferred training methods and focus on the recommended features for additional training. Furthermore, the findings are expected to benefit future MiChart rollouts throughout all of UMHS. The initial training for these rollouts can follow the preferred training methods previously described. During these initial trainings, features that are currently least used within the ED can be emphasized as well. The methodology to assess and improve MiChart usage efficiency and effectiveness developed by the team can be repeated throughout other UMHS departments to b assess MiChart usage and document progress made.

0

Appendix A: Epic’s Post-Live Checklist for Nurses

Nurse Post-Live Checklist Clinic Name/Location: Date:

Nurse Name:

Trainer Name: Analyst Name:

Navigating and Personalizing the System Uses time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.”

Clicks to personalize things like her main toolbar, activity tabs, navigators, and Chart Review reports.

Clicks to personalize things like her Epic menu, activity tabs, and SmartTool toolbar. Clicks Options in various navigator sections to personalize the sections to meet her needs. Knows and uses several keyboard shortcuts, such as F8 to advance to the next navigator section. Presses TAB to move between fields, options, and buttons rather than using the mouse. Presses ENTER after typing a search term instead of clicking Accept or Search. Sorts by clicking column headers to find information in places like Chart Review and In Basket. Searches for things like orders, reports, and recipients by completion matching on partial words.

Reviewing the Patient’s Chart Presses = and uses the Recent Patients list to find patients in the Patient Lookup window. Knows how to review a patient’s previous lab results using tools such as flowsheets or graphs. Can search for reports in SnapShot and add report buttons to the SnapShot and Schedule activities. Adds personal patient notes using sticky notes. Creates personal filters in Chart Review. Understands that available criteria are limited to what’s in

the current patient’s chart at that time.

Identifies and opens an attachment in Chart Review by looking for . Reviews multiple items at once in Chart Review by selecting multiple rows and clicking Review

Selected. Views and upload files to the chart using Media Manager and the Media tab in Chart Review.

Documenting a Visit Selects the pharmacy to use for the current visit, and clicks to indicate the patient’s preferred

pharmacy. Uses shortcuts to enter blood pressure and height. (Enters “120 80” for 120/80 and “5f8i” for 5’8.”)

1

Starts visit notes in the NoteWriter. Adds comments to specific findings or entire sections as needed. Writes follow-up instructions using SmartTools and Clinical References.

Charting with SmartTools Uses SmartPhrases and SmartLinks in charting. Knows how to refresh SmartLinks. Searches for SmartPhrases and SmartLinks using the SmartPhrase Butler. Creates and shares personal SmartPhrases. Can create a SmartPhrase based on an existing

SmartText. Knows naming convention for her specialty-specific SmartTools. Uses and easily navigates SmartTexts. Saves commonly used SmartTexts as her favorites.

Clicks to apply the default values for SmartLists when appropriate. Can change a SmartList section.

Clicks to copy a previous note or to copy a section of a previous note when appropriate.

Placing Orders Searches for medications by entering the first few letters of the name followed by the dose. Understands placing and releasing standing and future orders. Knows how to indicate that an order requires a cosign and assigns it to the correct provider. Knows how to order and document a facility-administered medication and immunizations. Knows how to document historical immunizations reported by the patient.

Using Preference Lists Clicks to add orders to her preference list. Edits orders’ display names so that they’re easy to

find. Adds common variations of an order to her preference list. Modifies personal medication, lab,

imaging, and diagnosis preference lists using the Preference List Composer. Selects the Facility List or Database tabs when searching for an order that’s not on her preference

list. Knows how to copy and share preference lists from the Preference List Composer.

Using and Personalizing In Basket Clicks and drags column headers to rearrange them according to her preferences. Knows how to change her In Basket folder order by clicking Settings and selecting the Display Order

tab. Jumps to new messages from the In Basket alert bar. Knows that indicates unread messages. Can create and update Out of Contact occasions to record outages and specify delegates.

2

Gives other users permission to attach to her In Basket. Can attach another In Basket to hers as needed.

Knows how to search for messages that have already been marked as Done. Can save In Basket searches for quick use in the future. Checks in and out of pools as needed. Sorts and filters messages. Knows how to take and put back responsibility for a message. Can pend messages.

Following Up on In Basket Messages Uses SHIFT+click to select multiple messages and sign them or mark them as Done at the same time. Can change the status of message back to Read or New by clicking Mark As and selecting a status. Knows how to right-click a message to add a flag or postpone the message. Quickly follows up on high-volume message types, such as Results, Result Notes, Rx Request, My

Open Encounters, Patient Calls, and Pt Advice Requests. Creates and uses QuickActions to write routine result notes, letters, QuickNotes, and MyChart

messages from various message types. Can create them on the fly and from the QuickAction Manager.

Knows how to access graphs or flowsheets from a Results message. Can review lab results in a flowsheet or graph from a Results message.

Reporting Knows how to access her dashboard to view reports. Runs Reporting Workbench reports and knows how to sort, filter, and save results. Can access a patient’s chart and take other actions, like writing a letter or MyChart message, from

results. Edits report criteria when necessary and saves commonly used reports. Searches for reports in the library and adds commonly used reports to her Favorites list. Uses the Run as User criteria to run Reporting Workbench reports for physicians.

3

Appendix B: Epic’s Post-Live Checklist for Providers

Physician Post-Live Checklist Clinic Name/Location: Date:

Physician Name:

Trainer Name: Analyst Name:

Navigating and Personalizing the System Uses time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.”

Presses TAB to move between fields, options, and buttons rather than using the mouse.

Presses ENTER after typing a search term instead of clicking Accept or Search.

Knows and uses several keyboard shortcuts, such as CTRL+SHIFT+O to jump to Meds & Orders.

Sorts by clicking column headers to find information in places like Chart Review and In Basket.

Searches for things like orders, reports, and recipients by completion matching on partial words.

Clicks to personalize things like her main toolbar, activity tabs, navigators, and Chart Review reports.

Clicks to personalize things like her Epic menu, activity tabs, and SmartTool toolbar.

Clicks Options in various navigator sections to personalize the sections to meet her needs.

Reviewing the Patient’s Chart Presses = and uses the Recent Patients list to find patients in the Patient Lookup window.

Knows how to review a patient’s previous lab results using tools such as flowsheets or graphs.

Knows how to use Synopsis to see correlations between results, medications, and other events. Can add important information to the Patient Spotlight.

4

Reviews patient information from outside organizations using CareWeb Charts.

Adds personal patient notes for all caregivers in the specialty using Specialty Comments in the Snapshot report.

Adds patient notes for all caregivers across the organization using Family comments in the Snapshot report.

Can search for reports in SnapShot and Schedule activities and add buttons to the report toolbar.

Creates personal filters in Chart Review. Understands that available criteria are limited to what’s in the current patient’s chart at that time.

Identifies and opens an attachment in Chart Review by looking for icon

Reviews multiple items at once in Chart Review by selecting multiple rows and clicking Review Selected.

Knows how to find and view patient images and scans on the Media tab in Chart Review.

Pins reports to the sidebar so she can review the report while charting in other activities.

Documenting Diagnoses and Problems Finds diagnoses quickly by completion matching or entering diagnosis codes.

Knows how to indicate which diagnosis is the primary diagnosis.

Clicks File to History when documenting a new problem if the problem should be part of the patient's medical history.

Places orders while reviewing the problem list.

Can copy diagnoses from the problem list to the visit diagnoses list and vice versa.

Using the NoteWriter

Adds comments to specific findings or entire sections as needed

Clicks to copy a previous note or to copy a section of a prevoius note when appropriate.

5

Creates and shares macros for typical exams and procedures. Knows how to designate a primary macro.

Annotates images on the Physical Exam tab.

Knows how to open a blank note to use one's personal note template, then pull in Review of Systems and Physical Exam Forms. (.ROSBYAGE) (.PHYEXAMBYAGE)

Opens a blank note to, while writing the note, see the sidebar with the previous note, specialty comments, and snapshot information.

Charting with SmartTools Uses SmartPhrases and SmartLinks in charting. Knows how to refresh SmartLinks to pull in new data.

Searches for SmartPhrases and SmartLinks using the SmartPhrase Butler.

Creates, shares, and "steals" personal SmartPhrases. Can create a SmartPhrase based on an existing SmartText.

Knows naming convention for her specialty-specific SmartTools.

Uses and easily navigates SmartTexts. Saves commonly used SmartTexts as her favorites.

Clicks to apply the default values for SmartLists when appropriate. Can change a SmartList section.

Clicks to copy a previous progess note when appropriate.

Placing Orders

Searches for medications by entering the first few letters of the name, followed by the dose.

Can quickly associate all orders with the same diagnosis by clicking the heading for that diagnosis.

Understands standing and future orders.

Clicks Edit Multiple to quickly update multiple orders (to make them all future orders, for example).

6

Knows how to reprint an order from Order Entry.

Knows to right-click anywhere on the order composer to close it, rather than having to find and click Accept.

Knows how to activate Reorder checkboxes for current medications in the Meds and Orders navigator.

Clicks Change or Reorder to renew or update a prescription, rather than starting from scratch.

Knows how to order a facility-administered medication.

Knows how to enter charges in the Charge Capture navigator section.

Uses OrderSets for routine visits. Selects only the appropriate items and can add orders on the fly.

Marks commonly used OrderSets as favorites and knows how to preview an OrderSet before using it.

Knows how to mark a pharmacy as the patient's preferred pharmacy by clicking .

Using Preference Lists Clicks to add orders to her preference list. Edits orders’ display names so that they’re easy to find.

Adds common variations of an order to her preference list.

Modifies her ordering preference list using the Preference List Composer.

Modifies diagnosis preference list using Preference List Editor.

Creates sections to organize her preference list.

Can find orders that aren't on her preference list using the Facility List and Database Lookup tabs.

Knows how to copy and share preference lists from the Preference List Composer.

7

Appendix C: MiChart Usage Survey for Nurses

Nurse Survey on MiChart-ASAP Usage

Hello. You are receiving this survey because you have been identified as a UMHS MiChart-ASAP user. The purpose of this survey is to collect data on your usage of MiChart-ASAP. The data will be used to determine new/improvements to the current MiChart-ASAP training sessions. No personal information will be shared. This survey will take approximately 10-15 minutes to complete and the research team is very grateful for your participation.

Please indicate your role:

Nurse Other:

Please indicate your age:

20-29 years old 30-39 years old 40-49 years old 50-59 years old Over 60 years old

Have you had previous experience using the Epic electronic health record system (outside of UMHS)?

No Yes

If yes, please explain:

a) Organization b) Role c) Years of experience

Have you had previous experience using another electronic health record system?

No Yes

If yes, please explain:

a) Organization b) Role c) Years of experience

8

_________________________________________________________________________

MiChart-ASAP Self-Assessment:

For this series of questions, please use the following scale (unless otherwise specified) to indicate how often you use the functions described while working clinically within the department: (0- I do not know how to perform this function, 1-Never, 2-Rarely, 3-Sometimes, 4-Frequently, 5-Always)

1. I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.”

0 1 2 3 4 5

2. I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports, Epic menu, and various sections. (Excluding aesthetic features such as backgrounds)

0 1 2 3 4 5

3. I press TAB to move between fields, options, and buttons rather than using the mouse.

0 1 2 3 4 5

4. I press ENTER after typing a search term instead of clicking Accept or Search.

0 1 2 3 4 5

5. I sort by clicking column headers to find information in places like Chart Review.

0 1 2 3 4 5

9

6. I search for things like orders, reports, and recipients by completion matching on partial words.

0 1 2 3 4 5

7. I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 0 1 2 3 4 5

8. I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals)

0 1 2 3 4 5

9. I have added buttons to the SnapShot report toolbar

No Yes

10. I verify vitals directly in MiChart-ASAP instead of manually typing 0 1 2 3 4 5

11. I start visit notes in the NoteWriter and add comments to specific findings or entire sections as needed. 0 1 2 3 4 5

12. I use SmartPhrases and SmartLinks in charting.

0 1 2 3 4 5

10

13. I create personal SmartPhrases (Also known as dot phrases)

0 1 2 3 4 5

14. I use OrderSets for NIPCO’s. I select only the appropriate items and can add orders on the fly.

0 1 2 3 4 5

15. I access the ED dashboard to view reports.

0 1 2 3 4 5

16. I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical history Medical: 0 1 2 3 4 5

Social: 0 1 2 3 4 5

Family: 0 1 2 3 4 5

Surgical: 0 1 2 3 4 5

17. I edit (add and/or delete) medications from the patient’s home medication list when appropriate

Add: 0 1 2 3 4 5 Delete: 0 1 2 3 4 5

18. I use the Trauma and Code Narrator solely to document trauma cases without taking notes on paper.

0 1 2 3 4 5

11

19. I review my documentation before completing a patient’s chart.

0 1 2 3 4 5

20. I edit/correct my documentation when needed before completing a patient’s chart.

0 1 2 3 4 5

21. I access MiChart-ASAP Tip Sheets and videos if I have questions. 0 1 2 3 4 5

___________________________________________________________________________

How would you identify your MiChart-ASAP proficiency level?

Low functioning user Medium functioning user High functioning user

How long has it been since your last training/refresher training session on MiChart-ASAP?

Less than 3 months 3-6 months Over 6 months

Do you feel the training you received prepared you to be able to use MiChart effectively? (1: strongly disagree - 5: strongly agree)

1 2 3 4 5

12

How beneficial would additional training on the MiChart-ASAP system be for you? (1: no benefit - 5: very beneficial)

1 2 3 4 5

What would be most beneficial for you to learn about MiChart-ASAP? What additional skills would you like to learn?

How often, if at all, do you feel training/refresher training sessions be offered?

Never Every 6 months Every year Other:____________

If further training were to be offered what method of training would you most prefer?

Classroom session with training instructor Mentoring session with colleagues Mandatory e-learning modules Elective, self-directed e-learning resources Written instructions and reference materials Other:_______________________________

What do you particularly LIKE about MiChart-ASAP?

What do you particularly DISLIKE about MiChart-ASAP?

Any additional comments/questions:

13

Appendix D: MiChart Usage Survey for Providers

Provider Survey on MiChart-ASAP Usage

Hello. You are receiving this survey because you have been identified as a UMHS MiChart-ASAP user. The purpose of this survey is to collect data on your usage of MiChart-ASAP. The data will be used to determine new/improvements to the current MiChart-ASAP training sessions. No personal information will be shared. This survey will take approximately 10-15 minutes to complete and the research team is very grateful for your participation.

Please indicate your role:

Physician-Resident Physician-Faculty Physician Assistant Other:

Please indicate your age:

20-29 years old 30-39 years old 40-49 years old 50-59 years old Over 60 years old

Have you had previous experience using the Epic electronic health record system (outside of UMHS)?

No Yes

If yes, please explain:

a) Organization b) Role c) Years of experience

Have you had previous experience using another electronic health record system?

No Yes

If yes, please explain:

a) Organization b) Role c) Years of experience

14

____________________________________________________________________________

MiChart-ASAP Self-Assessment:

For this series of questions, please use the following scale (unless otherwise specified) to indicate how often you use the functions described while working clinically within the department: (0- I do not know how to perform this function, 1-Never, 2-Rarely, 3-Sometimes, 4-Frequently, 5-Always)

1. I press ENTER after typing a search term instead of clicking Accept or Search.

0 1 2 3 4 5

2. I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.”

0 1 2 3 4 5

3. I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports, Epic menu, and various sections. (Excluding aesthetic features such as backgrounds) 0 1 2 3 4 5

4. I sort by clicking column headers to find information in places like Chart Review and In Basket.

0 1 2 3 4 5

5. I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results.

0 1 2 3 4 5

15

6. I review patient information using CareWeb. (Excluding access through MiChart-ASAP)

0 1 2 3 4 5

7. (If Question 6 is answered 2-5) When I access CareWeb (excluding access through MiChart-ASAP) I use CareWeb to review: Labs Documents Other Results Other:

8. I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals)

0 1 2 3 4 5

9. I have added buttons to the SnapShot report toolbar

No Yes

10. I find and view patient images and scans on the Media tab in Chart Review.

0 1 2 3 4 5

11. When using NoteWriter, I add comments to specific findings or entire sections as needed. 0 1 2 3 4 5

12. When using NoteWriter, I delete or edit SmartLinks. 0 1 2 3 4 5

16

13. When using NoteWriter, I create personal templates for documentation. 0 1 2 3 4 5

14. When using NoteWriter, I use the Review of Systems function.

0 1 2 3 4 5

15. When using NoteWriter, I use the Physical Exam tab.

0 1 2 3 4 5

16. I create and share macros for typical exams. 0 1 2 3 4 5

17. I use SmartPhrases and SmartLinks in charting. 0 1 2 3 4 5

18. I create personal SmartPhrases (Also known as dot phrases)

0 1 2 3 4 5

19. I add orders to my preference list. I can edit orders’ display names so that they’re easier to find.

0 1 2 3 4 5

20. I use the Facility List and Database Lookup tabs to find orders that aren’t on my preference list.

0 1 2 3 4 5

17

21. I use OrderSets for initial orders. I select only the appropriate items and can add orders on the fly.

0 1 2 3 4 5

22. I access the ED dashboard to view reports.

0 1 2 3 4 5

23. I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical history Medical: 0 1 2 3 4 5

Social: 0 1 2 3 4 5

Family: 0 1 2 3 4 5

Surgical: 0 1 2 3 4 5

24. I edit (add and/or delete) medications from the patient’s home medication list when appropriate

Add: 0 1 2 3 4 5 Delete: 0 1 2 3 4 5

25. I access MiChart-ASAP Tip Sheets and videos if I have questions. 0 1 2 3 4 5

26. (Faculty Only) While using the In-Basket function, I use protocol notes for patients on observation protocols. 0 1 2 3 4 5

18

27. (Faculty Only) While using the In-Basket function, I know which attestation tab to use for different circumstances (Ex: Resident, Attending, PA, and Subsequent teaching physician notes)

0 1 2 3 4 5

28. (Faculty Only) While using the In-Basket function, I send staff messages.

0 1 2 3 4 5

29. (Faculty Only) While using the In-Basket function, I forward charts to other faculty members.

0 1 2 3 4 5

30. (Faculty Only) While using the In-Basket function, I manage deficiencies that are not mine. 0 1 2 3 4 5

____________________________________________________________________________

How would you identify your MiChart-ASAP proficiency level?

Low functioning user Medium functioning user High functioning user

How long has it been since your last training/refresher training session on MiChart-ASAP?

Less than 3 months 3-6 months 6-9 months

Do you feel the training you received prepared you to be able to use MiChart-ASAP effectively? (1: strongly disagree - 5: strongly agree)

1 2 3 4 5

19

How beneficial would additional training on the MiChart-ASAP system be for you? (1: no benefit - 5: very beneficial)

1 2 3 4 5

What would be most beneficial for you to learn about MiChart-ASAP? What additional skills would you like to learn?

How often, if at all, do you feel training/refresher training sessions be offered?

Never Every 6 months Every year Other:____________

If further training were to be offered what method of training would you most prefer?

Classroom session with training instructor Mentoring session with colleagues Mandatory e-learning modules Elective, self-directed e-learning resources Written instructions and reference materials Other:_______________________________

What do you particularly LIKE about MiChart-ASAP?

What do you particularly DISLIKE about MiChart-ASAP?

Any additional comments/questions:

20

Appendix E: Nurse Focus Group Summary and Transcript Nurse Focus Group Summary:

Training:

• Positives: o Helpful to see the system and gain familiarity

• Negatives: o Too much information at once o Too long of a delay from training to go-live o Initial trainers did not understand nursing or ED process flow

• All are in favor of additional training o Specific focus on advanced features and customization o 6 months after initial training o Classroom session with instructor who knows the system

MiChart Usage:

• Many nurses had not customized their system o More comfortable with OrderSets for NIPCO’s and Favorites (Stars) o Not comfortable with SmartPhrases or Validation of vitals

• Most commonly ask coworkers for help o If cannot fix problem, will just write a blank note so it’s documented o Helpful to have super users around. Would like to see this more frequently.

• Believe people don’t want to change because they’re creatures of habit, don’t see the immediate benefit of the new methods, or don’t know how to use the new methods

MiChart Opinions:

• Hard to do blood and I/O’s • Easy to administer Meds • Would like to learn advanced features to be more efficient • Problems with students using the system

21

Nurse Focus Group Transcript: Responses are in Italics Training:

• What was most/least useful about your MiChart-ASAP training? Got us into MiChart for the first time Lot of information at once Hard to absorb lots of information at a short time Easier to use now, but lots of things I still don’t know because it was all once Don’t know SmartPhrase building, specifically with drop downs I can build SmartPhrases, but don’t use them Positive: Lot of information which would be helpful, Negative: Full month before we

actually used this information. Practice cases not totally applicable. Too long of a delay from education from implementation. That which you don’t use

you lose. • Do you use the system as trained? Why/Why not? Who knows? There’s way too many ways to do things. Initial trainers didn’t understand our flow or processes. They couldn’t apply it to our

work environment. I think it was good that they went through each scenario. Switched instructors midway through, after we complained, which was a little

confusing. A lot of people still have trouble with the blood

• Do you think a follow-up training would be helpful? Why/Why not? YES I would like that To go over things I still I don’t know I think across the board, but not everyone To learn the “cute” parts of MiChart like SmartPhrases. Perhaps first time training

just do basics. Follow-Up training about customization and such. Should standardize the best way to do things I seem to learn something new every time I sit down with someone else. I think a second learning session would be very helpful to learn advanced features

• If so, how long after the initial training should the follow-up take place? 6 months to a year I would say 6 months No longer than 6 months Made changes after go-live, changes every month After launch there was always something new to learn, something new to click At every meeting I was announcing changes

• What would be your preferred method of training for follow-up training? Exactly the same way as initial training Someone should be there so we can ask questions if we get stuck More applicable practice cases

22

Initial cases were not using nursing cases Trainers had no background in medicine or nursing Structure of education didn’t match up to what we actually practiced Trainers now ask “Is this what you want? Is this what you do?” which is good. The

initial trainers were outside contractors who knew system, but did not care about us.

MiChart usage and why it is used that way: • Have you personalized your system in any way? Why/Why not? No Yes. Probably within 3 days I knew which tabs I would use the most, so I put them

right at the top. I saved some things as Stars. I wish we could add more. I didn’t know you could do that! That sounds nice. I’ve built some SmartPhrases, but I never used them. I’m unfamiliar so I use the path of least resistance.

• Do you use SmartLinks and SmartPhrase in charting? Why/Why Not? Not sure what a SmartLink is. Provider only. Apparently we can build SmartPhrases, but I don’t know.

• Do you use advanced features such as validation of vitals directly in the system, OrderSets, Macros, etc.? Why/Why not? We use validation Validation doesn’t always seem to pull everything over Macro would be used in provider only We use OrderSets for NIPCO’s

• What steps do you take if you have difficulty performing a task in MiChart-ASAP? Forget the Help Desk Co-Worker or Help Desk Co-Worker or work around Or blank note just so it’s documented Reference sheets are outdated I feel like Tip Sheets are more Provider based No, there are Tip Sheets-RN just for nurses The one Desmond sends out are more providers I do call the Help Desk and start a ticket for some things I would like a phone call directly to someone who can help me right now. References

are vague for me to utilize. Having people around during Go-Live, the at the elbow support, was very helpful. I

think wherever they do go-live they need to have. The outside people were less helpful than our super users. Our super users were great because they understood how we use MiChart.

I think we should bring back Super Users. Perhaps have them come back around every 3 months to answer any questions. Let people know before they are coming so that we can prepare questions.

Sometimes people get annoyed if we ask for questions

23

• Why do you think people don’t change their ways with MiChart? (Culture change question)

Instead of using SmartPhrases, they create an entire sentence that doesn’t change so they created all their work but can’t customize it.

Some people are creatures of habit and don’t want to change. Some people have created work arounds to do what they always do without using the

system. Takes too much time to click and click and click I can type faster so I do a blank note. I don’t have SmartPhrases, I may be the slowest user in this group I’ll admit, and I’m

fine using the system when it makes sense. If something is missing I have to look for it and it’s time consuming. I’ll just use a blank note if I can’t figure out how to do it in the appropriate note.

A note is a note to me. They look the same. • What do you do when you see a problem/potential change in MiChart-ASAP? If there’s something in there that you want to add to the content, you should email.

MiChart opinions: • What is still hard to do with MiChart-ASAP?

Blood. You have double check before and after. You have to order it on paper and in the computer. It’s just a nightmare. If you click it, it’s hard to find your next step.

Screens are busy, font is too small. I have no interest in mastering this crap. Buttons change spots on every screen, no repetition.

Not a big fan of Code/Trauma Critical Care Narrator. Hard to navigate and see what you charted.

I don’t like any of the narrators. I always feel like I’m missing something in my Trauma Narrator Fluid I/O’s. If it’s the same type of fluid why does a new line get created every time

you start a new bag? Why can’t it be like centricity? Doesn’t tally it very quickly, I have to manually add it. Hard to figure out where I am in the system.

Documenting I/O’s. I just want it to have IV at the top like it used to. I add notes to make it look like the old system. I can count it faster in my MAR than I can in I&O’s.

I wish it didn’t have to be exact spelling. I’m a bad speller. I wish it was like Google searches where if you misspell something it asks “Did you mean this...” . Without exact spelling you’re done. However, notes do have spell check.

• What is easy to do with MiChart-ASAP? Turning it off Charting meds is easy. Triage process is getting easier for people. Where can I find a list of chief complaint phrases to increase my speed and

familiarity? If I knew now what I knew 6 months ago I would have been much more efficient.

Chief complaint list: I need to know all the options before I pick What term did you need to type to get result of the search? Sometimes I don’t know

what to type to get things to come up on partial matching search Moving patients to different areas

24

Ability to see status of unit, track board Snapshot. Snapshots great. Report View Can trend through results review so easily Functionality

• What could make you more efficient with MiChart-ASAP? Learning how to use anything Advanced training Super users even learn from teaching others I like when people use it differently and show me how to do it different ways I hope in training that they use a model very similar to model they will actually use

unlike we did in training • What do you like/dislike about MiChart-ASAP? I will like it more when I become more efficient with it. Once I figure it out I see how

efficient it is and that makes me happy. Better than chasing around with pieces of paper. Verbal orders haven’t increased Some people document in blank note meds they’ve done instead of doing it in MAR I look forward to the floors becoming familiar with MiChart. The floors don’t utilize

their viewing ability in MiChart. Once everywhere is live it will be better. Built in meds for narrators. Quick Meds. Can give and approve pre-approved meds in

a few clicks. Fluids got missed a lot in the past If I put in my own Bolus that I order myself there is not option to complete. If the

provider orders it I can complete, but not if I do it myself. Bolus is based on time so doesn’t allow you to complete, should drop off after 30 min.

I&O screen is a blur to me I like Chart Review. It works for me.

• Additional comments Struggled with students using it. Students can’t task complete in the system, can

acknowledge. I agree not giving order ability, but I think they should have the ability to collect labs. Nurses don’t actually collect so it’s inaccurate to say they did instead of the student nurse. I have to sign as student charts, which isn’t how the workflow actually works. Students only do half the flow, won’t learn workflow. This could be a problem once it goes upstairs where there are many more students. Creates extra work. Student can’t do anything with history.

25

Appendix F: Provider Focus Group Summary and Transcript Provider Focus Group Summary:

Training:

• Positives: o Exposure to the system

• Negatives: o Instructor’s do not understand workflow o Learned most from hands-on experience

• Would be helpful to know what the views of other roles are in the system (i.e. What nurses can and cannot see)

• Do not necessarily use as trained, have found more efficient or preferred methods • Interested in video training, but not another session with an instructor

o Topics could include: Trending Labs Going through old notes SmartPhrases Macros Better templates

MiChart Usage:

• Majority has personalized o Use SmartPhrases

• Differing opinions on use of OrderSets • Preference List not fully utilized • Will attempt to work around problems or ask coworkers for help

MiChart Opinions:

• Hard to search in Orders, find past patient information, and trend information • Easy to do orders, discharge process, expected referral • Real time dictation would improve efficiency • Would like to be able to build macros into a note

o Allow notes to be created with writer’s preferred order, but shown in standard reader’s order

• Many do not edit past medical history or home medications

26

Provider Focus Group Transcript: Comments from Focus Group are in Italics Training:

• What was most/least useful about your MiChart-ASAP training? Nothing Exposure to the application. Went to 3 or 4 sessions as a super user. Really helped

going to multiple sessions. Can’t just walk in and start using. Coverage for initial roll-out was good. Super users were everywhere. Lots of support. Practice Cases: Exposure, Able to play about and see interface was necessary.

Impossible to just walk in. I came in after Roll-Out, had 1 session of training, and little practice on my own. I

remembered nothing. I felt truly untrained. No learning occurred during the training session. Gave me a link to practice sessions (not very applicable though)

I had 6 months of real time usage prior to training. Training didn’t add anything I didn’t already know. Want to know how to order things for patient, discharge, write a note...which the training did not focus on. However, training was worse at my prior institution.

I went through initial training and roll out. I’ve heard from people who came in after that they really struggle coming in. Residents come in with complete inability to use the system. Ongoing training is really a challenge. They can usually log on, make sense of track board, but DON’T remember any of the processes. Hard because we don’t have time to teach them MiChart on the fly if they need a lot of help.

Instructors do not have an idea of what we do. They could not train on workflow, could only answer system functionality questions.

Training was going on while they were still refining the system. People who are trained now should have more workflow orientation around the

training. Instructors told us “This is not what you’re actually going to see” Would you use online modules useful? I find it hard to be motivated. Learned more my

first half shift than during my 4 hours of training. Learn by doing. Fastest way to learn is to have someone with you there the first time you use it.

MLearning was mostly passive. Interactive modules would instill memory of functions. Virtual charting would be useful.

Printed materials: Nobody looked at them ever again. “It stayed in my bag unopened” • What do you wish you learned during training? View of other roles. “What can nurses see in the system?” I want to know what other

people can and cannot do. How to get around the roadblocks. “For antibiotics you need weight, I didn’t know

that” I still don’t know how to do some things (Fix for weight was put in, but some people still don’t know about it)

• Do you use the system as trained? Why/Why not? I don’t remember how I was trained Found more efficient ways to use than what I was taught during training. Everyone

has their own way of using it. I dictate half the note in the room with the patient. I do

27

90% of documentation and place order while in the room with the patient. That way the patient can correct me at the end if I say something incorrect. 2 dictations and then start my MDM, by the end of my shift I have 7 in progress notes but I it’s only half of and MDM note which I can finish in the car on the way home. I carry 2 phones so I don’t get interrupted while dictating.

Have residents dictate while presenting. I do notes with partial dictation, system starts you at bottom so you go bottom to top,

so I don’t have to click refresh and have it bounce me around. I have a system that I use.

I do dictations as one big thing and then split them later. Later I copy and paste. • Do you think a follow-up training would be helpful? Why/Why not? Topics: Trending labs. Going through old notes. I still don’t know how to pull labs

into notes effectively. Dot phrases. I’ve done something but it won’t pull in. Better templates (Soon to be Dept. Policy to use MiChart templates, i.e. using ROS and Physical Exam tabs).

What other work arounds have people created? People would use note process if they knew how to have a Macro. They didn’t spend any time on Macros within the training. If they knew how to do it and how quick it was they would be motivated to do it.

• What would be your preferred method of training for follow-up training? Session probably not helpful, but individual video training for common problems that I

can follow would be helpful.

MiChart usage and why it is used that way: • Have you personalized your system in any way? Why/Why not?

Impossible not to I have dot phrases for discharge instructions I have dot phrases for everything If you’re not doing Macro physical review system you’re wasting time (about 30 min

per day) Sometimes you don’t know what you don’t know. Is there a way to place my 2 dictation

markers in each note? Because I have to create every note the same way. But I don’t think a macro can include dictation markers. (Takes about 1 min, about 20-30 minutes per day) Problem with multiple people on same note

I feel like I’m playing the worst video game ever • Do you use SmartLinks and SmartPhrase in charting? Why/Why not?

Yeah, you almost have to I use them for neuro exam, trauma exam MiChart doesn’t document what we want to see always Generic patients

• Do you use advanced features such as OrderSets, Macros, etc.? Why/Why not? I don’t use the Preference List (Still not great at that) I’m afraid I’ll forget what I put

in there and then stuff will get hidden. I prefer to not use OrderSets because I forget things if I do. Or I order unnecessary

things with OrderSets For protocol patients, OrderSets are nice. OrderSets might lead to ordering things you don’t need just because it’s included

28

Sometimes I have problems seeing all of the orders I just put in • What steps do you take if you have difficulty performing a task in MiChart-ASAP?

Swearing, being angry at the computer I haven’t found a good way to problem solve, I just try to work around it I ask co-workers a lot Sometimes I look at what MiChart has to say about them i.e. Tips and Videos (I do

send MiChart-ASAP the email) I didn’t know there was a Help Desk. It’s rarely helpful in real time. I have called the

Help Desk once for a computer related issue and they corrected my problem.

MiChart opinions: • What is still hard to do with MiChart-ASAP?

Searching for anything in Orders If you misspell it doesn’t correct at all Synonyms in my head are not the same as MiChart synonyms Old patient information, past information that is I settle for less than the amount of past information I would like to collect on a patient

because it’s hard I don’t know how to trend. I know how to trend some things, but not labs. Text results are hard to trend Can’t extend time period in Results Review

• What is easy to do with MiChart-ASAP? I love the expected referral on MiChart. I can put in orders, medication, initial bed

slip Orders are faster than hand writing at bedside. Can see everything at once Discharge process is very nice. Make referrals on same page, print everything all at

once. (Search is still an issue for Discharge Handouts, doesn’t recommend things that should be included. Have to go through whole list scrolling)

• What could make you more efficient with MiChart-ASAP? Real time dictation, voice recognition. We have to go through and edit all of my notes;

I spend at least an hour each night editing my notes for the day. If it were real time I could edit on the fly

I don’t dictate anymore. I type everything up. Dictation takes too long to pop up because I like to see it come up right away and edit it. Otherwise, I have to wait for the dictation to come back up. Dictation creates a two-step process I have to dictate, then come back and edit it, and sign off. I’m not a fast typist, but if I type it’s still faster than using dictation because I don’t have to go into the system twice.

Use of Dragon Dictation is of interest. Future version will be IPhone, IPad ready... but ASAP functionality will be pretty

limited. Wouldn’t be able to dictate or create note. You can maybe put in orders. If one person is in the chart now you can’t edit the chart. However, sometimes the

nurse is putting in vitals and I want to put in Orders. Can’t be within same function at the same time. If you’re in discharge and leave your

desk you’re technically in discharge so then the nurse can’t discharge the patient because you have that function open

Especially in trauma situations

29

• What do you like/dislike about MiChart-ASAP? Key past medical history on Clinical Snapshot Summary list can be bulky and not very accurate Scrolling through Clinical Snapshot not a problem, some people mind it though Crawl/Search system for Orders is poor. Have to type it in how MiChart recognizes it. 3 in 3 search system, start first two words, ex. APP SPL for Apply Splint Overall I think Epic is pretty reliable

• What would make creating a note faster?

Individual function to create note “Your Style”. Would have your macros already built into it. Some people put same 2 markers in for every note. I have to leave the note to get the past medical history in the note; I have to remember or remember to do it later. I have to leave the note and reopen it to add that past medical history.

If they have an extensive medical history you have to scroll all the way down to the bottom.

I wish I could move past Medical, Social, Surgical, History order Could create a difference between the way you create the note and the way the note is

displayed in its final form • If you have a preference for the way you create a note should be able to do it

in the way you want to • Then everyone’s data will reorganized in the same way no matter what way

you put it in. Reader preference would be how it’s displayed. Reader and writer preference may not match.

• As a Note creator you have scroll through multiple pages somewhere to get where you want to be. Should have the things you need to access at the top when you’re creating, that would be easier.

• Additional comments/questions I never edit past medical history, I hope the clinics and triage are doing that. For

residents and faculty that takes too much time. I would put “asthma” in HPI instead of Problem List. Edit home medications: Never, Frequently, Cross out sometimes if they aren’t taking

something, edit it in the Note but do not do it upstream in Medication section. I feel like these things should be done upfront.

30

Appendix G: Nurse MiChart Self -Assessment Feature Usage Rankings Note: The question “I have added buttons to the SnapShot report toolbar” was a Yes or No question. Therefore its scale is from 0 (No) to 1 (Yes).

All Question Text Mean St Dev I add medications to the patient’s home medication list when appropriate. 4.493333 0.777615 I use OrderSets for NIPCO's. I select only the appropriate items and can add orders on the fly. 4.013333 1.046143 I edit/correct my documentation when needed before completing a patient’s chart. 3.893333 1.072675 I delete medications from the patient’s home medication list when appropriate. 3.773333 1.447956 I review my documentation before completing a patient’s chart. 3.72 1.145614 I press ENTER after typing a search term instead of clicking Accept or Search. 3.64 1.290715 I use the history tab to enter and/or edit patient’s past medical history. 3.32 1.685871 I use the history tab to enter and/or edit patient’s past surgical history. 3.266667 1.750161 I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” 3.053333 1.895181 I press TAB to move between fields, options, and buttons rather than using the mouse. 2.853333 1.494977 I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals) 2.84 1.660674 I use the history tab to enter and/or edit patient’s past social history. 2.746667 1.778956 I access the ED dashboard to view reports. 2.68 1.534821 I sort by clicking column headers to find information in places like Chart Review. 2.653333 1.563894 I verify vitals directly in MiChart-ASAP instead of manually typing 2.6 1.542199 I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports, Epic Menu. 2.36 1.624475 I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 2.333333 1.545117 I use the history tab to enter and/or edit patient’s past family history 2.2 1.83067 I start visit notes in the NoteWriter and add comments to specific findings or entire sections as needed. 1.733333 1.788351 I search for things like orders, reports, and recipients by completion matching on partial words. 1.626667 1.642126 I use the Trauma and Code Narrator solely to document trauma cases without taking notes on paper. 1.533333 1.287501 I use SmartPhrases and SmartLinks in charting. 1.32 1.396521 I create personal SmartPhrases (Also known as dot phrases) 1.12 1.394584 I access MiChart-ASAP Tip Sheets and videos if I have questions. 0.946667 1.101514 I have added buttons to the SnapShot report toolbar 0.146667 0.356156

High Users Question Text Mean St Dev I press ENTER after typing a search term instead of clicking Accept or Search. 4.9 0.316228 I add medications to the patient’s home medication list when appropriate. 4.9 0.316228 I use OrderSets for NIPCO's. I select only the appropriate items and can add 4.7 0.674949

31

orders on the fly.

I use the history tab to enter and/or edit patient’s past medical history. 4.6 0.516398 I use the history tab to enter and/or edit patient’s past surgical history. 4.6 0.516398 I access the ED dashboard to view reports. 4.5 0.527046 I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” 4.4 1.264911 I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports, Epic Menu. 4.4 0.699206 I press TAB to move between fields, options, / and buttons rather than using the mouse. 4.3 1.251666 I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals) 4.3 0.823273 I review my documentation before completing a patient’s chart. 4.2 0.632456 I edit/correct my documentation when needed before completing a patient’s chart. 4.2 0.632456 I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 4.1 0.994429 I delete medications from the patient’s home medication list when appropriate. 3.9 1.852926 I use the history tab to enter / and/or edit patient’s past social history. 3.7 1.567021 I start visit notes in the NoteWriter and add comments to specific findings or entire sections as needed. 3.6 1.264911 I sort by clicking column headers to find information in places like Chart Review. 3.5 1.581139 I verify vitals directly in MiChart-ASAP instead of manually typing 3.4 1.173788 I search for things like orders, reports, and recipients by completion matching on partial words. 3 1.885618 I use SmartPhrases and SmartLinks in charting. 3 1.414214 I create personal SmartPhrases (Also known as dot phrases.) 3 1.632993 I use the history tab to enter / and/or edit patient’s past family history. 3 2 I use the Trauma and Code Narrator solely to document trauma cases without taking notes on paper. 2 1.154701 I access MiChart-ASAP Tip Sheets and videos if I have questions. 1.4 0.966092 I have added buttons to the SnapShot report toolbar 0.3 0.483046

Medium Users Question Text Mean St Dev I add medications to the patient’s home medication list when appropriate. 4.446429 0.760852 I edit/correct my documentation when needed before completing a patient’s chart. 4.035714 0.913819 I use OrderSets for NIPCO's. I select only the appropriate items and can add orders on the fly. 4.017857 0.999838 I delete medications from the patient’s home medication list when appropriate. 3.839286 1.358834 I review my documentation before completing a patient’s chart. 3.839286 1.022951 I press ENTER after typing a search term / instead of clicking Accept or Search. 3.5 1.190874 I use the history tab to enter and/or edit patient’s past medical history. 3.321429 1.608016 I use the history tab to enter and/or edit patient’s past surgical history. 3.321429 1.641586 I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” 2.964286 1.848481 I use the history tab to enter and/or edit patient’s past social history. 2.857143 1.710168

32

I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals) 2.821429 1.562133 I sort by clicking column headers to find information in places like Chart Review. 2.767857 1.401182 I press TAB to move between fields, options, and buttons rather than using the mouse. 2.678571 1.308811 I access the ED dashboard to view reports. 2.642857 1.380758 I verify vitals directly in MiChart-ASAP instead of manually typing 2.571429 1.570838 I use the history tab to enter and/or edit patient’s past family history. 2.339286 1.771253 I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 2.321429 1.349844 I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports, Epic Menu. 2.178571 1.478416 I start visit notes in the NoteWriter and add comments to specific findings or entire sections as needed. 1.571429 1.714719 I search for things like orders, reports, and recipients by completion matching on partial words. 1.553571 1.536462 I use the Trauma and Code Narrator solely to document trauma cases without taking notes on paper. 1.553571 1.32005 I use SmartPhrases and SmartLinks in charting. 1.178571 1.252011 I access MiChart-ASAP Tip Sheets and videos if I have questions. 1.017857 1.136039 I create personal SmartPhrases. (Also known as dot phrases.) 0.928571 1.157696 I have added buttons to the SnapShot report toolbar. 0.142857 0.353094

Low Users Question Text Mean St Dev I edit add medications to the patient’s home medication list when appropriate. 4.333333 1.118034 I use OrderSets for NIPCO's. I select only the appropriate items and can add orders on the fly. 3.222222 1.20185 I edit delete medications from the patient’s home medication list when appropriate. 3.222222 1.563472 I press ENTER after typing a search term instead of clicking Accept or Search. 3.111111 1.763834 I edit/correct my documentation when needed before completing a patient’s chart. 2.666667 1.581139 I review my documentation before completing a patient’s chart. 2.444444 1.509231 I press TAB to move between fields, options, and buttons rather than using the mouse. 2.333333 2 I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” 2.111111 2.14735 I verify vitals directly in MiChart-ASAP instead of manually typing 1.888889 1.452966 I use the history tab to enter and/or edit patient’s past medical history. 1.888889 1.964971 I use the history tab to enter / and/or edit patient’s past surgical history. 1.444444 1.878238 I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals) 1.333333 1.658312 I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports, Epic Menu. 1.222222 1.394433 I sort by clicking column headers to find information in places like Chart Review. 1 1.5 I use the history tab to enter and/or edit patient’s past social history. 1 1.322876 I access the ED dashboard to view reports. 0.888889 0.781736

33

I use the Trauma and Code Narrator solely to document trauma cases without taking notes on paper. 0.888889 1.054093 I start visit notes in the NoteWriter and add comments to specific findings or entire sections as needed. 0.666667 1.322876 I search for things like orders, reports, and recipients by completion matching on partial words. 0.555556 1.013794 I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 0.444444 0.726483 I use the history tab to enter and/or edit patient’s past family history. 0.444444 0.726483 I use SmartPhrases and SmartLinks in charting. 0.333333 0.5 I create personal SmartPhrases (Also known as dot phrases) 0.222222 0.440959 I have added buttons to the SnapShot report toolbar 0 0 I access MiChart-ASAP Tip Sheets and videos if I have questions. 0 0

34

Appendix H: Physician Assistant MiChart Self -Assessment Feature Usage Rankings All Question Text Mean St Dev I find and view patient images and scans on the Media tab in Chart Review. 4 1 When using NoteWriter, I use the Review of Systems function. 3.888889 1.691482 When using NoteWriter, I use the Physical Exam tab. 3.888889 1.691482 I use the Facility List and Database Lookup tabs to find orders that aren’t on my preference list. 3.555556 2.006932 I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 3.444444 1.333333 I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals). 3.333333 1.118034 When using NoteWriter, I add comments to specific findings or entire sections as needed. 3.222222 1.922094 I press ENTER after typing a search term instead of clicking Accept or Search. 3.111111 0.600925 I review patient information using CareWeb (Excluding access through MiChart-ASAP). 3.111111 1.615893 When using NoteWriter, I delete or edit Smartlinks. 3 1.581139 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical history-Medical. 3 1.224745 I create and share macros for typical exams. 2.888889 1.615893 When using NoteWriter, I create personal templates for documentation. 2.777778 1.922094 I use OrderSets for initial orders. I select only the appropriate items and can add orders on the fly. 2.777778 1.394433 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical history-Social. 2.777778 1.301708 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical history-Surgical. 2.555556 1.333333 I sort by clicking column headers to find information in places like Chart Review and In Basket. 2.333333 1.5 I use SmartPhrases and SmartLinks in charting. 2.333333 1.732051 I create personal SmartPhrases (Also known as dot phrases). 2.111111 1.900292 I access the ED dashboard to view reports. 2.111111 1.615893 I personalize things like the main toolbar, activity tabs, navigators. 1.888889 1.763834 I edit (add and/or delete) medications from the patient’s home medication list when appropriate-Add. 1.888889 1.615893 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical history-Family. 1.777778 1.394433 I edit (add and/or delete) medications from the patient’s home medication list when appropriate-Delete. 1.777778 1.563472 I use time and date shortcuts, such as “n”, “t-1”, “m+2” and “y+1”. 1.555556 1.740051 I add orders to my preference list. I can edit orders’ display names so that they’re easier to find. 1.555556 1.509231 I access MiChart-ASAP Tip Sheets and videos if I have questions. 1.222222 1.20185 When I access CareWeb (excluding access through MiChart-ASAP) I use CareWeb to review. 1.142857 1.345185

35

I have added buttons to the SnapShot report toolbar. 0.222222 0.440959 High Question Text Mean St Dev When I access CareWeb (excluding access through MiChart-ASAP) I use CareWeb to review. 0 N/A I use time and date shortcuts, such as “n”, “t-1”, “m+2”, and “y+1”. 5 N/A I find and view patient images and scans on the Media tab in Chart review. 5 N/A When using NoteWriter, I add comments to specific findings or entire sections as needed. 5 N/A When using NoteWriter, I create personal templates for documentation. 5 N/A When using NoteWriter, I use the Review of Systems function. 5 N/A When using NoteWriter, I use the Physical Exam tab. 5 N/A I use SmartPhrases and Smartlinks in charting. 5 N/A I create personal SmartPhrases (Also known as dot phrases). 5 N/A I use the Facility List and Database Lookup tabs to find orders that aren’t on my preference list. 5 N/A I use OrderSets for initial orders. I select only the appropriate items and can add orders on the fly. 5 N/A I access the ED dashboard to view reports. 5 N/A I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical history-Medical. 5 N/A I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical history-Social. 5 N/A I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical history-Family. 5 N/A I use the history tab to enter and/or edit patient’s past medical social, family, and surgical history-Surgical. 5 N/A I edit (add and/or delete) medications from the patient’s home medication list when appropriate-Add. 5 N/A I edit (add and/or delete) medications from the patient’s home medication list when appropriate-Delete. 5 N/A I personalize things like the main toolbar, activity tabs, navigators, Chart Review. 4 N/A I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 4 N/A I create and share macros for typical exams. 4 N/A I add orders to my preference list. I can edit orders’ display names so that they’re easier to find. 4 N/A I press ENTER after typing a search term instead of clicking Accept or Search. 3 N/A I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals). 3 N/A When using NoteWriter, I delete or edit SmartLinks. 3 N/A I access MiChart-ASAP Tip Sheets and videos if I have questions. 3 N/A I sort by clicking column headers to find information in places like Chart Review and In Basket. 1 N/A I review patient information using CareWeb (Excluding access through MiChart-ASAP). 1 N/A I have added buttons to the SnapShot report toolbar. 1 N/A

36

Medium Question Text Mean St Dev When using NoteWriter, I use the Review of Systems function. 4.142857 1.46385 When using NoteWriter, I use the Physical Exam tab. 4.142857 1.46385 I find and view patient images and scans on the Media tab in Chart Review. 3.857143 1.069045 I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 3.714286 1.112697 I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals) 3.285714 1.253566 When using NoteWriter, I add comments to specific findings or entire sections as needed. 3.285714 1.889822 I press ENTER after typing a search term instead of clicking Accept or Search. 3.142857 0.690066 I review patient information using CareWeb. (Excluding access through MiChart-ASAP) 3.142857 1.46385 When using NoteWriter, I delete or edit SmartLinks. 3.142857 1.772811 I use the Facility List and Database Lookup tabs to find orders that aren’t on my preference list. 3.142857 2.115701 I create and share macros for typical exams. 3 1.632993 When using NoteWriter, I create personal templates for documentation. 2.714286 1.889822 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical history-Medical. 2.714286 1.112697 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical history-Social. 2.714286 0.95119 I sort by clicking column headers to find information in places like Chart Review and In Basket. 2.571429 1.618347 I use OrderSets for initial orders. I select only the appropriate items and can add orders on the fly 2.571429 1.272418 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical history-Surgical. 2.428571 0.9759 I use SmartPhrases and SmartLinks in charting. 2.142857 1.573592 I create personal SmartPhrases (Also known as dot phrases) 1.857143 1.772811 I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports. 1.714286 1.799471 I access the ED dashboard to view reports. 1.714286 1.380131 I edit (add and/or delete) medications from the patient’s home medication list when appropriate –Add. 1.571429 1.272418 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical history-Family. 1.428571 0.786796 I edit (add and/or delete) medications from the patient’s home medication list when appropriate –Delete. 1.428571 1.133893 When I access CareWeb (excluding access through MiChart-ASAP) I use CareWeb to review. 1.333333 1.36626 I add orders to my preference list. I can edit orders’ display names so that they’re easier to find. 1.285714 1.380131 I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” 1.142857 1.345185 I access MiChart-ASAP Tip Sheets and videos if I have questions. 1 1.154701 I have added buttons to the SnapShot report toolbar. 0.142857 0.377964

37

Low Question Text Mean St Dev I review patient information using CareWeb. (Excluding access through MiChart-ASAP) 5 N/A I use the Facility List and Database Lookup tabs to find orders that aren’t on my preference list. 5 N/A I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals) 4 N/A I find and view patient images and scans on the Media tab in Chart Review. 4 N/A I press ENTER after typing a search term instead of clicking Accept or Search. 3 N/A I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Medical. 3 N/A I sort by clicking column headers to find information in places like Chart Review and In Basket. 2 N/A When using NoteWriter, I delete or edit SmartLinks. 2 N/A I use OrderSets for initial orders. I select only the appropriate items and can add orders on the fly 2 N/A I access the ED dashboard to view reports. 2 N/A I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” 1 N/A I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports. 1 N/A I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 1 N/A When using NoteWriter, I add comments to specific findings or entire sections as needed. 1 N/A When using NoteWriter, I create personal templates for documentation. 1 N/A When using NoteWriter, I use the Review of Systems function. 1 N/A When using NoteWriter, I use the Physical Exam tab. 1 N/A I create and share macros for typical exams. 1 N/A I use SmartPhrases and SmartLinks in charting. 1 N/A I create personal SmartPhrases (Also known as dot phrases). 1 N/A I add orders to my preference list. I can edit orders’ display names so that they’re easier to find. 1 N/A I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Social. 1 N/A I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Family. 1 N/A I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Surgical. 1 N/A I edit (add and/or delete) medications from the patient’s home medication list when appropriate –Add. 1 N/A I edit (add and/or delete) medications from the patient’s home medication list when appropriate –Delete. 1 N/A I access MiChart-ASAP Tip Sheets and videos if I have questions. 1 N/A When I access CareWeb (excluding access through MiChart-ASAP) I use CareWeb to review. 0 N/A I have added buttons to the SnapShot report toolbar. 0 N/A

38

Appendix I: Resident MiChart Self -Assessment Feature Usage Rankings All Question Text Mean St Dev When using NoteWriter, I add comments to specific findings or entire sections as needed. 3.740741 0.984206 I press ENTER after typing a search term / instead of clicking Accept or Search. 3.666667 1.568929 I review patient information using CareWeb. (Excluding access through MiChart-ASAP) 3.592593 1.366052 I create personal SmartPhrases (Also known as dot phrases) 3.555556 1.42325 I create and share macros for typical exams. 3.518519 1.503083 I use the Facility List and Database Lookup tabs to find orders that aren’t on my preference list. 3.444444 1.694637 When using NoteWriter, I delete or edit SmartLinks. 3.407407 1.217161 I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 3.37037 1.334401 When using NoteWriter, I create personal templates for documentation. 3.296296 1.636479 I use SmartPhrases and SmartLinks in charting. 3.222222 1.395965 I sort by clicking column headers to find information in places like Chart Review and In Basket. 3.185185 1.41522 I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals) 3.111111 1.502135 When using NoteWriter, I use the Physical Exam tab. 2.814815 2.149008 I find and view patient images and scans on the Media tab in Chart Review. 2.740741 1.430239 When using NoteWriter, I use the Review of Systems function. 2.703704 2.015609 I use OrderSets for initial orders. I select only the appropriate items and can add orders on the fly. 2.703704 1.203035 I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports. 2.444444 1.527525 I access the ED dashboard to view reports. 2.222222 1.625123 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Social. 2.037037 1.556166 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Medical. 1.925926 1.439175 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Surgical. 1.666667 1.300887 I edit (add and/or delete) medications from the patient’s home medication list when appropriate –Delete. 1.666667 1.414214 I add orders to my preference list. I can edit orders’ display names so that they’re easier to find. 1.555556 1.502135 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Family. 1.518519 1.369176 I edit (add and/or delete) medications from the patient’s home medication list when appropriate –Add. 1.518519 1.396985 When / I access CareWeb (excluding access through MiChart-ASAP) I use CareWeb to review. 1.44 1.121011 I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” 0.962963 0.979854 I access MiChart-ASAP Tip Sheets and videos if I have questions. 0.740741 0.902671 I have added buttons to the SnapShot report toolbar. 0.444444 0.50637

39

High Question Text Mean St Dev I sort by clicking column headers to find information in places like Chart Review and In Basket. 5 0 I review patient information using CareWeb. (Excluding access through MiChart-ASAP) 5 0 I create personal SmartPhrases (Also known as dot phrases) 5 0 I press ENTER after typing a search term instead of clicking Accept or Search. 4.5 0.707107 I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 4.5 0.707107 I find and view patient images and scans on the Media tab in Chart Review. 4.5 0.707107 When using NoteWriter, I add comments to specific findings or entire sections as needed. 4.5 0.707107 When using NoteWriter, I delete or edit SmartLinks. 4.5 0.707107 When using NoteWriter, I create personal templates for documentation. 4.5 0.707107 I edit (add and/or delete) medications from the patient’s home medication list when appropriate –Add. 4.5 0.707107 I edit (add and/or delete) medications from the patient’s home medication list when appropriate –Delete. 4.5 0.707107 I create and share / macros for typical exams. 4 1.414214 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Medical. 4 0 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Social. 4 0 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Family. 4 0 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Surgical. 4 0 I use SmartPhrases and SmartLinks in charting. 3.5 0.707107 I use OrderSets for initial orders. I select only the appropriate items and can add orders on the fly. 3.5 0.707107 When using NoteWriter, I use the Physical Exam tab. 3 2.828427 I use the Facility List and Database Lookup tabs to find orders that aren’t on my preference list. 3 2.828427 I access the ED dashboard to view reports. 3 2.828427 I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports. 2.5 0.707107 When using NoteWriter, I use the Review of Systems function. 2.5 2.12132 I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” 2 1.414214 I add orders to my preference list. I can edit orders’ display names so that they’re easier to find. 2 2.828427 I access MiChart-ASAP Tip Sheets and videos if I have questions. 2 1.414214 When I access CareWeb (excluding access through MiChart-ASAP) I use CareWeb to / review. 1.5 2.12132 I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals) 1.5 2.12132 I have added buttons to the SnapShot report toolbar. 0.5 0.707107

40

Medium Question Text Mean St Dev I press ENTER after typing a search term/ instead of clicking Accept or Search. 3.727273 1.485864 I use the Facility List and Database Lookup tabs to find orders that aren’t on my preference list. 3.727273 1.485864 I create and share macros for typical exams. 3.681818 1.286796 When using NoteWriter, I add comments to specific findings or entire sections as needed. 3.590909 1.007547 When using NoteWriter, I delete or edit SmartLinks. 3.545455 0.911685 I create personal SmartPhrases (Also known as dot phrases) 3.5 1.47196 I review patient information using CareWeb. (Excluding access through MiChart-ASAP) 3.454545 1.370689 I use SmartPhrases and SmartLinks in charting. 3.409091 1.259595 I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 3.318182 1.323285 I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals) 3.318182 1.358794 I sort by clicking column headers to find information in places like Chart Review and In Basket. 3.272727 1.241421 When using NoteWriter, I create personal templates for documentation. 3.272727 1.579084 When using NoteWriter, I use the Review of Systems function. 3 2 When using NoteWriter, I use the Physical Exam tab. 2.954545 2.081146 I use OrderSets for initial orders. I select only the appropriate items and can add orders on the fly. 2.772727 1.192509 I find and view patient images and scans on the Media tab in Chart Review. 2.681818 1.323285 I access the ED dashboard to view reports. 2.454545 1.438494 I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports. 2.409091 1.623022 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Social. 1.909091 1.540225 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Medical. 1.772727 1.377777 When I access CareWeb (excluding access through MiChart-ASAP) I use CareWeb to review. 1.55 1.099043 I add orders to my preference list. I can edit orders’ display names so that they’re easier to find. 1.545455 1.370689 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Surgical. 1.454545 1.143398 I edit (add and/or delete) medications from the patient’s home medication list when appropriate –Delete. 1.454545 1.184313 I edit (add and/or delete) medications from the patient’s home medication list when appropriate –Add. 1.318182 1.170525 I use the history tab to enter / and/or edit patient’s past medical, social, family, and surgical hist...-Family: 1.272727 1.202451 I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” 0.909091 0.921132 I access MiChart-ASAP Tip Sheets and videos if I have questions. 0.727273 0.827032 I have added buttons to the SnapShot report toolbar 0.454545 0.509647

41

Low Question Text Mean St Dev When using NoteWriter, I add comments to specific findings or entire sections as needed. 4.333333 0.57735 I review patient information using CareWeb. (Excluding access through MiChart-ASAP) 3.666667 1.527525 I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 3 1.732051 I create personal SmartPhrases (Also known as dot phrases) 3 1 I press ENTER after typing a search term instead of clicking Accept or Search. 2.666667 2.516611 I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports. 2.666667 1.527525 I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals) 2.666667 2.081666 When using NoteWriter, I create personal templates for documentation. 2.666667 2.516611 I find and view patient images and scans on the Media tab in Chart Review. 2 2 I create and share macros for typical exams. 2 2.645751 When using NoteWriter, I delete or edit SmartLinks. 1.666667 2.081666 When using NoteWriter, I use the Physical Exam tab. 1.666667 2.886751 I use SmartPhrases and SmartLinks in charting. 1.666667 2.081666 I use the Facility List and Database Lookup tabs to find orders that aren’t on my preference list. 1.666667 2.081666 I use OrderSets for initial orders. I select only the appropriate items and can add orders on the fly.. 1.666667 1.154701 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Medical. 1.666667 1.527525 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Social. 1.666667 1.527525 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Family. 1.666667 1.527525 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Surgical. 1.666667 1.527525 I sort by clicking column headers to find information in places like Chart Review and In Basket. 1.333333 1.154701 I add orders to my preference list. I can edit orders’ display names so that they’re easier to find. 1.333333 2.309401 I edit (add and/or delete) medications from the patient’s home medication list when appropriate -Delete 1.333333 1.527525 I edit (add and/or delete) medications from the patient’s home medication list when appropriate -Add 1 1 I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” 0.666667 1.154701 When / I access CareWeb (excluding access through MiChart-ASAP) I use CareWeb to review. 0.666667 0.57735 When using NoteWriter, I use the Review of Systems function. 0.666667 1.154701 I have added buttons to the SnapShot report toolbar 0.333333 0.57735 I access the ED dashboard to view reports. 0 0 I access MiChart-ASAP Tip Sheets and videos if I have questions. 0 0

42

Appendix J: Faculty MiChart Self -Assessment Feature Usage Rankings

All Question Text Mean St Dev While using the In-Basket function, I know which attestation tab to use for different circumstances... 4.5 1.082 When using NoteWriter, I use the Physical Exam tab. 4.083 1.461 When using NoteWriter, I add comments to specific findings or entire sections as needed. 3.944 1.013 While using the In-Basket function, I use protocol notes for patients on observation protocols. 3.806 1.754 I use the Facility List and Database Lookup tabs to find orders that aren’t on my preference list. 3.75 1.697 When using NoteWriter, I use the Review of Systems function. 3.639 1.726 I sort by clicking column headers to find information in places like Chart Review and In Basket. 3.5 1.424 I press ENTER after typing a search term instead of clicking Accept or Search. 3.417 1.481 When using NoteWriter, I delete or edit SmartLinks. 3.417 1.105 I use OrderSets for initial orders. I select only the appropriate items and can add orders on the fly... 3.083 1.538 I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals) 3 1.789 I find and view patient images and scans on the Media tab in Chart Review. 2.833 1.595 I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 2.806 1.527 I use SmartPhrases and SmartLinks in charting. 2.417 1.680 I access the ED dashboard to view reports. 2.333 1.690 I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports 2.25 1.500 I review patient information using CareWeb. (Excluding access through MiChart-ASAP) 2.139 1.477 I create personal SmartPhrases (Also known as dot phrases) 2.111 1.801 While using the In-Basket function, I send staff messages. 2.056 1.511 I create and share macros for typical exams. 2.028 1.844 When using NoteWriter, I create personal templates for documentation. 1.972 1.647 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Medical: 1.75 1.538 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Social: 1.75 1.519 While using the In-Basket function, I forward charts to other faculty members. 1.75 1.663 I edit (add and/or delete) medications from the patient’s home medication list when appropriate -Add: 1.694 1.636 I edit (add and/or delete) medications from the patient’s home medication list when appropriate -Delete: 1.694 1.546 I access MiChart-ASAP Tip Sheets and videos if I have questions. 1.667 1.373 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Family: 1.639 1.457 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Surgical: 1.639 1.515

43

I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” 1.556 1.681 I add orders to my preference list. I can edit orders’ display names so that they’re easier to find.... 1.472 1.502 While using the In-Basket function, I know what to do with deficiencies that are not mine. 0.611 0.494 I have added buttons to the SnapShot report toolbar 0.417 0.500

High Users Question Text Mean St Dev I use the Facility List and Database Lookup tabs to find orders that aren’t on my preference list. 5.000 0.000 While using the In-Basket function, I know which attestation tab to use for different circumstances... 5.000 0.000 I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports 4.600 0.548 When using NoteWriter, I add comments to specific findings or entire sections as needed. 4.600 0.548 I create personal SmartPhrases (Also known as dot phrases) 4.600 0.548 I use OrderSets for initial orders. I select only the appropriate items and can add orders on the fly... 4.600 0.548 While using the In-Basket function, I use protocol notes for patients on observation protocols. 4.600 0.548 When using NoteWriter, I use the Physical Exam tab. 4.400 0.894 I use SmartPhrases and SmartLinks in charting. 4.400 0.894 I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 4.200 0.837 I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals) 4.200 1.304 I find and view patient images and scans on the Media tab in Chart Review. 4.200 0.837 I create and share macros for typical exams. 4.200 1.095 I edit (add and/or delete) medications from the patient’s home medication list when appropriate -Add: 4.200 0.447 I edit (add and/or delete) medications from the patient’s home medication list when appropriate -Delete: 4.200 0.447 I press ENTER after typing a search term instead of clicking Accept or Search. 4.000 0.707 When using NoteWriter, I delete or edit SmartLinks. 4.000 1.000 When using NoteWriter, I use the Review of Systems function. 3.800 1.643 I access the ED dashboard to view reports. 3.800 0.837 I sort by clicking column headers to find information in places like Chart Review and In Basket. 3.600 1.517 When using NoteWriter, I create personal templates for documentation. 3.600 1.673 While using the In-Basket function, I send staff messages. 3.600 1.517 While using the In-Basket function, I forward charts to other faculty members. 3.600 1.517 I access MiChart-ASAP Tip Sheets and videos if I have questions. 3.600 1.140 I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” 3.200 1.483 I review patient information using CareWeb. (Excluding access through MiChart-ASAP) 3.200 1.789 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Medical: 3.200 1.483

44

I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Social: 3.000 1.225 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Surgical: 3.000 1.581 I add orders to my preference list. I can edit orders’ display names so that they’re easier to find.... 2.800 1.095 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Family: 2.800 1.304 While using the In-Basket function, I know what to do with deficiencies that are not mine. 1.000 0.000 I have added buttons to the SnapShot report toolbar 0.800 0.447

Medium Users Question Text Mean St Dev While using the In-Basket function, I know which attestation tab to use for different circumstances... 4.565 1.08 When using NoteWriter, I use the Physical Exam tab. 4.304 1.49 I sort by clicking column headers to find information in places like Chart Review and In Basket. 4.087 0.668 When using NoteWriter, I add comments to specific findings or entire sections as needed. 4.043 1.022 I use the Facility List and Database Lookup tabs to find orders that aren’t on my preference list. 3.87 1.576 When using NoteWriter, I use the Review of Systems function. 3.696 1.82 While using the In-Basket function, I use protocol notes for patients on observation protocols. 3.652 1.921 When using NoteWriter, I delete or edit SmartLinks. 3.478 1.201 I use OrderSets for initial orders. I select only the appropriate items and can add orders on the fly... 3.391 1.196 I press ENTER after typing a search term instead of clicking Accept or Search. 3.304 1.579 I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals) 3 1.859 I find and view patient images and scans on the Media tab in Chart Review. 3 1.537 I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 2.696 1.521 I access the ED dashboard to view reports. 2.652 1.555 I use SmartPhrases and SmartLinks in charting. 2.609 1.469 I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports, Epic Me... 2.13 1.29 I create personal SmartPhrases (Also known as dot phrases) 2 1.651 I review patient information using CareWeb. (Excluding access through MiChart-ASAP) 1.957 1.397 When using NoteWriter, I create personal templates for documentation. 1.957 1.609 While using the In-Basket function, I send staff messages. 1.913 1.311 I create and share macros for typical exams. 1.87 1.792 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Social: 1.826 1.586 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Medical: 1.783 1.536 I use the history tab to enter and/or edit patient’s past medical, social, family, and 1.739 1.514

45

surgical hist...-Family: I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Surgical: 1.652 1.526 While using the In-Basket function, I forward charts to other faculty members. 1.609 1.5 I access MiChart-ASAP Tip Sheets and videos if I have questions. 1.609 1.076 I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” 1.565 1.647 I edit (add and/or delete) medications from the patient’s home medication list when appropriate -Add: 1.522 1.473 I edit (add and/or delete) medications from the patient’s home medication list when appropriate -Delete: 1.522 1.344 I add orders to my preference list. I can edit orders’ display names so that they’re easier to find.... 1.435 1.59 While using the In-Basket function, I know what to do with deficiencies that are not mine. 0.522 0.511 I have added buttons to the SnapShot report toolbar 0.391 0.499

Low Users Question Text Mean St Dev While using the In-Basket function, I know which attestation tab to use for different circumstances... 4 1.309 While using the In-Basket function, I use protocol notes for patients on observation protocols. 3.75 1.753 I press ENTER after typing a search term instead of clicking Accept or Search. 3.375 1.598 When using NoteWriter, I use the Review of Systems function. 3.375 1.685 When using NoteWriter, I add comments to specific findings or entire sections as needed. 3.25 0.886 When using NoteWriter, I use the Physical Exam tab. 3.25 1.488 When using NoteWriter, I delete or edit SmartLinks. 2.875 0.641 I use the Facility List and Database Lookup tabs to find orders that aren’t on my preference list. 2.625 1.996 I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 2.25 1.488 I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals) 2.25 1.581 I review patient information using CareWeb. (Excluding access through MiChart-ASAP) 2 1.414 I sort by clicking column headers to find information in places like Chart Review and In Basket. 1.75 1.669 I find and view patient images and scans on the Media tab in Chart Review. 1.5 1.195 While using the In-Basket function, I send staff messages. 1.5 1.604 I use OrderSets for initial orders. I select only the appropriate items and can add orders on the fly... 1.25 1.165 I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports, Epic Me... 1.125 0.641 I create and share macros for typical exams. 1.125 1.356 When using NoteWriter, I create personal templates for documentation. 1 0.926 While using the In-Basket function, I forward charts to other faculty members. 1 1.512 I create personal SmartPhrases (Also known as dot phrases) 0.875 1.126 I add orders to my preference list. I can edit orders’ display names so that they’re 0.75 0.886

46

easier to find.... I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Medical: 0.75 0.707 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Social: 0.75 0.707 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Surgical: 0.75 0.707 I use SmartPhrases and SmartLinks in charting. 0.625 0.518 I use the history tab to enter and/or edit patient’s past medical, social, family, and surgical hist...-Family: 0.625 0.518 I edit (add and/or delete) medications from the patient’s home medication list when appropriate -Add: 0.625 0.744 I edit (add and/or delete) medications from the patient’s home medication list when appropriate -Delete: 0.625 0.518 While using the In-Basket function, I know what to do with deficiencies that are not mine. 0.625 0.518 I access MiChart-ASAP Tip Sheets and videos if I have questions. 0.625 1.061 I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” 0.5 1.069 I access the ED dashboard to view reports. 0.5 0.756 I have added buttons to the SnapShot report toolbar 0.25 0.463

47

Appendix K: Recommended Training Focuses Note: Features are ranked from most important to least important training focuses. All Roles Features Recommended for Additional Training: I access MiChart-ASAP Tip Sheets and videos if I have questions. I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports, Epic Menu, etc. I access the ED dashboard to view reports. I use the history tab to enter and/or edit patient’s past social history. I use the history tab to enter and/or edit patient’s past family history.

Nurses Features Recommended for Additional Training: I access MiChart-ASAP Tip Sheets and videos if I have questions. I create personal SmartPhrases (Also known as dot phrases) I use SmartPhrases and SmartLinks in charting. I use the Trauma and Code Narrator solely to document trauma cases without taking notes on paper. I search for things like orders, reports, and recipients by completion matching on partial words. I start visit notes in the NoteWriter and add comments to specific findings or entire sections as needed. I use the history tab to enter and/or edit patient’s past family history. I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports, Epic Menu. I verify vitals directly in MiChart-ASAP instead of manually typing I sort by clicking column headers to find information in places like Chart Review. I access the ED dashboard to view reports. I use the history tab to enter and/or edit patient’s past social history. I search reports in SnapShot (Ex: Patient timeline report, patient index, historical vitals) I press TAB to move between fields, options, and buttons rather than using the mouse.

PA Features Recommended for Additional Training: I access MiChart-ASAP Tip Sheets and videos if I have questions. I add orders to my preference list. I can edit orders’ display names so that they’re easier to find. I use time and date / shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” I edit delete medications from the patient’s home medication list when appropriate. I use the history tab to enter and/or edit patient’s past family history. I add medications from the patient’s home medication list when appropriate. I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports, Epic Menu. I access the ED dashboard to view reports. I create personal SmartPhrases (Also known as dot phrases)

48

I use SmartPhrases and SmartLinks in charting. I sort by clicking column headers to find information in places like Chart Review and In Basket. I use the history tab to enter and/or edit patient’s past surgical history. I use the history tab to enter and/or edit patient’s past social history. I use OrderSets for initial orders. I select only the appropriate items and can add orders on the fly. When using NoteWriter, I create personal templates for documentation. I create and share macros for typical exams.

Residents Features Recommended for Additional Training: I access MiChart-ASAP Tip Sheets and videos if I have questions. I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” I add medications from the patient’s home medication list when appropriate. I use the history tab to enter and/or edit patient’s past family history. I add orders to my preference list. I can edit orders’ display names so that they’re easier to find. I delete medications from the patient’s home medication list when appropriate. I use the history tab to enter and/or edit patient’s past surgical history. I use the history tab to enter and/or edit patient’s past medical history. I use the history tab to enter and/or edit patient’s past social history. I access the ED dashboard to view reports. I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports, Epic Menu. I use OrderSets for initial orders. I select only the appropriate items and can add orders on the fly. When using NoteWriter, I use the Review of Systems function. I find and view patient images and scans on the Media tab in Chart Review. When using NoteWriter, I use the Physical Exam tab.

Faculty Features Recommended for Additional Training: While using the In-Basket function, I know what to do with deficiencies that are not mine. I add orders to my preference list. I can edit orders’ display names so that they’re easier to find. I use time and date shortcuts, such as “n,” “t-1,” “m+2,” and “y+1.” I use the history tab to enter and/or edit patient’s past surgical history. I use the history tab to enter and/or edit patient’s past family history. I access MiChart-ASAP Tip Sheets and videos if I have questions. I delete medications from the patient’s home medication list when appropriate. I add medications from the patient’s home medication list when appropriate. While using the In-Basket function, I forward charts to other faculty members. I use the history tab to enter and/or edit patient’s past social history. I use the history tab to enter and/or edit patient’s past medical history. When using NoteWriter, I create personal templates for documentation. I create and share macros for typical exams.

49

While using the In-Basket function, I send staff messages. I create personal SmartPhrases (Also known as dot phrases) I personalize things like the main toolbar, activity tabs, navigators, Chart Review reports, Epic Menu. I access the ED dashboard to view reports. I use SmartPhrases and SmartLinks in charting. I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. I find and view patient images and scans on the Media tab in Chart Review.

50

Appendix L: Top 3 Defining Features for each Role

Nurses Top 3 Defining Features

Feature: High User

Average Score Medium User Average Score

Low User Average Score

I use time and date shortcuts, such as “n”, “t-1”, “m+2”, and “y+1.” 4.4 3.0 2.1

I trend a patient’s previous lab results using tools such as flowsheets, graphs, or results. 4.1 2.3 0.4

I use SmartPhrases in charting 3 1.7 0.3

Physician Assistant Top 3 Defining Features

Feature: High User

Average Score Medium User Average Score

Low User Average Score

I create and share macros for typical exams. 5 3 1 I trend a patient’s previous lab results using tools

such as flowsheets, graphs, or results. 4 3.7 1 I create personal SmartPhrases. (Also known as dot

phrases.) 5 1.9 1

Resident Top 3 Defining Features

Feature: High User

Average Score Medium User Average Score

Low User Average Score

When using NoteWriter, I create personal templates for documentation 4.5 3.3 2.7

I create personal SmartPhrases. (Also known as dot phrases.) 5 3.5 3

When using NoteWriter, I delete or edit SmartLinks 4.5 3.5 1.7

Faculty Top 3 Defining Features

Feature: High User

Average Score Medium User Average Score

Low User Average Score

I create personal SmartPhrases. (Also known as dot phrases.) 4.6 2 0.9

I use SmartPhrases and SmartLinks in charting 4.4 2.6 0.6 I edit (add and/or delete) medications from the

patient’s home medication list. 4.2 1.5 0.6