earlywarningsystem ewsoutreach team (cco). organizations like the american heart association,...

Early Warning System�EWS��

��ICU2018/03/06��

RRS��

best known approaches have been namedMedical Emergency Team (MET), RapidResponse Team (RRT), or Critical CareOutreach Team (CCO). Organizations likethe American Heart Association, Institutefor Healthcare Improvement (1), and Soci-ety for Critical Care Medicine are promul-gating some form of RRS, and many hos-pitals are trying to implement them.Although the structure may vary, they aresimilar in that they rely on the promptidentification and treatment of acutely crit-ically ill hospital patients.

Because of growing worldwide interestin these systems, several authors (MD, KH,RB) convened the first International Con-ference on Medical Emergency Teams(ICMET) in Pittsburgh in June 2005. Thisgathering of experts in patient safety, hos-pital and critical care medicine, and METsprovided a unique opportunity to analyzethe state of knowledge in this field, developa consensus around the basic requirementsfor such a system, and frame a researchagenda. In this article we relate the findingsof the ICMET.

METHODS

ICMET borrowed several components fromexisting consensus methods (2). As in the Na-tional Institutes of Health (United States) andFrench national processes, this consensusconference was organized around a small setof key questions. ICMET chairpersons devel-

oped a set of four key questions, each withseveral subquestions for the panels to answer.The panelists were selected on the basis ofexpertise in patient safety (7) or experiencewith one of the three major forms of RRS andwere divided into four groups, each of whichtook the primary responsibility for developingthe answers to one key question and associ-ated subquestions.

Before the consensus conference, panel-ists distributed to their group a set of refer-ences, literature reviews, draft responses,and other materials relevant to answeringtheir questions. Studies reviewed were cho-sen by our consensus group organizers, fol-lowing a comprehensive literature search byone of the authors (SG). An annotated bib-liography was created (Appendix 2). (N.B.Citations in this text from the annotated bib-liography are indicated by author; other cita-tions not annotated are numeric). Our searchstrategy did not include unpublished data ordata presented in abstract form only. Precon-ference interactions encouraged panelists tobegin thinking about their questions and an-swers and to allow them to gain an under-standing of the other members’ approaches(3). Levels of evidence of studies were consid-ered in discussion (4, 5); ICMET recommen-dations were not graded. The ICMET did notaim to make quantitative distinctions (andtherefore did not make recommendations) be-tween the various models of METs. Instead,our review focused on collecting informationregarding the general characteristics that weidentified as critical to understanding how to

implement an emergency response mecha-nism within the hospital for patients in need,as well as the organizational features thatshould be reported in the future.

ICMET panelists were faculty members ofthe First International Conference on MedicalEmergency Team (MET) Responses: Prevent-ing Patient Crises, Protecting Patients in Cri-sis. The consensus panel met immediately fol-lowing the last day of this symposium. On day1 of the consensus conference, the panelistswere instructed on the goals of the conference.Each group reported its preconference workand, with the full group, created a commonunderstanding of each question.

For 2 days, the four panels drafted answers totheir questions. Twice each day they convened,and each panel presented its progress to thegroup. The panelists critically reviewed eachpanel’s responses. In addition, during these ses-sions, the group helped to focus panels and toremove any overlap of issues addressed by morethan one panel.

Following the final group meeting at the endof the second day, the conference adjourned. Inthe weeks immediately after the conference,each panel modified its statements in responseto the final input. Each group generated a re-port. The answers to all questions were thendistributed to panelists for final comments.

RESULTS

What Is an In-Hospital MedicalEmergency?

An in-hospital medical emergency oc-curs when a hospitalized patient has dete-riorated, physiologically and/or psycholog-ically, to the point where there is animminent risk of serious harm. Such pa-tients urgently require (often critical care)resources that are not readily available ornot being implemented. Thus, a mismatchbetween patient needs and resources avail-able is the hallmark of an in-hospitalemergency. Clinical resources includeknowledge, skills, equipment, and person-nel. A medical emergency should bethought of in a broad context, includingsurgical, obstetrical, mental health, pediat-ric, and other events. An institution mayderive a set of criteria to help personnelidentify patients who are at risk.

Is There Evidence That MedicalEmergencies Occur In-Hospital?

There is considerable evidence that medi-cal emergencies occur in hospitals (Hillmanet al., Resuscitation 2001, Hillman et al.2002, Kause et al. 2004). This evidence isderived from observational epidemiologic

Figure 1. Rapid Response System structure. When patients have critical unmet needs and as a resultare at risk for imminent danger, the afferent limb detects the event and triggers a systematic response.The response provides resources to stabilize and triage the patient to a location where services meetthe patient’s needs. Data are collected to determine event rate, resources needed, and outcomes andto enable an analysis of events to prevent or prepare for future events. An administrative mechanismis needed to oversee all components and to provide resources to facilitate the system. MET, medicalemergency team; RRT, rapid response team; CCO, critical care outreach.

2464 Crit Care Med 2006 Vol. 34, No. 9

6��71-'24# /(50�$53

6��7+5'��//!5,.)"/��

6��7• $53��

6��7%&*0��/&')/ �/��

Findings of the First Consensus Conference on Medical Emergency Teams.Crit Care Med 2006;34(9)

RRSC�+

best known approaches have been namedMedical Emergency Team (MET), RapidResponse Team (RRT), or Critical CareOutreach Team (CCO). Organizations likethe American Heart Association, Institutefor Healthcare Improvement (1), and Soci-ety for Critical Care Medicine are promul-gating some form of RRS, and many hos-pitals are trying to implement them.Although the structure may vary, they aresimilar in that they rely on the promptidentification and treatment of acutely crit-ically ill hospital patients.

Because of growing worldwide interestin these systems, several authors (MD, KH,RB) convened the first International Con-ference on Medical Emergency Teams(ICMET) in Pittsburgh in June 2005. Thisgathering of experts in patient safety, hos-pital and critical care medicine, and METsprovided a unique opportunity to analyzethe state of knowledge in this field, developa consensus around the basic requirementsfor such a system, and frame a researchagenda. In this article we relate the findingsof the ICMET.

METHODS

ICMET borrowed several components fromexisting consensus methods (2). As in the Na-tional Institutes of Health (United States) andFrench national processes, this consensusconference was organized around a small setof key questions. ICMET chairpersons devel-

oped a set of four key questions, each withseveral subquestions for the panels to answer.The panelists were selected on the basis ofexpertise in patient safety (7) or experiencewith one of the three major forms of RRS andwere divided into four groups, each of whichtook the primary responsibility for developingthe answers to one key question and associ-ated subquestions.

Before the consensus conference, panel-ists distributed to their group a set of refer-ences, literature reviews, draft responses,and other materials relevant to answeringtheir questions. Studies reviewed were cho-sen by our consensus group organizers, fol-lowing a comprehensive literature search byone of the authors (SG). An annotated bib-liography was created (Appendix 2). (N.B.Citations in this text from the annotated bib-liography are indicated by author; other cita-tions not annotated are numeric). Our searchstrategy did not include unpublished data ordata presented in abstract form only. Precon-ference interactions encouraged panelists tobegin thinking about their questions and an-swers and to allow them to gain an under-standing of the other members’ approaches(3). Levels of evidence of studies were consid-ered in discussion (4, 5); ICMET recommen-dations were not graded. The ICMET did notaim to make quantitative distinctions (andtherefore did not make recommendations) be-tween the various models of METs. Instead,our review focused on collecting informationregarding the general characteristics that weidentified as critical to understanding how to

implement an emergency response mecha-nism within the hospital for patients in need,as well as the organizational features thatshould be reported in the future.

ICMET panelists were faculty members ofthe First International Conference on MedicalEmergency Team (MET) Responses: Prevent-ing Patient Crises, Protecting Patients in Cri-sis. The consensus panel met immediately fol-lowing the last day of this symposium. On day1 of the consensus conference, the panelistswere instructed on the goals of the conference.Each group reported its preconference workand, with the full group, created a commonunderstanding of each question.

For 2 days, the four panels drafted answers totheir questions. Twice each day they convened,and each panel presented its progress to thegroup. The panelists critically reviewed eachpanel’s responses. In addition, during these ses-sions, the group helped to focus panels and toremove any overlap of issues addressed by morethan one panel.

Following the final group meeting at the endof the second day, the conference adjourned. Inthe weeks immediately after the conference,each panel modified its statements in responseto the final input. Each group generated a re-port. The answers to all questions were thendistributed to panelists for final comments.

RESULTS

What Is an In-Hospital MedicalEmergency?

An in-hospital medical emergency oc-curs when a hospitalized patient has dete-riorated, physiologically and/or psycholog-ically, to the point where there is animminent risk of serious harm. Such pa-tients urgently require (often critical care)resources that are not readily available ornot being implemented. Thus, a mismatchbetween patient needs and resources avail-able is the hallmark of an in-hospitalemergency. Clinical resources includeknowledge, skills, equipment, and person-nel. A medical emergency should bethought of in a broad context, includingsurgical, obstetrical, mental health, pediat-ric, and other events. An institution mayderive a set of criteria to help personnelidentify patients who are at risk.

Is There Evidence That MedicalEmergencies Occur In-Hospital?

There is considerable evidence that medi-cal emergencies occur in hospitals (Hillmanet al., Resuscitation 2001, Hillman et al.2002, Kause et al. 2004). This evidence isderived from observational epidemiologic

Figure 1. Rapid Response System structure. When patients have critical unmet needs and as a resultare at risk for imminent danger, the afferent limb detects the event and triggers a systematic response.The response provides resources to stabilize and triage the patient to a location where services meetthe patient’s needs. Data are collected to determine event rate, resources needed, and outcomes andto enable an analysis of events to prevent or prepare for future events. An administrative mechanismis needed to oversee all components and to provide resources to facilitate the system. MET, medicalemergency team; RRT, rapid response team; CCO, critical care outreach.

2464 Crit Care Med 2006 Vol. 34, No. 9

1Event Detection2

• ��!:H�"A�( ��7�'• �5=/�( ��J&:H�( ��C��F�• ��D/PRTX,>KQP\_V�B�J&:H• Early Warning System aEWSb

1Triggering Mechanism2

• %��GV]M`C�.7O_Y^>3<;?9=F/RRS7��8I=4A4��73H0

• MET ��->��J�;9;�$B�930%96')8I=4A40aHillman 2005. MERIT studyb

• ')��GS`[EC*!��C��7�'0aZNTVL`WGPUTL`/ZPR`A@C�#b

Findings of the First Consensus Conference on Medical Emergency Teams.Crit Care Med 2006;34(9)

�RRS,��+�&)�• 3:<8,��+-936;53<,�*&)��"�1

–��+�16��+4;7+�0!,� �"��%2) (

Developing strategies to prevent inhospital cardiac arrest: analyzing responses of physicians and nurses in the hours before the event,Crit Care Med.1994 Feb;22(2)

• #2-��&(RRS"�1��+)/�'1Mature rapid response system and potentially avoidable cardiopulmonary arrests in hospital. Qual Saf Health Care. 2007;16(4)

1758

CHEST Recent Advances in Chest Medicine

Recent Advances in Chest Medicine

Adverse events on the wards, such as cardiac arrest and death, are rarely sudden and are often heralded

by abnormal vital signs hours before the event. 1-5 How-ever, these signs are often missed or not acted on appro-priately, even in hospitals with mature rapid response systems. 6 In 2007, the National Institute for Health and Clinical Excellence recommended that physio-logic track and trigger systems should be used to mon-itor all adult patients in acute hospital settings. 7 These systems, also known as early warning scores, typically

use vital sign thresholds to identify at-risk patients. Currently there are . 100 different published track and trigger systems, 8-10 most of which are hospital-specifi c modifi cations of the original Early Warning Score, 11 developed using expert opinion, and have dem-onstrated variable levels of reliability, validity, and use-fulness. 12 In the past few years, there has been a move toward scientifi cally derived risk scores and unifying the criteria across hospitals in some countries. 13-15

In this review, we discuss risk scores for use on the general inpatient wards, with a focus on those that were developed recently. In addition, we compare newly developed systems with more established risk scores, such as the Modifi ed Early Warning Score (MEWS) 16 and the criteria used in the Medical Early Response Intervention and Therapy (MERIT) trial 17 in our data-base of . 59,000 ward admissions. We limit our dis-cussion to objective vital sign-based risk scores designed for adult patients on the general hospital wards, but it should be noted that much progress has been made

Patients who suffer adverse events on the wards, such as cardiac arrest and death, often have vital sign abnormalities hours before the event. Early warning scores have been developed with the aim of identifying clinical deterioration early and have been recommended by the National Insti-tute for Health and Clinical Excellence. In this review, we discuss recently developed and vali-dated risk scores for use on the general inpatient wards. In addition, we compare newly developed systems with more established risk scores such as the Modifi ed Early Warning Score and the cri-teria used in the Medical Early Response Intervention and Therapy (MERIT) trial in our data-base of . 59,000 ward admissions. In general we found the single-parameter systems, such as the MERIT criteria, to have the lowest predictive accuracy for adverse events, whereas the aggregate weighted scoring systems had the highest. The Cardiac Arrest Risk Triage (CART) score was best for predicting cardiac arrest, ICU transfer, and a composite outcome (area under the receiver operating characteristic curve [AUC], 0.83, 0.77, and 0.78, respectively), whereas the Standard-ized Early Warning Score, VitalPAC Early Warning Score, and CART score were similar for predict-ing mortality (AUC, 0.88). Selection of a risk score for a hospital or health-care system should be guided by available variables, calculation method, and system resources. Once implemented, ensuring high levels of adherence and tying them to specifi c levels of interventions, such as acti-vation of a rapid response team, are necessary to allow for the greatest potential to improve patient outcomes. CHEST 2013; 143(6):1758–1765

Abbreviations: AUC 5 area under the receiver operating characteristic curve; CART 5 Cardiac Arrest Risk Triage; DNR 5 do not resuscitate; MERIT 5 Medical Early Response Intervention and Therapy; MEWS 5 Modifi ed Early Warning Score; SEWS 5 Standardized Early Warning Score; ViEWS 5 VitalPAC Early Warning Score

Risk Stratifi cation of Hospitalized Patients on the Wards Matthew M. Churpek , MD , MPH ; Trevor C. Yuen , BA ; and Dana P. Edelson , MD

Manuscript received June 26, 2012; revision accepted October 15, 2012 . Affi liations: From the Section of Pulmonary and Critical Care (Dr Churpek), the Health Studies Department (Dr Churpek), and the Section of Hospital Medicine (Mr Yuen and Dr Edelson), University of Chicago, Chicago, IL . Correspondence to: Dana Edelson, MD, 5841 S Maryland Ave, MC 5000, Chicago, IL 60637; e-mail: [email protected] © 2013 American College of Chest Physicians. Reproduction of this article is prohibited without written permission from the American College of Chest Physicians. See online for more details. DOI: 10.1378/chest.12-1605

journal.publications.chestnet.org CHEST / 143 / 6 / JUNE 2013 1759

were individually associated with at least 5% in-hospital mortality. The critical values they identifi ed were a sys-tolic BP , 85 mm Hg, heart rate . 120 beats/min, tem-perature , 35°C or . 38.9°C, oxygen saturation , 91%, respiratory rate , 13 or . 23 breaths/min, and level of consciousness recorded as anything but “alert.” They assigned one point for each critical value and found that having three simultaneous critical values was associated with 23.6% in-hospital mortality. In addition, the authors compared their score to two aggregate weighted risk scores, the MEWS ( Table 1 ) 16 and the VitalPAC Early Warning Score (ViEWS) ( Table 2 ), 25 and found similar accuracy for detecting in-hospital mortality (areas under the receiver oper-ating characteristic curves [AUCs] of 0.85, 0.87, and 0.86, respectively).

Aggregate Weighted Systems

Aggregate weighted scoring systems are the most complex of the early warning scores. They categorize vital signs and other variables into different degrees of physiologic abnormality and then assign point values for each category. They have the advantage of allow-ing for risk stratifi cation of patients and responses based on severity level but can be error prone when calculated manually. 26,27 Most published systems are expert opinion-based variations of the original Early Warning Score, 12 including the well-described MEWS and the Standardized Early Warning Score (SEWS) ( Table 3 ). 16,28

One of the most recently developed aggregate weighted risk scores is the ViEWS. 25 The ViEWS is similar to previously published early warning scores and includes heart rate, respiratory rate, temperature, and systolic BP but adds oxygen saturation and the use of supplemental oxygen ( Table 2 ). In addition, the weight ing for different vital sign abnormalities was adjusted based on the investigators’ analyses, prior liter-ature, and trial and error. 25 A study in the dataset from which the ViEWS was derived found that it outper-formed 33 other risk scores in a cohort of 35,585 acute medical patients for the outcome of death within 24 h of a ward vital sign set. In addition, an abbreviated ViEWS, without mental status, was externally vali-dated in a separate study in a Canadian hospital, where it was found to have an AUC of 0.93 for mortality and

in pediatric risk scores and in the development of sys-tems for specifi c patient populations. 18-21

Recently Developed Risk Scores for Ward Patients

Physiologic track and trigger systems are often divided into single-parameter, multiple-parameter, and aggre-gate weighted systems. There have been recent devel-opments in each of these categories. 8-10

Single-Parameter Systems

A single-parameter system is composed of a list of individual physiologic criteria that, if reached by a particular patient, trigger a response. Since any one abnormality on the list will trigger the response, these systems are the easiest to implement, requiring no score calculation. The fi rst such system was developed in the early 1990s in Liverpool, Australia and included vital signs, laboratory values, and specifi c conditions such as new arrhythmia and amniotic fl uid embolism. 22 The MERIT trial used a variation of these criteria, and it remains the most commonly described of the single-parameter tools today. 17 Both of these early sys-tems were developed using expert opinion. However, recently Cretikos and colleagues 23 used a case-control design to develop an evidence-based modifi cation to the MERIT criteria using data from the control arm of the trial. This model (respiratory rate ! 28 breaths/min, heart rate ! 140 beats/min, systolic BP " 85 mm Hg, or a decrease in Glasgow Coma Scale score of . 2 points), had a sensitivity of 59.6% and specifi city of 93.7% for predicting the composite outcome of cardiac arrest, death, or ICU transfer, compared with 50.4% sensi-tivity and 93.3% specifi city for the original criteria.

Multiple-Parameter Systems

Multiple-parameter systems use combinations of different physiologic criteria, without calculation of a score, to activate the rapid response system. These sys-tems are the least commonly described but have the advantage of allowing for risk stratifi cation and a graded response, without requiring a complex calculation. A recent example of this type of system was developed by Bleyer and colleagues 24 using vital sign cutoffs that

Table 1— Modifi ed Early Warning Score 16

Score 3 2 1 0 1 2 3

Respiratory rate, breaths/min … , 9 … 9-14 15-20 21-29 . 29Heart rate, beats/min … , 40 41-50 51-100 101-110 111-129 . 129Systolic BP, mm Hg , 70 71-80 81-100 101-199 … . 199 …Temperature, °C … , 35 … 35-38.4 … . 38.4 …Neurologic … … … Alert Voice Pain Unresp

Unresp 5 unresponsive.

&.&.�,��-��$%2) 1

Risk Stratification of Hospitalized Patients on the Wards

• 2007�\NIHCEZNational Institute for Health and Care Excellence:(��%�"�)*��[– ��!,@��'7C;@VUSWYO?-��#track and trigger systemsJ /GC39=��68

• ��<A100��@track and trigger systems 2.G• D=K>2MWQTX@EWSJ!,?0F0?��68E@

• 44��-�<!,J+1;criteriaJ&6-$�#?�3�5H8WRNRPL@� B= I:;3;/G

��track and trigger” systems

• Single-Parameter Systems • Multiple-Parameter Systems • Aggregate Weighted Systems

�� !



• 1990��0"7E;=AB5(!�#–>6<C96D"��"� ��"��+*-��-� %/32

• MERIT trial -:D8C?A@E<E.�1��,��&3'$2

• ��)1expert opinion4�$'��


Single-Parameter

Articles

any stage. Data collection continued during theimplementation period. At the end of this period, theMET system was activated in the intervention hospitalsonly and made available hospital-wide for the next6 months (study period). Apart from the obtaining ofapproval from the hospital ethics committee and thehospital management, and an undertaking by theresuscitation committee to maintain normal functioningof the cardiac arrest team, the study was not publicisedin the control hospitals. Management and resuscitationcommittees of the control hospitals agreed that theoperation of their cardiac arrest teams would continueunchanged during the implementation and studyperiods. Process and outcome data were obtained in allhospitals for the 6-month study period. Before theproject began, all data collectors were trained at thecoordinating centre with a standardised data collectionmanual. During the study, three data audits were doneto ensure the accuracy of data. These data audits targetedthe accuracy of the study data with reference to sourcedocumentation, outcomes of the study, and accuracy ofthe automated optical scanning data entry.

The primary outcome for the study was the compositeoutcome of the incidence (events divided by number ofeligible patients admitted to the hospital during thestudy period) of: cardiac arrests without a pre-existingnot-for-resuscitation (NFR) order, unplanned ICUadmissions, and unexpected deaths (deaths without apre-existing NFR order) taking place in general wards. Ageneral ward included any inpatient ward within thestudy hospitals. The coronary care unit was regarded as ageneral ward, as was a high-dependency unit that wasnot under the supervision of an intensive care specialist.

The ICUs, ICU-supervised high-dependency units,operating theatres, postoperative recovery areas, andemergency departments were not regarded as generalwards. Secondary outcomes consisted of: cardiac arrestswithout a pre-existing NFR order, unplanned ICUadmissions, and unexpected deaths.

A cardiac arrest was defined as when a patient had nopalpable pulse. An unplanned ICU admission wasdefined as any unscheduled admission to the ICU froma general ward. Unexpected deaths encompassed alldeaths without a pre-existing NFR order, including thosewith a preceding cardiac arrest. If a patient had morethan one event during their hospital stay, only one eventwas included in the composite measure. We excludedevents in patients younger than 14 years, patients whodied on arrival to hospital, or patients who had not beenformally admitted to hospital.

A standardised education and implementation strategywas used to introduce the MET into every interventionhospital. This strategy included education of clinical(medical and nursing) staff about the calling criteria, theimportance of these criteria in identification of patientsat risk, the need to call quickly if these criteria were met,and how to call the MET. We educated participating staffby using lectures, a MET videotape, and booklets, but wedid not educate them on the treatment of critically ill orunstable patients. Once MET implementation wascomplete, we provided reminders about the MET systemand how to call it by attaching the list of calling criteria toall identification badges of hospital staff andprominently displaying posters with the list of callingcriteria throughout the intervention hospitals.

The MET calling criteria are shown in the panel. Staffawareness of the introduction of the MET system wasmaintained by the use of regular reminders until thefirst day of the study period, after which awareness andeducation became the responsibility of the individualhospitals. The staff designated to form the MET variedbetween participating centres because of localcircumstances. The study protocol required that theMET should be at least the equivalent of the pre-existingcardiac arrest team and should consist of at least onedoctor and a nurse from the emergency department orICU.

Statistical analysisWith the assumption of an average 20 000 admissions peryear per hospital, the detection of a 30% reduction in theincidence of the composite primary outcome (from 3% to2·1%) with 90% power would need 18 hospitals with a 6-month follow-up. We used the method of Kerry andBland to account for clustering when the sample size wascalculated.14 The intraclass correlation coefficient used forthe sample size calculation (0·00127) was obtained from anon-randomised study of three hospitals.9

A weighted t test was used to assess cluster-leveldifferences in event incidence.14,15 Individual level

2092 www.thelancet.com Vol 365 June 18, 2005

Panel: MET calling criteria

AirwayIf threatened

BreathingAll respiratory arrestsRespiratory rate !5 breaths per minRespiratory rate "36 breaths per min

CirculationAll cardiac arrestsPulse rate !40 beats per minPulse rate "140 beats per minSystolic blood pressure !90 mm Hg

NeurologySudden fall in level of consciousness(fall in Glasgow coma scale of "2 points)Repeated or extended seizures

OtherAny patient you are seriously worried about that does not fitthe above criteria

Introduction of the medical emergency team (MET) system : a cluster-randomised controlled trial. Lancet 2005;June 18-24,365

MERIT trial MERIT trial��

Breathingrespiratory rate�28 breaths/min

Circulation heart rate�140 beats/minsystolic BP�85 mmHg,

or

Neurologya decrease in Glasgow Coma Scale score of 2 points

Original criteria ��50.4% → 59.6%��93.3% → 93.7%

The objective medical emergency team activation criteria: a case-control study. Resuscitation. 2007;73(1):62-72.



• �$! ��criteria+�(�*�"� �• RRS+��)�&/.,��'��• ��$��$�%��&�&0/-+��%��#�)�


Resuscitation 82 (2011) 1387– 1392

Contents lists available at ScienceDirect

Resuscitation

jo u rn al hom epage : www.elsev ier .com/ locate / resusc i ta t ion

Clinical paper

Longitudinal analysis of one million vital signs in patients in an academic medicalcenter!

Anthony J. Bleyera,∗, Sri Vidyab, Gregory B. Russellb, Catherine M. Jonesc, Leon Sujataa,Pirouz Daeihagha, Donald Hireb

a Section on Nephrology, Department of Internal Medicine, Wake Forest University School of Medicine, Medical Center Blvd., Winston-Salem, NC 27157, United Statesb Section on Biostatistics, Department of Public Health Sciences, Wake Forest University School of Medicine, Winston-Salem, NC, United Statesc Section on General Internal Medicine, Department of Internal Medicine, Wake Forest University School of Medicine, Winston-Salem, NC, United States

a r t i c l e i n f o

Article history:Received 21 January 2011Received in revised form 14 June 2011Accepted 25 June 2011

Keywords:Critical vital signsMortalityHospitalVIEWSMEWS

a b s t r a c t

Background: Recognition of critically abnormal vital signs has been used to identify critically ill patientsfor activation of rapid response teams. Most studies have only analyzed vital signs obtained at the timeof admission. The intent of this study was to examine the association of critical vital signs occurring atany time during the hospitalization with mortality.Methods: All vital sign measurements were obtained for hospitalizations from January 1, 2008 to June 30,2009 at a large academic medical center.Results: There were 1.15 million individual vital sign determinations obtained in 42,430 admissions on27,722 patients. Critical vital signs were defined as a systolic blood pressure <85 mm Hg, heart rate>120 bpm, temperature < 35 ◦C or >38.9 ◦C, oxygen saturation <91%, respiratory rate ≤ 12 or ≥ 24, andlevel of consciousness recorded as anything but “alert”. The presence of a solitary critically abnormalvital sign was associated with a mortality of 0.92% vs. a mortality of 23.6% for three simultaneous criticalvital signs. Of those experiencing three simultaneous critical vital signs, only 25% did so within 24 h ofadmission. The Modified Early Warning Score (MEWS) and VitalPAC Early Warning Score (VIEWS) werevalidated as good predictors of mortality at any time point during the hospitalization.Conclusions: The simultaneous presence of three critically abnormal vital signs can occur at any timeduring the hospital admission and is associated with very high mortality. Early recognition of theseevents presents an opportunity for decreasing mortality.

© 2011 Elsevier Ireland Ltd. All rights reserved.

1. Introduction

Rapid response teams have been implemented in a numberof hospitals for assistance with the care of critically ill patients.The designation of particular vital signs as “triggers” for alertingrapid response teams have been derived from landmark studies inthis field that were limited by relatively small datasets and oftenincluded only one measurement of vital signs1,2 or measurementof vital signs only early in the hospital admission.3–5 Studying vitalsigns throughout the entire hospital stay is important because rapidresponse teams are called not only at admission, but at any timethroughout the hospitalization.

The purpose of this study was to determine the prevalence andtime of occurrence of critical vital signs throughout the hospitalstay and examine their associated mortality. It was our intent to

! A Spanish translated version of the summary of this article appears as Appendixin the final online version at doi:10.1016/j.resuscitation.2011.06.033.

∗ Corresponding author. Tel.: +1 336 716 4513; fax: +1 336 716 4318.E-mail address: [email protected] (A.J. Bleyer).

determine if the simultaneous occurrence of more than one criticalvital sign at any time during the hospitalization was associated within-hospital mortality and to validate prior scores.3,6

In order to accomplish this, we were able to take advantageof the fact that vital signs at the medical center under study areentered electronically into a computer database. We were there-fore able to analyze a very large amount of data (>1 million vitalsign sets) in a large number of admissions (42,430). In addition,code status could be determined in individuals with the highestmorbidity.

2. Methods

This was a retrospective study approved by the Wake ForestUniversity School of Medicine Institutional Review Board.

2.1. Setting

A tertiary care 872 bed medical center with 108 intensive careunit beds, a 10 bed in-patient palliative care unit and 754 acute

0300-9572/$ – see front matter © 2011 Elsevier Ireland Ltd. All rights reserved.doi:10.1016/j.resuscitation.2011.06.033

1388 A.J. Bleyer et al. / Resuscitation 82 (2011) 1387– 1392

medical/surgical beds. The hospital is a level 1 trauma center per-forming cardiothoracic surgery and neurosurgery. Vital signs areentered electronically on all non-intensive care unit and all non-intermediate care unit floors. The hospital has a rapid responsesystem consisting of teams of two nurses with intensive care unitexperience and no physicians. At the time of the study, healthcareteams were encouraged to call rapid response for patients withthe occurrence of any critical vital signs or for patients whom thehealthcare team felt were unstable.

2.2. Data collection

All vital sign measurements, determinations of level ofconsciousness, and Glasgow coma scores were obtained for hos-pitalizations from January 1, 2008 to June 30, 2009. Vital signsobtained on individuals in the emergency room, intensive careunits, palliative care unit and intermediate care units were notincluded. Specifically, patients whose stay during the admissionincluded only the emergency room and intensive care unit werenot included. Information obtained included: systolic and dias-tolic blood pressure, temperature, heart rate, respiratory rate, pulseoximetry reading, level of consciousness, and Glasgow coma scoreif available. Readings were performed on hospital floors by nursesand nursing assistants who did not receive any special trainingwith regards to the study. Vital signs were entered electronicallyinto the hospital data system and were obtained based on thephysician orders. Automated blood pressure cuffs were used formost blood pressure measurements. Level of consciousness choicesincluded: alert, drowsy, sleeping, arousable, sedated, no response,and “see comment”, which allowed nurses to comment on the levelof consciousness. The comment field was in text and could not beanalyzed. Information on oxygen delivery was entered as a textfield in which the presence or absence of supplemental oxygen usecould be determined. The date, time, and location where each vitalsign was obtained were included in the database. A unique patientidentifier, the date of admission, date of discharge, and conditionat discharge (alive or expired) were also obtained. The data wasobtained from the hospital data repository. Due to the high volumeof data, it was not possible to check the accuracy of individual dataitems.

For each instance where three simultaneous critical vital signswere determined, a chart review was performed to determine thepresence of cancer, congestive heart failure, coronary artery dis-ease, dementia, or end-stage lung, heart, liver, or kidney disease,code status, and whether the rapid response team was called whenthe patient experienced these vital signs.

2.3. Analysis

We limited analysis to individuals with a respiratory rate >4and <60 bpm, systolic blood pressure >50 mm Hg and <250 mm Hg,temperature >90 ◦F (32.2 ◦C) or <106 ◦F (41.1 ◦C), pulse >30 bpmor <200 bpm, as readings outside these boundaries were rare andmore likely due to data entry error.

Categories representing different intervals were determined foreach vital sign, and the prevalence of admissions with vital signsin each range, together with proportionate mortality, were deter-mined. A critical vital sign was arbitrarily defined as the levelat which a patient who sustained the given vital sign during anadmission had a 5% or greater chance of mortality. An analysiswas performed to determine the proportionate mortality for indi-viduals who had the simultaneous occurrence of more than onecritical vital sign. To analyze factors associated with mortalityin patients with a trio of critical vital signs who were not DNR,a multivariate logistic regression model was created with out-come variables being death during hospitalization and independent

variables including vital signs at the time of determination of threecritical vital signs, comorbid conditions, and whether the rapidresponse team was notified. A backwards stepwise elimination wasthen performed, removing all parameters where the p value was>0.05.

A receiver operating characteristic (ROC) curve was constructedfor patients with one or more of these vital signs in regards to mor-tality. Similar ROC curves were constructed with an approximatedModified Early Warning Score (MEWS)3 for the same patients. Asthis was a retrospective study, we did not have the same scoringsystem for mental status as used in the MEWS study. For the AVPU(alert, voice, pain, unresponsive) score, we calculated a score of 0 ifthe patient was alert and a score of 3 if the patient was unrespon-sive. For patients who did not fit into these categories, a score of 1was given. An ROC curve was also generated for the VitalPAC EarlyWarning Score (VIEWS).7

3. Results

There were 1.15 million individual vital sign determinationsobtained in 42,430 admissions on 27,722 patients between January1, 2008 and June 30, 2009. A complete set of vital signs (level of con-sciousness, pulse oximetry reading, blood pressure, temperature,respiratory rate, and pulse) was present in 71.9% of measurements,and there was one missing vital sign out of 6 in 12.2%, 2/6 missingin 4.3%, 3–5/6 missing in 11.6%. The most common missing vitalsigns were temperature (17.8%) and pulse oximetry (16.5%). Theoverall mortality rate for all patients included in the study was2.74% and the mortality rate for all admissions studied was 1.79%.Of the patients studied, 76.4% were white, 19.8% African-American,2.43% Hispanic, and 51.1% male. The mean age of the patients was57.4 ± 17.6 years (range 18–104 years), and the mean length of staywas 6 ± 8.8 days (median 4 days, range 0–288 days).

Table 1 shows the level at which an admission with a given vitalsign sustained a mortality of 5%, 10%, or 20%. Vital sign ranges thatwere associated with 5% mortality are hereafter referred to as crit-ical vital signs. Supplementary Table 1 shows the number of vitalsigns that were obtained for each interval studied as well as theproportion of admissions that ended in mortality for individualswho had a determination in the given range. Fig. 1 shows simi-lar information graphically for systolic blood pressure, heart rate,

Table 1First vital sign range in which patients who experienced a vital sign in this rangesustained a minimum in hospital mortality of 5, 10, or 20%.a

Vital sign 5% mortality 10% mortality 20% mortality

Systolic blood pressure(mm Hg): low

80 to <85 65 to <70 55 to <60

Diastolic bloodpressure (mm Hg): low

20 to <30

Diastolic bloodpressure (mm Hg):high

120 to <130

Mean arterial pressure(mm Hg): low

40 to <50

Heart rate (bpm): high 120 to <130 140 to <150 150 to <160Temperature (◦C): low 34.4 to <35 33.9 to <34.4Temperature (◦C): high 38.9 to <39.4 39.4 to <40Respiratory rate (bpm):high

24 to 28 28 to 32 36 to <40

Respiratory rate (bpm):low

10 to 12 4 to 8

Oxygen saturation (%) 90 to <91 81 to <82Level of consciousness Not alert Sedated No responseGlasgow coma score 14 13

a Blank cells indicate that no vital sign range achieved corresponding level ofmortality. None of the high systolic blood pressure, high mean arterial pressure,and low heart rate categories were associated with a mortality >5%.

Ø��– RRSM#�=JWZP]B<@ �EXOU[RO\8';K@7J6

–�1$�,�CV]UY]TMFBD��6–�5�G��E��A/;K@7J6

Ø%$– �5�3�E1?"�EXOU[RO\E"�B5 ��!BE42M�-=J:B6

– HI�K>TQNM).=J:B6

ØVSO\`�L�9&(Ø*�

– �3`2008/1/1�2009/6/30– �0`42430�E�5^27722�E�+_�E115�EXOU[RO\M�.


medical/surgical beds. The hospital is a level 1 trauma center per-forming cardiothoracic surgery and neurosurgery. Vital signs areentered electronically on all non-intensive care unit and all non-intermediate care unit floors. The hospital has a rapid responsesystem consisting of teams of two nurses with intensive care unitexperience and no physicians. At the time of the study, healthcareteams were encouraged to call rapid response for patients withthe occurrence of any critical vital signs or for patients whom thehealthcare team felt were unstable.

2.2. Data collection

All vital sign measurements, determinations of level ofconsciousness, and Glasgow coma scores were obtained for hos-pitalizations from January 1, 2008 to June 30, 2009. Vital signsobtained on individuals in the emergency room, intensive careunits, palliative care unit and intermediate care units were notincluded. Specifically, patients whose stay during the admissionincluded only the emergency room and intensive care unit werenot included. Information obtained included: systolic and dias-tolic blood pressure, temperature, heart rate, respiratory rate, pulseoximetry reading, level of consciousness, and Glasgow coma scoreif available. Readings were performed on hospital floors by nursesand nursing assistants who did not receive any special trainingwith regards to the study. Vital signs were entered electronicallyinto the hospital data system and were obtained based on thephysician orders. Automated blood pressure cuffs were used formost blood pressure measurements. Level of consciousness choicesincluded: alert, drowsy, sleeping, arousable, sedated, no response,and “see comment”, which allowed nurses to comment on the levelof consciousness. The comment field was in text and could not beanalyzed. Information on oxygen delivery was entered as a textfield in which the presence or absence of supplemental oxygen usecould be determined. The date, time, and location where each vitalsign was obtained were included in the database. A unique patientidentifier, the date of admission, date of discharge, and conditionat discharge (alive or expired) were also obtained. The data wasobtained from the hospital data repository. Due to the high volumeof data, it was not possible to check the accuracy of individual dataitems.

For each instance where three simultaneous critical vital signswere determined, a chart review was performed to determine thepresence of cancer, congestive heart failure, coronary artery dis-ease, dementia, or end-stage lung, heart, liver, or kidney disease,code status, and whether the rapid response team was called whenthe patient experienced these vital signs.

2.3. Analysis

We limited analysis to individuals with a respiratory rate >4and <60 bpm, systolic blood pressure >50 mm Hg and <250 mm Hg,temperature >90 ◦F (32.2 ◦C) or <106 ◦F (41.1 ◦C), pulse >30 bpmor <200 bpm, as readings outside these boundaries were rare andmore likely due to data entry error.

Categories representing different intervals were determined foreach vital sign, and the prevalence of admissions with vital signsin each range, together with proportionate mortality, were deter-mined. A critical vital sign was arbitrarily defined as the levelat which a patient who sustained the given vital sign during anadmission had a 5% or greater chance of mortality. An analysiswas performed to determine the proportionate mortality for indi-viduals who had the simultaneous occurrence of more than onecritical vital sign. To analyze factors associated with mortalityin patients with a trio of critical vital signs who were not DNR,a multivariate logistic regression model was created with out-come variables being death during hospitalization and independent

variables including vital signs at the time of determination of threecritical vital signs, comorbid conditions, and whether the rapidresponse team was notified. A backwards stepwise elimination wasthen performed, removing all parameters where the p value was>0.05.

A receiver operating characteristic (ROC) curve was constructedfor patients with one or more of these vital signs in regards to mor-tality. Similar ROC curves were constructed with an approximatedModified Early Warning Score (MEWS)3 for the same patients. Asthis was a retrospective study, we did not have the same scoringsystem for mental status as used in the MEWS study. For the AVPU(alert, voice, pain, unresponsive) score, we calculated a score of 0 ifthe patient was alert and a score of 3 if the patient was unrespon-sive. For patients who did not fit into these categories, a score of 1was given. An ROC curve was also generated for the VitalPAC EarlyWarning Score (VIEWS).7

3. Results

There were 1.15 million individual vital sign determinationsobtained in 42,430 admissions on 27,722 patients between January1, 2008 and June 30, 2009. A complete set of vital signs (level of con-sciousness, pulse oximetry reading, blood pressure, temperature,respiratory rate, and pulse) was present in 71.9% of measurements,and there was one missing vital sign out of 6 in 12.2%, 2/6 missingin 4.3%, 3–5/6 missing in 11.6%. The most common missing vitalsigns were temperature (17.8%) and pulse oximetry (16.5%). Theoverall mortality rate for all patients included in the study was2.74% and the mortality rate for all admissions studied was 1.79%.Of the patients studied, 76.4% were white, 19.8% African-American,2.43% Hispanic, and 51.1% male. The mean age of the patients was57.4 ± 17.6 years (range 18–104 years), and the mean length of staywas 6 ± 8.8 days (median 4 days, range 0–288 days).

Table 1 shows the level at which an admission with a given vitalsign sustained a mortality of 5%, 10%, or 20%. Vital sign ranges thatwere associated with 5% mortality are hereafter referred to as crit-ical vital signs. Supplementary Table 1 shows the number of vitalsigns that were obtained for each interval studied as well as theproportion of admissions that ended in mortality for individualswho had a determination in the given range. Fig. 1 shows simi-lar information graphically for systolic blood pressure, heart rate,

Table 1First vital sign range in which patients who experienced a vital sign in this rangesustained a minimum in hospital mortality of 5, 10, or 20%.a

Vital sign 5% mortality 10% mortality 20% mortality

Systolic blood pressure(mm Hg): low

80 to <85 65 to <70 55 to <60

Diastolic bloodpressure (mm Hg): low

20 to <30

Diastolic bloodpressure (mm Hg):high

120 to <130

Mean arterial pressure(mm Hg): low

40 to <50

Heart rate (bpm): high 120 to <130 140 to <150 150 to <160Temperature (◦C): low 34.4 to <35 33.9 to <34.4Temperature (◦C): high 38.9 to <39.4 39.4 to <40Respiratory rate (bpm):high

24 to 28 28 to 32 36 to <40

Respiratory rate (bpm):low

10 to 12 4 to 8

Oxygen saturation (%) 90 to <91 81 to <82Level of consciousness Not alert Sedated No responseGlasgow coma score 14 13

a Blank cells indicate that no vital sign range achieved corresponding level ofmortality. None of the high systolic blood pressure, high mean arterial pressure,and low heart rate categories were associated with a mortality >5%.

Longitudinal analysis of one million vital signs in patients in an academic medical center


temperature, respiratory rate, level of consciousness, and pulseoximetry reading.

While admissions that had at least one of the critical vital signswere associated with approximately 5% mortality, individuals whoduring their admission had at maximum only one critical vital signat a given time had a mortality that was quite low (0.33%) (Table 2).The presence of one critical vital sign was also extremely high,noted in 18,412 out of 42,430 admissions (56.5%). The explanationfor this apparent paradox was that the mortality associated withone critical vital sign was clustered in a smaller group of patientswith multiple critical vital signs. Mortality was then calculated forgroups of critical vital signs (systolic blood pressure <85 mm Hg,heart rate >120 bpm, temperature <35 ◦C (95 ◦F) or greater than38.9 ◦C (102 ◦F), oxygen saturation <91%, respiratory rate ≥ 4 and≤ 12 or ≥ 24 and ≤ 60, and level of consciousness recorded as any-thing but “alert”), as well as age.

Table 2 shows the effect on mortality of the simultaneous pres-ence of multiple critical vital signs together with age. Increasedsimultaneous presence of abnormal vital signs, together with age,was strongly associated with higher mortality. The presence ofthree critical vital signs was associated with a 19-fold increase inmortality risk as compared with individuals who only sustained onecritical vital sign at a given time. The presence of three or more crit-ical vital signs occurred in 998/42,180 (2.4%) of admissions. Whena ROC curve was created with each critical vital sign being valuedat 1 point, the area under the curve (AUC) was 0.851. The presenceof three critical vital signs was especially concerning in individualsgreater than 70 years of age, where their presence was associatedwith a mortality of 33% and resulted in 112 deaths. We createda vital sign scoring system where each vital sign counted as onepoint, and added one point for ages 60 to <70, two points for 70 to<80, and 3 points for 80 to <90. The AUC curve was not appreciablydifferent with this scoring system (AUC 0.842), nor for a scoringsystem giving only one point for age >70 years (AUC 0.869) or onepoint for age 70 to <80 years and two points for age >80 years (AUC0.862) (see Supplementary Fig. 1).

We then determined the time of occurrence of the trio of criti-cal vital signs relative to admission and time to mortality after theiroccurrence. Fig. 2 reveals that 38% occurred within 48 h of admis-sion, but these events occurred throughout the admission, with 40%occurring greater than 5 days after admission. Fig. 3 shows the timeto mortality for patients with the simultaneous occurrence of threecritical vital signs. The mortality was 4.2% at 24 h and 7% at 48 h.Fig. 4 shows the proportion of patients who died based on the timeduring the hospital admission when the abnormal trio of vital signsoccurred.

ROC curves were calculated from our database for the VIEWSand the MEWS, with AUC 0.862 for the VIEWS and 0.865 for theMEWS (see Fig. 5). These scores provided very similar results.For individuals with a trio of critical vital signs, there were 998instances with 232 deaths (23.2%). For individuals with a corre-sponding VIEWS of ≥ 11, there were 1236 instances with 274 deaths(22.1%). Thus, the VIEWS was able to identify 42 more deaths withapproximately the same sensitivity.

Table 2In hospital mortality (%) associated with the simultaneous occurrence of critical vitalsigns. The number of patients who expired is noted in parentheses.

Number of simultaneouscritical vital signs

All ages Age >60years

Age >70years

Age >80years

0 0.24 (44) 0.40 (32) 0.62 (26) 0.92 (15)1 0.92 (174) 1.47 (134) 1.96 (100) 2.09 (44)2 6.95 (295) 9.84 (213) 11.5 (148) 13.0 (80)3 23.6 (186) 33.1 (128) 42.0 (99) 42.9 (48)4 or more 42.4 (61) 57.7 (41) 63.0 (29) 52.4 (11)

Fig. 2. Time from admission until occurrence of simultaneous critical vital signs for0 or 1 critical vital signs, 2 simultaneous critical signs, and 3 or more critical vitalsigns.

0

5

10

15

20

25

30

7654321

Time from admission to occurrence (days)

Perc

enta

ge

Fig. 3. Time from admission to mortality for individuals with three simultaneouscritical vital signs who expired during the hospitalization.

Further examination of the 998 cases with a trio of critical vitalsigns revealed that 15.8% of these patients were Do Not Resuscitate(DNR). For patients who were DNR, the mortality associated withthree simultaneous critical vital signs was 56.9%. For patients whowere not DNR, the mortality was 16.9%. In 39.6% of these patientswho were not DNR, the rapid response team was alerted within 24 hof three simultaneous critical vital signs, and the mortality rate ofthese patients was 28%. Of the patients in whom rapid responsewas not called, the mortality rate was 9.5%. Table 3 shows char-acteristics of those in whom the rapid response team was or was

0

5

10

15

20

25

30

3 to < 52 to < 31 to < 2<1 >= 107 to < 105 to < 7

Mor

talit

y (%

)

Day of hospitalization when critical vital signs occurred

Fig. 4. Proportionate mortality according to the time interval during hospitalizationwhen the trio of critical vital signs first occurred.


3�� 23.6%��


temperature, respiratory rate, level of consciousness, and pulseoximetry reading.

While admissions that had at least one of the critical vital signswere associated with approximately 5% mortality, individuals whoduring their admission had at maximum only one critical vital signat a given time had a mortality that was quite low (0.33%) (Table 2).The presence of one critical vital sign was also extremely high,noted in 18,412 out of 42,430 admissions (56.5%). The explanationfor this apparent paradox was that the mortality associated withone critical vital sign was clustered in a smaller group of patientswith multiple critical vital signs. Mortality was then calculated forgroups of critical vital signs (systolic blood pressure <85 mm Hg,heart rate >120 bpm, temperature <35 ◦C (95 ◦F) or greater than38.9 ◦C (102 ◦F), oxygen saturation <91%, respiratory rate ≥ 4 and≤ 12 or ≥ 24 and ≤ 60, and level of consciousness recorded as any-thing but “alert”), as well as age.

Table 2 shows the effect on mortality of the simultaneous pres-ence of multiple critical vital signs together with age. Increasedsimultaneous presence of abnormal vital signs, together with age,was strongly associated with higher mortality. The presence ofthree critical vital signs was associated with a 19-fold increase inmortality risk as compared with individuals who only sustained onecritical vital sign at a given time. The presence of three or more crit-ical vital signs occurred in 998/42,180 (2.4%) of admissions. Whena ROC curve was created with each critical vital sign being valuedat 1 point, the area under the curve (AUC) was 0.851. The presenceof three critical vital signs was especially concerning in individualsgreater than 70 years of age, where their presence was associatedwith a mortality of 33% and resulted in 112 deaths. We createda vital sign scoring system where each vital sign counted as onepoint, and added one point for ages 60 to <70, two points for 70 to<80, and 3 points for 80 to <90. The AUC curve was not appreciablydifferent with this scoring system (AUC 0.842), nor for a scoringsystem giving only one point for age >70 years (AUC 0.869) or onepoint for age 70 to <80 years and two points for age >80 years (AUC0.862) (see Supplementary Fig. 1).

We then determined the time of occurrence of the trio of criti-cal vital signs relative to admission and time to mortality after theiroccurrence. Fig. 2 reveals that 38% occurred within 48 h of admis-sion, but these events occurred throughout the admission, with 40%occurring greater than 5 days after admission. Fig. 3 shows the timeto mortality for patients with the simultaneous occurrence of threecritical vital signs. The mortality was 4.2% at 24 h and 7% at 48 h.Fig. 4 shows the proportion of patients who died based on the timeduring the hospital admission when the abnormal trio of vital signsoccurred.

ROC curves were calculated from our database for the VIEWSand the MEWS, with AUC 0.862 for the VIEWS and 0.865 for theMEWS (see Fig. 5). These scores provided very similar results.For individuals with a trio of critical vital signs, there were 998instances with 232 deaths (23.2%). For individuals with a corre-sponding VIEWS of ≥ 11, there were 1236 instances with 274 deaths(22.1%). Thus, the VIEWS was able to identify 42 more deaths withapproximately the same sensitivity.

Table 2In hospital mortality (%) associated with the simultaneous occurrence of critical vitalsigns. The number of patients who expired is noted in parentheses.

Number of simultaneouscritical vital signs

All ages Age >60years

Age >70years

Age >80years

0 0.24 (44) 0.40 (32) 0.62 (26) 0.92 (15)1 0.92 (174) 1.47 (134) 1.96 (100) 2.09 (44)2 6.95 (295) 9.84 (213) 11.5 (148) 13.0 (80)3 23.6 (186) 33.1 (128) 42.0 (99) 42.9 (48)4 or more 42.4 (61) 57.7 (41) 63.0 (29) 52.4 (11)

Fig. 2. Time from admission until occurrence of simultaneous critical vital signs for0 or 1 critical vital signs, 2 simultaneous critical signs, and 3 or more critical vitalsigns.

0

5

10

15

20

25

30

7654321

Time from admission to occurrence (days)

Perc

enta

ge

Fig. 3. Time from admission to mortality for individuals with three simultaneouscritical vital signs who expired during the hospitalization.

Further examination of the 998 cases with a trio of critical vitalsigns revealed that 15.8% of these patients were Do Not Resuscitate(DNR). For patients who were DNR, the mortality associated withthree simultaneous critical vital signs was 56.9%. For patients whowere not DNR, the mortality was 16.9%. In 39.6% of these patientswho were not DNR, the rapid response team was alerted within 24 hof three simultaneous critical vital signs, and the mortality rate ofthese patients was 28%. Of the patients in whom rapid responsewas not called, the mortality rate was 9.5%. Table 3 shows char-acteristics of those in whom the rapid response team was or was

0

5

10

15

20

25

30

3 to < 52 to < 31 to < 2<1 >= 107 to < 105 to < 7

Mor

talit

y (%

)

Day of hospitalization when critical vital signs occurred

Fig. 4. Proportionate mortality according to the time interval during hospitalizationwhen the trio of critical vital signs first occurred.


3�� 1�2��

A.J. Bleyer et al. / Resuscitation 82 (2011) 1387– 1392 1391

0

10

20

30

40

50

60

70

80

90

100

1009080706050403020100Critical Vital SignsMEWSVIEWS

Fig. 5. ROC curves comparing the scoring for the sum of critical vital signs, theMEWS, and the VIEWS.

not called. Patients for whom the rapid response team was calledin general had more comorbid conditions and had higher respira-tory rates and lower pulse oximetry readings at the time of the call.The mean duration of hospitalization prior to the abnormal vitalsigns was longer in patients in whom rapid response was calledvs. those who were not. In a multivariate best-fit logistic regres-sion model (see Supplementary Table 2) with the outcome variablebeing death during hospitalization, notification of rapid responsewas still highly associated with increased mortality. The AUC washigher for all individuals than for the group where DNR patientswere excluded (0.851 vs. 0.832).

4. Discussion

The simultaneous occurrence of three or more critical vital signswas more likely to occur early in the hospital admission (38% occur-ring within 48 h of admission), but could occur at any time, with40% occurring greater than 5 days after admission. We found that asimple summation score of critical vital signs was highly predictiveof in-hospital mortality. We were able to validate the MEWS andVIEWS as well, and show that these scores are predictive not only

at the time of admission but also at any point when vital signs aremeasured during the hospitalization.

Earlier, landmark studies in this area have identified vital signsas excellent predictors of survival and been instrumental in theimplementation of rapid response teams.8,9 Most prior studieshave used smaller data sets and focused on vital signs at thetime of admission. In this study, we were able to validate theMEWS and VIEWS not only at the time of admission, but as pre-dictors or mortality at any time point during the hospitalization. Inaddition, we were able to show that a summation score of the pres-ence of three critical vital signs was also an excellent predictor ofmortality.

Critical vital signs were common, with at least one critical vitalsign occurring in 56.5% of admissions. With grouping, the preva-lence of abnormal findings decreased, but the mortality remainedconcentrated in these smaller groups. This finding allowed betterdetermination of criteria for notifying rapid response teams. Forexample, visits by rapid response teams to all individuals with threeor more simultaneous critical vital signs would result in 55 patientsvisited per month; 13 of these 55 individuals, based on our his-tory, would expire. Many of these visits would occur within 48 hof admission, when early, aggressive care might help to improvesurvival.

In patients with a trio of critical vital signs, patients in whomrapid response teams were called had a higher mortality than inthe patients in whom the rapid response team was not alerted(28% vs. 9.5%, p < 0.01). This is likely due to the fact that the nursingstaff identified that these patients were more gravely ill and there-fore called rapid response teams. Patients in whom rapid responsewere called had more co-morbidities and lower oxygenation andincreased respiratory rates than patients in whom rapid responsewas not called. The average time to a rapid response team callin patients with three critical vital signs was 4.56 h, indicatingthat these vital signs may often precede a clinical deteriorationand should not be ignored. In hospitals with electronic entry ofvital signs, rapid response teams might be notified immediately inan electronic manner, bypassing judgments by local patient careteams as to the timing and appropriateness of referral to rapidresponse teams.

Shortcomings of this study include that these are results fromonly one center. While the results cannot be generalized to allhospitals, we were able to validate the MEWS3 and VIEWS,6

which have been found to be predictive in other settings. Another

Table 3Characteristics of patients with three critical vital signs in whom the rapid response team were or were not called.

Characteristic Rapid response called Rapid response not called p value*

Age, years (mean ± s.d.) 57.3 ± 16.4 55.2 ± 18.7 0.009Gender (% male) 53.9 52.7 0.73Race (% white) 72.0 73.8 0.30Congestive heart failure (%) 14.9 12.1 0.24Chronic obstructive pulmonary disease (%) 17.8 12.3 0.03Cancer (%) 40.2 38.8 0.68Dementia (%) 2.6 4.2 0.22End-stage kidney disease (%) 7.87 3.22 0.003Other end-stage disease (%) 5.23 9.91 0.01Total number of above comorbid conditions 0.007

0 32.7 40.01 46.4 46.52 or more 21.0 13.5

Temperature (◦C) (mean ± s.d.) 37.8 ± 1.52 37.9 ± 1.45 0.74Respiratory rate (mean ± s.d.) 26.2 ± 7.2 24.1 ± 6.2 0.0023Pulse oximetry reading (%) (mean ± s.d.) 90.6 ± 6.1 92.1 ± 5.4 0.003Mean arterial pressure (mm Hg) (mean ± s.d.) 63.8 ± 9.7 65.2 ± 9.5 0.84Pulse (mean ± s.d.) 124.7 ± 22.0 119.3 ± 21.3 0.5Duration of hospitalization prior to abnormal vital signs (days) (mean ± s.d.) 10.5 ± 18.3 6.5 ± 10.9 <0.001Mortality (%) 9.46 27.7 <0.001

* T test used for continuous variables and Chi-squared test used for discrete variables.


�� AUSs��3��$!#%"!&��/MEWS/VIEWS��0.85/0.87/0.86��

!"!#$%&' ()*!!+)*, #*-(*).#+)*, $/"0*1 !.#+).#&$1 2!(345,

%*6*% $7 '$1!'&$+!1*!! 2839: .%*)#;'$17+!*<;)*!($1<! #$6$&'*;)*!($1<! #$ (.&1;+1)*!($1!&6*5, !+((%*-*1#.% $/"0*12"=34

5 2.&);!>?@;A>?@5, B*&0C#, .1< +)&1* $+#(+# 7$)'.#C*#*)&D*< (.#&*1#!E FC* %*6*% $7 #C* (.#&*1#G! ')&#&'.%'.)* 1**<! .1< %$'.#&$1 B*)* '%.!!&H*< +!&10 <*H1&#&$1!()$6&<*< I" #C* =1#*1!&6* 9.)* J$'&*#"K 2%*6*% ?L 1$)-.%B.)< '.)* &1 .1 .'+#* C$!(&#.%: %*6*% ML '.)* &1 .1 .'+#* B.)<B&#C .<<&#&$1.% !+(($)#: %*6*% 4L '.)* 7$) . !&10%* $)0.17.&%+)* $) !#*( <$B1 7)$- . C&0C*) %*6*% $7 '.)*: %*6*% NL.<6.1'*< )*!(&).#$)" !+(($)# $) !+(($)# $7 #B$ $)0.1!"!#*-! .1< &1'%+<&10 .%% '$-(%*/ (.#&*1#! B&#C -+%#&O$)0.17.&%+)*5E

=7 )*'$)<*< B&#C&1 P C $7 $+) 6&!&#, B* +!*< #C* -$!#)*'*1# 6.%+*! $1 #C* (.#&*1#G! $I!*)6.#&$1 'C.)#E =7 1$#, B*-.<* 1*B $I!*)6.#&$1! $+)!*%6*!E 3+#'$-* .# N? <."!2<&!'C.)0*< .%&6*;<&*< &1 C$!(&#.%;&1(.#&*1#5 B.! )*#)&*6*<7)$- #C* C$!(&#.% )*'$)<! !"!#*-E Q* +!*< #C* 1$)-.%).10*! <*H1*< I" $+) &1#*1!&6* '.)* $+#)*.'C !*)6&'*2'.%%*< #C* R.#&*1# S# T&!U F*.- &1 $+) C$!(&#.%5L )*!(&).#$)").#* M?VMW I(-: XT K?VWW I*.#! -&1VM: !"!#$%&'()*!!+)* M??VMYW -- X0: #*-(*).#+)* NZE?VNYE>!9: !(34

"WK@: 839 .%*)#: +)&1* $+#(+# 2'.#C*#*)&D*< (.#&*1#!5?EKVN -% U0VM -&1VME

J#.#&!#&'.% .1.%"!&! B.! B&#C JRJJ 6M?E [$) #C* I.'UB.)<!#*(B&!* %$0&!#&' )*0)*!!&$1 2%&U*%&C$$< ).#&$5 -$<*%, .%%6.)&.I%*! */'*(# .0* B*)* #)*.#*< .! 1$)-.% $) .I1$)-.%2I&1.)" 6.)&.I%*!5: .0* B.! #)*.#*< .! . '$1#&1+$+! 6.)&.I%*EFC* $<<! ).#&$! #C+! ()$6&<*< )*%.#* #$ #C* &1')*.!*< )&!U $7-$)#.%&#" &7 #C* 6.)&.I%* &! .I1$)-.% 2+!&10 #C* <*H1&#&$1!%&!#*<5 */'*(# 7$) .0* BC*)* #C* $<<! ).#&$ &! #C* )&!U $7-$)#.%&#" '.+!*< I" I*&10 M ") $%<*)E S# *.'C !#*( &1 #C*)*0)*!!&$1, 7$) . 6.)&.I%* #$ I* )*-$6*< 7)$- #C* -$<*% &#C.< #$ C.6* . # 6.%+* 0)*.#*) #C.1 ?EME

Q* !+)6*"*< K>P I*<! $7 BC&'C WP B*)* +1$''+(&*< $1#B$ 6&!&#!E Q* */'%+<*< MN &1#*1!&6* '.)* +1&# 2=9\5(.#&*1#!, #C)** (.#&*1#! U1$B1 #$ I* ]1$# 7$) )*!+!'&#.#&$1G.1< $1* <+(%&'.#* $I!*)6.#&$1 '.+!*< I" . #).1!7*), %*.6&10>NN <.#. !*#! 7$) .1.%"!&!E ^.%+*! .)* 0&6*1 .! -*.1 2J_5 $)-*<&.1 2&1#*)`+.)#&%* ).10*5E

FC* 4Z (.#&*1#! BC$ <&*< B*)* $%<*) #C.1 #C* !+)6&6$)!2#a?E??M, +1(.&)*< $O#*!#5, -*.1 .0* YN 2).10* NPVWM5 ") %&Z? 2MPVWY5 ")E FC*&) -*<&.1 C$!(&#.% !#." B.! 4Z 2MZVNW5<."!, B&#C <*.#C .# . -*<&.1 $7 M?EK 2>V4M5 <."! .7#*) #C*

!#+<"E FC* $1%" (.#&*1# B&#C 1$ .I1$)-.%&#&*! BC$ <&*< <&<!$ 4M <."! .7#*) #C* !#+<"E b$)#.%&#" &1')*.!*< !&01&H'.1#%"B&#C #C* 1+-I*) $7 .I1$)-.%&#&*! 2#a?E??M, %$0&!#&')*0)*!!&$1: */(%.1.#$)" 6.)&.I%*L #$#.% 1+-I*) $7 .I1$)-.%O&#&*!5 2F.I%* M5E T*!(&).#$)" ).#* .1< XT B*)* #C*6.)&.I%*! -$!# 7)*`+*1#%" .I1$)-.% 2.I1$)-.% &1 K>@ .1<MN@, )*!(*'#&6*%", $7 '.!*! B&#C $1* .I1$)-.%&#" .1< &1WZ@ .1< ZN@, )*!(*'#&6*%", $7 '.!*! B&#C .# %*.!# #C)**.I1$)-.%&#&*!5E

=1 . !*(.).#* .1.%"!&!, #C* !*6*1 &1&#&.% 6.)&.I%*!2)*!(&).#$)" ).#*, XT, !"!#$%&' ()*!!+)*, #*-(*).#+)*, !(34

,839, .0*5 B*)* .1.%"!*< I" I.'UB.)< !#*(B&!* %$0&!#&')*0)*!!&$1E FC* $<<! ).#&$ 7$) -$)#.%&#" 7$) #C* H6* -$!#!&01&H'.1# 6.)&.I%*! B*)*, &1 <*')*.!&10 %*6*% $7 !&01&HO'.1'*L 839 >EZN 2WK@ '$1H<*1'* &1#*)6.%! MEYWVM4E??:#a?E?K5, XT NEPZ 2ME>WVWEWY: #a?E?K5, .0* ME?> 2ME?MVME?Y:#a?E?K5, !"!#$%&' ()*!!+)* NEMZ 2MEM4VPEW?: #a?E?K5 .1<)*)!(&).#$)" ).#* 4E>W 2?EW4VZEY45E

FC&! -$<*% C.< . !*1!&#&6&#" $7 YEY@, !(*'&H'&#" $7 [email protected]< . ($!&#&6* ()*<&'#&6* 6.%+* $7 ZZEY@E FC* !*1!&#&6&#" .1<!(*'&H'&#" $7 #C* -$<*% B.! '.%'+%.#*< +!&10 . '%.!!&H'.#&$1'+#O$77 $7 ?EK: #C.# &!, &7 #C* ()$I.I&%&#" $7 <*.#C $I#.&1*<7)$- #C* -$<*% B.! 0)*.#*) #C.1 ?EK #C* !+Ic*'# B.!'%.!!&H*< .! C.6&10 <&*<, '$16*)!*%" &7 #C* ()$I.I&%&#" $7<*.#C $I#.&1*< 7)$- #C* -$<*% B.! %*!! #C.1 ?EK #C* !+Ic*'#B.! '%.!!&H*< .! . !+)6&6$)E

\)&1* $+#(+# B.! $1%" )*'$)<*< 7$) (.#&*1#! B&#C '.#C*#*)!2'dK4, M4@5 .1< B.! 1$# &1'%+<*< &1 #C* %$0&!#&' )*0)*!!&$1-$<*%E 3/"0*1 #C*).(" B.! 0&6*1 #$ NW (.#&*1#! 2W@5E =1#C*!* #B$ 0)$+(! 2'.#C*#*)&D*< $) )*'*&6&10 $/"0*15 N?O<."-$)#.%&#" B.! MWE4@ .1< N?@, )*!(*'#&6*%"E

e!#&-.#*< %*6*%! $7 '.)* B*)* .6.&%.I%* 7$) NP> 2PPEY@5$7 .%% (.#&*1#!, BC&'C &1'%+<*< 4N 2PPEK@5 $7 #C* 4Z BC$<&*<E FC* N> (.#&*1#! #C$+0C# I" <.#. '$%%*'#$)! #$ I*)*'*&6&10 . %$B*) %*6*% '.)* #C.1 <*!&).I%* C.< . 0)*.#*)-$)#.%&#" 24?EZ@, #a?E?M, [&!C*)G! */.'# #*!#5 '$-(.)*<B&#C #C* N>W (.#&*1#! 2-$)#.%&#" KE>@5 c+<0*< #$ I*)*'*&6&10 .(()$()&.#* '.)*E

[$)#"O#C)** 2WEW@5 $7 #C* (.#&*1#! &1 #C* !#+<" B*)* !**1.# !$-* ($&1# <+)&10 #C*&) C$!(&#.% !#." I" #C* &1#*1!&6* '.)*$+#)*.'C #*.-E FC* #*.- )*6&*B*< !&/ $7 #C* (.#&*1#! BC$<&*<E =1 #C)** $7 #C*!* (.#&*1#!, &1')*.!*< #)*.#-*1# B.!'$1!&<*)*< &1.(()$()&.#*, $1* B.! .<-&##*< #$ =9\ .1< #B$

!"#$% & FC&)#"O<." $+#'$-*, %$'.#&$1 .1< <."! &1 C$!(&#.% I*7$)* .1< .7#*) #C* !#+<" .''$)<&10 #$ #C* 1+-I*) $7 .I1$)-.%&#&*! )*'$)<*<E 9=, '$1H<*1'*&1#*)6.%: =fT, &1#*)`+.)#&%* ).10*: g#d?E?PY: h#d?E?M>, i#d?E??M 2I" %$0&!#&' )*0)*!!&$15: .(.#&*1#! )*'$)<*< .! <&*< &1 C$!(&#.% B&#C&1 N? <."! $7 !#+<":I&1')*.!*< )&!U $7 -$)#.%&#" BC*1 '$-(.)*< B&#C 1$ .I1$)-.%&#&*!: '%$'.#&$1 '.(.I%* $7 <*%&6*)&10 %*6*%! $7 '.)* 2(.#&*1#! .%)*.<" &1 %*6*% N B*)* */'%+<*<5:<I*7$)* !#+<" d <."! &1 C$!(&#.% I*7$)* !#+<": .7#*) !#+<" d <."! &1 C$!(&#.% +1#&% <*.#C, <&!'C.)0* $) *1< $7 !#+<" (*)&$< 2N? <."!5

'()#%* +,"#-+*)"$./.%0

'()#%* +,1"/.%-/0

2345"6)+*/"$./6

7550 *"/.+I

89:; <=>'()#%* +,.-1"/.%-/0 "/

'()#%* +, 1"/.%-/0 ?"60 .- @+01./"$ 8)%5."- 8=AB>><

*%C+*5%5 8;> 8;>. 23 5"60 8;> D%E%$ 3' D%E%$ & D%E%$ F G%,+*% 0/(56 H,/%* 0/(56

? MNW 2N45 ?EY MP 2MN5 MNP M ? Z 2NVMK5 > 2MVMY5M MZ? 2NY5 >E> ZEN?g 2?EYYVK45 4Y 2MY5 MKK > M Y 2NE4V445 Z 24V4>54 PY 24?5 WE4 MNEWKh 2MEYVMMN5 MN 2MK5 PM K M M? 2NVMW5 Z 24V4M5N $) -$)* >Y 2MM5 4MEN NYE4>i 2>EZVN??5 MM 24N5 >M N N Z 24V4W5 M4 2NVN?5

RC"!&$%$0&'.% .I1$)-.%&#&*! .1< -$)#.%&#"

!!"

��

Physiological abnormalities in early warning scores are related to mortality in adultinpatients. British Journal of Aneasthesia 92(6): 882-4�2004�


• EWS6&/2�7�"4DEGK• I?FMC?N3�6�>�#*;• ,<-<6@GBLO5$�� 6��5�=+1J?NH'�:�19<1%;

• ��6 ��5�0%.��8$��6LEA6��>��2(;

• ��2��*;62!�%7�)8*%


• �� MEWS�Modifies Early Warning Score�ViEWS�Vital PAC Early Warning Score�CART�Cardiac Arrest Risk Triage�

Aggregate Weighted Systems Risk Stratification of Hospitalized Patients on the Wards


were individually associated with at least 5% in-hospital mortality. The critical values they identifi ed were a sys-tolic BP , 85 mm Hg, heart rate . 120 beats/min, tem-perature , 35°C or . 38.9°C, oxygen saturation , 91%, respiratory rate , 13 or . 23 breaths/min, and level of consciousness recorded as anything but “alert.” They assigned one point for each critical value and found that having three simultaneous critical values was associated with 23.6% in-hospital mortality. In addition, the authors compared their score to two aggregate weighted risk scores, the MEWS ( Table 1 ) 16 and the VitalPAC Early Warning Score (ViEWS) ( Table 2 ), 25 and found similar accuracy for detecting in-hospital mortality (areas under the receiver oper-ating characteristic curves [AUCs] of 0.85, 0.87, and 0.86, respectively).


Aggregate weighted scoring systems are the most complex of the early warning scores. They categorize vital signs and other variables into different degrees of physiologic abnormality and then assign point values for each category. They have the advantage of allow-ing for risk stratifi cation of patients and responses based on severity level but can be error prone when calculated manually. 26,27 Most published systems are expert opinion-based variations of the original Early Warning Score, 12 including the well-described MEWS and the Standardized Early Warning Score (SEWS) ( Table 3 ). 16,28

One of the most recently developed aggregate weighted risk scores is the ViEWS. 25 The ViEWS is similar to previously published early warning scores and includes heart rate, respiratory rate, temperature, and systolic BP but adds oxygen saturation and the use of supplemental oxygen ( Table 2 ). In addition, the weight ing for different vital sign abnormalities was adjusted based on the investigators’ analyses, prior liter-ature, and trial and error. 25 A study in the dataset from which the ViEWS was derived found that it outper-formed 33 other risk scores in a cohort of 35,585 acute medical patients for the outcome of death within 24 h of a ward vital sign set. In addition, an abbreviated ViEWS, without mental status, was externally vali-dated in a separate study in a Canadian hospital, where it was found to have an AUC of 0.93 for mortality and

in pediatric risk scores and in the development of sys-tems for specifi c patient populations. 18-21

Recently Developed Risk Scores for Ward Patients

Physiologic track and trigger systems are often divided into single-parameter, multiple-parameter, and aggre-gate weighted systems. There have been recent devel-opments in each of these categories. 8-10


A single-parameter system is composed of a list of individual physiologic criteria that, if reached by a particular patient, trigger a response. Since any one abnormality on the list will trigger the response, these systems are the easiest to implement, requiring no score calculation. The fi rst such system was developed in the early 1990s in Liverpool, Australia and included vital signs, laboratory values, and specifi c conditions such as new arrhythmia and amniotic fl uid embolism. 22 The MERIT trial used a variation of these criteria, and it remains the most commonly described of the single-parameter tools today. 17 Both of these early sys-tems were developed using expert opinion. However, recently Cretikos and colleagues 23 used a case-control design to develop an evidence-based modifi cation to the MERIT criteria using data from the control arm of the trial. This model (respiratory rate ! 28 breaths/min, heart rate ! 140 beats/min, systolic BP " 85 mm Hg, or a decrease in Glasgow Coma Scale score of . 2 points), had a sensitivity of 59.6% and specifi city of 93.7% for predicting the composite outcome of cardiac arrest, death, or ICU transfer, compared with 50.4% sensi-tivity and 93.3% specifi city for the original criteria.


Multiple-parameter systems use combinations of different physiologic criteria, without calculation of a score, to activate the rapid response system. These sys-tems are the least commonly described but have the advantage of allowing for risk stratifi cation and a graded response, without requiring a complex calculation. A recent example of this type of system was developed by Bleyer and colleagues 24 using vital sign cutoffs that

Table 1— Modifi ed Early Warning Score 16

Score 3 2 1 0 1 2 3

Respiratory rate, breaths/min … , 9 … 9-14 15-20 21-29 . 29Heart rate, beats/min … , 40 41-50 51-100 101-110 111-129 . 129Systolic BP, mm Hg , 70 71-80 81-100 101-199 … . 199 …Temperature, °C … , 35 … 35-38.4 … . 38.4 …Neurologic … … … Alert Voice Pain Unresp

Unresp 5 unresponsive.

1760 Recent Advances in Chest Medicine

MEWS was recently introduced in The Netherlands that includes nurse worry and urine output and was validated in a population of surgical patients using the highest score achieved for each patient during that hospitalization. 15 A score ! 3 had a sensitivity of 74% and specifi city of 82% for a composite outcome that included mortality, ICU transfer, and severe surgical complications. In another study, Kho and colleagues 33 developed a risk score by altering the vital sign weight-ings in the MEWS, adding BMI and age and removing mental status, based on prior literature and a review of calls to their hospital’s rapid response team. Using the maximum score for each patient on the wards, their model had an AUC of 0.72 for the combined out-come of code team activation, cardiac arrest, or ICU transfer. In addition, Cuthbertson and colleagues 34,35 used discriminant function analysis in two case-control studies, one in medical and another in surgical patients. They used the highest, lowest, and median vital sign values for patients in the 48-h period before transfer to the ICU (cases) or to a lower-acuity unit (control subjects). The resulting models were at least as accu-rate as the other published risk scores they compared their models to in both studies. Finally, using electronic medical record data, investigators from Kaiser Per-manente developed a 24-variable model that included vital signs, laboratory values, severity of illness scores, and longitudinal chronic illness burden scores. 36 Their model had an AUC of 0.78 in a validation dataset for detecting a combined end point of ICU transfer or death outside the ICU.

0.72 for ICU transfer within 48 h using admission vital sign data. 29

Another aggregate weighted scoring system was developed by Tarassenko and colleagues 30 using con-tinuous vital sign data in high-risk patients on the wards or step-down units. They derived a scoring system based on the distributions of respiratory rate, heart rate, systolic BP, and oxygen saturation in their data-set. Although they did not evaluate the accuracy of their system for detecting adverse outcomes, they are currently conducting a clinical trial using their system on the trauma wards at a teaching hospital. 30

Our group published an aggregate weighted system known as the Cardiac Arrest Risk Triage (CART) score ( Table 4 ). 31 The CART score was derived using logis-tic regression to detect in-hospital cardiac arrest and was validated for detecting ward-to-ICU transfers. In that study, the CART score outperformed the MEWS for detecting both cardiac arrest (AUC, 0.84 vs 0.76; P 5 .001) and ICU transfer (AUC, 0.71 vs 0.67; P , .001). However, it was not compared with other early warn-ing scores and has not yet been validated using exter-nal data.

There are several other recent additions to the liter-ature. These include the Worthing physiologic scor ing system, which was statistically derived using admission data. 32 When applied to a validation dataset, the system had an AUC of 0.72 for the outcome of in-hospital mor-tality. A disadvantage of this system is that it requires the measurement of oxygen saturation on room air, which is not always collected. In addition, a form of the

Table 2— VitalPAC Early Warning Score 25

Score 3 2 1 0 1 2 3

Respiratory rate, breaths/min , 9 … 9-11 12-20 … 21-24 . 24Oxygen saturation, % , 92 92-93 94-95 96-100 … … …Supplemental oxygen use … … … No … … Yes (any oxygen)Heart rate, beats/min … , 41 41-50 51-90 91-110 111-130 . 130Systolic BP, mm Hg , 91 91-100 101-110 111-249 . 249 … …Temperature, °C , 35.1 … 35.1-36 36.1-38 38.1-39 . 39 …Neurologic … … … Alert … … Voice

… … … … … … Pain… … … … … … Unresp

See Table 1 legend for expansion of abbreviation.

Table 3— Standardized Early Warning Score 28

Score 3 2 1 0 1 2 3

Respiratory rate, breaths/min , 9 … … 9-20 21-30 31-35 . 35Oxygen saturation, % , 85 85-89 90-92 93-100 … … …Heart rate, beats/min , 30 30-39 40-49 50-99 100-109 110-129 . 129Systolic BP, mm Hg , 70 70-79 80-99 100-199 … . 199 …Temperature, °C , 34 34-34.9 35-35.9 36-37.9 38-38.9 . 38.9 …Neurologic … … … Alert Voice Pain Unresp

See Table 1 legend for expansion of abbreviation.



has had a rapid response team in place since 2008, which is led by a critical care nurse and is separate from the team that responds to cardiac arrests. Respi-ratory therapy also responds to team activations and a hospitalist attending physician and/or pharmacist are available upon request.

We compared recently developed or validated single-parameter (ie, Cretikos et al 23 ), multiple-parameter (ie, Bleyer et al 24 ), and aggregate weighted (ie, Tarassenko et al, 30 ViEWS, SEWS, and the CART score 25,28,31 ) risk scores to previously validated systems that are commonly used (ie, the MERIT criteria and MEWS). 16,17 We were unable to compare all the recently developed systems because room air oxygen satura-tion, BMI, urine output, and accurate determination of patients admitted to the surgical service were not available. In addition, some aspects of the MERIT cri-teria were not captured in our dataset, such as a drop in Glasgow Coma Scale score, the presence of seizures, and airway emergencies.

Ward vital signs were extracted from our electronic health record (EPIC), and each of the previously mentioned early warning scores was calculated for every simultaneous ward vital sign set in the entire dataset. If a vital sign necessary for score calculation was missing, then the most recent value was carried forward. In addition, if there were no previous values, then a median value was imputed. Cardiac arrest was determined using a prospectively validated quality improvement database, and ICU transfer and mortal-ity were determined using administrative databases.

Accuracy was calculated using the AUC, sensitivity, and specifi city for detecting cardiac arrest, ICU trans-fer, mortality, and a composite outcome of any of these events using each patient’s highest score prior to the event or during their entire admission for those who did not experience an event. Ward patients transferred to the ICU from the operating room were not counted as an ICU transfer event. Of note, the CART score was developed to detect cardiac arrest using an older version of these data that account for approximately 80% of the patients in this updated dataset.

Results

During the study period, there were 59,643 admis-sions with ward vital signs, including 109 ward cardiac arrests, 291 deaths within 24 h of a ward vital sign, and 2,655 ward-to-ICU transfers. The included patients had a mean age of 55 ! 18 years; 56% were women, 43% were black, 36% were white, and 34% underwent surgery during the hospitalization. Results from the early warning score comparisons are shown in Table 5 , separated by outcome. We found a wide range of accu-racy, both across outcomes for a given system and across systems. In general, mortality resulted in the

Choosing the Optimal Scoring System Introduction

The accuracy of the system has strong implications for long-term success, both in terms of identifying more cases (ie, sensitivity) to minimize adverse events and preventing false alarms (ie, specifi city) to minimize resource expenditure and alarm fatigue. For example, consider a hospital that has 20,000 admissions per year and 1,000 events (ICU transfers, deaths, and ward car-diac arrests). An improvement in sensitivity of 5%, at the same specifi city level, would result in the detection of an additional 50 adverse events per year (1,000 events multiplied by the difference in sensitivity of 5%). In addition, an improvement in specifi city of 5%, at the same level of sensitivity, would result in 950 fewer “false alarms” per year (19,000 admissions who did not experience the event multiplied by the difference in specifi city of 5%). Multiplying these results over many hospitals across the country would result in a considerable public health benefi t and illustrates the importance of efforts to improve the accuracy of early warning scores.

Methods

Because the systems described previously represent recent advances, many have not been directly com-pared with one another in the same dataset. There-fore, we used our database of ward admissions from November 2008 until August 2011 consisting of both medical and surgical patients to compare the different early warning scores described previously. Our patient population has been previously described; in brief, it consists of all patients on the ward from a single urban academic center in the United States. 31 Our hospital

Table 4— Cardiac Arrest Risk Triage Score 31

Vital Sign Score

Respiratory rate, breaths/min , 21 0 21-23 8 24-25 12 26-29 15 . 29 22Heart rate, beats/min , 110 0 110-139 4 . 139 13Diastolic BP, mm Hg . 49 0 40-49 4 35-39 6 , 35 13Age, y , 55 0 55-69 4 . 69 9


�&6@><IJ=?@EH7'�

• �)M2008�11�-:2011�8�94• �621EWS;*�0� K �"7FLBADG;�,3�%/1+

• �$M!7��(�7�#AJBL7��"

• 2008�-:RRT; �+– Critical care nurse.��+Respiratory therapy /a

hospitalist attending physician / pharmacist –��7��CLH58�


1762 Recent Advances in Chest Medicine

aggregate weighted system, is determining how the score will be calculated. Options include calculation by hand, using a calculator or handheld device devel-oped specifi cally for the scoring system, and using the electronic medical record. Manual calculation, with or without a calculator, is the most commonly used method and the least expensive to implement. How-ever, studies suggest that calculation errors are not uncommon. 26,27 Preprogrammed applications decrease these errors but still require manual entering of the data, which can be redundant to workfl ow and error prone in its own right. 27 Completely automated sys-tems, such as those integrated into electronic medical records, are likely to be the least labor intensive from the clinician standpoint and least error prone. More-over, they have the potential to incorporate other patient data, such as demographic characteristics, location, and laboratory values and even be tied into automated noti-fi cation systems. 33,36 However, these systems require institutional resources to implement and may not be an option for most hospitals, especially those with paper-based medical records.

Current Controversies in the Field Outcome Selection for Development and Validation Studies

As demonstrated in the results of our comparison study , the reported accuracy of a scoring system will depend in large part on the outcome chosen for vali-dation. Therefore, comparison of systems requires a consistency in outcome selection. Although mortality is the easiest to predict, as evidenced by the higher AUCs, it may not be the most useful, since many deaths in the hospital are fully expected, and detecting those events is neither necessary nor helpful. However, most studies of early warning scores have not omitted do not resuscitate (DNR) patients from their analyses. Reasons for this may include diffi culty in identifying such patients in large datasets, the inability to deter-mine the exact time when the goals in patient care transition from life-saving (when early warning scores

highest AUCs, whereas ICU transfer resulted in the lowest. Overall, the aggregate weighted scoring systems outperformed the other systems for most outcomes, with the SEWS, MEWS, ViEWS, and CART score being the most accurate. In addition, the modifi ed MERIT criteria described by Cretikos and colleagues 23 were more accurate than the original MERIT cri-teria for all outcomes. Although the ViEWS, CART, MEWS, and SEWS were similar in performance across the outcomes, the CART score had the highest AUC for cardiac arrest (0.83), ICU transfer (0.77), and the composite outcome (0.78), whereas the CART score, ViEWS, and SEWS all had the same AUC for mortal-ity (0.88). The ViEWS was the second most accurate sys tem for detecting cardiac arrest (0.77), and the SEWS was the second most accurate for ICU transfer (AUC 0.75) and the composite outcome (AUC, 0.76). Since the CART score was derived using many of the patients from this dataset, we repeated the analysis for the CART score using only those patients not in the original study (ie, prospective validation) and found similar results (AUCs of 0.86, 0.76, 0.87, and 0.77 for cardiac arrest, ICU transfer, mortality, and the com-posite outcome, respectively). Sensitivity and specifi city values at cut-points closest to 85%, 90%, and 95% specifi city for the four most accurate systems for detect-ing cardiac arrest are shown in Table 6 . At a speci-fi city threshold of approximately 90%, the CART score had a sensitivity of 49%, compared with the ViEWS (41%), MEWS (39%), and the centile-based system (35%, data not shown). The SEWS and the multiple-parameter system by Bleyer and colleagues 24 did not have cut-offs near this level of specifi city. The MERIT criteria had a sensitivity and specifi city of 45% and 82% for detecting cardiac arrest compared with the modifi ed criteria proposed by Cretikos et al, 23 which had a sensitivity of 54% and specifi city of 84%.

Implementation of Early Warning Scores

An important part of the process of implementing a physiologic track and trigger system, especially an

Table 5— Accuracy of Track and Trigger Systems for Different Outcomes

Track and Trigger System Cardiac Arrest ICU Transfer Mortality Composite

MERIT 0.63 (0.59-0.68) 0.64 (0.63-0.65) 0.74 (0.71-0.76) 0.64 (0.64-0.65)Modifi ed MERIT 0.69 (0.65-0.74) 0.69 (0.68-0.70) 0.79 (0.76-0.81) 0.70 (0.69-0.70)Multiple parameter, from Bleyer et al 24 0.73 (0.68-0.78) 0.72 (0.71-0.73) 0.84 (0.82-0.87) 0.73 (0.72-0.74)Centile-based, from Tarassenko et al 30 0.70 (0.65-0.76) 0.71 (0.69-0.72) 0.83 (0.80-0.86) 0.72 (0.70-0.73)MEWS 0.76 (0.71-0.81) 0.74 (0.73-0.75) 0.87 (0.84-0.89) 0.75 (0.74-0.76)SEWS 0.76 (0.71-0.81) 0.75 (0.74-0.76) 0.88 (0.86-0.90) 0.76 (0.75-0.77)ViEWS 0.77 (0.72-0.82) 0.73 (0.72-0.75) 0.88 (0.86-0.91) 0.75 (0.74-0.76)CART score 0.83 (0.79-0.86) 0.77 (0.76-0.78) 0.88 (0.86-0.90) 0.78 (0.77-0.79)

Data are shown as area under the receiver operating characteristic curve (95% CI). CART 5 Cardiac Arrest Risk Triage; MERIT 5 Medical Early Response Intervention and Therapy; MEWS 5 Modifi ed Early Warning Score; SEWS 5 Standardized Early Warning Score; ViEWS 5 VitalPAC Early Warning Score.


7 &(421'��$,��'�� ,��AUC��ICU��) ,��#"�7 ��')�SEWS�MEWS�ViEWS��-*CART(421� ,��%�#"�7 aggregate weighted system)��(3456-/,�0$�"�7 Cretikos.'-#$� 0"MERIT�)�!+$(��'��$�(MERIT�-/,��%�#"


tem that had been previously in use in the develop-ment dataset. 30

Inclusion of Age in Risk Scores

Older age is a known risk factor for cardiac arrest and death and is often included in risk scores used in the ICU. However, age is only rarely included in cur-rent risk scores for ward patients. 39 Several studies have shown that the increased risk of adverse outcomes associated with increased age is independent of vital sign derangements, and the inclusion of age has been shown to improve the accuracy of risk scores, although to varying degrees. 16,24,31,32,39 Some concerns raised about including age in early warning scores are eth-ical in nature, 39 and including age could make it less likely for younger patients with vital sign abnormal-ities to be identifi ed. It is unknown to what degree this would occur, given that cardiac arrest and death are much more common in older age groups. In addi-tion, it is unknown whether vital signs prior to these events differ between younger and older patients, given the increased use of b -blockers in older patients and the physiologic changes that occur with aging.

Impact on Clinical Outcomes

The most important unanswered question is whether early warning scores improve outcomes. To defi nitively answer this question, a large randomized trial would be needed that used a well-validated risk score, and even then separating the effects of the specifi c risk score used in the study from the intervention would be diffi cult. Many before-and-after studies have been published highlighting the usefulness of early warning scores (usually concurrently implemented with rapid response systems), including improved vital sign docu-mentation 40,41 and improved patient outcomes. 37,42-44 However, these fi ndings have not been universal, 45,46 and it is currently unclear whether early warning scores improve important patient outcomes, such as hospital-wide cardiac arrest and mortality rates, or decrease costs. Importantly, delayed response has been identi-fi ed as one of the strongest predictors of mortality and unexpected ICU transfer in patients evaluated by the rapid response team. 47,48 Future efforts to improve the evidence base for early warning scores are needed, as are methods to improve adherence to vital sign docu-mentation and rapid response system activation after they are implemented. Automated implementation with built-in notifi cation, as described previously, is likely to help, but this remains to be demonstrated, and the cost-effectiveness needs to be studied. Finally, pairing the different risk strata to specifi c levels of interventions, such as increased monitoring, consultation by the ICU team, and automatic calls to the rapid response team, is essential, and different workfl ows have been proposed. 8

might be benefi cial) to comfort (when risk scores would not be useful), the belief that some DNR patients still desire other life-saving interventions, and previous stud-ies suggesting that rapid response teams can improve some aspects of end-of-life care. 37 ICU transfer rep-resents the most common outcome and thus results in the highest statistical power but is the least gener-alizable, given the heterogeneous criteria for admis-sion, resulting in the lowest AUCs. Moreover, it is, by defi nition, an event already recognized by hospital staff, albeit late on some occasions. In addition, crite-ria for ICU admission in some hospitals may include vital sign cutoffs, and so scores derived or validated in such hospitals would be affected by these criteria. Similar to mortality, cardiac arrest has the benefi t of being objectively defi ned. However, unlike mortality, it is always worth identifying and preventing, if only to institute a DNR order in some cases. As such, a cardiac arrest on the fl oor always represents a failure of the current system and thus may be the most clin-ically relevant of the outcomes to use for derivation and validation. However, reporting all four outcomes in future studies would allow readers to draw their own conclusions about relevant outcomes for their own practice.

Some authors have stated that the original early warn-ing scores were not developed to be highly accurate predictors of any specifi c outcome due to the many confounding events that occur during a hospitaliza-tion. 38 In addition, as described previously, investiga-tors have also used the distribution of vital signs to determine cutoffs for risk scores, arguing that deriving models based on outcomes, when used prospectively, will disadvantage those patients who would have been previously “salvaged” by the vital sign monitoring sys-

Table 6— Sensitivity and Specifi city at Selected Cut Points for the Most Accurate Track and Trigger Systems for Identifying Patients With Cardiac Arrest

Track and Trigger System Cut Point Sensitivity, % Specifi city, %

SEWS . 3 55 85 . 4 38 94 . 5 19 97MEWS . 3 67 80 . 4 39 91 . 5 20 96ViEWS . 8 60 83 . 9 41 91 . 10 29 95CART . 16 61 84 . 20 49 90 . 24 35 95

See Table 5 legend for expansion of abbreviations.

=�\;:F/3\��X-�[�Tdhiojq\>�

• hiojqf�[TdMN�OP�ETdH• hiojq\�<�N?7H• )�,fhiojq[TdQX\!C

– AOCW49ReUbK[G�*SaTJ– � G@\�P\)��]��*S&.WOVJd– _XgY\/3W�%McDNARfA�SVJZJ– �8'ZnsmpslMcDNAR�5f+�TdQX\�BR– �5\ksrNlife-saving Mccomfort ^26Td(0Z"$f1TQXN�B

– DNAR�5W`�\life-saving f�#Td� `Id– end-of-life care[LJVRRTN�TdQXW��Td�D`Id


*�?)(/!"?��<��>26EGIOKR?+�

• ICU� –�A�& >�1CFEGIOKR– ICU� ?criteriaBPHMSLHT?KNOJQ�4�;0D�&�4.71

• cardiac arrest –�' >�$7B81–�A%� >�=GIOKR3A7F=1–��7�=1��<��<?#�54.71– DNAR?�,;09:A�-7��8E��@0E


�9��&PHEHFB8�%@�<?,

• �%:��9PHEMAEIT5)?(• �%@PHEHFB8�<?6HFB9��-1(

• �%8�+� ��6'LCIQGCS9�69#!�:7*(

• �%8�*��-�0'βNRJDT9��

�=�.7?(/9>+7$%�6��9"5'��COSK�9LCIQGCS9�-�7?,;2�,34*7*(


Derivation of a cardiac arrest prediction model using ward vital signs. Crit Care Med. 2012;40(7):2102-2108.

Should age be included as a component of track and trigger systems used to identify sick adult patients? Resuscitation. 2008;78(2): 109-115.

A�d@?HClinical Outcomefd�G


• "gE;ZSj`�6^QaL LeIEWSelnxp}k �WiO`MO^KiJ

• �<'arandomized trialP�;^KiJ• -�d~uqurld%_I��d%k�SiT_e�F^KiJ

• �Rdbefore –and-after35bNL]EWSd#0�P�>Uj]LiPIC�eRRSd��g�!b:[]LiJ

• /!,^eIEWSP�8dE;alnxp}��(h)�.�k �V]LiOIrux�*b\aP[]LiO`MO�4O^KiJ

• Delayed responseeIRRSP��VY�8d)�._�$XcICU��d"g�L�+��^KiJ– ymv�sm�d=BdD�– 91bRRSf�2Ujituw}d&7– ruxz{o�|�u

• RRSH��G�96%��*HZVSVTRH�I6>H10�D�0G�:C=B7

• 4Aggregate Weighted Systems5I��6��#6ICU� Q�&@OVTRE?C6�HUVXYMN�'D8O– !G6ViEWS, SEWS, MEWS, and CART score

• 4Aggregate Weighted Systems5H��G;9C6�$D=O��L6+F-(��6/ Q)�?6%3H"�GPAB1�FW][Q2�\�.@O�,<8O7


JEK

NEWSfNational early warning score)

• 2012�Tddeb• NHSfNational Health Serviceh^_dbU��/aecb�%gDXV�U�/!6Q�,�Q�H�!A

• 4#– NEWSF<0IZM�"@� UEWSKS[Ntrack and

trigger systems FNHS�Q�+IZPCMA– LUMW@')�IZMEWSU1�@->=QU2 @�3U*��U$�R��U�9�U(�F�?RSOMA

– &BSb`]\�+JP;.�3\7�KYRG@��QU:58FSEOMA

National Early Warning Score (NEWS)*

3 2 1 0 1 2 3PHYSIOLOGICALPARAMETERS

Heart Rate

Temperature

Systolic BP

Respiration Rate

Level ofConsciousness

OxygenSaturations

Any SupplementalOxygen

≤40 41 - 50 51 - 90 91 - 110 111 - 130 ≥131

≤35.0 35.1 - 36.0 36.1 - 38.0 38.1 - 39.0 ≥39.1

≤90 91 - 100 101 - 110 111 - 219 ≥220

≤8 9 - 11 12 - 20

A V, P, or U

21 - 24 ≥25

≤91 92 - 93 94 - 95 ≥96

Yes No

*The NEWS initiative flowed from the Royal College of Physicians’ NEWS Development and Implementation Group (NEWSDIG) report, and was jointly developed and funded in collaboration with theRoyal College of Physicians, Royal College of Nursing, National Outreach Forum and NHS Training for Innovation

© Royal College of Physicians 2012

Please see next page for explanatory text about this chart.

Outline clinical response to NEWS triggers

NEWS SCORE CLINICAL RESPONSEFREQUENCY OFMONITORING

Minimum 12 hourly • Continue routine NEWS monitoring withevery set of observations

Minimum 4-6 hourly

• Inform registered nurse who must assessthe patient;

• Registered nurse to decide if increasedfrequency of monitoring and / orescalation of clinical care is required;

Increased frequencyto a minimum

of 1 hourly

• Registered nurse to urgently informthe medical team caring for the patient;

• Urgent assessment by a clinicianwith core competencies to assess acutelyill patients;

• Clinical care in an environment withmonitoring facilities;

Continuous monitoring ofvital signs

• Registered nurse to immediately informthe medical team caring for the patient –this should be at least at SpecialistRegistrar level;

• Emergency assessment by a clinicalteam with critical care competencies,which also includes a practitioner/s withadvanced airway skills;

• Consider transfer of Clinical care to alevel 2 or 3 care facility, i.e. higherdependency or ITU;

0

Total:1-4

Total:5 or more

or

3 in oneparameter

Total:7

or more

© Royal College of Physicians 2012

Please see next page for explanatory text about this chart.

Nat

iona

l Ear

ly W

arni

ng S

core

2 (N

EWS2

) © R

oyal

Col

lege

of P

hysi

cian

s 20

17

3

23

3

2

3

12

≥131121–130 111–120101–11091–10081–9071–8061–7051–6041–5031–40

≤30

CPulse Beats/min

≥131121–130 111–120101–11091–10081–9071–8061–7051–6041–5031–40≤30

3

1

2

3

1

AlertConfusion

VPU

DConsciousness

AlertConfusion VPU

Monitoring frequencyEscalation of care Y/N

Initials

MonitoringEscalation Initials

NEWS TOTAL TOTAL

≥2521–2418–20 15–1712–149–11

≤8

A+BRespirations Breaths/min

≥2521–2418–20 15–1712–149–11≤8

13

3

≥220201–219181–200161–180141–160121–140111–120101–11091–10081–9071–8061–7051–60

≤50

CBloodpressure mmHgScore uses systolic BP only

≥220201–219181–200161–180141–160121–140111–120101–11091–10081–9071–8061–7051–60≤50

3

1

DATETIME

DATETIME

≥39.1°38.1–39.0°37.1–38.0°36.1–37.0°35.1–36.0°

≤35.0°

ETemperature °C

≥39.1°38.1–39.0°37.1–38.0°36.1–37.0°35.1–36.0°≤35.0°

1

1

3

2

NEWS key FULL NAME

DATE OF BIRTH DATE OF ADMISSION10 2 3

32

≥9694–9592–93

≤91

≥97 on O2

95–96 on O2

93–94 on O2

≥93 on air88–9286–8784–85≤83%

A=AirO2 L/min

Device

≥9694–9592–93≤91

≥97 on O2

95–96 on O2

93–94 on O2

≥93 on air88–9286–8784–85≤83%

A=AirO2 L/minDevice

3

1

1

2

2

2

3SpO2 Scale 2†

Oxygen saturation (%)Use Scale 2 if target range is 88–92%, eg in hypercapnic respiratory failure

† ONLY use Scale 2 under the direction of a qualified clinician

Air or oxygen?

A+BSpO2 Scale 1 Oxygen saturation (%)

Score for NEW onset of confusion (no score if chronic)

NEW_NEWS!_v4.indd 1 11/07/2017 15:14

NEWS2

17

• RRSW�(FegmhqV�&I`_N[VX@1000?BN^25�WRRT doseF�6SI`_wCrit Care 2005, 9:R808-815.xA

• H`X@EWSW �,/�F�F^@EQ+� Z8=uRRS0�W<�F��I``Y@response doseX�bPRG_WOaDEz

• EWSWegmhqVQCRW:9FBPNF@EWSSRRSWegmhqXfivtUWOaDEzw�URRSWorfpsegmhqy�'J�UC��)u��)�$@*�-@RRS;�-@ICU��%$@5��3*�-2x

• ]X^@TdUV`N �,/�c\QEWSc�JNSJR\@M`c�.L_jklnW#4F>c!_WOS"[R KNA

earlywarningsystem ewsoutreach team (cco). organizations like the american heart association,...

Documents