Journal of Critical Care (2009) 24, 447–452

Cardiovascular Disease

The diagnosis of myocardial infarction in critically illpatients: An agreement studyWendy Lim MD, MSc a,⁎, Andrea Tkaczyk RN, MNb, Paula Holinski MDa,Ismael Qushmaq MDc, Michael Jacka MDd, Vikas Khera MDa,P.J. Devereaux MD, PhDa,b, Irene Terrenato PhDe, Holger Schunemann MD, PhDb,e,Diane Heels-Ansdell MSc b, Mark Crowther MD, MSc a, Deborah Cook MD, MSc a,b

aDepartment of Medicine, McMaster University, Hamilton, Ontario, CanadabDepartment of Clinical Epidemiology and Biostatistics, McMaster University, CanadacDepartment of Medicine, King Faisal Specialist Hospital and Research Center, Jeddah, Saudi ArabiadDepartment of Medicine, University of Alberta, Edmonton, Alberta, CanadaeDepartment of Epidemiology, Italian National Cancer Institute Regina Elena, Rome, Italy

C

0d

Keywords:Troponin;Critical illness;Intensive care unit;Electrocardiography;Decision making

AbstractPurpose: The aim of the study was to assess agreement among 4 intensivists in diagnosing myocardialinfarction (MI) in critically ill patients based on screening electrocardiograms (ECGs) and cardiactroponin (cTn) levels.Methods: Consecutive patients admitted to a medical-surgical intensive care unit (ICU) underwentsystematic screening with 12-lead ECGs and cTn measurements throughout their ICU stay.Independently, 4 raters interpreted the ECGs assessing for changes indicative of ischemia and thenclassified each patient as to whether they met diagnostic criteria for MI based on the screening cTnmeasurements and ECG results. A priori, 2 raters were designated the primary adjudicators, and theirconsensus was used as the reference for the agreement statistics. Agreement on MI diagnosis wascalculated for the 4 raters and expressed as raw agreement, κ (chance-corrected agreement) and ϕ(chance-independent agreement, calculated using pairs).Results: Among 103 enrolled patients, 37 (35.9%) had MI according to the primary adjudicators. Theraw agreement for diagnosing MI was 79% (substantial), κ was 0.24 (fair), and ϕ ranged from 0.12 to0.73 (slight to substantial).Conclusions: Diagnosing MI in the ICU remains a challenge due to variable agreement in 12-lead ECGinterpretation. Such variation in practice may contribute to underrecognition of MI during critical illness.© 2009 Elsevier Inc. All rights reserved.

⁎ Corresponding author. St Joseph's Hospital, Hamilton, Ontario,anada L8N 4A6. Tel.: +1 905 521 6024; fax: +1 905 540 6568.E-mail address: limwp@mcmaster.ca (W. Lim).

883-9441/$ – see front matter © 2009 Elsevier Inc. All rights reserved.oi:10.1016/j.jcrc.2008.08.012

1. Introduction

The universal definition of myocardial infarction (MI) asdefined by the Joint European Society of Cardiology/

448 W. Lim et al.

American College of Cardiology Foundation/American HeartAssociation/World Heart Federation (ESC/ACCF/AHA/WHF) Task Force is based on the rise and/or fall of cardiacbiomarkers such as cardiac troponin (cTn), in addition to one ofthe following: symptoms of ischemia, detection of ischemicchanges on an electrocardiogram (ECG), or imaging evidenceof infarction [1]. However, use of this definition for diagnosingMI in the intensive care unit (ICU) has limitations. Critically illpatients are usually endotracheally intubated, which inhibitscommunication of ischemic symptoms, and frequently receivesedatives and analgesic medications that can mask symptomsof ischemic chest pain. Use of various imaging techniques toassess for MI is not routinely performed because many of theaccompanying signs and symptoms ofMI such as hypotensionand arrhythmias are often nonspecific in the ICU setting [2-4].Furthermore, imaging investigations are usually only initiatedin patients in whom the diagnosis is suspected. Thus,intensivists frequently rely on ECG evidence of myocardialischemia in addition to biochemical markers of myocardialnecrosis to establish the diagnosis of MI.

Sepsis, hypotension, pulmonary embolism, and renalfailure have all been reported to cause cTn elevation in theabsence of MI [5-7] and are frequently encountered in theICU. It likely that one or more of the recognized causes ofcTn elevation are occurring in critically ill patients, but inmany cases, the etiology of cTn elevation is not specificallyinvestigated. Because cTn levels may be increased forreasons other than MI, interpretation of an elevated cTn levelin these patients is frequently uncertain and management isvariable. Further complicating the diagnosis of MI is thatinterpretation of the ECG in the ICU setting is variable, asshown in our prior study where we examined the intraraterand interrater reliability of ECG interpretation among 2raters and the influence of knowledge of the patient'stroponin values [8]. Interrater reliability for determiningmyocardial ischemia or infarction was slight (ϕ, 0.18) butincreased to moderate (ϕ, 0.52) when the ECG wasinterpreted alongside the patient's troponin values. Thissuggested that in critically ill patients, ECGs should beinterpreted with knowledge of the troponin values toincrease reliability. Specific, easily identifiable ECG find-ings such as bundle branch block also showed highreliability, compared to T-wave flattening [8].

In most ICUs, ECG and cTn measurements areperformed on patient admission and subsequently per-formed as clinically indicated based on the patient's clinicalcourse. Screening ECGs and cTn testing is not currently thestandard practice, and the effect of this screening on patientoutcomes is unknown. From our prior study, we recognizedthat ECG interpretation should be done alongside thetroponin values, but because these investigations were doneas clinically indicated, we did not always have access toboth ECG and troponin measurements. Furthermore,investigations ordered for clinical indications will typicallyresult in a higher pretest probability for detecting abnormaltest results compared to screening investigations. Therefore,

we conducted a screening study in which ECGs and cTnwere performed throughout the patient's ICU stay. Theobjective of this study was to assess agreement amongmultiple raters (4 physicians) in diagnosing MI in criticallyill patients based on screening ECG and cTn levels.

2. Methods

2.1. Patients

All consecutive critically ill patients admitted to a 15-bedgeneral medical-surgical ICU at St Joseph's Hospital,Hamilton, Ontario, Canada, from January 2 to March 1,2006, were screened by dedicated ICU research personnel. Allpatients were enrolled, and therewere no exclusion criteria. OnICU admission, we collected patient demographics, baselinedata, and any ECG and cTn levels ordered by the ICU team.We obtained screening ECG and cTn measurements after apredefined screening schedule throughout the patient's ICUstay if these investigations were not ordered by the ICU team.For these screening tests, we obtained deferred third personconsent from family members as soon as possible after ICUadmission. If consent was declined for the screening ECGs andcTn, we collected the ECGs and cTn that the ICU team orderedfor clinical purposes and the relevant data available in thepatient's medical chart. This study was approved by ourinstitutional research ethics board.

2.2. Screening schedule and investigations

Screening 12-lead ECGs (PageWriter, Hewlett-Packard,Palo Alto, Calif) and screening cardiac troponin (cTn)measurements were both performed according to thefollowing screening schedule: on ICU admission, daily forthe first week in ICU, followed by alternate days for theremainder of the first month, and then weekly thereafter untilICU death, discharge, or 2 months after the enrollment date.Research personnel assessed all enrolled patients on the dayof their scheduled screening test; if the ECG or cTn hadalready been performed by the ICU team, the results wererecorded by research personnel (ECG was photocopied, cTnlevel was recorded), and the test was not repeated on that day.If the ECG or cTn measurement had not been performed bythe ICU team on the scheduled screening day, these tests werecompleted by research personnel, such that on each scheduledscreening day, there was at least one ECG and one cTnmeasurement. If an ECG or cTn was measured by the ICUteam on a nonscheduled screening day, the test result wasrecorded, but no screening tests were completed by theresearch personnel.

To ensure that screening tests done for study purposes didnot influence patient management, screening ECGs wereprinted out and immediately placed in the research chart sothe results were not accessible to the ICU team. All cTnT

449Diagnosis of MI in critically ill patients

assays for this study were run in real time, with the cTnTresults ordered by the ICU team made available through thehospital computer laboratory system as per usual practice,whereas the screening cTnT results that were drawn for studypurposes were entered by the laboratory staff into a password-protected computer system. The results of the screening cTnwere not accessible to the ICU team and only available toresearch personnel after the study was completed.

Blood samples for cTnT measurements were drawn intoEDTA tubes, and plasma for sample analysis obtained aftercentrifugation of whole blood at 1500g × 15minutes. Cardiactroponin (cTn) T was measured using an electrochemilumi-nescence immunoassay (Roche Modular analytics E170[Elecsysmodule] immunoassay analyzer, RocheDiagnostics,Indianapolis, Ind). The analytical sensitivity (lower detectionlimit) of this assay is 0.01 μg/L. An elevated cTnT wasdefined as values greater than or equal to 0.04 μg/L, whichrepresents the assay coefficient of variation of 10%.

2.3. Electrocardiogram interpretation andMI determination

Electrocardiogram interpretation was independently per-formed by 4 raters, 3 of whom were intensivists (DJC, MJ, IQ)and 1 was a critical care trainee (VK). Two intensivists (DJC,IQ) were designated the primary adjudicators for establishing adiagnosis of MI; any disagreements between these 2 adjudica-tors was resolved by discussion. The primary adjudicators hadpreviously collaborated on a study involving ECG interpreta-tion [9]. For the interpretation, all raters were blinded to thepatient's name and clinical information. The automatedcomputer ECG interpretation printed on the top portion of allECGs was not removed to replicate clinical practice.

The entire series of 12-lead ECGs completed during thepatient's ICU admission were provided to each rater, alongwith the cTn value(s) measured on the day the ECG wasrecorded (the cTn value was written directly on the ECG). Ifthere was more than one cTn measurement, the time that thecTn was drawn was indicated along with the cTn value. Eachrater was asked to interpret the ECGs for ischemic changesmeeting the ESC/ACCF/AHA/WHF Task Force ECGcriteria for the definition of MI and to determine whetherMI was present. Myocardial infarction was diagnosed if therewere both ECG criteria for MI and cTn elevation. Each MIwas labeled as an ST elevation MI or non-ST elevation MI. Acalibration exercise was first performed with all 4 ratersusing 5 randomly selected ECGs. Disagreements werediscussed, and consensus was achieved.

2.4. Statistical analysis

We report continuous data using mean and SD andbinary data using proportions and 95% confidence inter-vals. All calculations were performed using MicrosoftExcel and PC-Agree.

We used 3 methods of assessing agreement due topotentially important limitations in some of these measure-ments. Crude agreement is the proportion of patients forwhich the raters agree with each other with respect to thepresence or absence of MI. Crude agreement has thepotential to mislead because any group of 2 or more ratersjudging the presence or absence of a finding will often agreepurely by chance. To address the limitation of misleadingresults due to chance, we used “chance-corrected agree-ment,” reflected in the κ statistic. κ eliminates the agreementdue to chance but is limited when the distribution of ratingsbecome extreme (eg, few ECGs have the abnormality ofinterest). This situation produces a high level of chanceagreement and makes any agreement above chance difficultto achieve, resulting in counterintuitively low estimates ofagreement. The ϕ statistic addresses this problem andreflects “chance-independent agreement” [10]. ϕ is basedon odds ratios and is not influenced when the proportion ofpositive ratings becomes extreme. To reduce bias, we added0.5 to each cell in our contingency table when calculating theodds ratio to determine ϕ [11].

We interpreted κ and ϕ results as follows: values less than0, poor agreement; 0 to 0.2, slight agreement; 0.2 to 0.4, fairagreement; 0.4 to 0.6, moderate agreement; 0.6 to 0.8,substantial agreement; 0.8 to 1.0, almost perfect agreement[10,12,13].

3. Results

3.1. Patients

For the 2-month study period, 103 patients wereadmitted and enrolled in the study. We obtained deferredconsent for 89 (86.4%) patients. No consent was obtainedfor 14 (13.6%) patients, and no screening cTn and ECGdata were collected on these patients; however, any cTn andECGs that the ICU team ordered for clinical care werecollected. The baseline characteristics of the enrolledpatients are presented in Table 1. The mean age of thepatients was 64.1 (±17.5) years with a mean AcutePhysiology and Chronic Health Evaluation II score of24.1 (±10.3). Most admissions were for medical reasons,and 44 (43%) patients were female. A total of 60 (58.2%)patients were mechanically ventilated, and 29 (28.2%)patients were receiving inotropic or vasopressor support atthe time of enrollment.

The analysis for the agreement statistics were based on thepatients identified who had at least one elevated cTn and atleast one ECG interpreted as having ischemic changes. Thiswas done based on the definition of MI that was used in thestudy—one or more elevated cTn measurements andischemic ECG changes. Therefore, a total of 45 patientswere identified for the agreement statistics. Of the 45 patients,8 patients did not provide consent and were included in the

Table 1 Patient clinical characteristics

Age, mean (SD) 64.1±17.5Female sex, n (%) 44 (42.7)APACHE II score, mean (SD) 24.1±10.3Medical admission, n (%) 65 (63)Medical history, n (%)Smoking 28 (27.2)Hypertension 54 (52.4)Diabetes mellitus a 25 (24.3)Hyperlipidemia 17 (16.5)Documented coronary disease/angina 3 (2.9)Prior MI 5 (4.9)Congestive heart failure 16 (15.5)Peripheral vascular disease 13 (12.6)Stroke/transient ischemic attack 10 (9.7)Baseline life support interventions, n (%)VentilationInvasive mechanical ventilation 60 (58.2)Noninvasive mechanical ventilation 4 (3.9)Inotropes and vasopressors, n (%)Epinephrine 3 (2.9)Dopamine b 6 (5.8)Norepinephrine 14 (13.6)Dobutamine 2 (1.9)Phenylephrine 3 (2.9)Vasopressin 1 (1.0)Hemodialysis, n (%)Intermittent dialysis 9 (8.7)Continuous renal replacement therapy 1 (1.0)

APACHE indicates Acute Physiology and Chronic Health Evaluation.a Managed with oral agents and/or insulin.b More than 3 μg/kg per minute.

450 W. Lim et al.

analysis based on an ICU-ordered cTn that was elevated orICU-ordered ECG interpreted as being ischemic.

3.2. Agreement in ECG interpretation

We present the agreement statistics in Table 2. Among all4 raters, the raw agreement was 79% and κ was 0.24 (fairagreement). ϕ was calculated between all pairs of raters andranged from 0.12 (slight) to 0.73 (substantial). ϕ was highestbetween the primary adjudicators (ϕ, 0.73, substantial

Table 2 Agreement among 4 raters for electrocardiogram interpretat

Agreement Primaryadjudicator1 and 2

Primaryadjudicator 1and intensivist

Primaryadjudicator 2and intensivis

Raw agreement 0.78 0.84 0.62κ (95% confidenceinterval)

0.47(0.22–0.71)

0 0

ϕ 0.73 0.39 0.12

We present raw agreement for each pair of reviewers. The raw agreement for all 4agreement reflecting the proportion of patients for whom all 4 raters agreed is 0.62assigned all patients as having MI).

agreement), was moderate between a critical care traineeand each of the 3 intensivists (ϕ, 0.42–0.70, moderate tosubstantial agreement), and was lowest between 2 intensi-vists at different institutions (ϕ, 0.12–0.39, slight to fairagreement). The raw agreement and κ statistics are alsopresented for each pair of raters. One rater (designated“intensivist” in Table 2) diagnosed MI in all patients,resulting in noncalculable κ estimates.

4. Discussion

In this study, 4 intensivists independently interpretedECGs that were performed using a scheduled screeningprotocol during the patient's ICU stay in conjunction withthe corresponding screening cTn measurements to classifypatients according to the presence or absence of MI occurringduring their ICU stay. We found that raw agreement washigh. κ between all 4 raters was factitiously low (κ = 0.24,fair) because of the high prevalence of MI in our sample of45 cases and hence the high level of chance agreement. Incomparison, ϕ showed moderate to substantial agreement.

In practice, ECG interpretation is subjective and requiresthe reader to visually analyze information and then consideradditional information (such as presence of elevated cTn) tomake a decision regarding the presence or absence of MI.Previous studies have examined the agreement in ECGinterpretation in assessing suitability for thrombolysis inacute coronary syndromes [14] and agreement for diagnos-ing MI in the emergency department [15]. Reliability of ECGinterpretation for thrombolysis was very good to substantialamong cardiologists compared to cardiology fellows andmedical residents [14]. In contrast, the reliability wasrelatively poor in the emergency department setting [15].We previously assessed the reliability of ECG interpretationin the ICU setting, with and without knowledge of cTn levelswhen evaluating specific ECG features [8]. To extend thiswork, the current study examined the agreement among4 raters for diagnosing MI, incorporating both ECGinterpretation and decision making with cTn values.

Multiple factors influence ECG interpretation, includingthe training and experience of the interpreting physician

ion for diagnosing MI

t

Primaryadjudicator 1and trainee

Primaryadjudicator 2and trainee

Intensivistand trainee

All4 raters

0.89 0.76 0.87 0.790.55(0.20–0.90)

0.40(0.16–0.65)

0 0.24(0.15–0.32)

0.62 0.70 0.42 -

raters is based on a mean of all 45 × 6 possible pairwise agreements. Raw. Some κ values are noncalculable because of extreme distributions (1 rater

451Diagnosis of MI in critically ill patients

[14,16,17] and knowledge of the clinical history [18]. In thisstudy, 3 raters were intensivists and 1 was a critical carefellow. The agreement between the fellow compared with theintensivists on the diagnosis of MI showed moderateagreement. In contrast, agreement was highly variablebetween intensivists. We did not evaluate agreement betweenmultiple trainees so we were unable to evaluate the level ofexperience on reliability. Furthermore, the raters wereblinded to the clinical history; consequently, we cannotevaluate the impact of clinical history on establishing adiagnosis of MI.

Limitations to this study include the relatively smallnumber of patients who were enrolled from a singlecenter. Because our ICU admits predominantly medical-surgical patients, inferences about ECG interpretation inother patient populations are limited. Second, becausecardiac catheterization or coronary artery interventionoccurred very infrequently, the reference standard for thediagnosis of MI of all raters was the consensus opinion ofthe 2 primary adjudicators using the ESC/ACCF/AHA/WHF Task Force criteria [1] rather than findings oncoronary angiography.

Strengths of this study include the quadruplicateindependent interpretation of ECGs to diagnose MIinterpreted with knowledge of cTn values, as in practice.We used frequent 12-lead ECG recordings occurringthroughout the patient's ICU stay to detect MI, which hasbeen shown to have higher sensitivity for detectingmyocardial ischemia compared to routine surveillancewith continuous monitoring with 5-electrode/2-lead ECGmonitoring ST-segment trends [19]. Finally, we usedrigorous analytic methods to measure agreement and usedmultiple agreement parameters to address the possibility ofimprecise results due to chance.

Adjudicating the presence or absence of MI in critically illpatients yielded substantial raw agreement and fair chance-corrected agreement (κ). Chance-independent agreement (ϕ)showed slight to substantial agreement between pairs.Diagnosing MI in the ICU remains a challenge due tovariable 12-lead ECG interpretation. Such variation inpractice may contribute to underrecognition and under-treatment of MI during critical illness.

Acknowledgments

We thank David Wei for his help with the data entry. Thisstudy was funded by a grant from the Regional MedicalAssociates of McMaster University (Hamilton, Ontario,Canada) and a grant from the Ontario Association of MedicalLaboratories (North York, Ontario, Canada). Dr W Limholds a Randomized Controlled Trials Mentoring Awardfrom the Canadian Institutes of Health Research (Ottawa,Ontario, Canada) with Dr D Cook. Dr D Cook is also aresearch chair and holds a Mentoring Award of the Canadian

Institutes for Health Research. Dr PJ Devereaux holds a NewInvestigator Award of the Canadian Institutes of HealthResearch. Dr M Crowther holds a Career Investigator Awardfrom the Heart and Stroke Foundation of Canada (Toronto,Ontario, Canada).

Appendix A. Author contributions

Obtaining funding: W Lim, D Cook, MA Crowther,PJ Devereaux

Conception and design: D Cook, W Lim, A Tkaczyk,P Holinski, E McDonald, PJ Devereaux

Data collection: P Holinski, A Tkaczyk, E McDonald,I Qushmaq, D Cook, M Jacka, V Khera, W Lim

Statistical analysis: I, Terrenato, D Heels-Ansdell, W Lim,PJ Devereaux, D Cook, H Schunemann

Drafting of article: W Lim, D CookCritical revision of article: PJ Devereaux, MA Crowther,

A Tkaczyk, E McDonald, P Holinski, I Terrenato, D Heels-Ansdell, M Jacka, V Khera, H Schunemann

Guarantor: D Cook

References

[1] Thygesen K, Alpert JS, White HD, Jaffe AS, Apple FS, Galvani M,et al. Universal definition of myocardial infarction. Circulation 2007;116:2634-53.

[2] Guest TM, Ramanathan AV, Tuteur PG, Schechtman KB, LadensonJH, Jaffe AS. Myocardial injury in critically ill patients. JAMA 1995;273:1945-9.

[3] Klein Gunnewiek JMT, van de Leur JJJ. Elevated troponinT concentrations in critically ill patients. Intensive Care Med 2003;29:2317-22.

[4] Arlati S, Brenna S, Prencipe L, Marocchi A, Casella GP, Lanzani M,et al. Myocardial necrosis in ICU patients with acute non-cardiacdisease: a prospective study. Intensive Care Med 2000;26:31-7.

[5] Jeremias A, Gibson CM. Narrative review: alternative causes forelevated cardiac troponin levels with acute coronary syndromes areexcluded. Ann Intern Med 2005;142:786-91.

[6] Roongsritong C, Warraich I, Bradley C. Common causes of troponinelevations in the absence of acute myocardial infarction: incidence andclinical significance. Chest 2004;125:1877-84.

[7] Ammann P, Fehr T, Minder EI, Gunter C, Bertel O. Elevation oftroponin I in sepsis and septic shock. Intensive Care Med 2001;27:965-9.

[8] Lim W, Qushmaq I, Cook DJ, Devereaux PJ, Heels-Ansdell D,Crowther MA, et al. Reliability of electrocardiogram interpretation incritically ill patients. Crit Care Med 2006;34:1338-43.

[9] Lim W, Qushmaq I, Cook DJ, Crowther MA, Heels-Ansdell D,Devereaux PJ. Elevated troponin and myocardial infarction in theintensive care unit: A prospective study. Crit Care 2005;9:R636-44.

[10] Cook RJ, Farewell VT. Conditional inference for subject-specific andmarginal agreement: two families of agreement measures. Can J Stat1995;23:333-44.

[11] Walter SD. Small sample estimation of log odds ratios from logisticregression and fourfold tables. Stat Med 1985;4:437-44.

[12] Landis JR, Koch GG. The measurement of observer agreement forcategorical data. Biometrics 1977;33:159-74.

452 W. Lim et al.

[13] Maclure M, Willett WC. Misinterpretation and misuse of the kappastatistic. Am J Epidemiol 1987;126:161-9.

[14] Massel D. Observer variability in ECG interpretation for thrombolysiseligibility: experience and context matter. J Thromb Thrombolysis2003;15:131-40.

[15] Snoey ER, Housset B, Guyon P, ElHaddad S, Valty J, Hericord P.Analysis of emergency department interpretation of electrocardio-grams. J Accid Emerg Med 1994;11:149-53.

[16] Salerno SM, Alguire PC, Waxman HS. Competency in interpretationof 12-lead electrocardiograms: a summary and appraisal of publishedevidence. Ann Intern Med 2003;138:751-60.

[17] Salerno SM, Alguire PC, Waxman HS, American College ofPhysicians. Training and competency evaluation for interpretation of12-lead electrocardiograms: recommendations from the AmericanCollege of Physicians. Ann Intern Med 2003;138:747-50.

[18] Hatala R, Norman GR, Brooks LR. Impact of a clinical scenario onaccuracy of electrocardiogram interpretation. J Gen Intern Med 1999;14:126-9.

[19] Martinez EA, Kim LJ, Faradat N, Rosenfeld B, Bass EB, PerlerBA, et al. Sensitivity of routine intensive care unit surveillancefor detecting myocardial ischemia. Crit Care Med 2003;31:2302-8.