reliability and validity of the respiratory...

34

ORIGINAL ARTICLEMalaysian Journal of Medical Sciences, Vol. 11, No. 2, July 2004 (34-40)

RELIABILITY AND VALIDITY OF THE RESPIRATORY SCORE INTHE ASSESSMENT OF ACUTE BRONCHIOLITIS

Ho Jen Chin & Quah Ban Seng

Department of PediatricsSchool of Medical Sciences, Universiti Sains Malaysia, Health Campus

16150 Kubang Kerian, Kelantan, Malaysia

Children with bronchiolitis from Hospitals Universiti Sains Malaysia (HUSM) andHospital Kota Bharu (HKB) were student using the Kristjansson Respiratory Scoreand the Wang Respiratory Score respectively. Saturation of oxygen (SaO2) wasmeasured with a pulse-oximeter while the child is breathing room-air. Twoobservers assessed the respiratory scores in all children independently. Thecorrelation between respiratory scores and SaO2 was assessed using Spearman’sRho, and the inter-rater reliability of respiratory scores determined using intraclass-corelation coefficient. There were 29 children in HUSM and 25 from HKB with amedian age of 8 months (IQR 4.5 months) and 9 months(IQR 7 months) respectively.In HUSM, the median Kristjansson Respiratory Score for both observers was 4(IQR 2), and the median SaO2 was 96% (IQR 3%). The correlation coefficientbetween the Kristjansson Respiratory Score and SaO2 for the first observer was –0.75 (p <0.001), and for the second observer –0.73 (p <0.001). In HKB, the medianWang Respiratory Score was also similar for both observers (median 4 IQR 4.5),and the median (IQR) for SaO2 was 96% (2%). The correlation coefficient betweenthe Wang Respiratory Score and SaO2 for the first observer was –0.41 (p = 0.04)and for the second observer –0.43 (p = 0.03). The inter-rater reliability betweenthe first and second observer was high for both the Kristjansson Respiratory (ICC0.89) and the Wang Respiratory Scores (ICC 0.99). In conclusion the validity ofthe Kristjansson Respiratory Score was high whereas the validity of the WangRespiratory Score was moderate in the assessment of the severity of acutebronchiolitis. Both respiratory scores and physical signs showed high agreementbetween observers. The Kristjansson Respiratory Score should be considered foruse by medical personnel in the assessment of the severity of acute bronchiolitis inchildren.

Key words : bronchiolitis, respiratory scores, validity, reliability

Submitted-12.5.2003, Revised-11.05.2004, Accepted-25.5.2004

Introduction

Bronchiolitis is a clinical syndromecharacterised by pulmonary hyperinflation andwheezing in association with a viral lowerrespiratory infection (1). It is an acute, highlycommunicable lower respiratory tract infection,which is the most common serious illness inchildhood and the most common reason forhospitalisation of infants beyond the neonatal period.Its morbidity is high, accounting for 17% of

hospitalisations in infants’ (9 admissions per 1000child-year) in New York State (2).

The assessment of the severity of bronchiolitisis necessary not only in daily practice but also inclinical trials. Children with severe bronchiolitis areat risk for severe hypoxaemia, and in daily practicepaediatricians are often faced with the task ofdeciding which children require hospitalisation soas to avoid unnecessary morbidity and mortality. Aspulmonary functions are difficult to perform ininfants, respiratory scores are often used to evaluate

35

the efficacy of medications for bronchiolitis. Variousrespiratory scores have been used to evaluate theresponse of acute bronchiolitis towards many typesof medication such as epinephrine (3), nebulisedalbuterol (4), nebulised metaproterenol (5),nebulised racemic adrenaline (6) and otherbronchodilators (7). Respiratory scores consistingof a combination of clinical symptoms and physicalsigns, are frequently used to estimate the severityof acute lower airway infection in children. Reportson the reliability of such signs have shownconsiderable disagreement among physicians,leading to unreliable clinical observations inrespiratory diseases. Therefore, it is important toinvestigate the inter-observer agreement on thepresence or absence of physical signs in acutebronchiolitis in order to derive a consistent diagnosisfrom different examiners irrespective of thesituation. A search of the medical literature (1966-1992) resulted in the identification of no less than16 clinical asthma scores (8). However, most scoreswere designed ad hoc based on clinical experienceand face validity, and lacked information onreliability and validity(8).

In this study we chose to evaluate two

respiratory scores, the Kristjansson RespiratoryScore (6) and the Wang Respiratory Score (9) forthe assessment of the severity of acute bronchiolitisbecause the physical signs used in these tworespiratory scores are simple and well categorised.The first objective of this study was to examine thevalidity of both the Kristjansson Respiratory Scoreand the Wang Respiratory Score by correlating thescores with oxygen saturation in room air in childrenwith acute bronchiolitis. The second objective wasto determine the inter-rater reliability of the aboveclinical respiratory scores in the assessment of theseverity of acute bronchiolitis.

Methodology

Children admitted to Hospital USM (HUSM)from June 2000 to July 2001 and Hospital KotaBharu (HKB) from July 2000 to June 2001 for acutebronchiolitis were recruited for the study. Childrenaged six to eighteen months who were admitted forthe first episode of acute wheezing or rhonchi,tachypnoea, and chest retraction, preceded by orassociated with cough, coryza, rhinorrhoea and hadan axillary temperature >37.5ºC were eligible for

Table 1: Clinical signs and scores for the Kristjansson Respiratory Score

Table 2: Clinical signs and score of the Wang Respiratory Score

RELIABILITY AND VALIDITY OF THE RESPIRATORY SCORE IN THE ASSESSMENT OF ACUTE BRONCHIOLITIS

Score

0

1

2

*a) Not affected if activity and feeding is normal, b) moderately affected if activity and feeding is less than normal,c) severely affected if child looks ill and feeds poorly.

<40

40-60

>60

None

Moderate (Costodia-phragmatic)

Severe (As in 1 plus rib& jugular retraction)

Vesicular

Wheeze =/-rhonchi/rale

Severe wheeze=/-rhonchi/rale

Normal

Pallor

Cyanosis

NotaffectedModeratelyaffected

Severelyaffected

Chest Recession Breath Sound SkinColour

*GeneralCondition

RespiratoryRate(breaths/minute)

Score

01

2

<3030-45

46-60

NoneTerminal expiration or onlywith stethoscopeEntire expiration or audible on expiration without stethoscope

NoneIntercostalrecessionTrachea-sternalrecession

Normal

3 >60 Inspiration and expirationwithout stethscope

Severe with nasal flow

Irritable/lethargic/poor feeding

Wheezing Retraction GeneralCondition

RespiratoryRate

36

the study. Children were excluded from the study ifthey had: i) an underlying disease that might affectthe cardiopulmonary status (e.g. bronchopulmonarydysplasia, prematurity, assisted ventilation duringthe neonatal period, congenital heart disease orimmunodeficiency); ii) asthma diagnosed by aphysician, iii) wheezing or cough that had previouslybeen treated with bronchodilators or corticosteroidswithin the preceding 2 weeks; or iv) had been treatedwith 35% or more of inspired oxygen by head boxor equivalent during the neonatal period.

Children admitted in HUSM were assessedwith the Kristjansson Respiratory Score (6) whereasthose admitted in HKB were assessed with the WangRespiratory Score (9). The Kristjansson RespiratoryScore consists of five clinical signs (Table 1) withscores for each sign ranging from zero to two. Thepossible total scores ranged from 0-10. Whistlingexpiratory sounds heard with and without thestethoscope was considered as rhonchi and wheezerespectively.

The 4-item Wang Respiratory Score consistsof respiratory rate, wheezing, chest retraction andgeneral condition (Table 2) and each clinical signwas scored from zero to 3 except for the generalcondition, which was scored zero for normal and 3

for irritability or lethargy. The possible total scoreranged from 0-12. The respiratory rate wasdetermined by counting the respiration for 30seconds in both respiratory scores.

Saturation of oxygen (SaO2), which was thegold standard for severity of acute bronchiolitis, wasmeasured with a pulse-oximeter (Nellor PuritonBennett NPB-295 in HUSM and MSA MedicalProduct, Pittsburgh, 8000 in HKB) while the childwas breathing room-air. The maximum SaO2 wasread for each child after the pulse-oximeter had beenattached to the patient for at least five minutes. Inchildren who were given bronchodilators at thecasualty, SaO2 was assessed 1 hour after treatment.

Two observers independently assessed allchildren on both respiratory scores. The firstobserver was the main investigator and the secondobserver was either a Paediatrician, PaediatricMedical Officer, House Officer, or Medical Student(fourth or fifth year). All observers were briefed onthe scoring system of each respiratory score and thepurpose of the study. All observations were takenwhen the child was awake and not crying.

Approval for the study was obtained from theHospital Director of both hospitals and the Researchand Ethics Committee, School of Medical Sciences,

Table 3: Correlation between each component of the Kristjansson Respiratory Score with the SaO2.

Table 4: Correlation between each component of the Wang Respiratory Score with SaO2


Clinical Signs First ObserverSpearmen’s Rho p Value

Second ObserverSpearmen’s Rho p Value

Respiratory Rate

Chest Recession

Breath Sound

Skin Colour

General Condition

- 0.39

- 0.54

- 0.52

- 0.48

- 0.74

0.04

0.002

0.004

0.008

<0.001

- 0.34

- 0.55

- 0.35

- 0.32

- 0.70

0.07

0.002

0.07

0.09

<0.001

Clinical Signs First ObserverSpearmen’s Rho p Value

Second ObserverSpearmen’s Rho p Value

Respiratory Rate

Wheezing

Chest Retraction

General Condition

- 0.27

- 0.40

- 0.28

- 0.48

0.09

0.05

0.18

0.01

- 0.31

- 0.40

- 0.34

- 0.48

0.13

0.05

0.10

0.01

37

Universiti Sains Malaysia. A verbal consent wasobtained from parents before the enrolment ofchildren in the study. The data was analysed usingSPSS for Windows version 9.0. The validity of therespiratory scores was assessed by correlating totalrespiratory scores of the main investigator with SaO2using Spearman’s Rho. A sample size of thirty wasestimated to detect a correlation coefficient of 0.5at an alpha error of 5% and beta error of 20%. Theinter-rater reliability of the scores (between first andsecond observer) was determined using a weightedkappa for the ordinal variables and intraclass-correlation coefficient (ICC) for the continuousvariables. A p value of less than 0.05 was consideredstatistically significant.

Results

There were 29 children from HUSM and 25from HKB with a median age of 8 months (IQR 4.5months) and 9 months(IQR 7 months) respectively.There were 19 (66%) boys and 15 boys (60%) inHUSM and HKB respectively. In HUSM there were28 (97%) Malays, but in HKB all children wereMalays.

In HUSM, the median (IQR) KristjanssonRespiratory Score for both observers was 4 (2), andthe median (IQR) for SaO2 was 96% (3%). Therewas a significant negative correlation between theKristjansson Respiratory Score and SaO2 for boththe first observer (r = –0.75, p <0.001) and secondobserver (r= –0.73, (p <0.001) (Figure 1).In HKB, the median (IQR) Wang Respiratory Scorefor both observers was 4 (4.5), and the median (IQR)

for SaO2 was 96% (2%). There was also a significantcorrelation between the Wang Respiratory Score andSaO2 both the first observer (r= –0.41, p = 0.04) andsecond observer (r= –0.43 p = 0.03) (Figure 2).However the magnitude of the correlation was lessthan the Kristjansson Respiratory Score.

Among individual components of theKristjansson Respiratory Score (Table 3) generalcondition had the strongest negative correlation withSaO2 for both observers. For the first observer,respiratory rate had the lowest correlation with SaO2but for the second observer, skin colour had thelowest correlation with SaO2.

Among individual components of the WangRespiratory Score (Table 4) general condition alsoshowed the highest correlation with SaO2 for bothobservers. However respiratory rate had the lowestcorrelation with SaO2 in both observers.

The inter-rater reliability between the first andsecond observer was high for the KristjanssonRespiratory (ICC 0.89). The inter-rater liability forindividual clinical signs Kristjansson RespiratoryScore also high with weighted kappa ranging from0.79 to 0.91 (Table 5). Respiratory rate showed thehighest reliability with kappa value of 0.91, followedby general condition (0.88), chest recession (0.84),breath sound (0.81) and skin colour (0.79).

The inter-rater reliability of the WangRespiratory Score between the first and secondobserver was high (ICC 0.99). Individualcomponents of the respiratory score also showedhigh inter-rater-raliability with weighted kapparanging from 1.00 to 0.90. Among the four physicalsigns, wheezing and general condition had thehighest reliability (weighted kappa of 1.00). All the

Table 5: The inter-rate reliability of the Kristjansson Respiratory Score and Wang RespiratoryScore between the First and Second observer.


KristjanssonRespiratory Score

WeightedKappa

WeightedKappa

Wang RespiratoryScore

Respiratory Rate 0.91 0.90Respiratory Rate

Chest Recession 0.84 1.00Wheezing

Breath Sound 0.81 0.97

Skin Colour 0.79

Chest Retraction

General Condition 0.88 1.00General Condition

38

components in the Wang Respiratory Score showedhigher inter-rater reliability compared to those ofKristjansson Respiratory Score.

Discussion

In this study, both Kristjansson RespiratoryScore and Wang Respiratory Score correlatedsignificantly with SaO2. However, the magnitudeof correlation for the Kristjansson Respiratory Scorewas higher than the Wang Respiratory Score. TheSaO2 determined by pulse oximeter is the single bestobjective predictor of severity in infants with acutebronchiolitis (10, 11) and was chosen as the goldstandard in this study. The severity of acutebronchiolitis is closely related to the degree ofhypoxaemia and hypercapnoea arising from anabnormal distribution of ventilation relative toperfusion. Thus both respiratory scores comprisedof clinical signs which represent the degree ofhypoxaemia. The average negative correlationbetween oximeter and the two respiratory scores wassimilarly observed by Hal et al (12).

In both respiratory scores, general conditionhad the highest correlation with SaO2. The moredetailed evaluation of general condition in theKristjansson Respiratory Score resulted in a highervalidity than that in the Wang Respiratory Score.Shaw et al. (10) showed that a simple clinicaldescription of the infants’ general appearance (“wellappearance”, “ill – not toxic” or “toxic”) by anattending physician was a helpful measure in

assessing the degree of illness in children with acutebronchiolitis. In the Kristjansson Respiratory Score,chest recession also showed a significant correlationwith the saturation of oxygen. It was postulated thatchest recession reflects the effort to improveoxygenation in hypoxaemic children with lowerrespiratory infection and the use of accessorymuscles had been shown to correlate strongly withthe severity of disease (10). In this study, chestretraction in the Wang Respiratory Score showed apoorer correlation with SaO2 than chest recession inthe Kristjansson Respiratory Score. A simpledescription of chest recession such as none, moderateand severe as in Kristjansson Respiratory Score hasbeen shown to be more valid than the more detaileddescription of chest retraction in the WangRespiratory score. In both respiratory scoresrespiratory rate and breath sounds (wheezing) onlyhad an average or non-significant correlation withSaO2. Children with acute bronchiolitis may beclinically well even when the inspiratory andexpiratory airway resistance is high (13), becauseincreased resistance is due to dynamic narrowingof the airways. Thus, the respiratory rate and breathsounds may not correlate well with SaO2 and are notgood indicators of disease severity. This finding wassimilarly observed by Hall et al (12) who showed apoor correlation of respiratory rate and diseaseseverity.

Skin colour was included in the KristjanssonRespiratory Score but not the Wang RespiratoryScore. However the correlation with saturation of

Figure 1: Scatter plot of the Kristjansson Respiratory Scoreversus the saturation of oxygen


39

oxygen was poor in both observers. The result inthis study concurs with the findings of Shaw et al(10) and Mulholland et al (11). This is not surprisingas skin colour is affected by many factors besidesperipheral tissue perfusion and oxygenation.Cyanosis which only occurs in children with severehypoxaemia may not be easily detectable in childrenwith anaemia and dark pigmentation. In summarythis study showed that general condition and chestrecession were better predictors of severity of acutebronchiolitis compared with respiratory rate, breathsounds and wheezing

Both the Kristjansson Respiratory Score andWang Respiratory Score showed high inter-raterreliability, with the Wang Respiratory Score havinga slightly reliability than the KristjanssonRespiratory Score. This difference is probably dueto the simplicity of the Wang Respiratory Score,especially the classification of general conditionwhich has only two scores. In the KristjanssonRespiratory Score, skin colour had the lowestreliability. This may be due to the difficulty inassessing skin colour which is also affected byanaemia and pigmentation of children in this region.In the Wang Respiratory Score, general conditionand wheezing had the highest reliability. This is mostprobably due to the simplicity of classification ofthese two signs. Physical signs, which can beobjectively evaluated, have been shown to have ahigher inter-rater agreement than subjective physicalsigns (14, 15). Spiteri et al (16) showed that thereliability of recognising physical signs in theexamination of the chest among 24 physicians waspoor. Wang et al (15) evaluated the inter-rater

agreement for respiratory signs in infantshospitalised with lower respiratory infections, andshowed that the agreement beyond chance were allbelow 50%. In these studies, poor inter-rateragreement was associated with poorly defined andpoorly categorised physical signs. The variation ofclinical experiences among different observers couldbe also affecting the reliability of a score.

Even though two different brands of pulseoximeter were used in the two hospitals, each scorewas evaluated against a single brand of pulseoximeter. As the readings of each pulse oximeterwere not expected to differ greatly from each other,the validity of both respiratory scores may becompared. Environmental factors such as roomtemperature and lighting, and physical conditionssuch as hypothermia and hypotension are notexpected to affect the oximeter reading as theenvironment of the hospitals are not unduly differentand the severity of bronchiolitis of children admittedin both hospitals is not markedly different. Despitethe differences in clinical experiences among theobservers in this study the reliability of therespiratory scores was high.

In conclusion the validity of the KristjanssonRespiratory Score was high whereas the validity ofthe Wang Respiratory Score was moderate in theassessment of the severity of acute bronchiolitis. Inthe Kristjansson Respiratory Score, generalcondition and chest retraction were the bestindicators of the severity of disease. Both respiratoryscores and physical signs showed high agreementbetween observers. Between the two respiratoryscores the Kristjansson Respiratory Score is more

Figure 2: Scatter plot of the Wang Respiratory Score versusthe saturation of oxygen


40

valid and reliable and should be considered for useby medical personnel in the assessment of theseverity of acute bronchiolitis.

Correspondence :

Dr. Ho Jen Chin MBBS (Malaya), MMed (USM),MRCPCH (UK)Department of Paediatrics,School of Medical Sciences,Universiti Sains Malaysia, Health Campus,16150 Kubang Kerian, Kelantan, Malaysia

References

1. Cengizlier R, Saraclar Y, Adalioglu G, Tuncer A. Effectof oral and inhaled salbutamol in infants withbronchiolitis. Acta Paediatr Jpn 1997; 39: 61-3.

2. McConnochie KM, Roghmann KJ, Liptak GS.Hospitalization for lower respiratory tract illness ininfants: variation in rates among counties in New YorkState and areas within Monroe County. J Pediatr 1995;126: 220-9.

3. Lowell DI, Lister G, Von Koss H, McCarthy P.Wheezing in infants: the response to epinephrine.Pediatrics 1987;79:939-45.

4. Gadomski AM, Aref GH, el Din OB, el Sawy IH,Khallaf N, Black RE. Oral versus nebulized albuterolin the management of bronchiolitis in Egypt. J Pediatr1994; 124: 131-8.

5. Alario AJ, Lewander WJ, Dennehy P, Seifer R, MansellAL. The efficacy of nebulized metaproterenol inwheezing infants and young children. Am J Dis Child1992; 146: 412-8.

6. Kristjansson S, Lodrup Carlsen KC, Wennergren G,Strannegard IL, Carlsen KH. Nebulised racemicadrenaline in the treatment of acute bronchiolitis ininfants and toddlers. Arch Dis Child 1993; 69: 650-4.

7. Wang EE, Milner R, Allen U, Maj H. Bronchodilatorsfor treatment of mild bronchiolitis: a factorialrandomised trial. Arch Dis Child 1992; 67: 289-93.

8. van der Windt DA, Nagelkerke AF, Bouter LM,Dankert-Roelse JE, Veerman AJ. Clinical scores foracute asthma in pre-school children. A review of theliterature. J Clin Epidemiol 1994; 47: 635-46.

9. Wang EE, Law BJ, Boucher FD, Stephens D, RobinsonJL, Dobson S, Langley JM, McDonald J, McDonaldNE, Mitchell I. Pediatric Investigators CollaborativeNetwork on Infections in Canada (PICNIC) study ofadmission and management variation in patientshospitalized with respiratory syncytial viral lowerrespiratory tract infection. J Pediatr 1996; 129: 390-5.

10. Shaw KN, Bell LM, Sherman NH. Outpatientassessment of infants with bronchiolitis. Am J Dis Child1991; 145: 151-5.

11. Mulholland EK, Olinsky A, Shann FA. Clinicalfindings and severity of acute bronchiolitis. Lancet1990; 335: 1259-61.

12. Hall CB, Hall WJ, Speers DM. Clinical andphysiological manifestations of bronchiolitis andpneumonia. Outcome of respiratory syncytial virus. AmJ Dis Child 1979; 133: 798-802.

13. Wohl ME, Stigol LC, Mead J. Resistance of the totalrespiratory system in healthy infants and infants withbronchiolitis. Pediatrics 1969; 43: 495-509.

14. Smyllie HC, Blendis LM, Armitage P. ObserverDisagreement in Physical Signs of the RespiratorySystem. Lancet 1965; 40: 412-3.

15. Wang EE, Milner RA, Navas L, Maj H. Observeragreement for respiratory signs and oximetry in infantshospitalized with lower respiratory infections. Am RevRespir Dis 1992; 145: 106-9.

16. Spiteri MA, Cook DG, Clarke SW. Reliability ofeliciting physical signs in examination of the chest.Lancet 1988; 1: 873-5.


reliability and validity of the respiratory...

Documents