a randomized trial of peer review: the uk national chronic obstructive pulmonary disease resources...

A randomized trial of peer review: the UK National ChronicObstructive Pulmonary Disease Resources and OutcomesProject: three-year evaluationjep_1639 599..605

Christopher M. Roberts MA MD FRCP ILTHE,1 Robert A. Stone BSc PhD FRCP,2

Rhona J. Buckingham MA,3 Nancy A. Pursey BSc (Hons),4 Derek Lowe MSc Cstat5 andJonathan M. Potter DM FRCP6

1Associate Director, Clinical Effectiveness and Evaluation Unit, Royal College of Physicians, London, UK; Professor of Medical Education, Bartsand the London School of Medicine and Dentistry, Queen Mary University of London, London, UK2Associate Director, Clinical Effectiveness and Evaluation Unit, Royal College of Physicians, London, UK; Consultant Physician, Taunton andSomerset NHS Foundation Trust, Taunton, UK3Project Manager, 4Project Coordinator, 5Medical Statistician, 6Clinical Director, Clinical Effectiveness and Evaluation Unit, Royal College ofPhysicians, London, UK

Keywords

COPD, National Health Service, peer review,quality improvement, randomized trial

Correspondence

Dr Jonathan M. PotterClinical Effectiveness and Evaluation UnitRoyal College of Physicians1 St Andrews PlaceRegents ParkLondon NW1 4LEUKE-mail: [email protected]

No competing interests are declared.

Accepted for publication: 1 December 2010

doi:10.1111/j.1365-2753.2011.01639.x

AbstractRationale Peer review has been widely used within the National Health Service to facili-tate health quality improvement but evaluation has been limited particularly over thelonger-term. Change within the National Health Service (NHS) can take a prolonged period– 1–2 years – to occur. We report here a 3-year evaluation of the largest randomized trialof peer review ever conducted in the UK.Aim To evaluate whether targeted mutual peer review of respiratory units brings aboutimprovements in services for chronic obstructive pulmonary disease (COPD) over 3 years.Methods The peer review intervention was a reciprocal supportive exercise that includedclinicians, hospital management, commissioners and patients, which focused on the qualityof the provision of four specific evidence-based aspects of COPD care.Results Follow-up at 36 months demonstrated limited significant quantitative differencesin the quality of services offered in the two groups but a strong trend in favour ofintervention sites. Qualitative data suggested many benefits of peer review in most but notall intervention units and some control teams. The data identify factors that promote andobstruct change.Conclusion The findings demonstrate significant change in service provision over 3 yearsin both control and intervention sites with great variability in both groups. The combinedquantitative and qualitative findings indicate that targeted mutual peer review is associatedwith improved quality of care, improvements in service delivery and with changes withindepartments that promote and are precursors to quality improvement. The generic findings ofthis study have potential implications for the application of peer review throughout the NHS.

IntroductionPeer review has been used both within the National Health Service(NHS) and overseas as a mechanism for affecting health qualityimprovement, but there is surprisingly little evidence for its effec-tiveness. Previous reports have detailed process [1] or the percep-tions of those involved as indicators of success [2]. In particular,there have been no long-term evaluations of peer review to estab-lish whether changes occur, are sustained and whether perceptionsof change are matched by reality.

The National Chronic Obstructive Pulmonary DiseaseResources and Outcomes Project (NCROP) has been a tripartiteinitiative between the Royal College of Physicians (RCP), BritishThoracic Society (BTS) and British Lung Foundation (BLF),which included a randomized control trial of peer review. Datafrom the original NCROP peer review study undertaken in 2007[3,4] suggested there was no statistically significant difference inservice change between the intervention and control.

Qualitative data collected using ‘change diaries’ did imply moreservice development within the intervention group, and an external

bs_bs_banner

Journal of Evaluation in Clinical Practice ISSN 1365-2753

© 2011 Blackwell Publishing Ltd, Journal of Evaluation in Clinical Practice 18 (2012) 599–605 599

evaluation suggested effects of peer review which, while notchanges in themselves, might act as precursors to change [5]. Thiswas apparent particularly around hospital palliative care and non-invasive ventilation (NIV) services. Interviewees acknowledgedthe time taken to achieve change within the NHS and measuringoutcomes 1 year after the intervention (as with the 2008 NationalCOPD Audit) might not allow adequate time for change to occur.Thus, in 2009, further funding was secured from The Health Foun-dation to investigate further whether longer-term quantitative orqualitative changes had occurred among the 100 participatingNCROP units in 2010.

AimsThe aim of the project has been to evaluate whether targetedmutual peer review of respiratory units brings about improvementsin services for chronic obstructive pulmonary disease (COPD)over a 3-year period.

The objectives of this study were to:• re-survey the NCROP units to measure quantitative changesover 3 years• re-survey the NCROP units to measure qualitative changes over3 years• evaluate the factors promoting or obstructing change

MethodsThe methodology and findings of the original NCROP are welldocumented elsewhere [3–6]. In summary, the project was over-seen by a Steering Group of COPD professionals and managersincluding Department of Health representation. All UK hospitalsadmitting acute COPD cases were invited to participate in a pro-spective randomized trial of focused mutual peer review. Peerreview teams consisted of lead Respiratory Consultant, anotherhealth care respiratory professional (usually a nurse or physio-therapist), hospital service manager, primary care commissioningrepresentative and a patient representative. The Steering Groupidentified four key COPD service areas from previous UK nationalCOPD audits [7,8] highlighted as having variable provision acrossthe country to be the focus of peer review. Each has a goodevidence base and national guideline recommendations [9–16].These services were non-invasive ventilation for acute type IIrespiratory failure treated in hospital, facilitated (early) dischargeschemes, pulmonary rehabilitation across both secondary andprimary care interface, and assessment for out-of-hospital oxygenrequirement. Quality indicators were identified for each, drawnmainly from national guidelines with others agreed by expert panelconsensus, a total of 46 in all. Another area ‘of interest’, palliativecare for COPD, was added at the request of BLF patient groups, torecord current practice and planned improvements but withoutindicators of quality. Quality indicator scores were derived byscoring unit responses to each of the 46 indicators [12 NIV, 11pulmonary rehabilitation (PR), 9 early discharge scheme (EDS),14 oxygen provision] as 2 = met in full, 1 = only partially met,0 = not met at all, by summing the scores for each service area andscaling service area totals from 0 to 100.

Reciprocal review visits were arranged within a 4-week period.A semi-structured programme for the visits was developed by theSteering Group. This directed discussion towards the COPD

service areas but also allowed for wider discourse about the COPDservice in general. In advance, visiting teams received a unit’sself-completed baseline pro forma during Spring 2007 describingtheir service provision and attainment (met in full, partially met,not met at all) of the quality indicators. A joint meeting at the endof each visit produced recommendations and a ‘service develop-ment plan’ supported by all parties was then required within 4weeks.

Units first submitted service provision data to NCROP inNovember 2005. The number of service areas provided by units(of non-invasive ventilation, pulmonary rehabilitation and earlydischarge scheme) ranged from 0 to 3 and to maximize the poten-tial for units to learn from each other those with 0/1 services werepaired to those with all 3 services, units with 2 services werepaired to units with all 3 services and then all remaining units werepaired. Where possible paired units were located within 2.5-hourtravel of each other, but not in adjacent districts. Each pair of unitswas randomized 3:2 to either the reciprocal peer review group (27pairs) or to the control group (23 pairs).

The 100 NCROP units were asked to participate in the re-surveyof 2010. The original 2007 NCROP pro forma was used for there-survey with certain additions. These were a revised set of orga-nizational questions and an extra section for intervention units torecord outcomes of the service development plans agreed at thepeer review visits in 2007. The final change diary was revised tocapture change in 2010 from the last submission in winter 2007/08. Ethical approval was sought as an amendment to the originalproposal in December 2009 and was granted by the Joint UCL(University College London)/UCLH (University College LondonHospital) Committees on the Ethics of Human Research (Com-mittee A) in February 2010.

Quantitative data were analysed using spss version 18. Giventhere were notable baseline differences between interventiongroups and control groups in pulmonary rehabilitation qualityscores it was appropriate to run adjusted analyses. Medianregression methods were used, specifically the ‘qreg’ procedurefrom Stata software with bootstrapping and with 2007 baselinescores categorized as quartiles for the baseline covariate. Quali-tative data were processed using grouped themed independentanalyses by two (CMR, JMP) of the researchers and changeswere assessed as being positive or negative. Themes werereviewed by the researchers, and collated under combinedheadings.

Results

Participation

Eighty-two of the original 100 units from the 2007 survey partici-pated in the 2010 re-survey. The 18 units that did not participate in2010 were different statistically from the 82 that did participate inthat fewer had their respiratory department in the Trust on a singlesite in 2007 (6/18 vs. 52/82, P = 0.03, Fisher’s exact test), pulmo-nary rehabilitation quality scores were better (median 82 vs. 77,P = 0.05, Mann–Whitney test) and there were greater numbersof pulmonary rehabilitation standards being fully met (median 11vs. 7, P = 0.02, Mann–Whitney test). There were no other statis-tically significant participation biases with regard to organizationaland composite quality score data. Reasons given by the 18 for

Peer review in COPD – three-year evaluation C.M. Roberts et al.

© 2011 Blackwell Publishing Ltd600

non-participation included the merging of acute trusts or unit nolonger having an acute admitting unit, the lead consultant from2007 had left, the lead consultant was on sick or maternity leave,and reports of single-handed clinicians feeling overwhelmed withworkload.

The data presented from now on in this report concerns justthose 82 units that participated in both the 2007 and 2010 surveys.

Quantitative data from the 2010 re-survey

Provision and organization of services

Many aspects of the organization of COPD care improved between2007 and 2010 (Table 1), but overall the intervention units werenot associated with any greater provision (retention or gain) thanthe control units.

Quality of services

There were many small changes noted among the 46 individualquality indicators that favoured the intervention units, althoughonly two of these were statistically significant at P < 0.05 [NIV– (1) a selection of nasal and full face masks types and nasalpillows available; (2) there is a policy for providing patient infor-mation about NIV to severe COPD patients while in a stablestate, e.g. in an outpatient setting or upon discharge from

hospital]. Some units changed from partially meeting or notmeeting an indicator in 2007 to meeting it in full in 2010, whilesome units changed in the opposite direction. Of the 46 indica-tors there were better directional changes reported for 36 infavour of the intervention group [7 NIV, 9 PR, 8 EDS, 12 long-term oxygen therapy (LTOT)] and for only 5 in favour ofthe control group (3 NIV, 1 PR, 0 EDS, 1 LTOT) with 5 setsof changes not favouring either group (2 NIV, 1 PR, 1 EDS, 1LTOT).

Quality scores for Pulmonary Rehabilitation improved signifi-cantly between 2007 and 2010 in intervention units (Table 2),although the overall provision of the service in hospitals reduced.However, there were notable baseline differences between inter-vention groups and control groups in pulmonary rehabilitationquality scores (not shown). Median regression methods unad-justed for baseline pulmonary rehabilitation quality score quar-tiles repeated the significant difference between groups(P = 0.001). When adjustment was made for baseline scores as acovariate the difference between groups was no longer signifi-cant (P = 0.16). Differences between groups in regard to NIV,EDS and LTOT remained non-significant following similar base-line adjustment.

There was evidence of marked improvements in the provision ofpalliative care for COPD patients over the 3 years (Table 3), butagain there were no clear results in favour of the interventiongroup.

Table 1 Organizational data at baseline 2007 and at year 3 for hospitals participating in the 2010 NCROP re-survey

If organizational featurewas PRESENT in 2007

If organizational featurewas ABSENT in 2007

% with feature RETAINED in 2010 % with feature GAINED in 2010

Intervention 2010 Control 2010 Intervention 2010 Control 2010

Respiratory department in the Trust on a single site 96% (26/27) 100% (25/25) 7% (1/15) 7% (1/14)Respiratory department in a dedicated area 96% (23/24) 93% (26/28) 12% (2/17) 18% (2/11)On-site palliative care 97% (34/35) 100% (33/33) 17% (1/6) 50% (3/6)On-site clinical psychology 100% (12/12) 86% (12/14) 10% (3/30) 0% (0/25)Written local guidelines for management of COPD 97% (31/32) 88% (23/26) 38% (3/8) 40% (4/10)Specialist respiratory ward 97% (35/36) 100% (34/34) 60% (3/5) 80% (4/5)Speciality triage 81% (22/27) 70% (14/20) 53% (8/15) 47% (9/19)Separate respiratory specialist on-call rota 100% (6/6) 100% (5/5) 3% (1/35) 3% (1/33)HDU available to COPD patients 82% (23/28) 93% (28/30) 38% (5/13) 56% (5/9)Funded smoking cessation programme in Trust 90% (27/30) 80% (16/20) 40% (4/10) 53% (10/19)Formal pulmonary rehabilitation % (PR) programme 76% (26/34) 76% (26/34) 33% (2/6) 20% (1/5)PR programme funded by NHS 88% (21/24) 96% (25/26) 71% (5/7) 100% (2/2)Access to EDS for patients with exacerbation of COPD 100% (22/22) 100% (29/29) 30% (6/20) 30% (3/10)Written self-management advice at discharge on responding

promptly to symptoms of exacerbation75% (15/20) 60% (9/15) 52% (11/21) 50% (12/24)

Local patient support group for respiratory conditions 91% (30/33) 91% (31/34) 43% (3/7) 80% (4/5)Access for patients to a palliative care service 94% (34/36) 94% (30/32) 60% (3/5) 71% (5/7)Unit provides ambulatory oxygen service 100% (24/24) 100% (19/19) 93% (13/14) 80% (16/20)Local PCO engages with respiratory services 94% (33/35) 97% (32/33) 50% (3/6) 67% (2/3)Respiratory interest group/network 77% (27/35) 82% (23/28) 33% (2/6) 80% (8/10)Mechanism to influence local commissioning of care 88% (28/32) 83% (20/24) 63% (5/8) 57% (8/14)PCO leads for respiratory care 70% (16/23) 67% (12/18) 33% (4/12) 45% (5/11)

NCROP, the National Chronic Obstructive Pulmonary Disease Resources and Outcomes Project; COPD, chronic obstructive pulmonary disease; EDS,early discharge scheme; NHS, National Health Service; HDU, high dependency unit; PCO, primary care organization.

C.M. Roberts et al. Peer review in COPD – three-year evaluation

© 2011 Blackwell Publishing Ltd 601

Qualitative data from the 2010 re-survey

Change diaries

Evidence of service change

Forty-one intervention and 37 control units (76% interventionunits and 80% control units) provided further change diary returnsin 2010. One hundred per cent of intervention and 95% controlunits reported at least one change for the better since 2007, and20% intervention and 22% control units at least one change for theworse (Table 4). Changes for the better were more likely to bemapped to the five main areas of service change indicated in theNCROP framework for the intervention units than in the controls.For pulmonary rehabilitation, 22% of control units reportedchanges to the better compared with 44% of intervention units. Forearly discharge schemes, 14% of control units reported changes tothe better compared with 27% of intervention units. For oxygen

services, 27% of control units reported changes to the bettercompared with 37% of intervention units. For non-invasive venti-lation, 14% of control units reported changes to the better com-pared with 39% of intervention units. For palliative care services,16% of control units reported changes to the better compared with24% of intervention units.

Impact of peer review on changes

Thematic analysis (Table 5) indicated very similar themes to thoseidentified in 2008 with the addition of a theme relating to thebenefit of national data – be it from the NCROP study or fromnational COPD audit. In 2008 there appeared to be a clear distinc-tion between community and hospital-based service develop-ments. The control and intervention groups had a similar numberof improvements in the community, but the intervention group hadmore hospital based improvements than the control group. In 2010

Table 2 Hospital variation in quality scores within four key COPD service areas, from the 2010 NCROP re-survey

Service

NCROP ‘intervention’ units NCROP ‘control’ unitsMann–WhitneytestMedian IQR n Median IQR n

2010 re-surveyNIV 77 66–92 42 75 63–88 39Pulmonary rehabilitation 91 82–95 42 86 77–95 39Early discharge* 89 0–100 42 89 72–94 39Early discharge (if EDS in 2010) 94 89–100 28 89 79–99 32Oxygen provision 89 82–96 42 82 64–93 39

Change (2010 minus NCROP 2007 baseline)NIV +13 -4 to 23 41 0 -4 to 13 38 P = 0.11Pulmonary rehabilitation +9 +5 to 16 41 0 -5 to 5 39 P = 0.001Early discharge* 0 0 to 8 41 0 -6 to 6 39 P = 0.28Early discharge (if baseline EDS) 0 -6 to 8 21 0 -8 to 6 29 P = 0.85Oxygen provision +11 0 to 20 41 +7 -4 to 11 39 P = 0.10

Quality indicator scores were derived by scoring unit responses to each of the 46 indicators (12 NIV, 11 pulmonary rehabilitation, 9 early dischargescheme, 14 oxygen provision) as 2 = met in full, 1 = only partially met, 0 = not met at all, by summing the scores for each service area and scalingservice area totals from 0 to 100.*Units with no EDS have a quality score of ZERO.NCROP, the National Chronic Obstructive Pulmonary Disease Resources and Outcomes Project; COPD, chronic obstructive pulmonary disease; EDS,early discharge scheme; NIV, non-invasive ventilation; IQR, interquartile range.

Table 3 Provision of palliative care services for intervention and control hospital units participating in the 2010 NCROP re-survey

If organizational featurewas PRESENT in 2007

If organizational featurewas ABSENT in 2007

% with feature RETAINED in 2010 % with feature GAINED in 2010

Intervention 2010 Control 2010 Intervention 2010 Control 2010

Are there any formal arrangements for patients with COPD toreceive palliative care in your area? % Yes

86% (18/21) 77% (10/13) 35% (7/20) 54% (14/26)

There is a policy for providing patient information about end-of-lifecare to severe COPD patients while in a stable state, e.g. in anoutpatient setting or upon discharge from hospital. % Yes

40% (2/5) 50% (2/4) 26% (9/35) 20% (7/35)

Are there any plans to develop/further develop palliative careservice for patients with COPD? % Yes

77% (20/26) 81% (17/21) 67% (10/15) 72% (13/18)

NCROP, the National Chronic Obstructive Pulmonary Disease Resources and Outcomes Project; COPD, chronic obstructive pulmonary disease.



Table 4 Sites reporting significant service changes within change diaries within the 3 years (2010) and the 1 year (2008) following peer review visitsin control and intervention units

Intervention 2008(n = 50)

Control 2008(n = 43)

Intervention 2010(n = 41)

Control 2010(n = 37)

Changes for worse 5 (10%) 7 (16%) 8 (20%) 8 (22%)Neutral 1 (2%) 1 (2%)Changes for better 44 (88%) 35 (81%) 41 (100%) 35 (95%)Changes in conjunction with PCT 15 (30%) 13 (30%)Changes for the better being initiated in the:

Community 24 (48%) 24 (56%)Hospital 37 (74%) 21 (49%)

Any change for the better relating to any of the COPD qualitymeasures, including palliative care

42 (84%) 30 (70%)

Pulmonary rehabilitation – community or hospital based 18 (36%) 9 (21%) 18 (44%) 8 (22%)Early discharge schemes – including admission avoidance schemes 13 (26%) 10 (23%) 11 (27%) 5 (14%)Oxygen therapy (all community based) 14 (28%) 20 (47%) 15 (37%) 10 (27%)Non-invasive ventilation community or hospital based

(mainly hospital based)18 (36%) 6 (14%) 16 (39%) 5 (14%)

Palliative care 16 (32%) 1 (2%) 10 (24%) 6 (16%)

COPD, chronic obstructive pulmonary disease; PCT, primary care trust.

Table 5 Examples of how peer review had helped bring about positive change in service and within departments

Grouped theme Examples

A. Positive changes in COPD services

National guidelines Many comments re BTS Guidelines on Oxygen Therapy, GOLD Standard Framework for Palliative Care,NICE Guidelines.

Use of national data and guidelines ‘Provided information on national perspective and acted as a driver for change’Raised the profile of COPD ‘The peer review raised the profile of our department both within the trust and county to allow us

service expansion. This coupled with a very active team in respiratory medicine allowed this expansionand vast improvement of the service’

External validation as anegotiating tool

‘NCROP helped us to provide important strategic data to compare our standard of care with the rest ofthe peer group and help us to build a local support mechanism to expand the service’.

A direct consequence of a peerreview recommendation

‘The NCROP peer review helped highlighting the lack of this service to the PCT representative andencouraged them to commission this’.‘Helped by Peer Review in my opinion – the decision to fund this by the xxxxxx PCT was veryborderline, so the Peer review opinion in favour may well have tipped the balance in the right direction.’

Shared learning/experience/materials ‘We are now using exchange visits although not formal peer review with the xxxxxxxx respiratory team toshare our learning . . .’

B. Changes within departments

Impact on morale ‘It was very good for team morale’.‘The COPD team felt valued as a result of review’

Team working ‘The experience of peer review visits was very useful for developing a strong sense of team philosophyincluding pride in our service, importance of being open to change and continued learning ..’

Increased self-awareness ofown service

‘Certainly in terms of personal development. It provided an opportunity for me to take ownership of aproject and develop leadership skills’

Linkages ‘The increased awareness of what is required of the trust as a result of the reviewers meeting managersaided this process’.‘Increased liaison with PCT as a member of the PCT admin was included in our peer review group andwas directly involved with the exercise’

Shared experience ‘Very useful to have seen how another department delivers care and enabled focus on getting the careright according to national standards.’

NCROP, the National Chronic Obstructive Pulmonary Disease Resources and Outcomes Project; COPD, chronic obstructive pulmonary disease;BTS, British Thoracic Society; GOLD, Global Initiative for Chronic Obstructive Lung Disease; NICE, National Institute for Health and Clinical Excellence;PCT, primary care trust.



that distinction has become blurred with evidence of much col-laboration between hospital services and community providers.The community providers included outreach from hospital, com-munity trusts, community arms of primary care trusts (PCTs) andprivate health-care organizations contracted by PCTs.

Control group. Within the control group respiratory teams werenot exposed to a peer review visit, but did collect data before andafter the peer review phase to intervention sites to assess perfor-mance with regard to the four quality markers of COPD care aswell as palliative care. While many felt that the NCROP projecthad had no effect on them (14/37, 38%), a significant number dididentify benefit. The main benefits related to: raising the profile ofCOPD, an opportunity to reflect on current practice, the benefit ofbenchmarking national data.

Intervention group. Within the intervention group there weremany units noting benefit for the service and for the departmentfrom NCROP (32/41, 78%). The benefits derived in part from thedata gathered before and after visits and in part from the peerreview visit itself and were cited as: raising the profile of COPD,an opportunity to reflect on current practice, the benefit of nationaldata, external validation as a negotiating tool, direct consequencesof peer review recommendations, shared learning and experiences,linkages to other organizations specifically the PCTs and finallyimproving morale due to recognition of achievement.

Achievement against the 2007 service

development plans

In 2007 the intervention units were asked to prepare service devel-opment plans to achieve change agreed at the peer review visits. In2010 data were received from 43 of 54 intervention units (80%).Action plan themes related predominantly to development of mul-tidisciplinary teams across primary and secondary care and tospecialist service provision. Most aims were fully achieved in atleast a third of units and partially achieved in a similar percentage.Palliative care plans were less prevalent although partially or fullysuccessful in 83%. Thirty-five per cent of planned actions werefully implemented by 2010, with a further 41% of planned actionspartially achieved and 19% of planned actions having not beenachieved at all.

DiscussionThe NCROP represents the largest ever voluntary peer reviewprogramme run in the UK. It was completed within a short timeframe and with positive engagement of the vast majority of clini-cians. Managers from acute trusts attended most peer reviewsand primary care organizations (PCOs) were represented in mostbut not all visits perhaps reflecting the restructuring of PCOs thatoccurred during the project.

Service change within the NHS can take a long time to manifestitself [17]. The hypothesis for carrying out a re-evaluation of theNCROP project at 3 years was that change that had not beendemonstrated at 1 year may become apparent over a 3-year period.The quantitative results have supported the premise but tend torefute the hypothesis. The study has demonstrated many changesover the 3 years. However, specifically with regard to the quanti-

tative impact of reciprocal peer review, there is only slightevidence at 3 years of a significant divergence between the inter-vention group and the control group.

There have been many powerful drivers for change within theNHS over the 3-year period since the NCROP peer review visits.Examples mentioned by many sites include changes in commis-sioning, the QIPP agenda (Quality Innovation, Productivity andPerformance) as well as specific national guideline requirementsand recommendations. In addition, the results show great varia-tion in the amount of change between sites. The feedback fromsites indicates – as in 2008 – that in some areas good progress ismade with effective working between hospital and communityservices as well as clinicians, managers and commissioners. Inother places there are perceived to be barriers that preclude effec-tive service delivery. Barriers include difficulties in establishingeffective working relationships, funding changes and servicere-design.

Against this backdrop of varying drivers for change within theNHS and varying ability to deliver change, it is perhaps not sur-prising that a randomized control trial of reciprocal peer reviewhas failed to demonstrate an independent effect on serviceimprovement.

The qualitative data support the trends seen in the quantitativedata, indicating that the intervention teams were more likely toreport positive service changes since 2007 than the control group.The majority of intervention teams reported ‘soft’ benefits ofinvolvement in the peer review process.

Within both intervention and control groups where there was nochange for the better a perceived lack of resources – whether ofpersonnel or money – was frequently cited as a reason. Loss ofstaff contributed to reduced service quality where key skilledworkers could or would not be replaced. Such obstacles wereassociated with a general feeling of frustration, disengagement anddisempowerment.

However, the qualitative data at 3 years has shown – as did thereview at 1 year – that there are many examples within the inter-vention group of processes being initiated, which would beexpected to support and foster change. Important examples of thisinclude the linkages developed between clinicians, managers andcommissioners, the use of national data to benchmark and providea basis for developing local plans, the opportunity to reflect on one’sown service in the light of the experience of others and see whereimprovements need to be made and can most effectively be made.

The NCROP peer review study was carried out in the clinicalcontext of respiratory disease. However, the findings with regard toimprovements to services and barriers to change, e.g. impact onteam working, shared learning, objective data on which to baseproposals for change, interaction between clinicians, managersand commissioners, are generic to all clinical conditions.

What emerges from this study of peer review is that, in the faceof the highly volatile state of service change within the NHS,quantitative analysis using a randomized control design does notprovide the whole picture. The qualitative analysis indicates thatpeer review as carried out within NCROP is associated with manyin-hospital changes and is felt by participants to have many ben-efits, both in driving change and on the department. The challengeremains to define which services are in a state to benefit from peerreview to ensure that the intervention is focussed where it can bemost cost-effective.



AcknowledgementsWe are grateful to the NCROP Steering Group, the COPD teamsand their patients who participated in this study and to the HealthFoundation which funded the project.

References1. Heaton, C. (2000) External peer review in Europe: an overview from

the ExPeRT project. International Journal for Quality in Health Care,12, 177–182.

2. Page, R. L. & Harrison, B. D. W. (1995) Setting up interdepart-mental peer review. Journal of the Royal College of Physicians, 29,319–324.

3. Roberts, C. M., Buckingham, R. J., Pursey, N. A., Stone, R. A. &Lowe, D. (2009) The National Chronic Obstructive PulmonaryDisease Resources and Outcomes Project (NCROP) Final Report.Royal College of Physicians, British Thoracic Society and BritishLung Foundation. Available at: http://www.rcplondon.ac.uk/clinical-standards/ceeu/Current-work/ncrop/Documents/The-National-Chronic-Obstructive-Pulmonary-Disease-Resources-and-Outcomes-Project-NCROP-Final-Report-updated%2027%2010%2009.pdf (lastaccessed 1 September 2010).

4. Roberts, C. M., Buckingham, R. J., Stone, R. A., Lowe, D. & Pearson,M. G. (2010) The UK National Chronic Obstructive PulmonaryDisease Resources and Outcomes Project – a feasibility study of largescale clinical service peer review. Journal of Evaluation in ClinicalPractice, 16, 927–932. Available at: http://www.rcplondon.ac.uk/sites/default/files/the-national-chronic-obstructive-pulmonary-disease-resources-and-outcomes-project-ncrop-final-report.pdf (last accessed31 January 2011).

5. Rivas, C., Taylor, S. & Clarke, A. (2008) Final report on the Quali-tative Sub-study of the National COPD Resources and OutcomesProject (NCROP). Barts and The London School of Medicineand Dentistry. Available at: http://www.rcplondon.ac.uk/sites/default/files/final-report-on-the-qualitative-sub-study-of-the-national-copd-resources-and-outcomes-project.pdf (last accessed 31 January 2011).

6. Roberts, C. M., Stone, R. A., Buckingham, R. J., Pursey, N. A.,Harrison, B. D. W., Lowe, D. & Potter, J. M. (2010) A randomised trialof peer review: the UK National Chronic Obstructive Pulmonary

Disease Resources and Outcomes Project. Clinical Medicine, 10 (3),223–227.

7. Roberts, C. M., Barnes, S., Lowe, D. & Pearson, M. G. (2003) Evi-dence for a link between mortality in acute COPD and hospital typeand resources. Thorax, 58, 947–949.

8. Price, L. C., Lowe, D., Hosker, H. S. R., Anstey, K., Pearson, M. G.,Roberts, C. M., on behalf of the British Thoracic Society and the RoyalCollege of Physicians Clinical Effectiveness Evaluation Unit (CEEu)(2006) The UK national COPD audit 2003. Impact of hospitalresources and organisation of care on patient outcome followingadmission for acute COPD exacerbation. Thorax, 61, 837–842.

9. British Thoracic Society (1997) BTS guidelines for the managementof chronic obstructive pulmonary disease. Thorax, 52 (Suppl. 5),S1–S28.

10. British Thoracic Society Standards of Care Sub-committee on Pulmo-nary Rehabilitation (2001) BTS statement on pulmonary rehabilita-tion. Thorax, 11, 827–834.

11. British Thoracic Society Standards of Care Committee (2002)Non-invasive ventilation in acute respiratory failure. Thorax, 57, 192–211.

12. Lacasse, Y., Goldstein, R., Lasserson, T. J. & Martin, S. (2006)Pulmonary rehabilitation for chronic obstructive pulmonarydisease. Cochrane Database of Systematic Reviews. Issue 4: Art.No: CD003793. DOI: 10.1002/14651858.CD003793.pub2.

13. NICE (2004) Chronic Obstructive Pulmonary Disease. National Clini-cal guideline on management of COPD in adults in primary andsecondary care. Thorax, 59 (Suppl. 1), 1–232.

14. Picot, J., Lightowler, J. & Wedzicha, J. A. (2004) Non-invasive posi-tive pressure ventilation for treatment of respiratory failure due toexacerbations of chronic obstructive pulmonary disease. CochraneDatabase of Systematic Reviews. Issue 3. Art. No.: CD004104. DOI:10.1002/14651858.CD004104.pub3.

15. BTS Home oxygen service in England and Wales 2006. Available at:http://www.brit-thoracic.org.uk/clinical-information/home-oxygen.aspx (last accessed 31 January 2011).

16. British Thoracic Society Standards of Care Committee (2007) Inter-mediate care- hospital-at-home in Chronic Obstructive PulmonaryDisease. Thorax, 63f, 200–210.

17. Rudd, A. G., Irwin, P., Rutledge, Z., Lowe, D., Wade, D. T. & Pearson,M. G. (2001) Regional variations in stroke care in England, Wales andN Ireland. Clinical Rehabilitation, 15, 652–572.

