Clinical practice guidelines for acute diarrhea in children: Methodological quality

Iván D. Flórez1. Javier Contreras1 Javier Sierra1, María E. Tamayo1, Luz H. Lugo1, Claudia Granados2, Germán

Briceño3, Juan M. Lozano4, Jorge L. Acosta5

1- Universidad de Antioquia; Medellín, Colombia 2. Pontificia Universidad Javeriana; Bogotá, Colombia

3. Fundación Cardio-infantil; Bogotá, Colombia 4. University of Florida, Miami, USA

5. Universidad del Norte, Barranquilla, Colombia

Background •  Acute diarrhea (AD) is the second most common disease in

children

•  AD is globally associated with high morbidity in developed countries and with high mortality in undeveloped countries.

•  The Colombian Ministry of Health financed the development CPG of some diseases in order to improve health care system and to do cost-effectiveness analysis. Diarrhea in children was one of them

•  We formed a guideline developer group (GDG) in order to develop the Clinical Practice Guidelines (CPG) of prevention, diagnosis and treatment of diarrheal diseases in children

Background •  Before developing a de novo-CPG it is

recommended to assess published guidelines in order to consider adaptation of a high quality CPG

•  There were a lot of CPG of diarrhea and gastroenteritis in children available in the literature

•  The quality of published guidelines on childhood diarrhea was assessed by Lo Veccio et al. in 2011, but it had some flaws

Copyright 2011 by ESPGHAN and NASPGHAN. Unauthorized reproduction of this article is prohibited.

Evaluation of the Quality of Guidelines for AcuteGastroenteritis in Children With the AGREE Instrument

!Andrea Lo Vecchio, !Antonietta Giannattasio, yChristopher Duggan, zSalvatore De Masi,§Maria Teresa Ortisi, jjLuciana Parola, and !Alfredo Guarino

ABSTRACT

Aim: The aim of the study was to assess the quality of clinical practiceguidelines (CPGs) using the Appraisal of Guidelines for Research andEvaluation (AGREE) instrument, a validated international tool.Materials and Methods: CPGs were identified by searching MEDLINE(1966–January 2009) and Embase (1988–January 2009), CPG databases,and relevant Web sites of agencies and organizations that produce and/orendorse guidelines. Included in the study were CPGs in English thataddressed the management of acute gastroenteritis in children. RetrievedCPGs were evaluated with the AGREE instrument for quality assessmentby 6 independent reviewers. AGREE consists of 6 domains for a total of23 items.Results: Nine CPGs were identified. Four were evidence based (EB) and2 of these included tables of evidence. Eight CPGs (88%) scored <50% for‘‘applicability,’’ 7 (77%) for ‘‘stakeholder involvement,’’ and 6 (66%) for‘‘editorial independence.’’ Compared with non-EB CPGs, EB CPGs hadhigher quality scores for all AGREE domains, with a better score for ‘‘rigorof development’’ (P< 0.001), ‘‘stakeholder involvement’’ and ‘‘clarity ofpresentation’’ (P< 0.01), and applicability (P< 0.05). Over time, thequality of guidelines tended to improve. The main recommendations ofCPGs were similar. However, there were differences in the treatment ofdiarrhea, namely based on the settings and circumstances in which CPGswere produced.

Conclusions: The overall quality of CPGs on acute gastroenteritismanagement in children is fair. Aims, target population, synthesis ofevidence, formulation of recommendations, and clarity of presentationare points of strength. Weak issues are applicability, including identifica-tion of organizational barriers and adherence parameters, and cost/efficacyanalysis.

Key Words: acute gastroenteritis, AGREE instrument, child, guidelines

(JPGN 2011;52: 183–189)

C linical practice guidelines (CPGs) are systematically devel-oped statements to assist practitioners in making decisions

about appropriate health care in specific clinical circumstances (1).Their purpose is to make explicit recommendations with a definiteintent to influence what clinicians do. The primary goal of CPGs inpediatrics is to improve the health of infants and children byensuring that they receive up-to-date, evidence-based (EB) care.CPGs are a major tool to improve the quality of care. Several studieshave shown that adherence to EB guidelines leads to improvementin the quality of care provided (2,3). For many health conditions,there is a gap between what medical science has shown to beeffective practice and what is actually done (4).

The number of CPGs is rapidly mounting also in pediatrics.However, the plethora of CPGs has been accompanied by growingconcern about differences among guideline recommendations andabout the quality of guidelines (5–8). How does one define thequality of guidelines? A ‘‘good’’ guideline should be scientificallyvalid, usable, and reliable, and should improve the outcome ofpatients; however, it is rarely known how a guideline performs inclinical practice. Evaluation of CPGs should include both methodsused to develop recommendations and applicability of recommen-dations (benefits, adverse effects, and costs).

An international group of researchers, the Appraisal ofGuidelines for Research and Evaluation (AGREE) Collaboration,developed and validated a specific instrument to assess the qualityof CPGs based on theoretical assumptions (9). A recent assessmentof the quality of pediatric guidelines with the AGREE instrumentdemonstrated better results for pediatric than for adult CPGs (10).The best performers were CPGs published and endorsed by theAmerican Academy of Pediatrics (AAP) or registered in theNational Guidelines Clearinghouse (NGC).

Acute gastroenteritis (AGE) remains a common cause ofmorbidity and mortality among infants and children worldwide. Inindustrialized countries, the disease is relatively mild and generallyself-limiting, but nevertheless can have a major effect on the qualityof life of infected children and their families. AGE is a majorcause of outpatient visits and hospital admissions in developedcountries, and consequently it has a substantial effect on healthcosts. Several guidelines for the management of AGE in childrenare available. However, only a minority of physicians fully comply

Received February 12, 2010; accepted April 2, 2010.From the !Department of Pediatrics University of Naples ‘‘Federico II,’’

Naples, Italy, the yDivision of Gastroenterology and Nutrition, Chil-dren’s Hospital Boston, and Department of Pediatrics, Harvard MedicalSchool, Boston, MA, the zDepartment of Epidemiology and Guidelinesof the Italian Institute of Health (Istituto Superiore di Sanita), Rome,Italy, the §Hospital of Sant’Anna of Como, Italy and member of theAccreditation and Quality Improvement Working Group of ItalianSociety of Pediatrics, and the jjHospital of Magenta, Azienda Ospeda-liera ‘‘Ospedale Civile di Legnano’’ Italy and member of the Accred-itation and Quality Improvement Working Group of the Italian Society ofPediatrics.

Address correspondence and reprint requests to Prof Alfredo Guarino,Department of Pediatrics University of Naples ‘‘Federico II,’’ ViaPansini 5, 80131 Naples, Italy (e-mail: alfguari@unina.it).

Two of the authors have been involved in the production of 2 guidelines thatwere included in this study (A.G. and C.D.). Neither guideline rankedfirst in the AGREE evaluation, and in addition, the concordance betweenraters was good, suggesting that there were no biases in the evaluation. Inaddition, A.G. promoted and coordinated the present study but did nottake part in the guideline assessment as evaluator.

The authors report no conflicts of interest. There is no potential conflict ofinterest in any part of the study and there was no form of payment givento anyone to produce the manuscript.

Copyright # 2011 by European Society for Pediatric Gastroenterology,Hepatology, and Nutrition and North American Society for PediatricGastroenterology, Hepatology, and Nutrition

DOI: 10.1097/MPG.0b013e3181e233ac

ORIGINAL ARTICLE: GASTROENTEROLOGY

JPGN " Volume 52, Number 2, February 2011 183

Copyright 2011 by ESPGHAN and NASPGHAN. Unauthorized reproduction of this article is prohibited.

Evaluation of the Quality of Guidelines for AcuteGastroenteritis in Children With the AGREE Instrument

!Andrea Lo Vecchio, !Antonietta Giannattasio, yChristopher Duggan, zSalvatore De Masi,§Maria Teresa Ortisi, jjLuciana Parola, and !Alfredo Guarino

ABSTRACT

Aim: The aim of the study was to assess the quality of clinical practiceguidelines (CPGs) using the Appraisal of Guidelines for Research andEvaluation (AGREE) instrument, a validated international tool.Materials and Methods: CPGs were identified by searching MEDLINE(1966–January 2009) and Embase (1988–January 2009), CPG databases,and relevant Web sites of agencies and organizations that produce and/orendorse guidelines. Included in the study were CPGs in English thataddressed the management of acute gastroenteritis in children. RetrievedCPGs were evaluated with the AGREE instrument for quality assessmentby 6 independent reviewers. AGREE consists of 6 domains for a total of23 items.Results: Nine CPGs were identified. Four were evidence based (EB) and2 of these included tables of evidence. Eight CPGs (88%) scored <50% for‘‘applicability,’’ 7 (77%) for ‘‘stakeholder involvement,’’ and 6 (66%) for‘‘editorial independence.’’ Compared with non-EB CPGs, EB CPGs hadhigher quality scores for all AGREE domains, with a better score for ‘‘rigorof development’’ (P< 0.001), ‘‘stakeholder involvement’’ and ‘‘clarity ofpresentation’’ (P< 0.01), and applicability (P< 0.05). Over time, thequality of guidelines tended to improve. The main recommendations ofCPGs were similar. However, there were differences in the treatment ofdiarrhea, namely based on the settings and circumstances in which CPGswere produced.

Conclusions: The overall quality of CPGs on acute gastroenteritismanagement in children is fair. Aims, target population, synthesis ofevidence, formulation of recommendations, and clarity of presentationare points of strength. Weak issues are applicability, including identifica-tion of organizational barriers and adherence parameters, and cost/efficacyanalysis.

Key Words: acute gastroenteritis, AGREE instrument, child, guidelines

(JPGN 2011;52: 183–189)

C linical practice guidelines (CPGs) are systematically devel-oped statements to assist practitioners in making decisions

about appropriate health care in specific clinical circumstances (1).Their purpose is to make explicit recommendations with a definiteintent to influence what clinicians do. The primary goal of CPGs inpediatrics is to improve the health of infants and children byensuring that they receive up-to-date, evidence-based (EB) care.CPGs are a major tool to improve the quality of care. Several studieshave shown that adherence to EB guidelines leads to improvementin the quality of care provided (2,3). For many health conditions,there is a gap between what medical science has shown to beeffective practice and what is actually done (4).

The number of CPGs is rapidly mounting also in pediatrics.However, the plethora of CPGs has been accompanied by growingconcern about differences among guideline recommendations andabout the quality of guidelines (5–8). How does one define thequality of guidelines? A ‘‘good’’ guideline should be scientificallyvalid, usable, and reliable, and should improve the outcome ofpatients; however, it is rarely known how a guideline performs inclinical practice. Evaluation of CPGs should include both methodsused to develop recommendations and applicability of recommen-dations (benefits, adverse effects, and costs).

An international group of researchers, the Appraisal ofGuidelines for Research and Evaluation (AGREE) Collaboration,developed and validated a specific instrument to assess the qualityof CPGs based on theoretical assumptions (9). A recent assessmentof the quality of pediatric guidelines with the AGREE instrumentdemonstrated better results for pediatric than for adult CPGs (10).The best performers were CPGs published and endorsed by theAmerican Academy of Pediatrics (AAP) or registered in theNational Guidelines Clearinghouse (NGC).

Acute gastroenteritis (AGE) remains a common cause ofmorbidity and mortality among infants and children worldwide. Inindustrialized countries, the disease is relatively mild and generallyself-limiting, but nevertheless can have a major effect on the qualityof life of infected children and their families. AGE is a majorcause of outpatient visits and hospital admissions in developedcountries, and consequently it has a substantial effect on healthcosts. Several guidelines for the management of AGE in childrenare available. However, only a minority of physicians fully comply

Received February 12, 2010; accepted April 2, 2010.From the !Department of Pediatrics University of Naples ‘‘Federico II,’’

Naples, Italy, the yDivision of Gastroenterology and Nutrition, Chil-dren’s Hospital Boston, and Department of Pediatrics, Harvard MedicalSchool, Boston, MA, the zDepartment of Epidemiology and Guidelinesof the Italian Institute of Health (Istituto Superiore di Sanita), Rome,Italy, the §Hospital of Sant’Anna of Como, Italy and member of theAccreditation and Quality Improvement Working Group of ItalianSociety of Pediatrics, and the jjHospital of Magenta, Azienda Ospeda-liera ‘‘Ospedale Civile di Legnano’’ Italy and member of the Accred-itation and Quality Improvement Working Group of the Italian Society ofPediatrics.

Address correspondence and reprint requests to Prof Alfredo Guarino,Department of Pediatrics University of Naples ‘‘Federico II,’’ ViaPansini 5, 80131 Naples, Italy (e-mail: alfguari@unina.it).

Two of the authors have been involved in the production of 2 guidelines thatwere included in this study (A.G. and C.D.). Neither guideline rankedfirst in the AGREE evaluation, and in addition, the concordance betweenraters was good, suggesting that there were no biases in the evaluation. Inaddition, A.G. promoted and coordinated the present study but did nottake part in the guideline assessment as evaluator.

The authors report no conflicts of interest. There is no potential conflict ofinterest in any part of the study and there was no form of payment givento anyone to produce the manuscript.

Copyright # 2011 by European Society for Pediatric Gastroenterology,Hepatology, and Nutrition and North American Society for PediatricGastroenterology, Hepatology, and Nutrition

DOI: 10.1097/MPG.0b013e3181e233ac

ORIGINAL ARTICLE: GASTROENTEROLOGY

JPGN " Volume 52, Number 2, February 2011 183

•  First version of the AGREE instrument •  Search was limited to English language CPG •  Databases and five web-sites of CPG or associations •  Date search limit: 2008 •  In 2009 were published: 1 guideline in Colombia

and 2 guidelines from NICE and ESPGHAN •  In 2009, were published some Cochrane Systematic

reviews about treatments in diarrhoea •  Authors included non-evidence based CPG •  This paper was not enough for us to do an assessment

of the quality of theses CPG

Problems:

Objective •  To assess the quality of CPG on acute

diarrhea or gastroenteritis in children using the AGREE-II instrument (Spanish version).

Methods •  Systematic review of CPG •  Databases:

o  Electronic databases: EMBASE, MEDLINE, LILACS o  National clearinghouses o  Non-electronic sources: Hand-searching:

•  Known CPG by authors, researchers, members of the group and clinical experts

•  Pediatrics, gastroenterology, familiar medicine and nursing congresses and textbooks

•  We searched from June to August of 2011

Methods •  We exclude guidelines that:

o  Were not Evidence-based

o  Had a different scope

o  Exclusive adult population

o  Diarrhea in children with chronic diseases

•  Each EB-guideline was independently assessed using AGREE-II (Spanish version) by three (3) clinical epidemiologists

•  Appraisers (MSc in Clinical epidemiology) o  Genera practicioner (1), Pediatrician (6)

Methods •  The assesment was made based on the information

presented in the published version of the CPG and on the web site of the organization, institution or journals

•  We acceded when possible to evidence tables and search strategies by contacting authors and institution by mail. 

Methods •  Spanish-AGREE II instrument has 23 items grouped in

6 domains and 2 final items for the global evaluation.

•  The score is calculated with proportions for each domain

•  There is not a minimum score to recommend a CPG

•  Rigor of development was chosen as the most important domain based on Colombian guidelines methodology manual

•  60% was the cut off point to consider a high quality guideline

Methods •  The score of each domain was calculated based in

the 3 appraisers evaluation

•  Differences between the scores in each items were analyzed by one of the appraisers.

•  When differences between appraisers were more than 4 points, they were asked to revaluate their score, in order to get a consensus

•  We calculated medians and interquartile ranges (IQR) of the scores in each domain

INSTRUMENTO AGREE II

INSTRUMENTO PARA LA EVALUACIÓN DE GUÍAS DE PRÁCTICA CLÍNICA

Consorcio AGREEMayo de 2009

APPRAISAL OF GUIDELINES fOR RESEARCH & EVALUATION II

INSTRUMENT

___________________________________________________________________________________

The AGREE Next Steps Consortium

May 2009

Results •  We found 63 diarrhea guidelines

•  17 of which (26.9%) were evidence-based.

•  The rest (46 CPG) were: protocols, narrative reviews, Non evidence –based guidelines, experts consensus, guidelines of associations and organizations (WHO), cost-effectiveness studies

•  10 in English language: Canada, India, UK, USA, Europe, Australia

•  7 in Spanish language: Colombia, México, Guatemala, Costa Rica, Spain

•  Peer-reviewed journals-databases (6), textbooks (2), guideline developer web-site (1), academic institutions or scientific association (4) or government organizations (4)

Results •  Domains 1 (scope and purpose) and 4 (clarity of

presentation) had the highest scores: 74 (60-94%) and 78 (56,5-91%), respectively

•  Domains 5 (applicability) and 6 (editorial independence), had the lowest: 13 (2-24,5%) and 8 (0-58%), respectively

•  Only five guidelines scored over 60% in the third domain (rigour of development) and were considered as “recommended”.

Results Recommended (5) Not recommended (12)

•  Domain 5 : 31 (6-69%)

•  Domain 6: 54 (0-97%)

•  Domain 1: 93,5 (89-100%)

•  Domain 4: 93,5 (89-95%)

•  Domain 3: 70 (60-42%)

•  Domain 2: 65 (42-95%)

•  Domain 5 : 10 (0-15%)

•  Domain 6: 8 (8-13%)

•  Domain 1: 63 (17-79%)

•  Domain 4: 69 (57-80%)

•  Domain 3: 27 (20-40%)

•  Domain 2: 26 (18-32%)

Discussion •  There is an urgent need to improve the quality of

CPGs in acute diarrhea in children by incorporating high quality standards in the development process based on the key points of a high-quality CPG

•  Most of CPG were from developed countries

•  There are several diarrhea CPG available in Spanish language but its quality is quite low in comparison to English languages ones

•  Two CPG were retrieved form Spanish textbooks

Discussion •  There are a lot of CPG that are called EB-guidelines

and they are not.

•  We retrieved CPG from databases, textbooks, developers websites, national clearinghouses. The search of CPG must be much wider than that for primary studies

•  Diarrhea is a common disease in children. It is desirable to have the best evidence-based recommendation available.

Discussion •  First (1th) and 4th domains scored high (upper 60%) in

both recommended and not recommended CPG

•  Fifth and 6th domains scored low in both recommended and not recommended CPG

•  Third domain must be the one that makes the difference

•  Developers are concerned about describing Scope and objectives and about the clarity of presentation, instead of being rigorous in methodology

•  We must be alert about the 3rd domain better than others that seem “attractive”: Clarity of recommendations and scope

Limitations •  We have conclusions that could not be reproducible

in other diseases •  We had a low number of CPG •  The 60% as a cut-point was recommended by the

Colombian guidelines, and the group agreed with this score.

•  What about another cut-points? 70 – 75 – 80 % ? •  AGREE collaboration doesn’t recommend a score,

so it could be controversial our cut-off point

Conclusions •  There is low quality in Spanish language CPG on

diarrhea •  Most of CPG on diarrhea were not evidence based •  Most of evidence based CPG had low to moderate

quality •  In AGREE-II, Third domain makes the difference •  Scope and purpose, and clarity could be good

even in low quality guidelines

ivoflorez@gmail.com

Danke!