critical assessment of pediatric neurosurgery patient

CLINICAL ARTICLEJ Neurosurg Pediatr 21:535–541, 2018

The concept and development of a global network be-gan in the 1960s at the Defense Advanced Research Projects Agency (DARPA) with a project known

as the Advanced Research Projects Agency Network (ARPANET).18 Twenty years later, the National Science Foundation Network (NSFNet), as well as other networks, arose with less access restriction.18 Beginning in 1995, these networks began to transition into a global network (the World Wide Web [WWW], also known as the “Web” or the “Internet,” henceforth referred to as the Internet in this study) with an infrastructure and backbone supported

by a consortium of commercial providers.18 Around this time, nearly 2 decades ago, the senior author (T.M.) and 2 colleagues wrote an article assessing the nascent Internet as a pediatric neurosurgery information resource.22 The authors concluded that the information available was vari-able but generally relevant. Since then, the popularity and scope of the Internet has grown exponentially in an ever-changing environment18—from the use of floppy disks to hard drives to laptop computers and mobile devices.

Today, Internet searches are rising as the use of desk-top and mobile search platforms expands. Thus, it is not

ABBREVIATIONS CRAAP = Currency, Relevance, Authority, Accuracy, and Purpose; IP = Internet Protocol.SUBMITTED April 23, 2017. ACCEPTED October 18, 2017.INCLUDE WHEN CITING Published online February 16, 2018; DOI: 10.3171/2017.10.PEDS17177.

Critical assessment of pediatric neurosurgery patient/parent educational information obtained via the InternetMichael Garcia, MA,1 Christopher Daugherty, PA-C,2 Bertha Ben Khallouq, MA,1 and Todd Maugans, MD1,2

1University of Central Florida College of Medicine; and 2Division of Pediatric Neurosurgery, Nemours Children’s Hospital, Orlando, Florida

OBJECTIVE The Internet is used frequently by patients and family members to acquire information about pediatric neurosurgical conditions. The sources, nature, accuracy, and usefulness of this information have not been examined recently. The authors analyzed the results from searches of 10 common pediatric neurosurgical terms using a novel scoring test to assess the value of the educational information obtained.METHODS Google and Bing searches were performed for 10 common pediatric neurosurgical topics (concussion, cra-niosynostosis, hydrocephalus, pediatric brain tumor, pediatric Chiari malformation, pediatric epilepsy surgery, pediatric neurosurgery, plagiocephaly, spina bifida, and tethered spinal cord). The first 10 “hits” obtained with each search engine were analyzed using the Currency, Relevance, Authority, Accuracy, and Purpose (CRAAP) test, which assigns a numeri-cal score in each of 5 domains. Agreement between results was assessed for 1) concurrent searches with Google and Bing; 2) Google searches over time (6 months apart); 3) Google searches using mobile and PC platforms concurrently; and 4) searches using privacy settings. Readability was assessed with an online analytical tool.RESULTS Google and Bing searches yielded information with similar CRAAP scores (mean 72% and 75%, respec-tively), but with frequently differing results (58% concordance/matching results). There was a high level of agreement (72% concordance) over time for Google searches and also between searches using general and privacy settings (92% concordance). Government sources scored the best in both CRAAP score and readability. Hospitals and universities were the most prevalent sources, but these sources had the lowest CRAAP scores, due in part to an abundance of self-marketing. The CRAAP scores for mobile and desktop platforms did not differ significantly (p = 0.49).CONCLUSIONS Google and Bing searches yielded useful educational information, using either mobile or PC platforms. Most information was relevant and accurate; however, the depth and breadth of information was variable. Search results over a 6-month period were moderately stable. Pediatric neurosurgery practices and neurosurgical professional orga-nization websites were inferior (less current, less accurate, less authoritative, and less purposeful) to governmental and encyclopedia-type resources such as Wikipedia. This presents an opportunity for pediatric neurosurgeons to participate in the creation of better online patient/parent educational material.https://thejns.org/doi/abs/10.3171/2017.10.PEDS17177KEY WORDS Internet; pediatric; educational information

J Neurosurg Pediatr Volume 21 • May 2018 535©AANS 2018, except where prohibited by US copyright law

Unauthenticated | Downloaded 05/12/22 12:20 AM UTC

M. Garcia et al.

J Neurosurg Pediatr Volume 21 • May 2018536

surprising that many people turn to the Internet to acquire educational information pertaining to their or their fam-ily members’ ailments, including pediatric neurosurgi-cal conditions. In fact, it is estimated that 59% of all US adults searched online for medical information in recent years.12,31 Although there is a growing interest within the medical community in evaluating the utility of online medical information,1–3,6,8,15,23 research on the accuracy and usefulness of online medical information is in the very early stages of scientific inquiry and requires con-stant revision due to the dynamic nature of the Internet. Research in this area is further complicated by structural (e.g., geographic region–specific searches) and method-ological challenges (e.g., purpose for medical information) that are difficult to address. Nonetheless, work of this type is important for patient care and education.

Currently, Alphabet’s Google is the most popular search engine, providing the results for roughly two-thirds (64%) of the nearly 17 billion monthly desktop Internet searches performed in the US. Microsoft’s Bing is the sec-ond most popular search engine in the US and provides approximately 31.9% of the results for desktop Internet searches performed in this country. According to experts, these 2 search engines supply the results for the majority of online searches performed on desktop platforms in the US.19 Thus, it is important that these search engines be considered in the design of scientific inquiry of this type.

Providing high-quality medical information is chal-lenging, but it is an important part of informed consent, and in some countries it is considered a legal responsi-bility.10 Many guidelines9,32,34 have been created with the intention of helping standardize online medical informa-tion; however, the variance in purpose and the state of medical information, along with the vast expanse of online material and diverse authorship, creates a unique conun-drum.7,25,30 The general goal of the guidelines is to provide benchmarks for content quality, currency, structure, and transparency (i.e., attribution to sources and identification of authorship and website ownership).

There are several tools that have been developed to evaluate written online medical information, such as the DISCERN instrument,1–4,6,8,23,24,28 the EQIP (Ensuring Quality Information for Patients) tool,5,23,27 and Minerva-tion’s LIDA instrument,2,15,16,26 as well as other novel tools29 and expanded versions of the aforementioned tools.5 The DISCERN instrument is limited in that it was developed by researchers for health professionals and patients to use in evaluating written literature focusing on treatment op-tions.4 The EQIP tool was designed to “provide a more rigorous method of assessing the presentation quality of patient information that is applicable to all information types, and prescribes the action that is required follow-ing evaluation.”27 The LIDA instrument was designed to assess medical information in 3 major areas: accessibil-ity, usability, and reliability.26 Since the existing tools are mostly intended to be used by health professionals and experts in the field, they arguably provide little to no use for patients. While they all have a significant level of in-terrater reliability, they also all have broad or sliding scor-ing rubrics. To circumvent the inherent methodological limitations of this work, in the present study, we explore

the use of the Currency, Relevance, Authority, Accuracy, and Purpose [CRAAP] test (https://www.csuchico.edu/lins/handouts/eval_websites.pdf), modified to provide a quantitative score of online medical information from a relatively dichotomous scoring rubric (compared with the other tools available).

The CRAAP test has been widely used as a “teaching tool” for college students to learn how to assess Internet resources for purposes of selecting credible sources for college papers and project development. Presumably, re-search of this type is also necessary to help our patients learn how to find reliable information about their medi-cal ailments; thus, CRAAP might be a promising tool in medicine. The objective of this study is to evaluate pe-diatric neurosurgical patient/parent educational material retrieved from the Internet.

MethodsThis was an observational study with a cross-section-

al study design (2 time points 6 months apart). Internet searches in pediatric neurosurgery were conducted using Google (Alphabet) and Bing (Microsoft), the most widely used search engines in the US.

Internet Search ProcedureTime 1

In mid-June of 2016, over the course of 2 weeks, desk-top Google searches were performed for 10 common pedi-atric neurosurgical keywords (concussion, craniosynosto-sis, hydrocephalus, pediatric brain tumor, pediatric Chiari malformation, pediatric epilepsy surgery, pediatric neu-rosurgery, plagiocephaly, spina bifida, and tethered spinal cord; also see Table 1) via central Florida IP (Internet Pro-tocol) addresses. The first 10 organic (nonadvertisement) search results (websites) for each keyword were recorded and used in this study (a total of 100 search results). As a second step, website search content was rated indepen-dently (using the CRAAP test) by 2 investigators (T.M., a board-certified pediatric neurosurgeon; and C.D., a pedi-atric neurosurgical physician assistant). Scoring discrep-ancies (6%) were resolved to reach a consensus score and create a final data set to be used for interrater reliability analysis with data from time 2.

Time 2In December 2016 and January 2017, the time 1 search

process was repeated by another investigator (M.G., a sec-ond-year medical student at the time of this research), this time using Google mobile and Bing desktop searches (Jan-uary 2017 and December 2016, respectively). Additionally, Google searches were performed on a desktop platform to compare matching search results. Finally, Google and Bing searches were also performed (December 2016) us-ing, respectively, the Chrome and Edge browser privacy modes, “Incognito” and “InPrivate.” These privacy modes (named differently by the 2 browsers) prevent the search engine from using saved search history, cookies, site data, and information to influence user experience. The results of searches using privacy modes were compared to the results of concurrent searches performed with the same


J Neurosurg Pediatr Volume 21 • May 2018 537

M. Garcia et al.

search engine without privacy mode. Time 1 and time 2 interrater reliabilities were tested using Cohen’s kappa (к). The value of Cohen’s kappa for interrater reliability for time 1 (T.M.:C.D.) was 0.90, and for agreement between time 1 (T.M./C.D. consensus data) and time 2 (M.G.) data it was 0.92.

Instrument: the CRAAP TestThe general purpose of the CRAAP test is to assess

the reliability and accuracy of Internet information. The CRAAP test asks 25 questions relevant to 5 domains: Currency (“the timeliness of the information”), Relevance (“the importance of the information for your needs”), Au-thority (“the source of the information”), Accuracy (“the reliability, truthfulness and correctness of the content”), and Purpose (“the reason the information exists”) (https://www.csuchico.edu/lins/handouts/eval_websites.pdf). The CRAAP test has been widely used in academic instruc-tion since its development in 2008. Although considered a qualitative assessment tool by the original authors, in this study, it was modified to include 23 quantitative items for scoring purposes (see Table 2 and Scoring section of this article).

In a research setting, the CRAAP test allows for a sys-tematic assessment of online content. The CRAAP test stands out from other similar tools (e.g., DISCERN), be-cause it was developed in consideration of naïve seekers of information. Moreover, the readability of the CRAAP test is approximately sixth-grade level according to the Flesch-Kinkaid scale. Thus, it is suitable for use with a broad patient population.

ScoringA score (0–3) and response anchors were assigned to

each item of the CRAAP test (e.g., “information updated,” no [0] or yes [1]). By assigning a numerical value, we were able to compute a total CRAAP test score. Search result scores could range from 0 to 34, with higher scores indicat-ing a better Internet information source (Table 2). Admit-

tedly, the use and modification of research tools have been comprehensively reviewed by many, and we acknowledge that there is no consensus on these methodological and statistical issues, but there is an existing body of literature with best practices.14,17,20,21,33 We have considered some of these best practices in our work, for example, the use of a 4-point scale over a 7-point Likert-type scale, which may induce participant reaction to the measurement scale. We used a combination of our statistical knowledge and clini-cal expertise when making changes to the CRAAP test; however, these methodological issues are not discussed in detail as we consider these issues beyond the scope of the present study. Cronbach’s alpha (a) was used to assess the reliability of the CRAAP test (a = 0.83). Subscale reliabil-ity results were as follows: Currency, a = 0.47; Relevance, a = 0.48; Authority, a = 0.68; Accuracy, a = 0.72; and Purpose, a = 0.70.

Categorization of Internet Search ResultsSearch results (websites) were sorted into 6 general

source categories (neurosurgery programs, divisions, and departments; journals and articles; government; reliable third-party; other third-party; irrelevant third-party). For thoroughness, general source categories were further di-vided into subcategories. For example, all websites con-taining the domain name “edu” were first sorted into the “neurosurgery programs/division/departments” and then further divided into 2 subcategories (“hospitals” and “uni-versities”) (see Table 3).

Website ReadabilityThe Internet Archive: Wayback Machine (archive.org),

which hosts archived copies of web pages, was used to identify website content changes before the readability analyses were conducted. All readability analyses were conducted in January 2017; to limit the influence of indi-vidual website code on the readability score, content from each source was copied and pasted into the Readable.io online analyzer. Lower grade-level (readability) scores

TABLE 1. Descriptive data by search term

Search Term

Google (desktop) Bing (desktop) Mobile (Google)Mean CRAAP

Score (SD)% CRAAP

Score RangeMean CRAAP

Score (SD)% CRAAP

Score RangeMean CRAAP

Score (SD)% CRAAP

Score Range

Pediatric epilepsy surgery 21.6 (5.4) 63.5 16–32 20.8 (6.3) 61.2 9–32 23.1 (4.8) 67.9 19–32Pediatric neurosurgery 21.8 (4.6) 64.1 16–29 22.3 (3.9) 65.6 14–29 21.5 (5.2) 63.2 13–29Craniosynostosis 23.7 (4.6) 69.7 16–33 26.9 (5.5) 79.1 17–34 25.0 (3.7) 73.5 19–33Tethered spinal cord 24.1 (4.8) 70.9 18–31 23.9 (4.8) 70.3 16–31 24.0 (4.5) 70.6 18–31Pediatric brain tumor 24.2 (5.3) 71.2 18–34 25.8 (5.0) 75.9 20–34 26.1 (5.5) 76.8 18–34Pediatric Chiari malformation 24.5 (3.3) 72.1 20–29 25.1 (2.0) 73.8 23–28 23.9 (3.5) 70.3 20–29Plagiocephaly 24.6 (4.8) 72.4 18–31 25.6 (4.7) 75.3 18–31 24.6 (4.8) 72.4 18–31Hydrocephalus 25.2 (5.0) 74.1 18–31 28.5 (2.8) 83.8 24–33 28.8 (2.8) 84.7 26–33Concussion 26.2 (7.1) 77.1 14–34 27.1 (6.9) 79.7 14–34 27.3 (5.8) 80.3 16–34Spina bifida 28.1 (5.0) 82.6 19–34 28.9 (3.5) 85.0 22–32 27.6 (4.6) 81.2 19–34Overall* 24.4 (5.0) 71.8 14–34 25.5 (4.5) 75.0 9–34 25.2 (4.5) 74.1 13–34

Data are ranked from lowest to highest Google (desktop) CRAAP score. The highest possible score is 34 (100%); range 0–34.* Overall averages for mean CRAAP scores and % CRAAP scores with total ranges.


M. Garcia et al.


indicate content that is easier to understand for a wider (or less-educated) audience and is thus considered more readable.

Statistical AnalysisComparisons of matching search results in 4 areas

(time, search engine, device, and privacy setting) are re-ported as percentage of matching search results and range (lowest and highest number of matching results of the top 10 search results) across all search terms.

CRAAP test scores are reported as means with standard deviations, mean percentage scores, and ranges. Two inde-pendent t-tests were conducted to assess the mean percent-age CRAAP test score differences between Google and Bing desktop and Google desktop and Google mobile. Sta-tistical tests were 2-tailed, and statistical significance was set at p < 0.05. Statistical analyses were conducted in IBM SPSS 24.0 and Microsoft Excel (version 16.0695.2117).

Website readability by source category is reported as the average readability grade-level score of the 5 readability tests (Flesch-Kincaid Grade Level, Gunning Fog Index, Coleman-Liau Index, SMOG Index, and Automated Read-ability Index), analyzed with an online analyzer (https://readable.io) that calculates grade-level readability.

ResultsComparison of Matching Search Results

Google desktop searches conducted 6 months apart yielded 72% matching results with a range of 2 to 9 of the top 10 organic search results for each keyword being the same at both time points. Concurrent Google and Bing desktop searches returned 58% matching search results (range 3–8). Concurrent Google desktop and Google mobile search re-sults yielded 91% matching results (range 8–10). Finally, privacy settings made little difference in search results, with 92% matching results (range 7–10) (Table 4).

TABLE 2. CRAAP test and scoring key

ElementPoints

0 1 2 3

Currency (3 items, 4 points) Date created* >5 yrs 1–5 yrs <1 yr Information updated No Yes Embedded links functional No YesRelevance (4 items, 5 points) Information answers questions appropriately No Yes Intended audience Not relevant Health care provider Patient/family Information appropriate level for needs No Yes Comfortable citing source No YesAuthority (5 items, 9 points) Identity of author/source None identified Patient Group connected to topic Authoritative resource Authors’ credentials None identified Patient Group connected to topic Authoritative resource Author qualified to write on topic No Yes Contact information of author provided No Yes URL identifies something relevant of source No YesAccuracy (6 items, 8 points) Derivation of information Unclear Personal experience Professional experience Evidence-based review Information supported by evidence No Yes Information has been reviewed or referred No Yes Information verifiable via another source No Yes Information unbiased, unemotional No Yes Grammar errors & typos Yes NoPurpose (5 items, 8 points) Purpose of information Sell something Persuade or entertain Inform Teach Intentions or purpose is clear No Yes Nature of information Propaganda Opinion Facts Point of view appears objective & impartial No Yes Political, ideological, cultural, religious biases Yes No

Based on items from Meriam Library–California State University, Chico: “Evaluating Information—Applying the CRAAP Test” (https://www.csuchico.edu/lins/handouts/eval_websites.pdf).A total of 23 items. Maximum CRAAP test score is 34 (range 0–34). Higher scores are indicative of a better Internet information source.* Scoring is based on length of time since site was created.



M. Garcia et al.

CRAAP Test ScoresMost Google and Bing searches yielded information

with CRAAP scores above 23.8 or 70%. Google and Bing searches for “spina bifida,” “concussion,” and “hydroceph-alus” yielded the highest CRAAP scores, while searches for “pediatric epilepsy surgery” and “pediatric neurosur-gery” yielded the lowest (Table 1).

There were no significant differences in CRAAP scores for searches using Google (overall mean percentage score 71.8%) and Bing (75.0%) (t [18] = 1.072, p = 0.299) despite the fact that the searches were conducted 6 months apart. Similarly, there was no significant difference between scores for searches using Google mobile (overall mean 74.1%) and Google desktop (71.8%) (t [18] = 0.846, p = 0.409) (Table 1).

CRAAP Test Scores and Website Readability by Source Category

Government websites generated the highest CRAAP scores (higher scores being better) and best readability (lowest readability grade level). Commercial medical in-formation websites (e.g., WebMD) also typically generat-ed high CRAAP scores with better than average readabil-ity. Hospital and university neurosurgery departments and divisions were the most common sources, but they scored below average on the CRAAP test and also received high-er grade-level readability scores, indicating material that was more difficult to understand. Physician groups and nonprofits scored just below average on the CRAAP test, but had better than average readability. Conversely, jour-nals and articles scored above average on the CRAAP test,

with content that was more difficult to understand, gener-ating the highest grade-level readability scores (Table 3).

DiscussionWe employed a novel evaluation tool, the CRAAP test,

to assess patient/parent educational information. While other scoring systems and instruments (e.g., DISCERN) have been validated and used in other studies,1–6,8,15,16,23, 24,

26–28 our results showed the CRAAP test to be a reliable tool in this study. Furthermore, using this instrument al-lowed us to obtain scores for the quality and accuracy of scientific content with less emphasis on treatment options and more focus on general patient education. Given the

TABLE 3. Descriptive data by source category

Category/GroupNo. of Results Mean CRAAP Score Average Readability Grade Level

Google Bing Google Bing Combined Google Bing Combined

Neurosurgery programs/divisions/departments 43 38 21.6 23.0 22.0 12.8 12.6 12.7 Hospitals 27 27 22.3 23.6 22.8 12.0 12.5 12.3 Universities 16 11 21.3 21.5 20.9 13.7 12.9 13.4Journals & articles 6 3 27.2 25.7 26.7 16.7 17.1 16.8Government 10 17 30.6 31.0 30.9 10.1 11.5 11.0Reliable 3rd party 22 26 28.2 28.5 28.5 12.2 11.7 11.9 Commercial medical information 9 14 28.8 28.7 28.8 11.4 11.1 11.2 Encyclopedias/wikis 3 6 30.0 30.7 30.4 12.8 13.1 13.0 Professional/health organizations 10 6 27.2 26.2 26.8 12.7 11.5 12.3Other 3rd party 11 11 23.2 23.7 23.5 11.0 11.8 11.4 Physician groups/clinics 3 5 22.0 21.4 21.6 11.0 13.0 12.3 Nonprofits/patient groups 8 6 23.6 25.7 24.6 11.0 9.8 10.5Irrelevant 3rd party 8 5 19.0 14.2 17.2 NA NA NA Person/no-evidence websites 1 0 24.0 NA 24.0 NA NA NA News & readership organizations 2 0 23.0 NA 23.0 NA NA NA Sales/product websites 2 2 18.0 13.5 15.8 NA NA NA Other/irrevelant/miscellaneous 3 3 15.3 14.7 15.0 NA NA NATotal/all 100 100 24.4 25.5 24.9 12.3 12.2 12.3

NA = not applicable.Number of results, mean CRAAP test score, and average readability by category and subcategory. Cells marked “not applicable” (NA) indicate either: a lack of sources to evaluate via CRAAP test (see column labeled “Mean CRAAP Score”) or sources not relevant to the study and thus not evaluated for readability (see column labeled “Average Readability Grade Level,” category “Irrelevant 3rd party”).

TABLE 4. Matching search results by time, search engine, device, and privacy setting

Searches Mean Range

Google-Google (6 mos apart) 72% 2–9Google-Bing (concurrent) 58% 3–8Google desktop–Google mobile (concurrent) 91% 8–10Google/Bing–Incognito/InPrivate (concurrent)* 92% 7–10

Data are reported as the mean percentage of matching search results over all 10 search terms and lowest and highest (range) number of matching results of the 10 top search results for all search terms in each category.* Matching search results for each search engine with and without privacy settings (Google using the Chrome privacy mode, Incognito, vs Google without the privacy mode and Bing with the Edge privacy mode, InPrivate, vs Bing without the privacy mode).


M. Garcia et al.


novelty of the research, we decided it would be careless to assess several scales at one time without knowing how each delivers results independently. Thus, we recognize the importance of replicating our findings using other scales.

Our analyses of pediatric neurosurgery patient/par-ent information via Google and Bing desktop and mobile searches demonstrates that available material is generally current, relevant, and accurate. The concordance between the 2 search engines was fair, with 58% of results being identical; however, CRAAP scores were not significant-ly different, implying that either search engine would be adequate for use by patients and parents. Privacy settings did not appear to affect search results, nor did the use of a desktop versus a mobile platform. The mobile platform is becoming increasingly relevant, since more searches are now performed on smartphones and tablets than with desktop or laptop computers, and this trend is expected to continue.11 Our results also appeared to be consistent over time, indicating that the addition of new information and updating of websites was not common.

The data presented here suggest that patients and par-ents should be directed to governmental sites for educa-tional materials. The reading level of material on these sites, however, may be challenging for some patients and families, which may be due in part to the complex nature of neurosurgical terminology and spelling. Wikipedia also scored very high and offered in-depth content; however, patients should be warned that Wikipedia is open to edit-ing, thus material does have the potential to be inaccurate. Reliable third-party websites (e.g., WebMD) scored close to governmental websites in both CRAAP test score and readability; however, authorship is frequently unclear.

We observed that—although they were the most preva-lent sources—information presented on the websites of neurosurgical academic and private practices and profes-sional organizations had lower CRAAP test scores. Most of these sites were dominated by innuendo and self-promo-tion. There are clearly opportunities for neurosurgical prac-tices and professional organizations to significantly expand patient educational resources and provide material that is comprehensive in scope and written at an appropriate read-ing level (no higher than 8th grade). Updating content is also paramount, as the evaluation and management of pe-diatric neurosurgical conditions is evolving continuously.

We acknowledge limitations to our study. We chose search terms that were very general. Although the senior author has found through conversations with parents that the search terms evaluated are the starting point for many searches, alternative strategies and search terms may be used. We also decided not to recognize HONcode (http://www.hon.ch/HONcode/) approval of websites in our anal-ysis. The HONcode badge denotes websites that have ap-plied to the Health on the Net Foundation and paid yearly membership fees to have their content assessed for reli-ability and credibility. Future studies may include analy-ses to determine the potential significance of third-party validation of content quality. It is also known that search results can vary by location and IP addresses of the de-vices upon which searches are conducted. Our searches were conducted in a single geographical location (central Florida); however, future studies should consider including

a comparison of search results across different regions of the country. Furthermore, with the diverse patient popula-tion of the US, information translation and online content in other languages are additional possible considerations for future studies. Also, tracking of search habits and tai-lored individual search results are of particular importance and will likely increase in the coming years. We attempt-ed to address this concern by comparing searches using browser privacy settings (Incognito and InPrivate), which limit storage of browsing history, cookies, site data, and/or “autofill” information entered into forms, although oth-er tracking methods (e.g., IP address) may still influence which search results are delivered. Good overall concor-dance in search results between all 3 observers, however, implies that individual search habits and histories may not heavily impact search results. Finally, website page size, the time it takes for a website to load, the user experience (usability.gov), content structure across devices, and the dynamic nature of the Internet are also important factors to consider in the future. Evaluation of archived versions of websites during readability analysis revealed only 5 sources, all from the same domain, that had updated their website design and content. Changes in content and web-site design may be influenced by popularity, novelty of the content provided, and general public interest. Analysis of advertising and possible censorship or weighted search rankings is currently beyond the scope of our research.

While there are many factors used by search engines to rank websites,13 incoming links to a web page constitute one of the most important factors that strengthen its search result position. For example, if many web pages link and reference another specific web page, the web page being referenced and linked to is more likely to be found in the top 10 search results or “hits.” Of course, many other factors influence search result position, and the specific algorithms used by Google and Bing to rank websites for the myriad key words and search combinations are not publicly available.

ConclusionsThis study reveals that Internet searches using Google

and Bing can yield potentially useful patient/parent educa-tional information. Based on the inferiority of organized, academic, and private practice pediatric neurosurgery website offerings, the pediatric neurosurgical community has significant opportunities to improve patient/parent ed-ucational information available via Internet searches.

AcknowledgmentsWe would like to thank Shalu Gillum, JD, MLS, AHIP, at the

Harriet F. Ginsburg Health Sciences Library (University of Central Florida College of Medicine) for her assistance with our literature review.

The CRAAP test was developed by librarians at the Meriam Library, California State University, Chico. We used this test with permission of the Meriam Library reference librarian.

References 1. Alamoudi U, Hong P: Readability and quality assessment of

websites related to microtia and aural atresia. Int J Pediatr Otorhinolaryngol 79:151–156, 2015



M. Garcia et al.

2. Azer SA, Alghofaili MM, Alsultan RM, Alrumaih NS: Ac-curacy and readability of websites on kidney and bladder cancers. J Cancer Educ [epub ahead of print], 2017

3. Cerminara C, Santarone ME, Casarelli L, Curatolo P, El Mal-hany N: Use of the DISCERN tool for evaluating web searches in childhood epilepsy. Epilepsy Behav 41:119–121, 2014

4. Charnock D, Shepperd S, Needham G, Gann R: DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Com-munity Health 53:105–111, 1999

5. Charvet-Berard AI, Chopard P, Perneger TV: Measuring quality of patient information documents with an expanded EQIP scale. Patient Educ Couns 70:407–411, 2008

6. Chi E, Jabbour N, Aaronson NL: Quality and readability of websites for patient information on tonsillectomy and sleep apnea. Int J Pediatr Otorhinolaryngol 98:1–3, 2017

7. Coiera E: The Internet’s challenge to health care provision. BMJ 312:3–4, 1996

8. Davis KS, McCormick AA, Jabbour N: What might parents read: sorting webs of online information on vascular anoma-lies. Int J Pediatr Otorhinolaryngol 93:63–67, 2017

9. Eysenbach G, Diepgen TL: Towards quality management of medical information on the internet: evaluation, labelling, and filtering of information. BMJ 317:1496–1500, 1998

10. Faden RR, Beauchamp TL: A History and Theory of In-formed Consent. New York: Oxford University Press, 1986

11. Fetto J: Mobile Search: Topics and Themes. New York: Hitwise, 2016 (http://hitwise.connexity.com/rs/371-PLE-119/images/hitwise-mobile-search-report-us.pdf) [Accessed Janu-ary 2, 2018]

12. Fox S, Duggan M: Health online 2013. Pew Research Center. January 15, 2013 (http://www.pewinternet.org/2013/01/15/health-online-2013/) [Accessed January 2, 2018]

13. Google: Search Engine Optimization Starter Guide. Mountain View, CA: Google, 2010 (https://static.googleusercontent.com/media/www.google.com/en//webmasters/docs/search-engine-optimization-starter-guide.pdf) [Accessed January 2, 2018]

14. Halpin G, Halpin G, Arbet S: Effects of number and type of response choices on internal consistency reliability. Percept Mot Skills 79:928–930, 1994

15. Jayaweera JMU, De Zoysa MIM: Quality of information available over internet on laparoscopic cholecystectomy. J Minim Access Surg 12:321–324, 2016

16. John ES, John AM, Hansberry DR, Thomas PJ, Agarwal P, Deitch C, et al: Colorectal cancer screening patient education materials—how effective is online health information? Int J Colorectal Dis 31:1817–1824, 2016

17. Komorita SS, Graham WK: Number of scale points and the reliability of scales. Educ Psychol Meas 25:987–995, 1965

18. Leiner BM, Cerf VG, Clark DD, Kahn RE, Kleinrock L, Lynch DC, et al: Brief History of the Internet. Reston, VA: Internet Society, 1997 (https://www.internetsociety.org/wp-content/uploads/2017/09/ISOC-History-of-the-Internet_1997.pdf) [Accessed January 2, 2018]

19. Lella A: comScore releases February 2016 U.S. desktop search engine rankings. comScore. March 16, 2016 (https://www.comscore.com/Insights/Rankings/comScore-Releases-February-2016-US-Desktop-Search-Engine-Rankings) [Ac-cessed January 2, 2018]

20. Masters JR: The relationship between number of response categories and reliability of Likert-type questionnaires. J Educ Meas 11:49–53, 1974

21. Matell MS, Jacoby J: Is there an optimal number of alterna-tives for Likert scale items? Study I: Reliability and validity. Educ Psychol Meas 31:657–674, 1971

22. Maugans TA, McComb JG, Levy ML: The internet as a pedi-atric neurosurgery information resource. Pediatr Neurosurg 28:186–190, 1998

23. McCool ME, Wahl J, Schlecht I, Apfelbacher C: Evaluating written patient information for eczema in German: compar-ing the reliability of two instruments, DISCERN and EQIP. PLoS One 10:e0139895, 2015

24. McKearney TC, McKearney RM: The quality and accuracy of internet information on the subject of ear tubes. Int J Pe-diatr Otorhinolaryngol 77:894–897, 2013

25. Meredith P, Emberton M, Wood C: New directions in infor-mation for patients. BMJ 311:4–5, 1995

26. Minervation: The LIDA Instrument - Minervation Validation Instrument for Health Care Web Sites (v1.2). Oxford, UK: Minervation, 2007 (http://www.minervation.com/wp-content/uploads/2011/04/Minervation-LIDA-instrument-v1-2.pdf) [Accessed January 2, 2018]

27. Moult B, Franck LS, Brady H: Ensuring quality information for patients: development and preliminary validation of a new instrument to improve the quality of written health care information. Health Expect 7:165–175, 2004

28. Nghiem AZ, Mahmoud Y, Som R: Evaluating the quality of internet information for breast cancer. Breast 25:34–37, 2016

29. Schairer WW, Kahlenberg CA, Sculco PK, Nwachukwu BU: What is the quality of online resources about pain control after total knee arthroplasty? J Arthroplasty 32:3616–3620, 3620.e1, 2017

30. Silberg WM, Lundberg GD, Musacchio RA: Assessing, con-trolling, and assuring the quality of medical information on the Internet: Caveant lector et viewor—Let the reader and viewer beware. JAMA 277:1244–1245, 1997

31. Tang H, Ng JHK: Googling for a diagnosis—use of Google as a diagnostic aid: internet based study. BMJ 333:1143–1145, 2006

32. U.S. Department of Health and Human Services: Quick Guide to Health Literacy. Washington, DC: U.S. Depart-ment of Health and Human Services, 2015 (https://health.gov/communication/literacy/quickguide/Quickguide.pdf) [Ac-cessed January 2, 2018]

33. Weng LJ: Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educ Psychol Meas 64:956–972, 2004

34. Winker MA, Flanagin A, Chi-Lum B, White J, Andrews K, Kennett RL, et al: Guidelines for medical and health infor-mation sites on the internet: principles governing AMA web sites. JAMA 283:1600–1606, 2000

DisclosuresThe authors report no conflict of interest concerning the materi-als or methods used in this study or the findings specified in this paper.

Author ContributionsConception and design: Maugans. Acquisition of data: Maugans, Garcia, Daugherty. Analysis and interpretation of data: all authors. Drafting the article: Maugans, Garcia. Critically revising the article: all authors. Reviewed submitted version of manuscript: all authors. Approved the final version of the manuscript on behalf of all authors: Maugans. Statistical analysis: all authors. Administrative/technical/material support: Maugans. Study supervision: Maugans.

Supplemental InformationPrevious PresentationsPortions of this work were presented at the 45th annual meeting of the AANS/CNS Section of Pediatric Neurosurgery, December 5–8, 2016, Orlando, FL.

CorrespondenceTodd Maugans: Nemours Children’s Hospital, Orlando, FL. [email protected].


critical assessment of pediatric neurosurgery patient

Documents