aep research and policy brief from anecdote to evidence...from anecdote to evidence: assessing the...

ArtsEducationPartnership

From Anecdote to Evidence:Assessing the Status and Conditionof Arts Education at the State Level

Among arts and education advocates, concernsabout a “narrowing of the curriculum” have grownsince the 2001 passage of the No Child LeftBehind Act (NCLB). A Center for Education Poli-cy analysis found that in school districts acrossthe country, the arts and other subjects are beingsqueezed out of the school day to make room formore math and reading. But beyond these basicgrim facts, most of what we know about the cur-rent status and condition of arts education in thenation’s public schools is based on sketchy infor-mation and anecdotes rather than on solid empir-ical evidence.

There is a pressing need for better and more com-prehensive state level information if the arts are toremain an integral component of what constitutesa well-rounded education for all students. Withoutit, it will be nearly impossible to make a convincingcase for the arts, assess current environments,monitor compliance with NCLB, or ensure equi-table access to existing programs. Good policystarts with good data and recently state policy lead-ers themselves have called for research-based evi-dence about arts education in their own and otherstates to help inform their decisions.

The field also recognizes the need. In 2004, theArts Education Partnership (AEP) issued The Artsand Education: New Opportunities for Research,which noted “[r]eliable information is unavailableabout student access to arts instruction, about thecurrent and predicted availability of qualifiedteachers, and about student performance.”

Fortunately, the data-free climate of policy and deci-sion making for the arts in education is starting tochange. In a growing number of states, state level edu-cation agencies and arts organizations have joinedforces with other critical partners to undertake com-prehensive statewide surveys of school districts andschools. Among the things they are learning: howmuch time students spend studying the arts; whoprovides the instruction they do receive and underwhat conditions; and how much money is spent onschool arts programs. The findings already have pro-duced measurable results in some states, such as achange in the number of arts credits required for highschool graduation or an added line item in the statebudget earmarked specifically for arts education.

This research and policy brief draws on the experi-ences in five states, each of which has been the subjectof a comprehensive arts education survey in recentyears. The states are Illinois, Kentucky, New Jersey,Rhode Island, and Washington. Distilled from the “les-sons learned,” the brief provides 20 principles designedto inform and guide new and continuing arts educa-tion data collection and survey research at the statelevel. A primary audience is those who may beinvolved in advancing such efforts in their own states.

In addition to an analysis of relevant written materi-als from the five states, the brief was informed by thedeliberations of a select group of arts educationresearch, policy, and communications experts. At theinvitation of the AEP, the group came together inJune 2006 for a one-day seminar in Chicago, sup-ported in part through a grant from the SpencerFoundation. Drawing on their direct involvement inthe research and survey work in one of the five states,the seminar participants provided valuable insightsinto the broader context in which the studiesoccurred as well as into the outcomes that followed.

AEP Research and Policy Brief

One MassachusettsAvenue, NWSuite 700Washington, DC 20001www.aep-arts.org

Richard J. DeasyDirector

Kathryn W. BeachlerAssociate forCommunications andPartnershipDevelopment

Karen S. GallagherSenior Associate forCommunications andPartnershipDevelopment

Andrew L. NelsonConsultant

Sandra S. RuppertSenior Associate forResearch and Policy

The mission of the ArtsEducation Partnershipis to demonstrate andpromote the essentialrole of the arts inenabling every studentto succeed in school,life and work in thediverse and globaleconomies and societiesof the 21st century.

FROM ANECDOTE TO EVIDENCE: ASSESSING THE STATUS AND CONDITION OF ARTS EDUCATION AT THE STATE LEVELARTS EDUCATION PARTNERSHIP/NOVEMBER 2006

Introduction

1

By Sandra S. Ruppert and Andrew L. Nelson

0610.AEPReport.qxd 11/1/06 1:47 PM Page 1

A necessary first step is to define a clear set of goalsand purposes for conducting a statewide survey ofarts education. Key aspects of the process aredescribed in the principles below.

1. Frame research goals within a broadereducation context.

In most cases, the survey research is part of a larg-er initiative related to education reform or schoolimprovement efforts. To understand the state con-text, it helps to remember that NCLB directlyaffects education policy and practice at the statelevel. The law also recognizes the arts as a core aca-demic subject, alongside mathematics, languagearts, science, and several other subjects. Further, itrequires a complete education for every child mustinclude rigorous instruction in all core academicsubjects. In addition to the federal law, 44 statesand the District of Columbia have policies thatdefine the arts as “core” or as an academic subjectin statute or administrative code, according to a 50-state analysis conducted in 2005 by the EducationCommission of the States (ECS).

2. Define the problem to be addressed in policy terms.

Determine opportunities and needs in the policyenvironment that will help bring about long-term,systemic change, rather than trying to win just theshort-term battles. Like the vast majority ofstates, Kentucky has policies in place that requireschools or districts to provide arts instruction forits students. One underlying research question

study leaders in Kentucky addressed in the base-line survey of arts education in the state was this:Are students being underserved in the arts? Theanswers allow them to draw inferences about lev-els of access to and participation in arts educationprograms in light of these policies and to makerecommendations. It helps to have an under-standing of how policy is made in the state and toask if there is political support behind the study,and an amenable policy environment in which toconduct it.

3. Gain the support and commitment of state leaders.

Defining the problem in policy terms can make iteasier to enlist the support of state leaders who canbe influential in moving the study forward. Theearly buy-in of state arts and state education enti-ties can be especially important. In Illinois, thestate principals’ association and the board of edu-cation endorsed the research and encouraged sur-vey participation. To ensure the study results havean impact beyond the advocacy level, the overallinitiative should include key leadership who cansteer the findings into the policy realm. Having anauthority, such as a governor, state legislature, orstate superintendent formally initiate the effort canadd legitimacy to the policy findings and futureactions. In Rhode Island, Governor Almondappointed 19 influential leaders from the arts, edu-cation, and business sectors to serve on his TaskForce on Literacy in the Arts and to issue policyrecommendations.

4. Create an organizational infrastructureto guide the work.

Forming a statewide steering committee that canassume “ownership” of the initiative can involve anumber of steps: Make the process of organizingthe study as transparent as possible and constructit with key stakeholders. Invite other organiza-tions, including funders, to become strategic part-ners. Consider involving a government relationsand/or public relations strategist at the beginningof the project. Their early input can be importantto converting the resulting data into effective,

20 Principles to Inform State LevelArts Education Survey Research

The 20 principles are organized according tofour basic research design components: clearresearch goals and purposes, appropriatemethods for data collection and analysis, effectivecommunications strategies, and direct policy andpractice implications.

Clear Research Goalsand Purposes

FROM ANECDOTE TO EVIDENCE: ASSESSING THE STATUS AND CONDITION OF ARTS EDUCATION AT THE STATE LEVEL ARTS EDUCATION PARTNERSHIP/NOVEMBER 20062


high-impact, clear, concise, and newsworthy mes-sage points that will resonate with elected officials,policy leaders, and the media. However, there is nosubstitute for the solid foundation a competentand credible staff provides, including a projectdirector as the point person who pulls all thepieces together.

5. Articulate explicit purposes for the study.

Going forward, the stated purposes for the studywill aid in determining the specific types of data tobe collected, how they will be analyzed, and themanner in which they will be reported to appropri-ate audiences. Achieving a sharp focus may requiremaking tough choices; it is unrealistic to expect thestudy to be all things to all people. These are amongthe commonly identified purposes for a statewidesurvey on the status and condition of the arts at theschool or district level:

• Raise awareness — Survey results can serve asthe basis for a statewide advocacy campaign orfor seeking additional support. In Illinois, a stat-ed purpose was to raise awareness among Illi-nois policymakers of the value of arts educationand to advocate for stronger state and local poli-cies, along with corresponding budget appropri-ations, to ensure that arts education is providedto students statewide.

• Monitor the general condition of arts educa-tion — The study can serve as a means forestablishing a “baseline” for tracking and meas-uring progress in the state, in some cases, basedon compliance with NCLB requirements. Theindicators also can be included on school or dis-trict level report cards. Kentucky’s first-everbaseline survey was designed to provide a“snapshot” of arts education and to inform thedevelopment of future programs and partner-ship efforts.

• Promote school or program improvement —In New Jersey, one purpose for the survey wasto provide schools and communities with toolsand resources to strengthen student learning inthe arts. The findings will be used to identifymodel arts programs and to serve as the basisfor establishing a clearinghouse on arts educa-tion where schools can share ideas. In Washing-ton, the foundation that helped support the

research was particularly interested in identify-ing school-based solutions to shared challengesin the funding, staffing, and scheduling of thearts in the day-to-day curriculum.

• Inform policy decisions — In Rhode Island,the governor-appointed task force was explicitlycharged with “mak[ing] policy recommenda-tions on how the arts can have a significanteffect on the educational agenda” of the state.

• Align resources — While not typically a pri-mary purpose, funding sources and levels forarts education and spending patterns over timecan reveal disparities in the level of arts educa-tion that schools are able to provide. This infor-mation can be useful in advocating for the allo-cation of new resources as well as for a betterinvestment of existing resources.

NEW JERSEY

A Well-Defined Purposeand Long-Range Planning

FROM ANECDOTE TO EVIDENCE: ASSESSING THE STATUS AND CONDITION OF ARTS EDUCATION AT THE STATE LEVELARTS EDUCATION PARTNERSHIP/NOVEMBER 2006 3

The Music for All Foundation, New Jersey StateCouncil on the Arts, New Jersey Department ofEducation, The Geraldine R. Dodge Foundation,and Playwrights Theatre of New Jersey formed theNew Jersey Arts Education Census Project with aclear purpose: “to gather, evaluate and disseminatequalitative and quantitative data regarding artseducation in the state of New Jersey.” At the pro-ject’s inception, these organizations established afive-goal process for collecting and disseminatingthe data. The goals include: (1) disseminate astatewide survey to all schools; (2) connect thisinformation to other state school and census data;(3) create an online arts education research center;(4) serve as a national model of state arts educa-tion research; and (5) create an ongoing program.By thinking through the collection and distribu-tion of the data before beginning their research,the Arts Education Census Project developed awell-defined purpose that will guide its researchand provide benchmarks that can be used to meas-ure progress.


12 Commonly Used Indicatorsto Assess the Status andCondition of Arts Education• Time/frequency provided for arts instruction

within school schedules

• Number and range of arts course offerings

• Percent of students participating in arts courses

• Number of credits in the arts required for highschool graduation

• Percent of certified or licensed teachers to teacharts education

• Availability of professional development work-shops and teacher planning time

• Frequency of arts-based field trips, residenciesand extra curricular activities

• Presence of designated arts classrooms and useof technology in arts learning

• Evidence of alignment of arts instruction withstate standards

• Presence of documented arts curricula

• Type of assessment tools for measuring studentarts performance

• Amount of school and outside funding for artsprograms

Once the goals and purposes for conducting sur-vey research are established, the next step is todesign the study itself, which can entail determin-ing the scope of work, the methodology to beemployed, and a timeframe and budget for execut-ing it. The conclusions and impact of the study willdepend largely on a carefully crafted and success-fully implemented research design. Some princi-ples that contribute to a well-designed study aredescribed below.

6. Work with an oversight committee to designand monitor the survey.

Some form of oversight committee or advisorygroup offers an important opportunity to involverepresentatives from the education, arts, art educa-tion, higher education, business, government, orcivic communities in the study. Among other func-tions, committee members can provide a “fresh setof eyes” and useful feedback during a pilot phase ofthe survey. Kentucky utilized a project oversightcommittee of state and local partner agencies forthe initial design and construction of the writtensurvey instrument. The committee’s primary focuswas to identify relevant categories for explorationand inclusion in the study. In Washington, a consor-tium of arts educators, school administrators, artsagency personnel, and teaching artists helped todefine key attributes of quality for the arts educa-tion components the researchers would investigate.

7. Seek research expertise early on.

Individuals with expertise in qualitative or quanti-tative research methods can provide invaluableassistance in the development of an appropriateand robust survey instrument. Typically, they alsocan perform sophisticated statistical analyses andassist in the interpretation of results. In Illinois, theresearch firm employed was able to devise an indexthat included measures for ranking arts educationby district and county. Research analysts alsoorganized a presentation of the data. To locate out-side help, prepare a request for proposals (RFP)and distribute it to various research firms that spe-cialize in practice-oriented or applied research.

8. Select the right survey tool and measures for the job.

Take care in identifying the indicators that will beassessed in the survey and, in most cases, trackedover time. For better or worse, it is generally under-stood that “what gets measured is what gets valued.”As shown below, what gets measured tends to bethose indicators that are quantifiable and can beeasily understood and interpreted by a wide audi-ence. They also are more amenable to presentationin tabular form and most allow for school or district

Appropriate Methods for DataCollection and Analysis



KENTUCKY

Achieving a High SurveyResponse RateEven though they were not mandated to do so bythe state Department of Education, 77% of Ken-tucky school districts completed the arts educa-tion survey that was developed by the Collabora-tive for Teaching and Learning with funding fromthe Kentucky Arts Council. The Kentucky Com-missioner of Education distributed the survey toall 176 of the state’s district superintendents viaemail, and superintendents also had the option ofobtaining a hard copy version from the Collabora-tive. To make the survey process as efficient aspossible, superintendents were told in advancewhat information they would need to have avail-able in order to fill out the questionnaire. Twoweeks before the survey deadline, the Commis-sioner sent a reminder email to the superintend-ents, and the survey deadline was then extendedby two weeks to enable telephone calls to a sam-pling of non-respondent districts, which increasedthe response rate by 16%.

level comparisons. Decide the best way to collectresponses — via the web, a paper-based survey, orboth ways. Increasingly, online surveys are becom-ing the norm; if using an online survey bear in mindthat human and technological aspects have to workin tandem. To supplement the quantitative data,consider gathering qualitative data through one-on-one or group interviews using a prepared questionguide or survey protocol to gain insights into peo-ple’s attitudes and perceptions about issues.

9. Identify the correct data sourceto answer questions.

Potential sources of information about the statusand condition of arts education in a state are numer-ous. Who receives the survey and who shouldrespond to it depend on what researchers want toknow. Appropriate respondents for statewide datacollected at the school or district level might includeprincipals, teachers, curriculum specialists, artsconsultants, or superintendents. Although the sur-veys in New Jersey and Washington went to chiefschool administrators and principals, it was recom-mended that they engage school or district arts staffin the process of information gathering. In Illinois,researchers used two slightly different versions ofthe same questionnaire — one designed for princi-pals and the other for superintendents. Given thesimilarity in the findings, however, researchers ques-tion whether the effort involved in analyzing twoseparate data sets was worth the extra investment oftime and resources. Some of the most difficult infor-mation to extract may be arts funding amounts andsources as they often are woven into multiple budg-et line items, or contributed by “soft” resources thatcan change from year to year.

10. Take steps to ensure an adequate response rate.

Among the actions that study leaders can take toincrease the survey response rate are these steps:

• In the introduction to the questionnaire, informrespondents of the reasons for the study andwho will see the results

• Let respondents know what information theywill need to gather in advance in order to com-plete the survey

• Include a glossary that gives clear definitions of

even commonly used terms, like “school day”and “curriculum,” and include detailed instruc-tions for completing the questions

• Address the concerns of respondents up front:Assure them the survey results will not be usedagainst them

• Keep the length of time needed to complete thesurvey short — 20 minutes is considered rea-sonable

• Provide letters of support or other validationfrom state leaders

• Send e-mail reminders or make follow-up callsas the response deadline approaches

• Be willing to apply pressure when needed. Toensure that data were gathered from all 2,400schools in the state, the New Jersey Commis-sioner of Education mandated participation inthe online survey.



WASHINGTON STATE

Gathering Quantitativeand Qualitative DataWhen conducting their statewide arts education sur-vey, researchers for Washington’s Arts EducationResources Initiative developed a methodology thatallowed them to collect both quantitative and quali-tative data regarding the state’s schools. After distrib-uting an online survey to all of Washington’s K-12principals, the researchers employed a collaborative-ly-defined “markers of arts education quality” tool toidentify 32 schools across the state that exhibited keyattributes. During these site visits, they interviewedprincipals, teachers, and arts specialists. The inter-views allowed researchers to gather in-depth infor-mation about specific school-based solutions tocommonly perceived challenges, such as tight build-ing schedules, fluctuating funding resources, and artscurricula documentation processes, which cannot beeasily collected in an online survey. The result of thismethodological approach is a detailed body of evi-dence that not only paints a picture of arts learning inWashington schools, but potentially links schoolswishing to maximize arts education programs withthose that already have created viable solutions.

Audience issues — how, what, and to whom infor-mation gets communicated — play a significant rolein determining the relevance and usefulness of thesurvey results. Following are principles for develop-ing an effective communications strategy that willhelp in reaching target audiences with informationthat makes sense out of oftentimes complex data.

11. Develop a communications plan to disseminatefindings.

Treat the release of survey results like a campaign.Set aside sufficient resources for production andmultiple means of communication with visuals thatmake it easy for elected officials and others to

absorb information. When developing a communi-cations plan, think about how you can graphicallyconvey the information for maximum impact.Often more data are collected than can actually bereported; it is not necessary to report the results ofevery survey question. Determine what is most rel-evant and newsworthy and highlight those findingsin your report and communication efforts. Be cau-tious, however — any graph or data organizer cantilt a story. Consider doing a press conferencelaunch of the results. It not only gets the attentionof policymakers, but it also can be an opportunityfor them and for schools to actively participate inthe study and become a partner in future efforts.

12. Define target audiences.

Study leaders need to plan how they will communi-cate with key messages developed specifically foridentified target audiences. Because most individu-als can absorb only a handful of major findings thatillustrate the status of the arts, create graphic andvisual representations that send a message at aglance. Determine what kinds of indicators willresonate with particular audiences. For example,the sort of information that teachers, arts supervi-sors, superintendents, and principals need to eval-uate and plan for their arts education programsmay be different from what arts-active parentsneed to select a school with an arts-rich environ-ment for their children or what advocates need tomake the case for the arts with state education pol-icymakers. In Washington, two distinct, but com-plementary, documents were created: one designedas an advocacy tool while the other contained in-depth research details and action steps.

13. Tap into existing data collection efforts.

In many cases, the data gathered through astatewide survey of the arts in education can besupplemented with information drawn from othersources. Part of the plan in New Jersey, for exam-ple, is to mesh survey results with existing educa-tional and census databases to create the New Jer-sey Arts Education Information and ResearchCenter. Using specially designed technology, thesystem will have the capacity to sort informationaccording to what people want to know, generateindividual reports, compare schools and districts,and geographically display the levels of arts educa-tion across the state.


Effective CommunicationsStrategies


ILLINOIS

Developing a CommunicationsPlan to Publicize Survey ResultsWhen Illinois Creates, a statewide coalition led bythe Illinois Arts Alliance, began its statewide artseducation survey of Illinois schools, it hired a pub-lic relations firm to develop a communications planthat would allow them to share the findings of thesurvey not only with school leaders and policymak-ers, but also with the media and public at large.After the state superintendents and principalscompleted the questionnaire, a press release sum-marizing the research findings was issued and aseries of press conferences were held throughoutthe state. As a result of the firm’s proactive efforts,stories were placed in almost every major print, tel-evision, and radio outlet, including the Chicago SunTimes and the Chicago Tribune. The pubic rela-tions firm also offered editorial assistance in thepublication of the Illinois Creates’ Arts at the Core:Every School, Every Student report, which cogentlydescribes the survey results.

14. Turn complex data into useful information.

To ensure the findings are easy to understand andinterpret, study leaders need to be able to weavecomplex data together into timely information thatpeople can use to make decisions. In Illinois, anindex was created that scored the schools in eachzip code based on 11 arts education measures.Researchers set up the variables so they could iden-tify disparities in many different ways, includingsocioeconomic, geographic, or racial categories.The findings showed significant disparities wereattributable to size; that is, the number of studentsthe school or the district served, and whether theschool was in a rural or urban setting.

15. Offer comparisons, but exercise caution in passingjudgment.

In presenting statewide results, the “big trends” areoften of most interest to general audiences. However,when tracking geographic disparities within thestate, for example, it is necessary for comparativepurposes to disaggregate the data into smaller units

of analyses, such as at the county, school district,school, or zip code level. Consider carefully the use ofratings and scales when it might lead to making judg-ments about “winners” and “losers:” The news caneffectively alienate particular audiences you want toreach. In Illinois, even though the scores were low forsome districts, study leaders identified places wherethings were working well within each district. Fur-ther, the study results offered recommendations forimproving arts education in the state.


Direct Policy andProgram ImplicationsSound policy depends on the availability of infor-mation that supports good decision making. Thefindings can and should lead to recommendationsthat have direct and actionable implications forpolicy and practice. Following are some ways inwhich data or information generated through astatewide survey can serve as an advocacy and pol-icy tool to effect change for arts education at theschool or district level.

16. Use the results to serve as a wake up call.

Hearing the hard cold facts about the status and con-dition of arts education in the state can be a way tocatch and keep the attention of key constituencies. InIllinois, based on geographic comparisons, the find-ing that the state ranked below national averages senta particularly powerful message to the state’s educa-tion leaders. The state superintendent of educationcalled the survey research a “wake up call” and vowedto use his “bully pulpit” to increase arts education inthe schools. In Washington, the study stimulated anew evidence-based conversation among state lead-ers and educators about how to realize the long-rangeand short-range goals to support arts education cur-ricula, professional development, and assessments.

17. Examine the policy implications.

The value of the survey results depends on how theyare applied to the task of making or interpreting pol-icy. In New Jersey, the findings will gauge how wellindividual elementary, middle, and high schools areadhering to a state mandate to provide a well-round-ed arts education. In Rhode Island, as a result of the


RHODE ISLAND

Using an Arts EducationSurvey as a Springboard forSchool ReformUsing questionnaires to gather baseline arts educationdata was only the beginning of the work of RhodeIsland’s Literacy in the Arts Task Force. After surveyingstate schools, institutions of higher learning, and com-munity organizations, the Task Force shared the surveyresults with these entities, allowing them to view theresults and better understand the status of arts educa-tion in the state and their own work. The surveys and thedissemination of their findings were the catalyst for theformation of the Rhode Island Arts Learning Network.This is a structure that can act as an advocacy organiza-tion to support and connect arts learning in home,school, and community, in order to promote equalaccess to arts learning in and out of school for all RhodeIsland children and youth. The network uses five region-al representatives to move its agenda forward and suc-cessfully advocated for a change in state graduationrequirements that mandates that all students demon-strate proficiency in an art form before graduation.

statewide survey and other activities undertaken bythe Task Force on Literacy in the Arts, study leaderswere able to advocate for and successfully change thearts graduation requirement so that it now applies toall students, not just the college-bound. Additionally,schools in the state of Washington have been advisedof how they can use data generated by the survey toargue for a line item in the district budget or for seek-ing support for arts programs.

18. Have a plan for technical assistance.

If possible, the state department of educationshould allow schools access to the data related tothem and support the schools with needed assis-tance after the research findings are published.Providing this support may require better coordi-nation of programming and allocation of resourcesin conjunction with more effective training andsupport to those who deliver these services. Basedon the 2005 ECS state-by-state analyses, a handfulof states require local districts to assess the arts;however, only Kentucky currently includes assess-ment in the arts as a component of its statewideassessment system. Partners there plan to use thesurvey findings as the basis for the development ofa school-based self assessment instrument gearedtoward improving the delivery and quality of artseducation programs. In Illinois, the state board ofeducation created a grant program for arts educa-tion that will provide school districts with planningdollars to improve and enhance their arts educa-tion programs.

19. Reframe the issues around equitable access to ahigh quality arts education.

New ways of framing the policy issues to focus onfair and equitable access to a high quality and well-rounded education for all students can emergefrom the survey research work. For example, studyleaders in New Jersey wanted to know “who hasaccess and who doesn’t” to arts instruction andlearning experiences in light of the state’s mandateto provide a well-rounded arts education. In RhodeIsland, arts learning is considered an integral com-ponent of the state’s “all kids to high standards”education agenda. Yet, the study found a lack ofequity in physical and programmatic access to artslearning opportunities. In Illinois, researcherslooked at meaningful correlations with drop outrates, attendance patterns, per pupil operating

expenditures, and other indicators to help make apersuasive case for arts education.

20. Build the capacity for implementation andsustainability.

Going forward, study leaders can be instrumental inthe development of new community-based struc-tures designed to implement the recommendationsgenerated by the survey research. The study findingsthemselves can serve as a community building tool.In spite of Rhode Island being arts rich, there was nostatewide coordination of arts learning for childrenand youth across the sectors of home, school, andcommunity. Based on recommendations of the taskforce, the Rhode Island Arts Learning Network wascreated to coordinate the effort. Established as apublic/private partnership, the network facilitatespublic engagement, assists in aligning resources, andfacilitates dialogue and action between and amongits partners. No existing organizations in the statehad the capacity to address these challenges.



20/20 CHECKLIST

Principles for Assessing theStatus and Condition of ArtsEducation at the State Level

The authors gratefully acknowledge the generoussupport of the Spencer Foundation and the impor-tant contribution made by the following individu-als who participated in the June 2006 seminarand/or who provided comments on an earlier draftof this brief: Julie Adrianopoli, Kathryn Beach-ler, Sherilyn Brown, Patricia Cirillo, Karen Gal-lagher, Ann Galligan, Dennis Horn, PeterMcWalters, Robert Morrison, Dale Schmid,Melanie Scofield, Gerri Spilka, Alene Valkanas,and Susy Watts. Special thanks to Richard J.Deasy, director of the Arts Education Partnership,for his substantial contribution to all aspects ofthis work.


ConclusionMost of our current knowledge about the status andcondition of arts education in the nation’s publicschools is drawn from anecdotes rather than empir-ical evidence. However, an increasing number ofstates either have undertaken or are in the processof conducting a comprehensive statewide survey ofthe quality and quantity of arts instruction andlearning experiences in their school districts andschools. This research and policy brief examines therecent experiences of five such states — Illinois,Kentucky, New Jersey, Rhode Island, and Washing-ton — to distill the lessons learned in hopes that itwill spur other states to action. The brief offers 20principles, organized according to four basicresearch design components, which can serve toinform and guide their efforts.

Here are 20 principles — 20 actions to consider —in creating a vision for a state level research surveyto assess the status and condition of arts educationin your state.

• Frame research goals within a broader educa-tion context

• Define the problem to be addressed in policy terms

• Gain the support and commitment of state leaders

• Create an organizational infrastructure to guidethe work

• Articulate explicit purposes for the study

• Work with an oversight committee to designand monitor the survey

• Seek research expertise early on

• Select the right survey tool and measures forthe job

• Identify the correct data source to answerquestions

• Take steps to ensure an adequate response rate

• Develop a communications plan to disseminatefindings

• Define target audiences

• Tap into existing data collection efforts

• Turn complex data into useful information

• Offer comparisons, but exercise caution in pass-ing judgment

• Use the results to serve as a wake up call

• Examine the policy implications

• Have a plan for technical assistance

• Reframe the issues around equitable access to ahigh quality arts education

• Build the capacity for implementation and sus-tainability

References

Acknowledgments

Arts Education Partnership, The Arts and Educa-tion: New Opportunities for Research, Washington,DC: AEP (2004).

Center for Education Policy, From the Capital tothe Classroom: Year 4 of the No Child Left BehindAct, Washington, DC: CEP (2006).

Education Commission of the States, Artscan data-base, Denver, CO: ECS. www.ecs.org/artscan(accessed August 10, 2006).


civic organizations and schools that works to raisearts education awareness to Illinois policymakersand the public at large about the importance of artseducation as a core learning area.

http://www.artsalliance.org/ed_research.shtml

KENTUCKY

In 2005, the Kentucky Arts Council (KAC) con-tracted the Collaborative for Teaching and Learn-ing to conduct the first-ever statewide survey ofarts education in Kentucky schools in order to“establish a set of baseline data which will informthe design of future KAC arts education programsand partnership efforts aimed at making the artsintegral to education in the commonwealth.”

Distributed to the superintendents of all 176 Ken-tucky school districts, the survey was based on theOhio Alliance for Arts Education’s 2001 report,Status of Arts Education in Ohio’s School Districts,and included six major categories of investigation:(1) Teaching, Support Personnel and Parent/Com-munity Involvement; (2) Curriculum and Instruc-tion; (3) Arts Core Content for Assessment andProgram of Studies; (4) Residencies, Field Trips,Extra-Curricular Activities, Use of Resources; (5)Facilities and Technology; and (6) Use of State-Level Arts Resources. Respondents were emailedthe survey and had the option of completing itonline or in hard copy form. Of the 176 districts,135 completed the survey for a 77% return ratethat represented 100 of Kentucky’s 120 counties.

The Collaborative for Teaching and Learning pub-lished the survey results in a 2005 report entitledStatus of Arts Education in Kentucky PublicSchools. Survey data are being used in a variety ofways, including in the early planning of a newschool arts self-assessment process to assist Ken-tucky schools with assessing their current compre-hensive school arts program, against a set of per-formance standards, to determine where gaps existand to plan for program improvement.

http://artscouncil.ky.gov/

NEW JERSEY

In 2004, the Music for All Foundation (MFA),working in partnership with the New Jersey State

ILLINOIS

In 2004, the Illinois Arts Alliance (IAF) and theChicago Community Trust launched Illinois Cre-ates, an arts education initiative designed “to advo-cate for stronger state and local policies and corre-sponding budget appropriations to ensure that artseducation is provided to students statewide.” Inspring 2005, Illinois Creates and the research firmMetro Chicago Information Center performed astatewide study aimed at gaining an understandingof the status of arts education in the state and bring-ing to light barriers to high quality arts education.The research project included two questionnaires,one that was mailed to all 881 Illinois superintend-ents and one that was sent to each of the state’s3,892 public school principals. Thanks in part to theendorsement and encouragement of the IllinoisPrincipals Association and the Illinois State Boardof Education, 234 superintendents (26.5%) and 751principals (19.2%) across the state completed thesurvey, giving the researchers a sound representa-tive sample of Illinois arts education practices.

In collaboration with Cypress Research Inc., Illi-nois Creates analyzed the survey results and com-pared them to State Board of Education data forschools and districts in order to ensure their valid-ity. The results of the study were published in the2005 report, Arts at the Core: Every School, EveryStudent. This publication summarizes the researchfindings and includes an index of arts educationlevels that scores schools in each state zip codebased on 11 arts education measures. The reportalso offers eight recommendations for improvingarts education in the state.

Since its inception and the completion of theresearch study, Illinois Creates has become a strongcoalition of nearly 300 education, business, and


Appendix

Profiles of State Level Surveyson the Status and Conditionof the Arts in Education inFive States


Council on the Arts (NJSCA), the New JerseyDepartment of Education (NJDOE), the GeraldineR. Dodge Foundation, and Playwrights Theatre ofNew Jersey (PTNJ), launched the New Jersey ArtsEducation Census Project, with additional supportprovided by NAMM, the International MusicProducts Association, the D’Addario Foundationfor the Performing Arts, and David Bryan of BonJovi. Building on the NJSCA’s and PTNJ’s 1996-98Mapping Project, the Arts Education Census Pro-ject is working to “gather, evaluate and disseminatequalitative and quantitative data regarding artseducation in the state of New Jersey.” In April2006, the NJDOE circulated online the New JerseyVisual and Performing Arts Survey to all 2,408schools in the state. Two surveys were disseminat-ed — one for schools that include grades K-6 andone for those serving students in grades 7-12. Bothsurveys included qualitative measures that exam-ine arts education policies and instruction, as wellas quantitative measures regarding student partic-ipation, enrollment, teachers, and certification.Ninety-eight percent of New Jersey schools and100% of the state’s school districts completed theonline survey.

The next step in the Arts Education Census Pro-ject’s plan is to join its survey results with existingeducational and census databases in the state tocreate the New Jersey Arts Education Research andInformation and Research Center (NJAERIC). TheCenter will offer researchers arts education data aswell as detailed information regarding New Jerseyschool and state demographics, thereby providing“a more comprehensive view of each school, dis-trict, region, and the state.” The Center will serve asan online clearinghouse for information about artseducation in the state and will be the rallying pointfor the creation of a statewide arts education advo-cacy group to take the results of the Census Projectand create actionable strategies to advance artseducation across New Jersey. One of the goals ofthe NJ Arts Education Census Project is to serve asa national model that will encourage other states toperform similar studies.

http://www.music-for-all.org/NJAEC/

RHODE ISLAND

In 1999, Rhode Island Governor Lincoln Almondissued an Executive Order calling for the creation


of The Governor’s Task Force on Literacy in theArts. In the Executive Order, the Governorcharged the Task Force “to examine the relation-ship between education reform and the arts, andto make policy recommendations on how the artscan have a significant impact on the educationalagenda of Rhode Island.” As a joint effort of theRhode Island State Council on the Arts and theRhode Island Department of Education, the TaskForce was composed of 19 leaders from state arts,education, and business communities appointedby Governor Almond. In order to provide a foun-dation for its efforts to assess arts education inRhode Island, the Task Force established the fol-lowing four areas of inquiry: (1) What role can anddo the arts play in overall education reform?;(2) What is the status of arts learning in schoolsand in community organizations? What is given,to whom, by whom, and to what effect?; (3) Whatis the status of teacher preparation and training,both for arts educators and classroom teachers,and for artists and community educators?; and(4) Is there a role for home and community inarts learning?

To address these questions, the Task Forcereviewed current arts education scholarship, metwith national arts education scholars, and held dia-logue sessions with teachers, arts practitioners,parents, and students from around the state. TheTask Force then surveyed state K-12 school dis-tricts, institutions of higher learning, arts educa-tors, and community organizations in an effort togather baseline arts education data. The results ofthese surveys were distributed to a variety ofrespondents, and the Task Force gathered theirfeedback regarding the data.

The culmination of the Task Force’s efforts was thepublication of the 2001 report, A Framework forAction and the creation of the Rhode Island ArtsLearning Network. The report summarized thefindings of the Task Force’s research efforts andidentified three goals for literacy in the arts to berealized by 2008. Using five regional representa-tives, the Rhode Island Arts Learning Networkworks to support and connect arts learning inhome, school, and community settings and hasadvocated for a change in state graduation require-ments that mandates students demonstrate profi-ciency in an art form before graduation.

http://www.riartslearning.net/generalinfo/about.php


AEP Research and PolicySeminar on Principlesfor Conducting StateLevel Research on theArts in Education

Sponsored byThe Spencer FoundationJune 22, 2006Chicago, Illinois

ILLINOIS

Julie AdrianopoliPublic Policy DirectorIllinois Arts Alliance

Melanie ScofieldSenior PartnerScofieldCommunicationsChicago, Illinois

Alene ValkanasExecutive DirectorIllinois Arts Alliance

KENTUCKY

Dennis HornVice President,Communications andArts for LearningCollaborative forTeaching and LearningLouisville, Kentucky

NEW JERSEY

Dale SchmidVisual and PerformingArts CoordinatorNew Jersey StateDepartment ofEducation

RHODE ISLAND

Sherilyn BrownEducation DirectorRhode Island StateCouncil on the Arts

Ann GalliganResearch DirectorRhode Island ArtsLearning Network

Peter McWaltersCommissioner ofElementary andSecondary EducationRhode IslandDepartment ofEducation

WASHINGTON

Gerri SpilkaDirectorOMG Center forCollaborative LearningPhiladelphia,Pennsylvania

Susy WattsConsultant forStrategic Planning andEvaluationTumwater, Washington

WASHINGTON

As a project of the Washington State Arts Com-mission (WSAC), the Arts Education ResourcesInitiative (AERI) was created in 2004 “to helpschools step up to the challenge of bringing highquality arts education to all their students.” A keycomponent of the AERI project is the findings of asurvey of Washington schools that was performedto gather baseline data regarding arts educationpractices and “to learn approaches and identifysolutions that support implementation of arts edu-cation so students can meet state standards.”

With funding from Washington Mutual, AERIresearchers Gerri Spilka and Susy Watts developedan online survey, which was distributed to all K-12principals in the state. Twenty-one percent of prin-cipals completed the survey, and their responsesprovided data regarding such issues as the intensi-ty of student arts experience in schools, theamount of external support available for arts cur-ricula development, school staffing and profession-al capacity to offer arts learning, and arts assess-ment practices.

After conducting the survey, WSAC, theresearchers, and the statewide Arts Implementa-tion Task Force (AITF) met and developed charac-teristics and attributes that mark a quality arts edu-cation. The researchers then visited 32 schools thatexhibited at least one of these attributes and con-ducted interviews with principals, curriculumsupervisors and coordinators, arts specialists,members of boards of education, general class-room teachers, and artists-in-residence. Theseinterviews allowed researchers to gather bothquantitative and qualitative data and added moredetail to the findings of the survey.

Based on the survey and interview results, AERIpublished a 2006 booklet entitled Arts for EveryStudent, which describes the elements of effective,sustainable arts education in six areas: (1) Curricu-lum; (2) Assessment; (3) Teaching Capacity; (4)Collaboration; (5) Scheduling; and (6) Funding. Inaddition, the report provides action items andmarkers of quality for each of these areas and high-lights the arts education practices of some of thesurveyed schools.

http://www.arts.wa.gov/aeri.html


The Arts EducationPartnership is anational coalition ofarts, education, busi-ness, philanthropic andgovernment organiza-tions that demonstratesand promotes theessential role of artseducation in the learn-ing and development ofevery child and in theimprovement of Ameri-ca’s schools. The Part-nership includes morethan 100 organizationsthat are national inscope and impact. Italso includes state andlocal partnershipsfocused on influencingeducation policies andpractices to promotequality arts education.

©2006 Arts EducationPartnership

For more informationon ordering additionalcopies of this researchand policy brief orother AEP publications,please visit our websiteat www.aep-arts.org.


ArtsEducationPartnership

From Anecdote to Evidence:Assessing the Status and Conditionof Arts Education at the State Level

Among arts and education advocates, concernsabout a “narrowing of the curriculum” have grownsince the 2001 passage of the No Child LeftBehind Act (NCLB). A Center for Education Poli-cy analysis found that in school districts acrossthe country, the arts and other subjects are beingsqueezed out of the school day to make room formore math and reading. But beyond these basicgrim facts, most of what we know about the cur-rent status and condition of arts education in thenation’s public schools is based on sketchy infor-mation and anecdotes rather than on solid empir-ical evidence.

There is a pressing need for better and more com-prehensive state level information if the arts are toremain an integral component of what constitutesa well-rounded education for all students. Withoutit, it will be nearly impossible to make a convincingcase for the arts, assess current environments,monitor compliance with NCLB, or ensure equi-table access to existing programs. Good policystarts with good data and recently state policy lead-ers themselves have called for research-based evi-dence about arts education in their own and otherstates to help inform their decisions.

The field also recognizes the need. In 2004, theArts Education Partnership (AEP) issued The Artsand Education: New Opportunities for Research,which noted “[r]eliable information is unavailableabout student access to arts instruction, about thecurrent and predicted availability of qualifiedteachers, and about student performance.”

Fortunately, the data-free climate of policy and deci-sion making for the arts in education is starting tochange. In a growing number of states, state level edu-cation agencies and arts organizations have joinedforces with other critical partners to undertake com-prehensive statewide surveys of school districts andschools. Among the things they are learning: howmuch time students spend studying the arts; whoprovides the instruction they do receive and underwhat conditions; and how much money is spent onschool arts programs. The findings already have pro-duced measurable results in some states, such as achange in the number of arts credits required for highschool graduation or an added line item in the statebudget earmarked specifically for arts education.

This research and policy brief draws on the experi-ences in five states, each of which has been the subjectof a comprehensive arts education survey in recentyears. The states are Illinois, Kentucky, New Jersey,Rhode Island, and Washington. Distilled from the “les-sons learned,” the brief provides 20 principles designedto inform and guide new and continuing arts educa-tion data collection and survey research at the statelevel. A primary audience is those who may beinvolved in advancing such efforts in their own states.

In addition to an analysis of relevant written materi-als from the five states, the brief was informed by thedeliberations of a select group of arts educationresearch, policy, and communications experts. At theinvitation of the AEP, the group came together inJune 2006 for a one-day seminar in Chicago, sup-ported in part through a grant from the SpencerFoundation. Drawing on their direct involvement inthe research and survey work in one of the five states,the seminar participants provided valuable insightsinto the broader context in which the studiesoccurred as well as into the outcomes that followed.

AEP Research and Policy Brief

One MassachusettsAvenue, NWSuite 700Washington, DC 20001www.aep-arts.org

Richard J. DeasyDirector

Kathryn W. BeachlerAssociate forCommunications andPartnershipDevelopment

Karen S. GallagherSenior Associate forCommunications andPartnershipDevelopment

Andrew L. NelsonConsultant

Sandra S. RuppertSenior Associate forResearch and Policy

The mission of the ArtsEducation Partnershipis to demonstrate andpromote the essentialrole of the arts inenabling every studentto succeed in school,life and work in thediverse and globaleconomies and societiesof the 21st century.

FROM ANECDOTE TO EVIDENCE: ASSESSING THE STATUS AND CONDITION OF ARTS EDUCATION AT THE STATE LEVELARTS EDUCATION PARTNERSHIP/NOVEMBER 2006

Introduction

1

By Sandra S. Ruppert and Andrew L. Nelson


A necessary first step is to define a clear set of goalsand purposes for conducting a statewide survey ofarts education. Key aspects of the process aredescribed in the principles below.

1. Frame research goals within a broadereducation context.

In most cases, the survey research is part of a larg-er initiative related to education reform or schoolimprovement efforts. To understand the state con-text, it helps to remember that NCLB directlyaffects education policy and practice at the statelevel. The law also recognizes the arts as a core aca-demic subject, alongside mathematics, languagearts, science, and several other subjects. Further, itrequires a complete education for every child mustinclude rigorous instruction in all core academicsubjects. In addition to the federal law, 44 statesand the District of Columbia have policies thatdefine the arts as “core” or as an academic subjectin statute or administrative code, according to a 50-state analysis conducted in 2005 by the EducationCommission of the States (ECS).

2. Define the problem to be addressed in policy terms.

Determine opportunities and needs in the policyenvironment that will help bring about long-term,systemic change, rather than trying to win just theshort-term battles. Like the vast majority ofstates, Kentucky has policies in place that requireschools or districts to provide arts instruction forits students. One underlying research question

study leaders in Kentucky addressed in the base-line survey of arts education in the state was this:Are students being underserved in the arts? Theanswers allow them to draw inferences about lev-els of access to and participation in arts educationprograms in light of these policies and to makerecommendations. It helps to have an under-standing of how policy is made in the state and toask if there is political support behind the study,and an amenable policy environment in which toconduct it.

3. Gain the support and commitment of state leaders.

Defining the problem in policy terms can make iteasier to enlist the support of state leaders who canbe influential in moving the study forward. Theearly buy-in of state arts and state education enti-ties can be especially important. In Illinois, thestate principals’ association and the board of edu-cation endorsed the research and encouraged sur-vey participation. To ensure the study results havean impact beyond the advocacy level, the overallinitiative should include key leadership who cansteer the findings into the policy realm. Having anauthority, such as a governor, state legislature, orstate superintendent formally initiate the effort canadd legitimacy to the policy findings and futureactions. In Rhode Island, Governor Almondappointed 19 influential leaders from the arts, edu-cation, and business sectors to serve on his TaskForce on Literacy in the Arts and to issue policyrecommendations.

4. Create an organizational infrastructureto guide the work.

Forming a statewide steering committee that canassume “ownership” of the initiative can involve anumber of steps: Make the process of organizingthe study as transparent as possible and constructit with key stakeholders. Invite other organiza-tions, including funders, to become strategic part-ners. Consider involving a government relationsand/or public relations strategist at the beginningof the project. Their early input can be importantto converting the resulting data into effective,

20 Principles to Inform State LevelArts Education Survey Research

The 20 principles are organized according tofour basic research design components: clearresearch goals and purposes, appropriatemethods for data collection and analysis, effectivecommunications strategies, and direct policy andpractice implications.

Clear Research Goalsand Purposes



high-impact, clear, concise, and newsworthy mes-sage points that will resonate with elected officials,policy leaders, and the media. However, there is nosubstitute for the solid foundation a competentand credible staff provides, including a projectdirector as the point person who pulls all thepieces together.

5. Articulate explicit purposes for the study.

Going forward, the stated purposes for the studywill aid in determining the specific types of data tobe collected, how they will be analyzed, and themanner in which they will be reported to appropri-ate audiences. Achieving a sharp focus may requiremaking tough choices; it is unrealistic to expect thestudy to be all things to all people. These are amongthe commonly identified purposes for a statewidesurvey on the status and condition of the arts at theschool or district level:

• Raise awareness — Survey results can serve asthe basis for a statewide advocacy campaign orfor seeking additional support. In Illinois, a stat-ed purpose was to raise awareness among Illi-nois policymakers of the value of arts educationand to advocate for stronger state and local poli-cies, along with corresponding budget appropri-ations, to ensure that arts education is providedto students statewide.

• Monitor the general condition of arts educa-tion — The study can serve as a means forestablishing a “baseline” for tracking and meas-uring progress in the state, in some cases, basedon compliance with NCLB requirements. Theindicators also can be included on school or dis-trict level report cards. Kentucky’s first-everbaseline survey was designed to provide a“snapshot” of arts education and to inform thedevelopment of future programs and partner-ship efforts.

• Promote school or program improvement —In New Jersey, one purpose for the survey wasto provide schools and communities with toolsand resources to strengthen student learning inthe arts. The findings will be used to identifymodel arts programs and to serve as the basisfor establishing a clearinghouse on arts educa-tion where schools can share ideas. In Washing-ton, the foundation that helped support the

research was particularly interested in identify-ing school-based solutions to shared challengesin the funding, staffing, and scheduling of thearts in the day-to-day curriculum.

• Inform policy decisions — In Rhode Island,the governor-appointed task force was explicitlycharged with “mak[ing] policy recommenda-tions on how the arts can have a significanteffect on the educational agenda” of the state.

• Align resources — While not typically a pri-mary purpose, funding sources and levels forarts education and spending patterns over timecan reveal disparities in the level of arts educa-tion that schools are able to provide. This infor-mation can be useful in advocating for the allo-cation of new resources as well as for a betterinvestment of existing resources.

NEW JERSEY

A Well-Defined Purposeand Long-Range Planning


The Music for All Foundation, New Jersey StateCouncil on the Arts, New Jersey Department ofEducation, The Geraldine R. Dodge Foundation,and Playwrights Theatre of New Jersey formed theNew Jersey Arts Education Census Project with aclear purpose: “to gather, evaluate and disseminatequalitative and quantitative data regarding artseducation in the state of New Jersey.” At the pro-ject’s inception, these organizations established afive-goal process for collecting and disseminatingthe data. The goals include: (1) disseminate astatewide survey to all schools; (2) connect thisinformation to other state school and census data;(3) create an online arts education research center;(4) serve as a national model of state arts educa-tion research; and (5) create an ongoing program.By thinking through the collection and distribu-tion of the data before beginning their research,the Arts Education Census Project developed awell-defined purpose that will guide its researchand provide benchmarks that can be used to meas-ure progress.


12 Commonly Used Indicatorsto Assess the Status andCondition of Arts Education• Time/frequency provided for arts instruction

within school schedules

• Number and range of arts course offerings

• Percent of students participating in arts courses

• Number of credits in the arts required for highschool graduation

• Percent of certified or licensed teachers to teacharts education

• Availability of professional development work-shops and teacher planning time

• Frequency of arts-based field trips, residenciesand extra curricular activities

• Presence of designated arts classrooms and useof technology in arts learning

• Evidence of alignment of arts instruction withstate standards

• Presence of documented arts curricula

• Type of assessment tools for measuring studentarts performance

• Amount of school and outside funding for artsprograms

Once the goals and purposes for conducting sur-vey research are established, the next step is todesign the study itself, which can entail determin-ing the scope of work, the methodology to beemployed, and a timeframe and budget for execut-ing it. The conclusions and impact of the study willdepend largely on a carefully crafted and success-fully implemented research design. Some princi-ples that contribute to a well-designed study aredescribed below.

6. Work with an oversight committee to designand monitor the survey.

Some form of oversight committee or advisorygroup offers an important opportunity to involverepresentatives from the education, arts, art educa-tion, higher education, business, government, orcivic communities in the study. Among other func-tions, committee members can provide a “fresh setof eyes” and useful feedback during a pilot phase ofthe survey. Kentucky utilized a project oversightcommittee of state and local partner agencies forthe initial design and construction of the writtensurvey instrument. The committee’s primary focuswas to identify relevant categories for explorationand inclusion in the study. In Washington, a consor-tium of arts educators, school administrators, artsagency personnel, and teaching artists helped todefine key attributes of quality for the arts educa-tion components the researchers would investigate.

7. Seek research expertise early on.

Individuals with expertise in qualitative or quanti-tative research methods can provide invaluableassistance in the development of an appropriateand robust survey instrument. Typically, they alsocan perform sophisticated statistical analyses andassist in the interpretation of results. In Illinois, theresearch firm employed was able to devise an indexthat included measures for ranking arts educationby district and county. Research analysts alsoorganized a presentation of the data. To locate out-side help, prepare a request for proposals (RFP)and distribute it to various research firms that spe-cialize in practice-oriented or applied research.

8. Select the right survey tool and measures for the job.

Take care in identifying the indicators that will beassessed in the survey and, in most cases, trackedover time. For better or worse, it is generally under-stood that “what gets measured is what gets valued.”As shown below, what gets measured tends to bethose indicators that are quantifiable and can beeasily understood and interpreted by a wide audi-ence. They also are more amenable to presentationin tabular form and most allow for school or district

Appropriate Methods for DataCollection and Analysis



KENTUCKY

Achieving a High SurveyResponse RateEven though they were not mandated to do so bythe state Department of Education, 77% of Ken-tucky school districts completed the arts educa-tion survey that was developed by the Collabora-tive for Teaching and Learning with funding fromthe Kentucky Arts Council. The Kentucky Com-missioner of Education distributed the survey toall 176 of the state’s district superintendents viaemail, and superintendents also had the option ofobtaining a hard copy version from the Collabora-tive. To make the survey process as efficient aspossible, superintendents were told in advancewhat information they would need to have avail-able in order to fill out the questionnaire. Twoweeks before the survey deadline, the Commis-sioner sent a reminder email to the superintend-ents, and the survey deadline was then extendedby two weeks to enable telephone calls to a sam-pling of non-respondent districts, which increasedthe response rate by 16%.

level comparisons. Decide the best way to collectresponses — via the web, a paper-based survey, orboth ways. Increasingly, online surveys are becom-ing the norm; if using an online survey bear in mindthat human and technological aspects have to workin tandem. To supplement the quantitative data,consider gathering qualitative data through one-on-one or group interviews using a prepared questionguide or survey protocol to gain insights into peo-ple’s attitudes and perceptions about issues.

9. Identify the correct data sourceto answer questions.

Potential sources of information about the statusand condition of arts education in a state are numer-ous. Who receives the survey and who shouldrespond to it depend on what researchers want toknow. Appropriate respondents for statewide datacollected at the school or district level might includeprincipals, teachers, curriculum specialists, artsconsultants, or superintendents. Although the sur-veys in New Jersey and Washington went to chiefschool administrators and principals, it was recom-mended that they engage school or district arts staffin the process of information gathering. In Illinois,researchers used two slightly different versions ofthe same questionnaire — one designed for princi-pals and the other for superintendents. Given thesimilarity in the findings, however, researchers ques-tion whether the effort involved in analyzing twoseparate data sets was worth the extra investment oftime and resources. Some of the most difficult infor-mation to extract may be arts funding amounts andsources as they often are woven into multiple budg-et line items, or contributed by “soft” resources thatcan change from year to year.

10. Take steps to ensure an adequate response rate.

Among the actions that study leaders can take toincrease the survey response rate are these steps:

• In the introduction to the questionnaire, informrespondents of the reasons for the study andwho will see the results

• Let respondents know what information theywill need to gather in advance in order to com-plete the survey

• Include a glossary that gives clear definitions of

even commonly used terms, like “school day”and “curriculum,” and include detailed instruc-tions for completing the questions

• Address the concerns of respondents up front:Assure them the survey results will not be usedagainst them

• Keep the length of time needed to complete thesurvey short — 20 minutes is considered rea-sonable

• Provide letters of support or other validationfrom state leaders

• Send e-mail reminders or make follow-up callsas the response deadline approaches

• Be willing to apply pressure when needed. Toensure that data were gathered from all 2,400schools in the state, the New Jersey Commis-sioner of Education mandated participation inthe online survey.



WASHINGTON STATE

Gathering Quantitativeand Qualitative DataWhen conducting their statewide arts education sur-vey, researchers for Washington’s Arts EducationResources Initiative developed a methodology thatallowed them to collect both quantitative and quali-tative data regarding the state’s schools. After distrib-uting an online survey to all of Washington’s K-12principals, the researchers employed a collaborative-ly-defined “markers of arts education quality” tool toidentify 32 schools across the state that exhibited keyattributes. During these site visits, they interviewedprincipals, teachers, and arts specialists. The inter-views allowed researchers to gather in-depth infor-mation about specific school-based solutions tocommonly perceived challenges, such as tight build-ing schedules, fluctuating funding resources, and artscurricula documentation processes, which cannot beeasily collected in an online survey. The result of thismethodological approach is a detailed body of evi-dence that not only paints a picture of arts learning inWashington schools, but potentially links schoolswishing to maximize arts education programs withthose that already have created viable solutions.

Audience issues — how, what, and to whom infor-mation gets communicated — play a significant rolein determining the relevance and usefulness of thesurvey results. Following are principles for develop-ing an effective communications strategy that willhelp in reaching target audiences with informationthat makes sense out of oftentimes complex data.

11. Develop a communications plan to disseminatefindings.

Treat the release of survey results like a campaign.Set aside sufficient resources for production andmultiple means of communication with visuals thatmake it easy for elected officials and others to

absorb information. When developing a communi-cations plan, think about how you can graphicallyconvey the information for maximum impact.Often more data are collected than can actually bereported; it is not necessary to report the results ofevery survey question. Determine what is most rel-evant and newsworthy and highlight those findingsin your report and communication efforts. Be cau-tious, however — any graph or data organizer cantilt a story. Consider doing a press conferencelaunch of the results. It not only gets the attentionof policymakers, but it also can be an opportunityfor them and for schools to actively participate inthe study and become a partner in future efforts.

12. Define target audiences.

Study leaders need to plan how they will communi-cate with key messages developed specifically foridentified target audiences. Because most individu-als can absorb only a handful of major findings thatillustrate the status of the arts, create graphic andvisual representations that send a message at aglance. Determine what kinds of indicators willresonate with particular audiences. For example,the sort of information that teachers, arts supervi-sors, superintendents, and principals need to eval-uate and plan for their arts education programsmay be different from what arts-active parentsneed to select a school with an arts-rich environ-ment for their children or what advocates need tomake the case for the arts with state education pol-icymakers. In Washington, two distinct, but com-plementary, documents were created: one designedas an advocacy tool while the other contained in-depth research details and action steps.

13. Tap into existing data collection efforts.

In many cases, the data gathered through astatewide survey of the arts in education can besupplemented with information drawn from othersources. Part of the plan in New Jersey, for exam-ple, is to mesh survey results with existing educa-tional and census databases to create the New Jer-sey Arts Education Information and ResearchCenter. Using specially designed technology, thesystem will have the capacity to sort informationaccording to what people want to know, generateindividual reports, compare schools and districts,and geographically display the levels of arts educa-tion across the state.


Effective CommunicationsStrategies


ILLINOIS

Developing a CommunicationsPlan to Publicize Survey ResultsWhen Illinois Creates, a statewide coalition led bythe Illinois Arts Alliance, began its statewide artseducation survey of Illinois schools, it hired a pub-lic relations firm to develop a communications planthat would allow them to share the findings of thesurvey not only with school leaders and policymak-ers, but also with the media and public at large.After the state superintendents and principalscompleted the questionnaire, a press release sum-marizing the research findings was issued and aseries of press conferences were held throughoutthe state. As a result of the firm’s proactive efforts,stories were placed in almost every major print, tel-evision, and radio outlet, including the Chicago SunTimes and the Chicago Tribune. The pubic rela-tions firm also offered editorial assistance in thepublication of the Illinois Creates’ Arts at the Core:Every School, Every Student report, which cogentlydescribes the survey results.

14. Turn complex data into useful information.

To ensure the findings are easy to understand andinterpret, study leaders need to be able to weavecomplex data together into timely information thatpeople can use to make decisions. In Illinois, anindex was created that scored the schools in eachzip code based on 11 arts education measures.Researchers set up the variables so they could iden-tify disparities in many different ways, includingsocioeconomic, geographic, or racial categories.The findings showed significant disparities wereattributable to size; that is, the number of studentsthe school or the district served, and whether theschool was in a rural or urban setting.

15. Offer comparisons, but exercise caution in passingjudgment.

In presenting statewide results, the “big trends” areoften of most interest to general audiences. However,when tracking geographic disparities within thestate, for example, it is necessary for comparativepurposes to disaggregate the data into smaller units

of analyses, such as at the county, school district,school, or zip code level. Consider carefully the use ofratings and scales when it might lead to making judg-ments about “winners” and “losers:” The news caneffectively alienate particular audiences you want toreach. In Illinois, even though the scores were low forsome districts, study leaders identified places wherethings were working well within each district. Fur-ther, the study results offered recommendations forimproving arts education in the state.


Direct Policy andProgram ImplicationsSound policy depends on the availability of infor-mation that supports good decision making. Thefindings can and should lead to recommendationsthat have direct and actionable implications forpolicy and practice. Following are some ways inwhich data or information generated through astatewide survey can serve as an advocacy and pol-icy tool to effect change for arts education at theschool or district level.

16. Use the results to serve as a wake up call.

Hearing the hard cold facts about the status and con-dition of arts education in the state can be a way tocatch and keep the attention of key constituencies. InIllinois, based on geographic comparisons, the find-ing that the state ranked below national averages senta particularly powerful message to the state’s educa-tion leaders. The state superintendent of educationcalled the survey research a “wake up call” and vowedto use his “bully pulpit” to increase arts education inthe schools. In Washington, the study stimulated anew evidence-based conversation among state lead-ers and educators about how to realize the long-rangeand short-range goals to support arts education cur-ricula, professional development, and assessments.

17. Examine the policy implications.

The value of the survey results depends on how theyare applied to the task of making or interpreting pol-icy. In New Jersey, the findings will gauge how wellindividual elementary, middle, and high schools areadhering to a state mandate to provide a well-round-ed arts education. In Rhode Island, as a result of the


RHODE ISLAND

Using an Arts EducationSurvey as a Springboard forSchool ReformUsing questionnaires to gather baseline arts educationdata was only the beginning of the work of RhodeIsland’s Literacy in the Arts Task Force. After surveyingstate schools, institutions of higher learning, and com-munity organizations, the Task Force shared the surveyresults with these entities, allowing them to view theresults and better understand the status of arts educa-tion in the state and their own work. The surveys and thedissemination of their findings were the catalyst for theformation of the Rhode Island Arts Learning Network.This is a structure that can act as an advocacy organiza-tion to support and connect arts learning in home,school, and community, in order to promote equalaccess to arts learning in and out of school for all RhodeIsland children and youth. The network uses five region-al representatives to move its agenda forward and suc-cessfully advocated for a change in state graduationrequirements that mandates that all students demon-strate proficiency in an art form before graduation.

statewide survey and other activities undertaken bythe Task Force on Literacy in the Arts, study leaderswere able to advocate for and successfully change thearts graduation requirement so that it now applies toall students, not just the college-bound. Additionally,schools in the state of Washington have been advisedof how they can use data generated by the survey toargue for a line item in the district budget or for seek-ing support for arts programs.

18. Have a plan for technical assistance.

If possible, the state department of educationshould allow schools access to the data related tothem and support the schools with needed assis-tance after the research findings are published.Providing this support may require better coordi-nation of programming and allocation of resourcesin conjunction with more effective training andsupport to those who deliver these services. Basedon the 2005 ECS state-by-state analyses, a handfulof states require local districts to assess the arts;however, only Kentucky currently includes assess-ment in the arts as a component of its statewideassessment system. Partners there plan to use thesurvey findings as the basis for the development ofa school-based self assessment instrument gearedtoward improving the delivery and quality of artseducation programs. In Illinois, the state board ofeducation created a grant program for arts educa-tion that will provide school districts with planningdollars to improve and enhance their arts educa-tion programs.

19. Reframe the issues around equitable access to ahigh quality arts education.

New ways of framing the policy issues to focus onfair and equitable access to a high quality and well-rounded education for all students can emergefrom the survey research work. For example, studyleaders in New Jersey wanted to know “who hasaccess and who doesn’t” to arts instruction andlearning experiences in light of the state’s mandateto provide a well-rounded arts education. In RhodeIsland, arts learning is considered an integral com-ponent of the state’s “all kids to high standards”education agenda. Yet, the study found a lack ofequity in physical and programmatic access to artslearning opportunities. In Illinois, researcherslooked at meaningful correlations with drop outrates, attendance patterns, per pupil operating

expenditures, and other indicators to help make apersuasive case for arts education.

20. Build the capacity for implementation andsustainability.

Going forward, study leaders can be instrumental inthe development of new community-based struc-tures designed to implement the recommendationsgenerated by the survey research. The study findingsthemselves can serve as a community building tool.In spite of Rhode Island being arts rich, there was nostatewide coordination of arts learning for childrenand youth across the sectors of home, school, andcommunity. Based on recommendations of the taskforce, the Rhode Island Arts Learning Network wascreated to coordinate the effort. Established as apublic/private partnership, the network facilitatespublic engagement, assists in aligning resources, andfacilitates dialogue and action between and amongits partners. No existing organizations in the statehad the capacity to address these challenges.



20/20 CHECKLIST

Principles for Assessing theStatus and Condition of ArtsEducation at the State Level

The authors gratefully acknowledge the generoussupport of the Spencer Foundation and the impor-tant contribution made by the following individu-als who participated in the June 2006 seminarand/or who provided comments on an earlier draftof this brief: Julie Adrianopoli, Kathryn Beach-ler, Sherilyn Brown, Patricia Cirillo, Karen Gal-lagher, Ann Galligan, Dennis Horn, PeterMcWalters, Robert Morrison, Dale Schmid,Melanie Scofield, Gerri Spilka, Alene Valkanas,and Susy Watts. Special thanks to Richard J.Deasy, director of the Arts Education Partnership,for his substantial contribution to all aspects ofthis work.


ConclusionMost of our current knowledge about the status andcondition of arts education in the nation’s publicschools is drawn from anecdotes rather than empir-ical evidence. However, an increasing number ofstates either have undertaken or are in the processof conducting a comprehensive statewide survey ofthe quality and quantity of arts instruction andlearning experiences in their school districts andschools. This research and policy brief examines therecent experiences of five such states — Illinois,Kentucky, New Jersey, Rhode Island, and Washing-ton — to distill the lessons learned in hopes that itwill spur other states to action. The brief offers 20principles, organized according to four basicresearch design components, which can serve toinform and guide their efforts.

Here are 20 principles — 20 actions to consider —in creating a vision for a state level research surveyto assess the status and condition of arts educationin your state.

• Frame research goals within a broader educa-tion context

• Define the problem to be addressed in policy terms

• Gain the support and commitment of state leaders

• Create an organizational infrastructure to guidethe work

• Articulate explicit purposes for the study

• Work with an oversight committee to designand monitor the survey

• Seek research expertise early on

• Select the right survey tool and measures forthe job

• Identify the correct data source to answerquestions

• Take steps to ensure an adequate response rate

• Develop a communications plan to disseminatefindings

• Define target audiences

• Tap into existing data collection efforts

• Turn complex data into useful information

• Offer comparisons, but exercise caution in pass-ing judgment

• Use the results to serve as a wake up call

• Examine the policy implications

• Have a plan for technical assistance

• Reframe the issues around equitable access to ahigh quality arts education

• Build the capacity for implementation and sus-tainability

References

Acknowledgments

Arts Education Partnership, The Arts and Educa-tion: New Opportunities for Research, Washington,DC: AEP (2004).

Center for Education Policy, From the Capital tothe Classroom: Year 4 of the No Child Left BehindAct, Washington, DC: CEP (2006).

Education Commission of the States, Artscan data-base, Denver, CO: ECS. www.ecs.org/artscan(accessed August 10, 2006).


civic organizations and schools that works to raisearts education awareness to Illinois policymakersand the public at large about the importance of artseducation as a core learning area.

http://www.artsalliance.org/ed_research.shtml

KENTUCKY

In 2005, the Kentucky Arts Council (KAC) con-tracted the Collaborative for Teaching and Learn-ing to conduct the first-ever statewide survey ofarts education in Kentucky schools in order to“establish a set of baseline data which will informthe design of future KAC arts education programsand partnership efforts aimed at making the artsintegral to education in the commonwealth.”

Distributed to the superintendents of all 176 Ken-tucky school districts, the survey was based on theOhio Alliance for Arts Education’s 2001 report,Status of Arts Education in Ohio’s School Districts,and included six major categories of investigation:(1) Teaching, Support Personnel and Parent/Com-munity Involvement; (2) Curriculum and Instruc-t

aep research and policy brief from anecdote to evidence...from anecdote to evidence: assessing the...

Documents