Download - Paul Baumann, Ph.D. Director, National Center for Learning and Citizenship Shelley Billig, Ph.D
LEARNING FROM THE PAST: Using the 2009 Learn and Serve cluster
evaluation to develop the field’s understanding of service-learning
implementation and outcomes
Paul Baumann, Ph.D. Director, National Center for Learning and Citizenship
Shelley Billig, Ph.D. Vice President, RMC Research Corporation, Denver
Susan Abravanel Vice President of Education, Youth Service America (YSA)
Teri DaryCo-Chair, National Coalition for Academic Service-Learning
2012 IARSLCE Annual ConferenceBaltimore, MD
September 25, 2012
INTRODUCTION
Session Goals• Provide an overview of three years of findings
for three participants in the 2009 LSA cohort (NCLC, YSA, Wisconsin DPI)
• Begin to examine how implementation strategy might play a role in program outcomes
• Begin to develop cross-cutting findings.
LSA 2009 COHORTLSA 2009 cohort initially included a number of states and national programs:
• States: AZ, HI, IA, IL, LA, MI, MN, OH, and WI• National Programs: NCLC, YSA (focus on STEM, middle schools, high-poverty schools)
Purpose: Implement and evaluate service-learning and Pre-K 12 schools
Cohort members agreed to use common quasi-experimental evaluation design and use the same survey scales to measure a set of core common outcomes:• Academic engagement• Academic performance• Dropout prevention• Acquisition of 21st century skills
Coordinated evaluation would help to address several shortcomings of S-L research (Billig, 2001; 2008; 2011; Bradley, 2005):• Lack of methodological rigor: Quasi-experimental or experimental design necessary to
establish causality and determine effect sizes• Lack of adequate sample: Large, diverse sample necessary to establish more
generalizable findings across states and various student populations.
RESEARCH QUESTIONS
1. What was the impact of service-learning participation on students?:
a. Academic engagement; educational aspirations; acquisition of 21st century skills; civic dispositions; interest and abilities in STEM; and interest in careers that require STEM skills
b. Academic achievement, dropout rates, attendance, and discipline referral rates
2. What program design characteristics influenced outcomes?3. What is the relationship between each programs’ theory of action
and implementation strategy, and the student outcomes for that program?
4. What conclusions, if any, can we draw by looking across the evaluation findings of the three programs?
METHODOLOGY
• LSA sub-grantees from each state were required to participate if they received a certain level of funding.
• Each recruited a matched-comparison site for every two programs (e.g., one middle school site for every two middle school sites served).
• Matches were based on demographics (roughly the same SES and racial/ethnic percentages) and achievement scores (scores on the state assessment in ELA and math) coming to the school year.
METHODOLOGY
• In year one, a retrospective pre/post-student survey was used; in years two and three, a traditional pre/post-student survey was used.
• Teachers and community members answered a post-only survey.
• Achievement data were collected from the sites or state, including test scores in language arts, math and science, attendance, disciplinary data, and graduation/dropout rates.
SCHOOLS OF SUCCESSNational Center for Learning and Citizenship
Theory of Action:• Guide schools from policy to action, and action to policy
o Encourage high-quality practice and sustainability through policy and district/building leadership
o Encourage policy and district/building leadership through models of high-quality practice.
Expectation for Change:• Changes to district/school policy (broadly conceived)• District/schools will begin new or grow existing service-learning
programs• Increase middle school students’ engagement in STEM related
classes.
SCHOOLS OF SUCCESSNational Center for Learning and Citizenship
Context:• Nine middle schools
o CA, KY (3), MI, MS, NY (2), PAo Two Charter Schools (MI, PA)o Three KY schools, all in one districto Mixture of rural and urban
• All high poverty ( > 50% FARMS)• Some high performing, some low performing • Several experienced with service-learning (CA, PA); the
remainder beginning with service-learning.
SCHOOLS OF SUCCESSNational Center for Learning and Citizenship
Implementation Strategy:• Role of leadership
o Recognition of leadership at multiple levels of the schoolo Worked primarily with leaders from district/building level, as well as key teacher
leaders.
• Professional developmento Three days of training annuallyo Focused both on implementation (how to do) as well as policy (how to encourage, lead,
fund, etc.).
• Sustainabilityo Focus on five elements that support sustainability:
Vision and leadership Curriculum and assessment Community and school partnerships Professional development Continuous improvement.
SCHOOLS OF SUCCESS FINDINGS
Service-Learning Quality Mean
SD
Link to Curriculum 3.40 .45Partnerships 3.23 .50Meaningful Service 3.54 .46Youth Voice 3.33 .61Respect for Diversity 3.25 .63Reflection 3.20 .47Progress Monitoring 3.08 .34
Teacher Perceptions of Service-Learning Quality in Their Classrooms (N = 6)
STUDENT RATINGS OF QUALITYService-Learning Quality N Mean SDOverall quality 285 3.02 .57I feel that my service-learning activities were meaningful. 253 3.13 .68My teacher made sure we linked service-learning activities to classroom subjects such as English, math, and/or science.
256 3.18 .72
The skills that I learned from my service-learning activities will be useful to me in the future.
246 3.11 .75
My service-learning activities were important to me. 248 3.02 .74I helped make decisions about my service-learning activities. 255 3.02 .70I helped come up with ideas for my service-learning activities. 261 3.00 .80
During my service-learning activities, we were expected to show respect for other people’s opinions.
255 3.22 .68
My class was asked to identify specific things we had learned during our service-learning activities.
241 3.00 .74
My class talked about several different ways to solve neighborhood or community problems.
237 2.85 .90
IMPACTSService-Learning Comparison
N
PostsurveyMean N
PostsurveyMean Significance
Academic Engagement 311 2.87 54 2.84 .673
Academic Competence 311 3.46 54 3.50 .533
Aspirations 299 3.62 54 3.74 .104
21st Century Skills 303 3.11 54 3.08 .603
Community Engagement 286 2.62 51 2.58 .523
STEM Content Area Interest and Abilities
292 2.95 53 2.94 .894
STEM Career Interest 274 2.72 52 2.67 .633
TEACHER PERCEPTION OF IMPACTSImpact area: N Mean SD
Improvement in academic performance. 6 3.17 .41Increased engagement in school. 6 3.17 .75Improved school attendance. 6 3.00 .63Improved social skills. 6 3.33 .52Increased awareness of community issues. 5 3.40 .55Increased sense of belonging to the community. 6 3.33 .52Increased commitment to working on social or
community problems.5 3.40 .55
Increased belief that they can make a difference. 5 3.20 .84Increased ability to work with others with diverse
backgrounds.6 3.33 .52
Increased respect for others. 6 3.33 .52Increased empathy and desire to help. 5 3.40 .55Increased problem-solving skills. 6 3.33 .52Increased leadership skills. 6 3.17 .75Increased ability to collaborate with others. 5 3.40 .55Improved ability to express ideas. 5 3.40 .55Improved career skills. 5 3.40 .55
STUDENT PERCEPTIONS OF SKILLS ACQUIRED
N=319
Student Perceptions of Skills Acquired Through Participation in Service-Learning (N = 319)
Tutoring skills
Career awareness
Science skills
Reading skills
Writing skills
0 20 40 60 80 100 120 140 160
32
80
87
101
105
123
133
135
138
Number of Respondents
MODERATORS OF OUTCOME
Moderator Outcomes Moderated df F SignificanceStudent Ratings of Program Quality
Academic Engagement
1, 278 12.309 .001**
Academic Competence
1, 278 12.934 .000**
21st Century Skills 1, 273 7.047 .008**
Community Engagement
1, 264 5.163 .024*
*p < .05, **p < .01.
Teachers and classes in middle schools
19 states most seriously challenged by HS graduation rates
≥50% free/reduced lunch demographic
Target:
YSA STEMESTER OF SERVICE
Training and resources
Unique partnership opportunities
Ongoing personalized consultation
Networking
Rigorous evaluation component
WHAT IS A STEMESTER OF SERVICE?
YSA Support:
Focus on incorporating science, technology, engineering or mathematics content learning
Service project targeting environmental or disaster preparedness issues
Career and workforce readiness component
A continuous Semester of Service™ strategy
Content:
WHAT IS A STEMESTER OF SERVICE?
19
SEMESTER OF SERVICE™ Integrated service and learning activities
12-to-14 week “semester”
National/international service days
Design based on the “IPARD/C” model
www.YSA.org/resources
Measure
Service-Learning Students Comparison Students
Signif
Effect Size
Cohen’s d)N
PresurveyMean
PostsurveyMean N
PresurveyMean
PostsurveyMean
Academic Engagement
1,049 2.95 2.99 348 2.86 2.82 .033* .19
Academic Competence
1,048 3.45 3.46 348 3.42 3.40 .297 --
21st Century Skills
1,030 3.09 3.14 343 3.09 3.07 .044* .11
Civic Dispositions
1,013 3.12 3.15 337 3.02 3.05 .841 --
Aspirations 1,003 3.48 3.50 326 3.50 3.52 .946 --STEM interest in courses
1,007 3.03 3.06 341 2.97 2.86 .000***
.18
STEM career 935 2.88 2.91 310 2.89 2.76 .000***
.18
4-point scale where 1 = Strongly Disagree, 2 = Disagree, 3 = Agree, and 4 = Strongly Agree. *p < .05, ***p < .001.
YSA FINDINGS
Student Perceptions of Skills Acquired Through Participation in Service-Learning (N = 1,055)
Tutoring
Career Awareness
Work Experience
Computer
Job
Reading
Writing
Math
Science
0 5 10 15 20 25 30 35 40 45 50
9.2
18.9
26.5
29.5
31.8
34.3
34.6
40.6
44.3
Percentage
Skill
s
YSA FINDINGS
ModeratorOutcomes Moderated F Significance
Effect Size (Cohen’s d)
Student Ratings of Program Quality
(Positive Moderator) Academic Engagement
243.16 .000*** 1.04
(Positive Moderator) Academic Competence
178.89 .000*** .91
(Positive Moderator) Aspirations 74.23 .000*** .59
(Positive Moderator) 21st Century Skills
253.01 .000*** 1.06
(Positive Moderator) Civic Dispositions
285.34 .000*** 1.13
(Positive Moderator) STEM Interest in Courses
162.12 .000*** .86
(Positive Moderator) STEM Interest in Career
119.626 .000*** .72
Moderators of Outcomes: Student Ratings of Program Quality ***p < .001.
[1]
YSA FINDINGS
You can see from the previous slides that high-quality service-learning is very effective… but regular service-learning has a significant but weak effect.
Most of our field has a .03 effect size… much too low.
How can we improve?
YSA FINDINGS
Measure
Service-Learning Students Comparison Students
Signif
Effect Size
Cohen’s d)N
PresurveyMean
PostsurveyMean N
PresurveyMean
PostsurveyMean
Academic Engagement
1,049 2.95 2.99 348 2.86 2.82 .033* .19
Academic Competence
1,048 3.45 3.46 348 3.42 3.40 .297 --
21st Century Skills
1,030 3.09 3.14 343 3.09 3.07 .044* .11
Civic Dispositions
1,013 3.12 3.15 337 3.02 3.05 .841 --
Aspirations 1,003 3.48 3.50 326 3.50 3.52 .946 --STEM interest in courses
1,007 3.03 3.06 341 2.97 2.86 .000***
.18
STEM career 935 2.88 2.91 310 2.89 2.76 .000***
.18
4-point scale where 1 = Strongly Disagree, 2 = Disagree, 3 = Agree, and 4 = Strongly Agree. *p < .05, ***p < .001.
Student Perceptions of Skills Acquired Through Participation in Service-Learning (N = 1,055)
Tutoring
Career Awareness
Work Experience
Computer
Job
Reading
Writing
Math
Science
0 5 10 15 20 25 30 35 40 45 50
9.2
18.9
26.5
29.5
31.8
34.3
34.6
40.6
44.3
Percentage
Skill
s
ModeratorOutcomes Moderated F Significance
Effect Size (Cohen’s d)
Student Ratings of Program Quality
(Positive Moderator) Academic Engagement
243.16 .000*** 1.04
(Positive Moderator) Academic Competence
178.89 .000*** .91
(Positive Moderator) Aspirations 74.23 .000*** .59
(Positive Moderator) 21st Century Skills
253.01 .000*** 1.06
(Positive Moderator) Civic Dispositions
285.34 .000*** 1.13
(Positive Moderator) STEM Interest in Courses
162.12 .000*** .86
(Positive Moderator) STEM Interest in Career
119.626 .000*** .72
Moderators of Outcomes: Student Ratings of Program Quality ***p < .001.
[1]
WISCONSIN DEPARTMENT OF PUBLIC INSTRUCTION
Theory of Action:• Create infrastructure of support
o Build capacity for quality and sustainability through district infrastructure and leadership
o Support quality practice through cascade professional development.
Expectation for Change:• Districts will infuse service-learning as a strategy to achieve
district goals• District will build an infrastructure of support and
leadership to guide practice in local schools.
WISCONSIN DEPARTMENT OF PUBLIC INSTRUCTION
Context:• Up to 10 districts funded per year for a maximum of 6
yearso Up to $20,000 grants per yearo Fundable expenses must support primarily infrastructure,
leadership, curriculum integration, and professional development.
• Geographic and demographic diversity• Collaborative network of support.
SCHOOLS OF SUCCESSNational Center for Learning and Citizenship
Implementation Strategy• Leadership
o Build capacity for leadership at teacher, school, and district levelso Increase capacity for teachers to serve as leaders within their
school/district and at the state level• Professional development
o Initial training by DPI during first semester of fundingo Ongoing professional development through in-person and online
environments• Sustainability
o All grants must demonstrate progress on five elements that support sustainability
Vision and leadership Curriculum and assessment Community and school partnerships Professional development Continuous improvement
Wisconsin Findings
• Will add this week.
Academic Engagement (7 items) GRADES 3-5 Pre sd Post sd Dif .003
Service-Learning 3.43 0.56 3.26 0.54 -0.17 Comparison 3.25 0.48 3.25 0.45 0.00
Academic Competence (7 items)
.002
Service-Learning 3.19 0.51 3.13 0.48 -0.06 Comparison 3.14 0.51 3.11 0.48 -0.03
Community Attachment (3 items)
.009*
Service-Learning 3.07 0.65 3.01 0.62 -0.06 Comparison 3.00 0.64 2.96 0.62 -0.04
Civic Skills (6 items) .011*Service-Learning 3.06 0.59 3.07 0.55 0.01 Comparison 3.08 0.56 3.01 0.56 -0.07
Civic Dispositions (4 items)
.001
Service-Learning 3.32 0.55 3.31 0.59 -0.01
Comparison 3.33 0.51 3.30 0.53 -0.03
N
Beginning of Year
End of Year
Average Difference g
2Mean SD Mean
SD
Academic Engagement 2.87 0.48 2.89 0.52 0.02 .001
Academic Competence 3.16 0.52 3.14 0.54 -0.02 .001
Community Attachment 2.44 0.51 2.50 0.53 0.06.014***
Civic Dispositions 2.95 0.56 3.01 0.58 0.06.015***
Respect for Diversity 3.26 0.58 3.34 0.53 0.08.019***
Civic Skills: Communication 2.94 0.52 3.00 0.49 0.06
.012***
Civic Skills: Leadership ( 2.93 0.62 3.00 0.60 0.07.015***
Civic Skills: Problem Solving 2.87 0.51 2.93 0.50 0.06
.013***
Service-Learning Engagement --- --- 2.89 0.69 --- ---
Service-Learning Quality ( --- --- 2.87 0.55 --- ---
2.502.44
2.382.41
1
2
3
4
Pretest Posttest
Service-Learning
Comparison
Changes Over Time in Service-Learning andComparison Students Ratings of Community Attachment (N = 1,174)
Frequency PercentageReading Skills 332 33.3
Writing Skills 387 38.8
Math Skills 225 22.5
Science Skills 203 20.3
Computer Skills 280 28.1
Tutoring Skills 165 16.5
Career Awareness 404 40.5
Job Skills 488 48.9
Work Experience 590 59.1
Percentage of Service-Learning StudentsReporting Skill Gains, Grades 6-12 (N = 823)
QUESTIONS FOR DISCUSSION
• What program design characteristics influenced outcomes?
• What is the relationship between each programs’ theory of action and implementation strategy, and the student outcomes for that program?
• What conclusions, if any, can we draw by looking across the evaluation findings of the three programs?
• What other questions do you have for us or for discussion here or in the future?
THANK YOU!
• To continue this conversation, please use the conference proceedings.
• Please contact us if you have additional comments or questions.– Paul Baumann: [email protected]– Shelley Billig: [email protected]– Susan Abravanel: [email protected] – Teri Dary: [email protected]