Data Collection and Evaluation to Promote Student Learning

PIRC Annual Conference Washington, DC August 3, 2011

Ron Mirr Zena Rudo

RM Consulting SEDL Iowa City, IA Austin, TX rmirr@mac.com zrudo@sedl.org

Participants will learn... ü Strategies to best showcase

PIRC accomplishments using data

ü Linking with student-level data

ü Action steps for using your data to support family and community engagement

Basics for Telling Your Story

•  Have quality data to describe what you did.

•  Be able to show how well you did it.

•  Know what factors influenced what you did.

•  Support the outcomes and benefits of what you did with data.

“Collecting data is only the first step toward wisdom, but sharing data is the first step toward community.”

Henry Louis Gates Jr. As used in the IBM/Linux Commercials

Basics for Telling Your Story

•  Include information from each:

-  Program delivery

-  Stakeholder satisfaction

-  Program outcomes

•  Be concise and identify highlights.

Basics for Telling Your Story

•  Use different perspectives to tell your story.

•  Use charts/graphs/photos/vignettes.

•  Grab the reader/listener with some emotion.

•  Tailor to your audience

Quality Data •  Accurate

•  Useful

•  Secure

•  Timely

•  Accessible

•  Understandable

•  Actionable

•  Meaningful

When it is a process that is an integral part of the learning environment.

Using Data to Tell Your Story

•  Tell a story about your PIRC’s services

•  Tell a story about what your PIRC has done

•  Tell the story about what your PIRC has accomplished

Your Story Is Only As Good As What You Collect •  What gets measured gets done.

•  If you don’t measure results, you can’t tell success from failure.

•  If you can’t see success, you can’t reward it.

•  If you don’t recognize failure, you can’t correct it.

•  If you don’t find successes and failures, you can’t learn and move forward.

What Did You Do? Counts of who you served:

–  Type (parent, teachers, administrators) –  Ethnicity –  Socio-economic status –  Location

Counts of the services you delivered: –  Type (workshops, newsletters, materials) –  Intensity (one time workshops, multi-session

programs)

Good

Tip: Don’t inflate counts by duplicating service recipients.

How Well Did You Do It? Quality of services delivered

–  Trainer/participant ratio

–  Participants report on quality

–  Use of evidence-based materials

Satisfaction with services –  Participant report on satisfaction

Good

Tip: Compare responses across participants by different characteristics, (i.e., role, ethnicity, location, and child’s age).

What Not To Do With Data

Source: USA Today.com. Reprinted by SEDL with permission of USA Today. All rights reserved.

Improved road conditions

Better-timed traffic signals

More highway lanes

Reduced construction delays

Improved signage

What Factors Influenced Your Work?

Internal –  Characteristics – Behavior

–  Communication – Attitude

External –  Environment – Culture

–  Situation – History

Better

Tip: Benchmark best practices to other successful programs.

Are Recipients of Your Services Better Off?

Changes that can happen more quickly –  Knowledge – Skill level

–  Awareness – Attitude

Changes that take longer to achieve –  Behavior

–  Condition

Better

Tip: Compare to similar groups not receiving services.

Are Students Better Off?

Establish a link between parent engagement and student outcomes:

–  Knowledge – Skill level

–  Awareness – Attitude

–  Behavior – Relationships

Best

Tip: Emphasize 3-4 strong results; however, don’t use terms that overstate the effect you made.

The Wyoming PIRC Story

Handout is available at http://www.ronmirr.com on the downloads page.

Collect student data based on what is known about student’s learning attributes from:

–  Kathy Hoover-Dempsey & Howard Sandler’s Model of Parental Involvement

–  Nebraska PIRC & 21st CCLC program study

–  Nancy Hill’s Project PASS and Project Alliance

Student Learning Attributes (Hoover-Dempsey)

•  Students believe they can learn –  I can do even the hardest homework if I try. –  I can learn the things taught in school. –  I can figure out difficult schoolwork.

•  Student want to learn because it’s important to them and not driven by external rewards –  I want to understand how to solve problems. –  I like to look for more information about school subjects. –  I want to learn new things.

Hoover-Dempsey, K.V., et. al. (2005). Why do parents become involved? Research findings and implications. Elementary School Journal, 106(2), 105-130. Kathy Hoover-Dempsey is currently at Vanderbilt University, Family-School Partnership Lab

Student Learning Attributes cont.

•  Students know how to manage their own learning –  I ask myself questions as I go along to make sure my homework makes sense

to me. –  I try to figure out the hard parts of my schoolwork on my own. –  I go back over things I don’t understand. –  I try to find a place that makes it easier to do my homework.

•  Students know how to ask for help, especially from teachers –  I can get along with most of my teachers. –  I can go and talk with most of my teachers. –  I can get my teachers to help me if I have problems with other students. –  I can explain what I think to most of my teachers. –  I ask the teacher to tell me how well I'm doing in class.

Parent-Child Conversations (Nebraska PIRC & 21st CCLC study)

Parent-child conversations that focus on learning – I talk to my family about my homework

– I talk to my family about what I'm learning in school.

Aspirational Items (Nancy Hill)

Current and future expectations –  I plan to continue my education after high school.

–  My family expects me to do well in school.

Hill, N. E., & Chao, R. K. (2009). Families, schools, and the adolescent: Connecting research, policy, and practice. Nancy Hill is currently at the Harvard Graduate School of Education, Radcliffe Institute of Advanced Studies.

Sample Elementary & Middle School Scores Item   Grade    

5  Grades  6-­‐8   D

I  plan  to  con3nue  my  educa3on  a)er  high  school   97%   76%   -­‐21%  

I  talk  to  my  family  about  my  homework   68%   48%   -­‐20%  

I  talk  to  my  family  about  what  I'm  learning  in  school.   77%   48%   -­‐29%  

I  like  to  look  for  more  informa3on  about  school  subjects.   68%   48%   -­‐20%  

I  go  back  over  things  I  don’t  understand.   87%   68%   -­‐19%  

I  can  explain  what  I  think  to  most  of  my  teachers.   78%   56%   -­‐22%  

I  ask  the  teacher  to  tell  me  how  well  I'm  doing  in  class.   61%   51%   -­‐10%  

How WY PIRC Uses Its Data Intensive 2 year program (Solid Foundation)

–  Baseline student data –  Use student data to prioritize areas of focus –  Monthly team meetings (parents and educators) –  Integrate data theme into all activities

•  Compact •  Parent Nights •  Conferences •  Home Visits

–  Reassess student data each year in the fall to make mid- course corrections

The CA PIRC (CABE) Story Project INSPIRE: Family-School Community Leadership Development Program

Level 1- Awareness: Critical information impacting their children (12 modules)

Level 2 – Mastery: In-depth understanding of critical information impacting their children (12 modules)

Level 3 – Expert: Development and re!nement of leadership knowledge and skills (16 modules )

Project INSPIRE Research Study Primary Research Questions:

1.  Is the intensive parent leadership program effective in helping parents increase the type, frequency, and intensity of support they provide their child’s learning?

2.  What impact, if any, does the type, frequency, and/or intensity of parent engagement impact student academic achievement?

Students’ ELA Scores Increase

Schools In Need of Improvement

Non Treatment Schools increase

Treatment School Increase

CA-PIRC Increase

Total Increase for Project INSPIRE Students

5.7 points Additional 4.8 points

Additional 8.8 points

13.6 points

Students’ Math Scores Increase

Schools In Need of Improvement

Non Treatment Schools increase

Treatment

School Increase

CA-PIRC Increase

Total Increase

for Project INSPIRE Students

4.9 points Additional 9.9 points

Additional 17 points 26.9 points

Two Year Trend in Students’ ELA and Math Achievement Gains

Parents Received Level 2 Services

CA  Standards  Test   2009`  Increase   2010  Increase  

English  Language  Arts   12.8   13.8  

MathemaKcs   18.5   26.6  

CA PIRC – CABE Moving the Field Forward

What We Learned:

1. Knowledge is Power – parents need information to participate fully in their children’s schooling.

2. Parents participate when they can work collaboratively with the school(s).

CA PIRC – CABE Moving the Field Forward

What We Learned:

3.  Parents are more willing to engage in learning about schooling when they are respected and valued as partners.

4.  Parental engagement does not happen nor is it sustained when there is no shared common language or structure at the school/district.

•  Describe two accomplishments that make your program high quality.

•  Describe two accomplishments that make your program successful.

•  What data do you have to show you’ve made these accomplishments?

•  What data challenges do you face to be able to showcase these accomplishments?

Your Data, Your Story

Told this story many times, but not with student data.

Haven’t yet been able to tell this story.

Told this story using student data.

Done That – What’s Next

Which best describes your PIRC story?

Simple Rules for Getting Student-Level Data

•  Know exactly what you want

•  Have a common understanding

•  Set goals

•  Know what data are available

•  Collect multiple forms

Simple Rules for Getting Student-Level Data

•  Get public data first

•  Find the right resource

•  Be polite, but persistent

•  Submit a written request

•  Use your connections

Using Your Story to Move Forward •  Spread the word

•  Start conversations

•  Assist families to understand

•  Ask questions

•  Get involved

•  Get additional funding

•  What data will be included and how? •  What are the key messages your data support? •  Who will you tell?

•  How will you tell your story?

The Mechanics of Telling Your Story

What Not To Do When Telling Your Story

Source: CartoonStock.com. Reprinted by SEDL with permission of CartoonStock.com. All rights reserved.

What Some May Say In Their Story

Retrieved on July 27, 2011 from: http://www.guy-sports.com/humor/jokes/jokes_academic.htm

"IT HAS LONG BEEN KNOWN"... I didn't look up the original reference. "IT IS GENERALLY BELIEVED THAT"... A couple of others think so, too. "A CAREFUL ANALYSIS OF OBTAINABLE DATA"... Three pages of notes were obliterated when I knocked over a beer glass. "IT IS HOPED THAT THIS STUDY WILL STIMULATE FURTHER INVESTIGATION IN THIS FIELD"... I am pleased to feed you this rubbish.

Using data to see what works and what does not work in achieving goals.

Data has no meaning on its own. Meaning is a result of human interaction with data.

If I’m Going to Remember One Thing From Today’s Session…

Questions and Answers