early learning data institute - wsa head start and eceap · 2014-04-03 · early learning data...

Early Learning

Data Institute Using the Cycle of Inquiry and

Action For Program Improvement

www..esd113.org ESD 113

Your partner for learning solutions

Welcome and Introductions

2

Overview of the Day

3

Part 1

The big picture

Getting Ready and Issues

Part 2

Application to your context

Plan and Evaluate

Diagnosing Causes

Digging Deeper

Today’s Objectives

1. Learn concepts related to:

a) Team readiness for use of data

b) Creating focusing questions

c) Gathering, displaying and analyzing data

d) Understanding underlying causes

2. Apply these concepts and related tools to real-life data and

consider the technical assistance/support you might provide.

4

To download materials:

esd113.org/datacoach

Early Learning Institute - link

on left

-or-

esd113.org/page/1529

5

Http://esd113.org/datacoach

Your Expectations???

What are you hoping to take

away after today? 6

Norms Promoting a spirit of inquiry

Paraphrasing

Probing for specificity

Paying attention to self & others

Presuming positive intentions

7

1.0 Getting Ready –

8

Don’t Be Puzzled!

9

Where We Are Going

10

The Cycle of Inquiry

and Action

Getting Ready

Identify an Issue

Understand the Issue

Diagnose Causes

Plan and Take Action

Evaluate Results

Getting Ready

Why?

Who?

How?

What?

Then What?

11

Self-Assessment

12

Debrief with Team, & Questions?

13

2.0 Identify Issues

14

Identify Issues: Overview

1. Formulate Initial Questions

2. Identify Potentially Useful Data

15

Identify Issues: Scenario

TOPPENISH EARLY LEARNING

CENTER

16

Identify Issues: Questions

1. Your response to the issue(s)

underlying the question is under

your direct control.

2. Significant in terms of helping

children.

17

Five year plans

Outcomes

for children Program

Management

Community

Involvement Family

Human Resources

Compensation

Fiscal Controls


Example Focusing Question:

Which domain of kindergarten

readiness provides the greatest

“opportunity” for growth?

19

Identify Issues: Influences

1. External: Expectations from

program review

2. External: Research/Best Practices

(National Training and Technical

Assistance)

3. Internal: Prior history/experience

4. Internal: Data review/analysis

20

Identify Issues: Overview

1. Formulate Initial Questions

2. Identify Potentially Useful Data

21

Identifying Issues: Data

22

1.

1.

2. 3.

4.






23

2.5 Sources of Data

24

To provide the team with a

protocol to follow when

attempting to find data to

answer a focusing question.

10 minutes

Debrief with Team, &

Questions?

25

3.0 Understand Issues

26

Understand Issues: Overview

1. Display data for understanding

2. Analyze the data

27

3.1 Data Display Quality Checklist

28

To provide the team with a

protocol to follow when

developing data displays.

Variable based on the

complexity of the data set and

the story to be told with the

data.

Understand Issues: Data Displays

Using the materials we have

provided and create a mock-

up of an effective data

display.

31

Understand Issues: Overview

1. Display data for understanding

2. Analyze the data

32

Understand Issues: Data Carousel

33

http://successatthecore.com/leadership-development/video.aspx?v=9

Factual Observations

A statement about what the data say without

any interpretation. Factual observations

are the first step in the data analysis

process and they provide the basis for

making sound inferences.

•Not all observations are “factual”

•Observations can be colored by biases,

assumptions, prejudice

34

Understand Issues: Data Carousel

Now, you get to experience a

mini-carousel. Review each of

the data display mock-ups, and

record your objective statements

in 3 categories:

strengths

concerns

additional data needed

35

Debrief with Team, & Questions?

36

Data Institute

Lunchtime!

37

4.0 Diagnose Cause(s)

38

Diagnose Causes

1. Identify and articulate the major

problem behind an underlying

issue.

2. Identify root cause or causes of a

problem.

39

Diagnose Cause(s): Questions





40

Diagnose Cause(s): Observations

New information

41

Diagnose Cause(s): Protocols

As a team, review the 20 reasons

protocol and apply it to your group’s

concern:

42

4.3A Significance &Control Protocol

43

4.3B Inter-relationship Protocol

44

The Problem of Practice

It is the adults who create and maintain learning

opportunities for the children they serve. The outcomes

that children experience, therefore, are determined by

the practices of adults in the learning environment.

45

Identifying the Problem: Two Sides of the Same Coin

Problem Statement

Expressed in terms of what children are having difficulty with.

E.g. Our children are not at age level in math skills.

Problem of Practice

Expressed in terms of the teaching, administrative, organizational, and structural practices related to the learner-centered problem.

E.g. Our staff do not have deep understanding of effective instructional stratgies in mathematics.

Debrief with Team, &

Questions?

47

5.0 Plan and Take

Action

48

Plan and Take Action

1. Clearly Describe the ‘End State’.

2. Plan to implement strategies.

49

S.M.A.R.T End States (goals)

SPECIFIC

MEASURABLE

ATTAINABLE

REALISTIC

TIMELY

54

5.7 Writing End Statements

(goals)

55

Elements:

What will change?

Who will it impact?

By how much?

By when?

SMART Goal Mad-Lib

To (insert change here) as measured by

(insert measurement instrument here) from (insert

current performance here) to (insert desired

performance here) by (insert date here).

56

Selecting Best Practice Strategies

59

HIGH IMPACT STRATEGIES

5.7 Developing an Action Plan

61

5.7 Developing an Action Plan

62

Develop a plan that will guide the specific steps of the strategy.

About 30 minutes to introduce and additional time to complete.

Purpose:

Time:

6.0 Evaluation

63

Evaluation

1. Identify data collection elements before you start implementation of the action plan(s).

2. Two parts:

Short term monitoring (formative) of the strategies

Long term plan goal evaluation (summative)

64

Quality of Implementation

.

65

“Science of implementation”

Billions spent on research-

based best practices

Quality of Implementation

From: Fixsen, D. L., Naoom, S. F., Blase, K. A., & Frances, W. (2007). Implementation: The

Missing Link Between Research and Practice. The APSAC Advisor . 66

“The quality of the implementation process

is the key link between sound research

and desired outcomes.”

“A sound intervention isn’t enough, those who will make

It happen and the systems that they work within

must be shaped to support implementation.”

Data Institute Wrap Up

67

A metaphor

68

Final Questions

69

To download materials

esd113.org/datacoach

Then click Early Learning

Institute link on left

-Or-

esd113.org/page/1529

70

Http://esd113.org/datacoach

Thank You

Dr. Dana Anderson Superintendent

Capital Region ESD 113

[email protected]

Sandy Nelson

360-464-6700

Assistant Superintendent, Early Learning

Capital Region Education Service District 113

[email protected]

71

Early Learning Data Institute

Day 2 Renewing use of the

Cycle of Inquiry

www..esd113.org

ESD 113 Your partner for learning solutions

Welcome and Session Overview

• Checking in with your use of the inquiry cycle

• Monitoring your action plans

• Leaders and change…

• Open space discussion – ‘I’d like to talk about…’

Norms

• Promoting a spirit of inquiry

• Putting ideas on the table

• Paying attention to self & others

• Presuming positive intentions

3

Your Expectations???

What do you hope to gain from our time together?

4

As a reminder

What have you done with the cycle of inquiry?

• In job (or role)-alike groups

– Directors

– Content experts

– Fiscal/Data

• How did you apply the cycle of inquiry?

• What did you learn?

Sharing out…

• Drop one sharing protocol----

– Select a reporter

– 3 ideas on notecard

– Cross out ideas if previously shared

– Sit if all covered

Time to move…

• Select a stage in the cycle of inquiry

• Prepare to share specific experiences/resources/questions related to your selected stage.

How did it go?

• What went well at this stage?

• What ‘opportunities’ presented at this stage?

• When (not if) you engage in the cycle of inquiry again, what will you do differently?

Questions and ?

• As a result of your discussion, what questions still remain?

Monitoring

So you’ve got a plan…

• How do you align your strategies to your goals?

• How do you monitor implementation?

• How do you adjust your plans?

Five year plans

Outcomes for children

Program Management

Community Involvement Family

Human Resources

Compensation

Fiscal Controls

Tying it together

Goal Area:

If we were to…

If we were to…

If we were to…

Then we would see…



Ultimately we would achieve

Strategies Short-term goals Long-term goals

Monitoring Dashboard

Effective Progress Monitoring

• Keep, drop, and create…

• Protocols for monitoring plan implementation

• Ends vs. Means

K

Kn

ow

an

d A

pp

ly 1. Every organization has a culture.

2. Either you will manage your culture or it will manage you.

3. Leaders must create the needed culture.

Culture

17

Results

Actions

Beliefs

Experiences

1. Define New Results

2. Identify New Actions

3. Identify New Beliefs

4. Provide New Experiences

Results Pyramid

18

19

• What do people need to stop doing? • What do they need to start doing? • What are the strengths that you want people to

continue doing?

20

“Never mistake motion for action.” Ernest Hemingway

If we want different results people have to act differently.

K

Kn

ow

an

d A

pp

ly

What Results are you looking for in your program?

• With a partner discuss:

–What do you want people to do differently?

–What barriers exist right now that keep that from happening?

21

K

Kn

ow

an

d A

pp

ly

Results

Actions

Beliefs

Experiences

- Connors, Roger; Smith, Tom (2011-01-04). Change the Culture, Change the Game

The Results Pyramid

22

37

Leaders must identify: • Existing beliefs that are hindering achievement

of new results • New beliefs that are needed to help the

organization move forward.

You cannot change beliefs simply by asking people to do it. Leaders must create experiences that will convince people to think differently.

38

Experiences create beliefs that drive actions that, in turn, produce results.

• Leaders provide experiences every day. • Leaders can be mindful of well planned

experiences targeted at desired beliefs.

25

You must be the change you wish to see in the world. Mahatma Gandhi

Results

Actions

Beliefs

Experiences

What results are we looking for in our program? What experiences will impact beliefs that may be interfering with our progress?

26

Open Space

Initiation

Marketplace

Wrap-up

27

The Marketplace

1. Data quality issues

2. Engaging others in process

3. Building staff capacity

4. Supporting change

5. Creating a vision

6. Monitoring plans

7. Managing data

8. Alignment of resources

My Partner For Learning Solutions 28

One rule of engagement

My Partner For Learning Solutions 29

I would like to talk with others about…

30

That’s a wrap

• What is one thing you will take away from today’s session?

• What will you do to revisit and manage your cycle of inquiry?

Data Institute

Day 1

1

Early Learning Data Team Toolkit 1. Getting Ready: Program Data Team Self-Assessment

1.4 – Agency or Program Data Team Self-Assessment Directions: This survey will yield the most information if an agency or program data team completes it as a group. To do so, follow the instructions below.

1. Several days prior to a meeting of the data team, print each page of this self-assessment (including the rubric on the following page) and provide a full copy to each member of the group.

2. In preparation for meeting together, each member of the team should individually complete the survey, assigning a rating from the rubric to each indicator.

3. As a group, discuss each page of the survey and agree on a rating for each indicator. It is not necessarily best to average the individual scores to get this final rating. If responses among individuals vary widely, engage in a discussion about which rating best represents the level of practice. This discussion can help the team begin the hard work of developing a common understanding of the work.

4. As a team, reach consensus on the evidence that supports the level at which each function is currently being accomplished. Those areas with relatively little evidence that the function is currently being discharged will be priority issues for the district data team to address in the next component of the Toolkit – Identify Issues.

Rubric for Assessing Each Indicator:

No Knowledge

Respondent/team has no knowledge about this indicator and cannot provide a judgment about its existence in the agency or program.

No Evidence There is no evidence that this indicator is in place within the agency or program.

Some Evidence

There is some evidence of this indicator in the agency or program, but the evidence indicates that the practice is far from standard procedure and has clear room for improvement in both quality and frequency.

Clear Evidence

This indicator has clear evidence of existence in the agency or program and is consistently practiced in many places. There is room for improvement in either quality or frequency.

Fully Developed

This indicator is evident in a variety of ways throughout the agency or program. The practice described is clearly a part of the agency or program culture and the way people operate within the district.

2


Vision and Policy Management

A mature data team has played a significant role in the development of the agency or program’s vision for data use and has gathered support for that vision from all stakeholders. The team has developed and promulgated policies which formalize the agency or program’s expectations for the use of data and the inquiry process throughout the agency or program and Head Start centers.

Rating Scale: NK=No Knowledge about this Indicator; NE=No Evidence; SE=Some Evidence; CE=Clear Evidence; FD=Fully Developed Indicators NK NE SE CE FD

The data team has worked with agency/program leadership to develop a written vision for data use that aligns with and furthers the wider agency/ program’s mission and vision.

The vision for data use was created with input from stakeholders

The data team has communicated the vision for data use widely within the agency/program and Head Start centers.

Agency/program and center stakeholders understand the vision for data use.

Agency/program and center stakeholders support the vision for data use.

The data team models expectations for data use as expressed in the vision.

The vision is supported by agency policies and published expectations that support the use of inquiry and data for instructional, program, and agency improvement.

The data team communicates and supports the agency’s/program’s policies for data use with all stakeholders.

3


Data Management

A mature district data team shares responsibility with the information technology (IT) staff to ensure that complete, accurate, and relevant data are provided to district and school staff in a timely manner to support the inquiry process and fulfill the district’s vision for data use. The team continually seeks feedback and information from data consumers in the district to ensure that the data needed to support the inquiry process are available in user friendly formats that are accessible to those who need the data to inform their decisions. Rating Scale: NK=No Knowledge about this Indicator; NE=No Evidence; SE=Some Evidence; CE=Clear Evidence; FD=Fully Developed Indicators NK NE SE CE FD The data team has established a cooperative relationship with the IT department and shares the responsibility for managing the agency’s/program’s and centers’ data.

The data team effectively coordinates the efforts of those who collect and manage data and those who are the consumers of the data.

The data team has inventoried the data currently available in the agency/program and centers and has shared this information with staff.

The data team, in collaboration with the IT department, has put systems in place to ensure the collection, storage, and dissemination in a timely manner of quality data.

The data team, in collaboration with the IT department, has provided the systems to ensure that appropriate stakeholders have access to relevant data.

The data team has created, published, and continually updated as appropriate a data dissemination schedule or similar document that includes information about when various data are updated and available, as well as expected uses of each data set.

The data team has the capacity to design and create meaningful data displays to support the staff as they engage in the inquiry process.

4


Inquiry, Analysis, and Action

A mature data team has played a central role in selecting or adapting an inquiry model that will be most effective at each level of the organization. The team provides professional development and other supports to help program and center teams initiate the inquiry process and coaches the teams through its phases as necessary. The data team models the use of the inquiry model in all of its investigations. Rating Scale: NK=No Knowledge about this Indicator; NE=No Evidence; SE=Some Evidence; CE=Clear Evidence; FD=Fully Developed Indicators NK NE SE CE FD The data team has collaborated with agency/program and center leaders to select or create an inquiry model to be used universally..

The data team has coached agency/program and center staff in the use of the selected inquiry model.

The data team uses the selected inquiry model to support their investigation of agency/program and center issues pertaining to data use.

The data team uses the selected inquiry model to support their investigation of teaching and learning issues.

The data team coach’s classroom and center teams as they develop progress indicators and use these indicators to monitor/evaluate the impact of their action plans.

5


Professional Development

A mature data team plans and conducts training and professional development activities that build the capacity of staff at all level to effectively use data. Team members extend the effect of these activities by serving as or providing other staff as data coaches who provide embedded professional development for groups that are conducting specific investigations. The team also works with those responsible for professional development in the agency or program to use data to inform decisions about the type of training that would be most helpful for the staff and to evaluate the impact of those trainings. Rating Scale: NK=No Knowledge about this Indicator; NE=No Evidence; SE=Some Evidence; CE=Clear Evidence; FD=Fully Developed Indicators NK NE SE CE FD The district data team has organized professional development/training activities to build the team’s own capacity to effectively use data and the inquiry process.

The data team has created and delivered professional development/training activities to build the capacity of agency, program and center-level staff to effectively use data.

Certain staff in the agency/program has been trained as data coaches to help all center teams use data effectively.

The data team has created and delivered professional development activities to build the capacity of agency, program and center-level staff to effectively use the inquiry process.

The data team has collaborated with agency, program, and center personnel who are responsible for planning and executing professional development activities for the staff to identify relevant activities through the analysis of child outcome data.

The data team has collaborated with agency, program and center personnel who are responsible for planning and executing professional development activities for the staff to use data to determine the effectiveness of professional development activities.

6


Monitoring and Communication

A mature data team supports communication that fosters learning and collaboration, while at the same time monitoring to ensure that people have the right data, tools, and support to do their work. The team uses monitoring information to periodically review what is and isn’t working and makes adjustments to policies, technologies, data collection, assessments, and professional development as necessary to continue the agency or program’s progression toward its vision. Rating Scale: NK=No Knowledge about this Indicator; NE=No Evidence; SE=Some Evidence; CE=Clear Evidence; FD=Fully Developed Indicators NK NE SE CE FD The data team has an established a communication plan or similar documentation and processes to facilitate communication about data use within the agency, program and center communities.

The data team has made the vision for data use available to all members of the community and has communicated the importance of the vision and the positive effect that the use of data and the inquiry process will have in the agency, program and centers.

The data team has developed and implemented systems to promote sharing of best data use and inquiry practices among all members of the agency, program and center communities.

The data team has established benchmarks and implementation indicators to monitor the progress toward the realization of the vision for data use.

The data team assesses the progress toward realization of the vision for data use and publishes an annual progress report.

The data team monitors the data use and inquiry practices that are being implemented by all teams at the agency, program and center levels.

The data team monitors the availability and accessibility of relevant data and data analysis tools and systems to support agency, program and center level inquiry.

7

Identifying Issues

8

Early Learning Data Team Toolkit 2. Identify Issues: Examples of Significant Issues and Related Questions

2.0 – Examples of Significant Program and Early Learning Questions

Directions:

As you review these issues and questions noted on the following pages, think about the two

criteria of a good issue as defined in Identify Issues:

1. The potential responses to the issue are under the direct control of the agency and/or

early learning program.

2. An issue that if addressed would have a significant impact on the school readiness of the

child.

9


Context Issues

While context issues are not under the direct control of the agency or program, they do represent issues

that need to be considered as they investigate issues over which they do have influence. In some cases

context issues like those below can result in blaming and frustration. Therefore, if investigating issues of

context, keep the focus on understanding existing conditions in an effort to change what is in the agency

or program control.

Context Issues

Enrollment

How have the demographic characteristics of the children and families in your

Head Start program changed over the past 5 years?

What new early learning programs in your community are impacting your

enrollment?

How have economic and resource changes in the community impacted

enrollment?

What do we know about why families withdraw their children?

New Families

and Children

Do we have the right procedures in place to welcome new children and families

into the program both at the beginning of the year and during the year?

What does our parent orientation communicate to families about the

importance of school readiness?

Does our introduction to the program adequately engage parents/caregivers in

the program; in the classroom; in governance; and/or in supporting their child’s development?

Special

Populations

Special Education: What are the characteristics and performance levels of

children who are receiving special education services? What services are we

providing to address the needs of these children?

English Language Learners (ELLs): Are ELLs who have been in the program for

at least 1 year meeting the “widely held expectations” for their age in English language and literacy? Mathematics?

Attendance: What is the relationship between attendance below 85% and the

school readiness scores?

Homeless: Do we have active and positive engagement with homeless shelters

and programs that supports the enrollment of homeless families and creates a

support system for families that may become homeless?

10


School Readiness Issues

School Readiness Issues

Enrollment

Years What does the child outcome data indicate about children who were in the

program for one year compared to children in the program for two years?

Gender What does the child outcome data indicate for school readiness of boys

compared to girls? Is it significant?

ELL What does the child outcome data indicate about school readiness for ELL

children?

Program

Model If your program has more than one program model, what does the child outcome

data indicate about the impact of those models?

Domains Over the last two to three years, which child development domains have

improved outcomes, which have remain essentially the same, and which have

decreased outcomes?

Instructional Supports, Classroom Management, and Emotional Environment

Instruction Issues

Instructional

Supports Which areas of instructional support indicate improvement in overall scores?

Which areas have not improved?

Intervention

and

Individualization

How do teachers use the data from their ongoing assessment to support the

growth of every child? What strategies are used to support children who are

not demonstrating growth and who will still be below the skills needed for

kindergarten readiness?

Differentiation How are children’s strengths identified to facilitate increased progress?

Time Is instructional time prioritized appropriately? Are there missed opportunities

for intentional instruction during regular routines?

Assessment Are ongoing (formative) assessments used to differentiate or individualize for

children??

11


Family Engagement Issues

Family Engagement Issues

Withdrawn

Families What are the characteristics and performance levels of children who leave the

program before the end of the school year?

Attendance What are the characteristics of children and families who have the highest and

lowest rates of absenteeism?

Policy Council

What is the attendance at Policy Council?

Are Policy Council members serving as positive and active communicators

between the center and the council?

Parent

Involvement

Is there a center committee? If so, what percentage of families are attending?

Are there parent identified activities? What is the attendance

Parent

Advocacy

Are there opportunities for parents to learn advocacy skills?

Are parents involved in local, state or national interest groups related to the

issues of early learning, poverty, child care, etc.?

Are there transition activities and processes for families entering public school?

Resource Allocation Issues

Resource Allocation Issues

Staffing What are the qualifications of the teachers? Is there a relationship between the

qualifications and aggregated child outcome scores by teacher?

Professional

Development

Based on child outcome data, what topics should be an integral part of our

professional development program this year?

Is job-embedded professional development occurring?

Are classroom staff receiving in-depth, over-time professional development in

one of the child outcome domains, such as math or literacy?

Does the plan for professional development link to the program’s long term and short term goals and objectives; and to the school readiness plan?

Program

Evaluation Based upon the program-wide child outcome report, is there a need to consider

specific curricula in specific domains?

Supplies and

Equipment

and Resources

Are classrooms adequately equipped?

Are there enough science and math materials?

Are there adequate literacy materials

Are your training funds reflective of the professional development needs?

Does the budget for materials and supplies support the school readiness goals

and child outcomes?

12

Early Learning Data Team Toolkit 2. Identify Issues: Identifying and Prioritizing Significant Issues

2.1 – Identifying and Prioritizing Significant Issues

Directions:

1. Using the information that you gained from the review of 2.0 Examples of Significant Early Learning Issues and Related Questions, work together to brainstorm issues that could be the focus of your inquiry. Capture these issues on chart paper.

2. As a team, reach consensus on the three top priority issues and circle them on the chart paper. From these three, choose the highest priority issue. This issue will guide the initial stage of your inquiry.

3. Write your issue statement in the top row of the separate document titled Data Institute Planning Summary.

Although the team will focus on one priority issue at this time, other issues that have been identified can be investigated as the inquiry progresses.

To identify and prioritize issues of importance to the program/learning environments.

Through the use of brainstorming techniques, the program data team will use this protocol to reach consensus on important issues that should be investigated, prioritize those issues, and help begin to shape the initial stages of the team’s inquiry.

About 30 minutes

Purpose:

Time:

13

1.

1.

2. 3.

4.

14

Early Learning Data Team Toolkit 2. Identify Issues: Developing Focusing Questions to Initiate Inquiry

2.4 – Developing Focusing Questions to Initiate Inquiry

Directions:

1. Write the issue that the team identified in 2.1 Identifying and Prioritizing Significant Issues at the top of a piece of chart paper. It can be formulated as either a statement or question.

2. As a group, brainstorm questions that stem from the issue statement/question. Capture these questions on chart paper. All items should be phrased as questions and relate to the Identify an Issue phase of the inquiry process. Circle your top three priority questions. From these three, choose the highest priority question. This question will guide the initial stage of your inquiry. Although the team will focus on one priority question at this time, other questions that have been identified can be investigated as the inquiry progresses.

3. When the team is finished, record this question on the second row of the separate document titled Data Institute Planning Summary.

To identify and prioritize focusing questions to initiate the inquiry process.

Through the use of brainstorming techniques similar to those used in 2.1 Identifying and Prioritizing Significant Issues, the data team will use this protocol to develop and prioritize questions that will focus the collection and analysis of data related to their priority issue. About 30 minutes

Purpose:

Time:

15


Page 2

Problem Statement Worksheet

Initial broad issue

Focusing question

Who is affected by this issue?

What do the data say about the focusing question?

Inferences regarding this issue?

Draft problem statement

16


Sample Issue Statement & Focusing Question from Scenario

Initial broad issue

Our State is measuring the “readiness” level of entering kindergarten

students and asking schools and communities to work together to

increase readiness of children.

Focusing question Which domain of kindergarten readiness provides the greatest


What do the data say

about the focusing

question?

39.5% of Native American students compared to 49.3% of

Hispanic students demonstrated readiness in math.


Hispanic students were ready in language.

36.2% of Hispanic students were measured as ready in all six

domains.

The scores in every domain were lowest for children with a

disability.

For all categories of students, language scores were lower than

literacy scores.

Math was the lowest domain for all children.

Which data informed

problem/concern was

selected for further

exploration?


Draft problem statement Less than 50% of children had basic math concepts expected at entry

to kindergarten, regardless of ethnicity, language or disability.

17

Early&Learning&Data&Team&Toolkit&& 2.&Identify&Issues:&Identifying&and&Locating&Data&Sources&&

2.5$Identifying$and$Locating$Data$Sources$Template$Directions:$Complete&this&template,&then&summarize&the&data&sources&on&row&4&of&the&separate&document&titled,&“Data%Institute%Planning%Summary”$

Focusing$Question$

What%was%the%kindergarten%readiness%for%each%center%of%the%Three%Rivers%Program%in%2011–2012?%<or<%

Write%a%focusing%question%of%your%own:$$

$

Data$Element$$ Where$Stored$Who$Can$Provide$

Access$Collected$by$Whom$ Collected$By$When$

&

&

& & & &

&

&

& & & &

&

&

& & & &

&

&

& & & &

&

&

& & & &

& & & & &

&

18

Understanding Issues

19

Early&Learning&Data&Team&Toolkit&& 3.&Understanding&Issues:&Attributes&of&a&Good&Data&Display&&

&&

Data$Display$Quality$Checklist$

Attributes$Attribute$Present$(Y/N)$

Comments$

Structural$Components:$ & &

Informative&title& & &

All&axes&are&labeled& & &

Population&size&noted& & &

If&data&are&presented&as&percentages,&the&numbers&used&to&calculate&the&percentages&are&also&provided.&

& &

All&variables&are&identified&and&labels&provided.&

& &

Dates&for&data&points&are&provided&if&applicable.&

& &

A&key&exists&that&identifies&all&symbols,&shading,&color,&etc.&

& &

Functional$Components:$ & &

Display&is&uncluttered&and&free&of&unnecessary&detail&and&extraneous&features.&

& &

Display&uses&an&appropriate&chart&style&(e.g.,&pie&chart,&clustered&bar&chart,&stacked&bar&chart).&

& &

The&display&communicates&the&story&that&the&author&wants&to&tell.&

& &

$

20

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

4 - 5 Year Old - PreK Meeting or Exceeding Age Expectations

Fall

Winter

Spring

21

|

HOW DO I CHOOSE WHICH TYPE OF GRAPH TO USE?

When to Use . . .

. . . a Line graph.

Line graphs are used to track changes over short and long periods of time. When smaller chang-es exist, line graphs are better to use than bar graphs. Line graphs can also be sued to compare changes over the same period of time for more than one group.

. . . a Pie Chart.

Pie charts are best to use when you are trying to compare parts of a whole. They do not show changes over time.

. . . a Bar Graph.

Bar graphs are used to compare things between different groups or to track changes over time. However, when trying to measure change over time, bar graphs are best when the changes are larger.

. . . an Area Graph.

Area graphs are very similar to line graphs. They can be used to track changes over time for one or more groups. Area graphs are good to use when you are tracking the changes in two or more related groups that make up one whole category (for example public and private groups).

. . . an X-Y Plot.

X-Y plots are used to determine relationships between the two different things. The x-axis is used to measure one event (or variable) and the y-axis is used to measure the other. If both variables increase at the same time, they have a positive relationship. If one variable decreases while the other increases, they have a negative relationship. Sometimes the variables don’t fol-low any pattern and have no relationship.

22

Early Learning Data Team Toolkit 3. Understanding Issues: Data Analysis Protocol

3.5A – Data Analysis Protocol

Directions:

Step 1: Prepare

1. Appoint a notetaker for this protocol and record the question that you are investigating on the Data Analysis Worksheet and on a piece of chart paper..

2. Gather the data set(s) and data displays that relate to the question under investigation and make them available to all team members.

Step 2: Observe

During the observation step concentrate on making objective observations about what is in the data. Do not attempt to make judgements about why the data may appear as they do.

1. Using the data set(s) and display(s) provided, take turns making factual observations about what the data say. The notetaker will record the observations under the question on the chart paper.

2. When expressing your observation, you might use sentence starters such as: “I see…,” “I observe…,” and “I notice…” Stay away from making inferences. Discuss ONLY the facts at this stage of the process. If you catch yourself using the terms “however,” “because,” or “therefore,” STOP and return to the sentence starters suggested above. It is okay to make observations that are based on the observations made by other team members.

The following questions will help you probe for deeper analysis: a. How do the data sets compare to each other? b. What are the commonalities among a given data set? c. What patterns or similarities are evident across different data sets? d. What inconsistencies or discrepancies (if any) are evident? e. Is there anything you expected to see but don’t? f. What is not represented in the data? g. What questions do the data raise?

3. The notetaker will record your observations under the question on the chart paper. You should record the observations on your Data Analysis Worksheet

Step 3: Interpret – Make inferences from the observations.

During the interpret step, your team will carefully work to make meaning from the data and your observations. Remember that inferences you are making need to be based on the evidence you observed in the data. When all observations have been made, review them as a team. Code or group the observations into categories of findings. Think about the following questions while organizing the observations.

a. What assumptions might be underneath what you are noticing in the data? b. What clues help explain why a certain population is meeting or missing targets? c. What areas in the data stand out as needing further explanation? Why? d. What patterns or themes do you see in the observations?

23


e. Which of these observations are most relevant and important to your inquiry? Why?

4. As a team, review the categorized findings. Make a list of what the team can now infer about the focusing question. The notetaker should record the list on chart paper. When the list is complete, record what the team can infer on the Data Analysis Worksheet. The inferences made by the team will help clearly identify the direction for further inquiry.

Step 4: New Questions or Tentative Conclusions

More often than not your team will end this protocol with a new set of questions to investigate. The data needed to extend the inquiry may be readily accessible or the set of questions may require a new round of data collection and organization.

The inferences that the team has generated are likely to raise more questions that need to be answered to support the inferences before a tentative conclusion can be made. If this is the case, generate questions as you did in 2.4

Developing Focusing Questions to Initiate Inquiry by following the steps below.

a. Brainstorm questions that arise from the observations and inferences that the team has made about the initial data set. Record these questions on chart paper.

b. From this group of questions, identify the questions that must be answered before any tentative conclusions about the problem that underlies the priority issue can be made. Record them on a new sheet of chart paper leaving room next to each question to record more information. Also record the questions on the Data Analysis Worksheet.

c. The clarifying questions the team has identified may be answered using the data already collected and displayed. It is more likely, however, that new data will need to be identified, collected, displayed, and analyzed. For each of the clarifying questions, brainstorm the data needed and record the data element(s) next to each question on the chart paper.

d. Complete the Data Identification, Collection, and Display Worksheet: Clarifying Questions to organize the next phase of your work.

e. Summarize your observations and your inferences on rows 5 & 6 of the separate document titled, “Data Institute Planning Summary”.

1. Build data displays as appropriate to facilitate analysis of the newly acquired data. 2. The data analysis process is iterative. Repeat the steps of this protocol to analyze the newly collected data. 3. Repeat the data analysis process until the team is confident that it can draw a tentative conclusion from its

observations and inferences.

24


Data Analysis Worksheet

Question:

Observations (Without judgment, what do you see?):

Inferences (What can the team now infer about the focusing question?):

Clarifying Questions or Tentative Conclusions

25


Data Identification, Collection, and Display Worksheet: Clarifying Questions

Clarifying Questions Data Needed Data Source Display Construction Plan

26

Early&Learning&Data&Team&Toolkit&& 3.&Understanding&Issues:&Data&Carousel&Protocol&

&

&

Directions:+

Step+1:+Preparing+for+the+Data+Carousel++1. Label&4&separate&chart&papers&with&the&name&of&the&data&domains;&Demographic;&Perceptual;&Outcome;&Program.&

(Note:+please+see+Identifying+and+Locating+Data+Sources+Template).&&&

2. Identify&at&least&two&data&sources&for&each&of&the&four&domains.&&&

3. Display&the&chosen&data&on&large&enough&so&the&data&can&be&read&easily&by&all&participants.&&&

4. Create&3&columns&under&the&displayed&data&on&each&chart&paper.&Label&one&Strengths,&another&Concerns,&and&the&third&Additional1Information1Needed.&&&

5. Break&the&participants&into&4&groups&and&assign&a&color&to&each&of&the&groups.&&a. If&the&activity&is&being&done&with&a&school&or&district&group,&the&groups&should&be&cross&departmental,&

cross&grade&level,&and/or&cross&function.&&b. Each&group&should&designate&a&facilitator&and&a&recorder.&&

&

6. Provide&one&or&two&color&markers&that&match&the&assigned&color&&for&each&group.&Markers&should&travel&with&the&group.&

Step+2:+Analyzing+the+Data+1. Each&group&should&select&one&of&the&data&domains&as&a&starting&point.&&Each&group&will&visit&each&station&and&

analyze&the&data&presented&at&that&station.&

2. The&facilitator&will&lead&their&group&in&making&objective&observations&about&what&the&data&say.&&&

a. The&recorder&will&record&each&narrative&observation&using&their&appropriately&colored&marker.&

b. The&narrative&statement&of&the&group’s&observations&must&be&factual,&not&evaluative.&&

i. Narrative&statements&should&be&simple&and&communicate&a&single&idea&related&to&the&given&characteristic.&

ii. The&statements&should&be&short&and&easy&to&read.&

iii. Each&narrative&observation&should&be&able&to&stand&alone.&&For&instance:&“From&1999&to&2002,&girls&scored&higher&than&boys&on&the&math&portion&of&the&4th&grade&MSP&assessment.”&

iii. Statements&such&as&“seventh&grade&reading&achievement&on&the&MSP&increased&by&34&percentage&points&between&2002&and&2003”;&NOT&“the&new&reading&program&contributed&to&a&34&point&increase&in&reading&scores&between&2002&and&2003.”&

3. Each&group&should&visit&all&of&the&stations,&repeating&this&process&at&each&station&until&they&have&returned&to&the&station&that&they&began&with.&

4. At&their&initial&station,&the&group&should&review&all&of&the&factual&observations&that&have&been&made&by&each&of&the&groups&about&the&data&related&to&that&characteristic&of&high&performing&schools.&

5. Each&group&should&reach&consensus,&based&on&all&of&the&observations,&on&what&the&data&say&about&the&characteristic&at&their&station.&

6. &Once&consensus&has&been&reach,&the&recorder&should&write&a&succinct&summary&and&present&the&group’s&analysis&to&the&larger&team.&

7. Summarize&your&observations&and&your&inferences&on&rows&5&&&6&of&the&separate&document&titled,&“Data%Institute%Planning%Summary”.&

3.5B+–+Data+Carousel+Protocol+

27

Early&Learning&Data&Team&Toolkit&& 3.&Understanding&Issues:&Placemat&Protocol&

&

& &

Directions:+

Step+1:+Preparing+for+Analysis+

1. Organize&participants&into&small&groups&of&3–5&individuals.&Groups&should&be&as&heterogeneous&

as&possible&(e.g.,&across&disciplines&and&grade&levels).&

2. Gather&several&data&sets/data&displays&that&relate&to&the&priority&issue&being&addressed&or&

general&displays&of&interest&if&a&priority&issue&has&not&been&identified.&&These&might&include:&

a. Child&Outcome&Data&

b. CLASS&reports&

c. Attendance&data&

&

Ideally,&data&sets&would&cover&:&

a. A&number&of&series&of&years&of&information&showing&trends&over&time&

b. Show&how&the&center&compares&to&the&program&wide&data&or&other&like&centers&

c. Compare&results&by&a&few&student&attributes&(e.g.,&gender,&special&education&status,&race).&

3. Produce&one&data&set&with&multiple&copies&per&group.&

4. Select&a&room&with&enough&tables&so&that&there&is&one&group&at&each&table.&

5. Provide&a&printed&placemat&for&each&table&(11”x17”&size&will&work,&but&posterTsized&paper&is&

better).&&The&placemat&is&provided&on&page&4&and&asks&the&questions:&

a. What&are&the&positives?&What&are&the&negatives?&

b. What&surprises&did&you&find&in&the&data&(both&positive&and&negative)?&

c. What&do&you&now&know&that&you&didn’t&know&prior&to&the&analysis?&

d. What&questions&arise&from&this&analysis&(anything&that&will&require&further&study)?&

e. What&questions&should&we&address&next?&

6. Provide&pens/markers&to&write&on&the&placemats.&Ideally,&each&groups&should&have&a&different&

color&marker&and&there&should&be&at&least&three&markers&per&group.&The&markers&will&travel&

with&the&group&to&identify&the&placemat&comments&made&by&each&group.&

Step+2:+Conducting+the+Analysis+

1. Place&a&data&set,&placemat,&and&pens/markers&at&each&table.&&All&of&these&items&except&the&

markers&are&intended&to&remain&at&the&table&for&the&entire&exercise.&

2. One&group&should&sit&at&each&of&the&tables.&

3. The&facilitator&should&provide&the&following&set&of&instructions&to&the&groups.&

a. On&each&table,&you&will&find&a&data&set&that&shows&current&and&past&data&for&our&school.&&You&

will&also&find&a&placemat&record&sheet.&&For&the&entire&session,&all&the&items&at&the&table&(with&

the&exception&of&your&group’s&colored&marker),&will&remain&at&the&table.&

b. As&a&group,&you&will&look&at&the&data&for&1–2&minutes&without&any&discussion.&&After&the&2&

minutes,&please&discuss&as&a&group&the&five&questions&on&the&placemat&record&sheet.&&During&

your&discussion,&please&record&your&answers&to&these&questions&on&the&placemat.&&Anyone&

3.5C+–+Placemat+Protocol+

28

Early&Learning&Data&Team&Toolkit&& 3.&Understanding&Issues:&Placemat&Protocol&

&

& &

can&write&on&the&placemat&at&any&time&and&in&fact&group&member&might&sometimes&do&so&

simultaneously.&&You&will&have&approximately&10&minutes&for&this&step.&

c. Once&completed,&groups&will&rotate&to&the&next&data&set&and&repeat&this&process.&&When&you&

record&your&findings,&you&will&do&so&on&the&same&record&sheet&as&the&previous&group(s).&&In&

this&way,&you&will&see&the&work&of&the&group(s)&before&you&and&can&add&to&their&thinking.&&

Each&group’s&comments&will&be&indicated&by&their&color.&

d. When&we&have&finished&with&all&the&data&sets,&we&will&collectively&review&what&we&have&

found.&

4. The&previous&exercise&will&take&about&15&minutes&to&complete&the&first&time.&Each&subsequent&

analysis&wil&likely&take&less&time.&Facilitators&are&urged&to&wander&from&table&to&table&to&participate&

in&or&listen&to&the&discussions&and&conversations.&&Probing&questions&can&be&asked&or&detail&

provided&as&to&the&meaning&of&the&actual&data&set.&&It&is&recommended&that&the&principal&not&

participate&in&providing&content&for&the&placemat.&&The&facilitator&can&adjust&the&time&required&

based&on&the&richness&and&direction&of&the&discussions&at&each&table.&

5. After&the&placemat&has&been&completed,&groups&will&rotate&to&the&next&data&set&and&repeat&the&

process.&&Remind&the&participants&that&they&must&leave&both&the&data&set&and&placemat&at&the&

table.&&At&the&new&table,&groups&will&consider&the&new&data&set&and&review&the&work&of&the&

previous&group(s)&before&adding&to&the&placemat&in&the&manner&described&above.&

6. Rotations&will&occur&until&the&groups&arrive&back&at&the&their&original&table.&&&

7. Provide&some&time&for&the&group&to&review&and&discuss&all&of&the&comments/statements&made&on&

the&placemat&at&their&original&table.&

8. By&group,&have&each&report&on&what&the&general&findings&were&for&the&data&set&at&that&table&as&

captured&in&the&placemat&record&sheet.&

Step+3:+Next+Steps+

1. The&next&step&for&the&principal&or&facilitor&is&to&collect&all&of&the&placemats&and&to&have&the&

information&from&them&recorded&in&a&more&shareable&format.&&&

2. Summarize&your&observations&and&your&inferences&on&rows&5&&&6&of&the&separate&document&titled,&

“Data%Institute%Planning%Summary”.&3. Once&completed,&it&is&recommended&that&this&information&be&shared&back&to&staff&for&further&

comment,&additions,&or&corrections.&&This&activity&has&engaged&the&entire&staff&in&considering&and&

analyzing&the&data,&recommending&additional&data&that&should&be&brought&forward,&and&suggesting&&

possible&strategies&that&may&be&used&to&address&the&priority&issue&or&issues&that&are&discovered&

through&the&analysis&process.&&

&

&

&

&

&

&

&

&

29

District'and'School'Data'Team'Toolkit''

'

3.5C%

Placemat%Record%Sheet%

'

30

Diagnose Causes

31

Early Learning Data Team Toolkit 4. Diagnose Causes: Why? Why? Why? Protocol

4.2A – Why? Why? Why? Protocol Introduction:

Identifying root cause in education rarely results in a single factor being identified that can easily be resolved. Protocols such as this one help a group of educators collaboratively discuss the most likely root causes of the problem under investigation. This discussion will help the team come to agreement about what is the most significant factor within the agency/program’s and center’s control to address.

Directions:

1. Write the evidence-based problem developed in 4.1 Problem Statement on chart paper. 2. Each member of the team will then write one or more responses to the question: “why might

this be happening?” Each response should be written on a separate sticky note. 3. Place the sticky notes in a row across the chart paper under the problem. Discuss the responses

and eliminate any that duplicate the same basic idea. Add any that appear to be missing. Rank-order the ideas from most plausible causes to least plausible. As you do this, think about

factors that are under the district’s/school’s control and which, if addressed, will solve the identified problem. If the team has difficulty determining rank order, they may want to consult tool 4.3A Determining Significance and Control, or 4.3B Interrelationship Protocol which are designed to help the team determine which root causes are most significant. This should be done after all potential root causes have been suggested.

4. For the most plausible reason, again write possible explanations of why this is happening on sticky notes. Place these in a row below the most plausible cause. You can revisit the other reasons later.

5. Again, rank-order the causes. Review all of the causes that you have associated with the initial, most plausible cause, and reach consensus on what the team believes to be the most likely root cause.

6. Summarize the root cause(s) in row 8 of the separate document titled “Data Institute Planning Summary”.

32

Early Learning Data Team Toolkit 4. Diagnose Causes: 20 Reasons Protocol

4.2B – 20 Reasons Protocol Introduction:

When using this tool, a data team will often disagree about why the given problem exists. During the brainstorming portion of the process, it is important to follow brainstorming norms. These norms will enable all participants to express their views without value judgments being made by others. When all of the 20 reasons have been recorded, the group can reflect and debate the various root cause suggestions and reach consensus on the most likely root cause.

Directions:

1. Each data team member should record the problem being investigated on their copy of the 20 Reasons Worksheet. Review the Example of Partially Completed 20 Reasons Worksheet so that each team member understands the desired outcome.

2. As a team, brainstorm reasons that might explain why the problem exists. During the brainstorming session, no judgment should be made about any of the suggested reasons. However, the facilitator must remind team members to suggest only significant reasons that are within the control of the agency/program.

3. It may be helpful for each team member to suggest a reason in turn. The recorder should capture each reason on the 20 Reasons Worksheet (computer or chart paper). Getting to 20 reasons is often difficult, but the team should persist. It is the last few reasons that often get to the heart of the problem.

4. When 20 reasons have been suggested, the team should take several minutes to individually reflect on all of the suggestions. Each team member should identify the reason that he or she feels is the root cause of the problem.

5. In turn, each team member should identify the reason that he or she believes is the root cause of the problem and justify his or her choice. The recorder should put a check by each reason so identified. If the same reason is identified by several team members, multiple check marks should be made next to the reason.

6. As a team, discuss the reasons with check marks and reach consensus on the root cause of the problem. It is possible that a given problem may have multiple root causes. If the team’s consensus is that several reasons explain the root cause of the problem, try to prioritize them or describe their relationship to each other.

7. If the team is struggling to prioritize the root causes, they may want to consult tool 4.3A Determining Significance and Control, and 4.3B Interrelationship Protocol that are designed to help the team determine which root causes are most significant. This should be done after all potential root causes have been suggested.

8. Summarize the root cause(s) in row 8 of the separate document titled, “Data Institute Planning Summary”.

33


Example of Partially Completed 20 Reasons Worksheet

# Possible Explanation Root

Cause?

1 New teachers are not individualizing for children

2 Classrooms are poorly equipped.

3 Professional development resources have been targeted at family support and health services for the last two years.

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

Problem: Overall child outcomes are below 80% for “widely held expectations” for age groups.

34


20 Reasons Worksheet

# Possible Explanation Root Cause?

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

Problem:

35

Early Learning Data Team Toolkit 4. Diagnose Causes: Fishbone Analysis Protocol

4.2C – Fishbone Analysis Protocol Introduction:

During the brainstorming section of this protocol, participants may come up with possible causes that do not fit easily into one of the previously identified categories. This can indicate a need to identify a new category or broaden an existing category. Do not discard an idea solely because it does not fit into a previously identified category. If necessary, add the new category and move on. The purpose of the major categories is to provide a structure to guide the brainstorming. These categories should be used to inspire, rather than restrict, participants’ thinking.

Preparation:

1. Distribute a copy of the Example of a Completed Fishbone Diagram Worksheet and a blank copy of the Fishbone Diagram Worksheet to each data team member.

2. Create a blank copy of the diagram on chart paper. Directions:

1. Write the problem under investigation in the box at the “head” of the fish.

2. Identify major categories that are logically associated with the problem and write them in the boxes in the diagram. The diagram has four “ribs” and boxes, but more or fewer boxes can be used depending upon the selected categories. Note: The following categories are often used: students, families, processes, curriculum, instruction, teachers. Remember to look for root causes that are under the agency/program and or center’s control.

3. For each category, brainstorm possible causes of the problem related to that category. Record the possible causes next to the appropriate “rib” in the diagram. Repeat this process for each of the categories.

4. Study the display that you have created. Are all of the reasons that have been identified under the control of the district and/or school? If not, place an “X” next to those not under agency/program or center control. As an alternative to this step, the team may want to consult tool 4.3A Determining Significance and Control that is designed to help the team determine which potential root causes are most significant AND most influenced by the agency/program or center.

5. As a data team, analyze each possible cause to determine whether it is a root cause by asking: a. Would the problem have occurred if this cause had not been present? b. Would the problem reoccur if the cause was corrected?

6. If the answer to both of these questions is no, you have found a likely root cause.

7. Place a checkmark next to each idea that is not a root cause and circle each idea that is a likely root cause of the problem.

8. If the team is struggling to prioritize the root causes, they may want to consult tool 4.3A Determining Significance and Control, and 4.3B Interrelationship Protocol which are designed to help the team determine which root causes are most significant. This should be done after all potential root causes have been suggested.

9. Summarize the root cause(s) in row 8 of the separate document titled, “Data Institute Planning Summary”.

36

Early Learning Data Team Toolkit 4. Diagnose Causes: Fishbone Analysis Protocol

Fishbone Diagram Worksheet

Problem:

37

Early Learning Data Team Toolkit 4. Diagnose Causes: Determining Significance and Control

4.3A Determining Significance and Control Directions:

1. Create a Significance/Control Matrix on a piece of chart paper similar to the one below. 2. Write each of the possible root causes identified in 3.5 Data Analysis (3.5A Data Analysis Protocol, 3.5B Data

Carousel Protocol, or 3.5C Placemat Protocol) on a separate sticky note.

3. Place each possible cause in a quadrant of the matrix based on the team’s judgment about its significance as a cause of the underlying problem.

4. When all of the possible root causes have been place in a quadrant of the matrix, revisit them starting with

Quadrant I (high significance). As a team, discuss the degree of control that the district or school has over each

of the causes in Quadrant I.

5. Reposition each of the Quadrant I possible root causes along the degree of control axis to indicate the control

that the district or school has over that cause.

6. Repeat this process for the possible root causes in each of the quadrants of the matrix.

7. When all of the possible root causes have been positioned on the significance and control axes, those in

Quadrant I represent the causes that are most significant and over which the program has the most control.

These are the root causes that should be easiest for the district or school to address and which, if appropriately

addressed, will have the greatest impact on student outcomes.

High Control Low Control

Low

Sig

nif

ica

nce

H

igh

Sig

nif

ica

nce

Lack of academic support for

traumatized children

Low use of proven instructional

techniques

Quadrant I Quadrant II

Parents were poor role models for

students

Cultural factors did not support the

importance of education

Quadrant III

Inadequate facilities

Transfer policies

Quadrant IV

Minority students

Mature faculty

Figure 1. Charting Significance and Control

38

Early Learning Data Team Toolkit 4. Diagnose Causes: Interrelationship Protocol

4.3B Interrelationship Protocol1 This is an excellent tool that will help the team prioritize the root causes and will give the team insight into which causes, when addressed, will have the most impact on the identified problem and priority issue.

Directions:

1. If you have not already done so, write each of the potential root cause statements on a separate sticky note.

2. Place 4 to 8 sticky notes in a circle on a piece of chart paper. Number them in sequential order.

3. Consider potential causes 1 and 2.

a. As a team determine if there is a relationship between these two causes. b. If there is a relationship, draw a line from cause 1 to cause 2. c. Repeat this process looking at the relationship between cause 1 and each of the other

causes.

4. After looking at the relationships between cause 1 and each of the other potential causes, repeat the process for cause 2 and each of the others, and so on until the relationship between each cause and each of the other causes has been considered.

5. You should now have a diagram on the chart paper with multiple lines running between causes.

Note that some causes may not be related to any of the other causes. No line will be connected to these causes.

6. As a team, consider the relationship that you have indicated between two causes. Reach

consensus on which is the dominant cause (e.g., which has influence over the other). Put an arrowhead on the end of the line pointing AWAY from the dominant cause. Repeat this process for each pair. Note that placing an arrowhead at both ends of a line will not be of value.

7. Count the number of arrowheads going AWAY from each of the causes and record the number

next to the cause. Rank order the causes based on these numbers (e.g. most away arrowheads to least away arrowheads)

8. The cause with the most arrowheads going AWAY from it has the most effect on all the others.

Thus, when addressed, this cause will have the greatest impact on the identified problem.

1 Adapted from Quality in Education, Inc.

39

Early Learning Data Team Toolkit 4. Diagnose Causes: Interrelationship Protocol

Sample Completed Interrelationship Chart

Cause 1

Cause 6 Cause 2

Cause 3

Cause 4

Cause 5

2 Arrows

0 Arrows

3 Arrows

1 Arrow

1 Arrow

0 Arrows

40

Diagnose Causes

41

Early Learning Data Team Toolkit 4. Diagnose Causes: Identifying the Problem(s) of Practice

Directions:

1. A completed Sample Problem of Practice Worksheet is presented on the next several pages. Review this sample before moving on to the next step.

2. When using the Problem Practice Worksheet start by writing the final root cause statement in the first row. 3. Then, brainstorm program or center practices that could be associated with the final root cause. For example:

“Content related to the root cause is not included in instruction.” Record your responses on chart paper. 4. As a team, reach consensus on the practices that are most likely related to the root cause. Record these

practices on the Problem Practice Worksheet. 5. From these practices, reach consensus on the practice that is most likely to have the greatest impact on the root

cause. Record multiple practices if they are related and can be addressed through one initiative.

Sample Problem of Practice Worksheet

Final root cause statement:

Children who are English Language Learners are missing critical social and content specific learning. These children are leaving the center unready to enter Kindergarten.

Practices that could result in the root cause of the problem:

Program has not developed an approach to support children with ELL needs. Materials in use are not culturally appropriate for the children served. Lack of classroom staff who speak the children’s first language. Bilingual staff members are not competent in English and continue to talk to children in their native language. Teachers at Riverside are not effectively implementing concept development instructional strategies.

Final problem(s) of practice (those that could be addressed through one initiative):

Teachers at Riverside are not effectively implementing concept development instructional strategies.

4.6 – Identifying the Problem(s) of Practice1

42

Early Learning Data Team Toolkit 4. Diagnose Causes: Identifying the Problem(s) of Practice

Problem of Practice Worksheet

Final root cause statement:

Practices that could result in the root cause of the problem:

Final problem(s) of practice (those that could be addressed through one initiative):

43


Page 1

Sample Problem of Practice Worksheet

Summarize Root Cause(s)

District has not communicated expectations to parents and early

learning providers.

Parents and early learning providers do not know what math skills are

expected at entry to kindergarten.

Instruction in math at preschool level is limited to counting and

identifying numerals.

Practices that could

result in the root cause of

the problem

Early learning teachers limit instruction to rote counting and one to one

correlation.

Math materials in early learning classrooms are limited due to lack of

resources.

Early learning teachers have not received professional development in

math.

There is lack of collaboration between early learning teachers in the

community and the public school teachers around math.

Final Problem of Practice:

The district teachers and early learning teachers do not have shared

understanding of the skills and concepts that entering kindergarten

students are expected to have.

Summary of Action Plan:

Goal: Increase the percentage of children ready in math from 48% to

58% in two years.

1. Joint committee of early learning and kindergarten teachers will be

formed.

2. Committee will identify kindergarten ready math concepts based upon

TS GOLD.

3. Joint professional development will be created.

Summary of Evaluation:

1. Evaluate the level of collaboration and connection between early

learning providers and the school district through a survey given.

Participants will determine if the action plan was successfully

implemented, creating relationships between early learning and school

teachers.

2. Are 58% of children scoring “ready” in the area of math on the

September 2015 WaKIDS assessment?

44

!

!

!

!

Plan!&!Take!Action!

45

Early Learning Data Team Toolkit 5. Plan for Action: Writing Measurable Goals

Directions:

1. Review Elements of a Well-Written Measurable Statement, below. 2. Using the Elements of a Well-Written Measureable Statement document as a guide, each data team member should create a well-written statement from the problem of practice scenario by:

a. Identifying each of the four elements of a well-written measurable statement b. Writing a clear target/goal statement

3. As a data team, reach consensus on the most appropriate measurable statement for the scenario.

Elements of a Well-Written Measurable Statement

Example Typical Goal or Desired Outcome Increase effective implementation of concept development practices by classroom staff members.

Example Typical Goal Statement Increase the percentage of staff members who implement concept development practices.

A clearer and more useful statement would address the following questions:

1. What will change? The percentage of staff members rated at level 5.

2. For what population? All teachers at the Riverside Center.

3. By how much? To reach 80%.

4. By when? Within 5 years.

The resulting measurable statement (compare to the typical statement above): To increase the percentage of teachers at the Riverside Center receiving a level 5 rating on concept development

to 80% within the next two years.

Practice Goal Writing

Data Scenario – summarize the scenario problem of practice and write a corresponding goal statement based on the scenario data.

Brief data scenario statement:

Elements: What will change? For whom? By how much?

By when?

Effective Goal Statement:

5.2 Writing Measurable Goals Statements

46

Early Learning Data Team Toolkit 5. Plan for Action: Strategies

5.3 Identifying Potential Strategies Directions:

1. The facilitator will write the root cause and problem(s) of practice statements that the team developed in Component 4 on chart paper to focus this brainstorming session.

2. Using the knowledge the team has gained , brainstorm possible strategies to address the problem(s) of practice. 3. Once the team has reached consensus on strategies that may address the problem(s) of practice, repeat this

process to identify strategies that will support the implementation effort and set the stage for eventually scaling-up the initiative and providing for its sustainability. Refer to the resources on Implementation Science, Scaling-up, and Sustainability in the Component 5 Handbook for guidance.

4. The facilitator will record the strategies for use with Tool 5.4 Rating Strategies’ Potential for Success.

5.4 Rating Strategies’ Potential for Success Directions:

1. The Strategy Rating Checklist contains several very basic characteristics that a strategy might need to be successful in your district and/or school. Review these characteristics and eliminate any that are not appropriate. Add characteristics that you believe are necessary for the success of the strategy. As a team, discuss the characteristics that should be eliminated and those that should be added to the checklist. Reach consensus on the final list of characteristics and record the characteristics on the checklist.

2. Write the name of each strategy that the team identified in 5.3 Identifying Potential Strategies as column titles in the Strategy Rating Checklist. Individually assess each strategy against the characteristics in the checklist by placing a check mark in the cell next to each characteristic that the strategy has. Repeat for each strategy.

3. Either project or have the facilitator duplicate the checklist on chart paper. As a team, reach consensus on the assessed characteristics of each of the strategies. Rank order the strategies based on the number of check marks each received. Those with the largest number of check marks are your high-impact strategies.

47

Early Learning Data Team Toolkit 5. Plan for Action: Strategies

Strategy Rating Checklist

Characteristics of Potentially Successful Strategies

Strategy

Example:

Differentiated

Instruction

through tiered

interventions (RtI)

Clearly addresses the

problem of practice. X

Is based on sound

research. X

Endorsed by other

authorities. X

Targets our population

of children. X

[Insert additional]

[Insert additional]

[Insert additional]

Total 4

48

Early Learning Data Team Toolkit 5. Plan for Action: Plan of Action Template

PLAN OF ACTION TEMPLATE

Goal:

Strategy #:

Activities: Steps to be taken. What will occur?

Who is Responsible? Who is involved? Who will provide leadership? Who will provide work?

Timeline: When will this strategy or action begin and end?

Resources Needed What existing and new resources will be used to accomplish the strategy?

Monitoring Effectiveness For each activity, what formative evidence will be gathered to demonstrate progress?

Expected Impact for this strategy: What measurable change will be seen as a result of the implementation of this strategy?

49

Early Learning Data Team Toolkit 5. Plan for Action: Plan of Action Template

PLAN OF ACTION TEMPLATE – TOPPENISH

Goal: Increase the percentage of children ready in math from 48% to 58% in two years as measured by WaKIDS. Strategy #: Joint committee of early learning and kindergarten teachers will be formed.





Expected Impact What measurable change will be seen?


District will assign responsibility for forming committee

Principal March 15, 2014 None Written evidence of assignment

District leader will invite kindergarten teachers and early learning providers to first meeting

District leader March 30, 2014 Contact list of early providers. Agenda for meeting

List of contacts Invitation sent Meeting date set

First committee meeting attendees include district and community teachers

District leader April 15, 2014 Sign-in of attendees

Strategy #2 : Committee will identify kindergarten ready math concepts based upon TS GOLD





Expected Impact What measurable change will be seen?


TS GOLD math indicators identified, and continuum provided to committee

District leader April 15, 2014 TS GOLD database Early Learning Specialist

Minutes of committee meeting indicate sharing of materials and discussion.

50

!

!

!

!

Evaluate!Results!

51

Early Learning Data Team Toolkit 6. Evaluate: Developing an Evaluation Plan

6.1 Developing an Evaluation Plan1 Directions:

1. Access copies of the action plan and the implementation monitoring plan that your team created.

2. Review the process for Creating an Evaluation Plan and the Example Evaluation Plan Template with your team.

3. As a data team, capture the required information in each section of the Evaluation Plan Template a. Since many of the required elements can be taken from documents that you have previously constructed, it would

be most efficient if the Evaluation Plan Template, action plan, and implementation monitoring plan are accessible in digital format on the same computer with projection capability.

b. Have a team member project the Evaluation Plan Template as the team discusses each section. As the team reaches consensus on the information to be included in each section.

c. Alternatively, the data coach can record the information on chart paper to be entered electronically later.

4. Once the plan is complete, assign responsibilities and timelines for the implementation of the plan by data team members. Creating an Evaluation Plan

1. Describe the initiative to be evaluated. a. Who will the initiative impact? b. What is the desired goal or long range desired impact of the initiative? a. What are the major strategies that will be used in the initiative?

2. Define the measurable improvement targets. a. How will success be measured? b. How will you know if your problem of practice was addressed?

3. Identify the data that will need to be collected and analyzed to provide evidence. a. What data need to be collected to provide evidence? b. How will the data be collected (e.g., tests, surveys, observations)? c. When and by whom will each data element be collected? c. How will the data be stored?

4. Explain how the data analysis will be conducted. a. How will the data be prepared for analysis (i.e., culled, organized, displayed)? d. Who will conduct the analysis of the data displays?

5. How will inferences, findings, and conclusions be made from the evidence? a. Have the goals been met? b. Why was the intervention successful? Why was the intervention unsuccessful? c. How can the intervention be improved? e. Who will develop findings and conclusions?

6. Describe the dissemination of findings, conclusions, and recommended next steps. a. What messages need to be communicated? b. To whom? c. When? d. How?

1 Portions of this tool were developed within the DATAUSE project (Using Data for Improving School and Student Performance) by the consortium of partners including: Public Consulting Group, University of Twente (the Netherlands), Institute of Information Management Bremen GmbH (Germany), Modern Didactics Center (Lithuania) and Specialist Schools and Academies Trust (UK). Please visit: www.datauseproject.eu

52

http://www.datauseproject.eu/


Evaluation Plan Template

Step 1: Summarize the plan.

Who will the plan impact?

What is the desired goal/end state?

What are the strategies being implemented? (From action plan or implementation monitoring plan)

53


Step 2: Define the measurable improvement targets.

How will success be measured?

How will you know if the problem of practice was addressed?

Step 3: Identify the data that will need to be collected and analyzed to provide evidence.

Data Needed How Collected? Collected by whom? Collected by when? How stored?

54


Step 4: Explain how the data analysis will be conducted.

How will the data be summarized and presented?

Who will conduct the analysis of the data displays?

Step 5: How will inferences, findings, and conclusions be made from the evidence?

Have the goals been met?

Why were the strategies successful? Why were the strategies unsuccessful?

How can the strategies be improved?

Who will develop findings and conclusions?

55


Step 6: How will the findings and conclusions be disseminated?

What messages need to be communicated?

To whom?

When?

How?

56

Toppenish School District Superintendent John Cerna

509.865.4455 306 BOLIN DR TOPPENISH 98948Educational Service District 105

Washington Kindergarten Inventory of Developing Skills (WaKIDS)

This displays student information for the Washington Kindergarten Inventory of Developing Skills(WaKIDS). You can view details for an individual school, school district, Educational Service District,

or the entire state. For information on WaKIDS, click on the WaKIDS web site (more info).

Caution: WaKIDS is required only in state-funded full-day kindergartens. Absence of data for schoolswith enrolled kindergarten students may indicate that full-day kindergarten students in these schools

are not state-funded. For this same reason, student totals at district and state levels may appearsmaller than their actual enrollments. In districts where only some of the kindergartners took part in

WaKIDS, the results do not represent all of the kindergartners in the district.

Select a category of students: All 2013-14 Chart

* Students above the black line demonstrate characteristics of entering kindergartners

*Colors in the above chart are not associated with the color band system used in WaKIDS Teaching Strategies GOLD.

Results with fewer than 10 students are not shown. Results not shown may also indicate data not available.



9/15/2011







Select a category of students: American Indian 2013-14 Chart










Select a category of students: Limited English 2013-14 Chart










Select a category of students: Special Education 2013-14 Chart



Page 1

Sample Issue Statement & Focusing Question from Scenario

Initial broad issue

Our State is measuring the “readiness” level of entering kindergarten

students and asking schools and communities to work together to

increase readiness of children.

Focusing question Which domain of kindergarten readiness provides the greatest


What do the data say

about the focusing

question?


Hispanic students demonstrated readiness in math.


Hispanic students were ready in language.

36.2% of Hispanic students were measured as ready in all six

domains.

The scores in every domain were lowest for children with a

disability.

For all categories of students, language scores were lower than

literacy scores.


Which data informed

problem/concern was

selected for further

exploration?


Draft problem statement Less than 50% of children had basic math concepts expected at entry

to kindergarten, regardless of ethnicity, language or disability.


Page 2

Summarize Root Cause(s)

District has not communicated expectations to parents and early

learning providers.

Parents and early learning providers do not know what math skills are

expected at entry to kindergarten.

Instruction in math at preschool level is limited to counting and

identifying numerals.

Practices that could

result in the root cause of

the problem

Early learning teachers limit instruction to rote counting and one to one

correlation.

Math materials in early learning classrooms are limited due to lack of

resources.

Early learning teachers have not received professional development in

math.

There is lack of collaboration between early learning teachers in the

community and the public school teachers around math.

Final Problem of Practice:

The district teachers and early learning teachers do not have shared

understanding of the skills and concepts that entering kindergarten

students are expected to have.

Summary of Action Plan:

Goal: Increase the percentage of children ready in math from 48% to

58% in two years.

1. Joint committee of early learning and kindergarten teachers will be

formed.

2. Committee will identify kindergarten ready math concepts based upon

TS GOLD.

3. Joint professional development will be created.

Summary of Evaluation:

1. Evaluate the level of collaboration and connection between early

learning providers and the school district through a survey given.

Participants will determine if the action plan was successfully

implemented, creating relationships between early learning and school

teachers.

2. Are 58% of children scoring “ready” in the area of math on the

September 2015 WaKIDS assessment?

early learning data institute - wsa head start and eceap · 2014-04-03 · early learning data...

Documents