program evaluation & recommendation...workforce and citizenry. lead and manage in public...

Empirical evidence suggests children that lack a third grade reading proficiency are less likely to graduate high school. The Enhanced School-based Mentoring program (ESBM) is part of We Read to Succeed (WRTS), a community-wide initiative lead by Big Brothers Big

Sisters of Harrisonburg-Rockingham County and the United Way aimed to improve the number of students in Harrisonburg and Rockingham County who are reading at grade level by the end of the third grade. ESBM is a one-to-one mentoring program designed for children aged 4-8 years old who are identified by the school system as being at-risk of falling behind on reading proficiency. The following evaluation provided a needs assessment, and detailed formative and summative research results. BBBS Director, Sue Totty, selected the program recommendations to improve ESBM.

Selected Competencies

This document evaluated the effect of ESBM on participant reading attitudes,

reading skills, reading preference, major GPA, and PALS scores, and provided

evidence-based data to make recommendations for program improvement.

Completion of this project demonstrated systems-based and strategic thinking to

generate governance strategies for local and global problems. Reading

proficiency is a global issue, and has been empirically linked to larger scale

academic failure and deficiency in personal development.

This evaluation required extensive research, critical thinking, and completion of

a complex statistical analysis to evaluate ESMB and to make evidence based

recommendations based upon the critical interpretation of past and present

research, administrative practices, and budgetary processes. Evaluation results

and recommendations were presented to BBBS director, Sue Totty, and two staff

members; they selected my group’s evaluation design and recommendations to

improve ESMB outcomes.

ESBM participants were identified by teachers, school administrators, or family

members as at-risk based on demographic and socio-economic factors.

Completion of the needs assessment demonstrated my awareness of how diverse

societal values and identities intersect to inform public and nonprofit

administration and my ability to effectively communicate with, actively listen to,

and resolve conflict between diverse stakeholders by utilizing multiple

technology modes for communicating with a diverse public.

Program Evaluation & Recommendation Origin: Program Evaluation

Produced: March 2016

Communicate and

interact productively with

a diverse and changing

workforce and citizenry.

Lead and manage in

public governance.

Analyze, synthesize,

think critically, solve

problems and make

decisions.

BIG BROTHERS BIG SISTERS ENHANCED SITE-BASED MENTORING

STEVEN COXSOPHIE KWOKAMANDA REEDY BILL WYATT

CONTENTS EXECUTIVE SUMMARY 6

INTRODUCTION 8

NEEDS ASSESSMENT 9

Risk Factors 9

Harrisonburg 9

The Need 11

FORMATIVE RESEARCH 13

Measures 13

Methods 15

Results 19

SUMMATIVE RESEARCH 21

Participants 21

Design and Procedure 23

Measures 23

Reading Attitudes 23

Strength of Relationship and Reading Ability 24

Reading Interest 24

Grades 24

PALS 24

Baseline to one year 25


Mentor perceptions of reading abilities 26

Mentor reported strength of relationship. 27

Reading interest 27

Major GPA 28

PALS Scores 30

Baseline to year two. 30

Reading Attitudes. 30

Mentor Perceptions of Reading Abilities. 31

Mentor Reported Strength of Relationship. 32

Reading Interest 32

Major GPA 33

Methods 34


Reading Abilities 34

Strength of Relationship 35

Reading Interest 35

Grades 35

PALS 35

Results 36


Reading Abilities 36

Strength of Relationship 37

Reading Interest 37

Grades 38

PALS 38

Recommendations 40

APPENDIX A: FORMATIVE DATA 41

APPENDIX B: SUMMATIVE DATA 43

APPENDIX C: REFERENCES 48

BBBS: Enhanced Site-Based Mentoring

EXECUTIVE SUMMARY Big Brothers, Big Sisters’ Enhanced School-Based Mentoring program was designed in an attempt to improve reading proficiency of at-risk youth, ages 4-8 in the Harrisonburg City and Rockingham County school districts. Our area (Harrisonburg City) has the highest limited English proficiency percentage of the student body in the state, comprising one third of the student population. Along with a high percentage of free and reduced price lunch recipients as an indicator of economic stress, it is evident that o u r a r e a i s a c a n d i d a t e fo r p o t e n t i a l l y underperforming in areas of reading success. Utilizing the one-to-one mentoring system used in their community based programs, BBBS designed a program of activities focused on reading that would enable the participants to better meet the reading proficiency measures by the end of their third grade year. For this program evaluation, we wanted to look at whether the program was being implemented properly by the mentors. As the crucial component in meeting the activity guidelines, we felt it was important to determine if the mentor’s actions and self assessments were indicating achievement of those goals.

In looking at the data from the monthly check-in reports, we found that half of the mentors were meeting the 15-20 minutes per week reading activity goal. We also found that the incorporation of sending books home with students decreased as the program continued which would indicate that the first 12 months of the program was the most beneficial to participants. We also looked at several of the outcome measures, but found only that GPA and PALS scores showed a statistically significant change during the course of participation.

We found that, when data is reported, mentors are engaging in the activities with their littles as required by the program guidelines. However during any

Page 6

Figure 1. Summary of overall results.


given period, data is missing for nearly one-third of the mentors. However, this activity did not necessarily translate into the anticipated outcomes. Our recommendations include two primary areas of focus for improvement in the program. First, we would suggest that mentors be recruited and selected in a manner proportionate to the ethnic and gender composition of the littles. Secondly, we recommend that the time the mentor spends reading to their little increase to at least 30 minutes

per session. Of the seven mentors who reported reading to their little at least 30 minutes during each visit, 5 of the 7 showed increases in their major GPA from baseline to 24 months while another maintained their 4.0 major GPA. Finally, we suggest that mentors receive more interactive and rigorous training in cultural, socioeconomic and gender diversity to assist them in relating to the challenges associated with those areas.

Page 7


INTRODUCTION The Enhanced School-Based Mentoring program (ESBM) is part of We Read to Succeed (WRTS), a community-wide initiative led by the United Way of Harrisonburg and Rockingham County (UWHRC). WRTS aims to improve the number of students in Harrisonburg and Rockingham County who are reading at grade level by the end of the third grade. WRTS focuses its efforts in the areas of school readiness, school attendance, and summer learning.

WRTS is governed by an advisory council of community leaders, educators, and policy experts. The ESBM is funded through a grant from UWHRC and is administered by the Big B r o t h e r s B i g S i s t e r s o f Harrisonburg-Rockingham County.

Big Brothers Big Sisters’ mission is “to provide children facing adversity with strong and enduring, professionally supported one-to-one relationships that change their lives for the better, forever (Totty, 2016).” BBBS is a one-to-one mentoring program that works with children in adversity. For every match, there is professional staff to support the mentor and mentee. This makes BBBS unique from other mentor programs in the area. BBBS served 638 children between the ages of 4 and 14 in 2015. Of the children mentored, 58 percent were girls while 42 percent were boys. 48 percent come from a single-parent home, 9 percent have at least one parent who is incarcerated and 75 percent live at or below 125 percent of the poverty level.

The ESBM seeks to leverage those one-to-one relationship to help advance the community goal of improving reading proficiency. The program is designed for children aged 4 - 8 years old who are identified by the school system as being at risk of

falling behind on reading proficiency. “Literacy and third grade reading test scores has become a buzz term,” said Sue Totty, executive director of Big Brothers Big Sisters of Harrisonburg-Rockingham County. Totty talks about attending a BBBS national conference where she heard a talk from a juvenile justice official who showed a map of 22 states who use third grade reading proficiency to predict the amount of prison space will be necessary moving

forward (Totty, 2016).

The ESBM began in 2013 with an initial cohort of 62 students. The participants are students at Spotswood and Smithland Elementary Schools. Each student was professionally

matched with a mentor. The mentors must commit to working with their little for at least 12 months or, in the case of a college student, for three academic semesters. Mentors are required to meet with their littles each week in the child’s school or in an approved after school setting. During their time together, the mentors are asked to read to their littles for 15 - 20 minutes, infuse reading into other activities, encourage reading at home, and send a book home with the little every month. BBBS provides books for the mentors to send home.

Mentors receive training on reading techniques and resources as well as attend an orientation at the child’s school prior to meeting with their littles. The child’s teacher is kept abreast of the mentor’s activity and schedule and the mentor has monthly contact with the BBBS match support specialist.

Children are assessed based on their reading reading attitudes, reading abilities, reading preference, mentor reported strength of relationship, PALS scores and grades.

Page 8

“Literacy and third grade reading test scores has become a buzz term.”

~ Sue Totty


NEEDS ASSESSMENT

Risk Factors Children of color and of low socioeconomic status lack third grade reading proficiency at a higher rate than their peers. According to the Annie E. Casey Foundation’s Early Warning Confirmed (2013), 74 percent of children who scored below the 25th percentile on the National Assessment of Educational Progress (NAEP) live in a low income household. Further, the share of low-income black, Hispanic and Native American children who scored below proficient on the NAEP was over 85 percent for each demographic. Socioeconomic status and ethnicity are among the key indicators of children who are at risk of falling behind in reading proficiency.

Harrisonburg The City of Harrisonburg is nestled in the picturesque Shenandoah Valley of Virginia. Surrounded by Rockingham County, Virginia’s leading agricultural producer (USDA, 2012), the Harrisonburg City Public School system has a minority population of 62 percent (Virginia Department of Education, 2015). Additionally, Harrisonburg has the highest percentage of limited English proficient population of students in Virginia with 33 percent of the student population participating in the program (Virginia Department of Education, 2015). As for socioeconomic status, 70.47 percent of Harrisonburg’s students qualify for free or reduced lunch (Virginia Department of Education, 2016). Harrisonburg’s free and reduced lunch percentage is well above the state average of 62.87 percent and ranks in the top 13 percent of all Virginia school districts.

The Commonwealth of Virginia uses the Phonological Awareness Literacy Screening (PALS) to assess reading readiness. The test is part of Virginia’s Early Intervention Reading Initiative (EIRI). According to Virginia Performs (2015), the PALS test is administered for the first time during the fall of semester of a student’s kindergarten year. The results are used to identify students who may be in need of additional instruction through EIRI programs. The PALS-K assesses the following literacy skills (University of Virginia, n.d.): rhyme awareness; beginning sound awareness; alphabet knowledge; letter sounds; spelling; concept of word; and word recognition in isolation.

Students receive a score for each fundamental and then the scores are added together and compared to

Page 9

Figure 2. Location of Harrisonburg, Va.


a benchmark of where students should be. If a student’s total score is lower than the benchmark, reading intervention services are recommended.

The statewide average on the PALS kindergarten benchmark was 87.3 percent (Virginia Performs, 2015) on the screening administered in 2014. This means that 87.3 of kindergartners in Virginia met or exceeded literacy fundamentals. The Hampton Roads region of Virginia had the highest PALS-K average with 90.5 percent of students meeting or

exceeding fundamentals. The Valley region, in which Harrisonburg is located, had among the lowest percentage of students who met or exceeded literacy fundamentals. The Valley region was tied with Southside Virginia for the second lowest PALS-K scores at 84.5 percent proficient among the eight regions measured (Virginia Performs, 2015). This illustrates the need for Harrisonburg and neighboring communities to invest in school readiness and early reading intervention strategies.

The Risks

Page 10

0

20

40

60

80

% LEP Student % Free / Reduced Lunch

Harrisonburg Virginia

Figure 3. Limited English Proficient and free and reduced lunch student population of Harrisonburg vs. statewide averages.


The Need Based on Harrisonburg’s high percentage of low income, minority students, ensuring that area students are proficient in reading by third grade is a major community concern. Research suggests that students who are not proficient in reading by the third grade and who experience poverty are six times more likely to not graduate from high school (Hernandez, 2011). A community coalition lead by the United Way of Harrisonburg and Rockingham County has been created to focus on increasing third grade reading proficiency among area students. The We Read to Succeed (WRTS) coalition, whose areas of emphasis include school readiness, school attendance, and summer learning, i n c l u d e s representatives f r o m t h e H a r r i s o n b u r g C i t y P u b l i c S c h o o l s , R o c k i n g h a m County Public Schools, Big Brothers and Big Sisters, Boys and Girls Club, Massanutten Regional Library, Harrisonburg-Rockingham Chamber of Commerce, Harrisonburg Rotary Club, James Madison University, Smart Beginnings Shenandoah Valley, United Way, and AmeriCorps VISTA.

There are a number of early reading intervention programs that serve the Harrisonburg student population. While a few of these programs utilize volunteers, most are administered by trained reading specialists or certified educators. As with most school districts around the country, Harrisonburg City Schools receives federal funding for the implementation of a Head Start program. This program is designed for 4-year old children. Though the program is primarily for disabled and at-risk youth, all may apply. Head Start is a full-day, school-based program that focuses on school readiness. The program is taught by professionals.

Similarly, Harrisonburg City Schools offers the services of reading specialists at each of the school district’s five elementary schools. According to a job announcement for the 2015 - 16 school year (Harrisonburg City Schools, 2016), reading specialists are required to have a Virginia postgraduate professional license with an endorsement as a reading specialist and at least three years of successful classroom teaching experience as well as certain character traits. For the 2015 - 2016 academic year, Harrisonburg City Public Schools had 21 reading specialists who were assigned to the five elementary schools. Based on cumulative enrollment figures for those schools (Harrisonburg

City Schools, 2015 a n d V i r g i n i a Department of Education, Office o f S c h o o l Nutrition, 2015), t h e m e a n s t u d e n t t o reading specialist

ratio is 1 reading specialist for every 131 students. When you look at the ratio of students to reading specialists using known risk factors affecting school readiness and reading proficiency at the schools with the greatest need, the ratio of students to reading specialists improves. At Spotswood Elementary School who has the highest percentage of free and reduced lunch (87.56 percent) and limited English proficiency students (60 percent), the student to reading specialist ratio is 1 to 111.

Other community organizations offer literacy programs. The United Way of Harrisonburg and Rockingham County provides grants to a number of local agencies that promote literacy. These grants include:

• G e n e r a t i o n s C r o s s i n g p r o v i d e s a n intergenerational learning environment to lower income children;

Page 11


• The Harrisonburg - Rockingham Child Day Care Center and the Roberta Webb Child Care Center make high quality child care and learning opportunities available to the community on a sliding scale based on a family’s ability to pay;

• Boys and Girls Club of Harrisonburg and Rockingham County provides small group reading instruction in an after school setting.

Given the high percentage of reading proficiency risk factors that affect the area, the demand for early reading intervention programs is high. Another model that may fill a service gap is the volunteer mentor model. We know from research that a consistent, supportive relationship with mentors have shown to have positive effects on a youth’s interpersonal relationships as well as cognitive and emotional development. Elbaum & Vaughn (2000) that college students and trained, reliable community volunteers were able to provide significant help to struggling readers. Based on this research, the Big Brothers Big Sisters’ proven volunteer mentor model could likely fill a gap for the Harrisonburg community.

With more than 30,000 college students attending schools within a 10-mile radius of Harrisonburg, Big Brothers Big Sisters of Harrisonburg and Rockingham County enjoys a potential volunteer base that could help the organization address the community’s early reading intervention need. In 2012, more than 70 percent of the Big Brothers Big Sisters mentors were James Madison University students (Young, 2013).

Page 12


FORMATIVE RESEARCH

Measures The current paper aimed to investigate whether the implementation of the program activities in the Enhanced School-Based Mentoring program (ESBM) was successful from the perspective of mentors. In other words, were the mentors doing what they were required to do with their littles during their meeting time? We felt that the mentor held the crucial position in the implementation process and had the most direct impact on determining whether the program activities with the participant were having an observable effect on the reading proficiency of the participant through various measurement instruments. Data was collected from both mentors and littles of the mentoring program in the form of survey. This data was then compiled by the lead project evaluator and made accessible for our classroom project. In order to capture the activities that participants were carrying out during the tutoring time, we examined the monthly check-in data, both quantitative and qualitative. The original plan was to divide the data into three cohorts: early match (1-4 months), early match (5-8 months), and late match (9-12 months). However, the fact that there were no data for the first three months made it difficult to create the early match category. Instead, we divided the monthly check-in into three periods: early match, mid match and long term match. Early match included all the responses reported by mentors for the first six months, while mid match was comprised of the monthly data from months seven through twelve. Long term match included all responses from mentors during the second year of the program. The primary purpose of dividing the monthly check-in data into three cohorts was to spread the observation and be able to make comparisons.

Page 13

Figure 4. Summary of formative research results related to mentor activity.


Page 14

Figure 5. Analysis of early match mentor. comments.

Figure 6. Analysis of mid match mentor comments.


We selected five of the monthly check-in activities as our performance indicators:

• Frequency Reading: How often did the mentors read with their littles?

• Average Time Reading: How long did the mentors read to their littles during each session?

• Quantity of Books Read: How many books did the mentor read to their little each session?

• Frequency of books sent home: How frequently did the mentor send home books with their little each month?

• Additional comments: Monthly comments from mentors reporting on their interactions with their littles.

Methods Average Time Reading. The dependent variable is the average time reading in the ESBM program. In order to capture the trend of the average time reading from the early match, we computed a mean of these three months’ reading time by using SPSS and named it as Mean_AverageTimeReading_EarlyMatch. Subsequently, we ran descriptive statistics for the mean created and each month average time reading. Table 4 showed that the average reading time of the first month (20.429; sd=6.399) and the second month (20.328; sd=7.21) were similar as the mean of three months together (20.388; sd=6.823). The average reading time of the third month (18.902; sd=5.186) was a little off compared with others. The standard deviation measured the dispersion of the sample by looking at how far observations are from the mean. A standard deviation that was closer to 0 indicated that data points tend to be very close to the mean of the set, while a high standard deviation indicated that data points were spread out over a

Page 15

Figure 7. Analysis of long term match mentor comments.


wider range of values. All four standard deviations showed that the minutes of reading time were relatively further away and spread out from the average reading time.

The average time reading data is continuous. We used paired sample T-test to observe the change over time between early match and mid match.

Quantity of Books Read.To determine if participant quantity of books read (QBR) changed overtime, a mode QBR value was calculated at early match, mid match, and long term match. The dependent variable mode QBR value is categorical and is parametric and correlated since the same mentor responded. The independent variable is time, which has three levels (early match, mid match, and long term match). Based on the variable characteristics, a Wilcoxon signed-rank test is the appropriate test of significance. The null hypothesis (Hₒ = MD = 0) states there is no change overtime between early match mode QBR scores and mid match mode QBR values, the alternative hypothesis (Hᴬ = MD ≠ 0) states that there will be a change overtime between the early match and mid match QBR values. In order for a Wilcoxon signed-rank test to meet statistical validity requirements, the test must meet four assumptions: the dependent variable should be measured at the ordinal level, the independent variable should consist of at least two categorical “related groups”, the paired observations for each participant need to be independent, and the difference scores should be from a continuous distribution.

Frequency of Books Sent Home. To determine whether books were being sent home by the mentors as designed, we looked to see if there was a change over time. The dependent variable of “SentBookHome” was categorical and was coded: 1=Never, 2=Once a month, 3=Once every other month. This variable is non-parametric and correlated since the same mentor responded. The independent variable is time, which has three levels

(early, mid, and long-term). Based on the variable characteristics, a Wilcoxon signed-rank test is the appropriate test of significance. The null hypothesis (Hₒ = MD = 0) states there is no change overtime between early match number of books sent home and that number at mid and long term match, the alternative hypothesis (Hᴬ = MD ≠ 0) states that there will be a change overtime between the early and mid match values for number of books sent home. In order for a Wilcoxon signed-rank test to meet statistical validity requirements, the test must meet four assumptions: the dependent variable should be measured at the ordinal level, the independent variable should consist of at least two categorical “related groups”, the paired observations for each participant need to be independent, and the difference scores should be from a continuous distribution.

Reading Frequency. Most of the data stayed the same. The results showed that the reading frequency did not have any significant change throughout the 2 years mentoring program. In other words, the program did not affect the reading frequency of participants.

Early Match and Mid Match. 11 mentors reported that the mode of early match for reading frequency is smaller than the mode of mid match for reading frequency; 4 mentors reported that the mode of mid match is greater than the mode of early match; and 29 mentors reported that the mode of mid match equals to the mode of early match.

Early Match and Long Term Match. 14 mentors reported that the mode of long term match for reading frequency is smaller than the mode of early match for reading frequency; 5 mentors reported that the mode of long term match is greater than the mode of early match; and 27 out of 46 mentors reported that the mode of long term match equals to the mode of early match. The fact that over half of the mentors reported no difference in mode between early match and long term match

Page 16


suggested that matches lasted more than 6 months did not have impact on the reading frequency of children.

Mid Match and Long Term Match. 9 mentors reported that the mode of long term match for reading frequency is smaller than the mode of mid match for reading frequency; 9 mentors reported that the mode of long term match is greater than the mode of mid match; and 29 out of 29 mentors reported that the mode of long term match equals to the mode of mid match. Two third of the mentors reported no difference in mode between mid match and long term match indicated that reading frequency of children did not improve in the second year.

A Wilcoxon Signed Rank-Sum test showed that there was a significant change in mentor reported reading frequency between the early match and mid match (Z=-2.085; p=0.037), and early match and long term match (Z=-2.158; p=0.031). However, there was no significance observed in mentor reported reading frequency between mid match and long term match (Z=-0.45; p=0.964).

Therefore, we were able to conclude that 1) there was a significant improvement in children’s reading frequency in the first 6 months after attending the

mentoring program; 2) the reading frequency of participants started to decrease after 6 months because the mode of reading frequency in mid match was smaller than in the early match; 3) there was no significant improvement in children’s reading frequency in the long term match.

Books Read by Children at Home. Although the majority of the children read 4 books per month throughout the program, some children have read up to 32 books per month at home. However, the number of books read dropped half to 15 in the second six months of the program. Interestingly, the data increased to 32 in the long term match.

Early Match and Mid Match. 20 mentors reported that the mode of mid match for quantity of books read is smaller than the mode of early match for quantity of books read. 18 mentors reported that the mode of mid match is greater than the mode of early match; and 5 mentors reported that the mode of mid match equals to the mode of early match. The numbers showed us that children from early match did slightly better than children from mid match.

Early Match and Long Term Match. 25 mentors reported that the mode of long term match for quantity of books read is smaller than the mode of early match for quantity of books read; 15 mentors reported that the mode of long term match is greater than the mode of early match; and 2 mentors reported that the mode of long term match equals to the mode of early match. There was significantly more mentors reported impressive reading proficiency of children in the early match than in the long term match.

Mid Match and Long Term Match. 21 mentors reported that the mode of long term match for quantity of books read is smaller than the mode of mid match for quantity of books read; 14 mentors reported that the mode of long term match is greater than the mode of mid match; and 9 mentors

Page 17

Figure 8. Statistically significant changes in reading frequency.


reported that the mode of long term match equals to the mode of mid match.

Wilcoxon Signed Rank-Sum test showed that while there was no significant change in mentor reported quantity of books read by the children between the early match and mid match (Z=-1.098; p=0.272), significant changes were observed between early match and long term match (Z=2.552; p=0.011), and mid match and long term match (Z=-2.386; p=0.017). Figure 9. Showed whether there was a significant change among the matches.

We were able to conclude that: 1) participants from the early match performed better in terms of reading more books at home than participants from mid match; 2) participants from the early match performed better in terms of reading more books at home than participants from long term match; 3) the improvement will not be further enhanced in the second year of the program.

Frequency of Books Sent Home. Early Match and Later match. 13 mentors reported that the mode of mid match for number of books sent home with

children is smaller than the mode of early match for number of books sent home with children; 9 mentors reported that the mode of mid match is greater than the mode of early match; and 20 out of 42 mentors reported that the mode of mid match equals to the mode of early match. The fact that almost half of the mentors reported no change in these two matches leads one to believe that, when data is reported, the mentors are doing what they’ve been instructed to do.

Early Match and Long Term Match. 18 mentors reported that the mode of long term match for number of books sent home with children is smaller than the mode of early match for number of books sent home with children; 6 mentors reported that the mode of long term match is greater than the mode of early match; and 18 mentors reported that the mode of long term match equals to the mode of early match. Three times more mentors reported better performance in the early match than in the long term match confirmed the effectiveness of the first 6 months of the mentoring program.

Mid Match and Long Term Match. 11 mentors reported that the mode of long term match for frequency of books sent home with children is smaller than the mode of mid match for frequency of books sent home with children; 8 mentors reported that the mode of long term match is greater than the mode of mid match; and 28 out of 47 mentors reported that the mode of long term match equals to the mode of mid match. More than half of the mentors reported no change between mid match and long term match indicated that, when data is reported, the mentors continue to send books home with the children in accordance with program guidelines.

The significant value (Z=-2.306; p=0.021) between the mode of long term match and early match indicated that we should reject the null that there was no significant difference. In other words, the frequency of books sent home with matches in the

Page 18

Figure 9. Statistically significant changes in the quantity of books sent home.


first 6 months was significantly larger than the frequency of books sent home with children with matches in the second year of the program. In addition, the Wilcoxon Signed Rank-Sum test proved that there was no significant change between early match and mid match (Z=-0.695; p=0.487), and mid match and long term match (Z=-0.513; p=0.608).

Results Early Match. 53 mentors reported that the average time reading of children in the program for the first 6 months was 21 minutes. Standard deviation that is close to zero indicates the data points tend to be very close to the mean of the set. Therefore, the standard deviation of 6.91 showed that the reading time of the sample tends to spread out from the average time. The minimum reading time for this group was 5 minutes and the maximum reading time was 40 minutes.

55 mentors reported that the majority of the participants read 5 times per month during the first 6 months and the middle number was also 5 times. The minimum times children read was 2 and the maximum times they read was 5.

52 mentors reported that most of the children read 4 books per month at home during the mid-match and the middle number of books read by the children at home was 6. The minimum quantity of books read at home was 0 and the maximum number was 30.

31 mentors reported that books were sent home every month. 14 mentors sent books home every other month. 4 mentors reported not send books home at all.

Mid Match. Table 2 showed that 48 mentors reported the average reading time for this mid match group was 20 minutes and the overall data points spread wide from the mean score (sd=6.17). The minimum minutes the children read were 12.5 and the maximum minutes was 45.

48 mentors reported that most of the children read 5 times during the first 6 months and the middle number was also 5 times. The minimum times the children read was once and the maximum times they read was 5.

48 mentors reported that most of the children read 4 books at home during the Mid Match and the middle number of books read by the children at home was 5. The minimum quantity of books read at home was once and the maximum number was 15.

33 mentors reported sending books home every month. 12 mentors reported sending books home every other month and 5 mentors reported never sending books home.

Long Term Match. Table 2 showed that 50 mentors reported that the average reading time for this long term (second year) match group was 20.1 minutes and the overall data points spread wide from the mean score (sd=6.76). The minimum minutes the children read were 10 and the maximum minutes was 45.

50 mentors reported that most of the children read 5 times in the second year and the middle number

Page 19

Figure 10. Statistically significant changes in frequency of books sent home.


was also 5 times. The minimum times the children read was once and the maximum times they read was 5.

50 mentors reported that most of the children read 4 books at home and the middle number of books read by the children at home was 4. While some participants read zero book during the second year, some other participants read 32 books.

38 mentors reported sending a book home every month. 7 mentors reported sending books home every other month and 5 mentors reported never sending any books home.

Average Time Reading. By looking at the average time reading across all three matches, the average reading minutes stayed the same. The standard deviations showed that the data were not concentrated around the average reading minutes. However, the minimum reading time increased 7.5 minutes in the mid match and dropped back to 10 minutes in the long term match. The results suggested that the mentoring program has had an impact on the reading time of children after being matched with the mentors for 6 months.

Page 20


SUMMATIVE RESEARCH Participants The Harrisonburg-Rockingham Big Brothers Big Sisters program partnered with the United Way to develop the Enhanced School-Based Mentoring Program (ESBM). ESBM is a semi-structured mentoring intervention for kindergarten through second grade children identified by their school as being at-risk of not reading at a proficient level by third grade. The study included 62 children from the Harrisonburg City Public School District (32 of the children were from Smithland Elementary School, and 30 were from Spotswood Elementary School) all of whom applied to the Harrisonburg-Rockingham County Big Brothers Big Sisters program in 2013, and

were assigned to the ESBM site-based intervention group by agency staff. Mentors and the child participants were surveyed at baseline, 12 months and 24 months.

Over half of the ESBM participants were boys (51.6%), sixty-three percent of the children were Hispanic, 19.3% were white, 12.3% were African American and 3.5% were Asian. Participants ranged in age from 5 to 8 (M = 6.290; s = .9301), most (90.3%) of whom were between the ages of 5 and 7, and 26 of the children were in first grade, 18 were in kindergarten and 18 were in second grade. Only 5.5% of the children had a physical disability, 11.8% had learning disabilities and 9.6% were reported as having an emotional disorder. Almost all of the

Page 21

Figure 11. Ethnicity of ESBM student and mentor participants.


children were reported as not having a chronic illness (96.3%) or a history of child abuse (97.6%), and 67.7% had not been in a juvenile detention center. More than half of the children in the ESBM program came from single, separated or divorced households (55%) and 86.7% had a family size of 2 to 5. Eighty-four percent of the children’s mothers had a high school education or less, 93.8% of the children’s fathers had a high school education or less, and 91.5% of the participants lived in a household with an income level of $29,999 or below. 42 of the 62 participants reported LEP status (67.7%). The other 20 participants did not report yes or no, so the data is considered missing. Of those that reported; 30 were girls and 12 were boys. Sixty-three percent of the participants’ guardians reported receiving food stamps, 69.4% reported receiving Medicaid assistance for the child and 61.3% of the children

received free or reduced lunch. Eighty-eight percent of the guardians reported that they were renting their home and 21.6% reported they had been unemployed in the last 6 months. Only 21.4% of the children participated in after school programs, 17.5% participated in literacy programs, 17.9% participated in food programs, 8.8% participated in after school tutoring and 91.2% of the guardians reported they did not participate in parenting classes.

Eighty-eight percent of the ESBM mentors were female, 93.5% were white, 3.2% were Hispanic, and 3.2% were Asian. Ninety-three percent of the mentors were college students, 91.8% attended James Madison University and over half (50.8%) had an anticipated graduation date of 2016. The mentors ranged in age from 18 to 66 (M = 21.710; s = 8.3340), most (95.2%) of whom were between the ages of 18 and 23. All of the mentors reported their primary

Page 22

Figure 12. Gender of ESBM student and mentor participants.


language as English, 8 of the mentors reported Spanish as a second language and 4 mentors reported “other” as a second language. Table 8 illustrates the demographic profile of the child participants and the mentors.

We hypothesized that mentoring would directly impact participants’ reading attitudes, reading abilities, reading preference, mentor reported strength of relationship, PALS scores and grades (major GPA). To test this hypothesis, SPSS statistical software was used to test for group differences from baseline to 12 months and from baseline to 24 months.

Design and Procedure The Harrisonburg-Rockingham Big Brothers Big Sisters agency and children from the Harrisonburg Public school district and Rockingham County Public school district were selected to participate in the study. The key selection criteria for inclusion in the study were kindergarten through second grade children that were identified by their school as being at-risk of not reading at a proficient level by third grade. All of the children enrolled in the 2013 BBBS mentoring program were encouraged to participate in the study. After being notified of the study, if the child agreed to participate and received parental consent, he or she was assigned randomly to either the ESBM intervention or to the less structured BBBS mentoring program. Parents were required to complete a guardian enrollment form which outlined religious affiliation, race, sexual orientation and marital status and were also asked to report any family issues that may impact a child’s match (i.e. physical disability, learning disability, behavioral or emotional issues, history of physical, sexual or emotional abuse, and issues with police). In addition to answering personal questions about their child, parents were also asked to identify if they received any type of public assistance, to provide their income level and to identify any substance

abuse, health issues, criminal history, or mental illness experienced in the family.

Agency staff matched volunteer mentors with children based on shared interests and same-race match preference. The major systematic differences between the mentoring programs, was that mentors in the more structured ESBM intervention, received vigorous training and support, used reading interventions with the children and were required to participate in a specific number of reading activities with their little each week.

Measures Reading Attitudes The Elementary Reading Attitude Survey is a 10-item scale containing questions related to attitude toward recreational reading or attitude toward academic reading, with each item beginning uniformly with “How do you feel…”, “How do you feel about reading for fun at home”, “How do you feel when you read a book on a rainy Saturday”, “How do you feel when you read a book in school during free time”. The pictorial image of Garfield was selected because of his familiarity and the comprehensibility of young children. Match specialists collected survey responses from the children at baseline (3 months), at twelve months and again at 24 months. The original assessment is coded on a 4-point scale, ranging from “very satisfying” (1) to “very dissatisfying” (4). However, for the purposes of this study the responses were recoded on a 4-point scale, ranging from “very satisfying” (4) to “very dissatisfying” (1). Cronbach’s α reliability coefficients of the subscales ranged from .74 to .89 (McKeena & Kear, 1990). A baseline sum reading attitudes score, 12 month sum reading attitudes score and 24 month score was calculated by summing the baseline, 12 month and 24 month scores using SPSS statistical software.

Page 23


Strength of Relationship and Reading Ability The Mentor Strength of Relationship Survey is a 28-item scale used nationally by Big Brothers Big Sisters to measure the strength of the mentor/mentee relationship and the extent to which the mentorship impacts mentees reading abilities and engagement. Match specialists administered the survey to mentors at baseline, twelve months and again at twenty-four months. Mentors were asked to report whether they agree with each statement regarding their “Little” for example “I am enjoying the experience of being a Big”, “My Little and I are interested in the same things”, “I feel confident handling the challenges of being a mentor”, and “I feel close to my little” responses were coded on a 6-point Likert scale, ranging from “strongly disagree” (1) to “strongly agree” (5) and included an “I don’t know” option. Questions 1-14 pertain to the mentor’s reported strength of the relationship and eleven of the statements are related to the Little’s reading skills or interest in reading (items 18 - 28). For the purposes of this study, the evaluation team added the last 11 items to the nationally used survey. Items, 2, 5, 6, 8, 9, 10, 12, 13, 22 and 26 were re-coded due to negative variance for example “I expected that being a mentor would be more fun than actually it is”, “Being a big is more of a time commitment than I anticipated”, or “I feel overwhelmed by my little’s family difficulties” responses were coded on a 6-point Likert scale, ranging from “strongly agree” (1) to “strongly disagree” (5) and included an “I don’t know” option. A baseline mean mentor reported strength of relationship score, 12 month mean mentor reported strength of relationship score and 24 month mean score was calculated using SPSS statistical software. Similarly, a baseline mean mentor reported perception of reading interest score was calculated in SPSS using survey items 18 – 28 at baseline, twelve months and twenty-four months.

Reading Interest The Activity Preference Questionnaire is a 10-item scale used to demonstrate preference between doing one of two activities. Match specialists administered the survey to child participants at baseline, twelve months and again at twenty-four months. Children were asked to choose the activity they prefer by placing a check next to the activity of choice and were instructed to only select one. Example questions include “I would rather watch a television program of my choice or read a book of my choice”, “I would rather read a book of my choice or play an outdoor sport of my choice”, “I would rather read a book of my choice or listen to music of my choice”. Responses were coded (1) activity selected or (2) activity not selected. Only items 2, 3, 5, 7 and 10 of the questionnaire pertain to reading and indicate whether the child had an interest in reading, each item was recoded to (1) ‘read a book’ or (0) ‘other activity’. A baseline mean mentor reported perception of reading score, twelve month mean mentor reported perception of reading score and twenty-four month mean score was calculated using SPSS statistical software.

Grades Major GPA information was provided to BBBS by the child’s elementary school and was determined by the child’s performance in the major subjects (Language Arts, Social Science, Math and Science). Major GPA information was collected from both schools by match specialists at baseline, twelve months and at twenty-four months. The grade scale used was as follows (A+ = 4.3 GPA, A = 4.0 (E), A- = 3.67, B+ = 3.33, B = 3.00 (S), B- = 2.67, C+ = 2.33, C = 2.00 (N), C- = 1.67, D+ = 1.33, D = 1.00 (U), D- = 0.67 and F = 0). A baseline mean major GPA, twelve month mean major GPA and twenty-four month mean major GPA was calculated using SPSS statistical software.

PALS The Phonological Awareness Literacy Screening (PALS) was created at the University of Virginia in

Page 24


1997 and provides a comprehensive assessment of children’s knowledge of the important literacy fundamentals that are predictive of future reading success. PALS is the state-provided screening tool for Virginia’s Early Interventional Reading Initiative and is used by 99% of school divisions in the state, there are three instruments PALS-PreK (for preschool students), PALS-K (for kindergartners), and PALS 1-3 (for students in Grades 1-3). For the purposes of this study we looked at the PALS-K and PALS 1-3. The PALS-K focuses on rhyme awareness, beginning sound awareness, alphabet knowledge, letter sounds, spelling, concept of word and word recognition in isolation. PALS 1-3 is a continuation of PALS-K, and also assesses passage reading, timing passage readings, comprehension (oral reading) and fluency. Student’s scores on specific tasks are added together to create a summed score for each of the assessments, the summed score is compared against a benchmark that represents minimum expectations for fall and for spring. PALS scores are reported on a 4-point Likert scale (1 = intensive, 2 = strategic, 3 = benchmark, and 4 = high benchmark) students that score intensive display the highest difficulty in reading comprehension, and students that score high benchmark are reading at the expected reading proficiency or better. PALS scores were provided to BBBS by the child’s elementary school at baseline, twelve months and at twenty-four months. Using SPSS statistical software, a baseline mean PALS score, twelve month mean PALS score and twenty-four month mean PALS score was calculated.

Baseline to one year Reading Attitudes To determine if participant’s reading attitudes changed overtime a cumulative reading attitudes score was calculated using the sum of the 10 Reading Attitudes Survey items at baseline and 12 months. The dependent variable cumulative reading attitude is continuous, and the independent variable is time which has two levels (baseline and 12 months), the

variable is parametric, and correlated since the same mentor responded. Based on the variable characteristics, a paired T-test is the appropriate test of significance. The null hypothesis (Hₒ = µ1 = µ2) states there is no difference between the mean baseline and mean 12 month response scores, the alternative hypothesis (Hᴬ = µ1 ≠ µ2) states that there will be a difference between the baseline and 12 month mean reading attitudes. In order for a Paired T-Test to meet statist ical val idity requirements, the test must meet four assumptions: one variable is continuous, and the other is dichotomous, the two distributions have equal variances, the observations are independent and the last assumption is the two distributions are normally distributed. The Paired T-Test met all of the test assumptions, the variable cumulative baseline reading attitudes (n = 62) had a mean score of 31.38 (s = 6.10), median score of 32.00, a range of 21.00 with a minimum score of 19.00 and maximum score of 40.00. The results of the informal test of normality indicated a negative skewness (-.292) and a negative kurtosis value (-.957). Cumulative reading attitudes at 12 months (n = 53) had a mean score of 32.37 (s = 6.27), median score of 33.00, a range of 28.00 with a minimum score of 12.00 and maximum score of 40.00. The results of the informal test of normality indicated a negative skewness (- 1.265) and a positive kurtosis value (1.684), both variables were normally distributed and we can assume equal variance. At baseline 32 of the participants reported a mean cumulative reading attitudes score of 31 or higher, with 33.9% reporting a score of 35 or higher. Thirty-five of the participants at 12 months reported a mean cumulative reading attitudes score of 32 or higher, with 39.6% reporting a score of 35 or higher (5.7% more of the participants at 12 months reported a score of 35 or higher) (Table 9). The results of the analysis included 53 observations with a mean cumulative reading attitudes score of 31.88 (s = 6.04) at baseline and a mean cumulative reading attitudes score of 32.37 (s = 6.27) at 12 months. The 12 month reported cumulative reading attitude

Page 25


scores were not significantly different than the baseline reported cumulative reading attitudes (t= -.475; p= .637). The null hypothesis was accepted; there was not a significant difference in cumulative reading attitudes at baseline and 12 months. Since the significance value for change in cumulative reading attitudes from baseline to 12 months is greater than 0.05, we can conclude that the average change in reading attitude scores of -.49057 may be due to chance variation and cannot be attributed to the ESBM program.

Mentor perceptions of reading abilities In order to determine if participant’s reading skills changed overtime a mean mentor reported reading skills score was calculated using items 18 - 28 of the Mentor Strength of Relationship Survey at baseline and 12 months. The dependent variable mean mentor reported reading skills is continuous and the independent variable time, has two levels (baseline and 12 months), and is parametric and correlated since the same mentor responded. Based on the variable characteristics, a paired t-test is the appropriate test of significance. The null hypothesis (Hₒ = µ1 = µ2) states there is no difference between the mean baseline and mean 12 month mentor reported reading scores, the alternative hypothesis (Hᴬ = µ1 ≠ µ2) states that there will be a difference between the baseline and 12 month mean reading skill scores. During the initial analysis, the Paired T-Test failed to meet all of the test assumptions. The variable mean mentor reported reading skills at baseline (n = 59) had a mean score of 3.636 (s = .544), median score of 3.636, a range of 3.18 with a minimum score of 1.36 and maximum score of 4.55. The results of the informal test of normality indicated a negative skewness (-1.244) and a positive kurtosis value (3.895). Most psychometric uses consider a skewness and kurtosis score less than +/-2 acceptable, indicating the variable doesn’t deviate in symmetry from the mean. A kurtosis value higher than 2 indicates that the variable is not normally

distributed, the first step in transforming an abnormal variable is to check for outliers or unlikely responses. A z-score transformation was performed in SPSS; using a z-score of 1.96 ensured with 95% confidence identification of all outliers. The case with the lowest response score (1.36) and a z-score of -4.177 was the furthest from the mean and removed. After removing the outlier and rerunning the normality tests, the variable mean mentor reported reading skills at baseline (n = 58) had a mean score of 3.675 (s = .457), median score of 3.636, a range of 1.91 with a minimum score of 2.64 and maximum score of 4.55. The results of the informal test of normality indicated a negative skewness (-.170) and a negative kurtosis value (-.625). The variable mean mentor reported reading skills at 12 months (n = 53) had a mean score of 3.700 (s = .481), median score of 3.636, a range of 2.36 with a minimum score of 2.36 and maximum score of 4.73. The results of the informal test of normality indicated a negative skewness (- .487) and a positive kurtosis value (.438), both variables were normally distributed and we can assume equal variance. At baseline 17 mentors (29.2%) reported a mean reading skills score of 4 or higher (with 4 indicating the mentor “agreed” the little’s reading skills were improving). At 12 months 18 mentors (34%) reported a mean reading skills score of 4 or higher, 4.8% more mentors reported a mean reading skills score of 4 or higher at 12 months than at baseline (Table 10). The results of the analysis included 51 observations with a mean mentor reported reading skills score of 3.6763 (s = .444) at baseline and a mean mentor reported reading skills score of 3.7153 (s = .483) at 12 months. The 12 month mentor reported reading skills were not significantly different than baseline mentor reported reading skills (t= -.529; p= .599). As per the results, the null hypothesis was accepted; there was not a significant difference in mean mentor reported reading skills at baseline and 12 months. Since the significance value for change in mean mentor reported reading skills from baseline to 12 months was greater than 0.05,

Page 26


we can conclude that the average change in mentor reported reading skill scores of -.03904 was due to chance variation and cannot be attributed to the ESBM program.

Mentor reported strength of relationship. To determine if the strength of the mentorship changed overtime a mean mentor reported strength of relationship score was calculated using items 1 to 14 of the Mentor Strength of Relationship Survey at baseline and 12 months. The dependent variable is mean mentor reported strength of relationship and is continuous. The independent variable is time which has two levels (baseline and 12 months), the variable is parametric and correlated since the same mentor responded. Based on the variable characteristics, a paired t-test is the appropriate test of significance. The null hypothesis (Hₒ = µ1 = µ2) states there is no difference between the mean baseline and mean 12 month mentor reported strength of relationship, the alternative hypothesis (Hᴬ = µ1 ≠ µ2) states that there will be a difference between the baseline and 12 month mean mentor reported strength of relationship scores. The Paired T-Test met all of the test assumptions, the variable mean mentor reported strength of relationship at baseline (n = 60) had a mean score of 4.11 (s = .379), median score of 4.07, a range of 1.64 with a minimum score of 3.21 and maximum score of 4.86. The results of the informal test of normality indicated a negative skewness (-.275) and a negative kurtosis value (-.289). Mean mentor reported strength of relationship at 12 months (n = 55) had a mean score of 4.14 (s = .422), median score of 4.14, a range of 2.14with a minimum score of 2.79 and maximum score of 4.93. The results of the informal test of normality indicated a negative skewness (-.782) and a positive kurtosis value (.985), both variables were normally distributed and we can assume equal variance. At baseline 43 mentors (71.1%) reported a mean mentor reported strength of relationship score of 4 or higher (with 4 indicating

the mentor “agreed” the little and big were a ‘good match’). At 12 months 41 mentors (74.5%) reported a mean mentor reported strength of relationship score of 4 or higher, 3.4% more mentors reported a mean mentor reported strength of relationship score of 4 or higher at 12 months than at baseline (Table 11). The results of the analysis included 55 observations with a mean mentor reported strength of relationship score of 4.13 (s = .380) at baseline and a mean mentor reported strength of relationship score of 4.14 (s = .422) at 12 months. The 12 month mean mentor reported strength of relationship scores were not significantly different than baseline mean mentor reported strength of relationship scores (t= -.098; p= .922). The null hypothesis was accepted; there was not a significant difference in mean mentor reported strength of relationship at baseline and 12 months. Since the significance value for change in mean mentor reported strength of relationship from baseline to 12 months was greater than 0.05, we can conclude that the average change in mentor reported strength of relationship of -.00609 is due to chance variation and cannot be attributed to the ESBM program.

Reading interest To determine if participant reading interest or preference changed overtime, a mean reading interest score was calculated using items 2, 3, 5, 7 and 10 of the Activity Preference Questionnaire at baseline and 12 months. The dependent variable mean reported reading interest is continuous and the independent variable time, has two levels (baseline and 12 months) and is parametric and correlated since the same participant responded. Based on the variable characteristics, a paired t-test is the appropriate test of significance. The null hypothesis (Hₒ = µ1 = µ2) states there is no difference between the mean baseline and mean 12 month reported reading interest, the alternative hypothesis (Hᴬ = µ1 ≠ µ2) states that there will be a difference between the baseline and 12 month mean reported reading interest scores. The Paired T-Test

Page 27


met all of the test assumptions, the variable mean reported reading interest at baseline (n = 62) had a mean score of .4089 (s = .298), median score of .4000, a range of 1.00 with a minimum score of .00 and maximum score of 1.00. The results of the informal test of normality indicated a positive skewness (.215) and a negative kurtosis value (-1.138). Mean reported reading interest at 12 months (n = 53) had a mean score of .4604 (s = .322), median score of .4000, a range of 1.00 with a minimum score of .00 and maximum score of 1.00. The results of the informal test of normality indicated a positive skewness (.115) and a negative kurtosis value (-.995), both variables were normally distributed and we can assume equal variance. At baseline only 2 participants (3.2%) reported a mean score of 1.0; (a score of 1 indicated they preferred reading over another other activity. At 12 months 6 participants (11.3%) reported a mean score of 1.0, 8.1% more participants reported greater reading preference at 12 months than at baseline (Table 12). The results of the analysis included 53 observations with a mean reported reading interest score of .4104 (s = .303) at

baseline and a mean reported reading interest score of .4604 (s = .322) at 12 months. The 12 month mean reported reading interest scores were not significantly different than baseline mean reported reading interest scores (t= -1.006; p= .319). The null hypothesis was accepted; there was not a significant difference in mean reported reading interest at baseline and 12 months. Since the significance value for change in mean reported reading interest from baseline to 12 months was greater than 0.05, we can conclude that the average change in reported reading interest of -.05000 was due to chance variation and cannot be attributed to the ESBM program.

Major GPA T o d e t e r m i n e i f p a r t i c i p a n t g r a d e s c h a n g e d o v e r t i m e , mean major GPA was calculated at baseline and 12 months. The

Page 28

Mean Major GPA

Baseline

12 Months

2.9 3 3.1 3.2 3.3

3.215

3.023

Figure 13. Statistically significant increase in major GPA from baseline to 12 months.


dependent variable mean major GPA is continuous and the independent variable time, has two levels (baseline and 12 months) and is parametric and correlated since the same mentor responded. Based on the variable characteristics, a paired t-test is the appropriate test of significance. The null hypothesis (Hₒ = µ1 = µ2) states there is no difference between the mean baseline and mean 12 month major GPA, the alternative hypothesis (Hᴬ = µ1 ≠ µ2) states that there will be a difference between the baseline and 12 month mean major GPA. The Paired T-Test met all of the test assumptions, the variable mean major GPA at baseline (n = 59) had a mean score of 3.00 (s = .440), median score of 3.00, a range of 2.00 with a minimum score of 2.00 and maximum score of 4.00. The results of the informal test of normality indicated a negative skewness (-.291) and a positive kurtosis value (.046). Mean major GPA at 12 months (n = 57) had a mean score of 3.21 (s = .462), median

score of 3.25, a range of 2.00 with a minimum score of 2.00 and maximum score of 4.00. The results of the informal test of normality indicated a negative skewness (-.513) and a positive kurtosis value (.065), both variables were normally distributed and we can assume equal variance. At baseline 36 participants (61%) reported a mean major GPA score of 3.0 or higher and 1 participant had a mean major GPA of 4.0. At 12 months 45 participants (78.9%) reported a mean major GPA score of 3.0 or higher and 3 participants (5.3%) reported a mean major GPA of 4.0. Seventeen percent more participants reported a 3.0 or higher major GPA at 12 months than at baseline (Table 13). The results of the analysis included 57 observations with a mean major GPA of 3.02 (s = .427) at baseline and a mean major GPA of 3.21 (s = .462) at 12 months. The 12 month mean major GPA was significantly different than baseline mean major GPA (t= -3.516; p= .001) (Figure 13). The

Page 29

Participant Mean PALS Scores Baseline to 12 Months

0

7

14

21

28

Increase Decrease No Change

15

10

27

Figure 14. Participant mean PALS scores from baseline to 12 months.


null hypothesis was rejected; there was a significant difference in mean major GPA at baseline and 12 months. Since the significance value for change in mean mentor reported reading interest from baseline to 12 months was less than 0.05, we can conclude that the average change in mean major GPA of -.19152 is not due to chance variation and can be attributed to the ESBM program.

PALS Scores T o d e t e r m i n e i f p a r t i c i p a n t P A L S s c o r e s c h a n g e d overtime, a mean PAL Score was calculated at baseline and 12 m o n t h s . T h e elementary schools were unable to provide 24 month PALS data to determine if there was a change in scores from baseline to year two. The dependent variable mean PALS score is ordinal with four levels (1 = intensive, 2 = strategic, 3 = benchmark and 4 = high benchmark) and is parametric and correlated since the same mentor responded. The independent variable is time, which has two levels (baseline and 12 months). Based on the variable characteristics, a Wilcoxon signed-rank test is the appropriate test of significance. The null hypothesis (Hₒ = MD = 0) states there is no change overtime between baseline mean PALS scores and 12 month mean PALS scores, the alternative hypothesis (Hᴬ = MD ≠ 0) states that there will be a change overtime between the baseline and 12 month mean PAL score. In order for a Wilcoxon signed-rank test to meet statistical validity requirements, the test must meet four assumptions: the dependent variable should be measured at the ordinal level, the independent variable should consist of at least two categorical “related groups”, the paired observations for each participant need to be independent, and the difference scores should be from a continuous distribution. The Wilcoxon signed-rank test met all

the test assumptions, the variable mean PAL score at baseline (n = 59) had a mode score of 1.00 median score of 2.00, a range of 2.00 with a minimum score of 1.00 and maximum score of 3.00. The variable mean PAL score at 12 months (n = 55) had a mode score of 2.00, median score of 2.00, a range of 3.00 with a minimum score of 1.00 and maximum score of 4.00. The Wilcoxon signed-rank test showed that ESBM participants elicited a statistically significant increase in PALS scores from baseline to one year (Z = -3.006; p = .003) (Figure 14). Of the 52 participants 27 saw an increase in their PAL score at 12 months, only 10 children had 12 month PAL scores that were lower than their baseline score and 15 participants saw no change in their score. The null hypothesis was rejected; there was a significant change in mean PAL scores at baseline and 12 months.

Baseline to year two. Reading Attitudes. To determine if participant’s reading attitudes changed overtime a cumulative reading attitudes score was calculated using the sum of the 10 Reading Attitudes Survey items at baseline and 24 months. The dependent variable, cumulative reading attitudes is continuous and the independent variable is time which has two levels (baseline and 24 months) and is parametric and correlated. Based on the variable characteristics, a paired t-test is the appropriate test of significance. The null hypothesis (Hₒ = µ1 = µ2) states there is no difference between the mean baseline and mean 24 month response scores, the alternative hypothesis (Hᴬ = µ1 ≠ µ2) states that there will be a difference between the baseline and 24 month mean reading attitudes. In order for a Paired T-Test to meet statistical validity requirements, the test must meet four assumptions: one variable is continuous, and the other is dichotomous, the two distributions have equal variances, the observations are independent and the last assumption is the two distributions are normally

Page 30


distributed. The Paired T-Test met all of the test assumptions, the variable cumulative baseline reading attitudes (n = 62) had a mean score of 31.38 (s = 6.10), median score of 32.00, a range of 21.00 with a minimum score of 19.00 and maximum score of 40.00. The results of the informal test of normality indicated a negative skewness (-.292) and a negative kurtosis value (-.957). Cumulative reading attitudes at 24 months (n = 22) had a mean score of 31.45 (s = 7.24), median score of 34.00, a range of 23.00 with a minimum score of 17.00 and maximum score of 40.00. The results of the informal test of normality indicated a negative skewness (-.825) and a negative kurtosis value (-.509), both variables were normally distributed and we can assume equal variance. At baseline 32 of the participants reported a mean cumulative reading attitudes score of 31 or higher, with 33.9% reporting a score of 35 or higher. Fourteen of the 22 participants at 24 months reported a mean cumulative reading attitudes score of 32 or higher, with 45.4% reporting a score of 35 or higher (11.5% more of the participants at 24 months reported a score of 35 or higher) (Table 14). The results of the analysis included 22 observations with a mean cumulative reading attitudes score of 31.77 (s = 5.52) at baseline and a mean cumulative reading attitudes score of 31.45 (s = 7.24) at 24 months. The 24 month reported cumulative reading attitude scores were not significantly different than the baseline reported cumulative reading attitudes (t= .162; p= .873). As per the results, the null hypothesis was accepted; there was not a significant difference in cumulative reading attitudes at baseline and 24 months. Since the significance value for change in cumulative reading attitudes from baseline to 24 months is greater than 0.05, we can conclude that the average change in reading attitude scores of .31818 may be due to chance variation and cannot be attributed to the ESBM program.

Mentor Perceptions of Reading Abilities.

In order to determine if participant’s reading skills changed overtime a mean mentor reported reading skills score was calculated using items 18 - 28 of the Mentor Strength of Relationship Survey at baseline and 24 months. The dependent variable mean mentor reported reading skills is continuous and the independent variable time, has two levels (baseline and 24 months), and is parametric and correlated since the same mentor responded. Based on the variable characteristics, a paired t-test is the appropriate test of significance. The null hypothesis (Hₒ = µ1 = µ2) states there is no difference between the mean baseline and mean 24 month mentor reported reading skills, the alternative hypothesis (Hᴬ = µ1 ≠ µ2) states that there will be a difference between the baseline and 24 month mean reading skill scores. The Paired T-Test met all of the test assumptions, the variable mean mentor reported reading skills at baseline (n = 58) had a mean score of 3.675 (s = .457), median score of 3.63, a range of 1.91 with a minimum score of 2.64 and maximum score of 4.55. The results of the informal test of normality indicated a negative skewness (-.170) and a negative kurtosis value (-.625). Mean mentor reported reading skills at 24 months (n = 15) had a mean score of 3.733 (s = .440), median score of 3.727, a range of 1.36 with a minimum score of 3.09 and maximum score of 4.45. The results of the informal test of normality indicated a negative skewness (- .025) and a negative kurtosis value (-.978), both variables were normally distributed and we can assume equal variance. The results of the analysis included 15 observations with a mean mentor reported reading skills score of 3.721 (s = .421) at baseline and a mean mentor reported reading skills score of 3.733 (s = .440) at 24 months. The 24 month mentor reported reading skills were not significantly different than baseline mentor reported reading skills (t= -.114; p= .911). As per the results, the null hypothesis was accepted; there was not a significant difference in mean mentor reported reading skills at baseline and 24 months. Since the significance value for change in mean mentor reported reading skills

Page 31


from baseline to 24 months was greater than 0.05, we can conclude that the average change in mentor reported reading skill scores of -.0121 was due to chance variation and cannot be attributed to the ESBM program.

Mentor Reported Strength of Relationship. To determine if the strength of the mentorship changed overtime a mean mentor reported strength of relationship score was calculated using items 1 to 14 of the Mentor Strength of Relationship Survey at baseline and 24 months. The dependent variable is mean mentor reported strength of relationship and is continuous. The independent variable is time which has two levels (baseline and 24 months), the variable is parametric and correlated since the same mentor responded. Based on the variable characteristics, a paired t-test is the appropriate test of significance. The null hypothesis (Hₒ = µ1 = µ2) states there is no difference between the mean baseline and mean 24 month mentor reported strength of relationship, the alternative hypothesis (Hᴬ = µ1 ≠ µ2) states that there will be a difference between the baseline and 24 month mean mentor reported strength of relationship scores. The Paired T-Test met all of the test assumptions, the variable mean mentor reported strength of relationship at baseline (n = 60) had a mean score of 4.11 (s = .379), median score of 4.07, a range of 1.64 with a minimum score of 3.21 and maximum score of 4.86. The results of the informal test of normality indicated a negative skewness (-.275) and a negative kurtosis value (-.289). Mean mentor reported strength of relationship at 24 months (n = 25) had a mean score of 4.10 (s = .505), median score of 4.28, a range of 2.14 with a minimum score of 2.79 and maximum score of 4.93. The results of the informal test of normality indicated a negative skewness (-1.210) and a positive kurtosis value (1.325), both variables were normally distributed and we can assume equal variance. At baseline 43 mentors (71.1%) reported a mean mentor reported strength

of relationship score of 4 or higher (with 4 indicating the mentor “agreed” the Little and Big were a ‘good match’). At 24 months 18 mentors (72%) reported a mean mentor reported strength of relationship score of 4 or higher, there was an increase in the number of mentors that reported a mean mentor reported strength of relationship score of 4 or higher from baseline to 24 months (Table 15). The results of the analysis included 25 observations with a mean mentor reported strength of relationship score of 4.20 (s = .319) at baseline and a mean mentor reported strength of relationship score of 4.10 (s = .505) at 24 months. The 24 month mean mentor reported strength of relationship scores were not significantly different than baseline mean mentor reported strength of relationship scores (t= 1.060; p= .300). The null hypothesis was accepted; there was not a significant difference in mean mentor reported strength of relationship at baseline and 24 months. Since the significance value for change in mean mentor reported strength of relationship from baseline to 24 months was greater than 0.05, we can conclude that the average change in mentor reported strength of relationship of .09868 was due to chance variation and cannot be attributed to the ESBM program.

Reading Interest To determine if participant reading interest changed overtime a mean mentor reported reading interest score was calculated using items 2, 3, 5, 7 and 10 of the Activity Preference Questionnaire at baseline and 24 months. The dependent variable mean reported reading interest is continuous and the independent variable time, has two levels (baseline and 24 months) and is parametric and correlated since the same participant responded. Based on the variable characteristics, a paired t-test is the appropriate test of significance. The null hypothesis (Hₒ = µ1 = µ2) states there is no difference between the mean baseline reported reading interest and mean 24 month reported reading interest, the alternative hypothesis (Hᴬ = µ1 ≠ µ2) states that

Page 32


there will be a difference between the baseline and 24 month mean reported reading interest scores. The Paired T-Test met all of the test assumptions, the variable mean reported reading interest at baseline (n = 62) had a mean score of .4089 (s = .298), median score of .4000, a range of 1.00 with a minimum score of .00 and maximum score of 1.00. The results of the informal test of normality indicated a positive skewness (.215) and a negative kurtosis value (-1.138). Mean reported reading interest at 24 months (n = 22) had a mean score of .4909 (s = .347), median score of .6000, a range of 1.00 with a minimum score of .00 and maximum score of 1.00. The results of the informal test of normality indicated a negative skewness (-.066) and a negative kurtosis value (-1.312), both variables were normally distributed and we can assume equal variance. The results of the analysis included 22 observations with a mean reported reading interest score of .3523 (s = .292) at baseline and a mean reported reading interest score of .4909 (s = .347) at 24 months. The 24 month mean reported reading interest scores were not significantly different than baseline mean reported reading interest scores (t= -1.333; p= .197). The null hypothesis was accepted; there was not a significant difference in mean reported reading interest at baseline and 24 months. Since the significance value for change in mean reported reading interest from baseline to 24 months was greater than 0.05, we can conclude that the average change in reported reading interest of -.13864 was due to chance variation and cannot be attributed to the ESBM program.

Major GPA To determine if participant grades changed overtime, mean major GPA was calculated at baseline and 24 months. The dependent variable mean major GPA is continuous and the independent variable time, has two levels (baseline and 24 months) and is parametric and correlated since the same mentor responded. Based on the variable characteristics, a paired t-test is the appropriate test of significance.

The null hypothesis (Hₒ = µ1 = µ2) states there is no difference between the mean baseline and mean 24 month major GPA, the alternative hypothesis (Hᴬ = µ1 ≠ µ2) states that there will be a difference between the baseline and 24 month mean major GPA. The Paired T-Test met all of the test assumptions, the variable mean major GPA at baseline (n = 59) had a mean score of 3.00 (s = .440), median score of 3.00, a range of 2.00 with a minimum score of 2.00 and maximum score of 4.00. The results of the informal test of normality indicated a negative skewness (-.291) and a positive kurtosis value (.046). Mean major GPA at 24 months (n = 51) had a mean score of 3.13 (s = .636), median score of 3.25, a range of 2.50 with a minimum score of 1.50 and maximum score of 4.00. The results of the informal test of normality indicated a negative skewness (-1.041) and a positive kurtosis value (.791), both variables were normally distributed and we can assume equal variance. At baseline 36 participants (61%) reported a mean major GPA score of 3.0 or higher and 1 participant had a mean major GPA of 4.0. At 24 months 39 participants (76.5%) reported a mean major GPA score of 3.0 or higher and 5 participants (9.8%) reported a mean major GPA of 4.0. Fifteen percent more participants reported a 3.0 or higher major GPA at 24 months than at baseline (Table 9). The results of the analysis included 50 observations with a mean major GPA of 3.06 (s = .417) at baseline and a mean major GPA of 3.11 (s = .630) at 24 months. The 24 month mean major GPA was not significantly different than baseline mean major GPA (t= -.556; p= .581). The null hypothesis was accepted; there was not a significant difference in mean major GPA at baseline and 24 months. Since the significance value for change in mean mentor reported reading interest from baseline to 24 months was greater than 0.05, we can conclude that the average change in mentor reported reading interest of -.05000 was due to chance variation and cannot be attributed to the ESBM program.

Page 33


Table 17 illustrates how the outcomes improved/changed in the cohort from baseline to 12 month survey data, and baseline to 24 month survey results.

Methods The main focus of the summative evaluation is to answer the research question; did the adequate mentor implementer group change significantly more than the non-implementer group? Specifically, what is the association between implementation and improvement in reading attitudes, reading skills, strength of relationship, reading preference, major GPA or PALS scores from baseline to 12 months?

Reading Attitudes The dependent variable cumulative reading attitudes from baseline to 12 months, was assessed using SPSS to calculate the sum of the 10 Elementary Reading Attitude Survey responses at baseline and at the 12 month check-in. In SPSS the correlated data was replaced with a gain score variable, by subtracting the cumulative reading attitudes scores at 12 months from the cumulative reading attitude scores at baseline to determine the change in participant reported reading attitudes overtime. Using SPSS, the dependent variable gain score was recoded into a binary measure (where 1 = improvement and 0 = no improvement) to help determine the probability of a case resulting in improvement, the recoded variable is ordinal and independent. The independent categorical variable mentor implementation type had two levels (adequate implementer and non-adequate implementer) and is independent. To be considered an adequate implementer, the mentor was required to have reported reading to their little during every visit for a minimum of 15 minutes (15 minutes is the standard time suggested by the ESBM program). Mentors that did not meet both of the requirements at each check-in were considered inadequate implementers. Using mentor check-in data a formula was created in Microsoft Excel to identify the mean frequency of minutes reported

spent reading during each visit and a separate formula was created to determine the mean frequency of reading reported for each check-in. The implementer variable was created using a third formula to identify adequate implementers (1) and non-adequate implementers (0) by recognizing the mentors that met both requirements. Based on the variable characteristics, a Chi-Square is the appropriate test of significance. The null hypothesis states there is no association between the variables; implementation type is independent of reading attitudes from baseline to 12 months. The alternative hypothesis states there is an association; mentor implementation is not independent of reading attitudes from baseline to 12 months.

Reading Abilities The dependent variable mean mentor reported reading abilities from baseline to 12 months, was assessed using items 18 - 28 of the Mentor Strength of Relationship Survey at baseline and 12 months. In SPSS the correlated data was replaced with a gain score variable, by subtracting the mean reading abilities scores at 12 months from the mean reading abilities scores at baseline to determine the change in mentor reported reading abilities overtime. Using SPSS, the gain score was recoded into a binary measure (where 1 = improvement and 0 = no improvement) to help determine the probability of a case resulting in improvement, the recoded variable is ordinal and independent. The independent categorical variable mentor implementation type had two levels (adequate implementer and non-adequate implementer) and is independent. Based on the variable characteristics, a Chi-Square is the appropriate test of significance. The null hypothesis states there is no association between the variables; implementation type is independent of reading abilities from baseline to 12 months. The alternative hypothesis states there is an association; mentor implementation is not independent of reading abilities from baseline to 12 months.

Page 34


Strength of Relationship Mean mentor reported strength of relationship from baseline to 12 months the dependent variable, was assessed using items 1- 14 of the Mentor Strength of Relationship Survey at baseline and 12 months. In SPSS the correlated data was replaced with a gain score variable, by subtracting the mean mentored reported strength of relationship scores at 12 months from the mean mentor reported strength of relationship scores at baseline to determine the change in perceived strength of relationship overtime. Using SPSS, the gain score was recoded into a binary measure (where 1 = improvement and 0 = no improvement) to help determine the probability of a case resulting in improvement, the recoded variable is ordinal and independent. The independent categor ical var iable mentor implementation type had two levels (adequate implementer and non-adequate implementer) and is independent. Based on the variable characteristics, a Chi-Square is the appropriate test of significance. The null hypothesis states there is no association between the variables; implementation type is independent of strength of relationship from baseline to 12 months. The alternative hypothesis s t a t e s t h e r e i s a n a s s o c i a t i o n ; m e n t o r implementation is not independent of strength of relationship from baseline to 12 months.

Reading Interest The dependent variable, mean participant reported reading preference from baseline to 12 months was assessed using items 2, 3, 5, 7 and 10 of the Activity Preference Questionnaire at baseline and 12 months. In SPSS the correlated data was replaced with a gain score variable, by subtracting the mean reported reading preference scores at 12 months from the mean reported reading preference scores at baseline to determine the change in reading preference overtime. Using SPSS, the gain score was recoded into a binary measure (where 1 = improvement and 0 = no improvement) to help determine the probability of a case resulting in improvement, the

recoded variable is ordinal and independent. The independent categor ical var iable mentor implementation type had two levels (adequate implementer and non-adequate implementer) and is independent. Based on the variable characteristics, a Chi-Square is the appropriate test of significance. The null hypothesis states there is no association between the variables; implementation type is independent of reading preference from baseline to 12 months. The alternative hypothesis states there is an association; mentor implementation is not independent of reading preference from baseline to 12 months.

Grades The dependent variable, mean major GPA was assessed using SPSS to calculate a mean major GPA score at baseline and 12 months. In SPSS the correlated data was replaced with a gain score variable, by subtracting the mean major GPA score at 12 months from the mean major GPA score at baseline to determine the change in major GPA overtime. The gain score was recoded into a binary measure (where 1 = improvement and 0 = no improvement) in SPSS to help determine the probability of a case resulting in improvement, the recoded variable is ordinal and independent. The independent categor ical var iable mentor implementation type had two levels (adequate implementer and non-adequate implementer) and is independent. Based on the variable characteristics, a Chi-Square is the appropriate test of significance. The null hypothesis states there is no association between the variables; implementation type is independent of major GPA from baseline to 12 months. The alternative hypothesis states there is an association; mentor implementation is not independent of major GPA from baseline to 12 months.

PALS The dependent variable, mean PALS scores was assessed using SPSS to calculate a mean PALS score

Page 35


at baseline and 12 months. In SPSS the correlated data was replaced with a gain score variable, by subtracting the mean PALS score at 12 months from the mean PALS score at baseline to determine the change in PALS scores overtime. The gain score was recoded into a binary measure (where 1 = improvement and 0 = no improvement) in SPSS to help determine the probability of a case resulting in improvement, the recoded variable is ordinal and independent. The independent categorical variable mentor implementation type had two levels (adequate implementer and non-adequate implementer) and is independent. Based on the variable characteristics, a Chi-Square is the appropriate test of significance. The null hypothesis states there is no association between the variables; implementation type is independent of PALS scores from baseline to 12 months. The alternative hypothesis states there is an association; mentor implementation is not independent of PALS scores from baseline to 12 months.

Results Reading Attitudes In order for a Chi-Square test to meet statistical validity requirements the test cannot be used on correlated data and the data are assumed to have a random distribution. I confirmed that the Chi-Square test met the test assumptions; cumulative reading attitude scores from baseline to 12 months were converted into a gain score so the data was no longer correlated. The binary cumulative reading attitude variable (n = 53) had a mode score of 1.00 with a range of 1.00 (minimum .00 = no improvement and maximum 1.00 = improvement) and indicated over half of the participants (52.8%) reported improvement in reading attitudes from baseline to 12 months. The implementation variable (n = 62) had a range of 1.0 with a minimum response of 0.0 (inadequate implementers) and a maximum response of 1.0 (adequate implementers) and the mentors were distributed equally among the

implementation groups. The results of the analysis on the association between participant reported reading attitudes from baseline to 12 months and implementation concluded participants with adequate implementers did not improve more than participants with inadequate implementers. We e x p e c t e d 1 3 p a r t i c i p a n t s w i t h a d e q u a t e implementers to report improved reading attitudes and 14 participants reported improvement, it was also expected that 14 participants with inadequate implementers would report improvement and 14 participants with inadequate mentors reported improvement in reading attitudes from baseline to 12 months. There is no association between improved reading attitudes and implementation (p-value = .662). A symmetric measures Somers’ D test was used (.060; p-value = .662) and indicated a weak indirect non-significant association. Therefore, the null hypothesis was accepted; there was no association between improved reading attitudes from basel ine to 12 months and mentor implementation.

Reading Abilities In order for a Chi-Square test to meet statistical validity requirements the test cannot be used on correlated data and the data are assumed to have a random distribution. I confirmed that the Chi-Square test met the test assumptions; mean mentor reported reading abilities scores from baseline to 12 months were converted into a gain score so the data was no longer correlated. The binary mean reading abilities variable (n = 52) indicated 28 mentors or 53.8% reported improvement in reading abilities from baseline to 12 months with a mode score of 1.00 a minimum score of .00 (no improvement) and maximum score of 1.00 (improvement). The implementation variable (n = 62) had a range of 1.0 with a minimum response of 0.0 (inadequate implementers) and a maximum response of 1.0 (adequate implementers) and the mentors were distributed equally among the implementation groups. The results of the analysis on the

Page 36


association between mentor reported reading abilities from baseline to 12 months and implementation concluded participants with adequate implementers did not improve more than participants with inadequate implementers. We expected 14 mentors with adequate implementation to report improved reading abilities and 16 mentors reported improvement, similarly it was expected that 14 mentors with inadequate implementation would report improvement and 12 inadequate mentors reported improvement in their Little’s reading abilities from baseline to 12 months. There is no association between improved reading abilities and implementation (p-value = .266). A symmetric measures Somers’ D test was used (.154; p-value = .260) and indicated a weak indirect non-significant association. Therefore, the null hypothesis was accepted; there was no association between improved reading abilities from baseline to 12 months and mentor implementation.

Strength of Relationship In order for a Chi-Square test to meet statistical validity requirements the test cannot be used on correlated data and the data are assumed to have a random distribution. I confirmed that the Chi-Square test met the test assumptions; mean mentor reported strength of relationship scores from baseline to 12 months were converted into a gain score so the data was no longer correlated. The binary mean strength of relationship variable (n = 55) had a mode score of 1.00, a range of 1.00 with a minimum score of .00 (no improvement) and a maximum score of 1.00 (improvement) and indicated 52.7% of mentors reported improvement in strength of relationship from baseline to 12 months. The implementation variable (n = 62) had a range of 1.0 with a minimum response of 0.0 (inadequate implementers) and a maximum response of 1.0 (adequate implementers) and the mentors were distributed equally among the implementation groups. The results of the analysis on the association between mentor reported

strength of relationship from baseline to 12 months and implementation concluded participants with adequate implementers did not improve more in strength of relationship than participants with inadequate implementers. We expected 14 mentors with adequate implementation to report improved strength of relationship and 15 adequate mentors reported improvement, similarly it was expected that 14 mentors with inadequate implementation would report improvement in strength of relationship and 14 inadequate mentors reported improvement in strength of relationship from baseline to 12 months. There is no association between improved strength of relationship and implementation (p-value = .680). A symmetric measures Somers’ D test was used (.056; p-value = .679) and indicated a weak indirect non-significant association. Therefore, the null hypothesis was accepted; there was no association between improved strength of relationship from baseline to 12 months and mentor implementation.

Reading Interest In order for a Chi-Square test to meet statistical validity requirements the test cannot be used on correlated data and the data are assumed to have a random distribution. I confirmed that the Chi-Square test met the test assumptions; mean reported reading interest scores from baseline to 12 months were converted into a gain score so the data was no longer correlated. The binary mean reading interest variable (n = 53) with a mode score of .00, range of 1.00 (minimum score of .00 = no improvement and maximum score of 1.00 = improvement) designated 23 mentors or (43.4%) reported improvement in reading interest from baseline to 12 months. The implementation variable (n = 62) had a range of 1.0 with a minimum response of 0.0 (inadequate implementers) and a maximum response of 1.0 (adequate implementers) and the mentors were distributed equally among the implementation groups. The results of the analysis on the association between mean reported reading

Page 37


interests from baseline to 12 months and implementation concluded participants with adequate implementers did not improve more in reading interests than participants with inadequate implementers. We expected 10 participants with adequate implementation to report improved reading interests and 12 reported reading improvement, similarly it was expected that 12 participants with inadequate implementation would report improvement and 11 reported improved reading interests from baseline to 12 months. There is no association between improved reading interests and mentor implementation (p-value = .523). A symmetric measures Somers’ D test was used (.088; p-value = .522) and indicated a weak indirect non-significant association. Therefore, the null hypothesis was accepted; there was no association between improved reading interests from baseline to 12 months and mentor implementation.

Grades In order for a Chi-Square test to meet statistical validity requirements the test cannot be used on correlated data and the data are assumed to have a random distribution. I confirmed that the Chi-Square test met the test assumptions; mean major GPA scores from baseline to 12 months were converted into a gain score so the data was no longer correlated. The binary mean major GPA variable (n = 57) with a mode score of 1.00, range of 1.00 (minimum score of .00 = no improvement and maximum score of 1.00 = improvement) indicated over half of the participants (52.6%) reported improvement in major GPA from baseline to 12 months. The implementation variable (n = 62) had a range of 1.0 with a minimum response of 0.0 (inadequate implementers) and a maximum response of 1.0 (adequate implementers) and the mentors were distributed equally among the implementation groups. The results of the analysis on the association between mean major GPA scores from baseline to 12 months and implementation concluded participants with adequate implementers

did not improve more in major GPA than participants with inadequate implementers. We e x p e c t e d 1 4 p a r t i c i p a n t s w i t h a d e q u a t e implementation to report improved major GPA and 13 reported improvement, similarly it was expected t h a t 1 5 p a r t i c i p a n t s w i t h i n a d e q u a t e implementation would report improvement and 17 reported improved major GPA from baseline to 12 months. There is no association between improved major GPA and mentor implementation (p-value = .357). A symmetric measures Somers’ D test was used (-.122; p-value = .353) and indicated a weak negative non-significant association. Therefore, the null hypothesis was accepted; there was no association between improved major GPA from baseline to 12 months and mentor implementation.

PALS In order for a Chi-Square test to meet statistical validity requirements the test cannot be used on correlated data and the data are assumed to have a random distribution. I confirmed that the Chi-Square test met the test assumptions; mean PALS scores from baseline to 12 months were converted into a gain score so the data was no longer correlated. The binary mean PALS score variable (n = 52) with a mode score of 1.00, range of 1.00 (minimum score of .00 = no improvement and maximum score of 1.00 = improvement) indicated over half of the participants (51.9% or 27 participants) reported improvement in PALS scores from baseline to 12 months. The implementation variable (n = 62) had a range of 1.0 with a minimum response of 0.0 (inadequate implementers) and a maximum response of 1.0 (adequate implementers) and the mentors were distributed equally among the implementation groups. The results of the analysis on the association between mean PALS scores from baseline to 12 months and implementation concluded participants with adequate implementers did not improve more in PALS scores than participants with inadequate implementers. We e x p e c t e d 1 3 p a r t i c i p a n t s w i t h a d e q u a t e

Page 38


implementation to report improved PALS scores and 12 reported improvement, similarly it was expected that 13 participants with inadequate implementation would report improvement and 15 reported improved PALS scores from baseline to 12 months. There is no association between improved PALS scores and mentor implementation (p-value = .405). A symmetric measures Somers’ D test was used (-.115; p-value = .402) and indicated a weak negative non-significant association. Therefore, the null hypothesis was accepted; there was no association between improved PALS scores from baseline to 12 months and mentor implementation.

Page 39


Recommendations The data suggests that there is not much positive movement in key indicators such as reading attitudes, reading preference, reading skills, strength of relationship, major GPA, and PALS scores. The formative evaluation shows that, when data is reported, mentors are engaging in the activities with their littles as required by the program guidelines. However during any given period, data is missing for nearly one-third of the mentors. Summative data shows that 50 percent of the mentors are not reading frequently enough or for a long enough period of time. Of the seven mentors who reported reading to their little at least 30 minutes during each visit, 5 of the 7 showed increases in their major GPA from baseline to 24 months while a sixth student maintained their 4.0 major GPA. It is recommended that mentors be required to read to their little at every visit for at least 30 minutes.

Another observation gleaned from the participant profile is that the demographics of the mentors are not representative of the demographics of the student participants. Research suggests that gender and ethnicity play a role in whether the mentor / protege relationship will be successful (Darling, 2006). For example, Darling suggests that a more structured program may benefit a male student with a female mentor. We recommend that gender and ethnicity be more of a factor in matching the mentor with their little. Further, BBBS should engage in targeted recruitment efforts in an attempt to recruit a more diverse mentor pool. Finally, mentors should participate in a rigorous training that fosters a better understanding of ethnic, socioeconomic, and gender diversity.

Page 40


APPENDIX A: FORMATIVE DATA Table 1. Summary statistics table of the performance indicators for Early Match

Table 2. Summary statistics table of the performance indicators for Mid Match

Table 3. Summary statistics table for the performance indicators for Long Term Match

Table 1. Early Match

Number MeanStandard Deviation Mode Minimum Maximum Median

Average Time Reading 53 21 6.91 x 5 40 x

Reading Frequency 55 x x 5 2 5 5

Books Read by Children

52 x x 4 0 30 6

Books Sent Home 51 x x 2 x x 2

Table 2. Mid Match


Average Time Reading 48 20 6.17 x 12.5 45 x



48 x x 4 1 15 5


Table 3. Long Term Match


Average Time Reading 50 20.1 6.76 x 10 45 x



50 x x 4 0 32 4


Page 41


Table 4. Summary statistics table of Average Reading Time

Table 5. Summary statistics table of Reading Frequency

Table 6. Summary statistics table of Books Read by Children

Table 7. Summary statistics table of Books Sent Home

Table 4. Average Time Reading

Number MeanStandard Deviation Minimum Maximum

Early Match 53 21 6.91 5 40

Mid Match 48 20 6.17 12.5 45

Long Term Match 50 20.1 6.76 10 45

Table 5. Reading Frequency

Number Mode Minimum Maximum Median

Early Match 55 5 2 5 5

Mid Match 48 5 1 5 5

Long Term Match 50 5 1 5 5

Table 6. Books Read by Children


Early Match 52 4 0 30 6

Mid Match 48 4 1 15 5

Long Term Match 50 4 0 32 4

Table 7. Books Sent Home


Early Match 51 5 x x 2

Mid Match 50 5 x x 2

Long Term Match 20 5 x x 2

Page 42


APPENDIX B: SUMMATIVE DATA Table 8. Demographic characteristics of participants

ESBM Intervention

Variable n Percent Variable n Percent

Child Age Mentor Age

5 to 7 56 90.3 18 to 23 59 95.2

8 6 9.7 35 to 66 3 4.8

Child Gender Mentor Gender

Male 32 51.6 Male 7 11.3

Female 30 48.4 Female 55 88.7

Child Ethnicity Mentor Ethnicity

White 11 19.3 White 58 93.5

Hispanic 36 63.2 Hispanic 2 3.2

African-American 7 12.3 Asian 2 3.2

Other Minority Groups 3 5.3

Mother’s Education Level Volunteer Occupation

High School Diploma or Less 50 84.7 College Student 58 93.5

Associate Degree or Some College 3 6.3 Community Member 4 6.5

Father’s Education Level Volunteer Primary Language

High School Diploma or Less 45 93.8 English 62 100

Assoc. Degree or Graduate School 3 6.3

Parent Marital Status Volunteer Second Language

Married 27 45 Spanish 8 66.7

Single, Separated or Divorced 33 55 Other 4 33.3

Page 43


Table 9. Reading Attitude mean scores above 35 at baseline and 12 months

Table 10. Reading Skills mean scores of 4 or above at baseline and 12 months

Income

$29,999 or below 43 91.5

$30,000 to $49,000 4 8.5

Family Size

2 to 5 52 86.7

6 to 9 8 13.7

Receive Free / Reduced Lunch

Yes 38 61.3

Receive Food Stamps

Yes 39 63.9

ESBM Intervention

Variable n Percent Variable n Percent

Survey Response

Reading Attitudes Baseline 12 Month Total

Score less than 35 30 32 62

Score 35 or higher 32 21 53

Total (n) 62 53

Survey Response

Reading Skills Baseline 12 Month Total


Score 4 “agree” or higher 17 18 35

Total (n) 58 53

Page 44


Table 11. Strength of Relationship mean scores of 4 or above at baseline and 12 months

Table 12. Reading Interest mean scores at baseline and 12 months

Table 13. Mean Major GPA at baseline and 12 months

Table 14. Reading Attitude mean scores above 35 at baseline and 24 months

Survey Response

Strength of Relationship Baseline 12 Month Total



Total (n) 60 55

Survey Response

Reading Interest Baseline 12 Month Total

0 score (no reading interest) 11 10 21

Score 1 (reading interest) 2 6 8

Total (n) 13 16

Survey Response

Major GPA Baseline 12 Month Total

Lower than 3.0 23 12 35

3.0 or higher 36 45 81

Total (n) 59 57

Survey Response

Reading Attitudes Baseline 12 Month Total


Score 35 or higher 21 10 31

Total (n) 62 22

Page 45


Table 15. Strength of Relationship mean scores of 4 or above at baseline and 24 months

Table 16. Mean Major GPA at baseline and 24 months

Table 17. Outcome measures from baseline to 12 months and baseline to 24 months

Survey Response

Strength of Relationship Baseline 12 Month Total



Total (n) 60 25

Survey Response

Major GPA Baseline 12 Month Total

Lower than 3.0 23 12 35

3.0 or higher 36 39 75

Total (n) 59 51

Program Outcome M SD p

Reading Attitude Mean Score Baseline 31.88 6.040.637

12 Months 32.37 6.27

Baseline 31.77 5.520.873

24 Months 31.45 7.24

Reading Skills Mean Score Baseline 3.67 0.4440.599

12 Months 3.71 0.483

Baseline 3.72 0.4210.911

24 Months 3.73 0.44

Page 46


Strength of Relationship Mean Score Baseline 4.13 0.380.922

12 Months 4.14 0.422

Baseline 4.2 0.3190.3

24 Months 4.1 0.505

Reading Preference Mean Score Baseline 0.41 0.3030.319

12 Months 0.46 0.322

Baseline 0.352 0.2920.197

24 Months 0.49 0.347

Major GPA Mean Score Baseline 3.02 0.4270.001

12 Months 3.21 0.462

Baseline 3.06 0.4170.581

24 Months 3.11 0.63

PALS Mean Score Baseline 1.84 0.8260.003

12 Months 2.3 0.742

BaselineNo data received for 24 months

24 Months

Program Outcome M SD p

Page 47


APPENDIX C: REFERENCES Darling, N., Bogat, G. A., Cavell, T. A., Murphy, S. E., & Sanchez, B. (2006). Gender, Ethnicity, Development, and

Risk: Mentoring and the Consideration of Individual Differences. Journal of Community Psychology, 34(6), 765-779. Retrieved March 17, 2016.

Elbaum, B., and Vaughn, S. (2000). How effective are one-to-one tutoring programs in reading for elementary students at risk for reading failure? A meta-analysis of the intervention research. Journal of Educational Psychology, 92(4), 605-619.

Harrisonburg City Public Schools. (2015). LEP Student Enrollment. Retrieved from http://harrisonburg.k12.va.us/Instruction/English-as-a-Second-Language/Enrollment-Statistics/2015-2016/12-18-15-LEP-Enrollment-Summary

Harrisonburg City Public Schools. (2016). Reading Specialist (SES - 2016-2017 School Year). Retrieved from https://harrisonburg.tedk12.com/hire/ViewJob.aspx?JobID=301

Hernandez, D. J. (2011). Double Jeopardy: How Third Grade Reading Skills and Poverty Influence High School Graduation (Rep.). Baltimore, MD: The Annie E. Casey Foundation. Retrieved from http://gradelevelreading.net/wp-content/uploads/2012/01/Double-Jeopardy-Report-030812-for-web1.pdf

McKenna, M. C., & Kear, D. J. (1990). Measuring attitude toward reading: A new tool for teachers. The Reading Teacher , 626-639.

The Annie E. Casey Foundation. (2013). Early Warning Confirmed: A Research Update on Third-Grade Reading. Baltimore, MD: Leila Fiester. Retrieved from http://www.aecf.org/resources/early-warning-confirmed/

Totty, S. (2016, January 19). Big Brothers Big Sisters of Harrisonburg-Rockingham County. Lecture presented at PUAD 606: Program Evaluation in James Madison University, Harrisonburg, VA.

U.S. Department of Agriculture, National Agriculture Statistics Service. (2012).Census of Agriculture. Retrieved from http://www.agcensus.usda.gov/Publications/2012/Full_Report/Volume_1,_Chapter_2_County_Level/Virginia/

University of Virginia, Curry School of Education. (n.d.). PALS: Phonological Awareness Literacy Screening. Retrieved from https://pals.virginia.edu/tools-k.html.

Virginia Department of Education. (2015). Fall Membership Data. Retrieved from http://www.doe.virginia.gov/statistics_reports/enrollment/fall_membership/report_data.shtml

Virginia Department of Education. (2016). School Nutrition Program Statistics & Reports. Retrieved from http://www.doe.virginia.gov/support/nutrition/statistics/index.shtml

Virginia Department of Education. (2015). Title III: Language Instruction for Limited English Proficient & Immigrant Students. Retrieved from http://doe.virginia.gov/federal_programs/esea/title3/index.shtml

Page 48


Virginia Department of Education, Office of School Nutrition. (2015). School Year 2015-2016 National School Lunch Program (NSLP) Free and Reduced Price Eligibility Report. Retrieved from http://doe.virginia.gov/support/nutrition/statistics/free_reduced_eligibility/2015-2016/schools/frpe_sch_report_sy2015-16.pdf

Virginia Performs. (2015). Assessing School Readiness in Virginia. Retrieved March 14, 2016, from http://vaperforms.virginia.gov/indicators/education/schoolReadiness.php

Young, L. (2013, October 10). Big Brothers Big Sisters and JMU team up to guide youth. The Breeze. Retrieved March 13, 2016, from http://www.breezejmu.org/life/big-brothers-big-sisters-and-jmu-team-up-to-guide/article_b2471a22-313b-11e3-b8a4-0019bb30f31a.html

Page 49

program evaluation & recommendation...workforce and citizenry. lead and manage in public...

Documents