Critique and Review of: Critique and Review of: Program Evaluation Report Program Evaluation Report Results-Based Model (RBM) Results-Based Model (RBM)

2002-20032002-2003Report found at: Report found at: http://www.idahotc.

com/files/rbm.pdf

Contained all the necessary Contained all the necessary component sections of a final component sections of a final evaluation report?evaluation report? Summary: a summary of results is provided Summary: a summary of results is provided

at the beginning of the documentat the beginning of the document Purpose: assesses team members from Purpose: assesses team members from

RBM pilot schools, analyze the effect of RBM pilot schools, analyze the effect of RBM on IRI scores on those with RBM on IRI scores on those with significant reading skill concerns in K-3significant reading skill concerns in K-3

Background information: explains that this Background information: explains that this the 4the 4thth year of implementation or the year of implementation or the Results-Based Model, a problem solving Results-Based Model, a problem solving approach to support students who are approach to support students who are having concerns with reading.having concerns with reading.

Contained all the necessary Contained all the necessary component sections of a final component sections of a final evaluation report?evaluation report? Description of the evaluation Description of the evaluation

study and design: provides a study and design: provides a general outline in Figure and general outline in Figure and Table 1, and an explanation of Table 1, and an explanation of survey creation techniques. survey creation techniques. Individual sections also Individual sections also contained descriptive contained descriptive information concerning the way information concerning the way data was gathered and used.data was gathered and used.

Contained all the necessary Contained all the necessary component sections of a final component sections of a final evaluation report?evaluation report? Discussion of the program and Discussion of the program and

results: the program and results results: the program and results were communicated throughout the were communicated throughout the documentdocument

Conclusions and recommendations: Conclusions and recommendations: unless the conclusion and unless the conclusion and recommendations were located in recommendations were located in the Appendix A-G sections, the the Appendix A-G sections, the evaluation failed to communicate evaluation failed to communicate significant conclusions or provide significant conclusions or provide recommendations to the reader.recommendations to the reader.

· Evaluation methodology - · Evaluation methodology - sound or questionable?sound or questionable? Survey: The staff survey is extremely Survey: The staff survey is extremely

thorough and provides valuable data for a thorough and provides valuable data for a wide variety of areas within the evaluation. wide variety of areas within the evaluation. It’s possible that these surveys are too It’s possible that these surveys are too lengthy resulting in participant stagnation. lengthy resulting in participant stagnation.

The use lists in the results sections The use lists in the results sections concerning areas of most improvement concerning areas of most improvement could be communicated in a better way. could be communicated in a better way. Perhaps an instance or significant example Perhaps an instance or significant example of data from the appendix section could of data from the appendix section could help represent the data areas listed.help represent the data areas listed.

· Evaluation methodology - · Evaluation methodology - sound or questionable?sound or questionable?

The information provided The information provided concerning the IRI Reading concerning the IRI Reading Skills Associated with RBM Skills Associated with RBM Intervention plan contains well Intervention plan contains well explained, sound data explained, sound data beneficial, and comprehendible beneficial, and comprehendible to the reader.to the reader.

· Readability - easy to read, · Readability - easy to read, follow and understand?follow and understand? The IRI Reading Skills Associated The IRI Reading Skills Associated

with RBM Intervention Plans section with RBM Intervention Plans section of the document contains a very of the document contains a very straightforward narrative about the straightforward narrative about the purpose of the analysis. The survey purpose of the analysis. The survey analysis section lacks this spelled analysis section lacks this spelled out text.out text.

It would be beneficial to have the It would be beneficial to have the purpose as clearly stated at the purpose as clearly stated at the beginning of the document as well as beginning of the document as well as in these sections.in these sections.

· Readability - easy to read, · Readability - easy to read, follow and understand?follow and understand? When comparing the evaluation When comparing the evaluation

information provided under the information provided under the survey section in comparison with survey section in comparison with that is written about the IRI skills, the that is written about the IRI skills, the survey information seems very survey information seems very disjointed and difficult to follow. disjointed and difficult to follow. Perhaps this is again because the Perhaps this is again because the Appendix A-G information is not Appendix A-G information is not included.included.

The IRI Skills section is much easier The IRI Skills section is much easier to follow and read.to follow and read.

Surprises?Surprises?

p.11 Table 2p.11 Table 2

Surprises?Surprises?

p.11, Table 2 is very busy and hard p.11, Table 2 is very busy and hard to follow. It would make more sense to follow. It would make more sense to have this information under the to have this information under the descriptions for each appendix descriptions for each appendix section.section.

Appendix A-G are missing, but this is Appendix A-G are missing, but this is likely an internet/upload problem, not likely an internet/upload problem, not a mistake by the evaluatora mistake by the evaluator

· Learned something new about · Learned something new about doing evaluation reports?doing evaluation reports?

Cut and dryCut and dry To the pointTo the point Be thoroughBe thorough Be careful when uploadingBe careful when uploading Clear communication is a must, Clear communication is a must,

especially goals, evaluation especially goals, evaluation questions, and conclusionsquestions, and conclusions

Need recommendationsNeed recommendations

· Suggestions to make the · Suggestions to make the evaluation better?evaluation better?

The evaluation desperately The evaluation desperately needs recommendations to needs recommendations to validate the evaluationvalidate the evaluation

Communicate goals of Communicate goals of evaluation in the introduction evaluation in the introduction more thoroughlymore thoroughly

Descriptive summary of resultsDescriptive summary of results