Technical Assistance on Evaluation to Child Welfare

Discretionary Grantees2012 Abandoned Infants Assistance

Comprehensive Support Services for Families Affected by Substance Abuse and/or HIV/AIDS

OverviewNovember 2012

James Bell Associates

James Bell Associates (JBA):James Bell Associates (JBA):Who We Are…Who We Are…

Small business located in Arlington, VA.Small business located in Arlington, VA. Conducting research and evaluation in public Conducting research and evaluation in public

health and child welfare since 1979.health and child welfare since 1979. 30+ employees and nationwide pool of 30+ employees and nationwide pool of

consultants.consultants. Experts in program evaluation, technical Experts in program evaluation, technical

assistance and training, applied research.assistance and training, applied research. DHHS clients include ACF, HRSA, NIH, ASPE, and DHHS clients include ACF, HRSA, NIH, ASPE, and

SAMHSA SAMHSA

JBA’s Evaluation TA ContractJBA’s Evaluation TA Contract TA provider to CW discretionary grantees since 1999.TA provider to CW discretionary grantees since 1999.

Number of grantees served annually varies (currently ~100).Number of grantees served annually varies (currently ~100).

Other priority areas supported include Adoption Other priority areas supported include Adoption Opportunities Programs, Child Abuse/Neglect Prevention, Opportunities Programs, Child Abuse/Neglect Prevention, Integrating Trauma-Informed and Trauma-Focused Practice Integrating Trauma-Informed and Trauma-Focused Practice in CPS, Promoting Safe and Stable Families, Diligent in CPS, Promoting Safe and Stable Families, Diligent Recruitment, Child Welfare and Early Education/Education Recruitment, Child Welfare and Early Education/Education Partnerships, Improving Service Delivery to Youth in CWS.Partnerships, Improving Service Delivery to Youth in CWS.

TA is also provided to Family Connection grantees under a TA is also provided to Family Connection grantees under a separate cross-site evaluation contract. separate cross-site evaluation contract.

Technical Assistance GoalsTechnical Assistance Goals Support the Children’s Bureau in disseminating Support the Children’s Bureau in disseminating

knowledge and promoting the use of effective knowledge and promoting the use of effective and evidence-based practices in the child and evidence-based practices in the child welfare field.welfare field.

Promote the implementation of rigorous, useful, Promote the implementation of rigorous, useful, and practical evaluations.and practical evaluations.

Strengthen the quality of evaluation reports and Strengthen the quality of evaluation reports and products available to the child welfare field. products available to the child welfare field.

JBA’s Evaluation TA Contract: JBA’s Evaluation TA Contract: What We ProvideWhat We Provide

Responsive TAResponsive TA to individual grantees to individual grantees upon request.upon request.

Proactive TAProactive TA to individual grantees to individual grantees upon request of FPO and/or JBA upon request of FPO and/or JBA recommendation.recommendation.

Group TAGroup TA to grantee clusters (e.g., to grantee clusters (e.g., facilitating conference calls & facilitating conference calls & Webinars regarding Webinars regarding research/evaluation topics of common research/evaluation topics of common interest).interest).

JBA’s Evaluation TA Contract: JBA’s Evaluation TA Contract: Important FactsImportant Facts

JBA is JBA is notnot a program monitor or a program monitor or auditor; our role is to work in auditor; our role is to work in partnership with the Children’s Bureau partnership with the Children’s Bureau to assist you in implementing a quality to assist you in implementing a quality evaluation.evaluation.

We do We do notnot conduct evaluations under conduct evaluations under this contract.this contract.

JBA’s services are available at JBA’s services are available at no costno cost to grantees.to grantees.

TA ApproachTA Approach

― Introductory mailingsIntroductory mailings― Evaluation TA packetEvaluation TA packet― Kickoff Meeting sessions/callsKickoff Meeting sessions/calls― Initial conference callsInitial conference calls― Develop TA plansDevelop TA plans― Ongoing communicationOngoing communication― Web-based portal/information sharingWeb-based portal/information sharing― Ongoing evaluation document reviewOngoing evaluation document review

Cluster TACluster TA Grantee meeting Grantee meeting

presentations and presentations and facilitated facilitated discussionsdiscussions

Webinars and Webinars and teleconferencesteleconferences

Summary documentsSummary documents Web-based portalWeb-based portal

Site-Specific TASite-Specific TA Conference calls Conference calls Document reviewDocument review E-mailE-mail Site visitsSite visits

TA ModalitiesTA Modalities

Proactive and ResponsiveProactive and Responsive

More About Site-Specific TAMore About Site-Specific TA Tailored to the specific interests and needs Tailored to the specific interests and needs

of grantees.of grantees.

Responsive TAResponsive TA: Grantee contacts JBA for : Grantee contacts JBA for advice or assistance with a specific advice or assistance with a specific evaluation issue. evaluation issue.

Proactive TAProactive TA: FPO asks JBA to contact : FPO asks JBA to contact grantee if a specific concern or TA need grantee if a specific concern or TA need is identified.is identified.

More About Cluster TAMore About Cluster TA Initiated based on interests and needs of FPO and Initiated based on interests and needs of FPO and

grantees.grantees. Focus is on peer learning, networking and Focus is on peer learning, networking and

information sharing.information sharing.ExamplesExamples::Facilitate collection and analysis of data on Facilitate collection and analysis of data on common outcome measures.common outcome measures.Facilitate discussion and information sharing Facilitate discussion and information sharing regarding research and evaluation issues or regarding research and evaluation issues or challenges of common interest.challenges of common interest.

Now that you have won your Now that you have won your grant …grant …

Enhancements to the evaluation described Enhancements to the evaluation described in your proposal are encouraged. in your proposal are encouraged. Components to revisit include:Components to revisit include: Logic ModelLogic Model Research/Evaluation DesignResearch/Evaluation Design Outputs/OutcomesOutputs/Outcomes Output/Outcome MeasuresOutput/Outcome Measures Data Collection Tools/InstrumentsData Collection Tools/Instruments

Consider Revising Your Evaluation Consider Revising Your Evaluation Design If:Design If: Expected data sources are not available.Expected data sources are not available.

Inadequate referrals pose challenges for random Inadequate referrals pose challenges for random assignment or comparison group designs.assignment or comparison group designs.

Original evaluation plan/logic model focused on Original evaluation plan/logic model focused on the “wrong” outcomes or measures.the “wrong” outcomes or measures.

Assessment instruments are invalid, unreliable, Assessment instruments are invalid, unreliable, or otherwise inappropriate. or otherwise inappropriate.

Some interventions/activities were never Some interventions/activities were never implemented and cannot be evaluated.implemented and cannot be evaluated.

Original evaluation plan was too ambitious.Original evaluation plan was too ambitious.

Back to the Drawing Board: Steps to Back to the Drawing Board: Steps to Take Before Changing Your Take Before Changing Your

Evaluation Evaluation Discuss your proposed changes with Discuss your proposed changes with

JBA.JBA. Contact your FPO to obtain approval Contact your FPO to obtain approval

before you make any changes. before you make any changes. JBA JBA cannot approve changes!cannot approve changes!

Document changes in subsequent Document changes in subsequent progress reports to CB and JBA.progress reports to CB and JBA.

Include JBA in the Distribution of Include JBA in the Distribution of Semi-Annual Progress ReportsSemi-Annual Progress Reports

Send a copy directly to Send a copy directly to your JBA TA liaison every your JBA TA liaison every six months (electronic six months (electronic copies preferred).copies preferred).

Allows JBA to stay informed Allows JBA to stay informed about your project and about your project and identify possible TA needs. identify possible TA needs.

QuestionsQuestions??Kim Keating, MEd

Sr. Research Associate/TA Liaisonkeating@jbassoc.com

703/528-3230

James Bell Associates3033 Wilson Boulevard, Suite 650

Arlington, VA 22201