developmental evaluation on the ground

Developmental Evaluation on the

Ground

Brigitte Scott, Ph.D.

Evaluation and Research Specialist

Military Families Learning Network

Military Families Learning Network

Personal Finance

Child Care

Family Advocacy

Network Literacy

Personal Finance

Family Development

Network Literacy

Community Capacity

Building

Nutrition & Wellness

Transition Support

Early Intervention

Special Needs

Life Span Special

Needs

Military Caregiving

MQP: DE Defined

“DE supports innovation development to guide

adaptation to emergent and dynamic realities in

complex environments.” —MQP, p. 1

Goal: support project, program, product, and/or

organizational development with timely

feedback

MQP: Core DE Question

What is getting developed and what are the

implications of what gets developed?

Complexity theory of change:

Bring people together who are knowledgeable and

committed and they will self-organize, take

action, and work together to create movement,

innovation, and change.

MQP: Key Characteristics of DE

Focus on development

Focus on developmentMQP

• NOT the same as continuous improvement (formative eval or accountability)

– Quality improvement is helping programs meet standards that have been set.

Development is when people are changing what they are doing, and the very nature of the standards are also changing. Standards are adapting to changing conditions.

Complex and dynamic environment

Complex and dynamic environmentMQP

Complex environment = lack of

central control

What to do to solve problems is

uncertain and there can be conflict

about how to proceed

Real time feedback

DE becomes part of intervention

Developmental Evaluator

Developmental EvaluatorMQP

• Works collaboratively with innovators to

conceptualize, design, and test new approaches

in a long-term, on-going process of adaptation,

intentional change, and development.

• Ask eval questions

• Apply eval logic

• Gather and report eval data

Developmental EvaluatorMQP

Primary Functions:

Elucidate the innovation and adaptation

processes

Track implications and results

Facilitate data-based decision-making

“There are no best practices. ‘Best’ is

entirely context-free.”—MQP, AEA 2014

Evaluation for MFLN

• Ongoing development

• Innovation in learning

– Internal practice

– External product

DE for MFLN: Why It Fits

Program development

Cooperative agreement environment

+

eXtension Initiative

Cooperative Agreement

“An opportunity to provide responsive and timely educational programming through being actively nimble, flexible, innovative, and creative in true partnership with our funders as they identify organizational priorities and ask us to engage in the construction of appropriate and necessary deliverables that meet the on-time needs of the target audience. The MFLN and funding partners are seen as true and honest equals in expertise, providing valued and accepted feedback bi-directionally. This often results in surpassing expectations for success in delivery.”

—Kyle Kostelecky, National Program Director, MFLN

Constant evaluative feedback

Constant evaluative feedback and

discussion

• Monthly reports (internal, DoD)

• Quarterly reports (NIFA)

• Annual reports (internal, DoD, NIFA)

• Webinar evaluation reports

• Quarterly webinar reports

• Weekly leadership team meetings

• Social Media Specialists meetings

• Net Lit support meetings

Evaluation used for decision-making

Evaluator is on leadership team

DE for MFLN: What It Looks Like

Continuous adaptation


Continuous adaptationSocial, political, economic, technological, and demographic

patterns



patterns

Emergent developments/needs of our target audiences



patterns





patterns



eXtension



patterns



eXtension

DoD


(not this. . . .)


(but this. . . .)


Inductive


InductiveChaotic and messy



Embrace forks in the road



Embrace forks in the road

Developmental moments

Track what’s going on

Understand what’s going on

Adapt to what’s going on


Everyone is an

evaluator

Inspired by Jeff Conklin, cognexus.org

Time

Complex development situations are ones in which this…

39Michael Quinn Patton

AEA eStudy webinar 2014

Michael Quinn Patton


And this…

Turns out to be this…

Time41





…looks like this

DE Successes for MFLN


• Very responsive to our partners’ programmatic

requests and innovations




• Doubling in size in 2015 with competitive award

– New CAs and new opportunities for innovation in

programming





– New CAs and new opportunities for innovation in

programming

• Programming


• Very responsive to our partners’ programmatic requests and innovations


– New CAs and new opportunities for innovation in programming

• Programming

• “Walk the talk”: model use of social media, collaborative learning, personal learning networks

DE Challenges for MFLN


• Innovation is challenging, slow, and . . . contentious



• Moving from program improvement mode to redesign

based on evaluation findings





• IRB challenges





• IRB challenges

• “In your face” evaluation





• IRB challenges


• Daily, evaluative thinking





• IRB challenges



• Platform delivery limitations





• IRB challenges




• Military culture





• IRB challenges




• Military culture

• Doubling in size in January 2015

DE Features for MFLN

• Reports

• Leadership

• Reflective discussion groups

• Transparency

• Collaboration

DE Features for MFLN

Patton, Michael Quinn. (2014).

“Intermediate Developmental Evaluation.”

AEA eStudy Webinar.

Patton, Michael Quinn. (2011). Developmental

Evaluation: Applying Complexity Concepts to

Enhance Innovation and Use. Guilford: New

York.

Brigitte Scott

@4ed_eval

[email protected]

www.blogs.extension.org/militaryfamilies

Icons made by Freepik:

http://www.flaticon.com. Flaticon is

licensed by Creative Commons 3.0.

mailto:[email protected]

http://www.flaticon.com

developmental evaluation on the ground

Government & Nonprofit