year two national 4-h science academy final report[1]

8/22/2019 Year Two National 4-H Science Academy Final Report[1]

1/48

National 4-H Science

Leadership Academy

Year Two

Regional 4-H Science Academies

Evaluation ReportJune 15, 2012

Mary E. Arnold, Ph.D.

Project Evaluator

Oregon State University

With assistance from

Courtney Archibeque, MpH

Brooke Nott, M.S.

Graduate Research Assistants

Oregon State University


2/48

i

Table of Contents

Evaluators Statement .. ii

Acknowledgements .. iii

Executive Summary .. iv

Year Two Regional 4-H Science Academy Overview .. 1

Knowledge and Skills for 4-H Science: Impact of the Regional Academies .. 8

Curriculum Focus .. 8

Evaluation Focus .. 10

Professional Development Focus ..... 11

Fund Development Focus .. 12

Summary .. 14

Confidence and Intention to Teach Others ... 15Curriculum .. 15

Evaluation .. 18

Professional Development ..... 21

Fund Development .. 23

Readiness to Facilitate 4-H Science .. 26

Summary .. 27

Science Liaisons Estimated Reach Beyond Regional Academies ..... 28

Estimates of Knowledge and Skill Dissemination . 28

Additional Resources Garnered for 4-H Science .... 29

Narrative Questions Content Analysis ... 33

Plans for Next Three Months .. 33

Additional Support Needs .. 35

Post-Academy Training Opportunity Ideas.. 36

Appendix 1: Feedback from New England on Effectiveness of Virtual Academy ..... 39


3/48

ii

Evaluators Statement

This document serves as the final evaluation report for the National 4-H Science Academy: Year

Two Regional Academies program sponsored by National 4-H Council with funding from the

Noyce Foundation. The academies were held in five locations across four regions of the country

between January and April, 2012. All academy participants were invited to participate in theprogram evaluation.

All data for the evaluation were entered by participants directly into an on-line data collection

system. Access to the system was provided by the evaluator to the participants for data entry,

but only the evaluator and her research assistants had access to the actual dataset. The

integrity and accuracy of the raw data rests with the individual participants. The integrity and

accuracy of the analysis and interpretation rests solely with me as the project evaluator. To this

end, I certify that the analysis and results presented in this document are complete and

accurate insofar as the data entered by the participants were as well. Any questions or

concerns about this report should be addressed to me.

Mary E. Arnold, Ph.D.

Project Evaluator, Oregon State University

June 15, 2012

[email protected]


4/48

iii

Acknowledgements

This program evaluation could not have taken place without the dedicated help and support of many

individuals.

First and foremost, I would like to thank National 4-H Council and the Noyce Foundation for theopportunity to conduct the Year Two Regional Academy evaluation. It was exciting to witness the

movement from the national to the regional level, and to document the important work that took place

as a result of the academies.

I would like to express my thanks and gratitude to Janet Golden, Eddie Locklear, Jo Turner, and Maila

Oliveria at National 4-H Council for their support and help with various aspects of the evaluation.

Thank you, also, to the lead planners at each of the five academies, for keeping me apprised of the plans

for your academies, and more than anything for your timely assistance with getting participant contact

information to me.

A very special thank you to my co-authors and graduate assistants: Ms. Courtney Archibeque and Ms.

Brooke Dolenc Nott. Your focused assistance with the data analysis and report preparation could not be

replaced. Thank you both especially for your cheerful willingness to concentrate your work time on the

report so we could meet the expected deadline.

I would like to thank each and every academy participant who contributed data. The sincerity with

which you approached the evaluation was evident in the data and information you provided. Without

your help, there would be nothing to report. So thank you for your willingness to help make this

possible.

Finally, thank you to the Noyce Foundation for the generous support of the National 4-H Science

Leadership Academy. The funding provided by the foundation made this important program possible. Asa result, 4-H programs across the country are more prepared to develop and sustain programs for youth

in science, technology, engineering, and math.


5/48

iv

Executive Summary

Year Two Regional 4-H Science Academy Overview

Five regional 4-H Science Academies were held during the second year of the National4-H Science Leadership Academy project:

o Northeast (Mid-Atlantic), State College, PA, January 24-26 (73 participants)o North Central-St. Louis, MO, February 7-10 (228 participants)o West-Davis, CA, March 26-28 (100 participants)o Southern- Huntsville, AL, April 9-12 (78 participants)o Northeast (New England), Virtual Academy, April 10-11 (36 participants)

515 invitations to participate in the academy evaluation were sent, with 427 returned,of an overall return rate of 82.9%:

Demographic data from the regional academies indicate that the primary audience forthe academy (county 4-H agents) was reached.

Academy attendance overall appears to represent the size and involvement of 4-HScience programs for each region.

Participants report a fairly even exposure to the academy topics, with slightly moreattending sessions on curriculum and professional development than the other areas.

This, however, matches the results of the national academy evaluation and follow-up in

which participants expressed the most interest in further training in the areas of

curriculum and professional development, especially in the area of scientific inquiry.

Evaluation and fund development participation was also fairly strong, along withpartnerships, which is a topic area added to the regional academies.

The lowest exposure was in the area of marketing, but this area was not explicitly statedin the academy request for applications (RFA).

An interesting finding is that over three-quarters of the academy participants werefemale. This may reflect the gender balance of the 4-H professional workforce,

especially at the county 4-H agent level. But it could also have implications for potential

capacity for engaging girls in 4-H Science.

Prior awareness and use of tools and resources for 4-H science revealed mixed results,with curriculum and webinars showing the greatest awareness and use. Other resources

revealed low level of awareness and use, particularly fund development and urbanscience resources. Since participants should have learned more about these resources at

the regional academy, it will be important to monitor levels of use of the resources in

future evaluations.


6/48

v

Knowledge and Skills for 4-H Science: Impact of the Regional

A positive result in the area of program evaluation is in using existing tools to evaluate4-H Science programs. This result dovetails nicely with the results of the national science

academy evaluation that showed a distinct need for ready-to-go instruments to

evaluate 4-H Science programs

A significant result appears to be in the area of teaching science inquiry to facilitators of4-H Science programs. This item shows the greatest change in reported level of ability,

with 74% reporting moderate or high levels at the end of the academy. This result

dovetails nicely with the results from previous academy evaluations that showed a

marked need for further training in science inquiry.

The emphasis on inquiry was evident in the results in the professional development andcurriculum areas. This builds nicely on the results of past academy evaluations that

revealed a need for more training on inquiry. It appears that the regional academies met

this request well, and served to further the infusion of science inquiry as an important

aspect of 4-H Science.

There appears to be continued concern in Fund Development (especially with thetoolkit) and Evaluation (especially in providing leadership for evaluation). Learning to

utilize the fund development toolkit and providing leadership for evaluation were two

explicitly stated outcomes for the academies, yet they appear to lag behind the rest of

the outcomes.

Confidence and Intention to Teach Others

The results indicate a positive personal and organizational readiness to facilitate 4-HScience

There is a general emphasis on furthering 4-H Science through curriculum andprofessional development.

There is less ability to facilitate 4-H Science in the fund development and evaluationareas than in curriculum and professional development.

Science Liaisons Estimated Reach Beyond Regional Academies

It is interesting to note the greater reach planned to staff in the areas of infusing scienceand evaluation science programs. This may be an early indicator of a move away from

emphasizing inquiry toward other topics, and could indicate that inquiry as a basis for

4-H Sciences is beginning to be understood and utilized more easily.

The lower levels of dissemination of information to staff related to the science plan ofaction, providing leadership for evaluation, and using the fund development toolkit is

consistent with other findings in this report, and again perhaps indicative of the


7/48

vi

specialist audience that needs to be reached about these topics. Clearly, not all staff will

benefit from this information and targeted dissemination efforts should be considered.

Virtual Academy

The Northeast-New England regional academy was held virtually. The evaluation of thisformat received mixed results, with several suggestions for improvement. Because the

Year Three academies are going to be virtual, the feedback and critique from the New

England Academy is presented in Appendix One of this report.


8/48

1

Year Two: Regional 4-H Science Academies Overview

Five regional 4-H Science Academies were held during the second year of the National 4-H

Science Leadership Academy project:

Northeast (Mid-Atlantic), State College, PA, January 24-26.North Central-St. Louis, MO, February 7-10

West-Davis, CA, March 26-28

Southern- Huntsville, AL, April 9-12

Northeast (New England), Virtual Academy, April 10-11

Academy planners submitted a proposal in response to the Regional 4-H Science Academy RFA

issued by National 4-H Council. The RFA outlined specific content, goals, and outcomes for the

regional academies. Each region submitted a proposal that sufficiently met the requirements

for the regional academies and received funding to plan and host the academy in their region.

The stated goal of the 4-H Regional Science Academies was to:

Build the capacity of state and local 4-H faculty, staff, and volunteers to offer high quality,

sustainable 4-H Science programs by helping them:

Design, implement, evaluate and sustain 4-H Science programs Develop strategies to secure funds and other resources to support 4-H

Science at the state and local levels

Assist Land-grant Universities and counties with developing and refiningtheir 4-H Science Plans of Action

Identify additional training and resources needed to support the 4-H ScienceMission Mandate

Furthermore, specific content was to be provided in order to meet the outcomes for the

academies in the following areas:

Fund Development

1. Every LGU will have a written vision and case for supporting 4-H Science financially.

2. LGUs will update their 4-H Science Plans of Action to include a revenue plan that articulates

the role of government grants, fees for service (if applicable) and philanthropic support as well

as a multi-year fund development plan which includes specific fundraising goals to achieve that

revenue plan.


9/48

2

3. Every LGU will have a plan for identifying, recruiting and engaging volunteer champions who

are passionate about 4-H and active in a variety of fund development activities in support of the

sustainable expansion of 4-H Science programming within that state.

Curriculum1. Participants will increase their skills to train others on curriculum components and program

quality.

2. Participants will be prepared to develop new, and revise existing curricula, incorporating

identified criteria and standards.

3. Participants will be able to train others to identify how science is infused through mission

mandate areas in 4-H.

Evaluation

1. Participants will learn to utilize existing tools and identify appropriate evaluation methods.

2. Participants will increase their knowledge to determine when and how to do evaluation that

meets their needs.

3. Participants will increase their knowledge and skills to provide leadership to and build

capacity for evaluation at Land-grant Universities and at the county level.

Professional Development

1. Participants will be able to utilize a train-the-trainer model to train staff and volunteers to

facilitate 4-H Science programs.

2. Participants will be able to train staff and volunteers to recruit, develop, and retain

volunteers to facilitate 4-H Science programs.

3. Participants will be able to train staff and volunteers to use 4-H Science professional

development tools and resources.

4. Participants will be able to transfer knowledge and skills learned across all three mission

mandates.

Partnership Development

1. Regional academy planning teams were encouraged to offer training in partnership

development.

The evaluator worked directly with the lead contact person for each academy to secure contact

information (names and e-mail addresses) for all academy participants. Participants were

contacted via e-mail and invited to take part in an on-line post-academy evaluation. Invitations

were sent immediately following the completion of each academy, with up to three follow-up


10/48

3

reminders to non-respondents over the course of 2-3 weeks. A total of 515 invitations were

sent, with 427 returned of an overall return rate of 82.9%:

Participant response rates for all the academies were sufficiently strong. Table 1.0 shows the

number of respondents and response rates by academy.

Table 1.0 Response Rates by Academy Invitations Respondents Response Rate

Northeast (Mid-Atlantic) 73 62 84.9%

North Central 228 185 81.1%

South 78 63 80.7%

West 100 87 87.0%

Northeast (New-England) 36 30 83.3%

Total 515 427 82.9%

Participants at the regional academies were overwhelmingly female (76.8%). Sixty five

respondents indicated they had attended the National 4-H Science Leadership Academy at theNational 4-H Conference Center in December, 2010; 351 respondents reported they did not

attend the national academy. Those who had attended the national academy were evenly

distributed according to the focus track they had attended at the national level:

17 attended the curriculum focus track 17 attended the evaluation focus track 15 attended the fund development track 16 attended the professional development track

By far, the majority of participants in the regional academies were county 4-H agents (221

reported this). Additional participant roles in 4-H are presented in Figure 1.0.


11/48

4

Figure 1.0 Participant Roles in the 4-H Program

* Note: Respondents could indicate more than one role; thirty-five (9%) indicated more than one role.

Respondents were provided the option of indicating their role was something other than those

listed. Table 1.1 provides a summary of the other roles identified by respondents.

Table 1.1 Other Roles Indicated

4-H Program Coordinator 26

Regional Agent or Specialist 17

State Level Educator 12

Community Worker/Educator 6CYFAR/Military 4

Student 2

County Director 1

Retired 1

As stated earlier, regional academies were designed to provide professional development

training for 4-H Science in: Curriculum, Evaluation, Fund Development, and Professional

Development. Content related to Partnerships and Marketing was strongly encouraged. Most

respondents reported experiencing content in the areas of Professional Development and

Curriculum at the academies. This is consistent with the content emphasis of the academies

that emphasized science inquiry in the curriculum and professional development areas. Figure

1.1 shows the number of respondents who reported participating in learning opportunities by

content area.

255

33 31 24

221

3617

4 3

63

0

50

100

150

200

250

Volunteer Partner

(funder,

program

supporter)

State 4-H

Liaison

State 4-H

Science

Specialist

State 4-H

Specialist

(not

science

specific)

County 4-H

Educator

(agent role)

County 4-H

Educator

(non-agent

role)

State 4-H

Program

Leader

State 4-H

Foundation

Staff

State 4-H

Foundation

Director

Missing


12/48

5

Figure 1.1 Number of Participants in Academy Learning Areas

* Note: Respondents could indicate participating in more than one area

One last demographic question asked participants about their pre-academy awareness and use

of tools and resources to support 4-H Science (see Figure 1.2).

Approximately 250 respondents (58%) were aware of resources and tools related tocurriculum and to the 4-H Science webinars. But considerably fewer had actually used

these tools: Webinars- 32%; curriculum website- 44%; and curriculum pieces -47%.

Slightly less (226; 53%) were aware of the state or LGUs science plan of action.

For professional development, 130 (30%) reported awareness of the resources, and 85(20%) reported using them.

Just 142 (33%) reported being aware of the fund development resources, and only 37(9%) reported using the resources.

Only 92 (22%) reported being aware of the web resource for science in urbancommunities, with just 32 (7%) reporting having used these resources.

204

111101

111

164

60 62

0

50

100

150

200

250

Curriculum Evaluation Fund

Development

Partnerships Professional

Development

Marketing Attended all

tracks


13/48

6

Figure 1.2 Frequencies for Awareness and Use of 4-H Sciences Resources- Pre- Academy

Summary

Demographic data from the regional academies indicate that the primary audience forthe academy (county 4-H agents) was reached.

Academy attendance overall appears to represent the size and involvement of 4-HScience programs for each region.

Participants report a fairly even exposure to the academy topics, with slightly moreattending sessions on curriculum and professional development than the other areas.

This, however, matches the results of the national academy evaluation and follow-up inwhich participants expressed the most interest in further training in the areas of

curriculum and professional development, especially in the area of scientific inquiry.

Evaluation and fund development participation was also fairly strong, along withpartnerships, which is a topic area added to the regional academies.

The lowest exposure was in the area of marketing, but this area was not explicitly statedin the academy request for applications (RFA).

An interesting finding is that over three-quarters of the academy participants werefemale. This may reflect the gender balance of the 4-H professional workforce,

especially at the county 4-H agent level. But it could also have implications for potentialcapacity for engaging girls in 4-H Science.

Prior awareness and use of tools and resources for 4-H science revealed mixed results,with curriculum and webinars showing the greatest awareness and use. Other resources

revealed low level of awareness and use, particularly fund development and urban

science resources. Since participants should have learned more about these resources at

254

142

252 255

92

130

226

137

37

187201

32

85

158

0

50

100

150

200

250

300

National 4-H

Science

Webinars

Fund

Development

Toolkit

4-H Science

Curriculum

Website

National 4-H

Science

CurriculumPieces

4-H Science in

Urban

CommunitiesWebsite

4-H Science

Professional

DevelopmentToolkit

State or LGU 4-

H Science Plan

of Action

Aware Of

Used


14/48

7

the regional academy, it will be important to monitor levels of use of the resources in

future evaluations.


15/48

8

Part Two

Knowledge and Skills for 4-H Science: Impact of the Regional Academy

The second part of the of the regional 4-H Science academy evaluation focused on the

knowledge and skills that participants reported leaving the academies with. Respondents were

asked to rate how much the academy contributed to their knowledge or skill related to

important learning items contained in the academy on a nominal scale of: (1) none; (2) low; (3)

moderate; and (4) high. For the purposes of this evaluation, we were interested in the number

of participants that reported that the academy contributed a moderate or high level to the

development of skill or knowledge about 4-H Science.

However, we knew that the backgrounds of those attending the academy in regards to 4-H

Science would vary considerably, so we also asked respondents to rate their level of knowledge

or skill related to important learning items prior to the academyon the same nominal scale.

For these questions, we were interested in the number of participants that reported none or

a low level of skill or knowledge prior to the academy.

In order to best understand the impact of the academies on personal skill and knowledge, both

of these questions should be considered in tandem. The figures below show the results for both

questions by content area.

Curriculum

Just under half of the respondents reported none or low levels of skill and knowledge related to

4-H Science curriculum before the academy (see Figure 2.0). This indicates that the majority of

respondents had considerable experience in curriculum before attending the academy.

However, the majority of respondents reported that the academy had contributed a

moderate or high amount to their skill and knowledge related to 4-H Science Curriculum (se

Figure 2.1).

This was particularly true in:

Why inquiry is critical to 4-H Science Program quality for 4-H Science Curriculum components for 4-H Science


16/48

9

Figure 2.0 Percent of Participants Report None or Low Before the Academy

Figure 2.1 Percent Reporting Academy Learning was Moderate or High

13.4 9.5

24.3 23.211.6 15.6 12.1

35.731.7

32.3 30.9

30.930.4

25.3

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.080.0

90.0

100.0

Curriculum

components for

4-H science

Program quality

for 4-H science

programs

Develop new

science

curriculum

Revise

curriculum

incorporatingscience

curriculum and

standards

Identify high

quality PYD

sciencecurriculum

Science across

all 4-H mission

mandate areas

Why science

inquiry is critical

in 4-H

Low

None

41.3 36.1 38.6 38.0 35.4 34.823.0

41.0 46.429.8 31.4

43.5 39.8 63.1

0.0

10.0

20.0

30.0

40.0

50.0

60.070.0

80.0

90.0

100.0

Curriculum

components

for 4-H science

Program

quality for 4-H

science

programs

Develop new

science

curriculum

Revise

curriculum

incorporating

sciencecurriculum and

standards

Identify high

quality PYD

science

curriculum

Science across

all 4-H mission

mandate areas

Why science

inquiry is

critical in 4-H

High

Moderate


17/48

10

Evaluation

About 60% of the respondents reported none or low levels of skill and knowledge related to 4-H

Science program evaluation before the academy (see Figure 2.2), indicating that most of the

respondents did not have strong skill for program evaluation coming into the academy. About

the same percentage reported that they learned a moderate or high amount related to 4-HScience evaluation, meaning at least 40% of respondents reported learning nothing or only

low levels of skill and knowledge related to evaluation (see Figure 2.3). Changes in skill and

knowledge for evaluation are difficult to interpret from these results, and may be most likely

due to the reality that it takes considerable time to develop good skills in program evaluation.

The most positive result is in the area of using existing tools to evaluate 4-H Scienceprograms. This result dovetails nicely with the results of the national science academy

evaluation that showed a distinct need for ready-to-go instruments to evaluate 4-H

Science programs

The least positive result was in providing leadership for 4-H Science evaluation. This islikely reflective of the academy audience; while 4-H agents may engage in evaluation

efforts, they are not typically the ones who provide leadership for such efforts.


22.6 19.1 20.5 20.8 21.9

39.8 42.6 40.3 38.3 37.8

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.080.0

90.0

100.0

Use exisiting tools for

evaluating 4-H sci.programs

Identify methods for

evaluating 4-H scienceprograms

How to evaluate a 4-H

science program

When to evaluate a 4-

H science program

Provide leadership for

4-H Science evaluation

Low

None


18/48

11



Professional development appears to be an area where many respondents had considerable

experience prior to the academy, with 50% or less reporting none or low levels of expertise (see

Figure 2.3). This was particularly the case in the area of using train the trainer models with staff

and volunteers, and for forming successful youth-adult partnerships. The ratings for post-

academy levels of knowledge and skill rose considerably with over 60% reporting moderate or

high levels for each skill (see Figure 2.4).

The most significant results appear to be in the area of teaching science inquiry tofacilitators of 4-H Science programs. This item shows the greatest change in reported

level of ability, with 74% reporting moderate or high levels at the end of the academy.

This result dovetails nicely with the results from previous academy evaluations that

showed a marked need for further training in science inquiry.

42.5 42.8 44.3 41.3 36.8

19.5 15.7 14.4 13.015.3

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.080.0

90.0

100.0

Use exisiting tools for

evaluating 4-H sci.

programs

Identify methods for

evaluating 4-H

science programs

How to evaluate a 4-

H science program

When to evaluate a

4-H science program

Provide leadership

for 4-H Science

evaluation

High

Moderate


19/48

12



Fund Development

Fund development is the area in which respondents reported the least amount of prior

expertise (see Figure 2.6). Over 50% of respondents reported no or low levels of knowledge and

skill for every item, and this is particularly true in using the fund development toolkit, where

81% reported no or low levels. At the end of the academy at least 60% of respondents reported

moderate to high levels of knowledge and skill for each item, with the exception of using the

fund development toolkit; just 45% of respondents reported moderate or high levels for this

item (see Figure 2.7).

14.4 15.3 14.4 15.5 18.7 10.3

23.4 23.732.2 35.6

33.3

31.4

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Train the trainer

model for

volunteers

Train the trainer

model for staff

Train staff to

recruit develop

and retain

volunteers

Train volunteers to

recruit develop

and reatain

volunteers

Teach scientific

inquiry to

facilitators of 4-H

sci. programs

Form YAPs to

support 4-H sci

programs

Low

None

41.9 37.5 42.7 39.8 31.444.1

29.530.6 22.9 23.6 45.8

29.2

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Train the trainer

model for

volunteers

Train the trainer

model for staff

Train staff to

recruit develop

and retain

volunteers

Train volunteers

to recruit develop

and reatain

volunteers

Teach scientific

inquiry to

facilitators of 4-H

sci. programs

Form YAPs to

support 4-H sci

programs

High

Moderate


20/48

13

These results suggest that the fund development area is one in which most 4-H agentsdo not have a great deal of experience. However, the reported increase in knowledge

and skills in fund development as a result of the academy is fairly strong, which

indicates that the academy served an important role in skill and knowledge

development in the fund development area. The lack of skill to understand and use the fund development toolkit remains a

concern. However, it is not entirely clear how much the use of the toolkit was

emphasized during the academies. The low rating could be simple a reflection of the

toolkit not being emphasized, or they could be indicative of confusion about how to use

the toolkit, or it could be a reflection of the fact that most 4-H agents are not focused on

fund development. Given the resources that have gone into developing the toolkit, it

will be important to keep monitoring its use by 4-H Science programs.


31.1 30.620.6

55.7

27.016.5

25.4 28.6 22.1

26.4 28.538.5

25.8

39.0

36.4

40.2 36.439.2

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Develop

state or LGU

Plan of

Action

Develop

county Plans

of Action

Why invest in

4-H science -

for donors

Fund

development

toolkit use

Volunteer

champions

for 4-H

science

Partnershipss

for 4-H

science

Identify

prospective

donors

Support fund

development

for 4-H

science

Work with

volunteers to

support fund

development

Low

None


21/48

14


Summary

Overall, these results indicated that the learning at the academy was strong and positivefor all of the content areas.

The emphasis on inquiry was evident in the results in the professional development andcurriculum areas. This builds nicely on the results of past academy evaluations that

revealed a need for more training on inquiry. It appears that the regional academies met

this request well, and served to further the infusion of science inquiry as an important

aspect of 4-H Science.

There appears to be continued concern in Fund Development (especially with thetoolkit) and Evaluation (especially in providing leadership for evaluation). Learning to

utilize the fund development toolkit and providing leadership for evaluation were two

explicitly stated outcomes for the academies, yet they appear to lag behind the rest of

the outcomes. It may be that there is a need to identify and match these two aspects of

the academy to the right audience. Who is it at each LGU that has the capacity and right

position description to further these outcomes? Given the specialized nature of these

outcomes, these results may be a function of missing the audience who can best use the

toolkit and/or provide leadership for evaluation. Future offerings in these areas should

be targeted at those who have the capacity to achieve them.

42.234.1

40.434.4

42.7 46.8 44.7 40.7 39.9

28.429.9

33.1

9.9

18.8

33.3

19.4 21.7 24.3

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Develop

state or LGU

Plan of

Action

Develop

county Plans

of Action

Why invest in

4-H science -

for donors

Fund

development

toolkit use

Volunteer

champions

for 4-H

science

Partnershipss

for 4-H

science

Identify

prospective

donors

Support fund

development

for 4-H

science

Work with

volunteers to

support fund

development

High

Moderate


22/48

15

Part Three

Confidence and Intention to Teach Others

One of the goals of the National 4-H Science Leadership Academy is to disseminate knowledge

and skills related to 4-H Science programming to professionals and volunteers throughout the

National 4-H system. This goal is reflected in the structure of the academy for Year Two, in that

resources were moved from the national to the regional level with the intention of reaching

more people through focused regional academies. As such, it is an additional goal that

participants in the regional academies will develop skills and confidence to continue to build

4-H Science programs locally by providing professional development opportunities for other

4-H professionals, volunteers, and partners at the local level. To this end, a section of the

regional academy evaluation focused on participants confidence to use information they

learned and confidence to teach the information to others. In addition, participants were asked

about their intention to teach others.

Participants were asked to rate their level of confidence to use each item and to teach it to

others. They were also asked to rate the likelihood that they would teach each items to others

as part of their 4-H Science program. The figures below show the percentage of respondents

who rated each item moderate or high.

Curriculum

Respondents reported fairly high levels of confidence related to each of the curriculum items

(see Figure 3.0). The greatest level of confidence was in understanding that inquiry is critical

component of 4-H Science (90%). The lowest was in developing curriculum that incorporates

science criteria and standards (69%). This is consistent with previous evaluation reports that

indicate states are not planning to develop their own science curriculum, but look for

curriculum that is developed by others with expertise in 4-H Science.


23/48

16

Figure 3.0 Percent of Participants Report Moderate or High Confidence to use Item

(N = 309)

Respondents also reported moderate to high levels of confidence to teach curriculum items to

others (see Figure 3.1). The highest level was in teaching why science inquiry is critical to 4-H

Science (91%); the lowest was teaching others how to develop curriculum that incorporates

science criteria and standards. The reported levels of confidence to teach others mirrors the

respondents own levels of confidence in each item and supports the emerging evidence that

science inquiry as a foundational part of 4-H Science is taking hold, while independent efforts

to develop 4-H Science curriculum is a lesser priority for many LGUs.

37.2 41.4 40.4 43.0 42.145.3

32.5

47.646.6

29.032.6

45.0 38.4 58.4

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Curriculum

components

for 4-H science

Program

quality for 4-H

science

programs

Develop

curriculum that

incorporates

science criteria

and standards

Revise

curriculum that

incorporates

science

curriculum and

standards

Identify high

quality PYD

science

curriculum

How science is

infused

through 4-H

mission

mandate areas

Why science

inquiry is a

critical

component of

4-H science

curriculum

High

Moderate


24/48

17

Figure 3.1 Percent of Participants Report Moderate or High Confidence to Teach Others

A similar picture emerges for respondents plans to teach others about 4-H Science curriculum

(see Figure 3.2). Most respondents intend to teach others about science inquiry for 4-H Science

(90%), followed by program quality (85%) and curriculum components for 4-H Science (83%).

Only 61% have intentions to teach about developing curriculum for 4-H Science.

Figure 3.2 Percent of Participants Report Moderate or High Intention to Teach Others

42.2 45.1 41.8 42.8 44.8 45.1 40.8

40.541.8

24.328.6

36.9 35.5 49.3

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Curriculum

components

for 4-H science

Program

quality for 4-H

scienceprograms

Develop

curriculum that

incorporatesscience criteria

and standards

Revise

curriculum that

incorporatesscience

curriculum and

standards

Identify high

quality PYD

sciencecurriculum

How science is

infused

through 4-Hmission

mandate areas

Why science

inquiry is a

criticalcomponent of

4-H science

curriculum

High

Moderate

37.5 39.3 34.3 37.343.0 42.0

34.1

45.8 46.0

27.031.0

36.8 39.7 56.0

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Curriculum

components for

4-H science

Program quality

for 4-H science

programs

Develop

curriculum that

incorporates

science criteria

and standards

Revise

curriculum that

incorporates

science

curriculum and

standards

Identify high

quality PYD

science

curriculum

How science is

infused through

4-H mission

mandate areas

Why science

inquiry is a

critical

component of

4-H science

curriculum

High

Moderate


25/48

18

In addition to determining intentionsforteaching content to others, it is important to also

determine intentions not to teach content to others. This provides a picture of potential

problems in reaching the information dissemination goals of the regional academies. Figure 3.3

shows the percentage of respondents who indicated they did not plan to teach content to

others. These ratings all fell below 15%, which is low, but still indicative that some respondentsdid not have plans to teach others, which could mildly impact the success of the academys goal

to disseminate knowledge and skill beyond those who attended the academy.

Figure 3.3 Percent of Participants Report No Intention to Teach Others

Evaluation

Respondents reported lower levels of confidence related to each of the program evaluation

items (see Figure 3.4). The greatest level of confidence was in understanding how to use

existing tools for evaluating 4-H Science programs (70%) and the lowest was providing

leadership for evaluation (61%).

9.311.9 12.2

9.06.5

8.0

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

50.0

Use train-the-trainer

model to train 4-H

science volunteers


model to train staff

in 4-H Science

Train staff to recruit

develop and retain

volunteers

Train volunteers to

recruit develop and

retain volunteers

Teach scientific

inquiry to 4-H

science facilitators

Form YAPs to

support 4-H Science

programs


26/48

19


Respondents reported even less confidence to teach others about evaluation (see Figure 3.5).

on the high side, 61% indicated confidence to teach others about using evaluation tools, while

on the low side, only 55% felt confident to teach others how to identify appropriate methods

for evaluating 4-H Science programs.


44.451.8 51.7 49.8 42.8

25.5 14.2 12.9 16.318.8

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

How to use exisiting

tools for evaluating 4-

H science programs

How to identify

appropriate methods

for evaluating science

programs

Determining how to

evaluate a 4-H science

program


H science program


evaluation of 4-H

science programs

High

Moderate

42.1 43.1 47.8 47.1 39.3

18.9 11.410.8 12.8

15.8

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.090.0

100.0


tools for evaluating 4-H science programs

How to identify

appropriate methodsfor evaluating science

programs

Determining how to

evaluate a 4-H scienceprogram


H science program


evaluation of 4-Hscience programs

High

Moderate


27/48

20

Despite the lower levels of reported confidence related to evaluation and teaching to others,

over 50% reported a moderate or high intention to teach these concepts to others (see Figure

3.6).


More respondents reported that they did not plan to teach others in the area of program

evaluation. As Figure 3.7 shows, the percentage ranged from 10 % for how to use existing

tools to 17.6% for providing leadership for program evaluation. These results could reflect the

fact that program evaluation is a complex topic that takes time to master, as well as that most

county agents are not typically involved in teaching evaluation to others. The bright note in

these results is that teaching others to use existing tools for 4-H Science programs appears to

be the topic that will be taught the most to others. This supports the findings of previous

evaluations that indicated having access to ready tools for evaluation should be a priority.

39.1 43.2 43.2 40.1 36.6

24.8 16.3 16.0 17.0 19.0

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0


tools for evaluating 4-

H science programs

How to identify

appropriate methods

for evaluating science

programs

Determining how to

evaluate a 4-H

science program

When to evaluate a

4-H science program

Provide leadership

for evaluation of 4-H

science programs

High

Moderate


28/48

21



Over 70% of respondents reported moderate or high confidence levels to use and teach items

related to professional development (see Figures 3.8 and 3.9). Intentions to teach others

dropped off slightly, especially around training staff and volunteers to identify, recruit and

retain volunteers for 4-H Science. Even so, over 65% reported moderate or high intentions to

teach others about all professional development items (see Figure 3.10).


10.1

15.3 15.0 15.617.6

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

50.0

How to use exisiting tools

for evaluating 4-H science

programs

How to identify

appropriate methods for

evaluating science

programs

Determining how to

evaluate a 4-H science

program

When to evaluate a 4-H

science program


evaluation of 4-H science

programs

41.538.4

51.0 50.843.1 44.6

38.137.0

25.3 22.6 38.8 38.6

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Use train-the-

trainer model to

train 4-H science

volunteers

Use train-the-

trainer model to

train staff in 4-H

Science

Train staff to

recruit develop

and retain

volunteers

Train volunteers

to recruit develop

and retain

volunteers

Teach scientific

inquiry to 4-H

science

facilitators

Form YAPs to

support 4-H

Science programs

High

Moderate


29/48

22



An analysis of the respondents who reported no intention to teach others about curriculum

topics revealed relatively low percentages ranging from 6.5% for teaching scientific inquiry to

12.2% for training staff to recruit volunteers. These results are similar to those found for the

curriculum area and are indicative of the general emphasis placed on furthering 4-H Sciencethrough curriculum and professional development training.

43.9 39.851.9 51.0 46.3 44.6

36.1 34.7 21.8 20.1 32.8 33.3

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Use train-the-

trainer model to

train 4-H science

volunteers

Use train-the-

trainer model to

train staff in 4-H

Science

Train staff to

recruit develop

and retain

volunteers

Train volunteers

to recruit develop

and retain

volunteers

Teach scientific

inquiry to 4-H

science

facilitators

Form YAPs to

support 4-H

Science programs

High

Moderate

35.1 31.539.5 43.2 43.3 39.4

39.537.4 27.6

22.236.1

36.3

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Use train-the-

trainer model to

train 4-H science

volunteers

Use train-the-

trainer model to

train staff in 4-H

Science

Train staff to

recruit develop

and retain

volunteers

Train volunteers

to recruit develop

and retain

volunteers

Teach scientific

inquiry to 4-H

science

facilitators

Form YAPs to

support 4-H

Science programs

High

Moderate


30/48

23


Fund Development

Things are a bit less positive in the fund development area, with respondents reporting

considerably less confidence to use some of the fund development content (see Figure 3.12).

This is particularly true for confidence to use the fund development toolkit, with only 48%

reporting moderate or high confidence.


(N=316)

9.311.9 12.2

9.06.5

8.0

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

50.0


model to train 4-H

science volunteers


model to train staff

in 4-H Science

Train staff to recruit

develop and retain

volunteers

Train volunteers to

recruit develop and

retain volunteers

Teach scientific

inquiry to 4-H

science facilitators

Form YAPs to

support 4-H Science

programs

47.8 46.4 52.836.3 45.0

51.9 51.3 46.3 48.4

25.9 28.1 19.6

11.313.1

29.613.9 15.7 16.5

0.010.020.030.040.050.060.070.080.090.0

100.0

Develop a

state or LGU

Plan ofAction

Develop

county Plans

of Action

Make case

for investing

in 4-Hscience for

donors

Utilize fund

development

toolkit

Identify

recruit,

engagevolunteer

champions

for 4-H

science

Develop

partnerships

for 4-Hscience

Identify

prospective

donors

Work with

others to

support funddevelopment

for 4-H

science

Work with

volunteers to


High

Moderate


31/48

24

Confidence to teach the fund development to others drops off even more (see Figure 3.13), for

each of the items. The one that remains strongest is developing partnerships for 4-H Science

where 78% of respondents indicated moderate or high confidence to teach to others. A similar

pattern emerges for intention to teach others (see Figure 3.14).


44.2 45.5 46.835.7

46.052.6 47.0 43.5 45.9

21.826.9

16.3

7.5

9.7

25.0

11.5 13.5 12.7

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Develop a

state or LGU

Plan of

Action

Develop

county Plans

of Action

Make case

for investing

in 4-H

science for

donors

Utilize fund

development

toolkit

Identify

recruit,

engage

volunteer

champions

for 4-H

science

Develop

partnerships

for 4-H

science

Identify

prospective

donors

Work with

others to

support fund

development

for 4-H

science

Work with

volunteers to

support fund

development

High

Moderate


32/48

25


The percentage of respondents reporting that they have no intention to teach others,

however, is highest for the fund development area content (see Figure 3.15). This is

particularly true with teaching others how to utilize the toolkit (22.8%), developing the state or

LGU plan of action (15.1%) and working with others to support fun development for 4-H Science

(15%). On a bright note, only 7.1 percent indicated no plans to teach about developing

partnerships for 4-H Science.

35.1 35.741.4

31.437.8

46.841.6 40.8 43.0

27.233.8 24.2

13.2

19.1

31.2

18.8 19.3 18.5

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Develop a

state or LGU

Plan ofAction

Develop

county Plans

of Action

Make case

for investing

in 4-Hscience for

donors

Utilize fund

development

toolkit

Identify

recruit,

engagevolunteer

champions

for 4-H

science

Develop

partnerships

for 4-Hscience

Identify

prospective

donors

Work with

others to


for 4-H

science

Work with

volunteers to


High

Moderate


33/48

26


Readiness to Facilitate 4-H Science at Participating Land Grant Universities

One last question in this section related to dissemination of knowledge asked participants

about their personal and organizational readiness to facilitate 4-H Science in their state.

Participants were asked to rate their level of agreement with several statements related to

personal and organizational readiness to facilitate 4-H Science programs at their LGU (N = 330).

92.4% agreed or strongly agreed that the team attending the regional academy fromtheir LGU is ready to facilitate 4-H Science programs

91.8% agreed or strongly agreed that there is a positive environment at their LGU for 4-HScience

92.5% agreed or strongly agreed that they have the skills needed to developpartnerships for 4-H Science

Figure 3.16 shows the percentage of respondents who agreed or strongly agreed with each

item.

15.111.7 11.4

22.8

14.8

7.1

13.015.0

12.6

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

50.0

Develop a

state or LGU

Plan of Action

Develop

county Plans

of Action

Make case for

investing in 4-

H science for

donors

Utilize fund

development

toolkit

Identify

recruit,

engage

volunteer

champions for

4-H science

Develop

partnerships

for 4-H

science

Identify

prospective

donors

Work with

others to

support fund

development

for 4-H

science

Work with

volunteers to

support fund

development


34/48

27

Figure 3.16 Percentage of Respondent Agreement for Personal and Organizational Readiness

Summary

The results indicate a positive personal and organizational readiness to facilitate 4-HScience

There is a general emphasis on furthering 4-H Science through curriculum andprofessional development.

There is less ability to facilitate 4-H Science in the fund development and evaluationareas than in curriculum and professional development.

61.854.9 59.8 54.9

45.6 51.559.8

29.733.5 32.6 32.0 46.2

26.421.3

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Skills to

develop

partnerships for

4-H Science

Ready to

facilitate 4-H

Science

Group

attending

academy is

ready to

facilitate 4-H

Science

LGU is

organizationally

ready to

support 4-H

Science

Positive

environment at

LGU for 4-H

Science

Adequate

support at LGU

for 4-H Science

Prepared to

form youth-

adult

partnerships to

support 4-H

Science

Strongly Agree

Agree


35/48

28

Part Four

Science Liaisons: Estimated Reach Beyond Regional Academies

Estimates of Knowledge and Skill Dissemination

Thirty-three participants in the regional science academies indicated they are the 4-H Science

Liaison for their state or LGU. Twenty-three of the 33 (70.0%) responded to questions asked of

the liaisons only. These questions asked liaisons to estimate the total number of people that

they plan to reach for 4-H Science using the tools and resources provided as part of the

academy. Liaisons were asked to report this information for their whole state or LGU, not just

for them personally.

High and low parameter estimates were calculated to create a range of estimated reach for

disseminating content area information to each audience. The Table 4.0 presents the estimated

ranges for reaching others with content. The estimates are organized by audience (staff,

volunteers, and partners). Cells highlighted in GREEN indicate the areas where the most reach is

planned; cells highlighted in RED indicate the areas where the least reach is planned.

Table 4.0 Liaison Estimates of Knowledge and Skill Dissemination

Topic Staff Volunteers Partners

Developing State or LGUs science Plan of Action 233-545 383-650 22-146

Science infused throughout three 4-H mission

mandates

628-1055 1664-2151 67-213

Evaluating 4-H Science Programs 628-1050 1426-1985 55-177

Teaching scientific inquiry to facilitators of 4-H

Science568-985 2046-2221 73-235

Making the case for investing in 4-H Science 222-520 945-1360 85-271

Providing leadership evaluating 4-H science programs 171-430 712-1060 19-113

Using the train-the-trainer model to train others 426-760 1086-1585 567-713

Developing partnerships for 4-H Science 407-755 1077-1545 551-729

Developing curriculum that incorporates inquiry,

science criteria and standards316-640 767-1125 28-116

Revising curriculum to incorporate inquiry, criteria

and standards 496-870 1248-1815 551-677

Utilizing the fund development toolkit 220-485 820-1135 27-105

Engaging champions for 4-H Science fund

development259-525 800-1205 209-410


36/48

29

Staff

The greatest planned reach for staff is in the areas of 1) teaching how science is infusedthroughout the mission mandate areas; and 2) evaluating science programs.

The least amount of planned reach is in the areas of 1) developing the science plan of action; 2)providing leadership for evaluating 4-H Science programs; and 3 utilizing the fund development

toolkit.

Volunteers

The greatest planned reach for volunteers is in the areas of 1) infusing science throughout thethree mission mandate areas; and 2) teaching scientific inquiry to facilitators of 4-H Science

programs.

The least amount of planned reach for volunteers is in developing the state of LGU science planof action.

Partners

The areas with the greatest planned reach for partners are: 1) using the train-the-trainer modelto train others; 2) developing partnerships for 4-H Science; and 3) revising curriculum to

incorporate inquiry, criteria and standards.

Summary

While it is important to note that the figures presented in Table 4.0 are only estimates and only

from twenty-three state Science Liaisons, they do paint an interesting and consistent picture of

the potential dissemination reach of skills and knowledge from the regional academies.

It is interesting to note the greater reach planned to staff in the areas of infusing scienceand evaluation science programs. This may be an early indicator of a move away from

emphasizing inquiry toward other topics, and could indicate that inquiry as a basis for 4-

H Sciences is beginning to be understood and utilized more easily.

The lower levels of dissemination of information to staff related to the science plan ofaction, providing leadership for evaluation, and using the fund development toolkit is

consistent with earlier findings in this report, and again perhaps indicative of the

specialist audience that needs to be reached about these topics. Clearly, not all staff will

benefit from this information and targeted dissemination efforts should be considered.

Additional Resources Garnered for 4-H Science

Liaisons were also asked about new partnerships, gifts, grants, and other donations that have

been leveraged to facilitate 4-H Science in their state. Liaisons reported the following funding

secured for 4-H Science:

Gifts: $159,000.00 (6 liaisons reporting)


37/48

30

Grants: $2,007,900 (14 liaisons reporting) Donations: $102,000 (2 liaisons reporting) In-kind estimates: $135,390 (5 liaisons reporting)

In addition to financial resources, Table 4.1 lists the new partnerships developed for 4-HScience by LGUs.

Table 4.1 New Partnerships formed by LGUs to Facilitate 4-H Science

Partnerships

Robotics support through National 4-H Council We are working with libraries and a state level contact for libraries - trying to increase 4-H

STEM programs in libraries and connect to literacy.

We've applied for Smith-Lever funding for a project to develop materials for a 4-H summerSTEM/literacy library program.

We are increasing efforts to connect with campus partners in departments and researchcenters.

We have secured funding for our statewide STEM Program Work Team to hold a springprofessional development/STEM Plan development retreat.

OJJDP 4-H Tech Wizards Walmart - HealthJam and Summer Nutrition Programs JCP RoboticsUSB Biotech ADM Grant to Support Think Green Environmental Program

Received grants from United Soybean Board for biotech, TecXite NSF subgrant forengineering, Gear-tech-21 NSF subgrants for robotics and Geospatial, and grants from MFA

and JC Penney for robotics.

Several corporate partners: Best Buy, 3M, CASE Int. Corporate Partnerships include Lowe's & Walmart. Institutional partnerships includeOklahoma State University and University of Oklahoma. Working to establish a relationship

with other higher education institutions.

- LGU departments (i.e. Physics, Education, Engineering, etc.) - 4-H Foundation partnering tofind funding for 4-H Science programs - LGU Engineering Alumni to partner with county

educators - Association with other state regional universities.

Tennessee Geographic Information Council is support of a GIS 4-H program. Foundation support from within the Texas 4-H Program and from outside foundation

sources have been very generous in the 4-H Science area.

We have been using our extension funds, resources we obtained from other mini-grants, andcollaborating with LGUs in the state for science resources (staff training and resource

materials and facilities) Partnership with a public utility to support the NYSD Industry partners to support robotics

program Grants to support science programs

California Afterschool Network - Professional development and 4-H curriculumdissemination. California Science Teachers' Association - 4-H workshops at their conferences

Grants from National 4-H Council FairPoint Communications - funds for equipment and volunteer training; UVM Extension


38/48

31

AmeriCorps & VISTA support funding; State 4-H Foundation equipment grants;

The University has created a new position that will focus entirely on 4-H Science. They havealso shifted and identified 4-H Science as a focus for all Extension staff.

JCPenney Foundation (4-H Robotics Grant) ESRI -GPS grant Funding from private source fordevelopment of junior master gardener program

Finally, Liaisons were asked to describe other developments in their 4-H Science program that

are direct results of the national and or regional 4-H science academy programs. These

developments are presented in Table 4.2


39/48

32

Table 4.2 Developments in 4-H Science as a Result of the National and Regional Science

Academy Programs.

Our staff are making the "subtle shifts" to inquiry based learning. Strategic planning at the state level Primarily work on our state STEM plan as a result of the National Academy. As a result of the

Regional Academy we will be doing more revisions to our state plan and holding a Program

Work Team retreat and a State Academy for county Educators and volunteers.

Simply organization and unifying state objectives and delivery methods. Also build networksand validate our work. Very beneficial!

Extension administration support of funds for staff professional development in 4-H Science(60 staff attended regional academy)

Determined to train all staff in the inquiry process in the coming year. To figure out how toincorporate the inquiry process into curriculum and teaching lessons for state wide specialists.

Hosted NC regional 4-H science academy with approximately 55 Missouri delegates. Networking with other LGU's and picking up on some of the resources has been a direct result

of these academy efforts.

Marketing logos, templates, - curriculum templates, new curricula related to science/STEM -Shared programs, curricula, etc. - Developed network of educators, support each other

Discussion about a 4-H STEM Specialist and this person's involvement on a campus-wideoutreach coordinators' committee.

As a result of participating in the National 4-H Science Academy, we have developed interestin increasing youth participation in science and agricultural science related careers. We have

also submitted a proposal to USDA/NIFA to build our extension staff capability to organize

youth in schools and communities with limited resources to develop interest in science. We

are utilizing the resources provided to us during the 4-H Science Academy to conduct effective

assessment of our science program

State-wide awareness of 4-H Science Mission Mandate Discussion on how to better organize 4-H programs and develop 4-H Project info sheets. more marketing to promote and recognize science efforts Development of the Science Rich 4-H Project Handbooks Resources leveraged in new state 4-H Science resource library and 17 new state 4-H Science

learning kits, new 4-H Science webpage, new state 4-H Science volunteer recruitment survey,

greater Extension investment in teacher & staff training, development of state 4-H Science

Post-programming technical assistance & support to extend 4-H Science learning. Increase in trainings offered to staff and volunteers around 4-H science. Have completed 4-H

Science training with at least 30 staff and over 150 volunteers.

Bigger focus on evaluating programs and efforts. More use of Train the Trainer withvolunteers Larger awareness and efforts with fundraising


40/48

33

Part Five

Narrative Question Content Analysis

Participants were asked three narrative questions: 1) list three things they planned to do

related to 4-H Science in the next three months; 2) list the main thing related to 4-H science

that they need additional support with the most; and 3) list three ideas/needs for post-

academy training opportunities. A content analysis of the 1663 responses to these questions

was conducted using the qualitative analysis software MAXQDA. For each question, comments

were categorized into common themes. The categories for each question, as well as the

percent in which each category was mentioned among the comments, are listed below (see

Tables 5.0 5.2). Some comments contained two or more themes and so one comment could

have several categories attached to it. The most frequently mentioned themes for each

question are described in slightly more detail with examples of typical comments under these

themes.

Three Plans to Implement in the Next Three Months

After analyzing the 889 comments, 19 different themes were decided upon that represented

the responses related to plans for the next three months (see Table 5.0). The most common

responses related to the following themes:

Planning, leading, developing, or assisting a science program (16.3% of responses)This included work on urban science programs, geospatial programs, science Saturdays, science spin

clubs, science camps, filmmaking, after-school programming, Youth Science Research Exhibition,

food science, garden projects, and science programs in general.

Recruiting, training, or informing volunteers and staff on science (14.4% of responses)This included teaching regional, state, and local volunteers, sharing with staff what was learned at

academy, training volunteers and staff to incorporate science in programs and lessons, recruiting

volunteers to facilitate science programs, training afterschool staff and agents, encouraging state

teams to develop trainings for staff and volunteers, designing a volunteer training, and leading

workshops.

Inquiry based learning (12.7% of responses)This included learning more about inquiry based learning, training others on inquiry based learning,

adding inquiry based learning to existing programs, using the inquiry method with all 4-H projects,

offering support with the inquiry based approach, providing hands-on inquiry based opportunities,

revising trainings to meet inquiry based standards, and adjusting lesson plans to include an inquiry

based approach.


41/48

34

Involving more science in existing programs (7.9% of responses)This included involving more science in camps, conferences, retreats, current projects (i.e. animals,

clothing, foods, shooting sports), workshops, judging events, afterschool programs, fairs, recognizing

science in all projects, and supporting volunteers in bringing out the science in their project.

Curriculum (7.4% of responses)This included developing 4-H science curriculum, using curriculum ideas to teach leaders, enhancing

curriculum, researching curricula, reviewing curricula to assure science model, adding inquiry based

approach to curriculum, informing others of curriculum to use, accessing websites for curriculum,

using the national curriculum development template, conducting webinars on 4-H Science

curriculum, and borrowing other states curriculum.

Table 5.0 Plans to Implement in the Next Three Months

PercentMentioned

Planning, leading, developing, or assisting science programs 16.3%

Recruiting, training, informing volunteers and staff on science 14.4%

Inquiry based learning 12.7%

Involving more science in existing programs 7.9%

Curriculum 7.4%

Securing partnerships 6.3%

Fund development (i.e. use fund development toolkit, identify opportunities, seek donors and

grants)

6.2%

State or county science plan of action (i.e. review, evaluate, or develop plan) 6.0%Evaluation of science programs (i.e. use evaluation tools, plan an evaluation, improve evaluation) 5.5%

Robotics (i.e. facilitate a robotics club, offer robotics workshops) 4.6%

Using webinars, on-line resources, and national website for information on science 2.6%

Networking with colleagues 2.2%

Obtaining and using science resources 1.8%

Promotion of science (i.e. make 4-H science more visible, include science in monthly newsletter,

promote science to youth)

1.6%

Obtaining more information on what was learned at academy 0.7%

Academy did not provide what was needed (i.e. no new information, too much lecture) 0.5%

Recruiting and training youth 0.4%Train others in Positive Youth Development 0.2%

Other (i.e. set goals, write articles for volunteers, survey needs in community, set dates for

meetings, write program descriptions)

2.7%


42/48

35

Response Themes for Additional Support Needs


the responses related to additional support needed (see Table 5.1). The most common themes

are listed below. It is interesting to note that Fund Development and Evaluation are the areas

in which most additional support is requested. This is a marked change from the Year Oneevaluation and follow-up that indicated the greatest need was in the area of scientific inquiry,

and perhaps yet another indicator that the knowledge and skill needs for facilitating 4-H

Science are shifting.

Themes:

More funds and fund development (19.7% of responses)This included funds for programs, staff, curriculum, and supplies, funding on county and state levels,

funding opportunities and training, and finding donors.

Evaluation support and tools (11.2% of responses)This included evaluations to show impact to partners and donors, how to use existing evaluation

tools, ready-to-use standard evaluations, activities that include evaluations, webinars on evaluation,

and understanding evaluation at different levels of program development and delivery.

More volunteers, staff, and manpower (9.7% of responses)This included finding, recruiting, maintaining, and training volunteers, more staff to run programs,

promoting volunteer opportunities, staff with evaluation experience, and volunteers with

commitment.

Materials, resources, and equipment (9.1% of responses)This included new equipment and computers, access to science materials and 4-H science resources,

easy and quick science resources, storage space, promotional materials, volunteer training resources,

tool-kits, one-page science activities, and curriculum.

Training related to science inquiry and hands-on science activities (7.9% of responses)This included more in-depth understanding of inquiry learning, inquiry based curriculum examples,

hands-on learning programs to share, tools to help volunteer and staff use inquiry-based learning,

applying inquiry based learning to areas that do not appearto be science related, teaching

traditional leaders to incorporate inquiry into existing programs, and hands-on demonstrations.


43/48

36

Table 5.1 Additional Support Needed

Percent

Mentioned

More funds and fund development 19.7%

Evaluation support and tools 11.2%

More volunteers, staff, and manpower 9.7%

Materials, resources, and equipment 9.1%

Training related to science inquiry and hands-on science 7.9%

Obtaining and developing curriculum and content (i.e. easy to use curriculum, how to use curriculum) 6.4%

Training volunteers and staff (i.e. state training for volunteers, training for staff to identify science in

programs)

5.5%

Time 4.8%

Institutional support, organization, and planning 4.2%

Utilizing science in already existing programs 3.3%

More information, workshops, or conferences on science 3.0%

Partnerships (i.e. establishing science partners, partnering with colleges and universities, adult-youth

partnerships)

2.7%

Promotion of science and programs 1.8%

Creating a plan of action 1.5%

Support of curriculum revision 1.5%

Access to materials at academy 0.9%

Applying what was learned at academy 0.9%

Other (i.e. update state website, knowledge on subject matter, consistency in implementation,

robotics)

5.8%

Response Themes for Three Ideas or Needs for Post-Academy Training Opportunities


the responses related to additional training opportunities (see Table 5.2). The most common

responses related to the following themes:

Recruiting and training of volunteers and staff (10.5% of responses)This included recruiting more committed volunteers, training volunteers, staff, and state leaders in

science, training on science needs, state and local training, face-to-face trainings, and training thatincludes a shadowing component.

Curriculum (9.8% of responses)This included training on national 4-H curriculum, curriculum that applies to different ages, easy-to-

use curriculum suggestions, adapting curriculum to inquiry-based, developing curriculum, hands-on


44/48

37

training using curriculum, how to use curriculum, online trainings for curriculum, and a place to

receive feedback on curriculum.

Sharing capabilities for resources, information, and success stories (9.4% of responses)This included resource site with links to other science 4-H web sites, blogs or eNewsletters that help

with sharing best practices, online community of academy attendees, regional and statewide sharing

of ideas, weekly tips on science learning, sharing of states marketing tools, games, and skills, website

listing key resources covered, contacts from academy, and shared materials.

Fund development (8.3% of responses)This included finding potential donors, funding support from NSF and/or USDA, funding sources and

opportunities, follow-up on fund toolkit, examples of a 4-H Science programs utilizing multiple types

of funding mechanisms, and training staff on fundraising.

Evaluation (7.9% of responses)This included how to use existing tools for evaluation, designing impact evaluations, evaluation

resources and strategies, how and when to evaluate programs, simple evaluations to go with

activities, and practical evaluation help.


45/48

38

Table 5.2 Three Ideas or Needs for Post-Academy Training Opportunities

Percent

Mentioned

Recruiting and training of volunteers and staff 10.5%

Curriculum 9.8%

Sharing capabilities for resources, information, and success stories 9.4%

Fund development 8.3%

Evaluation 7.9%

Inquiry-based learning (i.e. hands on inquiry training, mentor in inquiry training, integrating inquiry

into programs)

7.2%

Partnerships (i.e. developing and recruiting) 7.0%

Promoting and marketing science and information learned 4.5%

Specific science programs (i.e. Robotics and Gear teach 21) 4.3%

Webinars on various topics (i.e. curriculum, volunteers, funding) 4.0%

Informal science, integrating science into existing projects 3.8%

Hands-on science training 2.8%

Developing and implementing state/county plans 2.1%

Grant writing and grant opportunities 1.9%

Specific topics (i.e. PYD, citizenship and science, youth leadership, team building, learning styles) 1.9%

Another academy 1.7%

Train-the-trainer 1.5%

Technology (i.e. use of ScienceHub in my4-H.org, social media training) 1.3%

Resources (i.e. obtaining, training, and implementing) 0.9%

Follow-up on progress made by programs after the academy 0.9%

Time 0.6%

Professional development 0.6%

Science connected with other mandates 0.6%

Other (i.e. tour science museums, more handouts, individual follow-up, national science

requirements)

6.6%


46/48

39

Appendix 1: Feedback from New England on Effectiveness of Virtual

Academy

Table 13.0 Northeast New England additional questions

Frequency %How effective was the

virtual format of the

academy?

Not effective at all

Somewhat effective

Effective

Very effective

Missing

5

13

4

1

7

16.7

43.3

13.3

3.3

23.3

Do you think virtual

training is as effective as

face to face training?

Yes

No

Missing

7

15

8

23.3

50.0

26.7

Which type of training do

you prefer?

Virtual

Face to face, on site

No Preference

Missing

2

16

5

7

6.7

53.3

16.7

23.3

Why do you prefer virtual

training?

Reduced time away from office

Reduced Travel costs

More effective use of my time

Easier to focus on content that is presented

2

2

2

2

100.0

100.0

100.0

100.0

Why do you prefer face to

face training?More effective use of my time

Easier to focus on content that is presented

I learn more this way

Provides network opportunities

*Percentage was calculated as: the number who

responded/the number who said face to face

training was preferred

3

13

10

10

18.75*

81.25*

62.5*

62.5*

Open Ended Questions:

Comments about the Virtual Training format:

1. Test the technology FIRST. It was so frustrating to have presenters unfamiliar with the formatthey were using. There was no forethought about how to present to multiple locations. 2.

There was the ability to do hands on pieces- however this was not done. I think if the planning

team had spent more time thinking though ways to make this day more interactive the training

would have been much better.


47/48

40

At times the slide(s) on the screen were too small to read! I think we could just have easily taken part in this training from our own desks. It was difficult to

sit for long periods listening, and reading - incorporating more time to interact with the website

tools would have been helpful. As a region and state we rarely have time for discussion and

planning.

It could have been more effective if it was better organized. When we broke up into groups thedirections were consistently vague and hard to follow.

It was awful, totally boring and in my opinion misrepresented. Lapsed periods of engagement due to impersonal nature of this method of delivery. Too many

technical difficulties leading to distraction.

Not what I expected - way, way too much sitting and getting lectured to. Don't need someoneto read a handout to me.

Some of the presentations were helpful; others had technical problems that limitedeffectiveness. e.g. some slides had too much content, making them hard to read. Time was

wasted, even in effective presentations, when presenters had trouble loading their power points

or websites. URLs were not clearly displayed. Small group activities lacked sufficient direction

and structure.

The "hybrid" model of listening to a presenter and then spending time working with ordiscussing the information in our room was very effective. The slow pace of a webinar and the

slow conversation make is somewhat painful to sit through, but the savings in travel time is

worth it. Also, presenters could present less information and give people more time to explore

what they want to/need to know about that info.

The power points got tiresome after a while and the day could have been better organized withonsite personnel beforehand.

The technology was difficult, but in lieu of long distance travel, I thought it was OK. I think thehost could have had us do more experiential activities in between webinars

Too long Need active activities/break outs. Several things would have led well to engage in role-play like

approac

year two national 4-h science academy final report[1]

Documents