academic program: forestry and environmental resources · assessment/policy and administration)....

Academic Program: Forestry and Environmental Resources

College: Natural Resources

Year: 2012-13

Update: Use of Results for Program Improvement

FISHERIES, WILDLIFE, AND CONSERVATION BIOLOGY

Outcome(s):

1. Ability to apply critical thinking and knowledge of ecological principles to solve problems in fisheries, wildlife, and conservation biology

2. Ability to communicate proficiently in the kinds of writing and speaking required of fisheries, wildlife, and conservation biologists

3. Ability to work effectively in teams

Decision(s):

In the 2012-2013 Assessment Report for Programs in the Department of Forestry and Environmental Resources, the following three curriculum changes were proposed to address areas of concern in the Fisheries, Wildlife, and Conservation Biology (FWCB) program curriculum:

1. Addition of a new undergraduate program coordinator; 2. Improved undergraduate advising; and 3. Movement of summer camp to between the second and third years from between the third and

fourth years. All three of these proposed items have been addressed:

1. Lara Pacifici began as undergraduate coordinator of the Fisheries, Wildlife, and Conservation Biology program in July 2013.

2. Beginning in Fall 2013, all undergraduates attended one of seven group advising sessions where crucial information about courses and registration were shared and multiple FWCB faculty were present to help with course selection and general advisement. Students were encouraged to follow up with their assigned advisor for further professional advisement.

3. Students between their second and third years, as well as those between their third and fourth years will attend summer camp in 2014. In future years, all students will attend summer camp between second and third years. The FWCB curriculum was revised in Fall 2013 to reflect this change.

FOREST MANAGEMENT

Outcome(s):

To show that they can analyze the economic and social effects of forest management actions, giving particular attention to clients' objectives and the needs of various groups of people affected by the action.

Decision(s):

Data from the assessment of forest management program outcomes provided no clear reason to make alterations in the FOM program other than to emphasize the analytical intention of topic assignments in FOR405 during the fall 2013 semester. Emphasis on problem analysis in FOR405 was increased, and the resulting oral presentations were more analytical than in the previous fall semester. Possible changes in the FOR406 management planning course cannot be implemented until spring 2014.

NATURAL RESOURCES

Outcome(s):

Outcome #2: The ability to apply objective and critical thought to the solution of a variety of problems in natural resource management. Outcome #3: The ability to communicate proficiently in the kinds of writing and speaking required of natural resource professionals. Outcome #4: The ability to work effectively in teams.

Decision(s):

Paperwork to revise the two Natural Resources curricula in the Department of Forestry and Environmental Resources has been completed and accepted by the CNR Academic Affairs Committee for submission to the University Courses and Curriculum Committee. Course Action forms to revise NR301 and change the credits earned in NR360 are being completed. The entire curriculum package will be submitted in February for implementation during summer 2014.

Full report is included on following pages.

Assessment Report for Programs in the Department of Forestry and Environmental Resources

(DFER) 2012‐2013

This report addresses current assessment progress for three programs in DFER: Fisheries, Wildlife, and Conservation Biology (FWCB); Forest Management (FOM); and Natural Resources (NR) (ecosystem assessment/policy and administration). The Environmental Technology and Management (ETM) program report has been submitted separately. Progress in both FWCB and NR came as a result of revising assessment plans and actually implementing direct outcome measures in several courses (three in each curriculum). Outcome assessment in FOM continued its focus on direct measures in two senior courses. Results for each program are presented separately below.

Indirect measures include varied information gathering techniques such as focus groups of graduating seniors and university‐administered surveys. Numbers of participants in the instruments used for these indirect measures have been declining as a percentage of total students (numbers shown in Table 1.0). Thus, small numbers of responses generated by majors in our programs are probably unreliable indicators; for example, focus groups for graduating seniors held during the spring 2013 semester included 1 FOM graduate, 2 FWCB graduates, and 1 NR graduate. Timing of the scheduled sessions during the final exam period appeared to cause that problem and needs to be reconsidered. On the other hand, university survey participation has never been a high percentage of our students but at lower current levels offers even more unreliable data.

Table 1.0 Enrollment at Student Census Date 2012, by DFER Curriculum

Curriculum Totals Female

Environmental Technology and Management 182 56

Forest Management 80 7

Natural Resources Ecosystem Assessment 47 24

Natural Resources Policy and Administration 21 9

Conservation Biology 2 1

Fisheries 52 6

Wildlife Biology 134 60

All DFER 519 163

Fisheries, Wildlife, and Conservation Biology

Section 1 Student Learning Outcomes

Outcome 1: Ability to apply critical thinking and knowledge of ecological principles to solve problems in fisheries, wildlife, and conservation biology

Outcome 2: Ability to communicate proficiently in the kinds of writing and speaking required of fisheries, wildlife, and conservation biologists Outcome 3: Ability to work effectively in teams Section 2 Outcomes Assessed This Period The FWCB Assessment Map (Table 1.0) calls for assessments of the following during 2012‐2013:

Objectives 1 and 2 in FW 404, Forest Wildlife Management during fall 2012 Objectives 1 and 3 in FW 411, Human Dimensions of Wildlife Management during spring 2013 Objective 1 in FW 453, Principles of Wildlife Science, during spring 2013

Assessments of Objectives 1 and 2 in FW 404 during fall 2012 produced the following results: Outcome 1: Using scores for the Habitat Assessment (out of 20 points) and Habitat Management Recommendations (out of 20 points) portions of the rubric for the Stewardship Plan assignment in FW 404, we assessed student ability to integrate knowledge of wildlife‐habitat relationships to evaluate and develop a habitat management plan for an actual property. Groups of 3 students were required to submit 2 drafts, each of which received scores for the rubric. The averages for Habitat Assessment and Habitat Management Recommendations for the first draft were 14.9 and 14.4, respectively. For the final draft, the average scores were 17.5 and 16.6, respectively. Overall, groups improved in both categories following feedback from the instructor. However, groups scored lower on the Management Recommendations.

Table 2.0 FWCB Curriculum Map

Student Learning Outcome Statements

Course 1: FW 311

Course 2: FW 404

Course 3: FW 411

Course 4: FW 453

Year or semester to

measure Outcome

Other Measures

Outcome 1: Ability to apply critical thinking and knowledge of ecological principles to solve problems in fisheries, wildlife, and conservation biology

Stewardship plan

(assessment and mgt score)

Service learning

team project (community

partner evals)

Research question on study

design and analysis

(final exam scores)

Fall 2012 Spring 2013

Grad. Senior Survey, Senior Focus Groups, Alumni Survey

Outcome 2: Ability to communicate proficiently in the kinds of writing and speaking required of fisheries, wildlife, and conservation biologists

Team project (presentation

scores)

Stewardship plan

(grammar and clarity

score)

Fall 2012 Summer 2013


Outcome 3: Ability to work effectively in teams

Team project (peer

evaluation)

Service learning

team project (peer

evaluation)

Spring 2013 Summer 2013


Semester/ Year To Start in Course

Summer 2013 Fall 2012 Spring 2013 Spring 2013

Using scores for the Grammar and Clarity (out of 10 points) portion of the rubric for the Stewardship Plan assignment in FW 404, we assessed student ability to communicate in written form. The average

group score for the first draft was 7.7; for the second draft, groups improved their average score to 8.8. Extensive editing of the first draft contributed in large part to improved scores.

Assessment of Objective 1 in FW 453 during spring 2013 produced the following results:

Using scores for a question based on study design (see below), students were individually assessed based on a scale of 8 points. The results are as follows, based on scores from 46 students.

Min 1st Quiz Median Mean 3rd Quiz Max

2 3 5 4.525 5.250 8

The scores showed a slight improvement over the course of the semester; however half scored under 5 out of 8 points, indicating an inability to critically solve this problem.

Assessments of Objectives 1 and 3 in FW 411 during spring 2013 produced the following results:

Outcome 1: Four external observers were asked to evaluate students’ according to the following two questions, and those results are displayed in Table 2.1.

How would you describe the critical thinking skills demonstrated by students? How would you describe the students' ability to apply their knowledge of human dimensions of

wildlife principles to solving conservation problems?

Table 2.1 Observers’ ratings (#observers) of FW411 Students Relevant to Outcome 1.

Needs Improvement Competent Proficient Exemplary

Critical Thinking 100.00% (4)

Problem Solving 25.00% (1) 50.00% (2) 25.00% (1)

Outcome 3: The following questions were asked in order to generate data to assess this outcome. The first question was asked of four external reviewers, and the last four questions were part of the project rubric in the course. Results are shown in Table 2.2.

How would you describe the students' ability to work effectively in teams? Please rate your ability to effectively work in teams at the beginning of the semester Please rate your ability to effectively work in teams at the end of the semester Please rate your group members' ability to effectively work in teams at the beginning of the

semester Please rate your group members' ability to effectively work in teams at the end of the semester

Table 2.2 Ratings from FW411 Relevant to Outcome 3.


Ability to work in teams N=4 25% (1) 70.00% (3)

Your ability beginning N=51 33.33% 56.6% 9.80%

Your ability at end N=51 3.92% 70.59% 25.49%

Group members beginning N=51 32.69% 55.77% 11.54%

Group members at end N=51 9.8% 56.86 33.33%

Comparison of findings from previous years to determine improvement: We have no previous directly‐measured assessment results with which to compare the current data for FWCB, so these results establish a baseline for future comparison.

Indirect Evidence Collected for2012‐2013: Small numbers of students in focus group interviews (3 fall, 2 spring) make it difficult to judge concerns. Small percentages of students responding to surveys are similarly problematic for gaining meaningful insight about what to address as systematic problems versus idiosyncratic gripes. Are these small numbers indicative of broader issues? It is hard to discern any trend in the comments except the general concern for more hands on and practical experience rather than research focused instruction and the desire for better curricular and career counseling.

Interpretations and decisions from assessment data for 2012‐2013: Addition of the conservation biology concentration has potential to change the dynamics of the FWCB experience for students enrolled in this program. Students interested in non‐game and non‐commodity resource management topics and graduate study will have an experience differently focused than the traditional curricula in fisheries and wildlife biology. However, it seems that focus by the faculty on research and students’ desire for practical experience may be at odds. Section 3 Changes

Course changes that address some areas of concern: the only course change made since the last FWCB curriculum revision (Sum2’09) was hiring of a faculty member teaching FW453.

Curriculum changes to address areas of concern discovered during 2011—2013: Addition of a new program coordinator charged specifically with advising and professional development tasks for FWCB students may have an important impact on the undergraduate experience. Comments about advising have been a frequent complaint, and it is hoped students will find the advising situation improved with this new person taking charge. A plan to advance summer camp to between the second third years from between the third and fourth years may address a variety of issues such as preparation for internships and opportunities for curriculum‐related work both on and off campus.

Forest Management

In FOM, we continued to focus on the senior courses (oral presentations in FOR405 and field tours in

FOR406) to evaluate outcomes. We increased faculty and stakeholder involvement to measure

achievement in 2012‐2013, thereby increasing the reliability of data.

Section 1 Student Learning Outcomes

Table 3.0 Learning Outcomes for Forest management Curriculum

1. manage forests effectively. • to show that they understand forest community, structure, and function and can apply their

understanding to managing forests. • to show that they can apply the financial and regulatory knowledge of forestry to managing forests:

financial analysis; taxation policies; and local, state, and federal regulations, policies, and laws.

• to show that they can apply the technical skills of forestry to managing forests: inventorying a forest, scheduling silvicultural activities, developing silvicultural prescriptions, using GIS and GPS systems, identifying trees, and diagnosing and treating forest pests.

2. apply a broad perspective to forest management. • to show that they can analyze the economic and social effects of forest management actions, giving

particular attention to clients' objectives and the needs of various groups of people affected by the action.

• to show that they can assess the long‐term environmental effects of forest management actions, the impact on all the eco‐system components affected by the action: wildlife, water, soil, air, as well as trees.

• to show that they possess a strong ethical sense of stewardship for the environment. 3. apply objective and critical thought to the solution of a variety of problems in forest management. • to show that they can define and analyze a problem: the present state, the goal state, the

constraints, and the criteria for an effective solution to the problem. • to show that they can gather information from their own experience and from outside sources

pertinent to solving the problem, balancing the technical information and hard data with the vision, needs, and priorities of others affected by the problem.

• to show that they can synthesize and apply this information to generating possible solutions to the problem.

• to show that they can evaluate the possible solutions in terms of the criteria and make a case for a preferred solution.

4. communicate proficiently in the kinds of writing and speaking required of forest management professionals. • to show that they have command of the kinds of written communication appropriate to the field,

such as memos, business letters, data summaries, technical reports, feasibility studies, and management plans.

• to show that they can speak knowledgeably and effectively in the kinds of spoken communication appropriate to the field, such as impromptu presentations, technical reports, and field tours.

5. work effectively in teams. • to show that they can manage teamwork productively. • to show that they can lead and enhance the performance of others.

Section 2 Outcomes Assessed This Period

All outcomes identified above were to be assessed in this period through the combination of measurements in FOR405 and FOR406. Faculty members were invited to observe senior students presenting results of their assigned analysis papers in FOR405. [Averages of those results are not available in Sweden, where this report is being completed. They will be incorporated in August]. The focus of the assessment was on ability to present researched and analyzed material in a conference setting, taking questions and explaining approaches used.

The FOR406 field tour assessment tool has been in use for three decades, gathering information on the variety of technical components included in the curriculum and demonstrated in the planning effort students accomplish in that capstone course. Table 3.1 Assessment Tool Given to Faculty Participating in Senior Conference Talks in FOR405 ____________________________________________________________________________________ Composite Observed Score Above At Below Evidence of preparation 7 6 5 4 3 2 1 Satisfying level of engagement 7 6 5 4 3 2 1 Deliveries relatively distraction‐free 7 6 5 4 3 2 1

In For 406, in addition to faculty members who usually participate, we invited alumni professionals from varied professional organizations to serve as external reviewers during the field tours conducted in April. Table 3.2 presents the average rating from five of these reviewers using a 7‐point (7 high) scale. [Data from additional reviewers are in Raleigh and will be incorporated in the analysis so values in this table may change in August]. Table 3.2 Forest Management Senior Field Tour Assessment Ratings ______________________________________________________________________________________ 2011 2013 Knowledge of current resource status Average (n=8) Average (n=5) Timber (reference inventory, $ values, etc 5.63 4.4 Soil and water 4.75 3.8 Wildlife requisites 4.13 4.4 Knowledge concerning resource management Silviculture 5.75 4.8 Wildlife 4.25 4.4 Recreation and associated topics 4.50 4.8 Decision analysis Reference to quantitative methods 5.06 5.0 Understanding of markets and finances 4.75 3.8 Knowledge of regulatory limitations on practices 4.94 4.4 Knowledge of practical limitations on practices 4.81 3.6 Knowledge of operational considerations 5.38 3.4 Potential and probable consequences of proposed actions on timber product quality (growth and yield, etc.) 5.56 4.4 wildlife habitats (game and non-game species) 4.19 4.2 water quality 4.81 4.6 visual quality 4.88 4.2 landowner's economic situation 5.50 3.8 recreation opportunities 4.38 4.8 Recognition of professional ethical obligations to clients, society, and the environment 5.13 5.2 To organize and conduct audience-focused tour 5.38 4.8 To speak knowledgeably about chosen topics 5.38 4.0 To address questions on or off chosen subjects 5.25 4.0 Conformity with standards of normally spoken English 5.63 5.4 _____________________________________________________________________________________

Comparison of findings from previous years to determine improvement: data averages in Table 3.2 show modest change (<0.68) for twelve of the items, with six ratings increasing, but only slightly, from 2011 to 2013. However, the other ten scores changed anywhere between ‐0.95 to ‐1.98 points. The overall average change for 22 rated items was ‐0.63, so the average total score was 4.37 in 2013 compared to 5.0 in 2011. This rating is still on the high side of the expected range, as scored. Indirect Evidence Collected for 2012‐2013: As noted above, too few graduates participated in focus groups during the 2012‐2013 academic year to provide representative information. Of 16 alumni survey respondents in 2012, 93.8% reported being very or moderately satisfied that the FOM curriculum provided a broad perspective. To more specific prompts, they rated their preparation in five important curricular areas as shown in Table 3.3. Table 3.3 Percentages Reported from 2012 Alumni Survey.

Subject Area of Preparation Rated N Fair Average Good Excellent

Eco‐Bio Concepts, Principles, Processes 15 6.3 6.3 18.8 68.8

Measure, analyze and project trees/ Conditions

15 6.7 13.3 6.7 73.3

Scientific knowledge and Technical skill 15 6.7 6.7 6.7 80.3

Objective/critical thinking 15 6.7 13.3 80.0

Writing and speaking professionally 15 6.7 20.0 73.3

Interpretations and decisions from assessment data for 2012‐2013: At a faculty assessment day meeting in December immediately following the senior conference presentations in FOR405, some concern was raised that students were not uniformly able to go beyond mere reporting of gathered information to analyze a topic critically. Students’ ability to make an oral presentation was uniformly considered acceptable, but the weak content development and lack of analytical methods in some presentations suggested need for attention. The decline from 2011 in overall ratings for the 2013 field tours shown here may or may not be significant and will need to be examined again when all the data are processed, and when the missing data for April 2012 tours are included. The greatest drop in scores occurred in the area of decision analysis (‐0.95), which may relate to the concern about student’s analytical ability expressed in December. Section 3 Changes

Course changes that address some areas of concern: no significant concerns had been identified

in previous assessment using course‐based measurements, and no changes have been made in courses

in the last year. Several suggestions from alumni external reviewers of the field tours in April 2013 are

possible adjustments that could be made to better guide students as they create their management

plans in FOR406. These would be implemented in the spring 2014 semester.

Curriculum changes that address areas of concern discovered during 2011—2012: no curriculum

changes were implemented because no serious concerns about instruction or the curriculum were

identified during the previous assessment period.

Natural Resources (ecosystem assessment/policy and administration)

Meeting with Dr. Carrie Zelna in June 2012 to examine existing goals, outcomes, and measures for NR

program assessment, we decided to revamp the assessment plan for NR, as was noted in last year’s

assessment report. Then meetings with the other faculty involved in the natural resources curriculum

confirmed that the new plan shown below was workable and could be implemented last fall and this

spring. Results from that revision effort (Table 4.0) and measures taken in courses are discussed below.

Section 1 Student Learning Outcomes.

Outcome #1: The ability to apply a broad perspective to natural resource assessment. Outcome #2: The ability to apply objective and critical thought to the solution of a variety of problems

in natural resource management. Outcome #3: The ability to communicate proficiently in the kinds of writing and speaking required of

natural resource professionals. Outcome #4: The ability to work effectively in teams.

Table 4.0 Natural Resources Curriculum Map

Student Learning Outcome Statements

Course 1: NR300

Course 2: NR400

Course 3: NR484

Internship Year or semester to

measure Outcome

Other Measures

Outcome 1: ability to apply broad perspective to natural resource assessment

Group project, cases

Quizzes, Tests, Project

Test questions, Cases


Grad. Senior Survey, Senior Focus Groups Alumni Survey

Outcome 2: ability to apply objective and critical thought to solving a variety of problems in natural resource management

Group project, cases

Quizzes, Tests, Project

Test questions, Cases

Supervisor Evaluation

Fall 2013 Spring 2014 Summer 2013


Outcome 3: ability to communicate

Group project,

Presentations Technical Reports


Grad. Senior Survey,

proficiently in the kinds of writing and speaking required or natural resource professionals

cases Supervisor Evaluation

Summer 2013

Senior Focus Groups Alumni Survey

Outcome 4: ability to work effectively in teams

Project Project? Project Supervisor Evaluation

Spr/Fall 2013 Summer 2013


Semester/ Year To Start in Course

Spring 2013

Spring 2013 Fall 2012 Summer 2013

Section 2 Outcomes Assessed This Period

Outcome 2 was assessed in NR484 during the fall 2012 semester using a rubric applied to a portion of the final exam that culminated the semester‐long impact assessment project. Ratings are shown in Table 4.1.

Table 4.1 Rating Performance in Using Objective and Critical Thinking in Impact Assessment (n=14)


Technical Accuracy 25.35% 25.35% 43.40% Use of Evidence 25.35% 25.35% 43.40%

Outcome 3 was assessed in NR484 during the fall 2012 semester using a rubric applied to a written

assignment. Fourteen NR students were assessed in this course, and the results are shown in Table.

Table 4.2 Rubric B Assessing Writing Performance in NR484 Fall 2012 (n=14)


Content Development 7.00% 28.50% 43% 21.40% Style and Format 21.40% 14.20% 43% 21.40% Overall Presentation 14.20% 21.40% 43% 21.40%

Outcome 4 (Teamwork) assessment was implemented in Natural Resources Measurements (NR 300). Students peer‐assessed their teammates on an integral 1 (poor) to 3 (excellent) scale for five characteristics:

Shows up for planning meetings

Contributes to ideas and discussions Carries fair share of work

Completes assigned tasks on time

Overall rating of performance

For each student, the five scores were averaged and rescaled to 0‐100. Ninety percent of students scored 90 or above for the 2013 course offering.

Figure 4.1. Teamwork assessment at mid‐semester and end‐semester. Stacked bars show cumulative distribution of average score at each skill level. 90% of students scored 90 or better for teamwork; there was no change at this skill level between mid‐ and end‐semester. Number of students in each category shown in bars. Comparison of finding from previous years to determine improvement: the data from NR484 are consistent with data from prior years taken along with data for ETM students in that course. No previous measures were taken in NR300 or NR400.

Indirect Evidence Collected for 2012‐2013: While the focus group number of total NR students was small (6), the number represents a higher percentage of graduating seniors than for the other curricula, and the five in December were actually 71% of the class graduating then. The comments in the focus groups were widely varied, idiosyncratic and therefore sometimes contradictory about components or courses in the curriculum. What one person liked another didn’t, or what would add another had already received sufficient experience with. The focus on preparation for career performance in this curriculum was praised but suggestions for improvement in certain features and courses should be examined as NR curriculum revisions are being considered. The NR focus group suggested that overall the program is reaching its objectives aimed at preparation for professional positions and graduate education. Some students perceived weakness in the NRP curriculum in comparison to the NRE curriculum but that is perhaps a difference from person to person in terms of goals or attributes.

Interpretations and decisions from assessment data for 2012‐2013: NR484 needs to return its focus for NR students to developing their field‐based assessment skills. The addition of ETM students (now a majority in the course) diluted the biological assessment and analysis function of the course for NR and changed the focus of the planning exercises in the course. Corroborating comments from the ETM focus group suggest a change in focus and dynamics in the course is needed for those students as well. Otherwise, certain minor issues with the NR curricula have been pondered by the involved faculty for a while, and a plan for revising the curricula in minor ways is in process. Shifting hour allocation among the NR301‐NR360‐NR401 sequence and bolstering the content in NR301 is the main focus. Providing elective alternatives in several key points (second economics course; FOR353‐GIS410; CH220/221 or CH201/202) in the curriculum could address some students’ desire for more technical depth or focus. Section 3 Changes

Course changes that address some areas of concerns: NR484 will institute a split laboratory experience

in the fall of 2013, with NR and ETM students working on substantially different projects. This will return

the NR students to a field‐based practicum applying the NEPA process to site development scenarios.

Periodic submittal of reports that contribute to an environmental assessment document focused on

biological resources will increase guidance in writing and provide better performance evaluation.

Curriculum changes that address areas of concern discovered during 2011—2012: We intended to

address a problem identified by students concerning cost of summer enrollment in NR360 (the

internship) but did not accomplish this in 2012‐2013. Our plan is to alter credit allocation in NR301‐

NR360‐NR401 so only one credit is allocated to NR360, reducing summer fees, but also increasing credit

allocation and content requirements for NR301 and 401.

Conclusions from DFER Assessment in FWCB, FOM and NR Programs

After some delay, DFER created manageable plans for assessing outcomes in two programs: FWCB and

NR. Focus on upper division courses and activities in several key courses should be sustainable. We

gathered baseline data for these programs in 2012‐2013 and we will proceed with implementation in

2013‐2014. Assessment of internship documentation in NR360 will be initiated this summer and NR484

measures will be repeated in the fall semester.

In the meantime we have also realized the FOM plan devised over a decade ago may be too complex.

We have been able to sustain measurements in senior capstone courses but encountered a basic

problem with using lower division courses as points of measurement: transfer students constitute half

or more of our graduating cohorts. This situation is definitely the case in NR as well. Our main

assessment tool in FOM, the senior management field tours may be supplemented with a faculty team

review of management planning documents the students produce, which to this point has been

conducted solely by the faculty member teaching the FOR406 course. Organizing such an effort should

be a goal in 2014.

Monitoring the analytical skills of FOM majors, the advising afforded FWCB majors, and instituting

changes to the NR curricula are priorities for the next cycle.

academic program: forestry and environmental resources · assessment/policy and administration)....

Documents