assessing the effectiveness of your academic advising program tom grites assistant provost the...

Assessing The Effectiveness Of Your Academic Advising

Program

Tom GritesAssistant ProvostThe Richard Stockton CollegeTom.Grites@stockton.edu

What are your expected learning outcomes for this webinar?

A. Why do we assess?B. What do we assess?C. Where do we begin?D. What tools do I use?E. Who defines our success?F. What I need to do tomorrow.

Audience Poll

• Terminology and Rationale• Assessment as a Process• Focus on the Contexts• Non-Assessment• Summary

Overview

• Assessment

• Evaluation

• Measurement (the tools)

Terminology

Assessment(re: academic advising)

“Assessment is the process through which we gather evidence about the claims we are making with regard to student learning and the process/delivery of academic advising in order to inform and support improvement”

(Campbell, 2008)

Uses/Contexts

Assessment – tends to be more related to programmatic issues and outcomes

Evaluation – tends to be more related to people (advisor) skills, performance, and outcomes

It’s OK to use evaluation as part of the assessment process

• Formative – more associated with assessment; includes a wider range of efforts; requires more analysis; provides a broader perspective; focus on improvement

• Summative – more associated with evaluation; more limited effort; focus on “Does it work?” or “How well was job performed?”

Intentions (related to both)

The Rationale

• “…a lack of assessment data can sometimes lead to policies and practices based on intuition, prejudice, preconceived notions, or personal proclivities – none of them desirable bases for making decisions”

• (Upcraft and Schuh, 2002, p. 20)

“In God we trust; all others bring data.”

“An ounce of data is worth a pound of opinion.”

(Magoon, c. 1975)

More Rationale

• Accountability• Effectiveness• Accreditation• Trustees/Regents• Legislators

• Program Improvement (to monitor and improve student success) – the most important reason

Other Reasons

The Assessment Process:A Cycle

• Resources:

• Assessment of Academic Advising Package (3 CDs available from NACADA via www.nacada.ksu.edu)

• Assessment of Academic Advising Institute

(Feb 12-14, 2014 Albuquerque, NM)

• Complete set of advising constituents (students, staff and faculty advisors)

• Broad range of key offices (Registrar, Enrollment Management, similar advising units, certain campus referral resources, IR office)

• Critics, Antagonists, and Naysayers

• FYIs – Faculty Senate, Deans Council, Retention Committee, others as appropriate

Getting Started: Identify Stakeholders

The Advising HubThe Advising Hub

• Focus on student learning• Connect learning to mission, vision, values,

goals in your advising program– How will your program contribute to

student learning?– Who, what, where, when, how will learning

take place?• Define measures of student learning

– Gather evidence, set levels of expected performance

What Do We want To Know or Demonstrate as a Result of Academic Advising?

• Alignment with institutional and unit missions

• Specify goals and/or objectives

• Identify the outcomes expected (student learning and/or programmatic)

• Gather evidence (the measurements)

• Share findings, interpretations, and recommendations

• Begin implementation and re-start the cycle

The Assessment Process/Cycle

Mission/Purpose

• A working model…

• Academic advising is integral to fulfilling the teaching and learning mission of higher education. Through academic advising, students learn to become members of their higher education community, to think critically about their roles and responsibilities as students, and to prepare to be educated citizens of a democratic society and a global community. Academic advising engages students beyond their own world views, while acknowledging their individual characteristics, values, and motivations as they enter, move through, and exit the institution.

• (Preamble, Concept of Academic Advising, NACADA, 2006)

These need to emanate from and reflect the nature of the unit to be assessed (total institution, Advising Center and its clientele, College Dean’s Office, etc)

Examples:

• To assist students to become independent and lifelong learners

• To assist students in understanding the relevance of the total curriculum

• To assist students in making good decisions based on their own evidence (e.g., selecting a major)

Goals/Objectives(how we intend to achieve our mission)

Student Learning Outcomes – examples• All students will select an appropriate major by the end

of their third semester.

• All students will become engaged in at least one co-curricular activity each semester.

• All students will be able to identify and will select courses that enhance their human capital.

• At least 30% of the students will choose to participate in a service learning course.

• All (CC) students will be able to distinguish among the A.A., A.S., and A.A.S. degree programs

Identify Outcomes

• How many courses are in your Catalog? (A)

• How many courses are required to earn a degree from your institution? (B)

• What percentage of what your institution offers do students actually take in order to earn a degree? (B/A)

• Now, for each course a student takes, how many are eliminated?

A Task For You…re: Course Selection

• Programmatic/Process Outcomes – examples

• As a result of our advising services, the retention/persistence rate of first-year students will increase by 10% in the next 3 years.

• As a result of our intervention strategies, the percentage of students who are removed from academic probation will increase by 10% in the next academic year.

• After two advising sessions, all students will come to their future sessions with a degree audit already run and with a plan for meeting outstanding requirements

Outcomes (continued)

Everybody’s Favorite

All students will be able to understand, appreciate, and articulate the value of general education.

• Not all outcomes will necessarily occur as a direct result of what we do as advisors, so we need to know what other learning opportunities exist in order for the students to meet our stated goals/objectives.

• WHAT learning is to occur?• WHERE might it be learned?• By WHEN should it be learned?

*This process can also inform the kinds of evidence that need to be gathered for appropriate assessment.

Gather Evidence Mapping the Experience (Maki, 2004)*

The Advising HubThe Advising Hub

Qualitative – open-ended survey questions; focus groups; in-depth responses, but small N

Quantitative – descriptive, structured, numbers and statistics from surveys, demographics, etc; limited content responses, but large N

Types of Measurement and Data

Direct – observations; recorded data; pre-post information

Indirect – perceptions, inferences, even “inclinations”

Use Multiple Measures!!!

• Satisfaction Surveys (OK, but not enough)

• Institutional Data (changes of major, drop/add transactions, grades in gateway courses, retention and graduation rates, use of services provided elsewhere, advisor : advisee ratios, NSSE, etc)

• Office Data (number of appointments vs. walk-ins, nature of sessions, results of sessions, transcript analyses, other advisor tasks/activities; “What did you learn?”)

• Focus groups (of clients, of faculty advisors, others – a qualitative measure)

• The Advising Syllabus* can inform what evidence should be collected

*http://www.nacada.ksu.edu/Clearinghouse/AdvisingIssues/syllabus101.htm http://intraweb.stockton.edu/eyos/page.cfm?siteID=123&pageID=42#syllabus

Gather (Multiple) Evidence

Share the Results

Tips…

•Be sure that the stakeholders you identified earlier are informed throughout the process in order to enable their support in the decision-making for implementation of your recommendations.

•Academics have a preferred method of review, so it makes sense to conform to their expectations.

These elements are often best provided in a standard research report or journal format…

Purpose of the assessment projectMethod of data collectionResults foundInterpretation of the resultsRecommendations with timetable for and anticipated cost of implementationExecutive Summary or Abstract

Sharing the Results(Format and Content)

• Revise pedagogy or curriculum or policy/procedure

• Develop/revise advisor training programs

• Design more effective programming– advising, orientation, mentoring, etc.

• Increase out-of-class learning opportunities

• Shape institutional decision making– planning, resource allocation

How Results Will Inform Decision-Making

• Redesign the advising effort in the Orientation Program

• Develop a peer advising/mentoring program

• Streamline office procedures • Initiate proposals for policy changes• Improve communication with other

service offices and personnel• Request/Reallocate resources (human,

fiscal, and/or physical)

Sample Implementation Recommendations

You Did It!!

• This will complete the assessment cycle, which provides the evidence for change and improvement.

• Completion of the cycle may also provide new goals and objectives, new assessment strategies and tools, and other aspects that will be need to be included in beginning the next cycle.

(See Darling, 2005 handout)

You’ve Earned a Break

Please take a few minutes to submit any questions you may have at this point via the chat function.

Back to the Original Contexts

People…

Academic advising, as a teaching and learning process, requires a pedagogy that incorporates the preparation, facilitation, documentation, and assessment of advising interactions. Although the specific methods, strategies, and techniques may vary, the relationship between advisors and students is fundamental and is characterized by mutual respect, trust, and ethical behavior.

(Concept of Academic Advising, NACADA, 2006)

NACADA Core Values

Academic Advisors are responsible

to the individuals they advisefor involving others, when appropriate, in the advising processto their institutionsto higher education in generalto their educational communityfor their professional practices and for themselves personally

• SELECTION

• TRAINING

• EVALUATION

• RECOGNITION/REWARD

Assessment (Evaluation) of Advisors

• Use the best• Add from other resources/units• Target specific populations• Cross disciplinary lines• Develop mentors• Use other skills/expertise

Selection of Academic Advisors

Making a distinctionFaculty Advising (Programmatic;

Assessment)Faculty Advisors (Personal; Evaluation)

Inappropriate ComparisonsProfessional Academic AdvisorsPeer Advisors

No Improvement PlanTraining

Potential Pitfalls

Faculty vs. Professional Staff Advisors

• Too often all are expected or required to advise, but also teach, publish, seek grants, etc – no selection

• Training ranges from near nothing to perhaps a day or 2, but usually only a few hours

• Evaluation is not systematic

• Recognition/Reward is very limited in the tenure and promotion process; mostly intrinsic; can also be a reverse structure (better = more)

• They are hired via a search process and have specific job descriptions – they are selected

• Their training is systematic, intentional, and ongoing; staff development is expected

• They are evaluated through annual performance reviews

• They are rewarded with salary and benefits

• 37 % OF ALL INSTITUTIONS HAD NO PERFORMANCE EVALUATION MEASURES FOR

FACULTY IN THEIR ACADEMIC ADVISING ROLE

• 44 % in 2 yr public institutions

• 25 % in 4 yr public institutions

• 39 % in 4 yr private institutions

(Habley, 2004)

ASSESSMENT (Evaluation)

• Faculty Contract

• List of Responsibilities

• Availability of Resources

• Assignment of Advisees

• Recognition/Reward

PARAMETERS (faculty advisors)

• Self evaluation

• Student surveys (locally designed)

• Survey of Academic Advising (ACT)

• Academic Advising Inventory (NACADA)

• Student Satisfaction Inventory (Noel-Levitz)

• NACADA Clearinghouse

Tools for Assessment (and/or Evaluation)

Of Advisors

Back to the Original Contexts

• Program…

• “…a lack of assessment data can sometimes lead to policies and practices based on intuition, prejudice, preconceived notions, or personal proclivities – none of them desirable bases for making decisions”

• (Upcraft and Schuh, 2002, p. 20)

• Satisfaction Surveys • Institutional Data• Office Data• Focus groups• The Advising Syllabus • External Reviews• CAS Standards• Others…

Other Tools and Strategies

CAS Assessment Worksheet

An Economic Model

• Though not an outcomes-based model per se, this approach to assessment is a functional analysis based on the premise that every task an advisor performs and every operation that an advising unit conducts has some monetary value related to it.

• The analysis results in a comparison of the fiscal expenditures required to perform the tasks to the cost benefits as results.

• The model operates from the perspective of a threat to the existence of an advising unit, function or personnel. A quick example…

• Identify every function the unit performs

• Identify all possible alternatives for each function, if the unit was dissolved

• Determine the cost of those functions that cannot be replaced and who would perform them; estimates will sometimes be required

• Determine the cost of those functions that could be eliminated

(In Markee and Joslin, 2011)

Determining Your Worth

Where are the data?

Bill Gates – “colleges today know more about how many kids attend basketball games and which alumni give money than how many students showed up for economics class during the week…”

(jn review of Academically Adrift).

Where are the Data?

Jeff Selingo – “Think about it. Before we buy a car, we can find various measures on everything from gas mileage to results of safety tests. We can turn to objective sources to check comparisons of similar vehicles and see which cars hold their value over time. But when it becomes to potentially one of the most expensive purchases in a lifetime, the attitude from colleges has always been that we should just trust them on the quality of their product.” (p. 25)

• Student expectations, intentions

• Whether advising strategies actually can be attributed to different types of student success (removed from probation, successful choice of major, overcome a skills deficiency or harmful social habit, etc)

• Retention and graduation rates of transfer students

What Are We Not Assessing…And Should We Be?

Expectations vs. Experience

Expect ExperienceBe Undecided 7% 20%Change Majors 12 65-85Fail a course 1 16Extra time to complete degree 8 60Drop out 1 40Transfer institutions 12 28Work while in school 36 60Seek personal counseling 6 27Need tutoring 15 20Seek career guidance 5 25(Habley 2011)

• Use and value of articulation agreements – number of students who use them, are they updated

• Currency of academic policies, e.g., course repeats, course pre-requisite criteria, drop/add/withdrawal processes, academic warning, probation, and suspension policies

• Does advisor training result in better advising?

Non-Assessment (continued)

• Assessment is a process, not an event

• Collaboration and cooperation are necessary for productive assessment to occur

• “An ounce of data is worth a pound of opinion” (Magoon, c. 1975) Avoid the N of 1 syndrome

• The purpose and results of assessment should always be used for program and/or advisor improvement in order to realize maximum student development and success

Summary

Questions?

Tom.Grites@stockton.edu