1

WASC Evaluator Workshop

Fall Visits 2009

2

Workshop Outcomes

• Understand WASC’s three-stage process and how your visit fits into the process

• Be familiar with the WASC Standards and CFRs and how to use them

• Know how to prepare for and conduct an effective visit and produce a useful, high-quality team report

• Be prepared to make sound judgments about institutions under the Standards

• Be familiar with resources that support your work on a team

3

AgendaI. Context for the Visit/Accreditation

II. WASC Three-Stage Review Process

III. Standards and CFRs

IV. Preparing for the Visit

V. Conducting the Visit

VI. Developing Team Recommendations

VII. Writing the Team Report

VIII. After the Visit

4

Context for Accreditation and Visits

• The Continuing Evolution of the WASC Process and Standards

• The Accountability Movement– Retaining Peer Review

• The Impact of the Economy

• Value Added, Collaboration, and Ongoing Efforts to Refine and Improve

5

The WASC Review Process

6

Three-Stage Process

1. Institutional Proposal: Identifies priorities, themes/areas of emphasis, and outcomes. Aligns work with institutional plans and needs.

2. Capacity/Preparatory Review: Focuses on capacity (systems, policies, resources) and readiness for educational effectiveness.

3. Educational Effectiveness Review: Focuses on results, findings.

7

The Two Reviews

Capacity and Preparatory

• Preparatory = readiness for the Educational Effectiveness Review

• Capacity = purposes, integrity, stability, resources, structures, policies, processes

Educational Effectiveness

Demonstrating:• Student learning• Institutional learning• Evidence-based decision-

making

8

The CPR and EER as a Whole

• The CPR evaluates what an institution has for infrastructure (staff/faculty, resources, processes, facilities, systems, structures).

• The EER evaluates how well that infrastructure works and the results that the institution achieves.

9

Navigating Multiple Purposes (1)

Focusing on the institution

Applying the Standards and CFRs

Focusing on Proposal themes/topics

Evaluating capacity and effectiveness under Standards; addressing team-identified issues

Reviewing the whole institution

Focusing on specifics, e.g., distance education, samples of program reviews

Advancing institutional development

Addressing “compliance” matters

10

Navigating Multiple Purposes (2)

Allowing flexibility and experimentation on visit

Ensuring consistency and fairness among visits and quality control of visits and reports

Using the CPR to evaluate EE readiness

Leaving evaluation of educational effectiveness until EER

Supporting institutional creativity and excitement

Reporting to the Commission and serving the public

11

The Special Visit

• Intended to monitor institutional progress on issues identified by the Commission

• May or may not be connected to a sanction

• Limited to a few specific areas of concern

• Intended to assess how institution will move into compliance (if on sanction)

12

Understanding the Team’s Impact

Why were you chosen for a team?

• Peer review is the foundation of accreditation.

• The team report forms the basis for the Commission action and its letter.

• The team report and action letter inform the work of the institution for years to come.

13

Working with the Standards and CFRs

14

Understanding the Standards and CFRs

• Two Core Commitments: Capacity and Educational Effectiveness

• Standards: Broad, holistic, encompassing

• Criteria for Review: More specific and detailed

• Guidelines: Ways to demonstrate compliance with the relevant CFR

15

Using the Standards and CFRs

• Team judgments must be linked to specific Standards and CFRs

• CFRs must be cited in reports

• Standards and CFRs form the basis for Commission decisions

• Standards and CFRs provide a context for continuous quality improvement

16

STANDARD 1:Defining Institutional Purposes and Ensuring Educational Objectives

Institutional PurposesIntegrity

17

STANDARD 2:Achieving Educational Objectives

Through Core Functions

Teaching and LearningScholarship and Creativity

Support for Student Learning

18

STANDARD 3: Developing and Applying Resources

and Organizational Structures to Ensure Sustainability

Faculty and StaffFiscal, Physical & Information Resources

Organizational Structures & Decision-Making Processes

19

STANDARD 4: Creating an Organization Committed

to Learning and Improvement

Strategic Thinking and Planning Commitment to Learning and Improvement

20

Tool: Standards at a Glance

21

Comparing the Two Visits: Different Views of a CFR (2.6)

The institution demonstrates that its graduates consistently achieve its stated levels of

attainment and ensures that its expectations for student learning are embedded in the standards

faculty use to evaluate student work.

22

CFR 2.6: Two Views

Capacity and Preparatory

Has the institution defined expected levels of attainment for SL?

Are they embedded in the standards and measures for student work?

How does the institution know if students are meeting expectations?

What data are collected and how analyzed?

How is student learning measured?

Educational Effectiveness

What do data show about student learning?

Are data disaggregated and analyzed?

Did the students learn what the faculty intended them to learn? At what levels of performance?

Has the institution used data to make changes and/or improvements?

23

Changes in 2009:

• Implement changes to Institutional Review Process re: Student Success, Program Review and EE Sustainability

• Implement changes to CFRs• Clarify the scope of the CPR visit to review the

“infrastructure” for assessment of student learning• Examine Program Review and Program-Level

Student Learning in a systematic way• Allow teams more time together on visits

Tool: Table A & B (RB pg. 47)

24

Preparing for the Visit(Visit Guide, Part II, pp. 29-52)

25

Timeline For CPR/EER Reviews

12 weeks 2 months

Institution mails report to team and

WASC

Team holds conference

call

Site visit held and team report

written

Institution responds to

errors of fact in team report

Institution responds to final team

report

Commission acts at

February or June meeting

26

Roles and Responsibilities of Team Members and Staff

• Role of team chair (RB pg. 189)

• Role of team assistant chair (RB pg. 191)

• Role of assigned WASC staff liaison (VG pg. 7)

• Team assignments

27

Pre-visit Preparation

• Read all the documents from WASC– Standards, CFRs, policies, visit guide, rubrics– Background documents re: institution and purpose of

the visit, including Proposal and/or last action letter/team report

• Read the institutional report• Review the data portfolio and exhibits

– What to look for and highlight?Tools: Timeline (VG pg. 8, VG pg. 29)

28

Reviewing the Exhibits

• Enrollment data– Headcounts and FTE

• Graduation data• Faculty data• Key financial indicators• Inventory of Educational Effectiveness Indicators• Inventory of Concurrent Accreditation and Key

Performance Indicators

Tool: How to Review WASC Data Exhibits (RB pg. 61)

29

Reading the Report

• Has the institution done what it said it would do in its Proposal?

• Has it collected and analyzed data effectively?• Are its conclusions supported by evidence?• Are there serious problems or potential areas

of noncompliance?• Does the report contain recommendations for

further institutional action?

30

Developing Visit Strategies and Lines of Inquiry

• What are areas needing clarification and/or more information?

• What are the major issues challenging the institution?

• What is raised by the themes that needs to be verified or explored?

• What are the strategies that will be most effective?

31

Worksheet for Team Conference Call

• Organizes team’s responses to institutional materials • Helps team make preliminary evaluation under the

Standards• Provides basis for team to work toward consensus• Should be submitted in advance of call

Tool: Team Worksheet (VG pg. 43)

32

Team Conference Call

• Evaluate quality of institutional report and alignment with Proposal and previous action letter(s)

• Identify areas of good practice, improvement, and further inquiry

• Identify issues, strategies, evidence needed• Identify persons and entities to be interviewed • Make or refine team assignments• Plan visit logistics

33

Off-Campus Sites and Distance Education Programs

Prior to Visit: Sites will be identified and assignments made• Review substantive change action letters to determine if issues

have been identified• Develop plan for the review of the programs and/or sites

During Visit• Interview faculty, administrators and students• Evaluate facilities OR online infrastructure• Observe classes• Document visit and findings in appendix • Discuss important findings with team for inclusion in report, as

appropriate

Tools: Protocols (RB pg. 158, RB pg. 160) Forms (RB pg. 55, RB pg. 58)

34

Compliance Audit

• Required for:– Institutions seeking Candidacy and Initial

Accreditation– Some institutions under sanction

• Additional report submitted by institution in advance of the visit—with links to documents

Tool: Compliance Audit Checklist (RB, pg. 51)

35

Determining Strategy for CPR Visit

• What evidence is provided to show capacity and readiness for EE?

• Why was it chosen?• What are the strengths and weaknesses of the

evidence?• What other evidence do you want to review to evaluate

capacity and preparation for EE?• Do any issues arise with regard to the Standards?• Meetings: format/methodologies

36

Determining Strategy for EER Visit

• What evidence is provided to show EE?• Why was it chosen?• What are the strengths and weaknesses of the

evidence?• What other evidence do you want to see to evaluate

effectiveness?• Do any issues arise with regard to the Standards?• Meetings: format/methodologies

37

Drafting in Advance of the Visit

• Assistant Chair draft outline of team report and context sections

• Team members draft outline or text for which they are responsible, using data from institution, with space for additional data, analysis and conclusions

Tool: Team Reports (VG pg. 53)

38

Conducting the Visit

39

Process of Visit

• Team meets at start of visit to confirm roles, assignments, logistics, and agenda

• Team meets frequently re: observations, emerging recommendations, and issues

• Team members draft sections of report and turn in to assistant chair on the last day

• Team agrees on report recommendations and confidential recommendation to Commission

40

Visit Schedule

• Executive sessions and debriefings with team only• Meetings and interviews with key individuals and

groups• Open meetings with students, faculty and staff• Document review • Time for drafting report sections• Final exit meeting

Tool: Sample Visit Schedule

41

Confidential Email Account

• Set up by WASC as extension of open meetings

• Checked by assistant chair during visit

• Important emails shared with team and investigated

• Comments included in team report only if the institution has a chance to address them

42

Approaches Used on Visits

• Document review• Interviews and meetings

– Mini-questionnaires– Techniques for small and large meetings– Fishbowl exercises

• Audits

Plan visit methodologies in advance

as part of schedule.

43

Document ReviewUse to:

Check complianceEvaluate the level of institutional engagementExamine the evolution of a policy or processIdentify direct and indirect evidence of student and

organizational learningConfirm report claims

DO as much as possible in advance

44

Interviews

Use to:Gather informationExplore issuesBuild relationships with members of the

institutionValidate impressions and observations

45

Tips for Good Interviews

Decide on a protocol for interview Prepare questions and lines of inquiry in advance Ask questions that elicit information, stimulate

discussion, or require judgment Avoid interrogation, leading questions, or loaded

language Avoid consultation, giving solutions, or talking about

your institution Let them do the talking

46

Alternative Forms of Interview

FishbowlBrainstorm/free discussion on a salient topicGo-roundBundlingAudit

47

Evaluating Program Review and Student Learning

(EER Visits)

Tool: EE ToolkitTool: Suggested Approaches to Evaluating Program

Review on EER Visits

48

Rubrics: Assessment of Student Learning

1. Academic Program Learning Outcomes

2. Use of Portfolios in Assessing Program Outcomes

3. Use of Capstones in Assessing Program Outcomes

4. Integration of Student Learning Assessment into Program Review

5. Assessing General Education

Tool: Program Learning Outcome Rubric

49

Expectations for Two Reviews

Use:• ‘Student Learning’ questions (p. 2)• as a monitor to be sure you are within the

proper scope of the visit

Tool: Expectations for Two Reviews

50

Educational Effectiveness Framework

• Use with team to evaluate institution’s “place” • Use language of rubric to describe the institution in

the report• Ask the institution to evaluate itself and discuss• Confer with team toward end of visit to mark a copy

of the EEF • Submit the marked EEF confidentially to WASC Tool: EE Framework

51

The Exit Meeting

• Team chair communicates commendations and key recommendations that will be included in report

• Chair may ask team members to participate

• The meeting is not a dialog, discussion or debate

52

Developing Team Recommendations

53

Two Kinds of Recommendations

• Confidential Team Recommendation to the Commission for action

• Team recommendations at the end of team report, delivered at the exit meeting

Tool: Commission Decisions on Institutions (VG pg. 84; SVG Append. F)

Tool: Commission and Team Decision Indicators (RB pg. 173)

54

Team Report Recommendations

Should be:– Overarching and important– Supported by evidence – Linked clearly to Standards and CFRs– Supported by text in the report

- Distinguish recommendations from suggestions and observations embedded in the report

Tool: Educational Effectiveness Framework (RB pg. 176)

55

Confidential Recommendation to Commission(CPR)

Proceed to EER or reschedule EER visitConduct a Special Visit (not preferred)Add time to EER visitIssue a notice of concern or impose

a sanction

Tool: Confidential Team Recommendation Form (VG pg. 70)

56

Confidential Recommendation to Commission (EER)

Grant Candidacy, Initial Accreditation or Reaffirmation of Accreditation for specified term

Sanction or Notice of ConcernInterim Report or Special Visit

Tool: Confidential Team Recommendation Form (VG pg.72)

Tool: Commission and Team Decision Indicators (RB pg. 173)

57

Confidential Recommendation to Commission (Special Visit)

Varies with status of institution

Next steps

Removal or continuation of sanction (note two-year limit on sanctions)

58

Producing Effective Team Reports

59

Report Preparation Logistics

• Follow report template

• Start writing before the visit

• Complete your sections on site and give to assistant chair for editing together

Tool: CPR Report template

60

What is an effective team report?

Reflects a thorough assessment of the institution’s capacity, preparation, and/or effectiveness

Is evidence basedCites the Standards and CFRsProvides the basis for a sound and supportable

Commission decision Identifies important areas for institution to

address

61

Using Evidence in Team Reports

Use qualitative and quantitative evidence Select evidence carefully and purposefullyConnect evidence to an assertion or question Analyze information; do not just set forth dataLet evidence suggest improvementsUse evidence that speaks to the institution’s

themes and the team's questions

62

Tips for Writing Team Reports Consider multiple audiences: institution, Commission, and

next team Know your areas of responsibility, including length and

depth of your section Start writing before you arrive on campus Address priorities and goals set by the institution Address Commission’s concerns (last action letter) Make commendations, but don’t overdo it Use praise that doesn’t send wrong or mixed signal

63

More Tips on Team Reports….

Be sure to check facts Support findings and recommendations with

evidence --and tie them to CFRsEnsure evidence is sound and validDistinguish recommendations from suggestions or

observationsUse formal language and tone (e.g., not “we/they”)Don’t mention personnel by nameDon’t prescribe solutions

64

After the Visit

65

What happens next?

↓ AC prepares draft for chair, team and staff review; changes as needed

↓ Chair sends to institution for corrections of fact

↓ Chair finalizes draft and submits to WASC↓ Chair sends Confidential Team

Recommendation and completed EEF to WASC

↓ WASC sends report to institution

66

Then…↓ Staff prepares draft action letter, which is reviewed

by team chair

↓ Commission Panel reads report and documentation including institution’s written response, meets with institutional representatives at Commission meeting

↓ Panel makes recommendation to Commission, and Commission acts

↓ Staff finalizes draft action letter on behalf of Commission

67

Also after the visit…Send reimbursement forms to WASC within 30 days

– Hotel arranged and paid directly by institution– Travel / food reimbursed – Rental car must be approved in advance by

WASC staff– Spouse or assistant costs not covered– See policy for more details

Team members should not have any contact with the institution – About the visit OR– Consult with the institution for one year

68

Common concerns about Visits

• Some team members not well prepared

• The team “did not understand us”

• The CPR team moved into EER “territory”

• Team did not review all the evidence

• Team changed the schedule at the last minute, or did not stay on schedule

69

Common concerns about Team Reports

• The recommendations were too specific or were unfair

• The report did not show that the team reviewed the evidence

• The recommendations were not based on good evidence or supported in report text

• The report did not address all the important issues or themes

70

Resources for Teams

• Appendices of Visit Guide

• Team Materials and Institutional Report mailed 10-12 weeks in advance of visit

• WASC Website: www.wascsenior.org

• WASC Email Advisory (sent prior to visit)

• WASC Staff

71

Thank you for your service to the region