tesol 2010: assessing academic bridge courses powerpoint presentation

Assessing Academic Assessing Academic Bridge Bridge CoursesCourses

Jean M. Czaja & Brooke HaleyMarquette University

TESOL 2010—Boston, MAMarch 25, 2010

162B Boston Convention Center 3-3:45pm

Presentation OutlinePresentation Outline

Our Context Our Mandate & Timeline Marquette University’s General Assessment Process Steps for Designing Our Assessment Plan Examples of How We are Carrying Out Assessment in 2 of

Our Courses◦ How they are embedded into the courses

◦ sample assessment measures and rubrics

Data & Conclusions from Our First Assessment Cycle Actions We are Taking to Improve Students’ Learning Advice for Assessing Academic Bridge Courses

Our ContextOur Context

• Marquette University is a private Jesuit university located in Milwaukee, WI.

• We have a student body of approximately 8,000 undergraduate students and 3,500 post-graduate students.

• Of this total, about 500 (4.35%) are international. Approximately 150 (1.88%) are undergraduates, coming from 43 total countries.

• To support these undergraduates, we offer 4 advanced ESL academic bridge courses and an ESL section of Freshman English, which students take for academic credit:

• ESLP 82: Composition• ESLP 83: Reading• ESLP 84: Listening Comprehension• ESLP 182: Rhetoric and Composition for Non-Native Speakers• ENGL 1 (ESLP section): Rhetoric and Composition

• In AY 2009-2010, about 75% of the new international undergraduate students (54) took our English Placement Test and enrolled in our ESL academic bridge courses.

Our Mandate & TimelineOur Mandate & Timeline

• Marquette is accredited through the North Central Association of Colleges and Schools.

• As part of its accreditation, since Fall 2004 Marquette has been engaged in an initiative to improve:

(1) its overall assessment of students’ learning (2) the “culture of assessment” at Marquette

• The ESL Program first entered the assessment process in Fall 2006 when it created and listed specific course learning objectives on course syllabuses.

• Then in Spring 2008, the ESL Program wrote an assessment proposal & devised initial learning outcomes, performance indicators, and measures.

• In AY 2008-2009, the ESL Program carried out its first year of assessment, which we will be reporting on.

Three Levels of Marquette’s Assessment Three Levels of Marquette’s Assessment ProcessProcess

ESL ProgramInformation adapted from Marquette assessment literature

Marquette’s Assessment ProcessMarquette’s Assessment Process

At Marquette, student learning consists of 4 steps:

Feedback for continuous

improvement

Information adapted from the Marquette assessment website

Steps for Designing Our Assessment Steps for Designing Our Assessment PlanPlan

We devised our assessment proposal and plan for the university by:

1. familiarizing ourselves with the assessment terms & process used at Marquette

2. meeting with Foreign Language and Study Abroad representatives and examining their learning outcomes, indicators, and measures

3. speaking with ESL professionals, reviewing ESL assessment literature, and attending sessions at TESOL Assessment information is available for K-12, adult education,

and IEPs No guidelines for learning objectives & assessments of

university-level ESL courses

4. devising ESL Program learning outcomes, performance indicators, and measures for the university assessment committee

5. improving existing and creating new assessment specifications and course assessments, which are mapped to performance indicators

6. improving and creating rubrics for the course assessments

Our Assessment PlanOur Assessment Plan

MARQUETTE UNIVERSITY

UNDERGRADUATE CO-CURRICULAR

LEARNING OUTCOMES

English as a Second Language Program (ESLP)

At the completion of his/her ESL course(s), the student is able to:

1. Write using academic English

2. Read academic English texts

3. Listen to and speak using academic English

4. Use grammar, punctuation, and mechanics in academic writing

Program Learning Outcomes

Performance Indicators Measures Use of the Information


Produce a research paper Employ components of the writing process Use strategies to address writing problems

Final research paper in ESLP 82 Pre and post diagnostic tests Pre and post self-assessment of

skills and use of learning strategies

ESLP faculty will collect, analyze, aggregate data, and present a report to the Director and staff of the Office of International Education. The information will be discussed and commented upon by the staff and faculty for the purpose of advancement of students’ proficiency and program evaluation.

2. Read academic English texts

Comprehend literal and implied meanings of academic texts

Determine meaning of new vocabulary Evaluate and respond to ideas in academic

texts Employ components of the reading process Read under time constraints Use strategies to address reading problems

Comprehensive final exam in ESLP 83, which directly measures learning outcomes

Pre and post diagnostic tests Pre and post self-assessment of


Same as above

3. Listen to and speak using academic English

Employ note-taking techniques to take notes on and understand academic spoken texts

Participate in everyday interactions role-plays

Participate in small group discussions Deliver oral presentation on news broadcast Use strategies to address listening and

speaking problems

Comprehensive final exam in ESLP 84 which directly measures learning outcomes

Final (oral) presentation from last unit



Same as above


Understand conventions of standard U.S. English grammar, punctuation, and mechanics

Produce a research paper using conventions of standard U.S. English grammar, punctuation, and mechanics

Use strategies to address grammar, punctuation, and mechanics problems

Revision of ENGL 1 Paper #1 at the end of ESLP 182



Same as above

How We are Carrying Out Assessment in How We are Carrying Out Assessment in

ESLP 82: WritingESLP 82: Writing

Program Learning Outcome 1: Writing using academic English

Performance Indicators Measures

Produce a research paper

Employ components of the writing process

Use strategies to address writing problems

Final research paper in ESLP 82

Pre and post diagnostic tests

Pre and post self-assessment of skills and use of learning strategies

How We are Carrying Out Assessment in How We are Carrying Out Assessment in

ESLP 182: Rhet/Com for Non-Native ESLP 182: Rhet/Com for Non-Native SpeakersSpeakers

Program Learning Outcome 4: Use grammar, punctuation, and mechanics in academic writing

Performance Indicators Measures




Revision of ENGL 1 Paper #1 at the end of ESLP 182

Pre and post diagnostic tests

Pre and post self-assessment of skills and use of learning strategies

Data from First Assessment Cycle Data from First Assessment Cycle

Program Learning Outcome 1: Write using academic English

Measures Used to Assess the Outcome: Final Research Paper in ESLP 82

Total Number of Students Assessed: 12


Performance Indicators

Developing(CD, D, F)

Competent(B, BC, C)

Proficient(A, AB)


Produce a research paper

0 2 10

Employ components of the writing process

2 7 3

Use strategies to address writing problems

0 7 5

Data from First Assessment Cycle Data from First Assessment Cycle (cntd.)(cntd.)

Program Learning Outcome 4: Use grammar, punctuation, and mechanics in academic writing

Measures Used to Assess the Outcome: Revision of ENGL 1 Paper #1 at the end of ESLP 182 and 100-point diagnostic test given at the end of the semester

Total Number of Students Assessed: 23 and 22 respectively


Performance Indicators

Developing(CD, D, F)

Competent(B, BC, C)

Proficient(A, AB)



5 13 5


1 6 15


No data collected

No data collected

No data collected

Conclusions from First Assessment CycleConclusions from First Assessment Cycle

When we discussed the academic year 2008-2009 assessment results, we reached the following conclusions regarding student learning:

• The majority of the assessed students were ranked as competent or proficient in most categories.

• Some students did not meet the expectations, which makes us wonder why they did not make adequate progress and what actions we could have taken to help them to become competent or proficient.

• We feel that in general that most results represent an accurate assessment of students' learning. This was reinforced by students' performance in class and on other major assessments.

Actions We are Taking to Improve Actions We are Taking to Improve Students’ LearningStudents’ Learning

Action #1: We are strengthening course content in areas where we want more students to move from developing to competent on the performance indicators.

Action #2: We are determining what measures are really needed for the university assessment. We are fairly certain that in the future that diagnostic tests and questionnaires will NOT be reported to the university but rather used for:

giving students feedback on their skills encouraging students to reflect on their learning & how they can learn better improving our course planning—helping us to decide areas to emphasize

Action #3: We are reexamining and revising our assessment specifications, rubrics, and assessments to ensure that they measure students' actual proficiency.

#Action 4: We have made some measures carry more value towards students’ final grades so that students are motivated to do well on them and take them seriously.

Action #5: We have worked with the International Student and Scholar staff to adjust admissions guidelines so we do not admit students with too low of English proficiency to adequately progress in our courses and to succeed at Marquette.

Action #6: We are meeting with students who show problem behaviors and/or who receive low grades on their first assessments to give them advice and refer them to additional university resources.

Advice for Assessing Academic Advice for Assessing Academic Bridge CoursesBridge Courses

1. Give yourself enough time to plan and implement your assessment initiative.

2. Understand how assessment is carried out within your university and be sure that your plan will fit with the university’s plan (if applicable).

3. Create standardized course learning objectives and make sure that your staff actually emphasizes these areas when they teach these courses.

4. Devise assessment specifications and assessments that directly measure the course learning outcomes.

5. Create rubrics for all assessments to provide a standard way of assessing students’ performance.

6. Devise a standard procedure for collecting the evidence from students.

7. Set aside a specific time for your staff to analyze and honestly reflect on the evidence. Keep in mind the limitations of standardized assessments—they don’t tell the entire story.

Advice for Assessing Academic Advice for Assessing Academic Bridge Courses Bridge Courses (cntd.)(cntd.)

8. As a staff, decide on specific actions you will take to help improve students’ learning.

9. Embed the learning objectives, assessments, and rubrics into your

courses so teachers and students know clearly what they are learning and expected to be able to do.

10. Try to make the assessment initiative as simple, direct, and manageable as possible, and only gather as much evidence as needed.

Thank You & Electronic CopiesThank You & Electronic Copies

Thank you for coming to our presentation.

Please contact us for more information about assessment at Marquette. Visit http://www.marquette.edu/oie/eslp.shtml for a link to electronic copies of the handouts or other resources. Click on the “TESOL Presentations” link.