Towards a culture of data collection and analysis in mathematics support

centres

Professor Tony CroftDirector, Mathematics Education Centre,

Loughborough Universitywww.sigma-cetl.ac.uk

background

background

“higher education has little option but to accommodate to the students emerging from the current GCE [ie pre-university schooling] process”

Smith, 2004, Section 4.39,

Overview

Growth in activity

Difficulties of getting good and convincing data

What has been written about this ?

Measuring and modelling success

Caveats Shouldn’t be too fixated on trying to find hard measures of

success.

We are professionals – we recognise the difficulties our students have and respond to the challenges in different ways.

Mathematics support centres are but one possible way!

HE institutions have all sorts of support services – counselling, careers, personal tutors, English Language, and we don’t necessarily require these to justify their existence.

Why do we have these ?

Why do we have these ?

Because we care!

Because we want to encourage and support our students

Provision of excellent support is expected in a well-rounded institution, at a time of inclusion and widening participation

So in what follows, whilst I focus on collecting data and measuring/modelling success, I would not want to give the impression that I think these things are the most important aspect of what we do!

Continuing growth in UK activitySurveys:

Demand:

When sigma announced a competition for funding to develop new or existing centres, 14 high quality bids were received.

Sigma has now helped to establish new centres at

Beveridge 1993 42

Lawson et al 2001 46

Perkin & Croft 2004 66

Bath http://www.bath.ac.uk/study/mash/

Sheffield http://www.shef.ac.uk/mash/

Around the world

Australia:

there are 39 universities, of which 32 have some form of Mathematics & Statistics Learning Support.

(Helen MacGillivray, QUT, 2008)

Ireland:

13 institutions with mathematics support provision are reported in the audit of Gill Donoghue & Johnson

(CEMTL 2008)

Around the world

Switzerland’s first….. Opened September 2008

Evaluation, accountability

So with huge and rapid growth of mathematics support activity it is right and proper that we ask…

Are our efforts worthwhile, and how would we know ?Can we justify this activity ?

What data is it possible to collect ? Is trying to collect data worth the effort ? How can we interpret the data ? Modelling

Difficulties of getting data: systems and staff

This activity is still very young.

Drop-in activities were invariably started by concerned teachers who found a few hours in the week to offer additional support to students.

Staff if they were paid, were paid to tutor not to gather data, and certainly not to analyse it.

Very little long term strategy – cinderella service wherein staff rarely knew if they would continue to be employed from one year to the next.

Difficulties of getting data:departments and students Departments often reluctant to give out data.

Cooperation of departments is not always forthcoming.

Difficulties of accessing the data.

Moving goal posts from year to year.

There are many factors which influence students’ success in a mathematics module:

curriculum change; recruitment and selection criteria change; staff change; students change!; even things like time of day change!

There may be lots of students seeking help, but they come from many different courses, with very different needs, so it can be difficult to compare

Difficulties of getting data

Sent: 30 September 2008 12:43To: mlsc@lboro.ac.ukSubject: Maths for Physics

Hi,I’ve just started a Physics and Management course and am quite worried about the maths side of it as I am one of the very few if not the only undergraduate that doesn’t have a maths A level. I am willing to work very hard to get the skills needed but with maths tests already coming up for my course I want to get started as soon as possible. What it the best way to go about achieving my goal ? And is there anyone I can speak to about exactly what is needed for the course ? ( I haven’t yet spoken to my maths lecturer but will be seeing him tomorrow). Any help or advice about what’s the best course of action to take would be appreciated as it would be a weight off my mind.

Resource Issues / financial constraints Almost all centres were established on very modest budgets and their

purpose was to be up-front helping students.

“more than half of centres [in the Australian survey] do not have a full-time person employed”.

Many staff are sessional only (term only).

Short term funding.

Often staff recruited to maths support work do not have any experience of data collection and analysis, and are attracted to the work so that they can help students with their maths.

What has been written on Evaluation ? [Mathematics Learning Support] needs sufficient security to attract,

train and retain staff, and to play its part in the ongoing and longitudinal data collection and analysis that should be an integral part of its contribution to the university. All universities should ensure that such data collection and analysis are undertaken and performed correctly to provide vital information for university academic management. However, as reported, few of the facilities currently have the resources to undertake this important work.

Helen MacGillivray (QUT, Australia 2008) In the audit of Gill et al…

measures of success are listed, but [Carlow] there is no definitive way of measuring the effectiveness…messiness of the data.. [DCU].…LSU’s should have a research and development function that is adequately recognised for funding (ITT Dublin)

Measuring and modelling success

I will describe both soft and hard measures.

Measures of success – soft measures

Growth in the number of centres

Usage Data – attendance records, return visits, year of study, department, problem

Student Feedback Questionnaires – biased and usually positive

Module Feedback Questionnaires – reach a broader group of students, but often little is said about maths support.

External Review

Usage Data

Student visits

Usage Data

Students Visits Departments

External Review The Institutional Audit report of 2004 records:

“The audit team identified the following areas as being good practice in the context of the University: the work of the Mathematics Education Centre...”

“Having discussed the work of the MEC with members of staff across the University, the team came to the view that [its] contributions to the University's resources for staff development, and their work more generally, constituted a feature of good practice.”

http://www.qaa.ac.uk/reviews/reports/institutional/loughborough2004/Rg063loughboroughUni.pdf

The Institutional Audit report of 2008 records:

sigma CETL has its origins in the Mathematics Education Centre but has widened its concernfrom the teaching of mathematics to engineers to include support for mathematics educationacross the University. The ready accessibility of useful help was praised by both undergraduateand postgraduate students that met the audit team. Other students described the benefits of thesupport rooms and associated equipment. Postgraduate students were appreciative of the one-to-one help and individual study programmes provided for them by the Centre.

http://www.qaa.ac.uk/reviews/reports/institutional/Loughborough08/RG378Loughborough.pdf

NSS National Student Survey 2008

Comments relating to “maths support”

Department: Mathematical Sciences:

2 positive comments referring to MLSC (unspecified)

Department: Economics

Two positive comments citing “excellent maths support” (one) and “superb support in the form of the maths support centre” (one)

Department: Electronic and Electrical Eng:

One positive comment: “the uni is more than equipped to help and bring me up to speed with my maths skills that I may have previously missed or forgotten”.

The student voice video

Modelling success – hard measures

Studies with more substance are beginning to emerge which might inform methodology of data collection and analysis:

Dowling & Nolan Patel Croft & Pell Lee, Harrison, Pell & Robinson Parsons MacGillivray

Modelling success – hard measures

What sort of things might we count ?

% of support centre users who pass, against % of non-users who pass.

Mean exam score of users against mean exam score of non-users.

Pass rates of at-risk students who use the centre against those at risk that don’t.

Module grades of users. Number of users who just scrape a pass. Examine failing students to see if they attended the

centre. Retention rates longitudinally

Robinson

Bamforth et al (2007)

Bamforth et al (2007)

Bamforth et al (2007)

Dowling & Nolan (DCU)

Published in CETL MSOR 2006 Conference Proceedings Effectiveness of the Maths Learning Centre measured by usage

statistics, feedback including interviews.

Model:

pass rates of “at risk” students who did / did not visit the centre were compared over two years.

2004/5 2005/6

Number at risk 80 161

Number at risk who visited MLSC

41 95

Pass rate at risk who visited

53% 60%

Pass rate at risk who did not visit

25% 49%

Pell & Croft

Anecdotally:

Some of the students most in need of support fail to access it at all.

Many of those who make good use of the centres would pass anyway.

Analysed five first year engineering maths modules in 2004-5.644 students.

Data obtained on their module grade, number of visits to the centre.No access to entry grades.

Pell & Croft

21 students attended >=2 and achieved grade D 63 out of 74 failing students did not attend more than once

Pell & Croft

35% of those achieving A* sought maths help more than once 15% of E Grade Students and 15% of F Grade students sought help

more than once fail grade students, in addition to having ability problems have

attitudinal problems

National Audit Office 2007….

Staying the course: the retention of students in higher education.

“A university’s approach to retention should be a positive one and it should provide students with opportunities to improve their grades rather than just addressing gaps in their knowledge”

The data from Pell would suggest the support centres are already doing this rather well!

There are some things you just can’t measure….

I just wanted to share my good news that after five years of study at Loughborough I managed to gain a first class honours in Product Design and Manufacture. I believe that without the hours you dedicated to the maths learning support centre I would not have been able to pass the maths modules on my foundation year and first year. Your support, patience and encouragement were invaluable when it came to a subject that I had little confidence in when I first arrived at Loughborough.

Kathryn