universities in the ‘performance age’

1

Universities in the ‘Performance Age’

Session two:

Performance Management, Assessment and Rankings

Maison française d’Oxford, Oxford, February 17-18th

2

Assessments and Rankings in French Higher Education: a Case Study

Lise Gastaldi & Caroline Lanciano-Morandat

The Institute of Labour Economics and Industrial Sociology (LEST)

CNRS and Aix-Marseille University

3

Introduction

A presentation of the first works we realized within a larger research programme funded by the French ANR (National Agency for Research)

The Prestence project: From Prestige to Excellence, the fabric of academic quality

We investigate the consequences of a change in academic quality judgment devices with the increase of formal devices in addition to informal ones

With a focus on the devices dealing with higher education establishments

4

Introduction

Informal devices which are older Produce global judgments about quality in terms of

reputation or prestige Judgments are formulated

With no formalized list of criteria By various actors included non expert ones

Formal devices which are increasing currently Evaluations are based on formalized criteria With the aim to formulate more objective judgments by a

more systematic evaluation of quality

5

Introduction

Informal and formal evaluation devices can produce different judgments on one establishment

The Prestence research programme aims to investigate how an establishment is succeeding in producing academic quality, given different regimes of quality evaluation exist Putting the light on the organisational work of gathering and

combining various resources And investigating how the change of evaluation regime can

influence the evaluation of the establishments and the way they react in terms of strategy and organisational choices

6

Introduction

A comparative research between several establishments in various countries* and scientific fields*

I and Caroline focus on the chemistry and physics field We realized the first case study on an old and prestigious

French institution Characterising a singular organisational model and a focalisation

strategy with the production of specific outputs

Through this French case study, we address the question of the sustainability of singularity in higher education when establishments face more developed formal evaluation devices

7

Content

1. The French context of evaluation in higher education

2. The case study: a singular establishment

3. Aims and issues

4. Singularity facing formal evaluation devices

5. From evaluation to change on strategy and organisational path?

6. Conclusion and research perspectives

8

1. The French context of evaluation in higher education

We observe a strengthening of assessment We focus on establishments and recurrent

evaluations and we adopt a classification of these evaluation devices into two main categories Evaluations leaded by national public authorities

With a focus on public establishments Evaluations leaded by other actors (newspapers,

higher education institutions…)ARWU, THE, QS ranking, SIR…

9

The French context of evaluation in higher education

A classification that refers also to two main differences The imperative nature of the evaluation devices (or not)

Public establishments cannot avoid assessments leaded by national public authorities

If we consider rankings Some needs the establishments collaboration (it is the case of national

ones and of the THE ranking for example) The establishments can refuse to respond to the ranking producers requests,

even it can be a risky strategy given the growing audience of rankings Some doesn’t need the establishments collaboration and consequently the

establishments are evaluated even if they are not agreed

The form and the ‘philosophy’ of the evaluation process and its result

10

Differences between the two categories we distinguish

National public authorities assessments

Rankings

Evaluators identity

Peers and experts of higher education and concerned scientific fields

Experts in data mining

Judgment process

A substantive assessment with quite in-depth analysis of activities, organisation, governance, means, outputs…

Based on quantitative data and qualitative analysis

‘substantive subjectivity’

A mechanical judgment process through the automatic compilation of data dealing with inputs and outputs (considering the evaluated entity as a black box)

‘mechanical objectivity’ (Porter, 95)

Judgment result

A substantive evaluation report with quite developed analysis and advices

A very synthetic result: a score and a rank

11

The public assessment devices

Before 2007 There were various devices leaded by different

public entitiesSome was in charge of assessment of the trainingsSome was in charge of assessment of the research

activities and laboratories There was no unified evaluation system

12

The public assessment devices reforms

In 2007, the AERES was created and took the place of several former evaluation devices

Agency for research and higher education assessment

The AERES has the mission to evaluate all the research and higher education public institutions that are under a French Ministry authority

With a three levels evaluation: research laboratories, training programmes and establishments

An evaluation each 4/5 years by ad hoc experts committees compound by peers, higher education experts, industrial researchers…

13

The AERES assessments

First the entity evaluated make a report that presents itself, its activities, productions… with a self-evaluation

Then the experts committee visits the entity with meetings with the governance team, the academic and administrative staff and the students

They write an evaluation report, and the entity can respond These documents are published on the AERES website The committees mark laboratories and trainings on a A+, A, B, C

scale These evaluations can be a basis for the Ministry Higher

Education and Research decisions as funding, trainings certification…

With nowadays a direct link with the SYMPA model that determines the research budget allocated to an university in relation with the marks obtained by its laboratories

14

National and international rankings

National rankings in France Mainly dedicated to engineering and business schools For engineering schools (that are part of universities or

not) Rankings are realized and published by newspapers There are few old rankings and several new ones A larger audience

With an Internet diffusion With an increasing attention paid to rankings by students and their

families

International rankings (Shanghai, THE…) We also can observe a multiplication of rankings that

enjoy a growing audience

15

A functionalist lens with a reference to socio-economics works (Karpik, 96; Cochoy,99…)

Various actors (students and their family, public authorities, partners, academic staff, employers…) who invest resources in higher education want to have some references about the quality and performance of research centres, trainings and establishments

Higher education can be considered as experience goods It is difficult to have a precise idea of an entity quality when you are not inside It is a more and more complex and larger system if we consider the increasing number

of institutions and trainings in each country and if we consider the global ‘market’ of higher education

The stakeholders need quality references to do their choices among all these entities

And considering the new importance of accountability logics=> All these facts lead to strengthen evaluation

16

2. The case study: a singular establishment

A French engineering school located in Paris Created in 1882 in a specific context

East French regions, where chemistry schools were concentrated, passed into the hands of Germany

French chemical industrials wanted to create a new school to train engineers and researchers for national chemical industry

This project was presented to Paris city council and a new school was created under its authority

Nowadays An engineering school that is a ‘grande école’ with several research labs

on a same campus A pluridisciplinarity both in teaching (chemistry, physics, mathematics

and biology) and in research (chemistry, physics and biology)

17

Methodology: an in-depth case study

A collaborative research We collect various data between February 2010 and

November 2011 40 interviews Public and internal documents

That we analyzed in a qualitative way and with an historical perspective

We organised several restitutions to some actors in charge of the establishment governance to our Prestence project colleagues

18

A singular establishment

Institutional status: one of the two public higher education institutions that are under a local authority and not under a Ministry authority

A small institution 250 students 400 academic and administrative staff It is a small institution in France, and the gap is greater in comparison

with foreign universities A strong autonomy of the heads of laboratories behaving as

CEO of small firms Autonomy to conduct their research activities and manage their teams Autonomy to lead valorisation activities with for example the capacity to

take patents in their own name

19

A singular establishment

An organisational model characterised by strong interactions between research, teaching and industry

Research and teaching: lot of courses take place in the labs; the students spend 4 months in the school labs during the engineers training; the last year is dedicated to do a research Master degree; most of 60% of engineers do a PhD and the most of them will begin their career in industrial or academic research centres

Research and industry: researchers lead fundamental research but with a long term application preoccupation ; they have lots of research contracts and we can observe all the form of technology transfer: scientific consulting, patents, spin-offs…

Teaching and industry: with a six month stage in an industrial research department, with some courses linked to a specific firm needs and with various relations with an active network of industrial partners

An establishment that produces specific research and students prepared to work in the field of research and innovation

It is a singular establishment in the French context comparing to universities and also others engineering schools

20

3. Aims and issues

This establishment presents a singular organisational model and produces specific outputs

It is a well-evaluated establishment for a long time By informal judgment devices

Strong reputation and prestige in the academic world Strong reputation in the industrial world Some of its researchers are well-known by the public

By old formal judgment devices Publications French research institutions as CNRS or INSERM

However how is it evaluated by current formal evaluation devices?

21

Aims and issues

A main issue: Is singularity in the current higher education system sustainable?

Declined in two research questions with management and public management lens How does singularity face the current formal evaluation

devices? Or in other words: how do the current formal evaluation devices deal

with singularity? Could these evaluations have some consequences on the

singular establishments strategy and path?

We address these questions through one case study

22

4. Singularity facing formal evaluation devices

The establishment experiences some difficulties with current evaluation devices With the AERES and the CTI

The CTI: the ‘commission for engineer diploma’ – a specific French instance that evaluates engineering schools on the training side and does recommendations to the Ministry that gives the right for an establishment to deliver engineer diploma

With national rankings and especially the international ones

23

Confronting singularity to national public assessments

Most of the research centres are well-evaluated but we studied the case of a strong controversy about one laboratory evaluation

It is an historical laboratory with a strong reputation and positive CNRS evaluations

But the AERES assessed it as a ‘B’ laboratory All the laboratory members, the establishment directory and the head of

the CNRS chemistry department protested A case which illustrates some criticisms about the AERES

evaluations Difficulty to assess pluridisciplinary works Subjectivity of the committees members The short time allocated to visit each laboratory Some tensions around the evaluation of application-oriented research

24


Tensions with the CTI The CTI leads evaluations by comparing training

characteristics with standardsThe conformity with these standards is a condition to

obtain accreditation There are conflicts on

The training length: it is a 4 years training in this establishment compared to the 3 years standard

The training content: for the CTI students must follow courses in management and economics that are really low developed in this establishment

25


Due to its status there is no global evaluation of the establishment and no valorisation of its specific model with strong interactions between research, teaching and industry

26

Singularity facing rankings

First we should think that rankings can be more in favour of this kind of establishment

With the mechanical objectivity of the evaluation process that only counts the publications, there is no more tension with experts who can be subjective or not able to evaluate research in pluridisciplinary fields

Some rankings (some national or the THE ones) assert they evaluate all the dimensions and activities of an establishment

Nevertheless… The establishment is ranked between the 10th and 20th positions in

national rankings, consequently not among the most excellent ones It is only ranked in the ARWU, between the 200th and 300th positions;

it is not ranked in all the others rankings

27


When we compare the establishment profile and the rankings criteria

Strengths An old institution An historic and strong investment in research A high ratio staff / students A high quality of the students professional insertion

Weaknesses The small size (with a lot of criteria in absolute terms, with the importance

of reputation inquiry…) The low internationalisation with few foreign students and academic staff

Dimensions that are not really included in rankings The industrial relations and technology transfers The interactions between research and teaching

28


If rankings are objective, they incorporate normative definitions of higher education and excellence with usually A conviction of the superiority of a big size

establishment A focus on research outputs (evaluated with only

two criteria: number and impact of publications) A strictly academic definition of excellence The importance of internationalisation both of

students and scientific staff

29

Singularity facing various evaluation systems

Consequently we can ask if we would assist to a process of singularity reduction that will lead the establishments to more standard organisational models producing more standards outputs

30

5. From evaluation to change on strategy and organisational path?

Such a process of ‘desingularisation’ could only occur if formal evaluations lead some changes in establishments strategy or organisational model, and this depends on several elements:

The way the establishment obtains crucial resources and the existence or not of a link between its evaluations and its capacity to gather these resources

The way the establishment’s members sense the evaluation in general and the different devices and also their consequences on their access to resources

The strategy of the establishment in relation to the various evaluation systems, given it can adopt

An active or passive resistance strategy Or a strategy of adaptation; and in the case of we have to consider the capacity

of the establishment for engaging reforms in order to improve next evaluation

31

Karl Weick (1977, 1995…)

Weick and the theory of ‘sensemaking’ The process of sensemaking which is very important to understand how

individual and collective actors in organisation act and react to environmental change is composed of 4 stages

An environmental change An enactment process that leads the actors to take aware of the environment,

but the environment is complex and various The selection is the process by which actors try to reduce the environment

‘equivocity’ constructing interpretations and leading operations to give sense to the environmental change

The retention/memorisation: the actors retain some interpretations and act in consequence; and the interpretation of the environment which are memorized will influence the next processes of sensemaking when other environmental change will occur

Consequently it is important to adopt an historical perspective to analyse the actors reaction to a change in their environment

32

For a long time, a low sensibility to evaluation and especially to formal evaluations

On the research side At the beginning: laboratories are not affiliated to external entities Progressively some had been affiliated to universities or research entities

as CNRS or INSERM With positive evaluations by CNRS and INSERM

Nowadays it remains few laboratories that are not mixed research units and consequently that escape to formal evaluations

With the conviction that autonomy is a strength because it allows to lead application-oriented research and to exercise valorisation activities

On the teaching side The establishment was in a good position to bargain with the CTI in

order to maintain its singular characteristics

33

For a long time, a low sensibility to formal evaluations

The establishment enjoys positive informal evaluations by The academic world, due to its research works and

publications The industrials which pay for collaborative research

(financing projects and fellowships) and recruit engineers and doctors coming from the establishment

The students and their family, due to the establishment reputation and the very good conditions of professional insertion for the engineers coming out of it

The general public, due to its star scientists, 5 Nobel Prizes and its story (since Pierre and Marie Curie)

34


Thanks to These positive evaluations The autonomy some laboratories enjoy because they are not

affiliated to external institutions that allows them to lead valorisation activities (patents, spin-offs) without difficulties Valorisation activities as sources of incomes

The historical support of the main public authority that allocates every year comfortable means to the establishment

The establishment did not face problems to access resources

35


They have enough money and job positions to conduct their activities in good conditions With funding of public authorities With industrial research contracts With the revenues of valorisation

They do not have problems to recruit scientific staff They recruit high level students The students do not experience difficulties for their

professional insertion in higher education or in research and development departments in the network of industrial partners

36

Nowadays the context is changing

It is the end of the former victorious resistance against CTI orders: the establishment had to conform its engineering training to the 3 years standard length

Some evaluations by the AERES are reserved, with effects on internal hierarchy between laboratories and researchers

The fact the establishment does not appear in the international rankings except Shanghai one

The main public authority (the Paris city council) now heeds to these rankings and is concerned by the low international visibility of the establishment

With a stronger issue in terms of the return on the investment realized in financing this establishment

37

A progressive evolution of representations

A perception influenced by the establishment path Some people who continue to refer only to informal

judgmentsA reference to few academic colleagues in prestigious

institutions Few persons mention rankings

38

The reaction to academic evaluations

Even the conditions of evaluation had changed with the AERES creation, this kind of assessment is usual and well-accepted

Peers assessment Evaluation of the research programmes and the publications Evaluation leaded by national public authorities (even in this case the

Ministry is nor the main authority neither the main resources supplier)

Moreover their legitimacy, these evaluations condition a larger part of the resources (money and jobs position)

Directly in the case of laboratories associated to universities In a more indirect way with the influence of the AERES marks on the

capacity to obtain some financings on projects

39

The reaction to academic evaluations

In fact Some of the criteria of formal public authorities assessments

had been progressively integrated More recent evolutions can be analysed as a response to the

AERES evaluation process and norms Some laboratories mergers to constitute more important units The research activities and laboratories are now presented in a more

intelligible way for external evaluators (or widely for stakeholders) Through of a kind of map with different areas for each discipline that puts

laboratories in relation to them An internal reorganization of some labs to present a structure easier to

understand in a fast way for external evaluators

40

Public evaluations and future of singularity

The evolutions are too recent to conclude but we can ask if some of them could conduce to a reduction of singularity The aim to normalize the situation in integrating all

the laboratories in research units affiliated to external academic partners, with an automatic loss of autonomy for them

The reorganisation of labs and teams are not neutral on the scientific activities

41

Public evaluations and future of singularity

Moreover public evaluation systems could lead to deeper transformations if we consider that pluridisciplinary teams and application-oriented research are not well evaluated by all the committee experts

Then the form of the assessment result can have a performative effect given the report formulates analysis and gives advices on the governance, the internal organization and the scientific field: orientations, programmes, collaborations, publication strategy…

42

The reactions to rankings

Few persons inside the establishment are preoccupied by rankings, except the director and the man in charge of communication

A low attention is historically paid to national rankings due to several elements1- They are seen as non objective and non reliable because of

their way of collecting data

2- They are considered as non legitimate because they are not realized by peers or experts and not controlled by academic community or public authorities

43


3- National rankings have quite low impact on students recruitment In the French higher education the engineering schools

recruit by a special way: after a 2 or 3 years special training the students pass very selective exams

There are various exams to integrate the different engineering schools, and the establishment we study recruits since few years on the same exam that the most prestigious French establishment that is Polytechnic school

Whatever is their rank in rankings they succeed in recruiting excellent students

44


When the international rankings appeared, most of the people in this establishment not paid attention to them We may see here an effect of establishment path

Its members always have neglected national rankingsThey have not anticipated the new audience of these

international ones thinking the establishment prestige is sufficient to assure its durability

Indeed until now the establishment resources are not linked to its ranks in these rankings

45


However the directory board in which members of Paris city council are part of took decisions that can be seen as reactions face to this growing audience of rankings An increase of the students numbers and the aim to recruit

foreign ones A strategy to association with other higher education

institutions It is membership of ‘ParisTech’ that is a large association mainly of

engineering schools located in Paris and its suburbs It is also membership of ‘Paris Sciences et Lettres’ which is an

association of institutions in sciences and humanities fields located in Paris

46


We can see this strategy of association in the French context characterized by several mergers of universities with an evident preoccupation of increasing their size When the size is a key factor in rankings Given such mergers respond to Ministry pressure

So the establishment we study adopts an alternative strategy compared to merger in order to preserve its autonomy

We do not know now if this strategy is durable or if in the future this establishment will have to merge with university

47

The rankings and the future of singularity

For the moment rankings have low effects But

The aim to increase the number of students will ask the question of the capacity to maintain all the activities on the same campus while it is a key factor in the high proximity between students and research

If the establishment will merge with other institution and especially with a big university, the question of the capacity to maintain its specific model with the heads of research centres high autonomy that is a factor of the establishment attractiveness for academic staff will be asked

48

The creation of a own assessment committee

The directory board decides to create an international scientific committee in order to evaluate the establishment in all its activities This committee is composed of 10 international academic

and industrial star scientists Each year, they lead an overall evaluation on research,

teaching and valorisation activities They formulate qualitative assessment and numerous

advices. For example they can highlight research topics to investigate, strategic scientific field to develop…

49

The creation of a own assessment committee

We can interpret this own international committee creation in two ways As a way to say to national public authorities that the

establishment acknowledges the importance of evaluation even if it is not evaluated by the AERES

As a buffer To help the establishment to better understand the new international

evaluation norms And to help it to imagine some transformations that could improve its

evaluations but in the respect with its singularity

50

Why this need for such a buffer?

Rankings only evaluate outputs and in some cases inputs (as budgets or staff), often with few indicators

These criteria are not obvious to turn into concrete actions We have to more publish in higher level journals, ok… It would be better if we have more money, ok… We should have Nobel Prizes, ok…

Contrary to the AERES evaluations that give concrete practical and more easy-to-implement actions

The AERES evaluations are less normative but easier to understand and turn into action, whereas rankings are more normative and directive as they don’t give obvious ways of improvement

51

6. Conclusion and research perspectives

Do formal evaluations lead to a standardisation process? It seems to be the case. When evaluation is based on determined

criteria and implicit or explicit definition of academic quality (which tends more to a excellence and thus a restrictive definition of quality) it promotes norms and by the fact marginalises entities out of the line

But the singularity seems to be more problematical in the rankings evaluation regime

To some extent national public authorities assessments can take into account national, disciplinary or specific contexts and the establishments particularities and story

With the rankings, an establishment that does not fit the criteria is in a bad position, whatever the reasons that explain its specific characteristics

52

Conclusion

The sustainability of singularity mainly depends on the central question of resources access

How the evaluation condition resources access in a direct or more indirect way? Will an establishment be able to maintain its resources access even if it is not

well-evaluated? It depends on national systems, establishments status and stakeholders’

strategies (on the question of the importance they give to various evaluations and especially to rankings)

A key question seems to be the permeability between the two regimes of evaluation when the experts involved in public assessment systems are influenced by the rankings that modify the established hierarchy

We can question such a process of ‘desingularisation’ with social value lens: what is the social value of singularity versus a large standardisation?

When we consider the case we study that produces high level research (it is very well evaluated on the per capita criteria in the ARWU) with strong impact on industry and innovation, and that produces specific engineers who are immediately recruited by national and international firms

53

Research perspectives

Pursuing the work about these questions dealing with the future of singularity in the current higher education system

Others linked questions about The appropriation of evaluation devices by the

establishments The relations between industrial linkages and

excellence as it is defined by current formal evaluation devices

54

Research perspectives

New empirical investigations In the chemistry field

In France by others teams of the Prestence projectUniversity of Geneva, Switzerland, chemistry departmentPolytechnic of Turin

In other scientific fields

55

Thank you for your attention

[email protected]

[email protected]

universities in the ‘performance age’

Documents