italian research evaluation exercise 2010” and the …...vqr 2004‐2010 • evaluation of...
TRANSCRIPT
![Page 1: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/1.jpg)
Italian Research Evaluation Exercise “VQR 2004‐2010”
and the
University of Milano‐BicoccaBonaria Biancu
University of Milano‐Bicocca
![Page 2: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/2.jpg)
University of Milano‐Bicocca
• Young
(1998),
dynamic
and
multidisciplinary, offering Bachelors, Masters, PhDs
• 16 Departments
• +32,000 students• +900 scholars
and researchers
• 23rd in the Times
Higher
Education
World Ranking
of
universities
under 50 years
old
ICT Area ‐
University of Milano‐Bicocca 2
http://www.unimib.it/english
My
appointment:ICT Area ‐
Research
Information Systems
Office
![Page 3: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/3.jpg)
CAVEAT
• Data published
on ANVUR website on July 31st, 2013
but…
• Still
unstable
/ missing
• Slides based on ANVUR final report (07.31.2013 version)
• Focus on rating, not ranking
• Slides based on ANVUR final report (07.31.2013 version)
• Focus on rating, not ranking
ICT Area ‐
University of Milano‐Bicocca 3
![Page 4: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/4.jpg)
VQR 2004‐2010
• Evaluation of research quality of Italianuniversities and departments (and otherresearch institutions)
• Starting date: November 7th, 2011
• Ending date: July 31st, 2012– June 2nd, 2013
• Publication of VQR results report: July 16th,2013
ICT Area ‐ University of Milano‐Bicocca 4
![Page 5: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/5.jpg)
VQR 2004‐2010 Data Requests• Departments (pre‐ and post‐ University Reform Law and Bicocca
bylaws changes)• ‘Soggetti valutati’ i.e. academic staff in post on the date of the
census (11.7.2011)• 3 scientific products for every researcher (2 / 1 depending on the
scientific age or leaves; 6 for researchers with a joint appointmentboth with Bicocca and a research institution)
• Researchers/academics recruited or advanced• PhD, Specialization, Post‐doc students• Research and financial data:
– Income from competitive grants and contracts with private firms– University expenditure for research projects– Patents, spin‐offs, consortia– Other 3rd stream activities• Self‐evaluation Report
ICT Area ‐ University of Milano‐Bicocca 5
![Page 6: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/6.jpg)
Subject Area indicatorsand weights
w = mean of 4 indices (research
products and costs)
Subject Area wj (%)1 Mathematics 4,00%2 Physics 5,67%3 Chemistry 7,80%4 Earth Science 2,10%5 Biology 10,75%6 Medicine 19,47%9 Computer Engineering 10,73%
10 Literature 7,66%11a Philosophy, Pedagogy 4,94%11b Psychology 1,97%12 Law 5,07%13 Economics 5,12%14 Sociology 2,72%
7, 8a, 8b
Agriculture; Civil Engineering, Architecture ‐12%
Tot.: 0,88
ICT Area ‐ University of Milano‐Bicocca6
• IRAS1: Publications and patents: 0.5• IRAS2: Income from competitive
grants: 0.1• IRAS3: Publications and patents of
researchers recruited/advanced: 0.1• IRAS4: Internationalization: 0.1• IRAS5: PhD/Spec/Post‐doc students:
0.1• IRAS6: Expenditure of the university
for research projects: 0.05• IRAS7: Improvement on VTR 2001‐
2003: 0.05
![Page 7: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/7.jpg)
Internal procedure steps• Working group of technical and administrativestaff– ICT, Research, HR, Accounting&Budget,representative for Departments
• University and Departments’ Faculty Committeesto evaluate research products– Avoid submitting the same publication more thanonce
• All steps of publication selection and submissionmanaged via an internal CRIS software (SURplus)
ICT Area ‐ University of Milano‐Bicocca 7
Strong collaboration between Techical‐Administrative Staff and Scholars‐University Officials
![Page 8: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/8.jpg)
Internal procedure steps
• Working group of technical and administrativestaff– ICT, Research, HR, Account&Financial, deptsrepresentative
• University and departments’ Committees to evaluate research products– Avoid presenting same publication twice
• All steps of publications selection and submissionmanaged via an internal CRIS software (SURplus)
Focus on publications
ICT Area ‐ University of Milano‐Bicocca
Strong collaboration between Techical‐Administrative Staff and Scholars‐University Officials
Strong collaboration between Techical‐Administrative Staff and Scholars‐University Officials 8
![Page 9: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/9.jpg)
Selection and submission of publications
The Scholar
selects, using an
internal procedure
(CRIS module),3 to 20
publications
previously
registered in the
institutional
repository
The Rector + the
University Committee
verify that VQR
criteria are met, no
duplicates have been
selected
(following
negotiation) and give
the OK to the
submission
February to June 2012
ICT Area ‐
University of Milano‐Bicocca9
The Department
Committees verify
that every scholar has
completed the
procedure, VQR
criteria are met and
no duplicates have
been selected
(following
negotiation); finally
check the overall
quality of publications
![Page 10: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/10.jpg)
Publication selection
ICT Area ‐ University of Milano‐Bicocca
10
+ setting thepriority i.e.ranking 3 up to20 publications
![Page 11: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/11.jpg)
Additional information onselected publications
ICT Area ‐ University of Milano‐Bicocca 11
![Page 12: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/12.jpg)
Additional information on selected publications
ICT Area ‐
University of Milano‐Bicocca 12
Here InCites would have
helped! ‐> Having
normalized field citation
score ( / expected), IF
quartile, university
publications ranking
within Subect Areas etc.
Subject Categories
ISI/SCOPUS were used by
ANVUR to assign
publications to the correct
Panel and to evaluate
their impact factor and
citation score in context
Hopefully for the next VQRs we will save
time and perform even better…
![Page 13: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/13.jpg)
PDF upload or send request to the publisher
ICT Area ‐
University of Milano‐Bicocca 13
![Page 14: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/14.jpg)
University/Dept Committees ViewChecking completeness and correctness
ICT Area ‐ University of Milano‐Bicocca 14
![Page 15: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/15.jpg)
University/Dept Committee View Confirming or rejecting the proposed ranking
ICT Area ‐
University of Milano‐Bicocca 15
![Page 16: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/16.jpg)
Package of publications XML record
i.e. NO manualintervention by
academics other than inSURplus…
ICT Area ‐ University of Milano‐Bicocca16
+ one ormore PDF
files…
…
Sent to ANVUR FTP server
… Hence major control over the procedure by the ICT staff
![Page 17: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/17.jpg)
Overall FTE / publications
• 916 academic staff
• 21 departments– 16 after bylaws changes
• 2,309 products (over 2,339 expected)– Small number of low‐
or non‐productive staff
ICT Area ‐
University of Milano‐Bicocca 17
![Page 18: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/18.jpg)
Area# prodotti attesi (n)
% sul totale di Area
# prodotti M % su attesi# prodotti conferiti
# Prodotti P% su
conferiti
1 Mathematics 236 2,71 12 5,08 224 0 0,002 Physics 166 2,75 2 1,20 164 0 0,003 Chemistry 117 1,49 0 0 117 0 0,004 Earth Science 75 2,58 0 0 75 0 0,005 Biology 227 1,73 0 0 227 0 0,006 Medicine 292 1,06 1 0,34 291 4 1,37
missing Subject Areas 7 and 8a/8b (less than 10 products)9 Computer
Engin.33 0,24 2 6,06 31 0 0,00
10 Literature 55 0,39 0 0 55 0 0,0011.a Philosophy,
Pedagogy193 2,09 0 0 193 0 0,00
11.b Psychology 184 5,40 0 0 184 0 0,0012 Law 260 2,07 7 2,69 253 1 0,4013 Economics 360 2,94 6 1,67 354 0 0,0014 Sociology 130 3,09 0 0 130 0 0,00
TOTALE 2328 1,52 30 1,29 2298 5 0,22
Number of publications per Subject Area and % of missing / not suitable
ICT Area ‐ University of Milano‐Bicocca18
![Page 19: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/19.jpg)
Area# prodotti
attesi (n)
% sul totale
di Area
# prodotti M % su attesi# prodotti
conferiti
# Prodotti P% su
conferiti
1 Mathematics 236 2,71 12 5,08 224 0 0,002 Physics 166 2,75 2 1,20 164 0 0,003 Chemistry 117 1,49 0 0 117 0 0,004 Earth Science 75 2,58 0 0 75 0 0,005 Biology 227 1,73 0 0 227 0 0,006 Medicine 292 1,06 1 0,34 291 4 1,37
missing Subject Areas 7 and 8a/8b (less than 10 products)9 Computer
Engin.
33 0,24 2 6,06 31 0 0,00
10 Literature 55 0,39 0 0 55 0 0,0011.a Philosophy,
Pedagogy
193 2,09 0 0 193 0 0,00
11.b Psychology 184 5,40 0 0 184 0 0,0012 Law 260 2,07 7 2,69 253 1 0,4013 Economics 360 2,94 6 1,67 354 0 0,0014 Sociology 130 3,09 0 0 130 0 0,00
TOTALE 2328 1,52 30 1,29 2298 5 0,22
Number of publications per Subject Area and % of missing / not suitable
ICT Area ‐
University of Milano‐Bicocca19
«…
The percentage of missing products over expected ones (1,32%) is lower than the overall universities mean (4,68%), as is
the percentage of non evaluable products over the submitted ones (0,22% against an overall mean of 0,89%), thus denoting a particular care in selecting and submitting products»
ANVUR Final Report – Univ. of Milano Bicocca, page 4th
![Page 20: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/20.jpg)
Quality of research publicationsScore of the publications
Area I Quartile1 0,7 Q22 0,82 Q23 0,89 Q14 0,63 Q25 0,76 Q16 0,65 Q19 0,53 Q4
10 0,75 Q111a 0,69 Q111b 0,66 Q112 0,54 Q213 0,4 Q114 0,6 Q1
7, 8a, 8b < 10 publications ‐
Subject Areas(size‐
independent):1st Quartile: 82nd Quartile: 44th Quartile: 1
ICT Area ‐ University of Milano‐Bicocca20
![Page 21: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/21.jpg)
Quality of research publicationsExcellent publications
Area X Quartile1 1,12 Q22 1,12 Q23 1,17 Q14 1,57 Q15 1,12 Q26 1,34 Q29 0,84 Q4
10 1,63 Q111a 1 Q211b 1,3 Q112 0,92 Q213 1,02 Q214 1,84 Q1
7, 8a, 8b < 10 publications ‐
ICT Area ‐ University of Milano‐Bicocca21
Ratio ≥ 1for all but two Subject Areas
![Page 22: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/22.jpg)
Quality of research publications Ratio of quantity and quality and
IRAS 1
Area R Quartile IRAS1 IRAS1*1001 1,17 Q2 0,031687403 3,162 1,06 Q2 0,029066606 2,93 1,12 Q1 0,016720134 1,674 1,11 Q2 0,028623661 2,865 1,24 Q1 0,021431242 2,146 1,39 Q1 0,014727496 1,479 0,73 Q4 0,001779182 0,18
10 1,14 Q1 0,004477044 0,4511a 1,19 Q1 0,02477823 2,4811b 1,2 Q1 0,064709183 6,4812 1,09 Q2 0,022529619 2,2513 1,24 Q1 0,036191855 3,6114 1,35 Q1 0,041398247 4,14
7, 8a, 8b < 10 publications ‐ ‐ ‐
Subject Areas(size‐
independent):1st Quartile: 82nd Quartile: 44th Quartile: 1
ICT Area ‐
University of Milano‐Bicocca22
ANVUR ranking(size‐dependent):
1st in the segment: 11st Quartile: 6
2nd‐3rd Quartile: 6
![Page 23: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/23.jpg)
IRAS 3
• Mean score for recruited and advanced researchers within the Subject Area; mean score within the Subject Area /
BicoccaArea # Soggetti assunti o
promossi nel settennioCategoria Voto medio assunti o
promossi nel settennio /
Voto medio assunti o
promossi nel
settennio Italia (Rmobil.
Italia)
Posizione nel
segmento
dimensionale
Voto medio assunti o
promossi nel settennio /
Voto medio Area‐Bicocca
(Rmobil. Area)
1 47 Medie 1,14 5°
su 22 1,48
2 25 Medie 1,08 14°
su 20 1,16
3 18 Medie 1,07 9°
su 22 1,16
4 16 Medie 1,26 4°
su 17 1,51
5 50 Medie 1,11 6°
su 20 1,30
6 57 Piccole 0,99 11°
su 17 1,33
8 1 Piccole ‐ ‐ ‐
9 8 Piccole 0,58 14°
su 16 0,66
10 15 Piccole 1,07 6°
su 18 1,16
11 98 Grandi 1,05 2°
su 12 1,14
12 62 Medie 1,00 23°
su 30 1,15
13 87 Grandi 1,11 11°
su 36 1,41
14 31 Grandi 1,26 1°
su 15 1,38
Totale 515
Ratio ≥
1
for all but one
Subject Areas
ICT Area ‐
University of Milano‐Bicocca23
![Page 24: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/24.jpg)
IRAS 1‐7 IRFS 1 and IRFS 2
Università Prodotti attesi
% Prodotti attesi sul totale
IRFS1 x100
16 aree
IRFS2 x100
16 aree
IRFS1 x100 14 aree
IRFS2 x100 14 aree
Milano Bicocca 2.339 1,51995 1,95428 1,94394 1,94902 1,93740
Università
% Prodotti attesi sul
totale delle Università
IRAS1 x w x 100
IRAS2 x w x 100
IRAS3 x w x 100
IRAS4 x w x 100
IRAS5 x w x 100
IRAS6 x w x 100
IRAS7 x w x 100
Milano Bicocca 1,51995 1,71124 1,83654 2,03698 3,04352 1,65572 2,45750 2,37015
IRAS 1‐7
IRFS 1 and IRFS 2
Actually Bicocca governmental budget is underestimated
Here lies the importance of evaluation!
ICT Area ‐ University of Milano‐Bicocca24
![Page 25: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/25.jpg)
This talk is in loving
memory ofLuisella Sironi,THEBIGBOSS
of VQR Working Group
ICT Area ‐ University of Milano‐Bicocca 25
![Page 26: Italian Research Evaluation Exercise 2010” and the …...VQR 2004‐2010 • Evaluation of research quality of Italian universities and departments (and other research institutions)](https://reader034.vdocuments.site/reader034/viewer/2022050511/5f9b922083e0ad378f56cfdb/html5/thumbnails/26.jpg)
Thank you!
Bonaria BiancuICT Area – University of Milano‐Bicocca
Via Cozzi, 53 – Ed. U520125 Milan (Italy)
Website: http://www.unimib.itEmail: [email protected] phone: +39.0264486236Mobile phone: +39.3386940012