timeliness and quality

7th euroCRIS Strategic Seminar, Juergen Guedler

Brussels, September, 14-15th, 2009

Timeliness and qualityInformation products based on process-produced data – requirements on the quality assurance process

Presentation at the 7th euroCRIS Strategic Seminar

Brussels, September, 14-15th 2009



Agenda

1. DFG at a glance

2. DFG‘s main source of information

3. Timeliness and Quality

4. Scope of DFG information products



Third-party funding income of German

higher education institutions (HEI) by source (2006)



DFG Awards by scientific discipline (2007)

Mechanical and Industrial Engineering

6%

Geosciences6%

Mathematics3%

Physics10%

Chemistry7%Agriculture and

Veterinary Medicine

2%Medicine

21%

Biology13%

Social and

Behavioural Sciences

6%

Humanities10%

Construction Engineering and Architecture

1%

Computer Science,Electrical and System Engineering

8%Material Science

3%

Thermal and ProcessEngineering

4%

Life Sciences

EngineeringSciences

Soc.Sc. &Humanities

NaturalScience

Research fields:

► Soc.Sci.+Humanities 16%

► Engineering Sciences 22%

► Natural Sciences 26%

► Life Sciences 36%



DFG programme portfolio and budget (2007)

29.3%: Individual Grants Programme

54.6%: Coordinated Programmes

06.3%: Promoting Young Researchers

01.2%: Scientific Prizes

07.2%: Scientific Instrumentation and Infrastructure

01.4%: Encouraging Scientific Contacts (and other)

Programme portfolio and budget:

~2 Billion €, more than 1.000 programmes and 20.000 projects funded



Department ICentral Administration

Strategic Planning Committees

Executive Office

Quality Assurance and Programme

Development

InternationalAffairs

Press andPublic Relations

InternalAuditing

Berlin Office

Internal Advisory Committee

Extended Executive Board

Heads of Divisions and Executive-Level Offices

Executive BoardPresident Secretary General

Deputy Members:Heads of Departments I - III

Forum

Heads of Divisions

Budget and Accounting

Human Resources and Legal Affairs

Information Technology and Infrastructure

Information Management

Department IIScientific Affairs

Department IIICoordinated Programmes and Infrastructure

Research Centres

Research Careers

Scientific Library Services and Information Systems

Scientific Instrumentation and Information Technology

Humanities and Social Sciences

Life Sciences 1

Life Sciences 2

Physics, Mathematics, Geosciences

Chemistry and Process Engineering

Engineering Sciences

Dept. directly involved in the funding process

The DFG´s Head Office: An overview



The funding-process-chain

Whole chain is represented in theDFG-internal databases and documentmanagement systems

Advice

Submission

Formal examination

Reviewer

Statutatory bodies

Report

Proposal

Peer Review / Assessment

Performance Evaluation

Funding

Decision



Agenda

1. DFG at a glance






Facts and figures

► DFG internal electronic proposal processing system

► in use since 2005 (data coverage from 1982)

► used for all funding programmes

► used by about 400 out of 750 staff members directly involved in funding processes

► data source for all information products published by the DFG

ElektrA – the data source



Agenda

1. DFG at a glance






Timeliness and quality

► Data in ElektrA are very actual („day before“)

► Because of the use of data for funding-process related products (e.g. letters to applicants and reviewers) most information are of high quality

► But: less use in funding related products results in lower quality



Organisational model for quality assurance (QA)

► Quality responsibility is decentralised

● staff members are trained to produce good quality data

● software helps to avoid mistakes

► Nevertheless there is a need for a central quality assurance unit:

● development of complex quality assurance routines

● occasionally very high demand for timeliness (e.g. ad hoc statistics or „the day after“ information products like reports on decisions of DFG boards)

● most important: Training of data producing staff on how to avoid mistakes and – very helpful! – elucidation on the importance and use of infomation products based on these data



Arguments for investing high effort in quality

Only one:

► Quality of data ist quality of information products is acceptance and practical use for planning purposes of these products



Agenda

1. DFG at a glance






Statistical reporting based on the DFG internal databaseExample: Quarterly statistics for the statutory bodies

entsch. bew. Förderquote entsch. bew. Bew.-quote

N N % N Mio. € Mio. € % Summe

Allg. Forschungsförderung 4.495 2.704 60,2 1139,0 465,7 40,9

Einzelanträge 2.756 1.404 50,9 785,9 270,8 34,5

Forschungsstipendien 201 123 61,2 9,4 5,0 53,3

Heisenberg-Programm 66 32 48,5 11,3 5,7 50,8

Reinhart Koselleck-Projekte 24 3 12,5 28,7 3,9 13,6

Schwerpunktprogramme 682 448 65,7 185,2 91,5 49,4

Forschergruppen 226 206 91,2 67,1 52,3 77,9

Wiss. Bibliotheken und Informationssysteme 142 111 78,2 47,1 32,6 69,2

Sonstiges 398 377 94,7 4,2 3,8 89,8

Emmy Noether Programm 96 52 54,2 55,9 19,1 34,2

Forschungsgroßgeräte 130 113 86,9 31,6 27,2 85,9

Anzahl Summe

Verfahren

44

61

30 30

1318 17

13

29,5%

43,3%

29,5%

56,7%

0

10

20

30

40

50

60

70

Forschergruppen Schwerpunkt-programme

Sonderforschungs-bereiche

Graduiertenkollegs

Anzahl

0%

20%

40%

60%

80%

100%

Entschiedene Anträge Bewilligte Anträge Empfehlungsquote

20

30

40

50

60

70

80

Mai 07

Jul 07

Sep 0

7

Nov 0

7

Jan 0

8

Mrz

08

Mai 08

Jul 08

Sep 0

8

Nov 0

8

Jan 0

9

Mrz

09

Mai 09

%

Bewilligungsquote gesamt Halbjahresmittel gesamt

Halbjahresmittel Neuanträge Halbjahresmittel Fortsetzungsantr.

► Statistical reporting for the statutory bodies of the DFG:

● e.g. success rates by programme

● e.g. number of programmesfunded/rejected

● e.g. development ofsuccess rate over time



DFG internal data as a source for evaluative studies

Evaluative Study: „Women in the DFG“ (2008)

► Topic: Gender equality has been one of the aims of the foundation since 2002 (Article 1 of its statutes)

► Objective: Transparency, planning criteria (e.g. development for the modification offunding instruments)

► The project team: Prof. Thomas Hinz, Ina Findeisen, Kathrin Auspurg, University of Konstanz

► Target group: Statutory bodies, interested members of the public

► Data: Mainly based on process-produced data; many other sources of data included

► Study (german) and results in brief (engl.):www.dfg.de/dfg_im_profil/zahlen_und_fakten/statistisches_berichtswesen/gleichstellung/



Characteristics

► information system on DFG-funded projects

► different search functions

► shows all funded projects at a university, department, or institute since 1999 (~ 60.000 projects)

Contribution to QA-process

► three quality checks –

● first during proposal processing

● second by applicant before publishing

● constant email-feedback by users and applicants

GEPRIS – German Project Information System



Research Explorer (REx): www.research.explorer.dfg.de

Characteristics

► bilingual database (german / english) on research institutesat German HEIs and other institutions

► detailed information on nearly20,000 institutes at these institutions

► links to GEPRIS

Contribution

► daily by proposal processing within DFG

► periodically by QA-team for each university

► occasionally by user-feedback



The DFG Funding Ranking

Planning tool for universities and other actorsengaged in research planning and strategy.

Helps to identify research profiles by means of:

► Institutional Profiles: Information about research profiles and main research fields covered by German HEIs

► Regional Profiles: Unveils regional profiles and clusters of excellence

► Structural Profiles: Informs about networks in science

► Disciplinary Profiles: Detailed analysis on research activities in different disciplines and fields of research (e.g. biotechnology)



iele,

Objectives, approach and databases of the DFG Funding Ranking

Deutsche ForschungsgemeinschaftGerman government

European UnionEuropean Research Council

German Federation ofIndustrial Reserach Organizations

Alexander von Humboldt FoundationGerman Academic Exchange Service

iele,

Analysesby source

Analysesby recipients

dimensions of differentiation:

countries, states, regions, institutions.

Indicators cover approximately 90 percent of the third-party funding income of scientific and higher education institutions in Germany.

Huge set of additional non-monetary indicators (intern. attractivity, scientific expertise...)

No costly and laborious data collection from the research institutions but direct processing of data concerning funding activities of central sources

Subject-relatedAnalyses

Classification according tofour scientific disciplines

Specifics:

DFG: 48 research fields German government: 12 funding fields

EU: 8 funding fieldsAvH / DAAD a.o: 14 research areas



DFG Funding Ranking

Influence of programmes (e.g. ExIn)

Gender equality

International appeal

Networks and cooperation structures

Funding rankings and profiles

Participants in peer review processes

Aspects of statistical reporting within the DFG Funding Ranking



Disciplinary profiles of universities

► Example: DFG-funding in the humanities

► range of information:

● Volume of funding

● spectrum of disciplines of each HEI

● participation of HEI in „Excellence Initative“

► FU Berlin – LMU Munich:

Similar vol of funding

und

similar disciplinary profile

– different participation in ExIn



Networks and cooperation structures within DFG funded

programmes: humanities

► Very dense coll. cluster in Berlin

► = Berlin capitalizes its ressources best



Recipients in Germany

Ranking of countries

Land Mio. € kum. %

Deutschland

Großbritannien

Deutschland

Italien

Niederlande

Spanien

Belgien

Frankreich

Insgesamt

3.024,0

2.369,6

2.172,3

1.457,9

1.107,4

943,8

707,9

16.665,3

1,8

3,2

4,5

5,4

6,1

6,6

7,1

100,0

Art der Einrichtung

Hochschulen

Außeruniversitäre Einrichtungen

Max-Planck-Gesellschaft (MPG)

Fraunhofer-Gesellschaft (FhG)

Helmholtz-Gemeinschaft (HGF)

Leibniz-Gemeinschaft (WGL)

Bundeseinrichtungen

Weitere Einrichtungen

Industrie und Wirtschaft

Nicht-institutionelle Mittelempfänger

Ingesamt

Mio. €

947,9

1.173,5

154,3

216,4

316,9

94,5

128,1

263,2

900,9

1,7

3.024,0

%

31,3

38,8

5,1

7,2

10,5

3,1

4,2

8,7

29,8

0,1

100,0

6th Framework Programme of the European Union



ElektrA

management reports

individual Letters

funding ranking

GEPRIS

Research Explorer

applicants,reviewers

reportusers

Rankingrecipients

GEPRISusers

RExusers

Qualityassurance team

Feedback

qualityassurance

different products different usersone data source one qa-team

The quality assurance chain



Conclusions

► Good quality of data is costly and need personnel

► Decentralization and direct use of data for business related products is the best guarantor for a good data base

► Besides this it is necessary to centralize QA processes

► The more products with certain specifications and certain target groups are based on the same data, the more feedback is possible to improve quality

► every data problem causes trouble to a broad set of information products – so high data quality is a good reason for investing in well defined QA processes



Thank you for your attention!

more information:►DFG: www.dfg.de

►DFG-Funding-Ranking: www.dfg.de/ranking

► DFG- funded projects: www.dfg.de/gepris

► research institutes at German HEI: www.dfg.de/research_explorer

timeliness and quality

Documents