how can research and artistic performance of university staff members be measured ict-based?
DESCRIPTION
How can research and artistic performance of university staff members be measured ICT-based? Harald Lothaller University of Music and Performing Arts Graz (Austria). CRIS 2010 - Current Research Information Systems, Aalborg University, Denmark, June 2nd - 5th 2010. Background Design process - PowerPoint PPT PresentationTRANSCRIPT
How can research and artistic How can research and artistic performance of university staff performance of university staff
members be measured ICT-based?members be measured ICT-based?
Harald LothallerHarald Lothaller
University of Music and Performing Arts Graz University of Music and Performing Arts Graz (Austria)(Austria)
CRIS 2010 - Current Research Information Systems, Aalborg University, Denmark, June 2nd - 5th 2010
Background Design process Technical implementation Accompanying measures Conclusions
OvervieOvervieww
Development following DIRKS
University of Music and University of Music and Performing Arts GrazPerforming Arts Graz
One out of 22 public universities in Austria One out of 6 public universities of arts in
Austria One out of 4 public universities in the city of
Graz
BackgroundBackground
University of Music and University of Music and Performing Arts GrazPerforming Arts Graz
About 420 staff members in teaching, arts, and research
About 150 staff members in administration and services
Nearly 2300 students from 59 countries Fields of studies: Church music, Composition
and music theory, Conducting, Electrical engineering/Sound engineering, Instrumental studies (classical music), Jazz, Music education, Musicology, Performing art, Stage design, and Voice as well as Doctoral study programs and Non-degree programs
BackgroundBackground
Austrian Universities act 2002Austrian Universities act 2002 §13: …the Intellectual Capital Report (ICR) “shall,
as a minimum, present in itemised form (1) the sphere of action, social goals and self-imposed objectives and strategies; (2) its intellectual capital, broken down into human, structural and relationship capital; (3) the performance processes set out in the performance agreement, including their outputs and impacts”.
§ 14: “The universities shall develop their own quality management systems in order to assure quality and the attainment of their performance objectives.”
BackgroundBackground
Communication of universitiesCommunication of universities Middaugh, 2001, p.1:
“Colleges and universities have done a horrible job of communicating to both internal and external groups what faculty do and how well they do it.”
Middaugh, 2001, p.54: …information about the outcomes of research or public service activities “are integral to a full understanding of what faculty do”.
BackgroundBackground
Staff members’ viewStaff members’ view The main activities are teaching, research, and
the advancement and appreciation of arts Additional administrative duties are out-of-favor
and often disregarded The work, its quality, and its outcomes can
hardly be measured and this holds for quantitative approaches in particular (cf. Middaugh, 2001, p.80ff.)
Institutional rules sometimes are takes as conflicting with “[…] the freedom of scientific and artistic activities […]” (Austrian Universities act 2002, §2)
BackgroundBackground
Two main questionsTwo main questions
How can the research and artistic performance of university staff members be measured?
How can the staff be made to participate in an online tool designed to measure their performance?
BackgroundBackground
Our approachOur approach
“a structured and rigorous approach designed to ensure that records and information management is firmly based in the business needs of the organization” (National Archives of Australia, 2001a, p.5) a)Preliminary investigationb)Analysis of business activityc)Identification of recordkeeping requirementsd)Assessment of existing systemse)Identification of strategies for recordkeepingf)Design of a recordkeeping systemg)Implementation of a recordkeeping systemh)Post-implementation review
DIRKS DIRKS methodologymethodology
Preliminary investigationPreliminary investigation
…it was inevitable to get an overview concerning the role of the “organization, its structure, the business, regulatory and sociopolitical environments in which it operates, and major factors affecting its recordkeeping practices” (National Archives of Australia, 2001, A-4).
DIRKS – step ADIRKS – step A
Analysis of business activitiesAnalysis of business activities
Austrian University act 2002 explicitly points to eleven tasks universities have to fulfill within their sphere of action:
“1. advancement of sciences (research and teaching), and the advancement, appreciation and teaching of the arts
[…]11. provision of public information on the
performance of the tasks of the universities” (§3)
DIRKS – step BDIRKS – step B
Identification of recordkeeping Identification of recordkeeping requirementsrequirements
Clear focus on the measurement of research and artistic performance, but not teaching or financial aspects…
Two main aims: 1) The new tool should represent the wide-
spread range of activities of the university members.
2) The new tool should fulfill all requirements from the Austrian Universities act 2002 with regard to quality management and the ICR in particular
DIRKS – step CDIRKS – step C
Assessment of existing systemsAssessment of existing systems Paper & pencil approach for some years
Did not fully comply with the new requirements from the ICR
Was often criticized by staff members as to narrow
Contained a categorization system that gave us a starting point for designing the new tool
DIRKS – step DDIRKS – step D
Identification of strategies for Identification of strategies for recordkeepingrecordkeeping
“The purpose of Step E is to determine the most appropriate policies, practices, standards, tools and other tactics that your organisation should adopt to remedy weaknesses identified in Step D and ensure that they meet recordkeeping requirements identified in Step C” (National Archives of Australia, 2001, E-4).
DIRKS – step EDIRKS – step E
Design of a recordkeeping systemDesign of a recordkeeping system
DIRKS – step FDIRKS – step F
Step 1: revised system
Step 2: external
discussionsStep 3: internal
discussions
Project owner: ICR and management requirements in mind
Step 4: list of
attributes
Four steps:
Design of a recordkeeping systemDesign of a recordkeeping system
DIRKS – step FDIRKS – step F
At present, we use eight groups of categories:Artistic activities (11 categories)Research-related activities (6 categories)Pedagogic activities (6 categories)Publications (17 categories)Projects (3 categories)Functions and activities (16 categories)Awards (5 categories)Administration and service (8 categories)
Implementation of a Implementation of a recordkeeping systemrecordkeeping system
DIRKS – step GDIRKS – step G
Campus management system: CAMPUSonline From campus and elsewhere
(http://online.kug.ac.at) General information and internal area (with log-
in) New feature „Leistungen“ in 2007 Administration area for flexible configuration
• Transfer of categorizational system into CAMPUSonline
• Configuration on 4 levels
Implementation of a Implementation of a recordkeeping systemrecordkeeping system
DIRKS – step GDIRKS – step G
Level 2
Level 3
Level 4
Level 2
Implementation of a Implementation of a recordkeeping systemrecordkeeping system
DIRKS – step GDIRKS – step G
Transfer into CAMPUSonline was done all at once Subsequent testing phase:
• Test takers from different scientific and artistic fields
• Test takers with wide range of computer-handling skills
Adaptations and reductions were necessary• with regard to complex data entering possibilities• with regard to visibility of icons and order of
details Testing several alternatives gradually Theoretical importance versus practical situation!
Implementation of a Implementation of a recordkeeping systemrecordkeeping system
DIRKS – step GDIRKS – step G
Stähler, 2002, p.290f.: Innovation does not necessarily succeed • Even if there is a benefit for the customers or
users• Acquirement of knowledge is linked to time
(restricted factor)
Acceptance of innovation is high with• High benefits ( highlighted in
communication)• Low time demands for acquiring knowledge (
support)
Implementation of a Implementation of a recordkeeping systemrecordkeeping system
DIRKS – step GDIRKS – step G
Roll-out to all staff members in January 2008 Mainly three kinds of accompanying support:1) Detailed script of more than 60 pages
• Handed out in trainings and available for download
2) Small-group trainings • Up to 8 persons• From same department• Rather similar computer-handling skills
3) Personal assistance on demand by email or telephone
Implementation of a Implementation of a recordkeeping systemrecordkeeping system
DIRKS – step GDIRKS – step G
Communication is of high importance!• Following the given structures• Users as multipliers in addition• Ok, but could have been better…• Internal report after 6 month (and then every
6 month)
Highlighting two aspects in communication:1) Requirements from University act2) Personal advantages (e.g., online self-
presentation)
Post-implementation reviewPost-implementation review
DIRKS – step HDIRKS – step H
Presentation of persons, departments or the university to the respective community and to the public
Intellectual Capital Report Requirements are met well Better data in quality and quantity Analyses much easier
Internal reports every 6 month Evaluations of single persons or departments
Facilitates self-reports Presentations are more traceable for peer reviewers
etc. Provides quality information in addition to quantity
Post-implementation reviewPost-implementation review
DIRKS – step HDIRKS – step H
End of 2009: > 10.000 data entries Participation rate: 2/3 of academic staff
Enhancements are going on:• Adjustments due to user feedback• Expecting changes due to new governmental
rules for ICR• Quality assurance• Export function• FAQs via video/podcast
Post-implementation reviewPost-implementation review
DIRKS – step HDIRKS – step H
Cooperation with the University for Art and Industrial Design Linz (Austria)• CAMPUSonline too• Adjusting our system to their specific needs
(i.e. their scientific and artistic fields in particular)
• Transferring the system• Adjusting or building respective queries and
analyses• Successfully completed in June 2009
Successful implementationSuccessful implementation
ConclusionsConclusions
Tool provides fundamental data for performance statistics
Following DIRKS
Comply with quality Comply with practical standards requirements
Lessons learntLessons learnt
ConclusionsConclusions
To summarize our experiences, we have learned that (a)high effort for involving a wide range of persons pays off twice afterwards, (b)sometimes research and management interests in data collection have to be pared down in favor of usability in practice, and (c)extensive instructions, personal assistance, and highlighting beneficial features can make even skeptics or unskilled computer users participate in an ICT-based performance recording.
Harald Lothaller
Quality Management & ReportingUniversity of Music and Performing Arts
Graz (Austria)
Email: [email protected]: www.kug.ac.at
ContactContact