what is research metrics and do we need them?
DESCRIPTION
TRANSCRIPT
![Page 1: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/1.jpg)
WHAT
ARE R
ESEARCH
METRIC
S AND D
O WE
NEED THEM?
Open Science, Open IssuesInternational seminar, workshops and Open Science work group meetingAugust 18-22, 2014 @ Rio de Janeiro, Brazil
Leslie ChanUniversity of Toronto Scarborough@lesliekwchan
![Page 2: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/2.jpg)
"Not everything that can be counted counts and not everything that counts can be counted.”
Albert Einstein
![Page 3: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/3.jpg)
QUESTIONS• Is the diversity of research practices
and outputs, particularly those enabled by digital tools and network, recognized in the current academic reward system?
• If current metrics are so poor, why are they so hard to replace?
• Is “altmetrics” the answer to this “accounting” problem?
• Is altmetrics truly alternative? • How to capture “deep” or meaningful
data of research and scholarly activities?
![Page 4: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/4.jpg)
METRICS AS
Complex socio-technical system
Driver of behaviour (reactivity)
Mirror of specific value system
![Page 5: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/5.jpg)
http://news.utoronto.ca/university-toronto-ranked-first-canada-24th-world?utm_source=Bulletin&utm_medium=Email&utm_content=Lead
Shanghai Jiao Tong ranks universities by measuring objective indicators of academic or research performance. These include: alumni and faculty winning prestigious prizes and medals; highly cited researchers; articles published in Nature and Science; articles indexed in major citation indices, and the per capita academic performance of an institution.
![Page 6: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/6.jpg)
http://www.shanghairanking.com/index.html
![Page 7: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/7.jpg)
http://www.shanghairanking.com/index.html
![Page 8: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/8.jpg)
http://www.timeshighereducation.co.uk/world-university-rankings/
![Page 11: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/11.jpg)
So what does that mean?
![Page 12: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/12.jpg)
http://www.nature.com.myaccess.library.utoronto.ca/nature/journal/v454/n7203/full/454398a.html
![Page 13: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/13.jpg)
Nature Vol 454|24 July 2008http://www.nature.com.myaccess.library.utoronto.ca/nature/journal/v454/n7203/full/454398a.html
![Page 14: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/14.jpg)
RESEARCH PRIORITY, POLICYAND NATIONAL INTERESTS
![Page 15: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/15.jpg)
Rethink the framework and language of “accounting”
![Page 16: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/16.jpg)
CRITICAL ACCOUNTING
The very act of “counting” certain things and excluding others shapes a particular interpretation of social reality, which in turn has policy implications
For example, what should ”performance”, “impact” and ”prestige" include/exclude?
The answers to questions such as these define the “mission”, "health," ”values" and "performance" of an institution
![Page 17: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/17.jpg)
SOCIAL ACCOUNTING?
“A systematic analysis of the effects of an organization on its communities of interest or stakeholders, with stakeholder input as part of the data that is analyzed for the accounting statement”
Broadens the domain of items that are included in “accounting statements” (i.e. outputs) so that social organizations and institutions can better tell their story
Quarter, Mook & Richmond (2003)
![Page 18: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/18.jpg)
18
… SOCIAL ACCOUNTING
Knowledge as “public good” and implications for “open” scholarship and research
![Page 19: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/19.jpg)
BROADENING THE DEFINITION OF “SUCCESS” AND “VALUE”
Traditional value: economic returnScholarly value - reputation and citation
Institutional value - public mission, community outreach
Social value - equity, participation, diversity
Political value - evidence based policy, transparency, accountability
![Page 20: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/20.jpg)
… ACCOUNTING FOR SOCIAL IMPACT
What counts as “input”, cost or investmentWhat counts as “output” or benefits and impact
What are the positive and negative externalities
![Page 21: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/21.jpg)
BOYER’S SCHOLARSHIP OF ENGAGEMENT
Discovery
Integration
Application
Teaching
![Page 22: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/22.jpg)
"The scholarship of engagement means connecting the rich resources of the university to our most pressing social, civic and ethical problems, to our children, to our schools, to our teachers and to our cities..."
Ernest Boyer in The Scholarship of Engagement (1996)
![Page 23: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/23.jpg)
BOYER’S SCHOLARSHIP OF ENGAGEMENT
Discovery
Integration
Application
TeachingEngagement
![Page 24: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/24.jpg)
Opportunities for Open Scholarship
Public outreach and engagement
New forms of “impact”
Data sharing
New scholarly practices
Experimentations
Interdisciplinary and
Collaborative research
Professional development
Personalization
Curation
Student training
Service
![Page 25: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/25.jpg)
RETHINK
Knowledge as Public GoodScholarship as ServiceQuality and PrestigeSocial MetricsEducation as Emancipation
![Page 26: What is research metrics and do we need them?](https://reader034.vdocuments.site/reader034/viewer/2022051323/54985d77b4795927538b48f0/html5/thumbnails/26.jpg)
The beauty of the current moment is that new media has thrown all of us as educators into just this kind of question-asking, bias-busting, assumption-exposing environment. There are no easy answers, but we can at least be thankful for the questions that drive us on.
Michael Wesch (2009)
http://mediatedcultures.net/publications/from-knowledgeable-to-knowledge-able-learning-in-new-media-environments/