relevance in academic research dan remenyi phd visiting professor school of systems and data studies...
TRANSCRIPT
Relevance in academic researchDan Remenyi PhDVisiting ProfessorSchool of Systems and Data StudiesTrinity College [email protected]
Establishing the context•The issues of relevance and rigor are complex
concepts and need careful understanding.•Some words have a large semantic footprint
and relevance, rigor, validity, reliability, generalisability and integrity are good examples of these.
•No one can proceed with any understanding of research without having a clear agreement with regards the meaning of terms.
April 21, 2023
2
The effectiveness and efficiency parallel •A quick rule of thumb understanding of
relevance is doing the right thing or looking at the right issues
•A quick rule of thumb understanding of rigor is doing the thing right
•These are the colloquialisms coined by Peter Drucker to explain effectiveness and efficiency
•Thus relevance equates to effectiveness and rigor relates to ……… well… not so much efficiency but something more like adequacy
April 21, 2023
3
Relevance and Rigor•Relevance is a high level issue in that you
can say that the research project is relevant or not relevant as a whole …………it could also be said that the use of case studies was not relevant to the objectives of the study
•Rigor is an issue which permeates the whole project in that you can say that the literature review was not rigorously approached or the sample was not rigorously chosen, or the conclusions were not rigorously written up.
5
21 April 2023
Academic Research is a Challenge
Academic research
•What is the nature of academic research?▫It is an intellectual activity▫It is an attempt to answer a question▫It is a process of question, data and analysis
•Academic research has two primary attributes:-▫It adds something of value to the body of
theoretical knowledge…….and practice!▫It demonstrate a high level of scholarship
April 21, 2023
6
Something of value to the body of theoretical knowledge…….and practice•Something of value is reflected in the
concept of relevance.•Relevance needs to be perceived by both
the non-academic stakeholders and also the academic community.
•Without some contribution to theory the research will not be much valued by academics.
•Mode 2 brings both together.
Scholarship and rigor•Being well read and being able to
demonstrate that in conversation•Being able to craft an argument in writing•Scholarship is connected to rigor•An argument is a series of logical steps by
which someone is persuaded. In this sense is not confrontational.
•Academic work i.e. either a dissertation or a peer reviewed paper needs to argue for relevance, rigor, validity, reliability, generalisability.
Quality academic research• Dimensions of quality in academic research
addresses a pertinent question uses academic language with care - remember the
fog factor uses appropriate data is written up in intelligible English and correct
format is presented in an attractive way - structure invoking authority finds an acceptable/useful answer
• Being parsimonious quality academic research needs to be relevant and rigorous
April 21, 2023
9
Double hurdle
•Academic research needs to be relevant i.e. of interest and of use to a wide range of management at different levels and rigorous i.e. display academic excellence.
•Being both relevant and rigorous has been described as a synthesis strategy.
Research question
•The research question has been reached through a careful examination of and understanding of :1. The field of study2. The topic3. The specific questions and sub-questions
April 21, 2023
11
Some critical issues
• Something of value As judged by the academic community
• Body of theoretical knowledge Add something to established domains
• Parsimonious Expressed in a precise and economical way
• Authority Ideas/concepts/theories which have through the
peer review process• Relevance
Has value to a group of stakeholders• Rigor
Complies with the rules of the research paradigm
April 21, 2023
12
Three more issues•Validity ……………..Has the research
addressed the intended issues i.e. addressed the issues which are required to answer the research question?
•Reliability…………..Has the research been conducted sufficiently well (rigorously) to provide results in which we can have confidence?
•Generalisabilty……..To what extent can the research be applied else where?
April 21, 2023
13
Rationalism versus Empiricism• Rationalism (theory)
The mind is superior to the senses – dating from the Greeks……at least Plato…..thought experiments
Reason Logic Secondary data
• Empiricism If it cannot be held, felt or shaken etc it does not
exist! In its pure form it fails due to particle physics etc Positivism versus interpretivism Primary data
April 21, 2023
14
Assumptions - preliminary concepts• What are the assumptions under which academic research
is conducted?▫ The world is understandable.▫ We are interested in understanding it. ▫ Researchers will be open minded.▫ Researchers are capable of objective reasoning.▫ We can find data/evidence to explore the phenomena and
therefore understand them.▫ The nature of the understanding can be both broad and also
detailed.▫ Occam’s razor points out that too much complexity can
obscure what is happening.▫ We always have to use ceteris paribus – all other things being
equal.▫ That some degree of cause and effect can be
identified/assumed. ▫ We all operate in terms of our cognitive capacity which
changes, and hopefully develops/increases over time.
April 21, 2023
15
21 April 2023
16
The Issue of Integrity – Fleischmann and Pons
21 April 2023
17
Integrity and Professor Woo Suk Hwang And Professor Sir Cyril Burt
Sir Cyril BurtReported nonexistent studies on I.Q. tests in the 20’s and 30’s, and added fictitious collaborators to his papers
Some ideas to examine • Relevance ………..doing the right thing• Relevance ………..to whom or to what?
theory/academics/practitioners......Is it significant?
• Rigor ……………….doing the thing right• Is rigor a dimension of validity?• If the research is not valid then do not worry
about reliability? • If the research is not reliable can it be valid?• If the research is neither valid or reliable then is
it worthless?• What can go wrong question, data, analysis,
finding conclusions etc……threats to the research
April 21, 2023
18
Evaluation of relevance and rigor• Relevance is normally evaluated holistically i.e. is
the research important or is it not. Sometimes some aspects of the research may be more useful than others but this question is essentially a holistic one.
• Rigor is normally evaluated on an issue by issue basis. This requires comment of individual aspects of the research. But there is also a holistic view. There may be some tension between the individual evaluations and a holistic view of the research. The sum of the parts may be different to the whole.
April 21, 2023
19
Relevance• In information science “Relevance is the extent to
which some information is pertinent, connected, or applicable to the matter at hand. It represents a key concept in the fields of documentation, information science, and information retrieval”. Pehcevski and Larsen
• Although academic relevance is today of essential importance to research, surprising little has been written about it.
• It used to have been assumed that if the research is “good” then it is academically relevant and this attitude has only recently been challenged.
• The main drivers of this challenge to relevance may be seen in the debate surrounding Mode 1 and Mode 2 research.
April 21, 2023
20
A stumbling block issue•Previously relevance was not a visible issue.
If the research and his/her supervisor thought that the question was important then that was all that was needed.
•Now the research has to convince the examiners or the reviewers that it is adequately relevant. An argument has to be made.
•A research dissertation/paper could be refused/rejected on the grounds that the research is not adequately relevant
April 21, 2023
21
Weak and strong relevance•According to Blake Ives (US IS academic)
there are two forms of relevance - weak relevance and strong relevance - http://www.misq.org/archivist/vol/no20/issue3/edstat.html
•Weak relevance occurs when a research question addresses an issue which is known and documented
•Strong relevance requires an action research approach to answering the research question which produces concrete findings
A closer look at relevance•The statement that a research question is
relevant begs the question of “relevant to whom?” and “in which way?”
•A research question is relevant if there is:- A distinct interest among the stakeholders – but
who are the stakeholders? A material stakeholder group
The research question will provide interesting results and/or useful guidelines to professionals
Improvements to practice i.e. there is clear application for the findins
April 21, 2023
23
How are relevant research questions to be identified?• Relevant research questions may be found in:-
▫ the extant literature▫work place experiences▫special interest groups▫Business/professional/practitioner groups▫government requests▫general business press
• Remember that a relevant question leads to a solution for someone
• Is there “pure” or “basic” research in B&M Studies?
April 21, 2023
24
What do we need to find a relevant research question?• Imagination!• …….. But also needed is imagination to create from
these hints the great generalizations – to guess at the wonderful, simple, but very strange patterns beneath them all, and then to experiment to check again whether we made the right guess. Richard Feynman, Six Easy Pieces, p2, Penguin Books, London (1995)
• Finding a suitable research question should not be rushed.
• Activities which can help to find relevant research questions include:-
Talking to knowledgeable informants Focus groups Qualitative questionnaires Blue sky and scenario sessions Talking to your supervisor/s Literature………academic and popular press
21 April 2023
25
A caveat!• The relevance will be judged by both
academics and by practitioners but academics will/may have a greater say than practitioners.
• The degree of relevance is often tested by the “So What?” or “What’s new ....... and who cares?” question. This question needs to be directly answered in a research dissertation and in a peer reviewed academic paper.
• Relevance or the lack thereof will always be known post hoc or a posteriori but you have to address this issue on an a priori basis
April 21, 2023
26
The time dimension
•Relevance has a time dimension.•What is relevant today may not be
relevant tomorrow.•The time value of a research question is a
function of the field of study and the topic and directly influences the relevance of the research.
April 21, 2023
27
The argument
•Relevance may not be taken for granted.•The researcher needs to argue that the
research is relevant because it:- answers a well known /important question/s addressed a non-obscure question will produce a useful result usually a paradigm will allows several other questions to be
explored it will attract people and funding to the
university or business school etc
April 21, 2023
28
The argument is one of the key issues for establishing relevance and rigor
April 21, 2023
29
Assessing relevance
•Feedback from informants and peers is the most effective way of ensuring that the research is relevant.
•It is useful to have in mind the need to test this relevance occasionally through out the duration of the research
•Also test the findings by asking the stakeholders of the results of the research are likely to be useful to them.
April 21, 2023
30
Journal titles......Relevant?• Stark, S., Chernyshenko, O., & Guenole, N.
(Accepted). Can subject matter expert ratings of statement extremity be used to streamline the development of unidimensional pairwise preference scales. (18 no “?”) Organizational Research Methods
• Chernyshenko, O., Stark, S., & Guenole, N. (2007). Can the discretionary nature of certain performance criteria lead to differential prediction across cultural groups? (15 yes or no answer) International Journal of Selection and Assessment, 15, 175-184
• Be careful of papers with very long titles and many authors.
April 21, 2023
31