Fitting Oval Pegs into Round Holes:Tensions in Evaluating & Publishing

Qualitative Research in Top-Tier North American Journals*

Michael G. PrattUniversity of Illinois at Urbana-Champaign

* Forthcoming in Organizational Research Methods (July, 2008)

In my experience qualitative and quantitative papers are evaluated in precisely the same way -- both need to follow cannons of good empiricism and logic. [Qualitative researcher on being asked about his/her criteria for evaluating qualitative research.]

The problem is when it is not evaluated differently -- when one standard is used for all research. [Qualitative researcher on being asked what the difference is between how qualitative and quantitative research is evaluated].

Goals

I. Setting expectations

II. Background: Crux of the debate & pragmatic concerns

III. Study & results: examining criteria-in-use A. Author

B. Evaluator (reviewer or editor)

C. Explicit comparison: qualitative vs. quantitative

IV. Three fundamental tensions in publishing qualitative research

V. Publishing tipsA. Managing publishing tensions

B. Increasing your chances for a revise-and-resubmit

Setting Expectations

This talk will be unlike many other CARMA talks:

1. No equations2. Not about data analysis, but about

publishing research (see Cummings & Frost, 1985; Gephardt, 2004; Golden-Biddle & Locke, 1993, 2007; Lee, 1999; Suddaby, 2006)

Crux of the Debate

1. Should qualitative research be held to similar criteria as quantitative research?

2. If different, is it even possible to come up with a common set of criteria for evaluating all qualitative research?

Crux of the Debate

Similar Criteria

1. Research is research

2. One can approximate “quantitative” criteria (e.g., generalizability) or use analogous criteria (e.g., credibility vs. internal validity – Lincoln &

Guba, 1985)

Different Criteria

1. Paradigms are too different – “such a resolution appears highly unlikely and would probably be less than useful” (Lincoln & Guba, 2000: 185)

2. Qualitative methods are too disparate for a common set

So What?: Pragmatic Concerns

• Lack of consensus means it is difficult to publish

• If you do not publish…

• Publishing in top-tier North American journals are often critical for career success (note: criteria may differ by journal, discipline and by geography)

Study

• Sample 1: Polled authors of published articles that were at least partially qualitative, that appeared between 1995-2005 in the following journals:

– Academy of Management Journal– Administrative Science Quarterly– Strategic Management Journal– Organization Science

Sample 1

• Published Authors: 336 authors identified – 107 usable responses– 64 could not be reached (e.g., technical,

leaves, or loss of life)– Of the 272 that could be reached, I received

responses from111 (40.8%)– Of these 4 did not feel qualified

Sample 2

• Put out a call to RMNet to all authors who submitted an article during 1995-2005 to AMJ, ASQ, OS, or SMJ

• Received 22 additional responses who filled out the same survey

Survey

• On-line (Survey Monkey)

• Forced choice: topics– study design (clear purpose, methodological

completeness & appropriateness)– framing and contribution (rq’s, relationship to theory

and contribution)– stance as a researcher (voice, authenticity, data

presentation)

* Note: I also included space to comment on the criteria

Survey

• Open-ended: – For those who have had their qualitative papers accepted at top-

tier journals, briefly describe the 3-5 reasons for your acceptance. (AUTHOR)

– If you have been an editor or reviewer of a qualitative paper submitted to a top-tier North American journal, please list 3-5 criteria that you feel are the most important for evaluating qualitative research. (EVALUATOR)

– In what ways, if any, do you feel that qualitative research is evaluated differently than quantitative research? (QUAL. VS. QUANT).

– Other issues?

Analysis

• Quantitative – counts & percentages

• Qualitative– Frequency counts of themes (e.g. novelty)

OR – Iterative, grounded-theory approach (Locke,

2001; Strauss & Corbin, 1998)

Findings

• (3) Perspectives– Author– Evaluator (reviewer or editor)– Qualitative vs. Quantitative

Author Perspective: What Respondents View as Important

• Analyzed forced-choice and open-ended.• 10% cut-off• Table 1: Qualitative responses for top 3-5 criteria:

Criteria Used Frequency Contribution to Theory 52% Novelty (theory, context or methods) 41% Transparent / Exhaustive / Well-Articulated Methods 31% Good Writing (interesting / compelling) 31% Editor/Reviewer Qualities 17% Links to Existing Theory 14%

Author Perspective• Table 2: Quantitative

Question Asked Most Frequently Noted Responses % #

Which of the following comments related to STUDY DESIGN…

1. Missing or inadequately described methods 2. Missing or inadequately described analysis

65.4%56.2%61.5%75%

519

4812

Which of the following comments about the FRAMING and CONTRIBUTION…

1. Insufficient contribution to theory: Not enough links to existing theory2. Insufficient contribution to theory: Tied too closely to existing theory - nothing is 'new' here

68%68.8

49.3%56.2%

5111379

Which of the following comments related to YOUR STANCE AS A RESEARCHER…

1. Too much interpretation of data as compared with showing your data2. Too much showing of your data (e.g., quotes) as compared with interpretation of your data

52.7%53.8%32.7%30.8%

297

184

Author Perspective

• Convergence around: – Contribution to theory– Well-articulated & complete methods– Some overlap in terms of writing and clarity (e.g.,

articulating the purpose of article)

• Differences in:– Qualitative: role of reviewers & editors (as

champions)– Quantitative: showing vs. telling (Golden-Biddle &

Locke, 2007)

Evaluator Perspective

• Analyzed open-ended question about 3-5 most important criteria as editor or reviewer

• Used 10% cut-off in deciding whether to report a theme

Evaluator PerspectiveCriteria Used Frequency Representative Quotes

Contribution to Theory 49% “Contribution to theory”Transparent / Exhaustive / Well-Articulated Methods

49% “Clear explanation of data analysis and methods”; “transparency of analytical methods”

Good Writing (interesting / compelling)

46% “Quality of writing”; “Paper is well-written and well argued”; “compelling explanation”

Convincing / Enough Evidence

14% “Evidence must be convincing”; “details are important – convincing and interesting ‘story’ being told”

Appropriateness of Method 25% “Answering WHY a qualitative approach is necessary for the research question at hand”; “could not be told as well with quantitative data”

Data-Theory Fit / Data-Interpretation Fit

24% “tight links between data and proposed theory” / “must strike a balance between data and interpretation”

Links to Existing Theory 20% “Paper does not ignore existing literature and [does not] assumes [sic] it is the first considering these issues”

Novelty (theory) 17% “novel concepts”, “new or different theory”, “originality”

Evaluator Perspective• Again, convergence on most critical criteria: contribution to theory,

transparent & well-articulated methods

• Novelty becomes a little less important –and primarily linked with theory; explaining the appropriateness of methodology becomes more important

• Showing-telling tension more nuanced:

[T]he manuscript must strike a balance between data and interpretation - with the balance being a moving target depending on the topic and the richness of the data. Data are typically preferred to interpretation, but careful guidance in processing data is always appreciated.

• Showing vs. telling linked to “good writing”

Quantitative vs. Qualitative Perspective

• Some informants talked explicitly about how qualitative research differs from quantitative research

• This perspective on the data revealed new insights

s

Inappropriate (Qualitative)

Criteria

Higher

StandardsNo “Boilerplate”

Problems with Editor or Reviewers

Barriers to Publishing(e.g., length of manuscript)

Themes in Comparing Qualitative to Quantitative Studies

Qual. vs. Quant. Perspective: Themes

1. Overly high standards

If I held “quanto” papers up to my standards, ASQ and AMJ would go out of business, since I do not believe that a study of process (and most are) that does not directly measure or observe that process is acceptable science. I do not hold these papers up to that standard, so they get published! Bottom line, quant work has luxury of lower standards in some ways.

Qual. vs. Quant. Perspective: Themes

1. Overly high standards

I have never had a quantitative study rejected by a journal because I am a fairly good researcher and writer (to A, A- and B journals). On the other hand of the six times I have submitted qualitative work to the same level journals, I have had only 1 acceptance (the one that put me on your contact list).

Reasons for Overly High Standards

• Lack of common reviewing standards (no “boilerplate”)

I think that qualitative research does not have the commonly understood methodological shorthand that quantitative research does, so researchers have to explain and justify their methods much more (compared to writing something like "we conducted a multiple regression analysis"). Consequently, evaluations are more skeptical and looking for “proof” of adequate rigor..

Reasons for Overly High Standards

• Problems with editors & reviewers

The biggest issue is that qualitative papers are often reviewed by reviewers who are either not adequately trained in qualitative methods, or who may not regard qualitative research as being as valid as quantitative research. As such, qualitative research must meet a higher standard.

Even boring quantitative research gets published; careful, yet uneventful qualitative work will get ditched. Qualitative researchers are usually very stringent when reviewing, which is fine, but makes the bar much higher; oddly enough, constructs in quantitative can be daft and unrelated to any sense of reality and it's okay as long as the regressions have stars, clearly not so in qualitative!

Qual. vs. Quant Perspective: Themes

2. Inappropriate (quantitative) standards

In general, the positivist perspective is applied by most reviewers in evaluating qualitative research -- for example, complaints that research is not "scientific" and interpretations are "subjective.”

Reason for Inappropriate Standards

• Lack of “boilerplate”

In evaluating qualitative research, there is insufficient recognition of its variety in methodology and epistemology. So, review comments often assume or adopt one way of how the research should look that usually corresponds with the deductive quantitative model of doing and presenting research. More focus on generating possible alternative hypotheses for phenomenon observed and presented.

Reasons for Inappropriate Standards

• Lack of adequately trained evaluators

I believe most quantitative reviewers - which make most of the reviewers in top tier American journals - do not know enough about qualitative epistemology and methodology to actually appreciate qualitative work in its own terms. Thus, they use evaluations that better fit a quantitative approach.

Outcome of Overly High & Inappropriate Standards

• Barriers to publishing – especially length

The burden is higher to "prove" the results are valid. With quantitative research, people follow the numbers and fairly well-accepted protocols for analysis. In qualitative research, I have to walk the reader through every step of data collection and analysis and justify why I made each choice.

The editor kept asking for more and more quantitative analyses, although the qualitative analysis was good and based on well known methodological sources. Qualitative manuscript is longer because of the many quotes, but the editor did not accept longer paper than the standard.

s

Inappropriate (Qualitative)

Criteria

Higher

StandardsNo “Boilerplate”

Problems with Editor or Reviewers

Barriers to Publishing(e.g., length of manuscript)

Themes in Comparing Qualitative to Quantitative Studies

Key Tensions in Publishing Qualitative Research

1. Break from existing theory, while also being embedded in it.

2. Write well: showing vs. telling (Golden-Biddle & Locke, 2007)

3. Be expansive (detailed & transparent), but also adhere to journal formats

Oval Pegs in Round Holes

• At an abstract level, the criteria for qualitative and quantitative research are the same (e.g., contribution, good writing, & clear and appropriate methods)

• However, how you fulfill these abstract goals differs and creates specific frustrations (e.g., showing vs. telling) – which are compounded by differing criteria in different journals, disciplines & geographic areas

Publishing Tips

• … for qualitative papers (but many apply to quantitative, too).

Managing Publishing Tensions

1. Make “Oval” Pegs Seem “Rounder”

a. Mimicking quantitative format: intro, methods, results (findings), discussion (G-B & L, 2007)

or do “closet” qualitative research (Sutton, 1997)

b. Mimicking quantitative criteria (Kirk & Miller, 1986; LeCompte &

Goetz, 1982; Lee, 1999; Lincoln & Guba, 1985 as cited in Marshall & Rossman, 1999; Miles & Huberman, 1994)

• Generalize to theory, not sample (Yin, 2003)

• Naturalistic generalization (e.g., sparrow).• Use mixed-methods

Managing Publishing Tensions

2. Make Round Holes Larger (More Oval Friendly)

a. Better trained qualitative evaluators at journals

b. Change in doctoral student education

Managing Publishing Tensions

3. Focus on the Process of Fittinga. Create open theoretical frames --

reviewing and critiquing the literature in such a way that you are able to delineate what has been written -- and at the same time, create space for representing where research has largely been silent (Pratt, 2008: 18)

– Examples: Barley (1986); Pratt, Rockmann & Kaufmann (2006)

Managing Publishing Tensions

“Filling in” these theoretical frames speaks to more general issue of presenting theory vs. data

– interweaving throughout (e.g., Golden-Biddle & Rao, 1997)

– “data sandwich” (e.g., Gephart, 1978)

– “open-faced data sandwich” (e.g., Pratt & Rosa, 2003)

– first- vs. second-order findings (Gioia & Chittipeddi, 1991)

Managing Publishing Tensions

b. Creating a hands-on exhibit: Showing & telling within page limits -- invite reader to interact with your data in a structured way

– Translation trade-offs: Balancing “contribution-to-length” and “doing violence to experience”

Managing Publishing Tensions

c. Create checklists– “boilerplates” may be both unwanted and not

possible

– “Best papers” in AMJ & ASQ had some commonalities on what they mentioned

Managing Publishing Tensions: Proposed Checklist

1. Why this study? (Elsbach & Kramer, 2003)

a. Why are qualitative methods appropriate?b. Am I building, elaborating, or testing theory?

2. Why study here?a. What is the nature of the context I am examining?b. What was my rationale for choosing this context? (e.g., extreme case – Dutton &

Dukerich, 1991 vs. more prototypical cases – Uzzi, 1997).

3. What am I studying and why?1. Am I sampling events (Isabella, 1990), cases (Zbaracki, 1998), people (Rafaeli &

Sutton, 1988) etc.?2. What is my sampling strategy?

a. How did I study these things?1. How did I analyze the data?2. How did I link data with theory? (see Figure 2, Corley & Gioa, 2004)

Improving your Chances for a R & R

• Proviso: most first drafts will have misaligned theory and data

1. Have several people read it through first (including quantitative researchers)

2. Establish yourself as knowledgeable – remember you are the analytical instrument.a. Know the literature(s)b. Give hook + rationale for why it is important to fill it!c. Thoughtful & transparent methods (and better to cite

methods sources than empirical examples)d. # of pages is not that impressive

Improving your Chances for a R & R

3. Review the methods checklist – especially purpose and sampling criteria

4. Show your data (use power & proof quotes)

5. Tell a story … that even non-academics can understand

6. Know your outlet & its mission/ focusa. Examine editorial boardb. Look at previously published articlesc. Know about page limits vs. length-to-contribution

ratiod. Don’t spend too much time “gaming” it

Closing Thoughts

• No paper – even a sole authored paper – is a solitary exercise. Use your academic ties, the editor, and the reviewer to make the paper the best it can be.

• Publication is a matter of preparation, persistence, & luck!

Any Questions?

Relevant Literature• Alvermann, D., O’Brien, D., Dillon, D. (1996). On writing qualitative research.

Reading Research Quarterly, 31(1): 114-120). • Amber, A., Adler, P.A., Adler, P., & Detzner, D. (1995). Understanding and

evaluating qualitative research, Journal of Marriage and the Family, 57 (November): 879-893.

• Crestwell, J. (1998). Data analysis and representation. Qualitative inquiry and research design: Choosing among 5 traditions. Thousand Oaks, CA: Sage.

• Daft, R. (1985). Why I recommend that your manuscript be rejected and what you can do about it. In L. Cummings and P. Frost (Eds.) Publishing in the organizational sciences (pp. 193 - 209). Homewood Il: Richard D. Irwin, Inc.

• Denzin, N. & Y. Lincoln (Eds.) (2000). Handbook of Qualitative Research, 2nd Edition. Thousand Oaks, CA: Sage.

• Eisenhardt, K. (1989). Building theories from case research. Academy of Management Review, 14, 532-550.

• Fine, G. & Elsbach, K. (2000). Ethnography and experiment in social psychological theory building: Tactics for integrating qualitative field data with quantitative lab data. Journal of Experimental Social Psychology, 35, 51-76.

Relevant Literature• Gephardt, R. (2004). From the editors: Qualitative research and the Academy of

Management Journal, Academy of Management Journal, 47(4): 454-462.• Glaser , B. G. and A. L. Strauss (1967). The Discovery of Grounded Theory:

Strategies for Qualitative Research. Chicago: Aldine.• Golden-Biddle, K, & Locke, K. (1993). Appealing work: An investigation of how

ethnographic texts convince. Organization Science, 4(4): 595-616.• Golden-Biddle, K. & Locke, K. (1997). Composing qualitative research. Thousand

Oaks, CA: Sage. • Howe, K. & Eisenhardt, M. (1990). Standards for qualitative (and quantitative)

research: a Prolegomenon. Educational Researcher, 19(4): 2-9.• King, N. (2000). Commentary – Making ourselves heard: The challenges facing

advocates of qualitative research in work and organizational psychology. European Journal of Work and Organizational Psychology, 9(4): 589-596.

• Lee, T. (1999). Using qualitative methods in organizational research. Thousand Oaks, CA: Sage.

• Lee, T., Mitchell, T. & Sablynski, C. (1999). Qualitative research in organizational and vocational behavior. Journal of Vocational Behavior, 55: 161-187.

Relevant Literature• Lincoln, Y. (1995). Emerging criteria for quality in qualitative and interpretive research.

Qualitative Inquiry, 1(3): 275-289.• Locke, K. (2001). Grounded theory in management research. Thousand Oaks, CA:

Sage.• Marshall, C & Rossman, G. (1989). Designing qualitative research. Newbury Park, CA:

Sage. • McGrath, J. (1982). Dilemmatics: The study of research choices and dilemmas. In

McGrath, J. E., Martin, J., & Kulka, R.A. (Eds.), Judgment calls in research. Beverly Hills, CA: Sage.

• Miles, M. (1979). Qualitative data as an attractive nuisance: The problem of analysis. Administrative Science Quarterly, 24: 590-601.

• Miles, M. & Huberman, A.M. (1994). Qualitative data analysis, 2nd Edition. Thousand Oaks, CA: Sage.

• Perrow, C. (1985). Journalling careers. In L. Cummings and P. Frost (Eds.) Publishing in the organizational sciences (pp. 220 - 230). Homewood IL: Richard D. Irwin, Inc.

• Prasad, P. (forthcoming). Crafting qualitative research: Working in the postpositivist traditions. ME Sharpe.

 

Relevant Literature• Pratt, M.G. (2000). Some thoughts on publishing qualitative research. Research

Methods Forum. http://www.aom.pace.edu/rmd/pratt_files/pratt.htm• Smith, J. & Deemer, D. (2000). The problem of criteria in the age of relativism. In N.

Denzin & Y. Lincoln (Eds.) Handbook of Qualitative Research, 2nd Edition (pp. 877-896). Thousand Oaks, CA: Sage.

• Spradley, J. (1979). The ethnographic interview. New York: Holt, Rinehart & Winston, Inc.

• Stake, R. (2000). Case studies. In N. Denzin & Y. Lincoln (Eds.) Handbook of Qualitative Research, 2nd Edition (pp. 435-454). Thousand Oaks, CA: Sage.

• Strauss, A. & Corbin, J. (1998). Basics of qualitative research, 2nd Edition. Thousand Oaks: Sage.

• Suddaby, R. 2006. From the editors: What grounded theory is not. Academy of Management Journal, 49(4): 633-642.

• Sutton, R. 1997. The virtues of closet qualitative research. Organization Science, 8(1): 97 – 106.

• Van Maanen, J (1979). The fact of fiction in organizational ethnography. Administrative Science Quarterly, 24: 539-550.

• Yin, R. (2003). Case study research 3rd Ed. Beverly Hills, CA: Sage Publications.

Qualitative Methods

• Grounded theory

• Case study

• Ethnography

• Phenomenology

• Biography / Life history

• Action research

• And more….

Translation “Trade-Offs”

Potential for least amount ofviolence to experience

Potential for most amount ofviolence to experience

“Pure” Thick Description

“Numbers”as Evidence

“Quotes” as Evidence “Pure”Theoretical Description

•Example: Perspective 3: Qual vs. quant• Adds significant length• Power and proof quotes• More fully “informants”

• Example: Perspective 1: Authors• Take up less space •Taken-for-granted meanings?• Managing cut-offs• Cueing quantitative criteria

Perspective 2:Evaluators