Social science doctoral students’ needs and preferencesfor written feedback

Gulfidan Can • Andrew Walker

� Springer Science+Business Media Dordrecht 2014

Abstract The purpose of this study was to investigate social science doctoral students’

preferences and needs with regard to written feedback on academic writing and to develop

a written feedback categorization. In an exploratory mixed methods approach, qualitative

data collected during interviews were used to form a questionnaire to collect quantitative

data in two research-intensive universities. The results based on 276 doctoral students’

responses provided a clear list of feedback types needed by doctoral students, including

comments addressing their main idea, argumentation, clarity, and information coverage.

Their preferences varied on issues of autonomy, criticism, and ambiguity, all critical

factors in the transition to independence expected during their doctoral education. The

resulting written feedback categorization encompasses three aspects: function, focus, and

presentation. The findings of this study have the potential to guide supervisors, feedback

providers, and doctoral students as well as inform further research, including instrument

development and written feedback content analyses.

Keywords Doctoral students � Feedback categorization � Academic writing � Social

science � Autonomy � Critical feedback

Introduction

Written feedback to assist doctoral students’ academic writing is a common method used in

several university support mechanisms, including writing groups, doctoral supervision, and

G. Can (&)Computer Education and Instructional Technology, Middle East Technical University, UniversitelerMah. Dumlupinar Blv. No: 1, 06800 Ankara, Turkeye-mail: gcan@metu.edu.tr

A. WalkerInstructional Technology and Learning Sciences, Utah State University, Logan, UT, USAe-mail: andy.walker@usu.edu

123

High EducDOI 10.1007/s10734-014-9713-5

writing classes (Aitchison 2009; Biklen and Casella 2007; Cryer 2000; Kamler and

Thomson 2006; Mullen 2001; Parker 2009; Wisker 2005). Academic support bolstered by

constructive and regular feedback is found among the most important criteria for choosing

a doctoral supervisor in the social sciences (Zhao et al. 2007). While this method has the

potential to motivate and support them to contribute to their disciplines and to improve

their academic writing skills, providing effective feedback requires significant time and

effort (Hattie and Timperley 2007). However, while doctoral students complain about

feedback inadequacies, even well-crafted written feedback can be rendered ineffective

when students discard it without consideration (Ferguson 2009).

One way to improve the usefulness of written feedback and reduce wasted effort is to

target student perceptions and preferences. Feedback, like other instructional methods,

needs to be designed with analysis of internal conditions including learners’ needs and

attitudes (Gagne et al. 1992). While the research on perceptions of written feedback is

limited in undergraduate education (Carless 2006), even fewer studies have considered

doctoral education. Further, the research to date has only explored needs and preferences

through qualitative methods and has been conducted in individual disciplines. Although

deep exploration of individual programs promotes a wider understanding of students’

perceptions and preferences of feedback in those fields, an investigation without disci-

plinary restrictions is necessary to accommodate increasing interdisciplinary research

interests (Boden et al. 2011; Lattuca 2001).

Available studies are further limited by inconsistent labeling of feedback types iden-

tified during analyses, making it difficult to compare results. As Reigeluth and Carr-

Chellman (2009) have indicated, instructional theorists strive for well-defined methods to

provide practitioners instructional guidelines for situational cases. They specified methods

with their kinds, parts, and criteria. Feedback as a component method (Reigeluth and

Keller 2009) is frequently used in variety of instructional approaches, and therefore the

kinds of feedback need to be categorized to help make quality decisions for its utilization.

In the case of doctoral students’ academic writing, there is a need for a unified categori-

zation of feedback which should encapsulate feedback types reported in the literature and

be based on triangulated data.

Therefore, this study investigated social science doctoral students’ perceived needs and

preferences with regard to written feedback using a mixed methods approach and a large

group of participants from a variety of social science programs. The study also aimed to

provide a research-based triangulated categorization of written feedback for doctoral

students’ academic writing.

Review of literature

Few studies have explored graduate or doctoral students’ needs and preferences for written

feedback and those studies were conducted in the individual disciplines of Education,

Nursing, Applied Linguistics, Social Work, and Sociology. Considering the function of

feedback, participants preferred feedback that gives individualized support, and con-

structive, guiding suggestions compared to feedback that instructed or provided evaluation

criteria (Caffarella and Barnett 2000; Eyres et al. 2001; Ferguson 2009; Kumar and Stracke

2007; Li and Seale 2007; Stracke and Kumar 2010). Feedback that encouraged dialogue,

provided opinions and alternative views, and encouraged reflection were well received as

well as justifications for critics (Eyres et al. 2001; Ferguson 2009; Kumar and Stracke

2007; Page-Adams et al. 1995; Stracke and Kumar 2010). Students also wanted feedback

High Educ

123

to learn the conventions of their discipline and to develop an individual voice (Eyres et al.

2001; Kumar and Stracke 2007; Tierney and Hallett 2010). Furthermore, they needed

encouragement to maintain self-confidence (Eyres et al. 2001; Hunt 2001; Kumar and

Stracke 2007; Stracke and Kumar 2010; Tierney and Hallett 2010).

Regarding the focus of feedback, students sought feedback on the central aspects of

their work, especially their main argument and conceptual framework (Eyres et al. 2001;

Ferguson 2009; Page-Adams et al. 1995), but revealed feeling vulnerable when receiving

critiques on these aspects and, as such, preferred editorial feedback after such aspects were

established (Eyres et al. 2001; Stracke and Kumar 2010). They needed assistance orga-

nizing their paper, developing content (Ferguson 2009), ensuring clarity, using appropriate

academic and disciplinary language (Eyres et al. 2001), and improving writing style (Page-

Adams et al. 1995).

Studies in the literature have concentrated more on the presentation aspect of feedback.

Students responded well to supportive, encouraging, and polite tones (Caffarella and

Barnett 2000; Eyres et al. 2001; Ferguson 2009; Li and Seale 2007) compared to overly

negative ones (Ferguson 2009; Hunt 2001; Stracke and Kumar 2010; Tierney and Hallett

2010). They wanted criticism cushioned by positivity as well as recognition and interest in

their work, presented in an egalitarian tone (Eyres et al. 2001; Ferguson 2009; Hunt 2001;

Kumar and Stracke 2007; Li and Seale 2007; Stracke and Kumar 2010). Regarding

delivery, they stressed the need for timely feedback and feedback for multiple drafts

(Caffarella and Barnett 2000; Crossouard and Pryor 2009; Eyres et al. 2001; Tierney and

Hallett 2010). Although they complained about illegible hand-written feedback, their

attitudes toward electronic feedback were not consistent across studies (Crossouard and

Pryor 2009; Ferguson 2009). They preferred to discuss written feedback orally (Caffarella

and Barnett 2000; Page-Adams et al. 1995; Tierney and Hallett 2010). Doctoral students

preferred detailed, specific, and extensive feedback, especially on central issues of their

paper (Crossouard and Pryor 2009; Eyres et al. 2001; Ferguson 2009; Strijbos et al. 2010).

They were frustrated both with the large quantity of criticism on first drafts (Tierney and

Hallett 2010) and with minimal feedback (Eyres et al. 2001; Ferguson 2009; Tierney and

Hallett 2010). Feedback clarity and references to the text were essential (Ferguson 2009;

Stracke and Kumar 2010). Although contradictory feedback caused frustration (Caffarella

and Barnett 2000), it was considered fruitful, as the situation required further exploration

(Stracke and Kumar 2010).

The available studies to date have only used qualitative research methods. Half of them

are based on personal experiences while the remaining had limited numbers of participants.

Furthermore, their results may not be generalizable or comparable, as they were conducted

in individual disciplines. Therefore, this study investigated social science doctoral stu-

dents’ perceived needs and preferences for written feedback based on both qualitative and

quantitative data gathered from students in a variety of social science programs.

Written feedback categorizations

Although a variety of written feedback categorizations have been established for academic

writing (e.g. Ellis 2009; Hattie and Timperley 2007), they are not based specifically on

graduate students’ writing. Individual studies on graduate students’ written feedback

experiences provided different lists of feedback types and terms to label feedback cate-

gories, making it difficult to compare their results. One of the few available categorizations

was developed by Hyatt (2005) who categorized the purpose of comments by analyzing

masters’ level assignments of Education students: phatic, developmental, structural,

High Educ

123

stylistic, content-related, methodological, and administrative. Kumar and Stracke (2007)

also analyzed written feedback comments on a draft of a PhD thesis in Applied Linguistics

and suggested three categories: (a) referential (editorial, organization, and content),

(b) directive (suggestion, question, instruction), and (c) expressive (praise, criticism,

opinion). Li and Seale (2007) categorized supervisor criticism by analyzing oral and

written feedback, progress reports, emails, and a diary: direct criticism, indirect criticism,

criticism with caution inserted, and criticism with guidance or support inserted.

These categorizations are not unified and based on only qualitative data. Therefore, the

present study aimed to provide a research-based and triangulated categorization of written

feedback for doctoral academic writing that also encapsulates the categories found in the

literature.

Methodology

This study is part of a larger study exploring relationships between factors in social science

doctoral students’ written feedback practices (Can and Walker 2011). The research

question of the present study is, ‘‘What are the perceived needs and preferences of social

science doctoral students for written feedback of their academic writing?’’ The study

further aimed to develop a categorization of written feedback for social science doctoral

students.

A sequential exploratory mixed methods research design was used with the focus on the

quantitative stage (Creswell 2009). The purpose of the qualitative stage was to discover a

variety of written feedback types to form a questionnaire and develop a preliminary cat-

egorization. The purpose of the quantitative stage was to describe doctoral students’

preferences and needs by collecting data from a large group of participants, triangulate, and

improve the preliminary categorization.

Qualitative stage

An interview guide for standardized open-ended interviews was used for collecting data.

Purposeful sampling method was used. As key informants, department heads identified

students in their programs with considerable academic writing experience. Among 28

recommendations, 15 students (eight women and seven men; age M = 35) from the fields

of Instructional Technology, Special Education and Rehabilitation, Elementary Education,

Secondary Education, Theory and Practice of Professional Communication, Family Con-

sumer and Human Development, Psychology, and Sociology volunteered to participate.

Three participants were non-native English speakers. The participants were at various

stages of their programs. The mean number of journal articles they had published was 3.87

(SD = 2.77). Using constant comparative analysis, a preliminary categorization of written

feedback was developed in this stage (Glaser 1965; Strauss and Corbin 2008). Trustwor-

thiness criteria for qualitative research were used as a guide throughout (Lincoln and Guba

1985).

An online questionnaire was designed (Dillman 2007) by quoting items from the

interview transcripts or the written feedback examples participants brought to interviews.

After content validity testing with experts and pilot testing with ten students, some items

were eliminated due to problems with clarity and questionnaire length. In the final ques-

tionnaire, the section related to preferences and needs for written feedback included 45

questions grouped under (a) delivery preferences, (b) general feedback preferences,

High Educ

123

(c) specific feedback preferences, and (d) perceived needs. The internal consistency esti-

mate of reliability was satisfactory with Cronbach’s alpha = 0.84.

Quantitative stage

Data were collected from two research universities with doctoral programs in the same

state. Out of 32 programs contacted, 30 participated in the study. Participants included 276

doctoral students: 160 at an urban university (58 %, participation rate = 21 %) and 116

from a state university (42 %, participation rate = 35 %). Doctoral students from the

School of Education comprised 47 % of all participants; Social Science and Humanities,

30 %; Health Related programs, 17 %; and Business and Economics, 6 %.

The participants included 165 women (60 %) and 110 men (40 %), ranging in age from

22 to 83. For 13 percent of participants (n = 37), English was not the native language;

international students comprised 16 percent of all graduate students in both universities.

Among 223 employed participants (81 %), 124 performed work that required academic

writing (45 %). The average number of journal articles was M = 2.43.

After descriptive data analysis of each questionnaire item, Exploratory Factor Analysis

(EFA) was conducted to find latent variables underlying the items and to improve the

preliminary categorization (Costello and Osborne 2005; Fabrigar et al. 1999). EFA was

conducted for 27 items: 16 about general preferences (agreement scale) and 11 about

perceived needs (frequency scale). The section about specific feedback preferences was not

included in EFA because they failed to form meaningful factors due to their distinctive-

ness. Since multivariate normality assumption was not severely violated (skewness

range = -0.82 and 0.37; kurtosis range = -1.09 and 0.99) (West et al. 1995) the Max-

imum Likelihood method was used with oblique rotation. Scree test (Cattell 1966) and

parallel analyses (O’Connor 2000) were used while determining the number of factors for

rotation.

Results

Qualitative data analysis results

Constant comparative analysis on interview transcripts suggested three main categories,

each with three sub-categories (see Table 1). The main categories were function, focus,

and presentation, similar to the categorization names suggested by Narciss (2008) for

interactive instruction. For function, which addressed the intended purpose of the feedback,

three sub-categories emerged based on its scope, which ranged from feedback with

immediate and practical purposes for a specific paper to feedback for the improvement of

long-term academic skills and a student’s enculturation and socialization into a discipline.

Under focus, which relates to the aspects of academic writing, three-sub-categories were

defined based on complexity. While feedback for mechanics, references, and formatting

focused on surface features of academic writing, feedback for overall writing required

consideration of the principles of effective writing from a more holistic perspective, such

as consistency and writing style. At the highest complexity level for focus, feedback for

arguments and information related to development of arguments and coverage of infor-

mation. For the presentation main category, relating to the manner in which the feedback

was given, the tone sub-category included the affective dimensions of presentation;

High Educ

123

delivery addressed issues like time, media, and format; and quality was related to detail,

adequacy of amount, relevance, accuracy, feasibility, importance, and clarity.

Quantitative data analysis results

For the question, ‘‘When writing academic papers, how frequently do you feel that you

need written feedback for the following aspects of your papers?’’, the students most fre-

quently rated arguments and justifications, the conclusion, clarity and understandability,

inclusion and exclusion of information, and the introduction (see Table 2). Students’ rat-

ings were lower for simpler and more mechanical aspects such as grammar and sentence

structure, formatting, and references and literature decisions.

Compared to 17 % of participants who particularly preferred handwritten feedback,

45 % wanted electronic feedback (see Table 3). Participants appreciated clear, specific,

detailed, and straightforward feedback that supported their revisions (see Table 4). Most

students did not prefer feedback reflecting feedback providers’ bias or personal preference.

About half of the students preferred critical feedback over positive feedback; however,

most of them (83 %) liked when critical feedback was presented in a positive way. About

41 % of students were open to critical feedback that required drastic changes, such as the

direction of the paper or the student’s writing style.

The students’ ratings on specific feedback preference items showed consistency with

their ratings to general feedback preferences (see Table 5). Only about half of the students

preferred the feedback that asked them to change the direction of their paper. Similarly,

ratings on other examples showed that most participants preferred clear, specific, and

relevant information or direction in feedback. They sought comments that gave suggestions

or examples for revisions. Although almost all comments were appreciated by the majority

of the students, general comments without specific suggestions were least preferred, such

as, ‘‘I have a hard time following this section’’.

To explore justification for the six items that rated less than 60 % in the preference

scales, follow-up analyses were conducted to investigate their relationships with partici-

pant demographics and academic information. Correlation analysis was conducted for

continuous data (age, year in the program, perceived overall writing ability, and number of

journal articles), and Multivariate Analysis of Variance (MANOVA) was conducted for

categorical data (gender, native language, university, and program).

Table 1 Preliminary categorization of written feedback

Main category Sub-category

Function a. To evaluate or improve the students’ current written piece

b. To improve general academic skills

c. To help students with enculturation/socialization into their specific discipline

Focus a. Mechanics, references, and formatting

b. Overall writinga

c. Arguments and informationa

Presentation a. Tone/manner

b. Delivery

c. Quality

a These sub-categories and their contents were modified after EFA testing in the quantitative phase

High Educ

123

Correlation analyses using the Bonferroni method (p = 0.002) resulted in no significant

relationships. The highest correlation was between overall perceived writing ability (5-

point scale) and the item ‘‘I don’t appreciate written feedback that tries to change my

writing style.’’(p = 0.02)

A one-way MANOVA conducted for each independent variable (gender, native lan-

guage, university, program) resulted in significant differences among groups on the

dependent variables. As shown in Table 6, Multivariate Eta Square (g2) based on Wilks’s

value and Pillai’s trace were both large effect sizes (Cohen 1988).

Individual ANOVAs were conducted as follow-up tests using the Bonferroni method

with p = 0.008. For gender, the ANOVA on Item 5 (‘‘I don’t appreciate receiving con-

tradictory feedback from different people’’) was significant, F(1, 273) = 10.79, p = 0.001.

Female doctoral students’ agreement rating for this item was significantly higher than that

of male doctoral students (see Table 7). For native language, Item 3 and Item 4 also

resulted in significant ANOVA results. For Item 3 (‘‘I appreciate written feedback that

revises or edits my paper for me’’) the result was F(1, 273) = 9.97, p = 0.002, and for

Table 2 Perceived need for written feedback (1-never, 2-seldom, 3-sometimes, 4-often)

Aspects of academic writing M (SD) Neverf (P)

Seldomf (P)

Sometimesf (P)

Oftenf (P)

Arguments and justifications in mypaper

3.16 (0.69) 3 (1 %) 38 (14 %) 146 (53 %) 89 (32 %)

Conclusion 3.11 (0.80) 7 (3 %) 53 (19 %) 119 (43 %) 97 (35 %)

Clarity and understandability of thestatements

3.11 (0.75) 3 (1 %) 55 (20 %) 127 (46 %) 90 (33 %)

Inclusion or exclusion of information 3.08 (0.73) 4 (1 %) 51 (19 %) 141 (51 %) 80 (29 %)

Introduction, purpose and significanceof the paper

3.05 (0.79) 7 (3 %) 58 (21 %) 124 (45 %) 86 (31 %)

Consistency in the overall paper 2.99 (0.74) 4 (1 %) 64 (23 %) 138 (50 %) 70 (25 %)

Logical order and organization ofinformation and ideas

2.90 (0.78) 10 (4 %) 68 (25 %) 137 (50 %) 60 (22 %)

Transition and flow between sentences,paragraphs, or sections

2.79 (0.82) 12 (4 %) 91 (33 %) 113 (41 %) 57 (21 %)

Grammar and sentence structure 2.52 (0.96) 35 (13 %) 118 (43 %) 67 (24 %) 56 (20 %)

Formatting (tables, figures, pagedesign, fitting APA style, givingcitations, etc.)

2.51 (0.90) 36 (13 %) 106 (38 %) 92 (33 %) 42 (15 %)

References and literature decisions 2.45 (0.82) 34 (12 %) 109 (40 %) 109 (40 %) 24 (9 %)

Table 3 Delivery preferences

Method of giving/sending paper Method of receiving feedback Total (%)

Electronically (%) Handwritten (%) No preference (%)

Electronically 41 4 9 54

Personally 1 9 1 11

No preference 4 4 27 35

Total 45 17 37 100

High Educ

123

Item 4 (‘‘I appreciate critical/negative written feedback more than positive written feed-

back’’) the result was F(1, 273) = 14.58, p = 0.00. The results indicate that compared to

native speakers, English as Second Language (ESL) doctoral students appreciate critical

feedback more than positive feedback, and they are more open to direct revisions. How-

ever, these results should be considered cautiously as the group sizes were not equal.

EFA of the quantitative data was conducted to reach a final categorization of written

feedback (see Table 8). Parallel analysis resulted in three factors for both perceived needs

and general preferences. Scree plots showed four factors for frequency scale and two

factors for the agreement scale (see Fig. 1). The most meaningful factors were formed

according to the parallel analysis results. The total variance explained was 52 % for

perceived needs and 28 % for general preferences. Two items were eliminated due to

cross-loadings, and six items were eliminated due to low communality scores. The cor-

relations between factors ranged between -0.44 and 0.26.

A comparison of EFA results to the preliminary categorizations developed in the

qualitative phase resulted in revisions to the Focus category. The preliminary categori-

zation included the sub-categories, Overall Writing and Arguments and Information.

However, EFA results showed that Overall Writing had an item about arguments

(‘‘Arguments and justifications in my paper,’’ loading: 0.50). While arguments are

important for the conclusion and introduction of a paper, these sections also require

interpretations of results and literature, summaries, and original contributions. To improve

the distinctiveness between the factors, Arguments and Information was re-named Main

Table 4 General feedback preferences (1-strongly disagree, 2-disagree, 3-agree, 4-strongly agree)

M (SD) %Agreed

Positive statements

I appreciate straightforward written feedback 3.51 (0.54) 98

I appreciate written feedback that gives me clear instructions for how to revise mypaper

3.49 (0.58) 97

I appreciate written feedback that directs me to other related resources 3.39 (0.54) 97

I appreciate detailed/specific comments more than overall/general comments 3.29 (0.72) 85

I appreciate written feedback about grammar 3.16 (0.65) 88

I appreciate written feedback which is given based on only what is on the paper,not based on my previous papers

3.16 (0.70) 83

I appreciate balanced positive and critical/negative written feedback 3.12 (0.65) 86

I appreciate written feedback in which negative things are said in a more positiveway

3.08 (0.66) 83

I appreciate written feedback that revises or edits my paper for me 2.71 (0.78) 59

I appreciate critical/negative written feedback more than positive written feedback 2.58 (0.69) 51

Negative statements

I don’t appreciate written feedback that is given because of a personal preference 3.02 (0.74) 77

I don’t appreciate suggestions in written feedback that are hard for me to use whilerevising my paper

2.80 (0.77) 69

I don’t appreciate marks without text in feedback (such as underlined sentences,circle around a word, question mark, etc.)

2.79 (0.82) 64

I don’t appreciate written feedback that tries to change my writing style 2.70 (0.74) 59

I don’t appreciate written feedback that tries to change the direction of my paper 2.69 (0.75) 59

I don’t appreciate receiving contradictory feedback from different people 2.58 (0.84) 50

High Educ

123

Idea and Contribution. Under presentation, the sub-categories were enhanced. Autonomy

was aligned with the Tone/Manner sub-category and Clarity and Parsimony under Quality.

Since there were not adequate items to represent Function in the questionnaire due to

eliminations during pilot testing, the preliminary categorization was accepted. The final

categorization is presented in Table 9.

Discussion and conclusion

The study resulted in an agreed upon list of written feedback types that social science

doctoral students need and prefer. However, the students’ opinions varied on feedback

Table 5 Specific feedback preferences (1-strongly disagree, 2-disagree, 3-agree, 4-strongly agree)

I appreciate written feedback comments similar to this: … M (SD) %Agreed

Here is an article that supports what you’re saying here 3.59 (0.49) 100

I think this sentence should be said much earlier. It is important 3.35 (0.54) 97

Maybe you need a table here, listing X with each column showing Y. Just an idea 3.35 (0.57) 97

This section is really strong 3.27 (0.59) 93

This argument is unsupported. You need to cite more references 3.25 (0.57) 95

Have you thought about adding one more section to your paper about X literature? 3.25 (0.53) 95

You’re on the right track, this is a well-organized paper 3.22 (0.56) 93

It is not clear how this paragraph addresses your research question. You need toshow links to the research question

3.21 (0.58) 92

Break this into smaller, more focused paragraphs 3.17 (0.51) 95

Explain why you’re focusing on these dimensions. Not clear to the reader 3.10 (0.55) 89

This section is a bit dense, with lots of details. Are they all necessary? 3.01 (0.61) 84

Check the APA manual for this citation 3.00 (0.75) 82

A bit of wavering focus from this paragraph to this paragraph. Check forconsistency throughout

2.91 (0.71) 75

I have a hard time following this section 2.70 (.76) 61

I don’t agree with this paragraph. I think, … 2.69 (.76) 66

I’d like you to go in a little different direction, like this… 2.47 (.74) 48

Table 6 MANOVA results

Independent variables Test of homogeneity g2

Gender (male, female) F(21, 201,066) = 0.56,p = 0.95

Wilks’s K = 0.93, F(6,268) = 3.32, p \ 0.01

0.07 (large)

Native language (English asnative language: yes, no)

F(21, 13,811) = 1.70,p = 0.02 (unequalgroup sizes)

Pillai’s trace = 0.14, F(6,268) = 7.43, p \ 0.01

0.14 (large)

University (university 1, 2) F(21, 225,413) = 0.79,p = 0.74

Wilks’s K = 0.98, F(6,269) = 0.86, p = 0.53

0.02

Program (business, education,social sciences and humanities,health related)

F(63, 12,695) = 0.95,p = 0.58

Wilks’s K = 0.92, F(18,753) = 1.26, p = 0.21

0.03

High Educ

123

requiring fundamental changes to the direction of their papers and writing styles as well as

on contradictory and critical feedback. As these issues are relevant to doctoral students’

transitions to autonomous scholars, this section emphasized both agreements and differ-

ences in students’ preferences.

Based on the final categorization and on students’ agreed needs and preferences, the

main practical suggestion for feedback providers is to consider all three dimensions of

written feedback (function, focus, presentation) in their feedback design. To have inner

consistency in feedback, decisions with function should lead decisions with focus and

presentation. Surveying individual students’ needs and preferences for each dimension

may increase the likelihood of use. If surveying is unfeasible, priority should be given to

main idea and contribution, followed by overall writing in early drafts. Comments on

mechanics, references, and formatting should be reserved for final drafts. Feedback pre-

sentation in terms of tone, delivery, and quality requires careful attention, as it has a strong

influence on students’ use of feedback. In general, feedback providers should give balanced

positive and negative comments with specific, clear, and supportive guidelines for

revisions.

Regarding differences in participants’ perceptions and preferences, a key finding of this

study exemplified the dilemma identified in the literature between the purpose of doctoral

education to prepare creative, independent scholars and the formal criteria of its evaluation.

In the present study, while the majority of participants needed feedback frequently on

central aspects of their papers, 60 % were resistant to receive feedback that tried to change

the direction of their work or their writing style. As doctoral students develop an identity

and gain membership in the academic community (Lave and Wenger 1991), they need to

become autonomous, creative scholars, while standards require their conformity to formal

academic norms (Lee 2008). This paradox of ‘‘training students to be independent in an

authoritarian social structure’’ (Egan 1989, p. 203) may explain ambiguity in students’

perceived autonomy in their writing. A reflection of this paradox manifests through styles

Table 7 Means and standard deviations of the dependent variables (Z scores)

Dependent variables Gender Native language

Male(N = 110)

Female(N = 165)

English(N = 239)

Not English(N = 36)

1. I don’t appreciate written feedbackthat tries to change my writing style

-0.10 (1.02) 0.08 (0.97) 0.02 (0.99) -0.13 (1.09)

2. I don’t appreciate written feedbackthat tries to change the direction ofmy paper

0.01 (1.00) 0.01 (0.99) -0.06 (1.00) 0.41 (0.96)

3. I appreciate written feedback thatrevises or edits my paper for me

-0.08 (1.01) 0.04 (0.99) -0.07 (0.98)* 0.49 (1.04)*

4. I appreciate critical/negative writtenfeedback more than positive writtenfeedback

0.16 (0.98) -0.12 (0.99) -0.09 (0.96)* 0.57 (1.07)*

5. I don’t appreciate receivingcontradictory feedback from differentpeople

-0.24 (1.01)* 0.16 (0.97)* -0.04 (0.99) 0.27 (1.02)

6. I’d like you to go in a little differentdirection, like this…

-0.02 (1.01) 0.01 (1.00) -0.03 (0.98) 0.23 (1.13)

* Significant difference, p \ 0.008, based on individual ANOVA follow-up tests

High Educ

123

of supervision (Lee 2008; Wisker 2005) as well as feedback, which can function in con-

gruence with varying supervision styles. The perceived ownership and originality of

research products and a supervisor’s power as gatekeeper to a discipline and its physical

and social resources (Lee 2008) may further explain this resistance. Similarly, 40 % did

not appreciate feedback that included direct edits. These students may seek acknowl-

edgement of originality and style during their transition to independence. Regarding this

dilemma, supervisors should effectively communicate the function of their feedback to

their students and design adaptive feedback to guide their transition process.

Perceived autonomy in writing may also be influenced by perceived evaluation criteria,

as standards for doctoral degree completion are ambiguous (Lovitts 2007; Winter et al.

2000). Furthermore, conflicting opinions can exist on epistemologies and methods in the

same discipline (Lattuca 2001), departments may have specific success criteria (Golde

2005), and faculty may have the authority to define qualifications (Egan 1989). The

preference of the large majority of students for clear, detailed, practical, and

Table 8 Factor analysis results

Factor Perceived needs Loadings R2

Overall writing Clarity and understandability of the statements 0.82 0.82

Logical order and organization of information and ideas 0.73

Consistency in the overall paper 0.65

Inclusion or exclusion of information 0.60

Transition and flow between sentences, paragraphs, or sections 0.60

Arguments and justifications in my paper 0.50

Formatting andreferences

Formatting (tables, figures, page design, fitting APA style,giving citations, etc.)

0.94 0.75

References and literature decisions 0.58

Main idea andcontribution

Conclusion 0.86 0.77

Introduction, purpose and significance of the paper 0.71

Factor General preferences Loadings R2

Autonomy I don’t appreciate written feedback that tries change mywriting style

0.95 0.77

I don’t appreciate written feedback that tries to changethe direction of my paper

0.61

Clarity I don’t appreciate receiving contradictory feedbackfrom different people

0.58 0.60

I don’t appreciate suggestions in written feedback thatare hard for me to use while revising my paper

0.55

I don’t appreciate written feedback that is given becauseof a personal preference

0.50

I don’t appreciate marks without text in feedback (suchas underlined sentences, circle around a word,question mark, etc.)

0.47

Parsimony I appreciate straightforward written feedback 0.54 0.50

I appreciate written feedback that gives me clearinstructions for how to revise my paper

0.52

I appreciate written feedback that directs me to otherrelated resources

0.45

High Educ

123

straightforward feedback (97 %) as well as resistance to feedback based on personal

preferences (77 %) suggests a reaction to ambiguity of formal and informal standards. The

results also showed students’ varying tolerance for ambiguity and conflict. Half of the

students did not appreciate contradictory feedback from different people, though such

contradictions in social sciences are likely to occur. The study reported significant dif-

ferences between males and females regarding contradictory feedback, similar to the

findings of Seagram et al. (1998). Although ambiguous and contradictory feedback com-

plicates meeting standards and completing degree requirements, tolerance for ambiguity is

an important characteristic of creative people and ensures an easier transition to inde-

pendent researcher (Lovitts 2008). Therefore, departments should clearly outline formal

standards for degree completion and unwritten criteria for successful dissertations in their

programs. They should also help students handle contradictions in academic endeavors,

providing examples of experiences of successful scholars who frequently encounter

ambiguity and contradiction.

The students’ attitudes toward critical feedback were also mixed, suggesting varying

degrees in tolerance for failures. Although previous research has mostly represented stu-

dents with negative attitudes toward critical feedback on central issues of their papers, the

present study represented the opinions of students who preferred critical feedback over

positive feedback as well (51 %). The literature supports that independence and creativity

require risk-taking and perseverance after failures, yet graduate students are unaccustomed

to frequent revisions (Lovitts 2005), and social science doctoral and graduate students may

Fig. 1 Scree plots of two scales

Table 9 The final categorization of written feedback for doctoral students’ academic writing

Main category Sub-category

Function a. To evaluate or improve the students’ current written piece

b. To improve general academic skills

c. To help students with enculturation/socialization into their specific discipline

Focus a. Mechanics, references, and formatting

b. Overall writing

c. Main idea and contribution

Presentation a. Tone/manner

b. Delivery

c. Quality

High Educ

123

be reluctant to revise ideas or incorporate major changes in their papers (Torrance et al.

1992). Furthermore, students who have problems handling criticism generally face a more

difficult transition to independence (Lovitts 2008). The varying reactions to criticism

suggest its relation to writing strategies of doctoral students, some of which are more

productive than others (Torrance et al. 1994). Supervisors should analyze students’ writing

strategies, diagnose their possible problems, and encourage them to be more open to

making revisions.

With the ambiguous criteria of doctoral education, establishing help-seeking behaviors

and quality communication with supervisors and other members of the academic com-

munity becomes critical. Requesting feedback on research and writing is a way of

enculturation and socialization to aid the understanding of evaluation criteria. Knowledge

production is quickly becoming more collaborative and interdisciplinary, requiring a re-

evaluation of the image of an independent social sciences and humanities researcher,

working alone (Golde 2005; Johnson et al. 2000). Lovitts (2008) reported that students who

transition more easily are active seekers of feedback. Unfortunately, students’ help-seeking

behaviors are negatively affected by expectations to be independent and have original ideas

(Egan 1989; Gardner 2008). Departmental microenvironments should be improved to

facilitate students’ perseverance after failures (Lovitts 2008), socialization with feedback,

and collaborative writing. Seminars on receiving and responding to critical feedback would

also improve tolerance for conflict and ambiguity.

Supporting student independence is not the same as supervisor negligence. An auton-

omous student still needs monitoring, guidance, and academic support (Johnson and On-

wuegbuzie 2004; Lovitts 2005; Overall et al. 2011). As research self-efficacy is related to

the combination of a supervisor’s task-related academic and autonomy support (Overall

et al. 2011), an appropriate balance is essential. While supporting autonomy, feedback on a

variety of dimensions can be provided, starting with major issues. Heath (2002) reported

that doctoral students were more satisfied with feedback on papers submitted early in their

candidature. Supervisors should therefore address critical issues in early drafts, when there

is still room for negotiation, especially in the case of online doctoral programs or ABD

students with limited face-to-face contact. Although the preference of the majority of

students for electronic feedback may be due to its timeliness, it deprives students of face-

to-face negotiation with the provider on major issues.

Even though data were collected from a large group of participants in two different

universities in the United States, both universities were in the same state. Therefore, the

results should be considered cautiously while guiding the practice in other countries and

contexts. Systematic attention is needed in further research to explore possible reasons for

varying preferences considering interrelated issues, including (a) autonomy and originality

with formal criteria, (b) help-seeking behaviors, (c) feedback characteristics, and (d) stu-

dent characteristics. Furthermore, with evolving online writing groups, interdisciplinary

collaborative research writing (Gannon-Leary et al. 2011), and an increasing number of

online doctoral programs, more systematic research is needed on electronic written feed-

back and communication methods with respect to doctoral students’ collaborative writing.

The categorization resulting from this study is robust, as it combines all findings from

the literature and previous categorizations (Hyatt 2005; Kumar and Stracke 2007; Li and

Seale 2007) and is based on triangulated data. The function category, however, should be

confirmed with quantitative research data. This category may be explored in relation to the

motivations of the faculty for supervising doctorate students (Hockey 1996) as well as

supervision approaches (Lee 2008; Wisker 2005). The resulting categorization of written

feedback can be used for further research utilizing content or discourse analysis of written

High Educ

123

feedback. It can also be used by supervisors as a framework while designing written

feedback.

References

Aitchison, C. (2009). Writing groups for doctoral education. Studies in Higher Education, 34(8), 905–916.Biklen, S. K., & Casella, R. (2007). A practical guide to the qualitative dissertation. New York: Teachers

College Press.Boden, D., Borrego, M., & Newswander, L. K. (2011). Student socialization in interdisciplinary doctoral

education. Higher Education, 62(6), 741–755.Caffarella, R. S., & Barnett, B. G. (2000). Teaching doctoral students to become scholarly writers: the

importance of giving and receiving critiques. Studies in Higher Education, 25(1), 39–52.Can, G., & Walker, A. (2011). A model for doctoral students’ perceptions and attitudes toward written

feedback for academic writing. Research in Higher Education, 52(5), 508–536.Carless, D. (2006). Differing perceptions in the feedback process. Studies in Higher Education, 31(2),

219–233.Cattell, R. (1966). The scree test for the number of factors. Multivariate and Behavioral Research, 1,

245–276.Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommen-

dations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10(7),1–9.

Creswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods approaches (3rd ed.).Thousand Oaks, CA: Sage.

Crossouard, B., & Pryor, J. (2009). Using email for formative assessment with professional doctoratestudents. Assessment & Evaluation in Higher Education, 34(4), 377–388.

Cryer, P. (2000). The research student’s guide to success (2nd ed.). Philadelphia, PA: Open UniversityPress.

Dillman, D. A. (2007). Mail and Internet surveys: The tailored design method (2nd ed.). Hoboken, NJ:Wiley.

Egan, J. M. (1989). Graduate school and the self: A theoretical view of some negative effects of professionalsocialization. Teaching Sociology, 17, 200–207.

Ellis, R. (2009). A typology of written corrective feedback types. ELT Journal, 63(2), 97–107.Eyres, S. J., Hatch, D. H., Turner, S. B., & West, M. (2001). Doctoral students’ responses to writing critique:

Messages for teachers. Journal of Nursing Education, 40(4), 149–155.Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of

exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299.Ferguson, P. (2009). Student perceptions of quality feedback in teacher education. Assessment & Evaluation

in Higher Education, 36(1), 51–62.Gagne, R. M., Briggs, L. J., & Wager, W. W. (1992). Principles of instructional design (4th ed.). Belmont,

CA: Holt, Rinehart and Winston.Gannon-Leary, P., Fontainha, E., & Bent, M. (2011). The loneliness of the long distance researcher. Library

Hi Tech, 29(3), 455–469.Gardner, S. K. (2008). ‘‘What’s too much and what’s too little?’’: The process of becoming an independent

researcher in doctoral education. The Journal of Higher Education, 79(3), 326–350.Glaser, B. (1965). The constant comparative method of qualitative analysis. Social Problems, 12(4),

436–445.Golde, C. M. (2005). The role of the department and discipline in doctoral student attrition: Lessons from

four departments. The Journal of Higher Education, 76(6), 669–700.Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112.Heath, T. (2002). A quantitative analysis of PhD students’ views of supervision. Higher Education Research

and Development, 21(1), 41–53.Hockey, J. (1996). Motives and meaning amongst PhD supervisors in the social sciences. British Journal of

Sociology of Education, 17(4), 489–506.Hunt, C. (2001). Climbing out of the void: moving from chaos to concepts in the presentation of a thesis.

Teaching in Higher Education, 6(3), 351–367.

High Educ

123

Hyatt, D. F. (2005). ‘Yes, a very good point!’: a critical genre analysis of a corpus of feedback commentarieson Master of Education assignments. Teaching in Higher Education, 10(3), 339–353.

Johnson, L., Lee, A., & Green, B. (2000). The PhD and the autonomous self: gender, rationality andpostgraduate pedagogy. Studies in Higher Education, 25(2), 135–147.

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose timehas come. Educational Researcher, 33(7), 14–26.

Kamler, B., & Thomson, P. (2006). Helping doctoral students write: Pedagogies for supervision. New York:Routledge.

Kumar, V., & Stracke, E. (2007). An analysis of written feedback on a PhD thesis. Teaching in HigherEducation, 12(4), 461–470.

Lattuca, L. R. (2001). Creating interdisciplinarity: Interdisciplinary research and teaching among collegeand university faculty. Nashville, TN: Vanderbilt University Press.

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. New York: Cam-bridge University Press.

Lee, A. (2008). How are doctoral students supervised? Concepts of doctoral research supervision. Studies inHigher Education, 33(3), 267–281.

Li, S., & Seale, C. (2007). Managing criticism in Ph.D. supervision: a qualitative case study. Studies inHigher Education, 32(4), 511–526.

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.Lovitts, B. E. (2005). Being a good course taker is not enough: A theoretical perspective on the transition to

independent research. Studies in Higher Education, 30(2), 137–154.Lovitts, B. E. (2007). Making the implicit explicit: Creating performance expectations for the dissertation.

VA: Stylus Publishing.Lovitts, B. E. (2008). The transition to independent research: Who makes it, who doesn’t and why. The

Journal of Higher Education, 79(3), 296–325.Mullen, C. A. (2001). The need for a curricular writing model for graduate students. Journal of Further and

Higher Education, 25(1), 117–126.Narciss, S. (2008). Feedback strategies for interactive learning tasks. In J. M. Spector, M. D. Merrill, J J Gv

Merrienboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications andtechnology (3rd ed., pp. 125–144). Mahaw, NJ: Lawrence Erlbaum Associates.

O’Connor, B. P. (2000). SPSS and SAS programs for determining the number of components using parallelanalysis and Velicer’s MAP test. Behavior Research Methods, Instrumentation, and Computers, 32(3),396–402.

Overall, N. C., Deane, K. L., & Peterson, E. R. (2011). Promoting doctoral students’ research self-efficacy:combining academic guidance with autonomy support. Higher Education Research & Development,30(6), 791–805.

Page-Adams, D., Cheng, L. C., Gogineni, A., & Shen, C. Y. (1995). Establishing a group to encouragewriting for publication among doctoral students. Journal of Social Work Education, 31(3), 402–407.

Parker, R. (2009). A learning community approach to doctoral education in the social sciences. Teaching inHigher Education, 14(1), 43–54.

Reigeluth, C. M., & Carr-Chellman, A. A. (2009). Situational principles of instruction. In C. M. Reigeluth &A. A. Carr-Chellman (Eds.), Instructional-design theories and models (Vol. III, pp. 57–68). New York,NY: Taylor & Francis.

Reigeluth, C. M., & Keller, J. B. (2009). Understanding instruction. In C. M. Reigeluth & A. A. Carr-Chellman (Eds.), Instructional-design theories and models (Vol. III, pp. 27–39). New York, NY:Taylor & Francis.

Seagram, B. C., Gould, J., & Pyke, S. W. (1998). An investigation of gender and other variables on time tocompletion of doctoral degrees. Research in Higher Education, 39(3), 319–335.

Stracke, E., & Kumar, V. (2010). Feedback and self-regulated learning: insights from supervisors’ and PhDexaminers’ reports. Reflective Practice, 11(1), 19–32.

Strauss, A., & Corbin, J. (2008). Basics of qualitative research (3rd ed.). Thousand Oaks, CA: Sage.Strijbos, J., Narciss, S., & Dunnebier, K. (2010). Peer feedback content and sender’s competence level in

academic writing revision tasks: Are they critical for feedback perceptions and efficiency? Learningand Instruction, 20, 291–303.

Tierney, W. H., & Hallett, R. E. (2010). In treatment: Writing beneath the surface. Qualitative Inquiry,16(8), 674–684.

Torrance, M. S., Thomas, G. V., & Robinson, E. J. (1992). The writing experiences of social-scienceresearch students. Studies in Higher Education, 17(2), 155–167.

Torrance, M. S., Thomas, G. V., & Robinson, E. J. (1994). The writing strategies of graduate researchstudents in the social-sciences. Higher Education, 27(3), 379–392.

High Educ

123

West, S. G., Finch, J. F., & Curran, P. J. (1995). Structural equation models with nonnormal variables:Problems and remedies. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues andapplications (pp. 56–75). Newbury Park, CA: Sage.

Winter, R., Griffiths, M., & Green, K. (2000). The ‘academic’ qualities of practice: what are the criteria for apractice-based PhD? Studies in Higher Education, 25(1), 25–37.

Wisker, G. (2005). The good supervisor: Supervising postgraduate and undergraduate research for doctoraltheses and dissertations. Basingstoke: Palgrave Macmillan.

Zhao, C. M., Golde, C. M., & McCormick, A. C. (2007). More than a signature: How advisor choice andadvisor behaviour affect doctoral student satisfaction. Journal of Further and Higher Education, 31(3),263–281.

High Educ

123