the [mis] perceptions of graduate student research skills ......mentor perceptions and student...
TRANSCRIPT
RUNNING HEAD: MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS
The [Mis] Perceptions of Graduate Student Research Skills
.
Hurst, M., Feldon, D., Maher, M., Gilmore, J., Timmerman, B.,
Strickland, D, and Stiegelmeyer, C.,
Paper presented at the Annual Meeting of the American Educational Research Association,
Denver, CO April 30 – May 4, 2010
The work reported in this paper is supported in part by a grant of the National Science
Foundation (NSF-0723686) to David Feldon, Briana Timmerman, Stephen Thompson, Jed
Lyons, and Michelle Maher under the REESE program. The views in this paper are those of the
authors and do not necessarily represent the views of the supporting funding agency
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 1
Abstract
The mismatch between the academic preparation and realistic career prospects of
graduate students has been highlighted frequently in education literature (Golde &
Dore, 2001; Nyquist et al., 1999; Nyquist & Woodford 2000). Minimal attention,
however, has been placed on the mismatch between how graduate students and
their mentors perceive the students‘ research skills and the students‘ actual
research skills (Leggett et al., 2004). Using a mixed-method approach, this study
investigates the level of agreement between faculty mentors and their students‘
self-perceptions of research skills, knowledge, and dispositions. Further, it
compares both faculty and student assessments against independent ratings of the
students‘ work products. Results indicate that faculty mentors and their students
address non-overlapping facets of developing knowledge and skills at least 80%
of the time. However, when they do address the same issues, they agree less than
two thirds of the time. Neither faculty mentors‘ nor students‘ assessments
predicted strength or weakness on performance-based measures of research
proficiency at levels better than chance for most categories. Further, despite the
common assumption that mentors play a large role in students‘ research skill
development, collaboration with mentors is cited less frequently by graduate
students compared to other academic and social experiences. Implications for
graduate mentorship as both a facet of educational practice and as an area of
research are discussed.
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 2
Problem and Review of Literature
The growing mismatch between the training and preparation of graduate students for
future careers has fueled an interest among researchers, leading to recommendations for
improving graduate student education (Golde & Dore, 2001, Nyquist et al., 1999; Nyquist &
Woodford 2000). Recommendations include enhanced mentoring practices and increased
opportunities for students to gain skills in areas beyond the scope of their direct research mentor
(Bieber & Worley, 2006; Gaff, 2002). Commonly, however, the faculty mentor role in graduate
student preparation is considered fundamental to the development of research skills (Carnegie
Initiative on the Doctorate, 2001; Mervis, 2000; Pearson & Brew, 2002).
Through guidance, collaboration, and role modeling, faculty mentorship is associated
with students‘ increased self-efficacy for research and research productivity (Paglis, Green, &
Bauer, 2006). Such mentoring relationships are commonly believed to be directly beneficial for
graduate students‘ performance and skill development (Austin, 2002; Clark et al., 2000; Green &
Bauer; 1995). A direct causal relationship is often assumed, despite frequent acknowledgements
of the importance of peer mentorship and support from postdoctoral researchers in a research lab
other than the faculty mentor (e.g., Delamont & Atkinson, 2001; Delamont, Atkinson, & Parry,
1997; Sweitzer, 2009). Further, it is assumed that the explanations provided by the mentor are
foundational to students‘ skill development, despite concerns about the accuracy of self-reported
problem-solving strategies provided by experts (Author, 2007, in press; Nickerson, 1999).
Students consistently report the impacts their mentors have on their motivation and their
decisions to continue in or withdraw from graduate programs (Lovitts & Nelson, 2000).
However, few studies characterize or empirically test the effects of interactions between mentors
and mentees that focus on the development and assessment of research skills.
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 3
The current study builds upon a previous effort (Author et al., 2009) that identified and
defined research skills and dispositions described by graduate students and valued by faculty
mentors. Results of the previous study revealed misalignments among what faculty mentors
value in a researcher and reported research skills of their students. Many attributes and ‗soft
skills‘ such as confidence, integrity, ability to learn from failures, and, of particular note to
successful interdisciplinary efforts that define many current research investigations, collaborative
skills, were deemed as important, and perhaps more so, than research skills (Author et al., 2009).
This study investigates agreement between student mentees‘ and their faculty mentors‘
perceptions of the students‘ developing research skills, knowledge, and relevant personal
dispositions in science, technology, engineering, and mathematics (STEM) disciplines. Further,
these perspectives are compared against independently scored measures of research skills.
Theoretical Perspective
A commonly held belief among researchers in academia is that mentoring relationships
are beneficial for graduate students‘ performance and skill development (Austin, 2002; Clark, et
al., 2000; Green & Bauer; 1995). Moreover, graduate student instruction in conducting research
typically relies heavily on individual faculty mentors to provide mentorship and teaching of
research skills (Carnegie Initiative on the Doctorate, 2001; Mervis, 2000). These cognitive
apprenticeships guided by a mentor provide a mechanism through which graduate students
acquire research expertise, with the ultimate goal of the student becoming an independent
researcher (Pearson & Brew, 2002). A question arises, however, regarding the accuracy with
which mentors can convey their knowledge and an accurate perception of student research skills
which will in turn benefit and increase the knowledge of research skills among graduate students.
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 4
It is commonly understood that the mentor-mentee relationship is reciprocal in nature.
Specifically, in order for a positive and productive relationship to exist, a student must receive
clear expectations regarding the development of research skills from his or her mentor and also
assume responsibility for the development of those skills (Carnegie Initiative on the Doctorate,
2001; Hon Kam, 1997; Mervis, 2000). While performing authentic academic tasks, students
must receive explicit mentorship to cultivate and clarify necessary skills needed to become a
professional researcher capable of adapting to new scholarly challenges (Mervis, 2000; Pearson
& Brew, 2002, Willison & O‘Regan, 2007). Evidence suggests that meaningful research comes
from the development of research skills among students and the guidance of research mentors
(Willison & O‘Regan, 2007). ―Articulating coaching and mentoring as tasks for supervisors to
engage in allows for clearer differentiation of responsibilities and elaboration of what is involved
in helping students learn the craft of research‖ (Pearson & Brew, 2002, p.140).
While research skill development is critical to graduate students‘ research and academic
experience, students themselves are often not fully aware of their own skills and abilities needed
to become a professional researcher (Lovitts, 2007) and often have incorrect assumptions about
their actual skills. Previous research indicates that faculty mentors typically have clear
expectations of their mentees regarding their attainment of knowledge, skills and dispositions
(Kardash, 2000). However, Leggett and colleagues (2004) report that these goals are seldom
articulated effectively to students by their mentors. There is also evidence of a misalignment
between students‘ expectations of the role of the mentor and their own roles as developing
researchers (Lagowski & Vick, 1995). In addition, mentors often hold perceptions of a student‘s
research skills that differ from that student‘s actual skills. These perceptions may be influenced
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 5
by personality characteristics and types of experiences with the graduate student, and can impede
judgment of accurate skill assessment (Johnson, 2002).
Some guidelines for identifying research skills are available. A review of research
literature on criteria used by professional referees identifies several common qualities of
manuscripts and grant proposals that indicate competent and worthwhile science has occurred
(Cicchetti, 1991; Marsh & Ball, 1989; Marsh & Bazeley, 1999; Marsh & Bell, 1981; Petty,
Fleming, & Fabrigar, 1999). Additionally, Mullins and Kiley (1998, p.4) identified a list of
―generic‖ or ―employable‖ research skills, including (a) good communication/ presentation
skills, b) the ability to use fundamental and technical knowledge to applied systems, (c)
intellectual property management skills, (d) skills in the scientific method and linkage to the
broad context, and (e) good laboratory practice. The use of research proposals and scientific
assessments are also accurate indicators of skills, but often mentors receive relatively minimal
format training on mentoring practices and rely on their observations and interactions with their
graduate students to assess research skills as opposed to an objective assessment (Johnson,
2002).
Research literature identifying alignment or mismatch between student and mentor
perceptions of key research skills in graduate school is relatively rare particularly studies that
draw student and mentor participants from across the disciplinary spectrum. This study
addresses this gap in the literature by posing the following questions:
1. What is the level of agreement between faculty mentors‘ and student mentees'
perceptions of the mentees‘ research skills, knowledge, and related dispositions?
2. What is the relationship between those perceptions and student mentees‘ actual skill
type and level as measured through a performance-based skills assessment?
3. What types of experiences do graduate students frequently cite as contributing to the
development of research skills?
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 6
Methods
This study uses a multi-method approach to identify the alignment or misalignment of
perceived versus actual research skills. To pursue study objectives, qualitative data were
gathered from doctoral students and their respective faculty mentors through their participation
in individual semi-structured interviews. All interviews were tape-recorded and transcribed to
identify emergent research skill levels. Quantitative data collected included a scientific
reasoning assessment (Lawson‘s Test of Scientific Reasoning) and written research proposals.
Research proposals were assessed via a modified form of a previously tested rubric for assessing
scientific reasoning skills in written laboratory reports (Timmerman, Johnson, & Payne, 2007).
The current study is part of a larger ongoing project investigating factors affecting the
development of science and engineering graduate student research skills. The primary thrust of
the larger project is to assess what happens to graduate students‘ own research skills when they
teach in a classroom in an inquiry based fashion. The premise is that mentoring and teaching of
research skills to others will cause reflection and meta-cognition and consequential accelerated
improvement in graduate students' own research skills.
Data Source
Participants used for this analysis were recruited from a large, public research university
in the Southeastern United States and were enrolled as full-time graduate students in research-
oriented programs. As part of their participation, they gave consent for researchers to conduct
confidential interviews with their faculty mentors. In all, the sample consisted of 58 students and
their respective faculty mentors (n=54).A review of mentors' faculty ranks revealed that the
distribution was roughly even across the assistant, associate, and full professor levels (15
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 7
assistant, 15 associate, and 24 full). Table 1 displays the programmatic affiliation and year for
each of the 58 students.
[INSERT TABLE 1 ABOUT HERE]
Data Collection
Students and their mentors each participated in one-on-one, semi-structured interviews
with researchers early in the fall and late in the spring of an academic year. During the
interviews, students were asked to describe their strengths, weaknesses, and overall identity as
researchers. Mentors were asked to describe their participating students‘ research strengths and
weaknesses. Faculty mentors were also asked to describe what they valued in a researcher in
general.
To gain an authentic, performance-based measure of research skill, student participants
submitted written research proposals for projects in their respective areas of interest early in the
fall term and revised them for resubmission in late spring of the same academic year.
Data Analysis
Interviews: To compile interview data, transcripts were analyzed using the constant
comparison approach (Glesne & Peshkin, 1992). Members of the research team continuously
compared statements regarding mentee skills, knowledge representations, and research
dispositions both within and across mentor and mentee groups to identify emergent themes.
Perceptions recorded in the interview transcripts were classified into categories and
evaluated as indicating strength or weakness in that category. Redundant codes and
positive/negative characterizations of the same traits or skills were consolidated (e.g., ―analytic
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 8
skills‖ and ―data analysis‖ codes were merged into a single code, ―data analysis‖; ―focused‖ and
―unfocused‖ were coded as ―focus– strength‖ and ―focus-weakness,‖ respectively).A full list of
codes, organized by classification as disposition, skill, and knowledge is provided in Table 2.
[INSERT TABLE 2 ABOUT HERE]
Research proposals. Research proposals were assessed using a modified form of a
previously validated rubric for evaluating scientific research skills in written laboratory reports
(Author et al., 2007; Author et al., in press) and has been applied in other work (e.g., Author et
al., in press). Proposals were received from participants electronically, checked for plagiarism
using SafeAssign™ and assigned to raters based on subject matter. Raters possessed graduate
degrees in relevant STEM disciplines and had attained an inter-rater reliability above r = 0.80
when coding student written responses. At least two raters scored each proposal, and any
discrepant scores were resolved by discussion until consensus was reached (cf.Johnson, Penny,
& Gordon, 2000; Johnson, Penny, Gordon, Shumate, & Fisher, 2005).
Integrated analysis. Following transcript coding and research proposal scoring, analysis
of the data occurred in three phases. First, frequency analyses were conducted for codes
identified from paired mentor and mentee interviews. Due to mentor scheduling constraints, 45
of 54 mentors were interviewed in the fall, and 40 of 54 were interviewed in the spring.
Instances in which both members of a mentor-mentee pair discussed the same attribute as either
a strength or weakness were identified and agreement or disagreement was noted.
Second, the student attributes as identified in both the mentor and mentee interviews were
crosstabulated with the individual elements of the mentee‘s rated proposal score to reflect the
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 9
frequency with which interview-identified strengths and weaknesses aligned with rubric scores
above or below the sample mean.
Third, the extent to which interview data predicted performance data was assessed using
the Fisher Exact test. Frequencies within the 2 x 2 contingency table (interview-based
strength/weakness x rubric element scores above/below sample mean) were assessed against a
distribution with equal counts in all cells. Thus, a significant p-value indicated that interview-
based assessments were significantly better or worse than chance at predicting performance on a
given facet of the proposal. For the purposes of this study, a nontraditional significance
threshold value of p < 0.5 was used. The generally low frequency of any given code in the
dataset limits statistical power, which would otherwise generate an unacceptably high likelihood
of Type II error rates (Schunn & Anderson, 1999). This is particularly important given the weak
evidence for experts' perceptions of learners' abilities as accurate in the literature (e.g., Hinds,
1999).
All analyses were performed separately for the fall and spring data sets to determine
stability and the extent to which additional time influenced the alignment of assessments.
Results
Study results are presented in the order of the analytic steps detailed above. Together,
they represent a systematic investigation into the degree of alignment between faculty mentors'
and their graduate mentees' assessments of the students' research skill development. Further,
they reflect the degree to which either group can successfully predict the mentees' performance
on written research proposals evaluated against a rubric. The first section presents the results of
paired mentor and mentee response analysis conducted to identify alignment between mentor and
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 10
mentee in terms of perceived strengths and weaknesses. The second section compares the
responses of faculty mentors and their student mentees to performance-based measures of their
emerging skills as assessed by multiple blind reviewers. The third section results from the
application of a measure of stability are presented as fall and spring data are compared. Lastly,
the final section identifies types of experiences graduate students cite as impacting their research
skill development.
Comparison of Code Alignment in Mentor-Mentee Responses
Analysis of mentor-mentee response alignment indicated that most mentor-mentee
interview responses did not overlap. In the fall, only 9 out of 45 (19%) mentor-mentee responses
aligned. In the spring, only 4 out of 40 (10%) aligned.
In 8 of the 13 pairs (62%), when a disposition, skill, or conceptual knowledge was
observed independently by both mentors and mentees, the pair agreed on whether the mentioned
trait constituted a strength or a weakness of the mentee. For example, in one case of agreement
between a student and his mentor, the student indicated that a research strength of his was
implementing experiments in the field:
I think the implementation phase (is a strength), plus I enjoy that. So I would design an
experiment and I would be sitting here and planning out what I would do. I‘m pretty
excited about that. Of course I have to collaborate with a lot of different people to make
sure what I have here will actually work. What I find that I‘m very good at is having it
written down on paper and then going out into the field and actually execute what I have
and what I‘m supposed to do. To me, my strongest point is actually the implementation
of the experimental design that I prepare.
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 11
The student‘s advisor agreed with the student‘s assessment of research strengths, stating:
He is not afraid to tackle anything in the field at all. He has gone down to Belize with
just a boxful of equipment for three months and it works, and that is not something that
everyone can do. He is extremely independent, and he is not afraid to make contacts with
other people, and to seek out collaborators and to take on personal responsibility for
anything in his path, so he is really an independent operator.
In five pairs, however, a closer look revealed discordant understandings of mentee
development as a researcher. For example, a student described his strengths as a researcher by
stating:
I‘m good at asking questions when I don‘t know the answers. Pretty practical I think.
Pretty big overall, I have a good idea of what I want to do. I‘m not lost. I‘m pretty self-
motivated. I think I have the ability to be able to do the research on my own. I just need
someone to answer the questions that I occasionally have.
However, his advisor‘s comments countered this student‘s self-assessment:
I think he is a classic case of he has been trained by books to answer questions that are
based on those books and when you give him a more amorphous problem, something
more tiger-headed, it is a real challenge. He becomes very defensive. And that goes to
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 12
shut him down, I think. And he is unable to open his mind at that point to the problem at
hand.
In another misaligned case, a student stated, ―I don‘t feel really comfortable with me
being pretty much on my own right now. But right now I really describe myself as a weak
researcher who is striving to become a stronger one.‖ In contrast, her advisor observed that ―She
understands fully the concepts that she does need to design and complete a study. So, I would
say she is strong.‖
A Test of the Predictive Power of Perceptions
To determine the predictive power of mentors' and mentees' perceptions of mentee
research skills, the identified strengths and weaknesses reported by the faculty and students were
crosstabulated with elements of each mentee‘s rated proposal scores (see Table 3).
[INSERT TABLE 3 ABOUT HERE]
When compared to the sample population's mean scores, mentors‘ perceptions of
mentees‘ research skills were not always accurate predictors of their mentee‘s demonstrated
abilities. During the fall, 44% (47 of 108) of mentors‘ assessments of their mentees‘ research
skills were inconsistent with the mentees‘ norm-referenced performance. These misalignments
reflected mentors‘ perceptions of strength that corresponded to below average performance (n =
24) and their perceptions of weakness that corresponded to above average performance (n = 23)
on the written research proposals. Alignment between mentors‘ perceptions and mentees'
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 13
performance diminished in the spring, with mentors failing to predict performance 64% (34 of
53) of the time, with 19 instances of perceived strengths aligning with below average
performance and 15 instances of perceived weakness aligning with above average performance
(see Table 4).
[INSERT TABLE 4 ABOUT HERE]
Mentees' self-assessments were similarly inaccurate (see Table 5). In the fall semester,
mentees‘ perceptions of their research skills did not accurately predict their norm-referenced
performance 56% of the time (56 of 100 instances). Perceived strengths corresponded with
below average performance in 22 cases, and perceived weaknesses corresponded with above
average performance in 34 cases. Mentees' perception-skill alignment fell slightly to 46% (23 of
50 instances) in the spring. Mentees were more likely to underestimate their skills in the fall
(n=34 underestimations versus n=22 overestimations) and more likely to overestimate in the
spring (n=19 overestimations versus n=4 underestimations). Sixty percent of the time mentees
cited a research weakness, but when compared to actual research skills, they scored above the
population mean for those areas.
[INSERT TABLE 5ABOUT HERE]
Finally, when graduate students were asked about the types of experiences that
contributed to the development or increase of their research skills, approximately 40% cited
collaboration with their mentor. A larger majority cited experiences that did not involve a
mentor, such as trial and error in a lab, reviewing literature and observing other researchers or
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 14
graduate students in the lab, and collaborating with their peers. These results, coupled with the
mismatch between perception and actual skill level indicates a need for increased collaboration
between the student and mentor.
[INSERT TABLE 6 ABOUT HERE]
Conclusions
Faculty mentors are thought to play a key role in defining expectations for research skill
acquisition (Kardash, 2000), and to serve as role models for their graduate mentees (Paglis et al.,
2006). However, findings from the current study suggest that mentors‘ insights do not reliably
map onto their mentees‘ demonstrated abilities. Interpretation of the data indicates that mentors
often demonstrate a fundamental lack of understanding of their mentees‘ progress as developing
researchers, particularly in the critical first years of graduate education. Their perceptions of
their mentees rarely match their mentees‘ own perception of themselves as researchers. More
tellingly, when mentor perceptions are matched against an empirical measure of mentee research
skills, roughly one of every two mentors‘ perceptions are incorrect. This finding contradicts the
standard perceptions of mentorship‘s effects on the development of research skills.
Research in the field of science education suggests that students whose learning is not
closely monitored are more likely to acquire misconceptions that impact both their theoretical
understanding of key phenomena and their scientific problem-solving skills (Lohman, 1986;
Schwartz & Bransford, 1998). Unfortunately, once such misconceptions are acquired, they are
almost impossible to change without substantial effort being invested on both the part of the
student and a willing mentor (Bargh & Ferguson, 2000; Chinn & Brewer, 1993; Thorley &
Stofflet, 1996). It is therefore possible that those who "just don't get it" are held back by
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 15
incorrect or maladaptive knowledge that they acquired during the benign neglect of the first
couple years of their graduate training.
Increased interaction and collaboration between the graduate student and research mentor
may close the gap of perception versus actual skills level and provide the mentor with increased
opportunity to see the student engage in objective tasks in which the mentor can determine a
more accurate research skill level and provide the student with guidance to increase the research
skills
Significance
The current study is of educational significance in many ways. To create an effective
educational setting that addresses the cognitive and social demands of learning, the mentor must
be aware of the research skills a graduate student actually possesses. The study also has
implications for graduate mentor guidance. As doctoral education is characterized by cognitive
apprenticeship with a faculty mentor (Florence & Yore, 2004; Golde & Dore, 2001), the quality
of doctoral mentoring is of critical importance to the successful scholarly doctoral student
development.
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 16
References
Austin, A. E. (2002). Preparing the next generation of faculty: Graduate school as socialization
to the academic career. The Journal of Higher Education, 73(1), 94-122.
Bargh, J. A. & Ferguson, M. J. (2000). Beyond behaviorism: On the automaticity of higher
mental processes. Psychological Bulletin, 126(6), 925-945.
Bieber, J. & Worley, L. (2006). Conceptualizing the academic life: graduate student‘s
perspectives. Journal of Higher Education, 77 (6), 1009-1035.
Carnegie Initiative on the Doctorate. (2001). Overview of Doctoral Educational Studies and
Reports: 1990 - Present. Stanford, CA: The Carnegie Foundation for the Advancement of
Teaching.
Chinn, C. A., & Brewer, W. F. (1993). The role of anomalous data in knowledge acquisition: A
theoretical framework and implications for science instruction. Review of Educational
Research, 63, 1-49.
Cicchetti, D. V. (1991). The reliability of peer review for manuscript and grant submissions: A
cross-disciplinary investigation. Behavioral and Brain Sciences, 14, 119-135.
Clark, R., Harden, S., & Johnson, W. (2000). Mentor relationships in clinical psychology
doctoral training: Results of a national survey. Teaching of Psychology, 27, 262–268.
Cox, M. & Andriot, A. (2009). Mentor and undergraduate student comparisons of students‘
research skills. Journal of STEM Education, 10 (1 & 2), 31-39.
Davis, G., & Fiske, P. (2001). The 2000 national doctoral program survey. University of
Missouri-Columbia: National Association of Graduate-Professional Students.
Author. (2007). Implications of research on expertise for curriculum and pedagogy.
Educational Psychology Review, 19(2), 91-110.
Author. (in press). Do expert researchers tell it like it is? A microgenetic analysis of research
strategies and self-report accuracy. Instructional Science.
Florence, M., & Yore, L. (2004). Learning to write like a scientist: Coauthoring as an
enculturation task. Journal of Research in Science Teaching, 41, 637-668
Gaff, J. G. (2002). The disconnect between graduate education and the realities of faculty work:
A review of recent research. Liberal Education, 88(3), 6-13.
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 17
Golde, C. & Dore, T. (2001). At cross purposes: What the experiences of doctoral students
reveal about doctoral education (www.phd-survey.org). Philadelphia, PA: A report
prepared for The Pew Charitable Trusts.
Glesne, C. & Peshkin, A. (1992). Becoming qualitative researchers: An introduction. New York:
Longman.
Green, S. G., & Bauer, T. N. (1995). Supervisory mentoring by mentors: Relationships
with doctoral student potential, productivity, and commitment. Personnel Psychology,
48(3), 537–562.
Hinds, P. J. (1999). The curse of expertise: The effects of expertise and debiasing methods on
predictions of novice performance. Journal of Experimental Psychology: Applied, 5(2),
205-221.
Hodkinson, P. (2004) Research as a form of work: Expertise, community and methodological
objectivity. British Educational Research Journal, 30(1), 9-26.
Hon Kam, B. (1997). Style and quality in research supervision: the supervisor dependency
factor. Higher Education, 34, 81-103.
Johnson, R. L., Penny, J., Gordon, B., Shumate, S. R., & Fisher, S. P. (2005). Resolving score
differences in the rating of writing samples: Does discussion improve the accuracy of
scores? Language Assessment Quarterly: An International Journal, 2(2), 117-146.
Johnson, W. (2002). The intentional mentor: strategies and guidelines for the practice of
mentoring. Professional Psychology: Research and Practice, 33 (1), 88-96.
Kardash, C. (2000). Evaluation of an undergraduate research experiences: perceptions of
undergraduate interns and their faculty mentors. Journal of Educational Psychology, 92
(1), 191-201.
Lagowski, J. & Vick, J. (1995). Faculty as mentors. New Direction for Teaching and Learning,
(62), 79-85.
Leggett, M., Kinnear, A., Boyce, M. & Bennett, I. (2004). Student and staff perceptions of the
importance of generic skills in science. Higher Education Research & Development,
23(3), 295-312.
Lohman, D. F. (1986). Predicting mathemathantic effects in the teaching of higher-order
thinking skills. Educational Psychologist, 21(3), 191-208.
Lovitts, B. E. (2005). Being a good course-taker is not enough: A theoretical perspective on the
transition to independent research. Studies in Higher Education, 30(2), 137-154.
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 18
Lovitts, B. E. (2007). Making the implicit explicit: Creating performance expectations for the
dissertation. Sterling, VA: Stylus Publishing.
Lovitts, B., & Nelson, C. (2000). The hidden crisis in graduate education: Attrition from
Ph.D programs. Academe, 86, 44-51.
Manathunga, C. (2005). The development of research supervision: ‖Turning the light on a
private space‖. The International Journal for Academic Development, 10(1), 17-30.
Mervis, J. (2000). Graduate educators struggle to grade themselves. Science, 287(5453), 568-
570.
Mullins, G. & Kiley, M. (1998) Quality in postgraduate research: the changing agenda, in:
M. Kiley & G. Mullins (Eds) Quality in Postgraduate Research: managing the new agenda.
Proceedings of the 1998 Quality in Postgraduate Research Conference, pp. 1–13 (Adelaide,
Advisory Centre for University Education, University of Adelaide).
Nickerson, R. S. (1999). How we know—and sometimes misjudge—what others know:
Imputing one‘s own knowledge to others. Psychological Bulletin, 125(6), 737-159.
Nyquist J., Manning L., Wulff, D. H., Austin, A. E., Sprague, J., Fraser, P. K., Calcagono, C., &
Woodford, B. (1999). On the road to becoming a professor: The graduate student
experience. Change, 31, 18–27.
Nyquist, J., & Woodford, B. (2000). Re-envisioning the PhD: What concerns do we have?
Seattle, WA: University of Washington, Center for Instructional Development and
Research.
Paglis, L., Green, S., & Bauer, T. (2006). Does mentor mentoring add value? A longitudinal
study of mentoring and doctoral student outcomes. Research in Higher Education, 47(4),
451-476.
Pearson, M. & Brew, A. (2002). Research training and supervision. Development Studies in
Higher Education, 27(2), 135-150.
Petty, R. E., Fleming, M. A., & Fabrigar, L. R. (1999). The Review Process at PSPB: Correlates
of Interreviewer Agreement and Manuscript Acceptance. Personality and Social Psychology
Bulletin, 25(2), 188-203.
Schwartz, D., & Bransford, J. D. (1998). A time for telling. Cognition and Instruction, 16(4),
475-522.
Sweitzer, V. (2009). Towards a theory of doctoral student professional identity development: a
developmental networks approach. The Journal of Higher Education, 80 (1), 1-33.
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 19
Thorley, N., & Stofflet, R. (1996). Representation of the conceptual change model in science
teacher education. Science Education, 80, 317-339.
Author et al. (2007). Development of a universal rubric for assessing students' science inquiry
skills. Paper presented at the National Association of Research in Science Teaching.
Author et al. (in press). Development of a ‗universal‘ rubric for assessing students‘ scientific
reasoning skills using scientific writing. Assessment and Evaluation in Higher
Education.
Willison, J. & O‘Regan, K. (2007). Commonly known, commonly not known, totally unframed:
a framework for students becoming researchers. Higher Education Research &
Development, 26(4), 393-409.
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 20
Table 1
Graduate Student Participants by Programmatic Affiliation and Year in Program
Discipline 1st Year 2nd Year 3rd Year 4th Year or
Beyond Total
Education 0 1 2 0 3
Engineering 7 5 2 3 17
Math 5 0 1 1 7
Science 25 3 2 1 31
Total 37 9 7 5 58
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 21
Table 2.
Codes Used to Designate Disposition, Skill, and Knowledge
Disposition Skill Knowledge
Aggressive Communication Big Picture/Context
Collaborative Computer skills Conceptual Knowledge
Confident Data Analysis Terminology
Creative Defining Research Problem Theoretical Framework
Critical Thinking Field work
Enthusiastic Literature Review
Experienced Math/Statistics Skills
Focused Operating Lab Equipment
Hard Working Research Design
Independent
Inquisitive
Intuitive
Meticulous
Organized/Time Management
Persistent/Motivated
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 22
Table 3.
Comparison of Proposal Rubric Categories to Mentor-Mentee Strengths and Weaknesses in Disposition, Skill and Knowledge
Perceived Strengths and Weaknesses
Analysis
Skills
Big
Picture
Conceptual
Knowledge
Critical
Thinking
Data
Analysis
Defining
Problem
Literature
Review
Math/Stat
Skills
Research
Methods
Proposal Rating Categories
Introduction/Context Writer provides a clear sense of what is
known and what gaps exist in our
knowledge. Background information is
accurate, relevant and provides a clear
rationale for the objectives.
X X
X
Hypothesis Testable Research questions and expected findings
are clearly stated and plausible/testable
using proposed methods. Note For
experimental studies or when otherwise
appropriate, specific hypotheses should be
stated. As appropriate, plausible alternative
explanations / mechanisms / hypotheses
should be explained and the proposed
research design will allow investigators to
distinguish among them.
X
X
X
X
Methods-Validity/Reliability Appropriate controls and/or mechanisms to
ensure validity and reliability are present
and explained. Degree of replication /
sample size is explained and appropriate
for the research area.
X
X
X
Methods-Exp. Design Data collection plan / Experimental design
is likely to produce salient and fruitful
X
X
X
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 23
results (i.e. addresses the research
objectives posed).
Results-Data Selection Data produced by the research will be
comprehensive, informative, accurate and
relevant to the questions / hypotheses
posed.
If actual data are presented, use these
criteria Data produced by the research will
be comprehensive, accurate and relevant.
X
X
Results-Data Analysis Proposed interpretive framework and/or
statistical methods are appropriate for
research objectives. Rationale for the
choice of methods is explained clearly.
Expected evidence for data‘s validity,
reliability, and or statistical significance
(as appropriate to the proposed study) are
indicated.
X X X
X
Discussions/Conclusion Conclusion could be clearly and logically
drawn from predicted data. A logical chain
of reasoning from hypothesis/ design
element to predicted data to conclusions
are clearly and persuasively explained.
X
X X X X X
Broader Impacts How well does the activity advance
discovery and understanding while
promoting teaching, training, and learning?
How well does the proposed activity
broaden the participation of
underrepresented groups? What may be the
benefits of the proposed activity to
society?
X X
X
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 24
Limitations/Significance of
Findings Limitations of findings and remaining
questions to be answered in relation to the
phenomenon of interest are discussed.
Alternative explanations of the predicted
data are considered and weighted against
conclusions. How this study relates to
other knowledge in the field is clearly
discussed.
X
X
Primary Literature Use Relevant literature is reasonably complete
and present in both the
Introduction/context and Discussion
sections. Use of the literature demonstrates
the intellectual merit of the proposed
research and specifies how it relates to
other work in the field. Citations follow an
accepted format for the field and are
accurate. (Please indicate the citation style
you are using in the literature cited
section.) .
X
X
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 25
Table 4.
Comparison of Proposal Rubric Categories to Mentor-Mentee Strengths and Weaknesses in Disposition, Skill and Knowledge
Fall
Semester Introduction: Context Testable Hypothesis Validity/Reliability Experimental Design Data Selection Data Presentation
Faculty Student Faculty Student Faculty Student Faculty Student Faculty Student Faculty Student
CountFisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
Exact
Literature
Review
Strength3 0.999 4 0.429* 3 0.429* 3 0.464* 1 0.999 3 0.464* 1 0.486* 3 0.999 1 0.999 2 0.999 2 0.399* 2 0.999 Above Mean
0 2 0 3 1 3 2 3 2 4 0 4 At Mean or Below
Weakness3 0 2 0 1 0 3 1 2 1 1 0 Above Mean
1 2 2 2 3 2 1 1 2 1 3 2 At Mean or Below
Analytical
Skills
Strength1 0.999 0 0.999 0 0.999 1 0.999 1 0.999 1 0.999 Above Mean
1 2 2 1 1 1 At Mean or Below
Weakness0 0 0 0 0 0 Above Mean
1 1 1 1 1 1 At Mean or Below
Conceptual
Knowledge
Strength1 0.999 0 0.999 1 0.999 1 0.375* 1 0.429* 0 0.999 1 0.999 0 0.999 0 0.999 0 0.999 0 0.999 0 0.999 Above Mean
0 1 0 0 0 1 0 1 1 1 1 1 At Mean or Below
Weakness3 3 2 2 1 1 3 4 2 2 1 2 Above Mean
1 4 2 5 3 6 1 3 2 5 3 5 At Mean or Below
Critical
Thinking
Strength1 0.999 1 0.999 1 0.999 1 0.999 0 0.999 2 0.333* Above Mean
1 1 1 1 2 0 At Mean or Below
Weakness0 0 0 0 0 0 Above Mean
1 1 1 1 1 1 At Mean or Below
Big Picture
Strength2 0.999 1 0.999 2 0.999 2 0.999 2 0.999 1 0.999 2 0.999 1 0.999 2 0.999 2 0.999 1 0.999 2 0.999 Above Mean
0 2 0 1 0 2 0 2 0 1 1 1 At Mean or Below
Weakness2 3 2 3 1 3 2 3 1 2 1 3 Above Mean
0 2 0 2 1 2 0 2 1 3 1 2 At Mean or Below
Research
Methods
Strength1 0.999 1 0.999 1 0.999 1 0.999 1 0.999 0 0.333* Above Mean
1 1 0 1 1 2 At Mean or Below
Weakness1 1 0 1 0 1 Above Mean
0 0 1 0 1 0 At Mean or Below
Defining
the Problem
Strength0 0.999 1 0.999 1 0.999 1 0.999 1 0.999 0 0.999 Above Mean
1 0 0 0 0 1 At Mean or Below
Weakness0 0 1 0 0 0 Above Mean
1 1 0 1 1 1 At Mean or Below
Data
Interpret.
Strength1 0.999 0 0.333* 0 0.333* 1 0.999 0 0.333* 0 0.333*Above Mean
0 1 1 0 1 1 At Mean or Below
Weakness2 2 2 2 2 2 Above Mean
0 0 0 0 0 0 At Mean or Below
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 26
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 27
Table 4. Continued
Comparison of Proposal Rubric Categories to Mentor-Mentee Strengths and Weaknesses in Disposition, Skill and Knowledge-Fall Semester
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 28
Fall
Semester Data Analysis Valid Conclusions Broader Implications Limitations/Significance Primary Literature Total
Faculty Student Faculty Student Faculty Student Faculty Student Faculty Student Faculty Student
CountFisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
Exact
Literature
Review
Strength1 0.999 1 0.999 1 0.999 2 0.999 1 0.999 2 0.429* 1 0.333* 4 0.429* 3 0.999 2 0.429* 2 0.486* 2 0.999 Above Mean
1 5 2 4 2 4 1 2 0 4 1 4 At Mean or Below
Weakness2 0 1 0 2 2 0 0 3 1 1 0 Above Mean
2 2 3 2 2 0 4 2 1 0 3 2 At Mean or Below
Analytical
Skills
Strength0 0.999 1 0.999 0 0.999 1 0.999 1 0.999 0 0.999 Above Mean
2 1 2 1 1 2 At Mean or Below
Weakness0 0 0 0 1 0 Above Mean
1 1 1 1 0 1 At Mean or Below
Conceptual
Knowledge
Strength0 0.999 0 0.999 0 0.999 1 0.999 0 0.999 0 0.999 0 0.999 0 0.999 0 0.399* 1 0.999 0 0.999 Above Mean
1 1 1 0 1 1 1 1 1 0 1 At Mean or Below
Weakness2 1 2 4 2 3 0 2 3 3 1 Above Mean
2 6 2 3 2 4 4 5 1 4 3 At Mean or Below
Critical
Thinking
Strength0 0.999 2 0.333* 2 0.333* 2 0.333* 0 0.333* 1 0.999 Above Mean
2 0 0 0 2 1 At Mean or Below
Weakness0 0 0 0 1 0 Above Mean
1 1 1 1 0 1 At Mean or Below
Big Picture
Strength2 0.999 1 0.999 1 0.999 2 0.999 2 0.333* 3 0.196* 0 0.999 1 0.107* 2 0.999 1 0.999 2 0.999 1 0.999 Above Mean
0 2 1 1 0 0 1 2 0 2 0 2 At Mean or Below
Weakness2 3 1 3 0 2 1 5 2 3 1 3 Above Mean
0 2 1 2 2 3 1 0 0 2 1 2 At Mean or Below
Research
Methods
Strength0 0.999 0 0.333* 0 0.333* 0 0.999 1 0.999 0 0.333* Above Mean
1 2 2 2 1 2 At Mean or Below
Weakness0 1 1 0 1 1 Above Mean
1 0 0 1 0 0 At Mean or Below
Defining
the Problem
Strength1 0.999 0 0.999 0 0.999 0 0.999 1 0.999 1 0.999 Above Mean
0 1 1 1 0 0 At Mean or Below
Weakness0 0 1 1 0 0 Above Mean
1 1 0 0 1 1 At Mean or Below
Data
Interpret.
Strength0 0.333* 1 0.999 0 0.333* 1 0.999 0 0.333* 1 0.999 Above Mean
1 0 1 0 1 0 At Mean or Below
Weakness2 2 2 1 2 2 Above Mean
0 0 0 1 0 0 At Mean or Below
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 29
Table 5.
Comparison of Proposal Rubric Categories to Mentor-Mentee Strengths and Weaknesses in Disposition, Skill and Knowledge- Spring Semester
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 30
Spring
SemesterIntroduction: Context Testable Hypothesis Validity/Reliability Experimental Design Data Selection Data Presentation
Faculty Student Faculty Student Faculty Student Faculty Student Faculty Student Faculty Student
CountFisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
Exact
Literature
Review
Strength 0 0.999 4 0.999 0 0.999 2 0.429* 1 0.999 2 0.429* 0 0.999 2 0.999 0 0.999 2 0.999 0 0.999 3 0.999 Above Mean
1 2 1 4 0 4 1 4 1 4 1 3 At Mean or Below
Weakness1 1 1 1 1 1 1 0 1 0 1 0 Above Mean
1 0 1 0 1 0 1 1 1 1 1 1 At Mean or Below
Conceptual
Knowledge
Strength1 0.999 0 0.999 0 0.999 0 0.999 0 0.999 0 0.999 Above Mean
1 2 2 2 2 2 At Mean or Below
Weakness1 1 0 1 1 1 Above Mean
1 1 2 1 1 1 At Mean or Below
Big Picture
Strength0 0.999 2 0.999 0 0.999 2 0.999 0 0.999 1 0.999 0 0.999 2 0.999 0 0.399* 1 0.999 0 0.199* 1 0.999 Above Mean
1 1 1 1 1 2 1 1 1 2 1 2 At Mean or Below
Weakness1 0 2 0 1 0 2 0 3 0 4 1 Above Mean
3 1 2 1 3 1 2 1 1 1 0 0 At Mean or Below
Math/
Statistics
Strength 0 0.999 0 0.099* 0 0.099* 0 0.399* 0 0.999 1 0.999Above Mean
2 2 2 2 2 1 At Mean or Below
Weakness0 3 3 2 1 1 Above Mean
3 0 0 1 2 2 At Mean or Below
Data
Interpret.
Strength0 0.999 0 0.999 0 0.999 0 0.999 0 0.999 0 0.999 Above Mean
1 1 1 1 1 1 At Mean or Below
Weakness0 0 0 0 0 1 Above Mean
1 1 1 1 1 0 At Mean or Below
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 31
Table 5. Continued
Comparison of Proposal Rubric Categories to Mentor-Mentee Strengths and Weaknesses in Disposition, Skill and Knowledge- Spring Semester
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 32
Spring
SemesterData Analysis Valid Conclusions Broader Implications Limitations/Significance Primary Literature Total
Faculty Student Faculty Student Faculty Student Faculty Student Faculty Student Faculty Student
CountFisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
ExactCount
Fisher's
Exact
Literature
Review
Strength 0 0.999 0 0.999 0 0.999 2 0.999 0 0.999 4 0.999 0 0.999 2 0.999 0 0.999 4 0.999 1 0.999 2 0.999 Above Mean
1 6 1 4 1 2 1 4 1 2 0 4 At Mean or Below
Weakness1 0 0 0 1 1 0 0 1 1 1 0 Above Mean
1 1 2 1 1 0 2 1 1 0 1 1 At Mean or Below
Conceptual
Knowledge
Strength0 0.999 0 0.999 0 0.999 0 0.999 0 0.999 0 0.999 Above Mean
2 2 2 2 2 2 At Mean or Below
Weakness1 0 1 0 0 0 Above Mean
1 2 1 2 2 2 At Mean or Below
Big Picture
Strength0 0.999 1 0.999 0 0.999 3 0.250* 0 0.999 1 0.999 0 0.999 1 0.999 0 0.999 2 0.999 0 0.999 1 0.999 Above Mean
1 2 1 0 1 2 1 2 1 1 1 2 At Mean or Below
Weakness1 0 1 0 1 0 1 0 1 0 1 0 Above Mean
3 1 3 1 3 1 3 1 3 1 3 1 At Mean or Below
Math/
Statistics
Strength 0 0.099* 0 0.999 0 0.399* 0 0.999 0 0.999 0 0.399*Above Mean
2 2 2 2 2 2 At Mean or Below
Weakness3 1 2 1 1 2 Above Mean
0 2 1 2 2 1 At Mean or Below
Data
Interpret.
Strength0 0.999 0 0.999 0 0.999 1 0.999 1 0.999 0 0.999 Above Mean
1 1 1 0 0 1 At Mean or Below
Weakness0 0 0 0 0 0 Above Mean
1 1 1 1 1 1 At Mean or Below
MENTOR PERCEPTIONS AND STUDENT RESEARCH SKILLS 33
Table 6.
Cited Experiences Leading to a Perceived Improvement in Research Skills
Experience Type
Number of
Instances
Collaboration with mentor 22
Collaboration with non-mentor (i.e. peers, post doctoral students) 7
Independent scholarly work (i.e. reading journal articles,
coursework) 11
Trial and error 29
Presenting at or attending conferences 16