atej-nata baltimore presentation walker-mazerolle d7 · 7/1/2016 1 the peer review process: what...

7/1/2016

1

The peer review process: What editors and associate editors

need from reviewers

Stacy E. Walker, PhD, ATC

Stephanie M. Mazerolle, PhD, ATC, FNATA

1

Our background

Stacy Walker

• Associate Editor

– Journal of Athletic Training

– Athletic Training Education Journal

Stephanie Mazerolle

• Associate Editor

– Journal of Athletic Training

– International Journal of Athletic Training and Therapy

• Editorial board

– Journal of Strength and Conditioning Research

• Statistical consultant

– Athletic Training Education Journal

2

Our objectives

Understand the expectations and needs of associate and assistant editors when making a decision on a manuscript

Communicate specific suggestions and comments more effectively with authors and associate/assistant editors regarding their review of a manuscript

3

Topics to be covered

General comments

Specific comments

Confidential comments

Our decision making process

Take home points

4

5

Characteristics of a good reviewer

• Be an advocate for the author. Be supportive, yet constructive in the feedback presented.

• A quality review can take up to 3 hours, but no more than that time.

– Do you have the time

– Do you have the expertise

• Honesty, yet direct and tactful

6

7/1/2016

2

Overall

• Read through authors guide

• Avoid the word “you” = defensive

• Few if any change “that” to “this” comments

• Avoid liberal copyediting comments p. 5 ‐line 79 add comma after disability and after however. line 82 add comma after variables. line 92 add comma after 1950.

7

Logical

IntroIntro PurposePurpose MethodsMethods ResultsResults DiscussionDiscussion

8

*Remember the manuscript is on a continuum; all should be connected.

General Comments

9

General Comments

• The big picture

• Tell us the good, the bad, the ugly

• Identify the flaws (i.e. low sample size, not random sample, etc.)

• Discuss inconsistencies (i.e. use of terminology, spelling errors, abbreviations, etc.)

• Is there value in the study? Will it contribute to our existing body of literature?

10

General Comments: EXAMPLE

Author(s) "flip‐flop" or conflate "professional" and "post professional" education a few times, which makes it hard to stay on track with the narrative, and keep up points being made. As I read this paper, it is essentially a reflection or commentary on what (via respondents perceptions) is happening at the professional educational level, in this case the UNDERGRADUATE level (as all respondents were in a post professional MS degree program), yet the authors on a few occasions refer to the CAATE standards for Post Professional education to make/buttress their arguments? In fact, the CAATE S & G for PP Ed are even cited to support the argument for the some of the competencies being discussed...the role and purpose of EL AT education is different than for PP AT education, and thus using PP education issues, outcomes, and CAATE S & G only confuses the narrative.

11

General Comments: Talking Points

Highlights the concerns and is clear on the issues

Remove the use of “I” or personal conviction, and try to make the comments more about the greater audience

12

7/1/2016

3


The writing of this manuscript is redundant at times and could be revised to increase clarity and conciseness. Some specific examples are provided below (in the specific comments section). However, this is not an inclusive list. The entire document should be scoured for possible revision.

13


Gives snapshot of the evaluation of the manuscript and hints of the specifics to come later

In the specific comments section, reviewer then provides those examples (helpful for author understanding)

14

General Comments: EXAMPLES

No mention of John Dewey‐championed value/power of "reflection" in early 20th Century, advocated that "experience is meaningless without reflection", and thus education is limited if not central to the pedagogy. As one of this paper's central themes is reflection, I strongly urge the authors to visit Dewey's work on reflection and experience and incorporate this into their manuscript where appropriate. It will sign. Strengthen the theoretical threads of the narrative

15


Delivery is from a personal standpoint

No real direction or a valid reason why this information should be added

Check grammar/spelling

16


The instrument used to collect data in this project seemed more like a structured interview using an asynchronous online format than a traditional survey. In this case, traditional qualitative methods (saturation of data vs. response rate, direct quotes, etc.) should be used throughout the reporting process.

17


• Able to identify a concern related to the design of the study itself

• However, it would be more helpful to have more direction in the comments about the concern.– As presented the instrument used to collect data in the project seems to be a structured interview using an online platform, not a traditional survey. If the design was meant to be more qualitative data saturation and other information is needed. If the design was geared for a traditional survey method a response rate, and other credibility/validity information is needed.

18

7/1/2016

4

Specific Comments

19

Specific Comments

What is important here for us and authors

References specific concern with line #s

Instructs authors on what clarifications are needed to satisfy the concern

Expansion of general thoughts from the general comments section of the review

20

Abstract

Correct format?

Context sets up need for study

Purpose statement consistent with introduction

Results are different than conclusions

21

Specific Comments: EXAMPLE

Line 15: Not sure "online interviews" is the best way to describe the setting

22

Specific Comments: Talking Points

Line 15: Not sure "online interviews" is the best way to describe the setting

– interviews were via Skype, state alternative or why this wording was not recommended

23


Line 16: were their 23 of each or 23 total? How many were ATS and how many were AT? Clarify here please.

24

7/1/2016

5


Line 16: were their 23 of each or 23 total? How many were ATS and how many were AT? Clarify here please.

–clear and direct, states the confusion

25


The conclusion is not fully supported by results of the study: Page 2, lines 29‐30: "...suggesting that the interpretation of balance assessment in athletes and its use in preseason screening should be made with caution.” Please, remove last sentence.

26


The conclusion is not fully supported by results of the study: Page 2, lines 29‐30: "...suggesting that the interpretation of balance assessment in athletes and its use in preseason screening should be made with caution.” Please, remove last sentence.

–States concern, example, and direction

27


Line 8 ‐ should this read subjects and other participants rather than "patients"?

28


Line 8 ‐ should this read subjects and other participants rather than "patients"?

–The headers are standardized

29

Introduction

30

7/1/2016

6

Introduction

Should be 3‐4 paragraphs

Logical, inclusive of pertinent literature

Brief and sets up need for study

31


Sentence in lines 36‐39 is very awkward to read.

32


Sentence in lines 36‐39 is very awkward to read.

–references line numbers, but doesn’t explain why it reads awkward

33


Line 49, unpublished study should be cited and listed in reference

34


Line 49, unpublished study should be cited and listed in reference

–clear and tells them what is needed

35


Purpose: In line 58, "explore the experiences..." the term perspectives should be used instead of experiences to match the title of the manuscript

36

7/1/2016

7


Purpose: In line 58, "explore the experiences..." the term perspectives should be used instead of experiences to match the title of the manuscript

‐clear and explains what the concern is and why

37


Line 55: Could you further explain the concept of Facetime. This might make it easier for the reader to compare it to the 24/7

38


Line 55: Could you further explain the concept of Facetime. This might make it easier for the reader to compare it to the 24/7

Identifies the concern and its location in the manuscript

Concern could be strengthened by instructing authors on how they currently are interpretatingthe statement as is….

39


Line 43: The author uses the phrase "most important members of the AHCT are parents, but in line 31, which operationally defines the AHCT, parents aren't included. Either they are part of the AHCT, in which case they should be included in the definition, or they are not a part of the AHCT, in which case the authors should revise this sentence.

40


Identifies specific concern with line #s

Instructs authors on what clarifications are needed to satisfy the concern

Clear example of a strong “talking point” to the author and section editor

41

Methods

42

7/1/2016

8

Methods

• Connection made to purpose and research agenda

• Enough details to replicate

• Tables and figures must compliment and enhance the presentation of the findings

• Headings should include participants, research design, protocols and/or instrumentation, and, analysis

43


Line 68: "required parents to rate parents" ‐clarify if they are rating their own knowledge, or other parents' knowledge.

44


Line 68: "required parents to rate parents" ‐ clarify if they are rating their own knowledge, or other parents' knowledge.

–Clear and specific

45


Line 171: How were discrepancies in interpretation handled?

46


Line 171: How were discrepancies in interpretation handled?

‐ Not specific, how is author suppose to respond? What is expected?

47


Line 78‐Line 83: Please explain what types of revisions are made to the survey and why.

48

7/1/2016

9


Line 78‐Line 83: Please explain what types of revisions are made to the survey and why.

–Give example or guidance on what you are looking for?

– Please explain what types of revisions are made to the survey (re‐ordering of questions, additional questions, removal, etc.)? This information can provide insights to how the final instrument was developed. Explain why these changes were made (i.e. content review identified removal due to redundancy).

49


Consider expanding on how/why the three AT programs were "purposefully" selected. Were the three programs of convenience and consider discussing the Carnegie classifications for each school (this has the potential to influence clinical education opportunities).

50

Specific Comments: Talking Point

Direct with what information is missing and what authors would want to address concern

Consider, however, expanding on the importance of the information

– Why is more information on the classifications needed?

– Do they reference this in the limitations section, or other areas?

51

Specific Comments: EXAMPLES

A response rate of 16% is horribly low.

52


• Comment is direct

• Consider softening the tone OR providing some context to the comment

– The response rate of 16% is of concern, as a 20% response rate is usually viewed as appropriate and the benchmark. Can the author (s) provide context and rationale behind the lower rate. References could be helpful as well in the discussion.

53


Line 134 ‐ The only mention interviews I could find in Part I was in the validation process of the survey (Figure 1). The authors need to better develop this interview process. How were these 7 ATs selected? What questions were asked? Since this never arises in the Results or Discussion, I'm guessing it's simply a lack of clarity here in the Methods?”

54

7/1/2016

10


• Highlights the issue with the instrumentation

• Identifies where the concern is

• Consider presenting the range of line #s

• Questions are helpful, but provide some reference to WHAT information it needed to satisfy the concern

55


Line 102‐103: where the 3 parts in that order or where the randomized to avoid an order affect?

56


Check proper use of “grammar” (and spelling) when providing feedback and comments

Provides specific location and is clear on what they are asking the authors; however

– Comments reflect a clear misunderstanding of the research design (authors identified in methods that the data collection was structured)

57

Data Analysis

58

Data Analysis

Appropriate statistics

Provides alpha level

For Qualitative

– provide reader an understanding of how it was performed

– Trustworthiness

59

Specific Comment: EXAMPLE

What was the response rate for those who participated in the interview section of the research? It was clear that 34 did the interview but of how many overall participants from the main study? Do you feel the interview participants were a representative sample. While the break down regarding type of setting, district, etc is in the table, it would be helpful to comment on whether you believe there is a fairly representative sample or describe limitation in limitations section if necessary.

60

7/1/2016

11


Qualitative is driven by data saturation so this comment demonstrates lack of understanding

61


Sometimes the quotes and indentations are inconsistent. Are the quotes meant to be tabbed over or included in the article body paragraphs

62


Sometimes the quotes and indentations are inconsistent. Are the quotes meant to be tabbed over or included in the article body paragraphs

‐ AMA 40 characters go to block style

63


I appreciate the detail on the CQR process but please provide more detail on the actual coding process. What is a code? How did it turn into a theme? More details on this process, especially linking it to an actual example in the results, would be helpful.

64


I appreciate the detail on the CQR process but please provide more detail on the actual coding process. What is a code? How did it turn into a theme? More details on this process, especially linking it to an actual example in the results, would be helpful.

–No need to define these terms

65


Line 193‐199 Why weren't Median and IQR data not presented for this non parametric data?

66

7/1/2016

12


Line 193‐199 Why weren't Median and IQR data not presented for this non parametric data?

–Direct and states concern OR issue

67


Line 227‐228 Is this really "significant"? The different is only .16 between the 2 groups, and the SD is about .5 in both‐‐more than double the difference between groups.

68


Line 227‐228 Is this really "significant"? The different is only .16 between the 2 groups, and the SD is about .5 in both‐‐more than double the difference between groups.

– Important to tell author what is needed to address the question/comment raised

69


Page 5, Data Analysis and trustworthiness—The process described seems to veer somewhat away from the traditional model of multiple analyst triangulation. The first four transcripts were viewed by all the analyst, and 3 random others, but 14 of the transcripts (most of the data) were only coded by one individual? To address this please add additional references for support as well as details on the rationale for this approach.

70

Specific Comments: Talking points

Provided a specific page # and section of concern

Raises a questions, but also directs author on what they are requesting

71

Results

72

7/1/2016

13

Results

Reported results line up with data analysis section and purpose

Appropriate statistics

Appropriate theme explanation and number and description of quotes

73


Page 12, line 260‐261: This sentence makes little sense in the context of the previous sentence. Also, how is it possible to decrease patient care and increase clinical experience time? It would seem that more clinical experience would equal more patient care.

74


Information presented directs the author & section editor to the concern yet provides a degree of uncertainty on the ACTUAL sentence

The reviewer, however, does not clarify exactly what would satisfy the point of contention

75


Results section, page 6, line 131: I question the informality of a GA being assigned to shadow a more experienced staff member. This practice seems more formal in nature. Line 156 does in fact identify that the GA's were assigned to someone which at least from an organizational standpoint seems formal. If the conversations and socializing experiences arose in a more informal nature as GA and staff were working together in the AT room, labeling those interaction as a "informal" would seem more appropriate. Assigned shadowing however is designed to show new staff how things operate in the AT room and facilitate discussion of various topics. I do however agree with the informal nature of the "sink or swim/thrown in the fire approach" Suggest reconsidering what is identified as formal and informal processes.

76


• Remove the use of “I”

• Language should reflect recommendations and considerations and not personal thoughts and reflections

• Rephrased to: “as presented the informality theme may need some restructuring. As presented it could be interpreted as…”

77

Discussion

78

7/1/2016

14

Discussion

Relates back to the results and purpose –organized and logical

Does it expand the findings and how they relate to the literature and specualtion on why they had those findings?

Any statements made that are beyond the scope of the findings?

79


Discussion: The author provides good discussion of the concepts/statements presented in the results, but again the sections lack structure and need better developed introductory statements for each theme. The authors also need to break up paragraphs (paragraph to discuss findings and impact on XYZ; paragraph for recommendations/strategies).

80

Specific Comments: Talking points

A bit general

Be direct

– Introductory statements at the beginning of each theme would help orient the reader

– Suggest headers to organize the discussion

81

Specific comment: EXAMPLES

Lines 450 ‐ 454: Why do you believe this is? Can your further discuss this concept?

82


Lines 450 ‐ 454: Why do you believe this is? Can your further discuss this concept?

–Direct, asks for adding to the discussion

83

Specific Comment: EXAMPLES

Authors do not demonstrate any depth of analysis in the discussion

84

7/1/2016

15

Specific Comment: Talking Points

Authors do not demonstrate any depth of analysis in the discussion

–Vague and unclear as to what reviewer is expecting

85


Page 11 line 223 Not a correct interpretation of the reference.

86


Page 11 line 223 Not a correct interpretation of the reference.

– Why does the reviewer think it’s an incorrect interpretation?

87


Very few references to current literature used.

88


Very few references to current literature used.

–Vague and unclear

89


• Line 217: I am not sure the high school or college AT would agree with the larger amount of work for these professionals as they would feel that the workload is the same just in different ways.

• For example a pro person has more support staff for set up and manual labor but has more DR interaction and meetings, where as the high school person has all the manual labor set up without the DR interactions.

90

7/1/2016

16


• Delivery is from a “personal” standpoint, not from empirical view

• Concern may be valid and necessary to address, but the comment does not provide direction for the author.

• Consider providing data OR references to assist the author in further developing thoughts or discussion points

91

Confidential Comments

92

Confidential Comments

• Important to include comments here that you DO NOT want the authors to see

• Often the information shared in this section is valuable and usable in supporting the decision for the manuscript—WE WANT TO USE IT!– We are not blind to the authors

• Share apprehensions or thoughts on your decision– “I am on the fence with this article because although the information is new and necessary, the low N is of concern for power and generality”

93

Confidential Comments‐Helpful

• The authors despite adding more information to the manuscript, have failed to address the concerns raised and unfortunately have illustrated a few red flags.

• I reviewed a paper similar to this for another journal, therefore I am not sure if offers any new information.

• My reasons for rejection was because I believe the authors failed to triangulate the data, as described.

94

Confidential Comments‐Not Helpful

• The aims of the study are noble, but the investigation falls short on several key methodological and statistical grounds, some of which are remediable and some of which are not.

• The manuscript is lengthy and at times is difficult to follow. I do not believe the average educator will gain a high degree of usable benefit from the manuscript due to its low relevance. However, there are strategies that may resonate with some educators. As we have had them perform significant revisions, I was torn between additional minor revisions and rejection. My rejection side is based on relevance.

95

Process

96

7/1/2016

17

Article submittedArticle submitted

Assigned to Senior and/or Associate Editor Assigned to Senior and/or Associate Editor

*2 reviewers assigned *2 reviewers assigned

97*Editors do review paper and if not fit for review could be rejected

Timeline

Reviewer timeline depends on potential reviewers accepting/rejecting

Failure to respond quickly to an invitation to review slows down the timeline

98

How are decisions made?

Reviewer comments – not solely

Consultations

‐ expert

‐ statistical

Ratings

99

ATEJ Rankings 0‐5

• Overall importance of the paper to athletic training education

• Overall quality of content

• Overall quality of the written presentation

• Overall quality of the methods

• What is your recommendation?

100

101

JAT Ratings

Decide

Accept/Publish

Major revisions

Minor revisions

Reject

Specific to JAT – Refer to IJATT, ATEJ, or NATA news

102

7/1/2016

18

In closing….

103

Take Home Points

Avoid the word “you”

Explain why you feel the way you do, recommend, make sure the authors also understand what you are saying

What would you want a reviewer to say to you in this instance?

104

Please avoid

Statements that don’t explain fully “Reads awkwardly” – why is it awkward, explain why it’s awkward

Just asking a question – what do you want from the author? What needs to be done? Did you reach data saturation? – is a good questions but what needs to be done with this based on the question

Really vague statements – “the methods are clear”

105 106

Questions

107

Resources

• Knight KL, Ingersoll CD. Structure of a Scholarly Manuscript: 66 Tips for What Goes Where. Journal of Athletic Training. 1996;31(3):201‐206.

• Hertel J. Editorial: Keep It Simple: Study Design Nomenclature in Research Article Abstracts. Journal of Athletic Training. 2010;45(3):213‐214.

• Casa D. The five “I”s of discussion. NATA News. August 2001, p56.

108

7/1/2016

19

Contact Information

• [email protected]

• [email protected]

109

atej-nata baltimore presentation walker-mazerolle d7 · 7/1/2016 1 the peer review process: what...

Documents