how to revise a research paper and respond to reviewers damon chandler ecen 6001, spring 2009...

How to Revise a Research Paper and Respond to

Reviewers

Damon Chandler

ECEN 6001, Spring 2009

Oklahoma State University

Outline

• Understanding the review process

• Digesting the reviews

• Revising your paper

• Communicating your revisions to the reviewers and editor

• Conclusions

The Need for Peer Reviews

1. Ensure quality, checking that no mistakes in logic have been made

[Benos et al., 2003]

Submitted to Elsevier Health Series: Computer Methods and Programs in Biomedicine

The Need for Peer Reviews

1. Ensure quality, checking that no mistakes in logic have been made

2. Ensure that the work is original and significant

3. Ensure that the experimental methods are sound

4. Ensure that the results presented support the conclusions drawn

5. Ensure that no errors in citations to previous work have been made


Outline





• Conclusions

The Review Process

1. The paper is assigned to an editor

2. The editor selects reviewers

3. The reviewers are contacted and asked to review your paper

4. The reviewers review your paper

5. You receive the reviews, modify, and resubmit

6. The editor and the reviewers review your paper (again)

7. The editor makes his decision

An Invitation to Review

Dear Dr. Chandler,

As associate editor of the IEEE Transactions on Information Forensics and Security, I have received the above-referenced manuscript. I believe the topic is in your area of expertise, and hope that you are available to review it for me. The abstract is provided below for your reference. We ask that reviewers complete their evaluation within 4 weeks of receipt.

IEEE Transactions Reviews

I. Ratingsa) Suitability of the topic

b) Content

c) Presentation

d) Overall Rating

II. Recommendation

III. Detailed Comments

IEEE Transactions Ratings

I(a). Suitability of the topic: – Is the topic appropriate for publication in these

transactions?

– Is the topic important to colleagues working in the field?


I(b). Content:– Is the paper technically sound?

– Is the coverage of the topic sufficiently comprehensive and balanced?

– How would you describe the technical depth of the paper?

– How would you rate the technical novelty of the paper?


I(c). Presentation:– How would you rate the overall organization

of the paper?

– Are the title and abstract satisfactory?

– Is the length of the paper appropriate?

– Are symbols, terms, and concepts adequately defined?

– How do you rate the English usage?

– Rate the Bibliography


I(d). Overall rating:– How would you rate the appropriateness of

this paper for publication in this IEEE Transactions?

– How would you rate the technical contents of the paper?

– How would you rate the novelty of the paper?

– How would you rate the literary presentation of the paper?

IEEE Trans. Recommendation

Code Recommendation

A Publish Unaltered

AQ Publish in Minor, Required Changes

RQ Review Again After Major Changes

R Reject (A major rewrite is required; encourage resubmission at a later time.)

R Reject (Paper is not of sufficient quality or novelty to be published in this Transactions.)

R Reject (Paper is seriously flawed; do not encourage resubmission.)

IEEE Trans. Detailed Comments

• Please state why you rated the paper as you did in Sections I and II

• This is the “meat” of the reviews– Typically, two paragraphs to multiple pages

(some reviewers can get very detailed)– Your changes and responses will be based on

these detailed comments

J. Optical Society Reviews

I. Review Criteriaa) Originality and

significance

b) Accuracy and clarity

c) Appropriateness

II. Recommendation

III. Detailed Comments

http://josaa.osa.org/journal.cfm

J. Optical Society Review Criteria

I(a). Originality and significance– Does the paper contain enough new material

to warrant publication?– Does the paper contain enough

significant/useful material to warrant publication?

– Is the work placed in the proper context?– Are there adequate references?


I(b). Accuracy and clarity– Is the scientific development sound?

– Are the conclusions supported by the evidence?

– Is the paper clearly written, and assumptions and procedures clearly stated?

– Is the paper reasonably self-contained?

– Are values given for important experimental parameters?

– Are the figures and tables effectively presented?


I(c). Appropriateness– Is the paper appropriate for this journal?– Is there a more appropriate journal?

J. Optical Society Recommendation

Decision Recommendation

Accept Publish as is

Accept Optional minor revision

Revise Mandatory minor revision

Revise Mandatory major revision

Reject Encourage resubmission after substantive changes

Reject Strongly against publication

J. Optical Society Detailed Comments

• Same format as in other journals

• This is the “meat” of the reviews– Typically, two paragraphs to multiple pages

(some reviewers can get very detailed)– Your changes and responses will be based on

these detailed comments

More on the Detailed Comments

• Why the detailed comments?– Bordage, 2001: 151 research papers

submitted to Research in Medical Education from 1997-1998

– 40% of papers received satisfactory ratings, but were still recommended for rejection

• Detailed comments are a vital means of communication between the authors and reviewers


• Good detailed comments contain:

– One-paragraph summary (this demonstrates that the reviewer understands the main points)

– One paragraph on the positives

– Multiple paragraphs on major negatives (or potential major negatives)

– Listing of minor comments


• Damon’s Rule of Thumb: – Never write anything in a review that you

wouldn’t say in person

• Try to balance criticisms with encouragement:– “The author’s main conclusions are terrible.”– “I was impressed by the author’s novel

experimental approach. However, I do not feel that the main conclusions are supported by the results.”

Summary of a Good Reviewer

• The most helpful review is one that articulates the strengths of a paper while also assiduously identifying the limitations of the manuscript that can be addressed in a revision.

• Nevertheless, even if a paper is well received overall, reviewer comments on manuscripts are commonly “negative.”

[Roberts et al., 2004]

Outline





• Conclusions

“During my editorship, no article submitted to the Psychological

Bulletin has been accepted outright with no changes.”

[R. J. Sternberg, Yale University]

Positive Review Negative ReviewReviewer’s Average Scoreon 10+ Papers

Num

ber

of R

evie

wer

s

Assassins

Demoters

Zealots

Pushovers

Mainstream

Odds of Getting a Harsh Reviewer

-4 -3 -2 -1 0 1 2 3 40

0.1

0.2

0.3

0.4

Positive Review Negative ReviewReviewer’s Average Score

Pro

babi

lity

of R

evie

wer

harsh reviewers

Odds of Getting a Harsh Reviewer

reviewer harsh oneleast at Pr

reviewersharsh no Pr1

N

i

thi1

harsh t isn'reviewer Pr1

N)0.1(1

N843.01

5 %,58

4 %,50

3 %,40

N

N

N

Digesting the Reviews

Four-step process:

1. Read the reviews once, and then file them in a safe location

2. Don’t think about the reviews for at least a week (instead go skiing, golfing, etc.)

3. Read the reviews again

4. Discuss the reviews with your co-authors and create a plan-of-attack

Outline





• Conclusions

Deciding What to Change

1. You must address all comments• You can’t pick-and-choose which comments

to address• Even minor comments need to be addressed

2. Address does not always mean change• You and your co-authors should decide what

to change, and what to defend• Often, changing is the easiest route

(demonstrates openness to suggestions)


3. Change does not always mean revamp• Easy changes include:

Rewording Adding extra references Adding an extra paragraph, table, or figure Adding an appendix

• More difficult changes include: Modifying your central hypothesis Modifying your main algorithm Redoing an experiment


4. Always change technical errors• It’s the reviewer’s job to find these• Even minor errors can cast doubt

5. Always change errors in references• Skilled reviewers know the history better

than newer authors• You don’t want to get off on the wrong foot

with experts in the field by not citing the correct papers in the correct order


6. Always change parts which yielded “I didn’t understand”-type comments

• If the reviewer didn’t understand it, the readers might not either

• The effort required to defend this point will be more than the effort required to change the paper

• “I didn’t understand” is a polite way of saying “you didn’t explain clearly enough”


7. Always change parts which are have been mentioned by multiple reviewers

• If two or more reviewers make similar comments, the readers will likely have the same comments

• Repeated comments stand out to the editor• The effort required to defend this point will be

more than the effort required to change the paper

Revising the Paper

• Divide the comments into two categories:1. Easy changes

2. Difficult changes

• Do the difficult changes first– This might take some time (especially if you

need to repeat an experiment)

– Easy changes might be eliminated

• Consult with your co-authors on changes

Revising the Paper(Example of an Easy Change)

Reviewer did not like the memoryless

Gaussian assumption

Revising the Paper(Example of an Easy Change)

Reviewer did not like the memoryless

Gaussian assumption

Quick fix: Use a distribution-

dependent parameter

Revising the Paper(Example of an Easy, but Lengthy Change)

Revising the Paper(Example of a Difficult Change)

Outline





• Conclusions

Communicating Your Changes

• Letter to the editor– Summary of changes/defenses– Write this last

• Letters to each of the reviewers– Responses to each comment– Write these first

• Letter to typesetter (optional)

Letter to Editor(Opening Paragraphs)

February 5, 2003

Dear Dr. Said:

Enclosed please find documents which contain our responses to the reviewers of manuscript TIP-00290-2003 "Dynamic Contrast-Based Quantization for Lossy Wavelet Image Compression," D. M. Chandler and S. S. Hemami.

We have modified the manuscript to directly address all of the comments of the reviewers. The modifications are detailed in the individual letters to the reviewers.

(continued)

Letter to Editor(Summary of Changes)

To summarize our changes:

• We have performed and added the results of a psychophysical experiment... (Reviewers 1, 2, and 4)

• We have reworded our claims about the images produced... (Reviewers 2 and 4)

• We have generalized our rate-distortion function to account for... (Reviewers 3 and 4)

• We have added an appendix which shows the derivation of the approximation... (Reviewer 4)

(continued)

Letter to Editor(Closing Paragraph)

We sincerely appreciate the helpful comments and suggestions.

Thank you for facilitating this submission.

Sincerely,

Damon M. ChandlerSchool of Electrical and Computer EngineeringOklahoma State UniversityStillwater, OK 74074

Letters to Reviewers

• Open by thanking the reviewer for his/her time during the review

• Then, add a summary of changes

• Next, make a dialogue-type list of comments and responses– For changes: Indicate location (page and paragraph numbers)– For defenses: Be polite and write professionally (don’t write

anything you wouldn’t say in person)– Thank the reviewer abundantly (but don’t overdo it)

• Close by thanking the reviewer again

Letters to Reviewers (Opening Paragraph)

Responses to Reviewer 2 of TIP-00290-2003 “Dynamic Contrast-Based Quantization for Lossy Wavelet Image Compression,” D. M. Chandler and S. S. Hemami

Thank you for taking the time to review this paper. We have addressed all of your comments, as described in the dialogue below. In summary, we have performed a psychophysical experiment to assess the bit-savings afforded by using the proposed algorithm over baseline JPEG-2000. This experiment and its results are described in Section VI.D and Table III.

(continued)

Letters to Reviewers (Dialogue List)

Reviewer: <Insert reviewer’s comment>

Authors: <Insert your response>

Letters to Reviewers (Example of a Simple Change)

Reviewer: Each subband is assumed to be a "memoryless Gaussian source". This assumption is not very realistic. It would be interesting to know the impact of this assumption on the performance of the proposed method.

Authors: Thank you for pointing this out. We have removed the memoryless Gaussian assumption, and we have accordingly modified the rate-distortion function via a parameter which depends on the probability density function of the subband (Page 24, 3rd paragraph).

Letters to Reviewers(Example of a Simple Change)

Reviewer: Based on Fig. 6, the improvements of the proposed approach do not appear to be very impressive. Even though there are definitely some improvements, it is not clear how significant they are.

Authors: We are hoping that the printed journal version of this figure will be able to better-reproduce the visual gains. In addition, we have now made the images available online. We have also presented the results of a psychophysical experiment to quantify the bit-savings afforded by using the proposed algorithm over baseline JPEG-2000.

Letters to Reviewers(Example of a Backing Down)

Reviewer: On page 3, lines 1 to 3, this statement is incorrect as it indicates that existing quantization strategies rely only on a predetermined set of quantizer step sizes. This is not correct as there are existing perceptual-based algorithms that dynamically and adaptively compute the quantizer step sizes such as the work in Reference [10] (which also appeared as a journal version in the IEEE ITIP, 2000). So, this statement should be removed or rewritten.

Authors: Thank you for catching this. We have removed the statement. We have also updated Ref. 10 to refer to the journal version of the research.

Letters to Reviewers(Example of a Defense)

Reviewer: On page 2, the reviewer especially does not agree with item 1 where the authors stated that most experiments involving the HVS have quantified contrast sensitivity based only on simple targets. The authors imply in item 1 that more complex natural-image targets are needed which is not correct.

Also, Item 3 should be modified as several perceptual-based algorithm exist which account for masking in addition to the regular CSF and this include references [8] and [10].

Letters to Reviewers(Example of a Defense)

Authors: We appreciate and agree with many of your points. We do believe however, that our methodology is more of a top-down approach that has offered useful insights into the overall response of the HVS to the types of distortions induced via wavelet-based image compression presented against natural-image backgrounds.

We did not intend Items 1, 2 and 3 to characterize all HVS-based compression algorithms, and we apologize if it came across as such. As per your suggestions, we have added specific references to Items 1, 2, and 3. To Item 1, we have added references...

Letters to Reviewers(Closing Paragraph)

We sincerely appreciate your insightful and constructive comments and suggestions. We believe that these have greatly strengthened the paper.

Thank you again for taking the time to review this paper.

Sincerely,

Damon M. ChandlerSchool of Electrical and Computer EngineeringOklahoma State UniversityStillwater, OK 74074

Outline





• Conclusions

Conclusions

• When reviewing: – Be critical of technical content– Make an extra effort to be nice when criticizing

• When digesting reviews:– Try not to take things personally– Expect at least one harsh reviewer

• When addressing reviews:– Divide and conquer (do difficult changes first)– Communicate your changes, making an extra effort to

be professional and thankful

Backup Slides

How to be a Good Reviewer

• Evaluate the paper honestly, objectively, and critically

• Disclose any potential conflicts of interest

• Identify areas in which you are not an expert

• Write constructive and helpful reviews

• Maintain confidentiality


how to revise a research paper and respond to reviewers damon chandler ecen 6001, spring 2009...

Documents

biomedicine slide

bibliography slide

decision slide

paper appropriate

research paper

ieee transactions reviews

detailed comments slide

review process