The scope of a paper

Which results are the most surprising? What is the one result that other researchers might adopt

in their work? Are the other outcomes independent enough to be

published separately later on? Are they interesting enough to justify their being included?

Does it make sense to explaín the new algorithms first, followed by description of the previous algorithms in terms of how they differ from the new work? Or is the contribution of the new work more obvious if the old approaches are described first, to set the context?

What assumptions or definitions need to be formalized before the main theorem can be presented?

What is the key background work that has to be discussed? ,

Who is the readership? For example, are you writing for specialists in your area, your examiners, or a general computer science audience?

A writing-up checklist

Have you identified your aims and scope?Are you maintaining a log and notebook?Does the paper follow a narrative?In what forum, or kind of forum, do you plan to

publish?What other papers should your write-up resemble?Are you wríting to a well-defined structure and

organization?

Have you chosen a form for the argument and results?

Have you established a clear connection between the background, methods, and results?

How are results being selected for presentation?How do the results relate to your original aims?Have you used any unusual patterns of organization?Have the results been critically analyzed?Are the requirements for a thesis met?Do you and your co-authors have an agreed

methodology for sharing the work of completing the write-up?

Reading research literatureWhat is the main result?How precise are the claims?How could the outcomes be used?What is the evidence?How was the evidence gathered?How were measurements taken?How carefully are the algorithms and experiments

described?Why is the paper trustworthy?Has the right background literature been discussed?What would reproduction of the results involve?

A research checklistAre the ideas clear and consistent?Is the problem worthy of investigation?Does the project have appropriate scope?What are the specific research questions?Is there a hypothesis? What would disprove the hypothesis? Does it have any

improbable consequences?Are the premises sensible?Has the work been critically questioned? Have you

satisfied yourself that it is sound science?How are the outcomes to be evaluated? Why are the

chosen methods of evaluation appropriate or reasonable?Are the roles of the participants clear? What are your

responsibilities? What activities will the others undertake?

What are the likely weaknesses of your solution?Is there a written research plan?What forms of evidence are to be used?Have milestones, timelines, and deadlines been

identified?Do the deadlines leave enough time for your

advisor to provide feedback on your drafts, or for your colleagues to contribute to the material?

Has the literature been explored in appropriate depth? Once the work is largely done and your perspective has changed does it need to be explored again?

An experimentation checklistWhat is to be measured? How is it be

evaluated?What code has to be obtained? What data

has to be gathered? What has to be implemented?

Should the experimental results correspond to predictions made by a model?

What enduring properties might be observed by other people attempting to valídate the work with different hardware, data, and implementation?

Have appropriate baselines been identified?Do the results make sense? Are they

consistent with any obvious points of comparison?

Is the code going to be made publicly available? If not. why not?

What variables might influence the results? How do the experiments dis-tinguish between the effects of the variables?

Are statistical methods nccessary for validation of the results?

What is the population? How is a saniple to be taken?

Are noiebooks being kept? What is being recorded in the notebooks?

Is ethics clearance required?

Evaluation of papers

The process of evaluating a paper involves answering questíons such as:

Is there a contribution? Is it significant?Is the contribution of interest?Is the contribution timely or only of historical

interest?Is the topic relevant to the likely readership? |Are the results correct?Are the proposals and results critically analyzed?Are appropriate conclusions drawn from the

results, or are there other pos-sible interpretations?

Are all the technical details correct? Are they sensible?

Could the results be verified?Are there any serious ambiguities or

inconsistencies?What is missing? What would complete the

presentation? Is any of the material unnecessary?How broad is the likely readership?Can the paper be understood? Is it clearly written?

Is the presentation at an adequate standard?Dees the content justify the length?

A refereeing checklist

When you recommend that a paper be accepted, you should do the following:

Convince yourself that it has no serious defects.Convince the editor that it is of an acceptable

standard, by explaining why it is original, valid, and clear.

List the changes, major and minor, that should be made before it appears in print, and where possible help the author by indicating not just what to change but what to change it to (but if there are excessive numbers of errors of some kind, you may instead want to give a few examples and recommend that the paper be proofread).

Take reasonable care in checking details such as mathematics, formulas,and the bibliography.When you recommend that a paper be rejected, or recommend that it be resubmitted after major changes, you should do the following:

Give a clear explanation of the faults and, where possible, discuss how they could be rectified.

Indícate which parts of the work are of value and which should be discarded, that is, discuss what you believe the contribution to be.

Check the paper to a reasonable level of detail, unless it is unusually sloppy or ill-thought.

In either case you should do the following:Provide good references with which the authors should be familiar.Ask yourself whether your comments are fair, specific, and polite.Be honest about your limitations as a referee of that paper.Check your review as carefully as you would check one of your

own papers prior to submission.

An ethics checklistIs all the text yours?Are you the copyright holder for all figures and

illustrations?Have any authors been listed without their

knowledge?Have other potential authors been omitted? Do they

know that publication is proceeding without them?Is any of the material confidential?Was clearance obtained for any human studies?Is the scope of citation and attribution clear? Is there

a clear distinction between new work and previous knowledge?

Has other work with similar results been appropriately citcd and discussed?

If any material is shared with another paper, has the sharing been explained to the reader? Has it been explained to the editor?

Does the paper include materia! recycled from your eariier work?

Are other papers accurately described?Do you know which versión of the code was used

to run the experiments? y Could you run the experiments again and get the same outcome?

Are there any weaknesses or ¡imitations in tne experiments that need to be described? Wouíd you be prepared to show other researchers the raw experimental materials?

Are any claims overstated?

A presentations checklist

What is the key thing the audience should remember?Is there enough background material for the intended

audience'.'Is any material unnecessary?Could some of the material be left for people to read

about later?Is the talk self-contained?Does the talk have a motivating preamble?Have complex issues been explained in gentle stages?Are the results explained?Are the numbers necessary?Are more diagrams needed?

Are the slides simple? Do they have unnecessary ornamentation or distracting use of colour?

Is there any unnecessary animation?Are the font sizes reasonable?Are there enough examples?Have you rehearsed the talk?Have you prepared something to say about each slide?What were the limitations of the research?Do you explain why the research is interesting or important? Is there a clear conclusion?Have you memorized the talk? If you are asked a question you can't answer, how will you

respond?Have you rehearsed your manner? Will your enthusiasm show?Do you know how to use the equipment?