How to read and critique a technical paper?

3 phases to reading

• Determine if there is anything interesting at all in the paper.

• Determine which portion of the paper contains the interesting stuff.

• Read the whole paper.

Is there anything interesting?

• Ideally the abstract should tell you this, but frequently it does not.

• Need to jump about– Read conclusion– Read introduction– Look at the bibliography– Glance at the TOC (if any)

Which portion contains interesting stuff?

• Typically, a paper outlines its organization at the end of the introduction. Use this to determine which portion contains the exciting stuff.

Read the whole paper

• Read with the following questions in mind– How can I use this stuff?– Does this really do what the author claims

to do?– What if the assumptions and choices that

the author made are discarded (or made invalid)?

Context and problem statement

• What problems are the author trying to solve or trying to convince you of?

• Are they important problems?– Why?– Why not?

• What is the author’s thesis?

Related work evaluation

• Does the author describe other work in the field?

• If so, how does this research differ from the other work?

New idea

• What new idea is the author proposing?– Architecture– Algorithm– Mechanism– Methodology– Perspective

• Is the idea useful and practical?

What to evaluate?

• What need to be evaluated to confirm the worthiness of the new idea?– Runtime– Throughput– Resource utilization– Model validation

How to evaluate?

• How did the author go about conducting the evaluation?– Formalize and prove theorems– Run simulations– Artifact design and construction– Collect traces from existing systems

Was the evaluation correct and adequate?

• How was the data collection done?• Do you agree with the analysis of data?• Do you agree with the conclusions

about the data?• Do you have any new interpretation of

the data?• Can you suggest new ways to evaluate

the data?

Assumptions, drawbacks and extensions

• Can you think of other aspects of the idea that need to be evaluated?

• Can you think of extensions or modifications to the idea to improve it?

• How would you evaluate your improvement?

Assumptions, drawbacks and extensions

• Can you apply the idea or method of evaluation to your own project?

• Do the authors make any assumptions that are not valid or realistic?

• Can you come up with a more general solution that does not rely on one or more of the assumptions?

Future work

• Does the author indicate how the work should be followed up on?

• Does the paper generate new ideas?

• Does the paper implicitly or explicitly provide a new way of doing other things or of thinking about problems?