handling the literature prof carole goble [email protected] comp80122 28 january 2015
TRANSCRIPT
You will read a lot of papers
Most will be irrelevantMany will be poor
Some will be importantA few will change your life
You will find lots of papers or none
Find, Manage, Read, Synthesize ….
Purpose: Announce and Convince
Defend results are plausible or correct and method convincing and repeatable.
Review & Learn Verify the results empirically. Trust. Understand. Convince, comfort, credibility.
Reuse Use the explained and trusted results (data, method) for new / my science on demand. Compare. Extend.
Is it “true”?Can I repeat it?Am I convinced?Is it plausible?Can I reproduce it?
Can I use it?Is it a useful contribution?
Scholarly Communication Forms
• Making an impact– Demo, Magazine articles: reviewed– Blogs, twitter, forums: unreviewed– Technical reports
• Proposing an idea or view– Position statement, Commentary,
Perspectives, Magazine Department, Doctoral Consortiums
– Highly cited, editorialised, low rigour, established figures
• Presenting a preliminary research finding, on-going work, ideas, small extensions to existing work,
• Short paper, workshop paper, poster• Medium rigour, peer review
• Presenting a research finding• National Conferences:
– new ideas/applications/tools, medium extensions, more serious reviewing
• International Conferences: – mature work, serious reviewing, but
time-constrained, check track
• Journal article: – lots of mature work (e.g., 2
conference papers into 1 journal paper), serious reviewing, not time-constrained
– High rigour, peer-review
Scholarly Communication Forms
Additional material• Conference papers means a
presentation• Slides, videos• Web pages• Blogs• Technical reports• Other?
More forms• Position paper• Systems paper• Theory paper• Vision paper• End-to-End paper• Surveys papers• Summary papers• …
• Deep and narrow• Broad and shallow
Salami paper writing
Challenge: rebuilding a body of work
Finding and choosing papers
• Key players and key papers everyone cites
Finding and choosing papers
• Special repositories / libraries
Lab Note books
• Use one. A book. Or electronic.
• A wiki? A Blog?
http://www.atriumresearch.com/html/eln.htm
Finding and choosing papers
• Citation management
Managing your references• Use a reference management systemhttp://en.wikipedia.org/wiki/Comparison_of_reference_management_software
Places to find tools
https://www.force11.org/catalog
Utopia Documents: http://getutopia.com
Kristian Garzi
a PhD student speaks about tools
I can’t find any papers!
• Adjacent fields?
• Different terminology?
• Related topics?
Synthesize: beyond Shopping Lists
• Annotated Bibliography
• Literature review framework– Categories– Clusters – Timelines
• Comparisons on aspects– Cross cutting the papers
Mind-mapping tools can help• http://cmap.ihmc.us/• http://www.mindjet.co
m/uk/mindmanager/• http://freemind.sourcef
orge.net/wiki/index.php/Main_Page
• http://www.biggerplate.com/
http://www.biggerplate.com/mindmaps/NwUuYpZx/critical-literature-review-template
Smart Reading• Understand the context of the paper.• Beginnings and endings.• Survey the structure.• Use figures and tables. Generate them if
absent.• Decide when to read every word. • Summarise the paper.• Read out loud.• Explain the paper to your cat.• Multiple reading passes. Over time.• Set aside time to think about it and digest it. • Get help. Talk to people. Set up a reading
group.
Where does it fit with my work
• Is it relevant? If not why not?• How does it fit with your framework?• Yes – you will need a framework!• Can you relate the terminology and
notation to yours?• Keeping a record of the contribution.
When will the paper become relevant?
• Over time• An ongoing framework• Revisiting
Hints for Reviewing Papers• The answer to each question tells you something about the technical content of
the paper• The ease of extracting the answer to each question tells you something about the
quality of the writing.
Questions• Is this a vision/position/direction paper, or a measurement/implementation paper? • If you know the area well, can you mentally slot this paper somewhere in the
taxonomy? ("Differs from X as follows; has the following in common with Y;" etc.) If the paper is radically brilliant, new, or iconoclastic work, this question may not apply.
• Can you summarize the single most important contribution in one or two sentences?
Issues• Will this advance the state of the art? • Did you learn anything new? • Does it provide evidence which supports/contradicts hypotheses? • Experimental validation? • How readable is the paper? • Is the paper relevant to a broader community? Goals of Review• Guide the program committee in selection process • Help authors (to revise paper for acceptance, to understand rejection, to improve
further research and future projects) John Ousterhout's Hints for Reviewing Papers
Make yourself a template• Author housekeeping stuff• Paper genre• Problem statement/motivation• Key ideas• Technical contribution• Technical flaws• Evaluation• Presentation• Comparison
– To authors’ other work– To third party’s work– To your work
• How would I extend this paper?• What questions does it raise?• Future work of author.• What else?• Author log
Make yourself a template
Well-established class of problems, e.g., FO theorem proving, image retrieval etc.
Novel class of problems (is it really new?)
Single problem or many problems
Implicit or explicit new way of thinking about a problem?
• Author housekeeping stuff• Paper genre• Problem statement/motivation• Key ideas• Technical contribution• Technical flaws• Evaluation• Presentation• Comparison
– To authors’ other work– To third party’s work– To your work
• How would I extend this paper?• What questions does it raise?• Future work of author.• What else?• Author log
Make yourself a template
Implicit or explicit new way of doing things?
New, i.e., developed by authors?existing?
Developed by authors or others?
New combination of existing techniques?
Good or better/worse than X and why
• Author housekeeping stuff• Paper genre• Problem statement/motivation• Key ideas• Technical contribution• Technical flaws• Evaluation• Presentation• Comparison
– To authors’ other work– To third party’s work– To your work
• How would I extend this paper?• What questions does it raise?• Future work of author.• What else?• Author log
Make yourself a template
What is the author's thesis? What are they trying to convince you of?
Summarize the author's argument.
How does the author go about trying to convince you of the thesis?
• Author housekeeping stuff• Paper genre• Problem statement/motivation• Key ideas• Technical contribution• Technical flaws• Evaluation• Presentation• Comparison
– To authors’ other work– To third party’s work– To your work
• How would I extend this paper?• What questions does it raise?• Future work of author.• What else?• Author log
Make yourself a template
Does the author describe other work in the field?
If so, how does the research described in the paper differ from the other work?
• Author housekeeping stuff• Paper genre• Problem statement/motivation• Key ideas• Technical contribution• Technical flaws• Evaluation• Presentation• Comparison
– To authors’ other work– To third party’s work– To your work
• How would I extend this paper?• What questions does it raise?• Future work of author.• What else?• Author log
Make yourself a template• Author housekeeping stuff• Paper genre• Problem statement/motivation• Key ideas• Technical contribution• Technical flaws• Evaluation• Presentation• Comparison
– To authors’ other work– To third party’s work– To your work
• How would I extend this paper?• What questions does it raise?• Future work of author.• What else?• Author log
Empirical (run tests): test suite and testing must match problem targeted
Theoretical: correct and understandable/convincing and relevant
Does the paper succeed?
Are you convinced of the thesis by the time that you have finished reading the paper?
Make yourself a template• Author housekeeping stuff• Paper genre• Problem statement/motivation• Key ideas• Technical contribution• Technical flaws• Evaluation• Presentation• Comparison
– To authors’ other work– To third party’s work– To your work
• How would I extend this paper?• What questions does it raise?• Future work of author.• What else?• Author log
Does the author indicate how the work should be followed up on?
Does the paper generate new ideas?
What are your tips?
Reading groupsPrinting and physical markup
Not printing and electronic markupRead everything 3 times