My motive for reading any given paper is generally not “for the sake of reading the paper from start to finish”. And these days I am not reviewing any papers for journals or conferences. So I start by identifying my purpose.
Prerequisite #0: Define your purpose.
- Am I trying to learn a new area of expertise?
- Am I working on a design and want to understand other approaches?
- Am I trying to identify best practices?
- Am I trying to identify and evaluate the correctness, efficiency, and usability of the newest research in a given niche?
- Am I working on a specific problem and want to understand how others have solved related problems?
- Do I have a specific list of questions that I am trying to answer? Should I draft such a list?
Prerequisite #1: Find a paper to consider reading.
- If you are trying to learn a new area:
- Try to find a dissertation on the topic, preferably recent, but older dissertations can be quite useful launch points, both for understanding and for further reading.
- Dissertations usually contain background chapters that present the material in an intelligible way to a non-expert (where here, non-expert just means “someone not in the niche Y of discipline X”).
- Even if the related work section is dated, dissertations often include pointers to the seminal papers in the field, which can be extremely helpful to read.
- Find the seminal paper or papers in the field. Use a search engine that can give you a list of work that cites these papers. Start with more recent citations, surveys, and systematizations of knowledge (SoKs).
- Identify top venues.
- Try to find a dissertation on the topic, preferably recent, but older dissertations can be quite useful launch points, both for understanding and for further reading.
- If you are not (completely) new to the area:
- Track publications at top venues. If your job involves staying on top of current research, devote some time to a cursory read of titles and abstracts. Also see Guideline #3.
- Devote some time, regularly, to a cursory perusal of pre-print abstracts.
- Join a forum that has an active reading discussion group.
Guideline #1:First make sure you know whether and why you want to read the given paper.
- Why do I think I should consider reading the specific work in question, e.g.,
- Am I trying to learn a new area of expertise?
- Does the paper contain a comprehensive overview of the literature, or at least friendly-looking background and related works sections? (Papers that do not have these features are, in my experience, not the best place to start.)
- Does the list of references seem comprehensive and recent? Do I think the paper will act as a guide for further reading?
- Is the paper a seminal work in the field? Do lots of other papers rely on it?
- Where was the paper published? Who are the authors? Do I have any faith in the quality level of the paper based on this information?
- Am I deciding to implement a protocol related to the one developed and analyzed in the paper?
- The protocol from the paper itself?
- Or a protocol that solves the same or a similar problem?
- Am I trying to solve a related problem?
- Would it suffice to understand the claims in the paper at a high level, contextualized against other, related papers?
- Am I trying to learn a new area of expertise?
- Why do I want to read the specific work in question, e.g.,
- Does the paper seem interesting or well-written?
- Does it contain implementation considerations and pitfalls?
- Does it claim to solve a problem I think is cool or particularly impactful?
- Are the authors likely to have a unique perspective on the topic?
- Do I know the authors personally and want to support them by engaging with their work?
- Do I know the authors from previous work and know from experience that I am likely to glean something useful from reading the paper?
- Does the paper seem interesting or well-written?
Guideline #1 and Prerequisite #0 have some overlap, but I strongly believe it is useful to iterate on your understanding of your purpose through repetition.
Guideline #2: Reading a paper thoroughly from start to finish is often not the most effective way of accomplishing your purpose.
S. Keshav has a page on reading papers in three passes. This is a good place to start. I tend to read papers in multiple passes, sometimes over the course of years. Here is my routine:
- Pass #0: Many papers don’t get more than a casual perusal of the abstract, authorship list, and publication venue. Papers most often remain in this category:
- When the paper sounds just vaguely interesting because of the title, venue, or authorship list, but I don’t have any real need, interest, or time to delve deeper.
- When the answer to the question “Would it suffice to understand the claims in the paper at a high level, contextualized against other, related papers?” is Yes. Hopefully the abstract spells out the claims in sufficient detail; I can always return to this paper later: see Guideline #3.
- Pass #1:
- Familiarize yourself with the structure of the paper: what are the headers and subheaders? Can you already tell what the paper claims to show? If so, identify the claims explicitly.
- Read some bits and pieces: I start with the introduction, skim through any theorem statements or claims, and read the conclusion.
- Identify the claims and research performed. At this point, I usually stop reading if I can’t identify the claims or experiments performed, or find the referenced implementation.
- Evaluate plausibility of claims: Determining correctness can be difficult. I find it often involves a thorough reading of the paper and a relatively deep understanding of the literature and field in question. For this pass, I focus on plausibility.
- Do I think the results are reasonable or persuasive?
- Are there any red flags? e.g., Are graphs and figures intelligible? Are clear definitions and theorem statements given? Does the list of references seem reasonably complete? Does the full version of the paper actually exist? How about the implementation?
- If I don’t know the field well, sometimes I leave this at “I can’t really tell, but at least it was peer-reviewed at a top venue in Field X”, etc.
- Pass #2: Sometimes I check to see if a talk recording is available and watch it instead of, or as part of, this pass. And no, I don’t feel the need to watch the video from beginning to end or at normal speed. But on this pass:
- Contextualize the paper against related work: Are the claims as interesting as the abstract and introduction attempt to portray? What don’t the authors address in this paper? What are the pros and cons of this approach versus related work?
- Read the discussion section.
- Read the related work section.
- Take notes: Identify and note down any insights gleaned or references that look intriguing.
- Answer your questions: If you have a list of questions you are trying to answer, did this paper help you answer any of them? If so:
- Track this.
- Verify (to the best of your ability) that the result you are relying on is correct. If you can’t tell, more background research, thinking, and hopefully, discussion with others, is required.
- Contextualize the paper against related work: Are the claims as interesting as the abstract and introduction attempt to portray? What don’t the authors address in this paper? What are the pros and cons of this approach versus related work?
- Pass #n for n \geq 3: These passes align pretty closely to Keshav’s “third pass”. This is where I try to understand the details of the paper, verify the correctness of any results of interest, try to identity where I agree or disagree with unstated assumptions, etc. But this is not a single pass for me and it’s almost never front to back:
- If I had a specific question, e.g., if I wanted to verify a particular result or check my understanding of a particular concept, it’s helpful to pinpoint that part of the paper and only focus on the relevant parts.
- A friend of mine from mathematics advises:
- “If I’m looking to check a particular fact in a reference, I’ll try to start at the particular theorem and gradually expand my scope until everything I need is defined.”
- “Otherwise if I’m generally interested in the paper, I’ll do multiple passes. First pass is literally flipping the pages to just see the layout and the headings, then progressively more detail on each pass. But also on each pass there’s a point where I give up on absorbing/understanding, and that point is further along on each pass.” (Emphasis mine.)
Guideline #3: Use a paper management software system that includes decent search functionality, take organized notes for papers you read in any depth, and consider your equipment choices.
Many of these systems have deeply disturbing privacy implications if you enable cloud features rather than just running locally, however. I don’t advise ignoring this issue entirely, because intellectual freedom thrives in the absence of surveillance.
But it’s very helpful to be able to efficiently search through a large set of papers you’ve spent some time reading in the past. And taking notes on each pass (either directly in the pdf, or in some other form) can help you return to a paper later, regain context quickly, and achieve a deeper understanding on your next pass more efficiently.
To that end, consider what medium and tools work best for your understanding:
- Have paper and a writing implement handy to sketch out mathematics, figures, and questions. White boards are nice, but keep in mind that you eventually erase white boards. (You might need alcohol to do so, yes, but eventually.)
- Experiment with different mediums. I used to be addicted to paper, and there are still no hand-held devices that have appropriate dimensions for paper-reading in my opinion, but there is an assortment of e-readers, tablets, and monitors to consider.
- I avoid reading papers in depth on my computer; I just don’t absorb the content as easily as in printed form or on an e-reader device.
- But when I do read on the computer—which is often the case for passes #0–2—I want the text to be as crisp as possible, so I invested in a capable monitor. I wouldn’t trade crisp text for anything.