“How many citations will I have to screen?”

I get asked this question a lot at the beginning stages of a review project. It’s a fair question: researchers want to know what their screening workload is going to look like, and screening abstracts is a tedious process.

There’s some great research happening right now on how to make screening less tedious – using text mining to automate or semi-automate the process, for example. These are promising approaches, but for now, most reviews need a human eye to oversee study selection for at least part of the process.

I used to get flustered when I was asked this question because I was afraid of giving the wrong answer and underestimating the amount of work a researcher would later have to do. On the other hand, if I overestimated the number of citations to screen, a researcher might want to change the search strategy to lower the number of citations or otherwise change the methodology.

I also didn’t have a very good answer. It’s hard to estimate the number of citations to be screened without downloading them from all sources and de-duplicating first. I’ve sometimes estimated the total number of included studies at the end of the project by screening a random sample (such as 100 citations) and calculating the ratio of the sample to the total number of citations to be screened (e.g. if there is one relevant article in a random sample of 100, and 1000 articles total to screen, I would estimate about 10 studies to be included at the end of the project – for more on getting a truly random sample of citations, see my previous blog post on this topic).

Tired of giving bad answers to this question, I’ve crunched the numbers for a few of the review projects I’ve worked on in the last year. For each of the projects, I found the number of citations downloaded from the Ovid MEDLINE search only, then the number of citations left to screen at the title/abstract stage after citations were downloaded from all databases and duplicates removed.

The results are below:Capture

The ratio of MEDLINE search only to total citations to screen ranged from 239% to 1333%. However, the last two columns represent projects that had less of a biomedical focus (social sciences and computer science, respectively). The MEDLINE searches were still relevant in these cases, but we didn’t expect the majority of our studies to come from MEDLINE. Thus, if we exclude the major outlier, with more of a social sciences focus, the results are a little more consistent.

For each of the projects above, I searched 6-7 databases, except project 1, where I searched 14(?!). However, the ratio of citations for project 1 is not exceptionally different than that of the other projects. For now, I can’t see a discernible difference between screening burden and number of databases searched, but possibly more data is needed.

My overall take-away from this exercise is that, for the searches that I run, the screening burden of a systematic review tends to be about 2.5x to 5x that of the original MEDLINE search. In the future, this is the advice that I’ll be giving my researchers to help them better plan their resources and time. I can breathe a small sigh of relief, too, knowing that the information that I give my researchers is just a little more evidence-based than it was the day before.


Finding a random set of citations in EndNote

Have you ever been asked to find a random set of citation from EndNote? This happens most often to me when researchers are testing out screening procedures, and want to ensure they are all interpreting the screening guidelines the same way. The researchers will all screen the same random set of 10-20 articles and compare results before screening the entire set.

So: what’s the best way to go about this? Sorting from a-z on any given field and selecting the top 10-20 articles isn’t likely to be truly random. For example, sorting by date will retrieve only very new or old articles. Sorting by record number is one possible way to do it, but also isn’t truly random as it will retrieve articles added to the database most or least recently.

Here’s how I take a truly random sample of citations from EndNote.

First, create an output filter in EndNote

The output filter will include only the citation record numbers. Don’t worry, you only have to do this once, and in the future it will all be set up for you!

  1. In EndNote, go to Edit –> Output Styles –> New Style
  2. In the resulting screen, click “templates” under the heading “bibliography”
  3. Then, put your curser in the box below “generic”. Then, click “insert field” –> “Record Number” –> then press enter so that you curser goes to the next line in the text box.
  4. Go to “file” –> “save as” and save it to something descriptive like “record-number-only”.

Next, export your record numbers.

  1. Back in the main EndNote screen, click the dropdown box at the top of the screen, then “select another style”, and search for your previously created Output Style.
  2. Then click “choose”. Ensure that your output style name is displaying in the dropdown box!
  3. Select “all references” to make sure all your references (that you want to create a subset from) are displayed. Then click one of the references and press ctrl + a (or cmd + a on a mac) to select all references.
  4. Right-hand click and select “copy formatted”.

Create your random subset!

  1. Open excel, and press ctrl + v (or cmd + v on a mac) to paste all your record numbers.
  2. in the cell to the right of your first record number, insert the formula =rand(). This will create a random number from 0 to 100.
  3. Hover the cursor over the bottom-right corner of the cell until it makes a cross. Then click and drag all the way down to the last row that contains a record number
    Insert a row at the top and click “sort & filter” –> “filter” on the menu bar.
  4. Then, sort the second row (with the random numbers) from smallest to largest (or largest to smallest).
  5. You now have a randomly sorted list! Select and copy the top x number of cells in the first column (however large you want your sample to be).

Format your record numbers to put back into EndNote.

  1. Paste your subset of record numbers into word (paste as text, not a table!)
  2. Click “replace” on the main toolbar to bring up the find and replace box.
  3. Beside the box “find what”, write ^p (the up-carrot symbol followed by “p”).
    Beside the box “replace with”, insert a semi-colon followed by one space.
  4. Then click “replace all”.
  5. You should have a string of record numbers separated by semi-colons.

Put them back into EndNote!

  1. Go back to your EndNote Library.
  2. Right-hand click in the sidebar and select “create smart group”
  3. Give it a nice title, like “random set” 😃
  4. In the first dropdown box, select “record number”, then “word begins with”, then paste in your formatted record numbers separated by semi-colons.
  5. Click “create”.
  6. All done!

I hope you found this useful. It might sound complicated, but this process really only takes a few seconds once you have gone through it a few times.

Do you have a more efficient or a different way of doing it? What kinds of formatting and database problems do you come across in your position? Feel free to send me a message or tweet at me.

Til next time,


I was asked recently on Twitter how to isolate the remainder of citations for screening after using this method. It’s very easy!

creating a combination group in EndNote
creating a combination group in EndNote

To isolate the rest of your citations, simply make a combination group by doing the following:

  1. First, create a new group in EndNote calls “All Refs”, and drag ALL your citations from the library into it by going to “All References”, selecting all by clicking ctrl/cmd + A, and dragging them into your new group.
  2. Right-hand click on “My Groups” in EndNote, then click “Create From Groups”. Name this group “remainder to screen”, or something else that makes sense to you.
  3. In the first drop-down menu, select your “All Refs” group.
  4. Then select “NOT” from the boolean operators dropdown menu.
  5. Then select the group that holds your random subset in the second dropdown menu.

How to make screening less painful

You know that feeling when you are running searches for a patron, and want to pick out some of the most relevant papers for them, but it’s a Friday afternoon and your eyes are tired and zomg screening is the worst?

I’ve got a tip to make this process marginally less awful. This life-saving tip comes from a wonderful colleague at the College of Physicians and Surgeons of British Columbia.

First, run your search(es) and download your citations into EndNote (or another citation management program)* for screening. Generally, I only use this tip for general scoping (not for systematic review screening), so I usually end up downloading less than 200 citations for this process, and sometimes as few as 30.

Next, export your citations into a .rtf format with an export filter that includes both the citation information and the abstract. To do this in EndNote (v7), go to the dropdown menu at the top of the page and choose “select another style…”, then search for “annotated”. Click the one with the category “generic”, then click “choose”. You will notice that the preview pane for each citation now contains the citation’s information and its’ abstract.

EndNote dropdown menu and preview pane

Next, export your references by clicking the blue arrow on the top bar. First, press ctrl + a to select all the references. Then save the filetype as .rtf and select “annotated” as your output style. Save the file wherever, then navigate to that folder and open it. It should automatically open in Microsoft Word (or the word processing program of your choice). The file should contain all of your references, with abstracts.

exporting your references from EndNote

Now comes the fun bit! Press ctrl + H (or click “replace” in the main top bar). Under “find what”, type one of the main terms for your first concept. Then click anywhere in the “replace with” box, but instead of typing anything, click “More >>” to expand the options, then click the “format” dropdown box, then “highlight”. The word “highlight” ought to appear below the “replace with box”.

Find and Replace in Microsoft Word

Still with me? Okay. Click “replace all”. Repeat this step with other terms that might be found in the titles and abstracts of the citations (but only for your first concept!). Once you have reached relative saturation, click the highlighter icon in the main top bar, and select a different highlighter colour. Next, repeat the same process as above with your second main concept, until you have reached relative saturation.

Ta da! At this point, you ought to have a pretty colour-coded document which helps you easily see the main concepts from your search. Screening this word document will be much less straining on the eyes and take less time because the main concepts have already been identified for you.

Word document, ready for screening

This trick works better for some topics than others. My example above which uses the concepts of caring and attachment works pretty well. However, complex interventions or other areas with ever-changing terminology might not work as well.

Pro tip: in some cases, it is useful to send this colour coded document to your patron, and let them make decisions about what citations are relevant.

Another pro tip: instead of formatting with a highlighter, which only comes in garish colours (why, microsoft? why??), you can also format the text in any way you want. For example, you can put the relevant terms in bold or italics, or make the text itself different colours.

That’s it for today. Have you ever done this, or something similar? Do you have any protips for screening more quickly and efficiently? Send them to me on Twitter or through the Contact Me form!

* But seriously, if you’re not using EndNote, get on that.