Introduction {#s1}
============

The Internet has emerged as an important tool for modern clinical practice. It provides health professionals with access to current, best evidence and facilitates the translation of research to clinical care. According to a recent survey, nephrologists search for medical information to guide the treatment of patients at least once per month \[[@SFS152C1]\]. To access best evidence, physicians can no longer rely on browsing a few key journals as relevant literature is widely dispersed. In nephrology alone, relevant literature is published across over 400 journals \[[@SFS152C2]\].

Numerous tools are available to healthcare professionals when searching for primary literature online. These include subscription-free databases such as PubMed, Google Scholar and Scirus (which are freely accessible via the Internet), and subscription-based bibliographic databases such as EMBASE, MEDLINE and ISI Web of Knowledge. Subscription-free resources enable global access to important clinical information. This is especially important in developing nations where physicians and institutions may lack the funds needed to maintain database and journal subscriptions.

While nephrologists increasingly rely on bibliographic databases for clinical information, evaluations of database performance are limited \[[@SFS152C3], [@SFS152C4]\]. Evaluating the performance of a bibliographic database involves the consideration of several factors \[[@SFS152C5]--[@SFS152C8]\]. For example, an important article may not be retrieved from a database such as PubMed because (i) the article of interest is not contained in the database holdings, or (ii) the search strategy failed to retrieve the article. To thoroughly evaluate the search performance of a database, it is first necessary to consider the 'content coverage', which quantifies the availability of relevant literature contained in the database. To date, no such evaluation has been performed for renal content \[[@SFS152C9]--[@SFS152C14]\].

Here, we evaluate and compare the availability of renal literature in six major bibliographic databases (PubMed, Google Scholar, Scirus, EMBASE, Ovid MEDLINE and ISI Web of Knowledge). We searched each database for 2195 journal articles, meeting articles and conference abstracts pertinent to renal medicine. We compared the searching features and content coverage among the subscription-free and subscription-based databases, and also compared how often full-text journal articles were available for free in the subscription-free databases.

Materials and methods {#s2}
=====================

Bibliographic database characteristics {#s2a}
--------------------------------------

We considered six databases: three subscription-free databases (PubMed, Google Scholar and Scirus) and three subscription-based databases (EMBASE, Ovid MEDLINE and ISI Web of Knowledge). We searched official home pages to determine the content and searching characteristics of each bibliographic database \[[@SFS152C15]--[@SFS152C22]\].

Set of renal studies {#s2b}
--------------------

A content coverage analysis quantifies the proportion of important literature in a specific field that is contained in a specific database \[[@SFS152C6]\]. As users in different medical disciplines have specialized needs, discipline-specific evaluations are the most informative. To avoid bias, we identified studies independently of the bibliographic databases to be tested by extracting the citations of the primary studies included in high-quality, renal systematic reviews. A well-conducted systematic review uses a variety of methods to identify all high-quality primary studies for a particular clinical question.

We compiled renal systematic reviews from peer-reviewed journals using the EvidenceUpdates service \[[@SFS152C23]\]. This service directs users to high-quality systematic reviews and meta-analyses that meet strict methodological criteria and have a high potential for clinical relevance \[[@SFS152C16]\]. We searched the EvidenceUpdates service in November 2009 and selected the option to view all reviews for the discipline of nephrology; our search yielded 207 systematic reviews. Two nephrologists used a standardized checklist previously developed by our team to independently assess whether each systematic review was pertinent to renal care \[[@SFS152C24]\]. A sample of 151 renal reviews was identified and included in the study (Appendix eList 1). The reviews were published between 1995 and 2009; 146 (97%) reviews targeted therapies for renal conditions. For each included review, we extracted the citations of all the primary studies. This included abstracting the full title, including any non-English titles, author names, year of publication, page numbers and journal or meeting title. We also noted whether the citation referenced a journal article, an article from a meeting or a meeting abstract. In total, we extracted 2838 citations to primary studies from all reviews. After removing duplicates, 2195 unique citations remained, which included 1577 (72%) journal articles, 280 (13%) meeting articles and 338 (15%) meeting abstracts; all published between 1963 and 2008. Among the journal articles, 34 (2%) were published in a language other than English.

Study availability in each bibliographic database {#s2c}
-------------------------------------------------

We searched for each citation in each bibliographic database (all searches performed between March and October, 2010). Our method of searching was similar to previous coverage studies \[[@SFS152C12], [@SFS152C13]\]. Using advanced search features where necessary, one author searched for each study in each of the six major scientific databases: PubMed, Google Scholar, Scirus, EMBASE, Ovid MEDLINE and ISI Web of Knowledge. This involved various combinations of the manuscript\'s title (both English and non-English titles), the authors\' names, journal title and the year published. All links to candidate matches to confirm a true match were followed. In Google Scholar, we always selected the option to view all versions of a candidate article, and all links were attempted. If a citation was not found in a database, a second rater performed further searches to confirm its absence.

For each found article, two raters recorded information about the citation (authors, journal name, issue, volume and date of publication) and whether the complete abstract was available. In addition, when searching for journal or meeting articles in the three subscription-free databases, the raters documented whether the link(s) provided to access the article resulted in free full-text viewing, or if payment or subscription was required for full-text viewing. When searching the subscription-free databases, we ensured that no-pay-for privilege access was used. For articles not found in any of the databases, we further searched the Cochrane Library database to determine whether the articles were included in their holdings.

Analyses {#s2d}
--------

We compiled and analyzed all citations using Microsoft Excel 2007 (Microsoft Corporation, Redmond, WA). We compared the proportion of journal articles, meeting articles and meeting abstracts contained in the six databases. To assess potential biases in coverage, we further considered articles published in a language other than English and articles published before 1995 \[[@SFS152C8]\] (this year marked milestones in computer and Internet advancements: Yahoo! was incorporated and Microsoft introduced the Windows 95 operating system and Internet Explorer version 1.0) \[[@SFS152C17], [@SFS152C25]\]. For subscription-free databases, we also compared the proportion of full-text articles that were available free of charge. As PubMed is the most frequently used database by nephrologists \[[@SFS152C1]\], we further examined the articles not included in this database and analyzed whether other databases contained these missed articles in their holdings.

We used the chi-squared test to determine whether the proportions of articles available in the databases were statistically different. If a test yielded a significant result, we conducted pairwise tests between the database with the largest yield and the remaining four sources. To reduce the risk of Type I error, we used a conservative P-value of 0.01 to interpret significance for all comparisons. We used the Wilson\'s score method to calculate 95% confidence intervals (95% CI) for all proportions \[[@SFS152C26]\].

Results {#s3}
=======

General characteristics of the six databases are contrasted in [Table 1](#SFS152TB1){ref-type="table"}. Table 1.Characteristics of six bibliographic resourcesCharacteristic^a^PubMed^b^Google Scholar^b^Scirus^b^EMBASE (Ovid)^c^MEDLINE (Ovid)^c^ISI Web of Knowledge^c^ContentNumber of journalsOver 23 000 cited; 5511 indexed (through MEDLINE)NANAOver 5000551123 000DisciplinesBiomedicalMultidisciplinaryMultidisciplinaryBiomedicalBiomedicalMultidisciplinaryPeriod covered1948--presentNANA1974--present1948--present1864--presentNumber of entries20 423 752NA410 million science-specific web pagesNearly 13 million19 398 391NABiomedical databases includedMEDLINE, OldMedline, 'in-process' and 'out-of-scope' citationsNAPubMed, LexisNexis, ScienceDirect, otherEMBASEMEDLINE, OldMedline, 'in-process' and other non-indexed citationsWeb of Science, Biological Abstracts, BIOSIS Previews, ISI Proceedings, MEDLINEFrequency of updatesDailySeveral times a weekNADailyDailyDailyDeveloper/owner or provider and countryNational Library of Medicine, USAGoogle Inc., USAReed Elsevier, The NetherlandsDeveloper: Elsevier\
Provider: Wolters Kluwer\
NetherlandsWolters Kluwer, The NetherlandsThomson Reuters, CanadaSearch FeaturesNumber of entries which can be displayed/accessed for each searchNo limit10001000No limitNo limitNo limitTracks the number of times articles are cited by other publicationsNoYesNoNoNoYesAllows users to view citing articlesNoYesNoYesYesYesAllows use of controlled vocabulary (e.g. MeSH terminology)YesNoNoYesYesYes (limited)Indicates whether articles are available as free full textsYesYesNoN/AN/AN/AAllows article citations to be imported into bibliography managers (e.g. Reference Manager)YesYes (can import only one citation at a time)NoYesYesYesProvides searching by limits (e.g. age, publication type)YesYes (very limited)Yes (very limited)YesYesYesAllows linking to institutions for subscription access (e.g. link to university library)YesYesYesYesYesYesProvides email alerts for prespecified searchesYesYes (introduced in 2010)NoYesYesYesAllows users to view related articles for an article of interestYesYesYesYesYesYes (not all records)Allows access to search filters (e.g. clinical queries)YesNoNoYesYesNoWhen searching, algorithm searches the full text of publicationsNoYes (not all records)Yes (not all records)NoNoNoSorts results by relevanceNoYesYesYes (very limited)Yes (very limited)NoProvides spell checking for misspelled search termsYesYesYesNoNoNoStores search historyYesNoNoYesYesYes[^1][^2][^3][^4]

Content coverage {#s3a}
----------------

We describe the results separately for journal articles, meeting articles and meeting abstracts.

### Journal Articles {#s3a1}

The proportion of journal articles contained within the holdings of each of the six databases were similar, ranging from 96 to 97% ([Figure 1](#SFS152F1){ref-type="fig"}); content coverage did not differ between the subscription-free databases and the subscription-based databases (P = 0.6). All databases contained similar proportions of articles published before 1995 (88--94%), while the few articles published in languages other than English (*n* = 34) were most prevalent in EMBASE (82%) and least prevalent in ISI Web of Knowledge (29%); P \< 0.01. Five journal articles were not contained in any of the databases; we examined the methods of the originating reviews and determined that three of the citations originated from a specialized database in China \[[@SFS152C27]\], and two were found via hand searching \[[@SFS152C28], [@SFS152C29]\]. Fig. 1.Proportion of all journal articles, older articles and non-English language articles included in the six bibliographic databases. Error bars represent 95% confidence interval calculated using the Wilson\'s score method. \*Global chi-squared test: P = 0.6. \*\*Global chi-squared test: P \< 0.01; pairwise chi-squared tests compared with Google Scholar--EMBASE: P \< 0.01, Scirus: P = 0.4, PubMed: P = 0.4, Ovid MEDLINE: P = 0.4, ISI Web of Knowledge: P = 1. \*\*\*Global chi-squared test: P \< 0.01; pairwise chi-squared tests compared with EMBASE--Google Scholar: P \< 0.01, Scirus: P = 0.5, PubMed: P = 0.5, Ovid MEDLINE: P = 0.5, ISI Web of Knowledge: P \< 0.01. Detailed breakdowns of the formats (citation only, full abstract, subscription or payment full-text option, free full text) that the articles were found in the subscription-free databases are presented in the [Supplementary data Figure S1](http://ndtplus.oxfordjournals.org/lookup/suppl/doi:10.1093/ckj/sfs152/-/DC1).

Of the 1577 journal articles, 60 (4%) were not contained in PubMed, 43 (72%) originated from journals not indexed by the database and 14 (23%) were published in years when the source journals were not indexed by PubMed. Failure to locate the final three articles in PubMed resulted from errors in indexing. While the database indicated that the originating journal volumes were available (and other articles from the journals volumes were found), the three articles could not be located. EMBASE contained the largest proportion of articles not indexed by PubMed (73%; [Table 2](#SFS152TB2){ref-type="table"}). When combined, EMBASE and PubMed contained 99% of all journal articles. Table 2.Proportion of articles not contained in PubMed that were contained in the other bibliographic databasesProportion (%) of articles (*n* = 60)Ovid MEDLINE0Scirus16ISI Web of Knowledge40Google Scholar53EMBASE73

### Meeting articles and meeting abstracts {#s3a2}

Table [3](#SFS152TB3){ref-type="table"} lists the proportion of meeting articles and meeting abstracts included in the six databases. While over 98% of meeting 'articles' were contained in all the databases, the distribution of meeting 'abstracts' showed significant variation, ranging from 0 to 37% (P \< 0.01). ISI Web of Knowledge contained the largest proportion of meeting abstracts (37%, 95% CI 32--43%). The largest source of meeting abstracts among those found by ISI Web of Knowledge originated from the American Society of Nephrology Renal Week meetings. When combined, 57% of abstracts were not included in any of the databases. Further investigation revealed that the Cochrane Library database included 162 of the 193 (84%) abstracts not found by any of the databases. Table 3.Proportion of meeting articles and meeting abstracts included in the six bibliographic databasesProportion (%) of articles (95% CI^a^)Meeting articles (*n* = 280)^b^Meeting abstracts (*n* = 338)^c^EMBASE100.0 (98.7--100.0)0.0 (0.0--1.1)Scirus99.6 (98.0--99.9)4.2 (2.5--6.8)ISI Web of Knowledge99.3 (97.4--99.8)37.3 (32.3--42.6)Google Scholar99.3 (97.4--99.8)9.5 (6.8--13.1)PubMed98.9 (96.9--99.6)0.0 (0.0--1.1)Ovid MEDLINE98.9 (96.9--99.6)0.0 (0.0--1.1)[^5][^6][^7]

### Availability of free full-text publications {#s3a3}

Among the subscription-free databases, free full-text access to journal articles (via links to external sources offering free full-text access) was greatest in Google Scholar (38%), followed by Scirus (36%) and PubMed (33%); P \< 0.01 ([Figure 2](#SFS152F2){ref-type="fig"}). A similar pattern was seen for meeting articles (P = 0.3; [Figure 2](#SFS152F2){ref-type="fig"}). Altogether, free full-text access to journal articles among the subscription-free databases was 39%, with free access increasing after the year 2000 when almost half of the articles were available for free ([Figure 3](#SFS152F3){ref-type="fig"}). Fig. 2.Proportion of journal articles and meeting articles available as free full texts in Google Scholar, Scirus and PubMed. Error bars represent 95% confidence interval calculated using the Wilson\'s score method. \*Global chi-squared test: P \< 0.01; pairwise chi-squared tests compared with Google Scholar--Scirus: P = 0.2, PubMed: P \< 0.01. \*\*Global chi-squared test: P = 0.3. Fig. 3.Proportion of journal articles available as free full texts in Google Scholar, Scirus or PubMed by era.

Discussion {#s4}
==========

In this study, we evaluated the availability of high-quality, renal practice evidence in six major bibliographic databases (PubMed, Google Scholar, Scirus, EMBASE, Ovid MEDLINE and ISI Web of Knowledge). We found that over 99% of renal journal articles were contained in at least one of these databases.

We were surprised to learn that the availability of renal journal articles did not differ between subscription-free databases and subscription-based databases. Further, nearly 40% of full-text journal articles could be accessed freely through links provided by the subscription-free databases. Free access to full-text articles increased from 20 to 49% for articles published before 1980 compared with articles published in 2000 or later. These findings indicate that journals appear slow to grant free access to older materials, which typically involves conversion of print materials to electronic PDF files. The increasing availability of free access to literature is particularly important for developing nations where physicians and institutions may lack the resources needed to maintain subscriptions to databases and journals \[[@SFS152C30]\]. Even in developed countries such as Canada and the United States, the burden of paying for knowledge is felt: some academic databases and journals have raised their fees for university subscriptions by up to 400% over a 1-year period \[[@SFS152C31]--[@SFS152C33]\].

In contrast to the excellent coverage of journal articles, we found that meeting abstracts are poorly indexed. Abstracts were nonexistent in most databases, with the exception of ISI Web of Knowledge, which indexed 37% of meeting abstracts. This finding suggests that authors of systematic reviews are not solely relying on bibliographic databases but are also searching conference websites and supplements as is recommended. To assist with comprehensive searching for reviews, The Cochrane Collaboration provides detailed guidelines on searching for gray literature (including hand searching) through the Cochrane Handbook, available online at <http://www.cochrane-handbook.org/>.

To our knowledge, this is the first study to quantify the content coverage of renal literature in the major bibliographic databases. While previous evaluations of other medical disciplines showed greater variation in content coverage, we found very little variation with respect to journal articles \[[@SFS152C13], [@SFS152C14]\]. For the purpose of conducting systematic reviews, numerous studies recommend searching several bibliographic databases to develop comprehensive reviews \[[@SFS152C13], [@SFS152C34]--[@SFS152C40]\]. Although our results indicate only a minor increase in content yield when searching multiple databases (the greatest increase in yield was only 3% when combining MEDLINE and EMBASE), these results only reflect differences in the renal content of the databases. Differences between this study and earlier evaluations may be due to improved database performance over time, possibly due to competition between databases.

We designed this study to overcome the limitations of previous publications. We used systematic reviews to identify an unbiased, representative set of renal clinical studies. However, due to the lengthy systematic review process, publication process and required time to complete the current study, this resulted in articles being tested that were at least 2 years old and were cited by at least one publication. Nonetheless, we reduced the likelihood that possible selection bias influenced the results by using this objective method of sampling renal literature. Because we selected primary studies included in systematic reviews, we were unable to test the coverage of articles that some physicians find relevant (studies of lower methodological quality, narrative reviews, case reports, animal studies, commentaries etc.). However, we engaged in the widely accepted principles of the hierarchy of evidence to identify the most important articles to retrieve when answering clinical questions. Having chosen articles from English language systematic reviews, we likely limited the number of non-English renal articles in our sample (only 34 journal articles were published in a language other than English). Thus, our results may not generalize well to non-English renal content. Finally, we acknowledge that the Internet is a dynamic environment and encourage future research to evaluate whether the performance of databases varies with time, as well as to monitor the improvement of the online sources.

As shown in Table [1](#SFS152TB1){ref-type="table"}, bibliographic databases vary with respect to search features and each offers certain advantages. All the databases allow access to institutional library links, which can be used to retrieve the full text or publications within their library\'s subscription packages. In general, the subscription-based databases provide a more structured interface, which enables more complex searches, such as combinations of two or more searches, use of complex search filters and a controlled vocabulary. They also allow a user to search several databases simultaneously and provide more options for efficient data management. In contrast, the subscription-free databases, such as Scirus and Google Scholar, offer a simpler search interface, but limit the number of entries, which can be displayed or accessed for each search. With the exception of PubMed, subscription-free databases also lack specialized features for clinicians. PubMed, EMBASE and MEDLINE provide indexed content that is directly relevant to clinicians, including extensive clinical controlled vocabularies (e.g. MeSH), limits, access to discipline-specific search filters and the 'Clinical Queries' filters, which help clinicians find articles of high methodological quality for clinical questions of therapy, diagnosis, prognosis and etiology \[[@SFS152C15], [@SFS152C41]--[@SFS152C43]\].

Content coverage analysis is only the first necessary step in evaluating the quality of bibliographic databases. A comprehensive content coverage of a database does not directly translate into better search performance; indexing database systems, syntax, searching capabilities and available limits all influence the successful retrieval of evidence. For example, the EMBASE database is well known for its extensive indexing of studies in pharmacology and drug therapy: a search performed in March 2011 for the term 'valsartan' identified 6620 records in EMBASE and only 1852 in PubMed. Future research should focus on the other properties of bibliographic databases that are of interest to clinicians, namely the ability to retrieve relevant literature when conducting a search, the relative effort required to retrieve this literature and the ability to find newly published evidence.

In conclusion, the major bibliographic databases appear sufficient in indexing the majority of renal articles. However, poor indexing of meeting abstracts means that a comprehensive search will require searching other sources in addition to bibliographic databases. Finally, nephrologists searching for full-text articles should first search the subscription-free databases before paying for access.

Supplementary Data {#s5}
==================

[Supplementary data is available online at](http://ndtplus.oxfordjournals.org/lookup/suppl/doi:10.1093/ckj/sfs152/-/DC1) <http://ckj.oxfordjournals.org>.

Conflict of interest statement {#s6}
==============================

None declared.

Supplementary Material
======================

###### Supplementary Data

Dr. Shariff was supported by a Canadian Institutes of Health Research (CIHR) Doctoral Research Award and the Schulich Scholarship for Medical Research from the Western University. Dr. Garg was supported by a Clinician Scientist Award from the CIHR.

Funding. Funding for this study was provided by The Physicians\' Services Inc.

[^1]: NA, not applicable.

[^2]: ^a^As of March, 2011.

[^3]: ^b^Subscription-free databases available for free access via the Internet.

[^4]: ^c^Subscription-based database; requires a subscription to access.

[^5]: ^a^CI, confidence interval calculated using the Wilson\'s score method.

[^6]: ^b^Global chi-squared test: P = 0.6.

[^7]: ^c^Chi-squared test: P \< 0.01; all pairwise chi-squared tests compared with ISI Web of Knowledge: P \< 0.01.
