Athorough review of existing scientific literature is

Transcription

1 Beyond MEDLINE for Literature Searches Vicki S. Conn, Sang-arun Isaramalai, Sabyasachi Rath, Peeranuch Jantarakupt, Rohini Wadhawan, Yashodhara Dash Clinical Scholarship Purpose: To describe strategies for a comprehensive literature search. Organizing Construct: MEDLINE searches result in limited numbers of studies that are often biased toward statistically significant findings. Diversified search strategies are needed. Methods: Empirical evidence about the recall and precision of diverse search strategies is presented. Challenges and strengths of each search strategy are identified. Findings: Search strategies vary in recall and precision. Often sensitivity and specificity are inversely related. Valuable search strategies include examination of multiple diverse computerized databases, ancestry searches, citation index searches, examination of research registries, journal hand searching, contact with the invisible college, examination of abstracts, Internet searches, and contact with sources of synthesized information. Conclusions: Extending searches beyond MEDLINE enables researchers to conduct more systematic comprehensive searches. JOURNAL OF NURSING SCHOLARSHIP, 2003; 35:2, SIGMA THETA TAU INTERNATIONAL. [Key words: literature searching, publication bias, research synthesis] * * * Athorough review of existing scientific literature is essential for determining the evidence for practice and for designing research that will fulfill the potential of cumulative knowledge. Many nurses conducted literature searches as students, but usually those searches were of limited scope. Often more comprehensive searches are necessary to fully assess the state of existing knowledge. This paper is focused on diverse strategies to conduct an extensive literature search. Rationale for Comprehensive Literature Searching The volume of scientific literature has grown exponentially in the last 50 years. Practicing nurses are expected to base their nursing care on scientific evidence. Nurse researchers strive to conduct studies that will provide the scientific evidence to improve nursing care. Neither can fully meet their goals if the literature they retrieve is narrow in scope or is not representative of existing science. Finding the existing science is challenging. Many nurses begin the search with computerized databases, typically MEDLINE, CINAHL, and PsycINFO. These are excellent starting points. Unfortunately, searches that include only these sources exclude many valuable studies and the research reports retrieved by these databases will be a biased sample of existing studies. If the studies that were easily retrieved through the most popular computerized databases accurately represented all research, searching these databases would be adequate. However, extensive evidence shows that these studies more often include statistically significant findings than do equally rigorous research found in other locations (Chalmers et al., 1990; Dickersin, Chan, Chalmers, Sacks, & Smith, 1987; Dickersin, Min, & Meinert, 1992; Easterbrook, Berlin, Gopalan, & Matthews, 1991; Gotzsche, 1989; Hubbard & Armstrong, 1997; Kleijnen, & Knipschild, 1992; Rosenthal, 1979; Simes, 1986, 1987; Serling, 1959; Serling, Rosenbaum, & Weinkam, 1995; Sugita, Kanamori, Izuno, & Miyakawa, 1992). Limited searches are also less likely to identify studies with small samples, which are often intriguing pilot studies of innovative interventions or studies conducted with difficultto-recruit samples (Chalmers et al., 1990; Dickersin et al., 1987; Easterbrook et al., 1991; Scherer, Dickersin, & Langenberg, 1994; Thornton & Lee, 2000). Thus reviewing only the easily located studies leads to a biased view of existing Vicki S. Conn, RN, PhD, Alpha Iota, Potter-Brinton Professor & Associate Dean for Research, Sinclair School of Nursing, University of Missouri, Columbia, MO; Sang-arun Isaramalai, RN, PhD, Assistant Professor, Prince Songkla University, Songkla, Thailand; Sabyasachi Rath, MBBS, Graduate Student, Health Informatics, Peeranuch Jantarakupt, RN, MS, Alpha Iota, Doctoral Candidate, School of Nursing, Rohini Wadhawan, BA, BDS, Graduate Student, Health Services Management, Yashodhara Dash, MBBS, Graduate Student, Health Informatics, all at the University of Missouri, Columbia, MO. Financial support was provided by a grant from the NIH NINR RO1NR07870 to Vicki Conn, principal investigator. Correspondence to Dr. Conn, S317 School of Nursing-MU, Columbia, MO conn@missouri.edu Accepted for publication January 7, Journal of Nursing Scholarship Second Quarter

2 research. A balanced review of existing studies requires more extensive searching to locate a broader scope of research. Comprehensive search strategies are needed to locate large numbers of studies for extensive or systematic reviews, such as would be necessary for a valid meta-analysis. More limited strategies might be appropriate for other types of literature reviews. Managing a Comprehensive Literature Search A clearly articulated plan is essential for conducting a comprehensive search. A written search protocol is helpful, based on preliminary perusing of computerized databases related to clear research questions. The objectivity of the review will be enhanced if a priori inclusion criteria and terms are specified. For example, in a search for studies of interventions to increase physical activity, two concepts would be required if the title were: physical activity and intervention. Terms that would indicate physical activity include exercise, exertion, exercise therapy, physical education, physical training, aerobic, endurance, or fitness. Intervention terms include behavior, adherence, compliance, patient education, health promotion, health education, behavior therapy, lifestyle, program, evaluation, effect, outcome, intervention, or treatment. A system to manage the search process and products is necessary. Bibliographic software, such as EndNote or Reference Manager, is useful for both predefined and customized fields. For example, a field named status could be added to indicate whether the article has been obtained, or is needed from a library. Another field labeled outcome measure might indicate specific outcomes of interest for the review. Term lists that are linked to fields can be created to structure the language. Structured language can be helpful for later searching within the data file. The nature of the review will determine whether multiple fields and lists of associated terms are worthwhile. Highly differentiated fields and standardized term lists facilitate a comprehensive meta-analysis of a field of research with many potential eligible studies. Bibliographic software can be used to record when report-linked search and review activities have been completed, such as ancestry searches, coding, or author contacts. Overlap between products of search strategies will likely be extensive. Bibliographic software that can locate duplicate entries is useful to make the process as efficient as possible. A staged method of determining that studies are eligible for the review can be helpful (Cook et al., 2001). Potential eligibility would indicate a research report with a title or abstract indicating that the study might meet the criteria for inclusion. This judgment can often be made from abstracts when studies are available through computerized databases. Probable eligibility would be determined after reading the entire research report. Definite eligibility would occur when information from the research report is determined to be useful in the review. For example, in literature reviews for a metaanalysis, definite eligibility would be declared when datacoding shows that the independent and dependent variables and effect size can be extracted from the research report. Search strategies should be completely documented to prevent duplication of effort. Ample time must be allowed for contacting authors for information about additional studies. Systematic literature reviews are the first step in systematic research. Search Strategies Diverse search strategies are available to locate research. Recall and precision are accepted measures of the utility of information retrieval (Eysenbach, Tuische, & Diepgen, 2001). Recall the proportion of relevant studies retrieved by search strategies indicates the sensitivity of the search (Eysenbach et al., 2001). A comprehensive search will have high recall. Precision is the proportion of relevant studies in the retrieved set (Eysenbach et al., 2001; Hersh & Hickam, 1993). Precision indicates the specificity of the search. Recall and precision vary dramatically, often inversely, by search strategy. The purpose of the literature search will determine the appropriate balance between recall and precision. The Table shows important attributes of some search strategies. Table. Attributes of Strategies for Literature Searches Strategy Precision Recall Comments Computerized High for limited Generally high Experienced librarians databases databases, lower for can assist with complex general databases structured languages. Ancestry Moderate High Does not generally broaden searches the diversity of studies. Citation index Very low for broad High Most efficient for littlesearches topics, high for narrow, studied areas with few highly specific topics seminal works. Research High High Excellent source that registries avoids publication bias. Journal hand High with well- High Labor intensive but yields searches trained searchers higher than computerized databases. Invisible college High Low to moderate May locate student contacts depending on projects and reports not response rate yet published or indexed. Presentation High for highly High for Less promising results abstract searches specialized specialized for general conferences. conferences conferences Internet searches Low unless Low May locate grey literature specialized terms and studies in progress. Computerized Database Searching Computerized database searching using MEDLINE, CINAHL, and PsycINFO are familiar to many nurses. Morrisey and DeBourgh (2001) provided an excellent introduction to this form of searching. This discussion will focus on issues related to conducting a comprehensive search. Search skills improve with practice and most database users achieve satisfactory recall of relevant citations. This situation sometimes leads to what Booth and O Rourke (1999) referred to as the satisfied but 178 Second Quarter 2003 Journal of Nursing Scholarship

3 inept searcher. Many users overestimate the quality of their searching performance (Hersh & Hickam, 1993). Experienced reference librarians typically retrieve about twice as many citations as do less experienced users (Hersh & Hickam, 1993), because databases are highly structured with complex indexing rules (Dickersin, Scherer, & Lefebvre, 1994). Experienced searchers are also best able to generate the independent search strategies sometimes required for varied databases because of lack of standardization of databases (Sindhu & Dickson, 1997). A reference librarian can assist with developing free-text strategies for databases without well-constructed thesauruses (Hek, Langton, & Blunden, 2000). Thus one important strategy for conducting a comprehensive search is to enlist the assistance of an experienced health sciences reference librarian. Another important strategy is to treat searching as an iterative process in which eligible studies are examined for potential keywords or indexing terms to further develop the search process (Sindhu & Dickson, 1997). Acceptance that a highly sensitive search (high recall) will likely have low specificity (diminished precision) is necessary for comprehensive searching. Still, even searches conducted by highly skilled librarians result in only about half of the eligible citations (Avenell, Handoll, & Grant, 2001; Dickersin et al., 1994; Jadad, Moher, & Klassen, 1998; McManus et al., 1998; Poynard & Conn, 1985; Sindhu & Dickson, 1997). Indexing problems most likely cause this low yield. Poor reporting, coding systems that do not address some topics, new areas of science without index terms, inadequate descriptions of research to allow indexers to correctly classify studies, and inevitable lack of consistency among coders all contribute to inadequate indexing (Dickersin et al., 1994; Jadad et al., 1998; McManus et al., 1998). Data consistently indicate that MEDLINE searches will not retrieve all the citations found by hand-searching journals indexed by MEDLINE. Furthermore, significant numbers of studies are not indexed in MEDLINE. McDonald and colleagues (2002) found that 30% (n=6,554) of reports of randomized controlled trials located by extensive hand searching for the Cochrane Collaboration were indexed in MEDLINE. Sensitivity and precision of database searches vary dramatically by scientific area (Dickersin et al., 1994). Research on the effectiveness of computerized database searching has been almost entirely focused on MEDLINE. CINAHL has been studied infrequently. One search of nonpharmacologic nursing interventions resulted in 14% of eligible studies using CINAHL, but all the CINAHL-identified studies were also obtained through other sources (Sindhu & Dickson, 1997). Another nursing search produced no citations through CINAHL, but citations were readily retrieved from MEDLINE and other databases (Avenell et al., 2001). Numerous specific databases are available. For example, a recent meta-analysis of interventions to increase physical activity among aging adults included searches of an exercise-specific database (SPORT Discus) and American Association of Retired Persons Ageline for gerontology research (Conn, Valentine, & Cooper, 2002). Databases that will allow access to international studies are important. For example, EMBASE (Excerpta Medica Database) includes journals from 70 countries (Dickersin et al., 2002; Hek et al., 2000). The overlap between MEDLINE and EMBASE is only 34% (Dickersin et al., 2002). Databases outside health care might be useful depending on the topic (e.g. Educational Resources Information Center [ERIC]). Databases not primarily focused on research might also be useful. For example, CHID (Combined Health Information Database) contains information about health promotion programs that could be useful in research on health behavior interventions. More general databases will increase the sensitivity of a search, but precision will decline (Dickersin et al., 1994). An experienced reference librarian can assist with decisions about computerized databases. Ancestry Searching Ancestry searching refers to the systematic review of citations from studies included in the review and from review articles. The most thorough approach is to review the text of the research report, especially the literature review and discussion, for citations to other potentially useful studies. Then each reference is reviewed for other potentially useful studies. These dual strategies are more effective than simply reviewing the reference list because titles often misrepresent content (Cooper & Ribble, 1989; Sindhu & Dickson, 1997). Ancestry searches expand the number of eligible studies. However, reliance on ancestry searching without adequate attention to other search strategies can yield a biased set of studies. Studies with statistically significant findings are more likely to be cited in reference lists than are studies without significant findings (Gotzsche, 1987). For example, Ravnskov (1992) found that studies to test cholesterol-lowering interventions to prevent coronary disease published after 1970 included citations of positive or inconclusive studies, but they never included contrary evidence. Reference lists of studies generally contain studies similar to the one in the original report. Thus an ancestry search might enlarge the number of retrieved studies but it does not always broaden the characteristics of studies in the review. Ancestry searching tends to have high recall and moderate precision. Citation-Index Searching Citation-index searching is based on the cumulative nature of research. Citation indexes include references used in published articles. This form of searching is most effective if a few important works are widely cited by other researchers and are not widely cited by those not conducting similar research. Because citation indexes are based on the citing practices of authors, they generally yield studies similar to the original citation. Citation-index searching might be helpful in identifying subject headings for MEDLINE searches. Recall and precision vary dramatically by topic area. Searching Research Registries Research registries are a high-recall and high-precision source of information about studies. Research registries provide information about studies regardless of the statistical significance of findings because registration occurs before study completion (Egger & Smith, 1998). Specialty organizations often manage registries. The Cochrane Journal of Nursing Scholarship Second Quarter

4 CENTRAL register contains information about more than 300,000 controlled trials (Lefebvre et al., 2002). Advocates of research synthesis suggest mandatory research registration to avoid publication bias. In Australia, all studies approved by ethics committees are entered into a prospective study registry (Stern & Simes, 1997). Registries are a valuable source of information about studies. Journal Hand Searching Journal hand searching requires examining journal article titles, abstracts, and complete articles. Journals with high potential recall are selected for hand searching because a majority of eligible studies in a systematic review are from a small number of journals (Hek et al., 2000). More varied studies can be retrieved by searching more diverse journals. Given the laborious nature of hand searching, researchers have attempted to document whether hand searching yields more studies than do other methods. Studies consistently have shown higher recall from hand searching than through computerized databases and other search strategies (Dickersin et al., 2002; Helmer, Savoie, Green, & Kazanjian, 2001; McManus et al., 1998; Poynard & Conn, 1985). Langham, Thompson, and Rowen (1999) reported that they found 84% of the total articles in their review by hand searching, but only 68% through MEDLINE. The Cochrane collaboration identified 18,000 controlled trials, not indexed in MEDLINE, through hand searching European journals (Lefebvre et al., 2002). Hand searching has both high precision and strong recall when well-trained reviewers conduct the searches. Journal hand searching may yield publications that have not yet been indexed in computerized databases (Avenell et al., 2001). Labor intensiveness is the predominant disadvantage of hand searching (Jadad et al., 1998). Contact with the Invisible College The invisible college is composed of geographically dispersed investigators conducting research in a specific area of science. Experts are easily identifiable from repeated publications. Contact with experts might be used to solicit the names of others who should be contacted (Sindhu & Dickson, 1997). Experts in the field can identify additional relevant published and unpublished research (Helmer et al., 2001; Jadad et al., 1998). Experts are often aware of relevant graduate student work that might not be published but would be appropriate for inclusion. McManus and colleagues (1998) found that 24% of the studies included in their systematic review would have been missed entirely if experts had not been contacted. Contacting experts yields high precision and variable recall, depending on response rates. The largest drawback is variable response rates. McManus and colleagues (1998) reported a 53% response rate for queries sent to academic researchers. Promising copies of the review if experts provide citations might increase response rates (Conn et al., 2002). Searching Conference Proceedings Presentations at conferences are based on recently completed research, but often reports of these studies do not appear in the literature for several years. Many are never published, especially those without statistically significant findings (Callaham, Wears, Weber, Barton, & Young, 1998). Some associations journals include conference abstracts in regular or supplemental issues. Generally, bound copies of proceedings are available from large organizations. Once studies are identified, contact with authors might yield unpublished full reports, prepublication full reports, or copies of presentation materials with additional information about the studies. This search strategy produces high recall and precision with specialized conferences closely linked to the scientific topic (Langham et al., 1999; McManus et al., 1998). Less promising results are obtained from more general conferences. Internet Searching World wide web (WWW) searching is valuable for locating studies in progress or recently completed that are not yet published, presentations at conferences, and researchers who might be contacted for other studies, electronic journal articles, and other links for information. For example, Eysenbach and colleagues (2001) found clues concerning unpublished studies on departmental or institutional homepages with descriptions of research activities, researchers personal homepages, online conference proceedings or abstracts, announcements about grant awards, press releases by departments or institutions about ongoing or recently completed studies, online published research reports, and Websites for recruiting participants. WWW searching might yield high recall for unpublished and grey literature (studies not formally published or published in difficult to access locations). Suitable search engines must be powerful enough to handle complex queries because Web documents are not indexed with key words in a controlled vocabulary (Eysenbach et al., 2001). The large number of irrelevant documents on the WWW necessitates carefully designed and highly specific search strategies (Eysenbach et al., 2001). Useful search engines must be able to truncate word stems, link synonyms with an or, and conduct proximity search with near. Searching is an iterative process in which results from one attempt are used to expand or narrow a subsequent search. Near generally limits the search more than and. Spelling variants might result in more hits. The number of possible synonyms can be increased to further expand the search. Similarly, searches may be narrowed by reducing the number of synonyms, removing truncated terms or providing longer word stems, and sometimes by using not to exclude sites with particular terms (Eysenbach et al., 2001; Robinson & Dickersin, 2002). Eysenbach and colleagues (2001) provided an excellent overview of these strategies and examples of searches. Despite rapid proliferation of search engines, Internet searching remains challenging (Sigouin & Jadad, 2002). No single search engine includes all of the indexable WWW sites (Lawrence & Giles, 1998). Scant work has addressed the precision and recall of Internet searching. Eysenbach and colleagues (2001) searched with Alta Vista to find studies for Cochrane reviews. They reviewed 429 Web pages to find clues 180 Second Quarter 2003 Journal of Nursing Scholarship

5 to 14 studies. The specificity of the topic determines whether the signal-to-noise ratio makes WWW searching useful. Researchers should be aware that the quality of information on the Internet varies dramatically (Eysenbach & Diepgen, 1998; Hersh, Gorman, & Sacherek, 1998). Securing International Literature North American and British studies are most easily retrieved through standard computerized databases and ancestry searches. Searching beyond these traditional sources is necessary to broaden the base of science available for nursing knowledge. For example, an in-progress meta-analysis is focused on the effects of interventions to increase physical activity among adults with chronic illnesses (Conn, ). Through extensive literature searching the researchers have retrieved potentially useful research reports in 87 journals published outside North America and the United Kingdom. Depending on the purpose of the literature review, reports published in languages other than English are needed. Dickersin and colleagues (1994) found that 20% of the studies for which they found citations were not in English. Searching international literature is consistent with the global focus of nursing. Translators with university affiliations can be especially helpful, because of their scientific knowledge, in managing research reports in other languages. Journals in some countries have enormous linguistic, financial, and production difficulties (Zielinski, 1995). World Health Organization, UNESCO, and the Food and Agriculture Organization have developed projects such as ExtraMED, ExtraSCI, and AgROM Extra to help address the problem of accessing research completed in developing countries (Zielinksi, 1995). Grey literature is difficult to locate. SIGLE (System for Information on Grey Literature in Europe) is a bibliographic database covering European nonconventional (so-called grey) literature in the fields of pure and applied natural sciences and technology, economics, social sciences, and humanities. The purpose of a literature review will indicate whether extensive searching for research across the globe is necessary. Finding the Fugitive Literature Many search strategies will reveal clues to other studies that might be useful. For example, in the area of health promotion research, one common hint is a description of a program without outcome data. Or an article might include outcomes other than the ones of interest to the reviewers. The clues provide a beginning point for a search for the fugitive literature. The first strategy is to carefully inspect the reference list for a possible relevant citation. Computerized searches for all authors of the reference might lead to additional reports eligible for inclusion in the review. Sometimes unusual words are used that can be searched in text fields of computerized databases (e.g., calling an exercise program WOMENACT). Studies may be located in the National Institutes of Health database of funded studies (Computer Retrieval of Information on Scientific Projects [CRISP]) by searching for author names. Authors may be contacted to seek further information. A careful system of recording completed strategies will make the search for fugitive literature manageable. References to research reports are often found outside traditional peer-reviewed journals. Other possible sources include company reports and press releases, government monographs and reports, policy documents, publications of special panels or commissions, and regulatory documents. Other Considerations Research Studies of Variable Quality The quality of information retrieved from any source must be carefully scrutinized. For example, promotional material on the WWW may be disguised as peer-reviewed research (Eysenbach et al., 2001). Decisions about methodologic quality of retrieved research are separate from strategies to locate studies. Easterbrook and colleagues (1991) said that studies published in peer-reviewed journals are not necessarily of higher methodologic quality than are other studies. Studies retrieved from any source, including MEDLINE, have to be evaluated in regard to inclusion criteria that may include methodologic characteristics (de Vet et al., 2001). In metaanalysis methods are coded for empirical analysis as potential moderators of effect size. Filters have been proposed as a strategy to retrieve only the most rigorously conducted studies (Booth & O Rourke, 1999). Unfortunately, filtering studies might indicate more about the quality of reporting than the quality of studies. Filtering is an inexact science at best. Even if filters were effective, research reviewers would not be in universal agreement about what should be used. In metaanalysis, inclusion of all relevant studies is consistent with the aims of science. Resource Allocation and Extensive Searching Researchers planning a comprehensive review of the literature should not underestimate the difficulty or expense of conducting a well-designed and carefully implemented review (Dickersin et al., 1994; Jadad et al., 1998). Predicting the time necessary to conduct searches or to fully consider citations is difficult. Jadad and colleagues (1998) suggested bibliographic databases are the most cost-effective source, hand searching is an intermediate cost strategy, and direct contact with investigators or organizations is generally quite costly. Fewer resources are required for very narrow searches or for searches in areas in which little research has been conducted. The most common strategies used for systematic reviews are computerized databases, ancestry searches, journal hand searches, and contacting experts (Avenell et al., 2001). Jadad and colleagues (1998) recommended that researchers select as many search strategies as their resources allow. Conclusions A systematic and thorough review of existing research is as important as is systematic collection of data for advancing knowledge. Limited and biased searches are inefficient and Journal of Nursing Scholarship Second Quarter

6 inadequate for building the cumulative nature of science. Conclusions about the efficacy of nursing interventions should not be based on inadequate reviews of existing evidence. A broad array of search strategies are available and they extend far beyond the customary MEDLINE search. The purpose of the literature review will determine the extent of search that is appropriate for any given project. References Avenell, A., Handoll, H.H., & Grant, A.M. (2001). Lessons for search strategies from a systematic review, in The Cochrane Library, of nutritional supplementation trials in patients after hip fracture. American Journal of Clinical Nutrition, 73, Booth, A., & O Rourke, A.F. (1999). Searching for evidence: Principles and practice. Evidence Based Medicine, 4, Callaham, M.L., Wears, R.L., Weber, E.J., Barton, C., & Young, G. (1998). Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. JAMA, 280, Chalmers, I., Adams, M., Dickersin, K., Hetherington, J., Tarnow-Mordi, W., Meinert, C., et al. (1990). A cohort study of reports of controlled trials. JAMA, 263, Conn, V. ( ). Chronic illness exercise interventions: A metaanalysis. National Institutes of Health National Institute of Nursing Research, RO1NR Bethesda, MD. Conn, V., Valentine, J., & Cooper, H. (2002). Interventions to increase physical activity among aging adults: A meta-analysis. Annals of Behavioral Medicine, 24, Cook, A.M., Finlay, I.G., Edwards, A.G.K., Hood, K., Higginson, I.J., Goodwin, D.M., et al. (2001). Efficiency of searching the grey literature in palliative care. Journal of Pain & Symptom Management, 22, Cooper, H., & Ribble, R. (1989). Influences on the outcomes of literature searches for integrative research reviews. Knowledge: Creation, Diffusion, Utilization, 10, de Vet, H.C., van der Weijden, T., Muris, J.W., Heyrman, J., Buntinx, F., & Knottnerus, J.A. (2001). Systematic reviews of diagnostic research. Considerations about assessment and incorporation of methodological quality. European Journal of Epidemiology, 17, Dickersin, K., Chan, S., Chalmers, T.C., Sacks, H.S., & Smith, H., Jr. (1987). Publication bias and clinical trials. Controlled Clinical Trials, 8, Dickersin, K., Manheimer, E., Wieland, S., Robinson, K. A., Lefebvre, C., & McDonald, S. (2002). Development of the Cochrane Collaboration s CENTRAL Register of controlled clinical trials. Evaluation & the Health Professions, 25, Dickersin, K., Min, Y.I., & Meinert, C.L. (1992). Factors influencing publication of research results. JAMA, 267, Dickersin, K., Scherer, R., & Lefebvre, C. (1994). Identifying relevant studies for systematic reviews. British Medical Journal, 309, Easterbrook, P.J., Berlin, J.A., Gopalan, R., & Matthews, D.R. (1991). Publication bias in clinical research. Lancet, 337, Egger, M., & Smith, G.D. (1998). Bias in location and selection of studies. British Medical Journal, 316, Eysenbach, G., & Diepgen, T.L. (1998). Towards quality management of medical information on the Internet: evaluation, labeling, and filtering of information. British Medical Journal, 317, Eysenbach, G., Tuische, J., & Diepgen, T.L. (2001). Evaluation of the usefulness of Internet searches to identify unpublished clinical trials for systematic reviews. Medical Informatics & the Internet in Medicine, 26, Gotzsche, P.C. (1987). Reference bias in reports of drug trials. British Medical Journal Clinical Research, 295, Gotzsche, P.C. (1989). Multiple publication of reports of drug trials. European Journal of Clinical Pharmacology, 36, Hek, G., Langton, H., & Blunden, G. (2000). Systematically searching and reviewing literature. Nurse Researcher, 7, Helmer, D., Savoie, I., Green, C., & Kazanjian, A. (2001). Extending the search to find material for the systematic review. Bulletin of the Medical Library Association, 89, Second Quarter 2003 Journal of Nursing Scholarship Hersh, W.R., & Hickam, D.H. (1993). A comparison of two methods for indexing and retrieval from a full-text medical database. Medical Decision Making, 13, Hersh, W.R., Gorman, P.N., & Sacherek, L.S. (1998). Applicability and quality of information for answering clinical questions on the Web. JAMA, 280, Hubbard, R., & Armstrong, J.S. (1997). Publication bias against null results. Psychological Reports, 80, Jadad, A.R., Moher, D., & Klassen, T.P. (1998). Guides for reading and interpreting systematic reviews: II. How did the authors find the studies and assess their quality? Archives of Pediatrics & Adolescent Medicine, 152, Kleijnen, J., & Knipschild, P. (1992). Review articles and publication bias. Arzneimittel-Forschung, 42, Langham, J., Thompson, E., & Rowen, K. (1999). Identification of randomized controlled trials from the emergency medicine literature: comparison of hand searching versus MEDLINE searching. Annals of Emergency Medicine, 34, Lawrence, S., & Giles, C. (1998). Searching the Web. Science, 280, Lefebvre, C., Lusher, A., Dickersin, K., & Manheimer, E. (2002). Literature searches. Lancet, 359, 896. McDonald, S., Lefebvre, C., Antes, G., Galandi, D., Gotzsche, P., Hammarquist, C., et al. (2002). The contribution of handsearching European health care journals to the Cochrane Controlled Trials Register. Evaluation & the Health Professions, 25, McManus, R.J., Wilson, S., Delaney, B.C., Fitzmaurice, D.A., Hyde, C.J., Tobias, R.S., et al. (1998). Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. British Medical Journal, 317, Morrisey, L.J., & DeBourgh, G.A. (2001). Refining literature searching skills for the advanced practice nurse. AACN Clinical Issues: Advanced Practice in Acute & Critical Care, 12, Poynard, T., & Conn, H.O. (1985). The retrieval of randomized clinical trials in liver disease from the medical literature. A comparison of MEDLARS and manual methods. Controlled Clinical Trials, 6, Ravnskov, U. (1992). Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. British Medical Journal, 305, Robinson, K.A., & Dickersin, K. (2002). Development of a highly sensitive search strategy for the retrieval of reports of controlled trials using PubMed. International Journal of Epidemiology, 31, Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, Scherer, R.W., Dickersin, K., & Langenberg, P. (1994). Full publication of results initially presented in abstracts. A meta-analysis. JAMA, 272, Serling, T. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance. Journal of the American Statistical Association, 54, Serling, T., Rosenbaum, W., & Weinkam, J. (1995). Publication decisions revisited: the effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician, 49, Sigouin, C., & Jadad, A.R. (2002). Awareness of sources of peer-reviewed research evidence on the Internet. JAMA, 287, Simes, R.J. (1987). Confronting publication bias: a cohort design for meta-analysis. Statistics in Medicine, 6, Simes, R.J. (1986). Publication bias: the case for an international registry of clinical trials. Journal of Clinical Oncology, 4, Sindhu, F., & Dickson, R. (1997). The complexity of searching the literature. International Journal of Nursing Practice, 3, Stern, J.M., & Simes, R.J. (1997). Publication bias: Evidence of delayed publication in a cohort study of research projects. British Medical Journal, 315, Sugita, M., Kanamori, M., Izuno, T., & Miyakawa, M. (1992). Estimating a summarized odds ratio whilst eliminating publication bias in metaanalysis. Japanese Journal of Clinical Oncology, 22, Thornton, A., & Lee, P. (2000). Publication bias in meta-analysis: Its causes and consequences. Journal of Clinical Epidemiology, 53, Zielinski, C. (1995). New equities of information in an electronic age. British Medical Journal, 310,