Traditional author co-citation analysis: A discussion of the sampling problem

Transcription

1 Traditional author co-citation analysis: A discussion of the sampling problem Jeppe Nicolaisen 1 Royal School of Library and Information Science, Institute for Information Studies, Birketinget 6, DK-2300 Copenhagen S., DENMARK Drawing a sample of data that adequately represents the population from which it was drawn can be a difficult task. Biases are often introduced in this initial phase of statistical analyses. The paper addresses the sampling problem in relation to traditional author co-citation analysis. It identifies common biases (misrepresentation of different theoretical orientations within the domain of study, national differences, misrepresentation of document types, ISI data) and discusses possible alternatives ( authoritative lists of authors, PFNETs, domain analysis). It concludes that conducting a proper domain analysis before starting the sampling procedure seems to be the only good answer to the sampling problem. Keywords: Author co-citation analysis, Sampling, Domain analysis. 1 INTRODUCTION A traditional author co-citation analysis usually consists of six separate steps (McCain, 1990: 434): Selection of authors Retrieval of co-citation frequencies Compilation of a raw citation matrix Conversion to correlation matrix Multivariate analysis of correlation matrix Interpretation and validation Common biases in traditional author co-citation analyses are often introduced with the first step. To conduct such analyses one has to have something to analyze. This usually means that one has to draw a sample of authors. Drawing such a sample can be a difficult task. However, it is often necessary as it is often impossible, impractical, or extremely expensive to collect data from all the potential units of analysis covered by the research problem. Fortunately, analysts should be able to draw precise inferences on all the units (a set) based on a relatively small number of units (a subset) when the subset accurately represent the relevant attributes of the whole set. To achieve such precision the analyst needs to deal effectively with three issues: Definition of the population Selection of a representative sample Determination of the sample size A population is usually defined as the aggregate of all cases that conform to some designated set of specifications (Chein, 1981: 419). For example, by the specifications authors and having published in Journal of Documentation one can define a population consisting of all authors having published in Journal of Documentation. As the specific nature of the population depends on the research problem, the population often has to be defined in terms of a) content, b) extent, and c) time. If the research problem only concerns authors having published articles in Journal of Documentation during the last 25 years, the population consequently has to be defined in terms of a) authors, b) having published articles in Journal of Documentation, c) in the period A single member of a sampling population (e.g., an author having published in Journal of Documentation) is called a sampling unit. The essential requirement of any sample is that it should represent the population from which it was drawn. A sample is considered to be representative if analyses made using sampling units produce results similar to results achieved using the entire population of units. A welldesigned sample ensures that if a study were to be repeated on a number of different samples drawn from a given population, the findings from each sample would not differ significantly. 1 jni@db.dk, URL:

2 A sample may be a probability sample or a non-probability sample. The distinguishing characteristic of probability sampling is that for each sampling unit of the population, one can specify the probability that the unit will be included in the sample. In non-probability sampling one cannot specify the probability of each unit s inclusion in the sample, and there is no assurance that all units have the same chance of being included. Chein (1981: 421) argues that if a set of units has no chance of being included in the sample then the definition of the population must be restricted. To estimate the proper size of a sample, the analyst needs to determine what level of accuracy is expected of his or her estimates. The literature on sampling provides different techniques for determination of sample size. This paper addresses the sampling problem in relation to traditional author co-citation analyses. It identifies common biases and discusses possible alternatives. 2 COMMON BIASES If an analyst sets out to construct an author co-citation map that illustrates clusters of units in some field or discipline of study, the analyst make sure that the sample used for analysis is constructed in a way that makes it possible to conclude from the sample to the population. Such a conclusion is only possible if the sample has been drawn adequately and thus reflects the population. This implies that the sample must be designed in a way that gives all units an equal chance of being selected. To give an example: In order to visualize a whole discipline by author co-citation analysis, the sample of authors must be a random sample of authors who have published in the field during some period of time. Taking a specific example for further explanation: One would therefore expect White and McCain s (1998) famous author co-citation analysis of information science ( ) to be based on a random sample of authors having published information scientific books, papers, reviews, etc. during the specified period. However, this is not the case. Like most other author co-citation analyses it is based on a so-called judgment sample. White and McCain (1998) limited their sample to the 120 most cited authors in twelve selected international journals. The main problem with judgment samples is that the chance that a particular sampling unit will be selected for the sample depends on the subjective judgment of the researcher (Frankfort-Nachmias & Nachmias 1996: 184). A judgment sample is consequently a non-probability sample. In the case of White and McCain (1998), it means that we have no way of knowing whether their results actually visualize information science or whether they just visualize the artificial sample 2. However, there are good reasons to believe that author cocitation maps based on highly cited units do not provide adequate illustrations of whole disciplines or subdivided fields. The main reason is that highly cited authors and documents do not represent the average author and document. A simple example borrowed from Hjørland (1981: 184) explains why: Examining the history of psychology we find it has been marked by shifting theoretical orientations. For example, during the last ten years there has been a rapidly increasing interest for classical authors such as Sigmund Freud, C.G. Jung and Karl Marx. When such shifts occur it is obvious that they influence the distributions of citations. Now, in contrast to earlier periods dominated by, for instance, behavioristic theories, it is these persons and their successors who in particular are being cited [my translation]. Hjørland s (1981) claim about shifting theoretical orientations having marked the history of psychology is, for instance, supported by Robins, Gosling and Craik s (1999) empirical analysis of trends in psychology. Examining the distribution of references in four leading psychological journals between 1977 and 1996, the authors found that a shift had occurred around More precisely they found that the behavioral school, which previously had been the most cited school, lost approximately half its share of annual citations from 1977 to During the same period, the cognitive school almost doubled its share of citations. Since 1980 the cognitive school was found to have increased its share further. Thus, in 1996 the cognitive school had been cited almost 500 times in the four leading journals whereas the behavioral school had been cited only about 100 times. The authors found that two other schools, the psychoanalytic and the neuroscientific, had been cited approximately by the same annual frequency throughout the investigated period, but considerably less than the two other schools. Thus, if one draws a sample of highly cited authors 2 It is not entirely clear whether White & McCain (1998) acknowledge the limitations of their sample. They write that they do not regard their sample to be wholly definitive of information science, past, present, or future (White & McCain, 1998: 332). But they go on to claim that the 120 authors are sufficient for their main purpose: [T]o convey [ ] the main subdivisions of the field as they have evolved in the past quarter-century (White & McCain, 1998: 332).

3 from articles published in 1996 in these four journals, it will not reflect adequately the population of cited authors, as it will probably include only the highly cited authors working within the cognitive school. Hjørland (1981) ends his paper by concluding that there are important national differences, and that some research paradigms prefer the journal article as communication medium while others prefer the monograph. This is another important observation. If this holds true for other disciplines as well 3, it more or less disqualifies SCI, SSCI, and A&HCI as sample populations for co-citation analyses, which seek to map larger areas (as, for instance, White and McCain (1998) did). The reason is that the citation indexes almost exclusively index references from international journals. Thus, if one draws a sample of highly cited authors from these indexes it is likely to contain a number of those authors, who are being frequently cited in the international journal literature, but not necessarily in national journals or monographs. A number of empirical studies have demonstrated this to be the case. The bibliometric consequences of scholars national orientation are, for example, well illustrated by Webster s (1998) analysis of a Polish sociological citation index (PSCI) and the SSCI. By counting and comparing the number of citations to Polish sociologists in the two indexes between 1981 and 1995 she was able to conclude: Lists of the top 20 most cited Polish sociologists in each index had 12 names in common. The most cited sociologist in the PSCI (253 citations) was ranked as number 41 in the SSCI (19 citations). The most cited sociologist in the SSCI (254 citations) was ranked as number 20 in the PSCI (41 citations). Lists of the top 20 most cited documents by Polish sociologists in each index contained none in common. Warsaw sociologists dominate Polish sociology. 60% of the citations in PSCI went to Warsaw sociologists. 80% of the citations to Polish sociologists in SSCI went to Warsaw sociologists. Webster s (1998) findings strongly suggest that bibliometric indicators based on SSCI paint one picture of Polish sociology, and the PSCI another. Frohmann (2004: 128) argues that citation studies may reveal useful discursive characteristics and regularities of a specific document type, but that such studies do not reflect the real world of research science. A number of studies have in fact demonstrated that indicators constructed from journal references alone will differ from indicators that include book references as well. In one of these, Cronin, Snyder and Atkins (1997) constructed a database comprising references from 90 books randomly chosen among those reviewed in top sociological journals and published between 1985 and The authors compared lists of the 26 authors most cited in the books and in the top 24 sociology journals, and found: Nine authors featured on both lists. The five authors ranked 22 to 26 on the book list did not appear among the top 532 authors most cited in the top journals. These findings suggest that there are two distinct populations of highly cited authors in sociology: One consisting of authors cited in the journal literature, another of authors cited in the monographic literature. Given the citation indexes limited coverage of monographic citing material, the latter population may regularly go unrecognized. The majority of co-citation analyses are based on ISI data. ISI data are normally used for two related purposes: 1. for selecting a sample for further analysis, and 2. for detecting how many times the sample units are co-cited. Reliance on ISI s citation indexes is clearly problematic. Such a sample may at best be regarded a fractionized sample, and any results based on such a sample has limited generalizability. 3 ALTERNATIVES A number of researchers have sought to bypass the sampling problem and have made use of authoritative lists of authors instead. White and Griffith (1981), for example, constructed a map of information science using a list of 39 authors taken in considerable part from the anthology Key Papers in Information Science, edited by Griffith (1980). The table of contents supplied 22 names, and the authors added 17 more, which they judged to be major contributors to the field (White & Griffith, 1981: 164). The 3 Hicks (1999: 193) argues that social science research is characterized by more competing paradigms and a national orientation.

4 authors note that the resulting list is biased towards established figures with multiple contributions over the years. This is probably true. Another, and perhaps more important bias, has to do with the fact that researchers from different research traditions tend to evaluate the merit of the same papers in a different way. Thus, if one asked researchers from competing research traditions to point out the key papers of their discipline, one would probably end up with quite different lists of papers. Consequently, co-citation maps based on so-called authoritative lists probably fail to provide a full picture of disciplines and specialties with different competing research traditions. Pathfinder networks (PFNETs) made with raw citation counts and visualized through spring embedders appear to have considerable advantages compared to the traditional way of conducting author cocitation analyses (Chen, 1998; White, 2003; Schneider & Borlund, 2004). A key advantage is that PFNETs in principle are able to present an unlimited number of authors and their relations. A long acknowledged weakness of the traditional method is that it is only able to represent a small set of entities (see Schneider, 2004: 130). Thus, PFNETs seem to provide the technical requirements necessary for sidestepping the sampling problem. Yet, a simple technological fix is not enough to deal adequately with the sampling problem. What is needed is first and foremost knowledge about the domain of study (the population). The domain analytic approach to information science is a metatheory that provides a theoretical framework for the explanation of all phenomena of relevance to the discipline. Its main creator or architect is Birger Hjørland. Hjørland gave the first oral presentation of domain analysis in 1993 at the ASIS 56th Annual Meeting in Columbus, Ohio, and wrote the first article on the metatheory together with Hanne Albrechtsen two years later (Hjørland & Albrechtsen, 1995). Hjørland has subsequently made a number of important contributions alone and with others that have contributed to establish domain analysis as a very promising metatheory. One of the absolute presuppositions in the domain analytic approach is the claim that tools, concepts, meaning, knowledge organization, cooperation patterns, communication forms, information needs, and relevance criteria are shaped in discourse communities, for example in scientific disciplines, which are parts of society s division of labor (e.g., Hjørland & Albrechtsen, 1995, p. 400). This absolute presupposition places the focus of information science on the sociocultural world rather than on the psychology of individuals or on the processes of computers. Exponents of the domain analytic approach see different objects as being informative relative to the social division of labor in society. Information thus becomes a subjective concept, but in a collective sense, as exponents of the domain analytic approach believe that criteria for what counts as information are formed mainly by sociocultural processes. The domain analytic view consequently implies that information scientists should study knowledge domains either individually or comparatively in order to understand the information activities of their actors. The domain analytic approach offers a theoretical framework for the committed information scientist who wants to explore and understand the information phenomena he or she is engaged in studying. It recognizes individuals as primary sociocultural beings acting in different domains, and is based on the notion that all domains are constituted by three dimensions (Hjørland & Hartel, 2003, p. 239): 1. Ontological theories and concepts about the objects of human activity. 2. Epistemological theories and concepts about knowledge and the way to obtain knowledge, implying methodological principles about the ways objects are investigated. 3. Sociological concepts about the groups of people concerned with the objects. In order to properly understand information activities, the information scientists needs to be acquainted with the domain of these activities and especially with its three dimensions and how they structure and constrain individual actions. A domain analysis may be approached in a variety of ways and from a number of potential starting points (cf. Hjørland, 2002). Ontological theories and concepts about the objects of human activity are, for instance, studied by philosophers and historians of science. Epistemological theories and concepts about knowledge and the way to obtain knowledge are also studied by philosophers and historians of science. Sociology and sociological concepts are, for instance, studied by sociologists of science, scientometricians, anthropologists, and scientists from a wide variety of other social sciences. The serious information scientists should consequently keep an open mind about being informed by these disciplines in order to be able to make proper studies of the three dimensions 4. 4 The Epistemological Lifeboat (Hjørland & Nicolaisen, 2005-) is a concise online encyclopaedia that attempts to guide information science students and researchers into the highly relevant but complex field of epistemology/philosophy of science. It is intended as a lifeboat or a philosophy for dummies. That is obviously not enough for serious studies, but it provides an overview and refers the reader to further sources of information.

5 4 CONCLUSION Conducting a proper domain analysis before starting the sampling procedure seems to be the only good answer to the sampling problem. A domain analysis will, for instance, inform the researcher about different theoretical orientations within the domain, national differences, and document type preferences, which should enable the researcher to draw a sample that adequately represents the domain under study. ACKNOWLEDGMENT This work was supported by a travel grant from the Nordic Research School in Library and Information Science (NORSLIS). REFERENCES [1] Chein, I. (1981). An introduction to sampling. In: Kidder, L.H. et al. (eds.), Research Methods in Social Relations. New York, NY: Holt, Rinehart, & Winston: [2] Chen, C. (1998). Generalised similarity analysis and pathfinder network scaling. Interacting with Computers, 10: [3] Cronin, B., Snyder, H. & Atkins, H. (1997). Comparative citation rankings of authors in monographic and journal literature: A study of sociology. Journal of Documentation, 53(3): [4] Frankfort-Nachmias, C. & Nachmias, D. (1996). Research Methods in the Social Sciences. London, UK: Arnold. [5] Frohman, B. (2004). Deflating Information: From Science Studies to Documentation. Toronto, CAN: University of Toronto Press. [6] Griffith, B.C. (1980). Key Papers in Information Science. White Plains, NY: Knowledge Industry Publications. [7] Hicks, D. (1999). The difficulty of achieving full coverage of international social science literature and the bibliometric consequences. Scientometrics, 44(2): [8] Hjørland, B. (1981). Bibliometriske analyser i psykologien [Bibliometric analyses in psychology]. Nordisk Psykologi, 33(3): [9] Hjørland, B. (2002). Domain analysis in information science: Eleven approaches - traditional as well as innovative. Journal of Documentation, 58(4): [10] Hjørland, B. & Albrechtsen, H. (1995). Toward a new horizon in information science. Journal of the American Society for Information Science, 46(6): [11] Hjørland, B. & Hartel, J. (2003). Afterword: Ontological, epistemological and sociological dimensions of domains. Knowledge Organization, 30(3/4): [12] Hjørland, B. & Nicolaisen, J. (2005-). The Epistemological Lifeboat. Available at: [13] McCain, K.W. (1990). Mapping authors in intellectual space: A technical overview. Journal of the American Society for Information Science, 41(6): [14] Robins, R.W., Gosling, S.D. & Craik, K.H. (1999). An empirical analysis of trends in psychology. American Psychologist, 54(2): [15] Schneider, J.W. (2004). Verification of Bibliometric Methods Applicability for Thesaurus Construction. Aalborg, DK: Royal School of Library and Information Science. PhD Thesis. [16] Schneider, J.W. & Borlund, P. (2004). Introduction to bibliometrics for construction and maintenance of thesauri: Methodical considerations. Journal of Documentation, 60(5): [17] Webster, B.M. (1998). Polish sociology citation index as an example of usage of national citation indexes in scientometric analysis of social sciences. Journal of Information Science, 24(1): [18] White, H.D. (2003). Pathfinder networks and author cocitation analysis: A remapping of paradigmatic information scientists. Journal of the American Society for Information Science and Technology, 54(5): [19] White, H.D. & McCain, K.W. (1998). Visualizing a discipline: An author co-citation analysis of information science, Journal of the American Society for Information Science, 49(4): [20] White, H.D. & Griffith, B.C. (1981). Author cocitation: A literature measure of intellectual structure. Journal of the American Society for Information Science, 32: