Information-Seeking Strategies of Novices Using a Full-Text Electronic Encyclopedia

Information-Seeking Strategies of Novices Using a Full-Text Electronic Encyclopedia Gary Marchionini College of Library and Information Services, University of Maryland, College Park, MD 20742 An exploratory study was conducted of elementary school children searching a full-text electronic encyclopedia on CD-ROM. Twenty-eight third and fourth graders and 24 sixth graders conducted two assigned searches, one open-ended, the other one closed, after two demonstration sessions. Keystrokes captured by the computer and observer notes were used to examine user information-seeking strategies from a mental model perspective. Older searchers were more successful in finding required information, and took less time than younger searchers. No differences in total number of moves were found. Analysis of search patterns showed that novices used a heuristic, highly interactive search strategy. Searchers used sentence and phrase queries, indicating unique mental models for this search system. Most searchers accepted system defaults and used the AND connective in formulating queries. Transition matrix analyses showed that younger searchers generally favored query refining moves and older searchers favored examining title and text moves. Suggestions for system designers were made and future research questions were identified. Introduction Electronic systems for storing and retrieving information are used by increasing numbers of people who are not information specialists (end-users), and as Ojala [28] pointed out, this trend is likely to accelerate. The use of Electronic Information Systems (EN) by novice or casual users is driven by hardware developments, for example, personal computers, optical storage, etc. and software developments, for example, menu driven user friendly interfaces, pseudo-intelligent front ends, etc. The availability of inexpensive, easy to use full-text systems presents opportunities and challenges for information professionals and end-users alike. The information-seeking process is a complex interaction among several factors and success is dependent on methods as well as tools. The challenge to information professionals is to design and implement efficient and effective search systems and databases for end- Received December 29, 1986; revised March 4, 1987; accepted March 10, 1987. % 1989 by John Wiley & Sons. Inc. users. The challenge to end-users is to understand the many facets of the information-seeking process so that they can make full use of these emerging systems. This article reports the results of research on information-seeking strategies used by novice users searching a full-text electronic encyclopedia. The research was conducted from a cognitive process perspective. The study examined the following questions: 1. Can novices use such a system successfully with little formal training? 2. What features of full-text retrieval do novices apply? 3. What are the relationships among user, task and search pattern? 4. What search patterns are exhibited and how are these patterns related to information-seeking strategies? Answers to these questions will give immediate guidance to designers of full-text retrieval systems meant for end-users, and to designers of instructional materials for their use. Results and methods of data collection and analysis will also contribute to an emerging cognitive theory of information-seeking. Information-Seeking Theory Information-seeking is a special case of problem solving. It includes recognizing and interpreting the information problem, establishing a plan of search, conducting the search, evaluating the results, and if necessary, iterating through the process again. Studies of where and how people look for information (see, for example, [10,24]) highlight the interaction of personal factors such as experience and knowledge, and the information need. As with problem solving in general, understanding the informationseeking process requires exploration of human cognition, and we lack direct methods for such exploration. A general procedure is to observe behavior in well-controlled situations and use the observations to construct a model of the cognitive process. By incrementally modifying the conditions of observation the model is refined and generalized. This article reports on a series of observations of a group of novice users applying a single database and JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE. 40(1):54-66, 1989 CCC 0002-8231/89/010054-13$04.00

search system to preselected search tasks. The results are therefore specific to this combination of users, tasks and tools but provide a base for future study of informationseeking and the design of similar systems. Three overlapping bodies of literature support a theory of informationseeking and provided guidance in the conduct of this study: online searching, electronic information systems, and cognitive science. Online Searching The literature related to online searching is rich and varied. Many studies have examined the search behavior of expert searchers [ 1,2,16,26]. Fenichel [ 151 reviewed the online searching literature and illustrated the complexity of studying information-seeking. She noted that there is great variation in approaches taken even within controlled settings and that even experienced searchers did not take full advantage of system features (see [3,20] for additional reviews). End-users who do not use an intermediary are typically novice or casual users of the retrieval system and their direct use of such systems has been considered by several researchers [5,23,25,33]. The results thus far show that novices use simple and direct strategies for conducting search. There is a need to systematically examine searches conducted by novices to understand their thinking so that features for improving their strategies can be built into future search systems and instructional materials. Fidel and Soergel [ 171 described a framework for online searching which defined the elements for an informationseeking theory. Their framework included: setting, user, request (task), database, search system, searcher, search process, and outcome. Each of these factors is complex, having many facets which are not easily quantified. Moreover, these factors are interdependent. This framework, with some modification, was used as the basis for constructing the research reported here. One simplification caused by considering novice users was that the searcher and user were the same person. Electronic Information Systems (EIS) Two developments pertinent to this research provide exciting potentials for information storage and retrieval: fulltext databases and CD-ROM storage systems. Full-Text Databases. Full-text databases offer great opportunities to professionals and end-users because they can provide primary information efficiently. Full-text databases on CD-ROM makes large scale end-user access possible. However, at present, there is no conclusive evidence about the performance of full-text database systems; there are few results about novice users applying such systems; and, there is no research on what search strategies are best applied in full-text environments. Questions about the performance of full-text systems have yielded both positive [32,36] and negative [4] results. Even user satisfaction results are mixed (see [12] for positive indications and [ 191 for negative results). Tenopir [ 351 summarized differences between searching full-text and controlled vocabulary databases. She suggested the use of proximity operators before boolean operators in fulltext systems, and attention to the use of synonyms and specific natural language terms. She recommended the combining of full-text and controlled vocabulary techniques, and pointed out how little is really known about full-text searching. Full-text databases have great implications for end-user information-seeking because full-text systems can accommodate and compensate for end-users typically simple search strategies. However, because extensive full-text searching is not feasible manually, we have no experiential models upon which to develop conceptual models of fulltext searching. Using a full-text database could significantly affect a users cognitive information-seeking system in a variety of ways. For example, a full-text database can be thought of as having an exhaustive index and the user can modify his/her information-seeking system by generalizing the existing rules for using paper-based indexes, eventually affecting the use of all indexes. On the other hand, the electronic database can be viewed as a sequentially searchable main file, which empowers a simple facet not practical in large, print databases; thus calling for modification of the user s internal rules for selecting tactics and terms. Moreover, it is possible that the full-text lookup could be perceived as something entirely new, requiring additional strategies to be added to the user s informationseeking system. Perhaps subjects who fail to generalize are confused by the full-text indexing and their performance suffers accordingly. Novice users experience with such systems can begin to illustrate how users cope with fulltext databases and provide guidance for database and search system designers as well as to the end-users of these systems. CD-ROM Technology. The ability to store up to 600 megabytes of data on a single surface twelve centimeters in diameter with seek times under two seconds is clearly cause for attention. See Chen, P. [ll] for a technical description of CD-ROM technology and Chen, C. [9] or Desmarais [14] for overviews of applications. The first widely distributed text database delivered on CD-ROM was Grolier Electronic Publishing s The Electronic Encyclopedia. The text of the twenty-volume set occupies about 60 megabytes. The database is full-text searchable through a fully inverted, 50 megabyte index of all non-stop words in the database. This encyclopedia is geared toward middle school and young adult readers. Kister [22] pointed out that the best feature of the print version of this encyclopedia was the excellent use of graphics. It is somewhat ironic that the electronic version is text-only. The search system was designed to require no instruction, and it is delivered on floppy disk with the drivers to enable various CD-ROM players. The software represents the encyclopedia as hypertext, a highly interactive network of articles. It is menu-driven and supports both direct lookup of articles and full-text searching. The full-text JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-January 1989 55

searching component supports boolean (and, or and not), character masking, right truncation, and proximity functions. As Connolly [ 131 pointed out, at approximately onethird the cost of the print version, this encyclopedia is attracting much attention in schools and libraries. Because cost is not such a constraint when using a system like the electronic encyclopedia, searchers could employ more display tactics in conducting the search, including actually reading articles online. Just as water and electricity seek paths of least resistance, so humans seek the path of least cognitive load. Because this system displays article titles together with frequency of occurrence of query terms, users can scan these lists using this frequency data to make relevance judgments about whether to examine the text of articles. Moreover, when an article is displayed, the terms from the query formulation (which caused the hit) are highlighted on the screen, thus facilitating browsing via scanning of the text for relevance. By supporting scanning which is a recognition task, this system provides a low cognitive load strategy for searching. Systems which encourage much refinement of queries demand higher cognitive load since the task of query formulation involves at least recall procedures which themselves are executed more slowly (Card, Moran & Newell, [7]). In essence, it takes less concentration and cognitive effort to scan lists of items from which to make a choice, than to identify and recall synonyms and combine facets using logical connectives. Whether systems that support a scan and select strategy yield effective results requires investigation in a variety of databases and settings. Mental Models Of all the factors relevant to information-seeking, the human information seeker is the most complex. Variables such as intelligence, experience, motivation, and a host of other individual characteristics certainly affect information-seeking performance. The perspective taken in this research is based upon a theory of cognition which proposes that performance in applying principles or tools is dependent on dynamic internal representations of those principles or tools called mental models. Norman s research [27] demonstrated that mental models are ill-defined, incomplete, and can be illogical. In general, a mental model is a cognitive representation of a problem situation or system which is active in the sense that it can take inputs from the external world and return predictions of effects for those inputs. It can be run to allow predictions which then determine what actions should be taken. Mental models serve the dual purposes of representing entities and relationships which are refreshed and extended by experience, and simulating the possible effects of acting on these entities and relationships. Thus, mental models allow us to both understand problem situations and predict consequences of actions contemplated for solving the problems. Johnson-Laird [21], in a seminal work, explained a mental model theory for inferencing from both empirical and theoretical perspectives. Young [39] provided a taxonomy for mental models which attempted to unify much of the cognitive science research on representations for complex systems and processes. Mental models for specific mathematical and physical systems have been explicated and Borgman [5] examined how mental models for an information storage and retrieval system are best acquired (see [18] for a collection of studies, and [6] for a general review). For this research, a system of mental models for the general problem of seeking information is assumed. This system controls the combination of several specific mental models related to a particular information problem. Such a system can be described functionally and structurally. Functionally, an information-seeking system controls search by: extracting key concepts from the information problem, identifying criteria for search success, selecting candidate information sources, monitoring lookup (search) and examination procedures, and using results to modify itself. Structurally, an information-seeking system includes a set of mental models associated with various information sources (databases and accompanying search systems), a set of mental models pertinent to a particular information problem (task domain knowledge), an historical record of past applications of the information-seeking system (selfawareness which allows analogy and checks context), and a set of rules for combining these components and monitoring progress. The information-seeking system is assumed to be a control mechanism which can be run when instantiated with inputs for a particular problem. Perhaps the most interesting characteristic of an information-seeking system is its constant evolution. Questions about how the informationseeking system changes structurally and functionally have theoretical and practical implications. Each time the information-seeking system is applied, it must adapt. The execution (run) of an individual s information-seeking system for a particular information problem is considered an Information-Seeking Strategy (ISS). Each informationseeking strategy leads to modification of the general information-seeking system. An ISS is clearly task driven, the task serving as stimulus to activate the informationseeking system. An ISS is manifested behaviorally by the actions taken in conducting a search-a search pattern. A single action of the information-seeking strategy is considered a tactic. Tactics are manifested behaviorally by individual moves made during a search, for example, lookup a particular term, examine a citation or article, etc. Before the complexities of mental models for particular types of search systems can be understood, the databases, search systems, and task domains must be described and understood. The purpose of this research was to begin these descriptions from the perspective of novice users. Method The general procedure taken was to introduce students to an electronic encyclopedia; assign two search tasks; 56 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-January 1989

collect data through observation and keystroke capturing techniques as subjects conducted searches for the tasks; summarize data for group and task by success, time and number of moves; and analyze search patterns by examining query formulations and search moves. Hypotheses relating groups and tasks on the dependent variables success, time and number of moves were tested, but since the research was mainly exploratory with respect to informationseeking strategies, descriptive data were presented and research questions were generated. Twenty-eight third and fourth graders and twenty-four sixth graders in a talented and gifted program in an urban school setting participated in the study. Subjects were exposed to two forty-five minute explanation/demonstration sessions on use of the electronic encyclopedia. They were then assigned to pairs by their respective teachers and reported to the media center on two separate occasions, conducting one of two assigned searches on each occasion. Student pairs conducted their searches in a separate, quiet area of the center using an IBM-PC computer driving a Phillips 100 CD-ROM player and color monitor. The search topics were given to them upon arrival by a project member who observed and recorded the team s actions during the search. The observer gave a brief review of system commands before students began their searches but answered no questions and made no comments once the searches began. User keystrokes were captured in an unobtrusive manner during their searches. Subjects All students had had previous computer experience, including an introduction to keyboarding and some computer assisted instruction. None of the students had ever used an electronic encyclopedia, and the print version of this particular encyclopedia was not available in the school. The key subject characteristics were the cognitive information-seeking system, the knowledge base(s) for the task domains, and mental models for an encyclopedia and the search system. The effectiveness of a search was presumed to be dependent on the accuracy of mapping these internal representations onto the actual tasks, source, and system used. Rouse ]31] pointed out that... human information processing abilities plateau at a relatively early age while information-seeking abilities continue to improve. (~131). Since mental models develop with experience it was reasonable to assume that young children would have less developed information-seeking systems and thus exhibit less sophisticated and successful information-seeking traces than older students. Thus, it was assumed that outcomes for older subjects (sixth graders) would be superior to outcomes for younger subjects because they had more highly developed mental models for information-seeking and broader task domain knowledge bases. Tasks Information-seeking is problem driven; the problem situation must certainly affect the information-seeking strat- egy applied and the outcomes of searching. From a cognitive perspective, an information need occurs when a knowledge base for a task domain is activated and requires instantiation or modification. The information processing system is called into action by passing relevant facets of the task domain to it for completion. The interplay between task domain knowledge and the information-seeking system is manifested in the terms used in conducting a search. For this research, the problem situation was controlled; search topics were assigned. Note that this created a somewhat artificial setting for the information-seeking system in that search was externally motivated and the search statement itself presented language which suggested query terms. Two tasks were designed for use by all student pairs. These tasks were pilot tested and refined before the research began. Students were told to imagine that they were media specialists asked by a teacher to find particular information. Both tasks were stated in paragraph form (See Appendix A). One task required students to find a fact - the first year speed skating was introduced into the Olympic games. This task required students to combine three facets (concepts): place, activity and time. This was termed a closed task. The other task required students to find information about women who have traveled in space. Three main facets were also combined for this task: person, place, and activity. This was termed an open task since there were many possible names and associated facts to retrieve. Both questions were designed to be motivating to elementary student interests. All student pairs conducted both searches except one third grade pair who conducted only the open task due to absences from school. The order of the search was alternated for each group so that half the groups did the closed search first and the other half did the open search first. It was hypothesized that outcome would be dependent on task, i.e., one task would take more time and be considered more difficult by students; and that information-seeking strategies would differ for the two tasks. Outcome Measures Three outcome measures were considered: success, time to complete a search, and total number of moves (tactics) used. Subjects recorded relevant information and titles as they conducted the search. For the closed search, subjects were judged to be successful if they found the correct fact, otherwise they were judged to be unsuccessful. Judging the open search was more difficult since it was possible to find a variety of information about the several women who have explored space. Two measures for success were used for the open task: a dichotomous scale, as in the closed search-subjects were judged to be successful if they found information about at least one female space traveler; and the number of relevant articles they listed on their search worksheets. Students were given a maximum of 45 minutes to complete each search. The observer noted starting and stopping time for each search and interrupted searches if 45 minutes JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-January 1989 57

passed and subjects had not yet completed their search. Total number of moves were determined by corroborating observer notes and keystroke data. These data served as dependent variables for comparing groups and type of task variables. It was hypothesized that success, time taken to complete search, and total number of moves made would be dependent on user group and task. To test these six hypotheses, non-parametric statistical procedures were used. Because success was not measured on an interval scale, and neither success nor time were found to be normally distributed, Kolmogorov-Smirnov tests for two independent samples were used at an alpha level of 0.05 for all hypotheses related to outcomes (Siegal, [34]). The SPSS X computer program was used to conduct the data analyses. Search Process It was assumed that the strategy a user applies is dependent on his/her mental models for the task, database, and search system and how the information-seeking system manipulates those models. The discrete moves made by searchers are considered as traces of this information-seeking strategy. Which moves are used and in what combinations depend on the interaction of user, setting, task, database, and search system factors. The process used by a professional searcher using an online system is surely distinct from the process used by an end-user searching a CD- ROM based encyclopedia. The set of tactics available to users are constrained by the search system as well as the users mental models. The search process is difficult to understand because it is composed of the interaction of all the factors discussed above. An approach to analyzing the results of a humanmachine interaction is to have the machine record the interactions unobtrusively. Rice and Borgman [30] discussed the advantages of this type of data collection and Card, Moran and Newell [7] made extensive use of the method in building a theory of information processing systems. Although the method of data collection is appealing, problems of analyzing these large quantities of data remain. One approach to analyzing search data is to examine key aspects of the data in discrete, descriptive fashion. In this study, this approach was taken for use of system features, query terms used, type of query formulation, and use of system feedback. These data were extracted from keystroke traces and observer notes and frequencies or proportions used to summarize them by group and task. Another approach to organizing and analyzing keystroke level data is to define a state map of possible moves and characterize each search pattern as a sequence of state changes according to the state map. By assuming that arrival at a certain state is dependent on the previous state, the search pattern can be modeled as a Markovian process. Transition matrices for various lengths of sequences can then be formed and compared. This method was first used by Penniman [29] to examine user search and system response patterns in a bibliographic database system. Tolle [37] used the method to describe use of different online catalogs, and Tolle and Hah [38] used the method to compare NLM databases. Borgman [5] used transition matrices as one of the measures for comparing training treatments and Chapman [8] used the method to compare groups of searchers. A variation of this technique was used in this study to identify search patterns and, by inference, information-seeking strategies. The actual procedure was to develop a state map, count occurrences of moves in each state, form transition matrices, examine these frequencies and matrices, collapse the state map and original matrices into a simpler state map and corresponding matrices and compare them across groups and tasks. First, the system options for querying and viewing results were considered and a state map of tactics (moves) constructed. Examination of all possible actions resulted in a state map definition which took into account moves assumed to be related to the informationseeking strategy. It is important to note that the state map was constructed from the user s rather than the system s point of view. The states were abstractions of user behavior rather than simply keystrokes or menu choices. Although some of the states correspond to a single keystroke, for example, show titles and show text, most required a sequence of keystrokes, for example, enter or edit an entire query formulation. By this method, what would appear as a single system state- the key which initiated searchwas considered as six states from the user s viewsix ways of entering or refining queries. Since the focus was on query formulation and revision, the codes represented types of query revisions rather than system features. Thus, the possible states were constructed to include all possible intellectual moves rather than system moves. The state map developed included the following actions which were grouped under the two categories of lookup and examine: Lookup-original term, new term, broader term, narrower term, synonym, reorder terms, change relation (system defaults); Examine-show titles, show text. Note that lookup was used to group moves that initiated or refined queries, or altered system defaults; the query moves were always completed by pressing the function key which signaled the system to begin search with the entered query and the change relation moves were signaled by one of two function keys. Since users in this study were not permitted to print or save their results, examine was used to group only two moves which allowed them to view results of search; these moves each corresponded to a function key. This state map was used to tabulate frequencies of moves for each state and to form first order transition matrices for each search. A first order transition matrix was formed by crossing the state map with itself, thus forming a nine by nine grid defining all possible two-step moves. The matrix was completed by counting all two-step moves for each search. Since judgments about how queries were changed (for example, whether a change was a broadening or narrowing) could only be made by humans, all coding of individual transition matrices was done by hand. As 58 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-January 1989

searches were coded, patterns began to emerge. Two patterns in particular were noted. Some searches exhibited heavy concentrations in the upper left quadrant (lookup/ lookup) i.e., these searchers made more query formulation and refinement moves than examine moves. Others were more heavily concentrated in the lower right quadrant (examine/examine). This led to concentration on the general moves, lookup and examine. The nine by nine first order matrices were collapsed into two by two matrices with the following cells: lookup/lookup, lookup/examine, examine/ lookup, and examine/examine. The ratios of each resulting cell to the total number of moves were formed and used to make comparisons between groups and tasks. Two other ratios were formed to allow comparisons based upon the patterns described above. One ratio compared number of lookup moves to number of examine moves. The other represented the tendency to stay in a state and was formed by dividing the sum of lookup/lookup and examine/examine by the sum of lookup/examine and examine/lookup. To conduct comparison tests with data which reflected frequency of state use or state change, assumptions about population distributions were made which allowed parametric statistical tests to be made. The overall sample distribution of moves was found to be normally distributed via a Kolmogorov-Smirnov goodness of fit test (2 =.595,p =.87) which supported the assumption of population normality. The level of measurement was clearly interval and thus a t-test procedure was used to test differences between group and task types of moves. Because hypotheses were not stated a priori, these analyses yielded suggestive rather than demonstrative results. Results Results were organized into three groups: 1. Descriptive results for success, time, number of moves, and use of system features; 2. Analyses of query formulation from three perspectives: distinct facets used in query formulation, combination of facets used in query formulation, and use of system feedback to modify query formulation; 3. Examination of search patterns through transition matrices, in particular, the use of lookup and examine moves. Descriptive Results Data for success, time and number of moves were summarized and compared by group and task. Success. Most subjects were successful in finding the required information. Frequency of success is reported by group and task in Table 1. Two-thirds of all the searches were judged to be successful on the dichotomous scale. Sixth graders were clearly more successful than the younger subjects. These differences were statistically significant (Kolmogorov-Smimov 2 = 1.78, p =.003). It is not surprising that older, more experienced subjects were TABLE 1. Frequency of success by group and task. Successful Unsuccessful Open Closed Open Closed Grades 314 6 5 8 8 Grade 6 11 10 1 2 Total 17 15 9 10 more successful in finding information. What portions of these differences are due to task domain knowledge base, information-seeking experience, and system manipulation require further study. Since subjects were asked to record all relevant articles, a second measure of success was possible for the open task which required multiple articles for full consideration. The mean numbers of articles listed were 1.4 by the younger group and 2.9 by the older group. Older subjects consistently found more relevant articles and these differences were statistically significant (Kolmogorov-Smimov 2 = 1.574, p = 0.014). Subjects were equally successful on both tasks; that is, fifteen searcher pairs were successful on both, six were unsuccessful on both, and only four were successful on only one of the tasks. No statistical difference was found (Kolmogorov-Smimov Z = 0.099, p < 0.99). Although task certainly plays a role in determining what queries are formulated, either the user factor was more dominant in determining success or the two tasks assigned in this study were not discriminating enough to affect success. Time. The amount of time taken to conduct searches was dependent on both group and task. The mean time taken for all searches was 36 minutes (39.4 minutes for younger group, 32.2 minutes for the older group). The differences in time taken by the two groups to complete a search was statistically significant. (Kolmogorov-Smimov Z = 1.419, p = 0.036). It should be noted that 18 of the younger group searches were stopped at the 45 minute maximum and only 8 of the older group searches were stopped. It is likely that even greater differences would have been found had there been no time limits on searches. That younger subjects would take more time was predictable since their reading rates when examining articles were generally slower. Future studies should consider individual components such as query formulation time and reading time rather than only the aggregate search time. Time was also dependent on task. The mean time for the open search was 41.1 minutes and the mean time for the closed task was 30.7 minutes. Differences in time between the two tasks was statistically significant (Kolmogorov-Smimov Z = 1.598, p = 0.012). Since subjects were looking for multiple facts in the open task, it is not surprising that they took longer to search for those facts. Moves. The number of moves taken to complete each search was examined by group and by task. The mean number of moves taken for all searches was 21.O (22.4 for the younger group, and 19.5 for the older group). Differences between the groups on number of moves made were not statistically significant (Kolmogorov-Smirnov JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-January 1989 59

L = 0.644, p = 0.802). Although older subjects took fewer moves in general, they typically found more relevant information for the open task; their moves were more efficient. As expected, number of moves was dependent on task. The mean numbers of moves for the tasks were 24.7 for the open task and 17.2 for the closed. These differences were statistically significant (Kolmogorov-Smirnov z = 1.598, p = 0.012). Overall, results for this sample produced predictable patterns with respect to success, moves, and time. An interesting relationship between success and number of moves was found (Pearson R = -0.43. p < 0.01). Success was more likely to occur with a small number of moves. This supports the rather obvious notion that success is related to the quality of moves rather than the quantity. Use of System Features. The search system offers a variety of features to control search. The main screen used for query formulation consists of six windows: function key menu to aid in move selection; option menu to allow scope changes; relation menu to allow proximity changes and enable the NOT operator; result window to display intermediate results of a search; query formulation window which provides five lines for queries; and a status window which shows a trace of screens displayed, i.e. where one is in the program. The query formulation window begins the second through fifth lines with the words along with to explicate that terms on separate lines are linked by AND. To change these defaults to NOT, the relation menu must be selected and NOT chosen from within that menu. No subjects used the NOT feature. To use the OR connective, terms are linked on a single line by a comma, but there is no explicit indication of this feature anywhere except in the manual. Likewise, masking and right truncation features are explained only in the manual. Table 2 presents a summary of what system features were used by group and task. Note that most subjects generally used the default conditions of the program. Forty of the fifty one searches (78%) used the AND connective. Given the subjects willingness to explore other features of the system, this nonuse of OR and NOT should be considered in future studies. In particular, the system s handling of these features should be carefully considered. The use of the truncation feature illustrates that students did learn from the TABLE 2. Number of subjects using of system features. Third/Fourth Sixth Feature Open Closed Open Closed AND 12 (49) 9 (53) 11 (73) 8 (40) OR 0 0 1 (3) 0 MASK 0 0 0 0 TRUNCATE 5 (20) 4 (14) 2 (15) 0 SCOPE 2 (2) 0 1 (4) 0 PROXIMITY 1 (1) 2 (5) 5 (10) 5 (11) Note: Figures in parentheses are total number of times each feature was used. instructional demonstrations since that was the only introduction to that feature and students were not given access to the manual. Use of proximity features is interesting, but based on observer notes, it is clear that many of these efforts were meant for exploring the system rather than focusing or expanding search. Two sixth grade pairs were particularly fascinated by proximity selection and changed proximity in both of their searches. When success was considered, use of features other than AND differed for the younger and older subjects. Of the fourteen sixth grade searches that used some system feature other than AND, eleven were successful. Of the fourteen third/fourth grade searches that used some system feature other than AND, only one was successful. Clearly, the younger users who were successful attended (wisely) to the query formulation and feedback and ignored the complexities of the system by accepting defaults. Study of the correct use of powerful search features deserves attention from both system designers and educators if novices are to take full advantage of electronic information systems. Overall, these summary data reveal reasonable and consistent results. If anything was surprising, it was the generally high level of performance of these young novices. Query Formulation Examination of query formulations served a primary role in exploring users information-seeking strategies. Query formulation was examined from three perspectives: the use of key facets for each task; the actual formulation of queries by combining facets, other vocabulary, and system features; and the effects of system feedback on refinement of the original query formulation. Task Facets. Consideration of the selection of terms provided an indication of how well the task was understood and internally represented - an inferred look at users task domain knowledge base. The two tasks used in this study were constructed to be concrete and minimally complex. Although stated in paragraph form, with distracting information present, subjects typically used terms present in the task statement. This surely presented a skewed look at subject knowledge bases since the task was formulated for them. The open task required subjects to find information about women who have travelled in space. The facetsperson, place, and activity-were easily grasped by subjects and five terms were commonly used to represent these facets. Table 3 presents the proportion of occurrence for these five terms for all open task queries by age group. Note that multiple terms often occurred in single queries and thus the percentages do not sum to 100. In general, subjects chose appropriate terms. Other terms used less often included: lady, NASA, spacecraft, human, pilot, etc. The fact that most subjects were able to identify reasonable terms for the tasks indicated that their task domain knowledge and ability to extract key facets from this task statement was good and the results of their searches were due 60 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-January 1989

TABLE 3. Proportion of occurrence of key terms in open task. TABLE 5. Query type by group and task. Term Third/Fourth Sixth Third/Fourth Sixth Woman/women 32% 50% Female 18% 22% Space 53% 52% Traveler 27% 20% Astronaut 11% 19% Note: Percentages are based on the total number of queries. There were 191 queries for third/fourth graders and 151 queries for sixth graders. more to their information-seeking experience and use of the EIS itself. The closed search required subjects to find the year that speed skating was introduced into the Olympic Games. For the closed search, sixth graders made many fewer queries than the third/fourth graders. These results are presented in Table 4. The older subjects were much more likely to use the specific term speed skating, which resulted in 10 occurrences of that term in four articles, including articles on ice skating and Olympic Games, either of which when selected, displayed speed skating on the same page as the date, i.e., completed the task. Whether the ability to select an appropriate level of conceptual specificity is a result of physiological differences between these two age groups as suggested by developmental psychology, or is due to information-seeking experience should be explored in future studies. Like the open task, this task involved three facets-place, activity and time. The abstract facet, time, was largely ignored and activity at specific and general levels dominated search. For a well-defined activity, like speed skating, this was sufficient to allow quick success. Terms in Query Formulations. A second view of term selection was gained by examining the actual query formulations. Since subjects were novices from both information-seeking and search system vantages, it was not surprising that so many used natural language queries. Some subjects actually entered full length questions to the system. Examination of all queries allowed classification into six categories-a single term on a single line, single terms on multiple lines (terms connected by AND), a phrase consisting of terms and/or adjectives on a single line, phrases on multiple lines (phrases connected by AND), a sentence (term(s), verb(s) and possibly modifiers) on a single line, and sentences on multiple lines. Term here actually meant facet since expressions such as speed TABLE 4. Proportion of occurrence of key terms in closed task. Term Third/Fourth Sixth skating 14% 2% speed skating 52% 77% Olympics 37% 59% event 19% 17% Note: Percentages are based on the total number of queries. There were 139 queries for third/fourth graders and 66 queries for sixth graders. Quev be Open Closed Open Closed one term 35 (18%) 23 (17%) 22 (15%) 12 (18%) one term per line 30 (16%) 30 (22%) 40 (26%) 16 (24%) one phrase 51 (27%) 43 (31%) 35 (23%) 11 (17%) one phrase per line 20 (10%) IO (7%) 27 (19%) 17 (26%) one sentence 52 (27%) 26 (19%) 22 (15%) 5 (8%) one sentence per line 3 (2%) 7 (5%) 5 (3%) 5 (8%) skating and space travel were counted as single terms. Table 5 presents frequencies and proportions of each type of query by group and task. Younger subjects were more likely to use actual sentences to query the system. In a sense, these novices assigned considerable intelligence to the system. This reflected a lack of understanding on the part of these users about how the system worked- a poorly defined mental model of this system. This is not at all surprising, since none of the subjects had any previous online searching experience. One possible explanation is that subjects used their existing mental model for an encyclopedia as a base and added a computer component. The computer component was intelligent since all subjects knew that computers are interactive. Future studies should examine how use of natural language (sentences and phrases) lessens as users become more experienced with the system. Less than half of all the query formulations were single terms or single phrases. These would likely be the type of queries formulated by these subjects using a print encyclopedia because they had reasonable amounts of experience with how lookup is performed in a print encyclopedia. Clearly, subjects recognized that the electronic encyclopedia was conceptually different than the familiar print version. Feedback Effects on Query Formulation. A major advantage of an interactive search system is the ability to plan for and use intermediate results in conducting a search. To examine how subjects used system feedback to modify their queries, an analysis of initial queries and first refinements was conducted with respect to the system feedback. Because the subjects were novices, the initial query likely represented a best approximation of an information-seeking system run. Since no strategies like the building block approach were apparent, this is particularly reasonable. The results of original query formulation and subsequent modification were categorized into five cases: no hits on initial query and no hits on subsequent query (labeled 0->O); no hits on initial query and some hits on subsequent query (labeled 0->hits); some hits on the initial and no hits on the subsequent (labeled hits->o); hits on both queries (labeled hits->hits); and hits on the first query and immediate success (labeled hits->succ). Table 6 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-January 1989 61

TABLE 6. Results of first two queries by group and task. Third/Fourth Sixth Results Open Closed Open Closed O-0 8 4 7 3 0 --, hits 4 1 3 1 hits + 0 2 3 1 3 hits -+ hits 0 2 1 3 hits + succ 0 3 0 2 presents these results by group and task. Most of the searches (43%) yielded no hits on the first two queries. These results were typically due to the use of phrases and sentences in forming queries. In general, subjects eventually dropped modifiers and verbs and found title lists to examine; they adapted their information-seeking tactics. Searchers who began with no hits and adjusted their query formulation to find hits would seem to have good understandings of the system and be making progress toward success. Most (seven out of nine) of the searches in this category were on the open task, and seven out of nine were eventually successful. For five of these searches, progress was made via the second query. Two of the searcher pairs broadened their original query to space, thus retrieving 894 articles -an overreaction to having no hits. The other two searches illustrate some unique characteristics of this full-text system. One pair of searchers began with women astronauts, a reasonable term, which yielded no hits. The modified query, woman in space is conceptually more specific which logically would not be a good choice if the aim was to move to a broader topic. However, this query retrieved six articles (all relevant). Moreover, had the revised query used the plural form women in space, no hits would have been reported. The other search in this category reinforces the somewhat arbitrary nature of full-text retrieval when phrases or multiple terms are used. The original query was a phrase, year of speed skating, which yielded no hits. The revised query, year AND of speed AND skating yielded three articles. (Note that the default proximity condition limited occurrences to a paragraph.) This awkwardly constructed query was actually useful in retrieving relevant information. It is clear that some thesaurus should accompany such systems to help control the arbitrary nature of written language and help users identify word forms and synonyms. Some insights into subjects lack of strategy and followthrough was gained by examining the nine searches which began with some hits but which yielded no hits after reformulation. These searches would seem to indicate counterproductive moves and only four of the nine were eventually successful. Most of these searches (seven of nine) involved the closed task. For two of the searches, the original query formulation did not yield fruitful beginnings-one retrieved 894 articles, the other a single false drop. One pair of subjects did not bother to examine a good set of 16 titles at all, and another pair selected a false drop and ignored the relevant articles before reformulating their query. The other five searches were all similar in that they selected a relevant article, Olympic Games, which brought up a page of text that indicated in two sentences that the winter Olympics began in 1924 and included speed skating as an event. None of these subjects (three pairs of sixth graders and two pairs of third/fourth graders) extracted this fact from the two sentences and thus continued with their search. Only two of these pairs eventually succeeded in finding the required fact. The extent of such failure to extract information given appropriate text, and whether this inability to extract information once its context was retrieved is due to subjects losing sight of the goal because they were focused on the system, the situation, or reading, bears future investigation. Observer notes indicated that some subjects used terms they discovered in reading text to refine subsequent queries. For example, names of astronauts were subsequently used in queries and the term astronaut itself was used in queries immediately following examination of the article space exploration in which it appeared. The observer also noted subjects comments about selecting articles to examine because of high frequencies of term occurrence. These observations demonstrated that searchers used feedback from the system in examining title lists and article text as well as from system reports of number of hits immediately after a query. Search Pattern Analysis In this study, a search pattern was the set of moves users made during an entire search session. A general characteristic of search patterns that makes them difficult to compare is that they are unique entities. In this study, they varied in length from two to 51 moves, were distinct with respect to query formulation, how the queries were ordered, use of system features, and yielded a variety of outcomes. Using the state map described in the methodology section, frequency counts for each state (move) were analyzed and order one matrices were formed and discussed. The state map used to generate the transition matrices was then collapsed to focus on two conceptual states-lookup and examine. New transition matrices were formed and analyzed by forming ratios among various cells and comparing those ratios by group and task. State Map Frequency Analysis. Table 7 presents the mean number of times users entered each state (made a particular move) organized by group. The mean differences between groups was statistically significant for only one type of move-narrow. Younger subjects in general spent more time refining their queries, and narrowing was the most frequently made move. Narrowing was used by subjects even when no hits were obtained. This illogical action may be due to the inverse relationship between use of AND and resulting outcome, i.e., adding terms linked by AND (the default in this system) restricts searchactually retrieves less articles. This relationship was obvi- 62 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-January 1969