Phase 2 Volume I Use of Electronic Science Journals in the Undergraduate Curriculum: An Observational Study National STEM Educational Digital Library Program Sponsored by the National Science Foundation As part of the National Science Digital Library September 30, 2004 Principal Investigator: Dr. Carol Tenopir, Professor NSF No.: 01-55 DUE: 0121575
Acknowledgements The research team for phase 2 of this project includes UT faculty, graduate students, and representatives from OSTI. I would like to thank these dedicated team members: Richard Pollard and Peiling Wang, University of Tennessee School of Information Sciences faculty members; Julia Krummen, Dan Greene, Beverly Simmons, Kyoungsik Na, Elizabeth Kline, Matt Grayson, Michael Schlitz, and Yan Zhang, all graduate research assistants; Lynn Davis, Karen Spence, Rita Hohenbrink, and Mary Schorn, OSTI staff members. Many OSTI staff members answered our questions and participated whenever asked. A special thanks to Walter Warnick and R.L. Scott of OSTI who made this cooperative UT-OSTI project possible, and to Karen Hunter and Michael Mabe of Elsevier who arranged research access to ScienceDirect.
Table of Contents Acknowledgements... i Table of Contents... ii Executive Summary... 1 1. Introduction... 2 1.1 Objectives... 3 1.2 Literature Review... 3 1.3 Research Design... 7 2. Data Analysis... 17 Part I Data analysis in Observer... 17 Part II Data analysis using Microsoft Access... 19 Part III Verbal transcription and data analysis in QSR N6 software... 21 3. Preliminary Results... 24 4. Conclusions and Further Analysis... 33 5. Bibliography... 35 - ii -
Executive Summary Phase 2 of a three-phase project funded by the NSF National Science Digital Library Project observed five different groups of participants as they searched an e-journals system for scholarly science journal articles for simulated class-related assignments. These groups were undergraduate and graduate engineering, chemistry, and physics students; undergraduate Speech 210 students; librarians; and science faculty. All groups searched the ScienceDirect system and some graduate students searched the Energy Citations Database. Think-aloud protocol was used to capture affective and cognitive state information, while online monitoring provided an automatic log of interactions with the system. Pre- and post-search questionnaires and a learning style test provided additional data. Preliminary analysis shows differences in search patterns among undergraduates, graduates, and faculty. All groups used the basic search functions most frequently. Graduate students on average spent more time per session and viewed more pages. Further analysis, including analysis of affective and cognitive reactions is continuing. - 1 -
1. Introduction A two-year project for the National Science Foundation/National Science Digital Library project (NSDL) attempted to discover how faculty and librarians can encourage sustained use and understanding of scholarly literature by science students, including the role of journals in the undergraduate curriculum and what e-journal system features encourage use. Phase 1 used focus groups of students, faculty, and librarians to reveal current use behavior and opinions of what they think is needed in electronic journals systems and class assignments to encourage sustained use by undergraduate science students (Tenopir 2003b). The phase 1 study concluded that e-journals should be incrementally introduced to students starting at the time they declare a major (http://web.utk.edu/~tenopir/nsf/presentations.html.) E-modules developed by the library and faculty could introduce the structure and content of articles, including links to glossaries and encyclopedias, tutorials about the publishing process, and study of the structure of articles. Phase 2, reported here, observed undergraduate students, graduate students, faculty, and instructional science librarians as they searched for journal articles for a simulated class assignment in a major online journals system. Phase 3 provides an overview of library work on instruction modules and conclusions on benchmarks for successful web-based learning modules to help undergraduate science students learn about scholarly journals (http://web.utk.edu/~tenopir/nsf/nsf_phase3_final.pdf.) - 2 -
1.1 Objectives Research questions for phase 2 include: 1. What do undergraduate and graduate science students understand about the structure, purpose, and content of scholarly journal articles? 2. How do undergraduate and graduate science students search for journal articles to solve a specific class-related task? 3. What features of online systems and web search engines do undergraduate and graduate science students understand, use, and value? 4. How do science faculty and science librarians search for and use science journal articles for a simulated classroom assignment and how do their search patterns and understanding differ from those of students? The answers to these questions will help us design more useful electronic journal systems, better instructional materials and coursework involving scholarly journals, and lead to more understanding and use of scholarly electronic journals by undergraduate students. 1.2 Literature Review There is abundant evidence that scholarly journals are not only widely read by working scientists, but they are extremely useful and important to scientists' work, whether that work be teaching, research, administration, or other activities (Tenopir and King 2000). Studies show that many faculty and most students prefer electronic journals - 3 -
over print journals, and the convenience of linked desktop access likely results in a greater amount of reading of journal articles (Tenopir 2003b; Tenopir et al. 2003). Phase 1 of this study confirmed that undergraduate students turn to electronic sources first, in particular the Web, for their coursework (Jones 2002). Their understanding and use of scholarly journals depends on the requirements imposed by their class work and instructions from professors (Tenopir et al. 2003). Recent studies have shown that undergraduate students often use the sources that are most convenient to them, rather than carefully selecting the highest quality materials. Easy availability of full-texts of articles is the one overriding factor that undergraduate students take into account when selecting a digital resource for research even if another source may provide indexing and abstracting data for higher quality literature (Tenopir 1999). Observational testing of students and faculty attempting to solve a simulated classrelated assignment helps to provide a deeper understanding of how journal articles fit into the undergraduate science curriculum and interaction with features of e-journals systems. Observational studies of online searching behavior have been well documented. Early studies of information retrieval often focused on systems and technologies, but many studies now take user-oriented approaches to investigate this complex activity (Bates 1996; Bates, Wilde, and Siegfried 1993a; Bates, Wilde, and Siegfried 1993b; Belkin, Oddy, and Brooks 1982; Borgman, Hirsh, and Hiller 1996; Dervin 1992; Ellis 1992; Fidel 1987; Harter 1992; Ingwersen 1996; Kuhlthau 1993; Marchionini 1989; Wang 1999). - 4 -
Observational studies may be case studies or experimental. Callison (1997) cites an increase in the use of the case study method to investigate the process used by undergraduate students to locate information. He contends that the case study method has become established as the primary research technique used to document student thought processes in topic focus and source selection. Direct observation is a primary tool common to the case study method. The think-aloud user protocol is a useful tool for experimental observational studies. The purpose of the protocol is to gain insight into the behavior and experience of subjects performing online searching or using any particular tool or product. Subjects are told what tasks to accomplish but not how to accomplish them. Discovering whether and how participants accomplish the assigned tasks and gathering data about their experience during the experiment is the goal (Covey 2002). Despite being criticized as soft data, concurrent-verbalization is the only method to obtain subjects thoughts while they perform specific tasks (Wang, Hawk, and Tenopir 2000). Nahl and Tenopir (1996) used both the think-aloud user protocol and online monitoring in their study of online searching. Their results demonstrated the importance of affective and sensorimotor information needs as complements to the cognitive information needs of users involved in online searching. In this study, subjects verbalizations were recorded on audio tape. Subjects employed the think-aloud protocol as they searched by indicating their search topics, their purposes, their motivations, what uses they were making of the database, whether they were satisfied with the retrievals and the results of each search, and other reactions. Session audio tapes were then transcribed. Online monitoring provided an automatic transaction log of commands. This - 5 -
dual data recording method is often used for unobtrusive observation of online searching (Oldroyd and Citroen 1977; Penniman and Dominick 1980; Rice and Borgman 1983). Ericsson and Simon (1993) address the validity of verbal reports in this kind of study. They maintain that recent research based on explicit information processing models of the cognitive process has caused think-aloud verbalizations to be viewed in a new light. Making careful verbatim transcripts of the recorded tapes preserves raw data in a hard data form. The process begins with tape recording, containing essentially all the auditory events that occurred during the experimental session. The authors refer to the transcription step as preprocessing. At the next step, preprocessed segments are encoded into the terminology of the theoretical model, usually by human judges. This kind of concurrent verbal report a think-aloud report is a close reflection of the cognitive process. The concurrent report reveals the sequence of information heeded by the subject without altering the cognitive process. On-line monitoring, the other half of this observational technique, is a highly useful technique for studying, evaluating, and improving systems and user/system interfaces. Penniman and Dominick (1980) define monitoring as the process of collecting data associated with the functioning and/or usage of a system. Evaluation is defined as the process of analyzing the functioning and/or usage of a system so that decisions can be made concerning the effectiveness of the system in satisfying its design objectives. Objectives for monitoring can be multiple and include comparison of systems and/or data base structures, efficiency evaluation of system/database interface, analysis of usage of the system and/or data base, and analysis of user success/satisfaction. - 6 -
Studies involving detailed observations of users engaged in searching entail intense interaction and observation, reducing the number of subjects which can reasonably be observed (Shaw 1996). However, Shaw proposes that the depth of information available in such observational studies provides a rich sense of the nature of searching and the context and evolution of information needs. Although the small number of subjects often prevents the use of statistical measures of significance, the data gleaned reveal important findings through the repeated observation and statements of the subjects. 1.3 Research Design This study was conducted in a laboratory setting. The participants were given simulated search tasks based on their academic status. Their search processes and concurrent verbalizations were recorded as audio and video data. Participants The participants of this project consisted of undergraduate science students, undergraduate speech students who had taken at least one science course, graduate students, faculty in the fields of chemistry, physics and engineering, and instructional science librarians at the University of Tennessee, Knoxville. Participants in the Phase 1 focus groups of this project were personally invited to participate in Phase 2. In addition, students were recruited through flyers passed out and posted in the chemistry, physics and engineering buildings in the campus. Emails and personal telephone contact were used to recruit hard-to-reach respondents. All participants were offered monetary - 7 -
compensation for their participation. In total, 9 faculty members, 10 graduate students, 10 undergraduate students, 7 Speech 210 undergraduate students, and 5 librarians participated in this project. Among the graduate students, five were PhD students and five were Masters students. Test-bed System: ScienceDirect ScienceDirect, an Elsevier e-journals system, was launched in 1997. (http://www.info.sciencedirect.com/licensing_options/index.shtml). It has evolved from a web database of Elsevier Science journals to one of the world's largest providers of scientific, technical and medical (STM) literature. ScienceDirect online is a complex information retrieval (IR) system with many search functions and features. As of September 2004, it covers over 1800 journals from Elsevier, with over 6 million articles and over 60 million abstracts from all fields of science. Articles are available online before appearing in print. Elsevier provided research access to the system for the duration of this project. Test-bed System: Energy Citations Database The Office of Scientific and Technical Information (OSTI) created a proprietary full text test-bed comprised of content from all across the Department of Energy complex. The test-bed included approximately 28,000 citations, 15,000 conferences and technical reports with links to full text, 8,600 American Physical Society (APS) journal citations with links to their abstracts, and 5,000 APS journal citations with links to table of contents pages. The subject categories included inorganic, organic, and analytical - 8 -
chemistry; radiation chemistry, radiochemistry, and nuclear chemistry; physics of elementary particles and fields; nuclear physics and radiation physics; atomic and molecular physics; condensed matter physics, superconductivity and superfluidity; materials science; geosciences; fossil-fueled power plants; specific nuclear reactors and associated plants; general studies of nuclear reactors; power transmission and distribution; energy storage; direct energy conversion; advanced propulsion systems; engineering; particle accelerators; isotopes and radiation sources; energy conservation, consumption, and utilization. Lab settings The College of Communication and Information Usability Lab consists of a test area and an observation area. These two areas are separated by a freestanding partition budgetary constraints having precluded installation of a one-way mirror found in many laboratories of this type. On entering the lab, participants are directed to the welcome section of the test area where they are given an introduction to the study and asked to complete pretest paperwork, including an Informed Consent statement (see Volume II, Section 3.1.4.) Participants then proceed to the workstation to perform the tasks described in the task scenario (Volume II, Section 1.) A lab assistant stationed in the observation area conducts the test session and records the participant s interaction with the system as he or she works through the tasks. On completion of the assigned tasks the participant returns to the welcome area to complete post-test paperwork and receive compensation for their time. - 9 -
The participant s workstation consists of a Microsoft Windows -based personal computer and a laser printer, both of which are connected to the campus network. The test area also contains two small Pan/Tilt/Zoom video cameras. A floor camera, which is used to capture the participant s facial expressions, is mounted on a tripod outside the participant s main field of view. A ceiling-mounted camera is positioned behind the participant and is used to capture images of documents on the workstation table. Both cameras are controlled from the observation area using a remote control device. Participants wear a wireless microphone so that their verbalizations can be captured as part of the think-aloud protocol used in the study. A screenshot of an Observer software session is included in Volume II, Section 5. The observation area houses audio/video equipment used to capture and record the test sessions. A scan converter captures the participant s workstation screen display as an S-video signal. This signal is mixed with video from the floor camera to produce a picture-in-picture display of the participant s screen with their face in a small foreground window. The mixer also combines audio from the participant s microphone with the picture-in-picture video. Digital recordings of the test sessions are made using a dual format S-VHS/MiniDV tape recorder. While digital tape recording produces high-quality results, use of the MiniDV format limits standard record time to eighty minutes per cartridge. However, our tape recorder also has a non-standard LP recording mode which doubles the recording time of the MiniDV cartridges should this be necessary. Tape recordings of the test sessions are used to provide redundancy for the primary recordings which are made to digital media files. Digital media files provide greater - 10 -
control during data collection than is possible using tape. For example, media files can be positioned more accurately than tape and can be repeatedly repositioned without degrading the recording. A Microsoft Windows -based workstation with a real-time MPEG encoder card is used to record the test sessions to media files. MPEG-2 encoding was chosen for high quality results and compatibility with the Observer Video-Pro data-collection software from Noldus Information Technology. Procedure for ScienceDirect All the participants used the same computer set-up which includes a Pentium personal computer, a standard keyboard, a three-key mouse and a wireless microphone. Microsoft Internet Explorer 6.0 was used as the web browser. A VCR was used to record the participants searching process and vocalizations in the ScienceDirect system. In order to minimize possible stress, a research assistant was trained as an observer to collect data, prompt participants to think aloud, and to be available to assist with questions about procedure. Before each session started, the observer introduced the project and its purpose briefly to the participant. The lab settings and sequence of tasks were also introduced. Each participant was informed that he/she would be videotaped and an Informed Consent Statement was signed before the testing session started. After signing the consent statement, participants were asked to finish Kolb Learning Style Inventory questionnaires (Kolb 1984; Smith and Kolb 1986). - 11 -
Participants were then led to the work station to finish two known-item searches on the Web under the observation of the research assistant (observer). This was to make sure that the participant had basic on-line surfing skills and also to help them warm up. After the two preliminary searches were finished, a task scenario was provided to the participant. The various task scenarios for the different groups of participants are available in Volume II, Section 1. (Librarians used the Faculty task scenario.) Each task scenario consists of four tasks. The first task was topic selection and description. A list of topic suggestions was provided to both undergraduate students and graduate students. Faculty participants were asked to think of a topic that was suitable for students to use in a class assignment. In order to make the test searching situation as realistic as possible, all participants were permitted to choose any topic of interest, either from one of their own projects or from the lists provided (if the participant was a graduate or undergraduate student.) The second task was to search for relevant literature and provide a description of the topic chosen by the participant. The default page in the Web browser was set for ScienceDirect s homepage. There was no restriction or control on how the participants interacted with the system. They could stop one task and move to the next task once they were satisfied with what they found. The participants were required to think aloud while interacting with the search system. This technique helps researchers explore participants cognitive and affective reactions beyond their searching behavior. The third task required participants to describe the topic of their research paper, project, or assignment, and describe again what they would study in the project or paper. - 12 -
They were also asked to write down the title of a projected paper if they already had one in mind. The fourth task asked for the participants opinions of the test-bed system, ScienceDirect. The task scenario provided participants with five open-ended questions to answer about their use and opinion of ScienceDirect. At the end of each task, the participants were required to tell the observer they were moving to the next task so that the observer could monitor their progress. An Observer Problem List (see Volume II, Section 3.1.14) was prepared for the observer to record any problems with the procedures, the equipment or the test session. The whole searching process and the participants verbalizations were recorded by VCR. Each participant was given a random identity number to assure anonymity. All the materials about each participant are filed together by these identification numbers. After each session, a checklist was used to make sure each participant s package file was complete (see Volume II, Section 3.1.15.) While observing the searching process, the observer was responsible for directing participants to think aloud whenever they were silent for a while. Observers also resolved unpredictable problems met by the participants and made sure the equipment was working normally. Procedure for Energy Citations Database (ECD) Some graduate students searched ECD instead of or in addition to ScienceDirect. The procedures were the same, but the Web browser in this case was set to ECD and, since only graduate students used ECD, only the graduate student task scenario was used. - 13 -
Data A variety of information was collected for each participant, including academic standing (academic level and major field of study), learning style (as measured by the Kolb Learning Style Inventory), transaction process data of how the user interacted with the ScienceDirect system (audio/videotaped searching process), the participant s own description of the topics before and after searching, verbalization (think-aloud data), query data, and a post-search questionnaire to measure the participant s impression of the ScienceDirect system. Academic Standing: Academic standing data includes participants academic level and major field of study. The participants academic levels were determined during the initial contact. Participants indicate their major fields on the pre-search questionnaire before starting the tasks in the task scenario. Learning Style: Learning style is the way in which individuals learn and how they deal with ideas and day-to-day situations. We all learn in different ways. Knowing a participants learning styles can help researchers better understand how the participants solve problems, how they manage conflicts, and how they negotiate with IR systems. Dimensions of learning style can be measured by standard tests; in this study we chose the Kolb Learning Style Inventory (version 3) to measure participants learning style. For more information, see David Kolb s Experiential Learning (Kolb 1984). - 14 -
Four types of learning modes are defined in the Kolb Learning Style Inventory: 1) Concrete Experience (CE), 2) Reflective Observation (RO), 3) Abstract Conceptualization (AC), and 4) Active Experimentation (AE). These learning modes make up a four-phase cycle of learning. The Kolb Learning Style Inventory test consists of 12 rating questions. Each question has 4 options; each of which corresponds to a type of learning mode. The participants rank options from 1 through 4 according to their learning mode preferences with a rank of 1 being the least preferred mode and a rank of 4 indicating the most preferred mode. No tie is permitted. The sum score of each row should equal 10, and the sum score of the whole test should equal 120. The final scores for AC-CE and AE-RO indicate how much participants rely on each of the four different learning modes. Based upon their scores, the participants learning styles can be determined. Individuals with both Concrete Experience (CE) and Reflective Observation (RO) learning preferences tend to have a Diverging style. They prefer to consider a situation from different perspectives and tend to diverge from conventional solutions, coming up with alternative possibilities. Individuals tending to approach the learning process by focusing on Abstract Conceptualization (AC) and Active Experimentation (AE) probably prefer the Converging style. They may enjoy gathering information to solving problems and tend to converge on the correct solution. Individuals using approaches that include Reflective Observation (RO) and Abstract Conceptualization (AC) prefer the Assimilating style. They are interested in absorbing the learning experience into a larger framework of ideas and tend to assimilate information into theories or models. Individuals whose primary learning modes involve Active Experimentation (AE) and Concrete Experience (CE) have the Accommodating - 15 -
style. They want to put their ideas into action, finding still more uses for whatever has been learned. They tend to accommodate, or adapt to, changing circumstances and information. Not everyone falls into one of the four dominant styles. There are some people who can achieve balance along two or more dimensions of the learning process. A balancing style may indicate a person who is comfortable with a variety of learning modes. Videotaped sessions: Each participant s searching process, key strokes, and continuous screen shots with synchronized verbalization were videotaped. Videotaping of information-seeking transactions can capture the process to be replayed during data analysis. Compared to other recorders, the VCR provides higher storage space and reduces the risk of losing data. After a session was finished, the content of the VCR tape was stored on DVD for data analysis. Cognitive and Affective Reactions: In information seeking and human system interaction research, it is important to know the underlying cognitive and affective process related to the observable behavior. The think-aloud technique provides a valid and reliable way to obtain subjects thoughts while they perform specific tasks (Wang 1999). The Observer software generates a number for each movement between pages made in each session. We added this number to each paragraph in the verbalization. (Each paragraph is an exact transcription of what the participant said during the time period the record represents.) - 16 -
The bottom-up method was used to analyze the verbalized thinking-aloud data. Bottom-up analysis, like grounded theory, begins with the data (verbalizations and movements) from which the coding schema is developed. Twenty-nine nodes were created to depict the participants affective status. Each node stands for one kind of emotional reaction. The cognitive reactions were coded in the same way as the affective ones except that the cognitive schema was predefined by the researchers. Coding schemes are listed in Volume II, Section 4. Queries: Queries were analyzed in the context of verbalized think-aloud data. The number of search terms, the use of query operators, query modification, use of search features and query errors are studied to gain insight into patterns of user queries and the characteristics of system users. 2. Data Analysis Both quantitative and qualitative data were collected and various ways are employed to analyze the data. The data analysis could be roughly divided into three parts. Part I Data analysis in Observer The Observer Video-Pro software from Noldus Information Technology was used to mark the videotaped searching process. In Observer, video files were transcribed to text files. The whole searching process was divided into slices based on participants - 17 -
movements. When participants moved from one page state to another, one marker was inserted at the point of changing and a new record was produced. The text files start at the point the searching starts. Time stamps, which are the corresponding times in the videotape, were generated automatically by the Observer as the record identifier. In addition to the time stamp, each record has two other fields -- page state and comment. Page state is discussed in the following paragraph. The number of results, participants search actions, navigational behavior, document usage, error descriptions and other related information were recorded in the comment field. A screen shot of a data analysis session in Observer is in Volume II, Section 5. Ten codes were created to code the ten possible page states of the ScienceDirect system. They are Home Tab, Search Tab, Journals Tab, Abstract Tab, Reference Tab, Other Tabs, Search Results, Document, Error and Null, among which the Other Tabs is used to code two page states: My Alert and My Profile. The decision to include two page states under Other Tabs was based on the consideration that these two pages are not often used by common users. Furthermore, codes were developed to depict actions on each page state. These codes fall into three categories: search, navigation, and document. The search codes specifically refer to the search actions, such as quick search (QS), search all source (All), search for a journal title (SJT), search within (Srw), etc. The navigation codes are used to mark the navigational movement within the system, such as back (B), forward (F), next page (NP), results list (RL), etc. The third category is document, which describes the document components, such as tables (Dta), figures (Dfi), introductions (Din), method (Dme) of a document and the actions conducted on the document. The actions are further - 18 -
categorized into three groups. One is the use of document, such as Email article (EA), or export citation (EC). One is leaving the document, such as Cited by (CB) or Reference Link (Drl). The third is document format, such as PDF (pdf), Fulltext + Links (FT), Summary Plus (Spl), etc. The coding system was developed in the process of marking the videotapes. Once a function was used by a participant, a corresponding code was be added to the coding list. Therefore, the codes are quite practical and the coding system is also very flexible. It is an open system; new codes can be added without any difficulties or conflicts. Keeping the application of the codes consistent is the most important issue in the transcribing process. In order to achieve this purpose, meaning and descriptions are attached to the corresponding code and the codes were stored in the database to verify their uniqueness. Part II Data analysis using Microsoft Access The structured text files generated in Observer were imported into Microsoft Access. A relational database was constructed to integrate and search the data. Figure 1 shows the composition of the database. - 19 -
Figure 1. Relational Database Structure for Observer System Data The Subject table has two fields. The field called Subject is each participant s ID. The Status field is the academic standing (faculty, graduate student, undergraduate student, speech 210 student, or librarian) of each participant. The table Allrecord records the participants movements from one page to another. The field ID (the combination of participant ID and the sequence number generated automatically by Observer ), is the record identifier. It is. It was set to be the primary key of the table; the fields Time and Pagestate store the values of time stamp and page state respectively from Observer. The Session field is the sequence number of - 20 -
the record in Observer. The table Allquery has four fields: 1) ID ; 2) Subject ; 3) Search - the type of search of the corresponding query; and 4) Query - the copy of queries submitted by the participants. The ID and Subject fields are the same as those in the Allrecord table. The ID and Subject fields in the table Allcomment are the same as they are in Allrecord and Allquery ; the Comment field records the actions of the participants on each page status. Individual participant s movements between pages, query, and actions on each page could be grouped together through subject number. Through the ID number, each page status, participant s actions on that page status, and the queries submitted through that page were connected to each other. SQL statements can easily integrate the data in these separate four tables or pull out the data from one or more specific tables. Part III Verbal transcription and data analysis in QSR N6 software Verbal Transcription Participant verbalization provides valid and reliable data with which to study the underlying cognitive and affective processes related to observable behavior. In the sessions, participants were required to think aloud while interacting with the system. A student assistant transcribed the think-aloud data into text files. The verbalization was sequenced by the record number generated by Observer. Thus, what the participant said can be linked to the time he or she was on a particular page status and performed certain actions. If the participant said nothing, a blank record was generated. Each record of verbalization is one paragraph. - 21 -
Coding schemas Affective coding schema A list of 29 terms indicating emotional status was generated after examining the verbalization data. To facilitate analysis, affective words were grouped into three categories: positive, negative and neutral. The affective coding schema is attached in Volume II, Section 4.3. Cognitive coding schema Based on what initiated the cognitive reactions, cognitive events are grouped into four general categories: 1) System, 2) Search Strategy, 3) Search Results, and 4) Task. The cognitive coding schema is attached in Volume II, Section 4.2. Coding in QSR N6 software QSR N6 Release V 6.0 was used to create, organize and manage the affective and cognitive nodes. QSR gives researchers the ability to mark the verbalization text, add new points to existing nodes, create, delete, edit, merge or divide nodes and to generate reports including document text, coding summary and coding stripes. Before the coding process starts, the text files produced in the process of verbal transcription were imported into the software. The coding unit in the QSR software is set as paragraph. Because each paragraph of the verbalization corresponds to each record in Observer, the coding results of affective and cognitive reactions can be easily connected to the participants behavioral data. After the verbalization text files were - 22 -
imported into QSR, a unit was added to each file to indicate each participant s academic standing and academic major. Meanwhile, the queries were appended to the end of the corresponding text units so that queries could be analyzed in the context of verbalized think-aloud data. Two cross-coders worked to code the affective reactions that were recorded during the search. One coder examined the verbalization, developed the affective coding schema and coded the affective reactions. Then, the other coder coded verbal transcriptions of ten participants with the developed coding schema. The codes assigned by the two coders were used to calculate the coding reliability of the affective coding. In the cognitive coding process, the coding schema was predetermined by the researchers. Two coders assigned cognitive codes to the transcripts based on the predefined coding schema. Five verbal-rich files were assigned to both coders. The cross coding result of the five files were used to calculate the reliability of the cognitive coding. Both affective and cognitive coding reliability were calculated by the formula: A= M/( N i /n). M is the number of coding events agreed on by two coders; N i is the number of coding events assigned by the ith coder; and n is the number of coders. - 23 -
3. Preliminary Results Forty-one participants completed sessions on ScienceDirect. Table 1 shows participants prior experience with the test-bed. Eight graduate students, six faculty members, and all five of the librarians had used ScienceDirect before. None of the undergraduates had used ScienceDirect prior to this study. Altogether, 22 participants were first-time users. Table 1. Experience with ScienceDirect Academic Used ScienceDirect before? # Status Never used Used before Faculty 9 3 (33.3%) 6 (66.7%) Graduate 10 2 (20%) 8 (80%) Undergraduate 10 10 (100%) Speech210 7 7 (100%) Librarian 5 5 (100%) Total 41 22 (53.7%) 19 (46.3%) Table 2 shows participants academic status and major fields. Table 3 displays participants learning styles as revealed by the Kolb Learning Style Inventory. Table 2. Participant major fields Major Academic # Astronomy Status Chemistry Engineering Librarian Other & Physics Faculty 9 1 1 4 Graduate 10 2 8 Undergraduate 10 2 5 3 Speech210 7 2 2 2 1 Librarian 5 5 Total 41 7 11 17 6 1-24 -
Table 3. Participant learning styles Academic Learning Style # Status Accommodator Assimilator Converger Diverger Faculty 9 1 1 6 1 Graduate 10 6 3 1 Undergraduate 10 2 4 2 2 Speech210 7 2 2 3 Librarian 5 1 4 Total 41 6 17 14 4 Search topics. The topics searched by faculty were related to the topics they used for undergraduate classroom teaching. For example, faculty chose to do searches on: Special relativity and quantum mechanics Mass spectrometry forensics Vapor-liquid equilibrium data for acetone-ethanol system Depleted urania-active electronic devices Supply chain and scheduling At the end of the searches, one faculty member commented: ScienceDirect seems to point at a level of reference that is somewhat above the level of sophomore and senior undergraduate courses. The topics searched by PhD and Masters students were mostly related to their research projects. For example, graduate students searched on: Magnetic nanostructures Major techniques for synthesis of metal oxide Air pollution in Knoxville area Data mining and scheduling Lean manufacturing as it applies to non-manufacturing applications - 25 -
The topics searched by undergraduate students were selected from the list provided to them in the search scenario (see Volume II, Sections 1.4 and 1.5.) These included: Air pollution due to industrial and automobile waste Stellar wind Black holes Radiation and food safety Water treatment Session length. As Table 4 shows, session length varied, with undergraduate speech students spending the most total time and spending the most time per page viewed. Undergraduate science students spent the least time per page visited. Graduate students visited the largest number of pages on average. Faculty members spent the least total time and visited the fewest pages on average. Table 4. Session length Academic Status # Faculty 9 Graduate 10 Undergraduate 10 Speech210 7 Librarian 5 Total 41 Session Length (Minute) 17.03 (10.92) 28.52 (17.44) 20.44 (8.19) 37.34 (14.73) 21.97 (7.41) 24.73 (13.99) Number of Visited Pages Mean (SD) 26.89 (19.35) 51.90 (27.83) 39.60 (16.87) 44.43 (25.26) 31.80 (13.07) 39.68 (22.62) Time/Page (Minute) 0.69 (0.22) 0.58 (0.22) 0.53 (0.12) 0.92 (0.21) 0.77 (0.36) 0.67 (0.25) - 26 -
Pausing behavior. The tempo of the actions can easily identify the pausing behavior of the searchers. Figure 2 shows five examples of pausing behavior. The faculty participant (top) illustrated here paused at the ScienceDirect Reference Tab. The graduate student had two big pauses; both at the time he/she was browsing search results. For the undergraduate, the biggest pause occurred when he/she was browsing a selected document. The three pauses of the speech 210 participant all occurred when he/she was browsing a selected document. The pauses made by the librarian are not as long as the example participants in the other four groups and the pauses are evenly distributed at search tab, selected document page and search result page. - 27 -
Use of Electronic Science Journals in the Undergraduate Curriculum: Figure 2: Sample participant pausing behavior - 28 -
Definition of pausing: Stopping on a page is considered a pause when the length of time spent on the page is twice longer than the average time spent on each page for that session. Table 5 shows the average length of pauses for each group in the study. Undergraduate speech students averaged the longest pauses while graduate students had the shortest pauses. Further analysis of the verbal protocols may help reveal the differences in search pause patterns. Table 5. Average length of pauses by group Academic status Average length of pause (seconds) Faculty 126.55 Graduate 109.67 Undergraduate 134.23 Speech 210 197.79 Librarian 146.18 Interactions. During the process, participants interacted with the system in four ways: 1) choosing, 2) conducting search, 3) accessing documents, and 4) help. Their frequency of use of these four types of interactions is listed in Table 6. As the data in Table 6 illustrate, many actions were taken during the interaction process. - 29 -
Table 6. Types of interactions Academic status # Choosing system Faculty 9 22 (15.17) Graduate 10 76 (25.25) Under- 69 10 Graduate (26.54) Speech 210 7 56 (27.45) Librarian 5 21 (19.81) Total 41 244 (24.02) Note: Percentage is based on row total Conducting search Accessing document Total use (%) 57 64 (39.31) (44.14) 112 112 (37.21) (37.21) 70 121 (26.92) (46.54) 51 93 (25.00) (45.59) 25 56 (23.58) (52.83) 315 446 (31.00) (43.90) Help 2 (1.38) 1 (0.33) 4 (1.96) 4 (3.77) 11 (1.08) Choosing a system state. At the time of our study, the system offered six states (subsystems) with different functions selectable from the top tabs. (After the sessions were complete, a seventh state was added.) A user must be in a specified state in order to interact with the system. For example, the default state Home will allow the user to conduct QuickSearch, select subject areas, and access individual journals directly. Other states from which a user could choose included Search, Journal, Abstract and Reference, My Profile, and Alerts. Overall, the most used state was Search (Figure 3), but faculty participants used Home most. - 30 -
Abstract tab Home tab Search tab Reference tab Other tabs Journals tab Figure 3: System states and participants Conducting searches. The system provides several search features. The following features were used by our participants: QuickSearch, BasicSearch, AdvancedSearch and SearchWithinResults. The default, QuickSearch, on top of all pages under the navigation tabs, is the simplest with one query box for entering queries. The BasicSearch allows a Boolean search with only one operator linking two terms (the operator can be AND, OR, or AND NOT.) The terms can be limited to any of 10 fields (such as Author, Journal, Title, etc.) Searches can be limited to specific document type (journals, abstract databases, etc.), subject areas, and publication years. The AdvancedSearch, intended for expert searchers, looks similar to BasicSearch but allows combinations of multiple terms and multiple Boolean operators. SearchWithinResults allows the user to modify a query or a new query can be executed within the previous result set. The use of these features by the participants is summarized in Table 7. - 31 -
Table 7. Use of search features Academic status Faculty 9 # Graduate 10 Undergraduate Speech 210 7 Librarian 5 10 Total 41 Quick Search 45 (78.95) 35 (31.25) 35 (50.00) 23 (45.10) 1 (4.00) 139 (44.13) Basic Search 8 (14.04) 42 (37.50) 22 (31.43) 22 (43.14) 22 (88.00) 115 (36.83) Total Use (%) Advanced Search 8 (7.14) 2 (2.86) 10 (3.17) Search Within Results 4 (7.02) 27 (24.11) 3 (4.29) 2 (3.92) 2 (8.00) 38 (12.06) Search Journal Title 8 (11.42) 4 (7.84) 12 (5.58) Faculty mostly used QuickSearch and did not use AdvancedSearch at all. Graduate students used BasicSearch more than QuickSearch, and used Search Within Results most. Undergraduates used mostly QuickSearch and BasicSearch. Speech 210 students used QuickSearch and BasicSearch with near equal frequency and used other modes very infrequently. Librarians used BasicSearch most and other modes very infrequently. Accessing documents. Most participants did not elaborate on their search strategies or search results before evaluating and viewing documents. The number of search results or hits ranged between 0 and 8015. There were 15 times that queries brought back more than 10,000 documents and the searches were interrupted by the system automatically. - 32 -
The 41 participants accessed a total of 434 documents. The first and second documents on the search results are most likely to be selected -- the first document was selected a total of 52 times (11.98%), and the second document was selected 50 times (11.52%). The third document on the results list was selected 32 times (7.37%), and the fourth document was selected 32 times (7.37%). The highest position on the results list for a viewed document was 204th. There were 14 instances when a document with a position beyond one hundred on the results list was accessed (3.23% of total instances.) Using help. Only three participants accessed Help. One faculty member used Search Tips when he/she received an error message after submitting a search using QuickSearch, but then continued to do searches in QuickSearch throughout the rest of the search. This participant commented after the session that help is not easy to read. Another searcher tried to open Help unsuccessfully. 4. Conclusions and Further Analysis Preliminary results from this observational study show there are differences in how undergraduate science students, undergraduate speech students, graduate students, librarians, and science faculty interact with an electronic journals system for class related tasks. On average, undergraduate speech students spent the most time per session and spent the most time on each page. Graduate students viewed the greatest number of - 33 -
pages on average. Undergraduate speech students averaged the longest pauses while graduate students had the shortest pauses. The tempo of session functions will be analyzed in more depth. All groups used QuickSearch or BasicSearch modes most of the time, but graduate students also used SearchWithinResults quite a bit. Searching and accessing documents were common types of system interactions; using Help was unusual (and even when Help was used, it was not found to be useful.) Perhaps the resistance to using Help or incidents of unhelpful Help messages (as stated in verbalization transcripts) explains in part why the majority of affective reactions were negative. These results are only preliminary. In-depth analysis of all types of data collection continues and will be reported in journal articles. Currently we are integrating the behavior data with the verbalized thoughts to help shed light on motivations and reasons for specific behaviors. - 34 -
5. Bibliography Atkinson, J. D., III, & Figueroa, M. (1997). Information seeking behavior of business students: A research study. Reference Librarian(58), 59-73. Bates, M. J. (1996). The Getty End-User Online Searching Project in the humanities: Report no. 6: Overview and conclusions. College & Research Libraries, 57(6), 514-523. Bates, M. J., Wilde, D. N., & Siegfried, S. (1993a). An analysis of search terminology used by humanities scholars: The Getty Online Searching Project report number 1. Library Quarterly, 63(1), 1-39. Bates, M. J., Wilde, D. N., & Siegfried, S. (1993b). A profile of end-user searching behavior by humanities scholars the Getty Online Searching Project Report No.2. Journal of the American Society for Information Science, 44(5), 273-291. Belkin, N. J., Oddy, R. N., & Brooks, H. M. (1982). ASK For Information Retrieval: Part I. Background and Theory. Journal of Documentation, 38(2), 61-71. Bell, S. J. (1999). Understanding preferences for search system interfaces and the role of emotive forces: A research challenge. College & Research Libraries, 60(5), 440-452. Bhavnani, S. K., & Bates, M. J. (2002). Separating the knowledge layers: Cognitive analysis of search knowledge through hierarchical goal decompositions. In Asist 2002: Proceedings of the 65th Asist Annual Meeting, Vol 39, 2002 (Vol. 39, pp. 204-213). Medford: Information Today, Inc. Borgman, C. L., Hirsh, S. G., & Hiller, J. (1996). Rethinking online monitoring methods for information retrieval systems: from search product to search process. Journal of the American Society for Information Science, 47(7), 568-583. Callison, D. (1997). Evolution of methods to measure student information use. Library & Information Science Research, 19(4), 347-357. Case, D. O. (2002). Looking for information : a survey of research on information seeking, needs, and behavior. Amsterdam ; New York: Academic Press c2002. Covey, D. T. (2002). Usage and usability assessment: Library practices and concerns: Council on Library and Information Resources. Dervin, B. (1992). From the mind's eye of the user: The sense-making qualitativequantitative methodology. In J. D. Glazier & R. R. Powell (Eds.), Qualitative research in information management (pp. 61-84). Englewood, CO: Libraries Unlimited. - 35 -