Content analysis (qualitative, thematic) (Last updated: 9/4/06, Yan Zhang)

Transcription

1 Content analysis (qualitative, thematic) (Last updated: 9/4/06, Yan Zhang) Introduction As one of today s most extensively employed analytical tools, content analysis has been used fruitfully in a wide variety of research applications in ILS field (Allen and Reser, 1990). Similar to other fields, content analysis has been primarily used in ILS as a quantitative research method until recent decades. (Please refer to the previous chapter for quantitative content analysis.) Essentially reducing text into numbers, the quantitative techniques often are criticized for missing syntactical and semantic information embedded in the text (Weber, 1990). Qualitative content analysis takes effect at the place where quantitative presentation reaches its limits. Qualitative content analysis has been defined as: a research method for the subjective interpretation of the content of text data through the systematic classification process of coding and identifying themes or patterns (Hsieh and Shannon, 2005:1278). an approach of empirical, methodological controlled analysis of texts within their context of communication, following content analytic rules and step by step models, without rash quantification (Mayring, 2000:5). The definitions illustrate that qualitative content analysis emphasizes an integrated view of data and their specific contexts. It allows researchers to interpret social reality in a subjective but scientific manner. The validity of the inference is ensured by complying with a systematic coding process. Qualitative content analysis goes beyond merely counting words or extracting objective content from texts to examine themes and pattern that appear or are latent in the manifest content. Comparing qualitative content analysis with its rather familiar quantitative counterpart would help enhance our understanding of the method. First of all, the research areas from which they branched out are different. Quantitative content analysis was used widely in mass communication as a way to count manifest textual elements, while qualitative content analysis was developed primarily in anthropology and psychology to explore the meanings underlying physical messages. Secondly, quantitative content analysis is deductive, trying to test hypothesis or questions generated from theories or previous researches, while qualitative content analysis is mainly inductive, grounded the examination and inference of topics and themes to raw data. In some cases, qualitative content analysis tempts to generate theory. Thirdly, the data sampling technique required by the two approaches is different. Quantitative content analysis requires that the data is selected using random sampling rules so as to ensure the validity of statistic inference, while samples for qualitative content analysis usually consist of purposively selected cases which reflect the research questions being investigated. Last but not the least, the product of the two approaches is different. Quantitative approach produces numbers that can be manipulated by various statistic methods. Qualitative approach produces descriptions or typologies, most of the time, along with expressions from subjects reflecting how they view the social world. By this means, the perspectives of the producers of the text could be better understood by the investigator as well as the readers (Berg, 2001). Qualitative content analysis pays 1

2 attention to unique themes that illustrate the range of the meanings of the phenomenon rather than statistic significance. In real research work, the two approaches are not mutually exclusive. Rather they are encouraged to be used in combination. As suggested by Smith that qualitative analysis deals with the forms and antecedent-consequent patterns of form, while quantitative analysis deals with duration and frequency of form (Smith, 1975: 286). Weber (1990) also pointed out that the best content-analytic studies use both qualitative and quantitative operations. Inductive vs. deductive Qualitative content analysis involves a process to condense raw data into categories based on valid inference and interpretation. This process by large uses inductive reasoning, by which themes and categories emerged from raw data under researchers careful examination and constant comparison. But qualitative content analysis is not insulated from deductive reasoning. Generating concepts or variables from theory or previous studies is also very useful for qualitative research, especially at the inception of data analysis (Berg, 2001). Hsieh and Shannon (2005) discussed three approaches to qualitative content based on the degree of involvement of inductive reasoning: (1) conventional content analysis, where coding categories are derived directly and inductively from the raw data. It is a grounded theory approach. Its purpose mainly is to develop theories; (2) directed content analysis, where initial coding starts with a theory or relevant research finings. In data analysis process, researchers immerse themselves into the data and allow the themes to emerge from the data. The purpose of this approaches usually is to validate or extend a conceptual framework or theory; (3) summative content analysis, which starts with words or content counting, then extends the exploration to include latent meanings and themes. This approach seems quantitative at early stage, but it intends to explore the usage of the words in an inductive manner. The process of qualitative content analysis As a data analysis method, qualitative content analysis often begins at early stage of qualitative data collection. The early involvement helps researchers move back-andforth between concept development and data collection, which directs the new data collection toward a way more meaningful for answering research questions (Miles and Huberman, 1994). To achieve valid and reliable inferences, qualitative content analysis involves a set of systematic and transparent procedures for processing data. Some of the steps overlap with the traditional quantitative content analysis procedures (Tesch, 1990). Generally qualitative content analysis process can be divided into the following steps: Step 1: Arranging data for qualitative content analysis. Qualitative content analysis can be used to analyze various types of data. But generally the data needs to be transformed into written text format before analysis could start. If the data for analysis come from existing text, the choice of the content must be justified by what the analyst wants to know (Patton, 1990). In ILS field, qualitative content analysis is more often used to analyze interview transcripts in order to reveal or model people s information related behaviors and thoughts. When transcribing interviews, the following questions arise: (1) should all questions of the interviewer or only the main questions from the interview 2

3 guideline be transcribed; (2) should the verbal be transcribed literally or only in a summary; (3) should observations during the interview, sounds, pause, as well as audible behavior be transferred or not (Schilling, 2006). The answers to these questions are constrained by the research questions. Step 2: Deciding the unit of analysis. Unit of analysis refers to the basic unit of text to be classified during content analysis. Messages have to be unitized before it could be coded. Therefore, defining coding unit is one of the most fundamental and important decisions for content analysis (Weber, 1990). Changes of the unit will affect coding decisions as well as comparability of outcomes with other similar studies (Wever, et al, 2006). In traditional quantitative content analysis, there are six commonly used coding units: word, concept, sentence, paragraph, whole text (item), and theme. These units can be used in combination in one study (Berg, 2001). Qualitative content analysis usually uses theme as the meaningful unit for analysis rather than physical linguistic units. The themes can be expressed in single words, phrases, sentences, paragraphs, or entire documents. When using theme as coding unit, the researcher is primarily looking for the expressions of an idea (Minichiello, 1990). Thus, codes will be assigned to a chunk of message of any size as far as they together represent a theme or issue of interest to the researcher. Step 3: From units to categories: developing categories and coding scheme. Categories and coding scheme can be derived from three sources: the data themselves, previous related studies, and theories. Coding schemes can be developed both inductively and deductively. In studies where no theories are available, researchers need to generate categories inductively from the data. Inductive content analysis is particularly appropriate for studies that intend to develop theory rather than describing phenomenon or verifying existing theories. In most cases, researchers have a preliminary model or theory to start their inquiries. An initial list of coding categories can be generated from the model or theory and the model or theory could be modified within the course of inductive analysis as new categories emerge (Miles and Huberman, 1994). When developing categories inductively from raw data, researchers are suggested to use constant comparison method in that it is not only able to stimulate thoughts but also able to make differences between categories apparent (Glaser and Strauss, 1967). In the literature, adoption of schemes developed in previous studies is called for by researches because it makes possible the accumulation and comparison of research findings across different studies. In traditional content analysis, categories are required to be mutually exclusive because confounded variables would violate some statistical procedures (Weber, 1990). But in reality clear categories of a simple item could be very difficult. Qualitative content analysis allows fuzzy boundaries between categories. This means that a body of text can belong simultaneously to more than one category (Tesch, 1990). However, it still applies that the categories should be defined in a way that they are internally as homogeneous as possible and externally as heterogeneous as possible (Lincoln and Guba, 1985). To ensure the coding consistency especially when multiple coders are involved, it is recommended to develop an instruction dictionary, which often consists of categories names, definitions or rules for assigning codes, and examples (Weber, 1990). Some dictionaries have an additional field for taking notes when coding goes on. Step 4: Test coding on sample text. The best test of the clarity and consistency of category definitions is to code a data sample. After the sample is coded, the coding 3

4 consistency needs to be checked, in most cases, by inter-coder agreement. If the level of consistency is low, the coding rules must be revised. Doubts and problems concerning the definitions of categories, coding rules, or categorization of specific cases need to be discussed and resolved within research team (Schilling, 2006). Coding sample text, checking coding consistency, and revising coding rules is an iterative process and will continue until sufficient coding consistency is achieved (Weber, 1990). Step 5: Code all text. When sufficient consistency has been achieved, the coding rules can be applied to code all the text. During the coding process, researchers need to check the coding constantly to prevent drifting into idiosyncratic sense of what the codes mean (Schilling, 2006). Step 6: Access coding consistency. After coding the whole text, coding consistency needs to be rechecked. It is not safe to assume that if the sample was coded in a consistent and reliable manner, so do the whole corpus of text. Human coders are subject to fatigue and are likely to make more mistakes as the coding proceeds. Also, their understanding of the categories and coding rules may change subtly over the time, which may lead to greater inconsistency (Miles and Huberman, 1994; Weber, 1990). Step 7: Drawing conclusions from the coded data. This is a process of making sense of the generated categories. At this stage, researchers make inferences and present their reconstructions of meanings. Activities may involve exploring properties and dimensions of categories, identifying relationships between categories, uncovering patterns, and testing categories against the full range of data (Bradley, 1993). Miles and Huberman (1994) provide a dozen of tactics for generating meanings from data display, such as partitioning variables, subsuming particular into the general, seeing plausibility, clustering, and making metaphors. For full discussion of each tactic, please refer to Miles and Huberman (1994). Step 8: Reporting. For the study to be replicable, researchers have to monitor and report their own analytical procedures and processes as completely and truthfully as possible (Patton, 1990). In cases of qualitative content analysis, researchers need to report their decisions and practices concerning the coding process, as well as the methods used to establish the trustworthiness of their studies. Qualitative content analysis does not produce counts and statistic significance, but rather uncover patterns, themes, and categories of social realities. Presenting research results from qualitative content analysis is challenging. Although it is a common practice to use typical quotations to justify conclusions, it is not always satisfying (Schilling, 2006). Miles and Huberman (1994) introduced and illustrated a set of other options for data display, including matrices, graphs, charts, and conceptual networks. The form and extent of reporting will finally depend on the specific research questions (Patton, 1990). When presenting qualitative content analysis results, researchers also need to achieve a balance between description and interpretation. Description gives readers background and context and thus needs to be rich and thick (Denzin, 1989). Qualitative research is fundamentally interpretative and interpretation represents researchers personal and theoretical understanding of the phenomenon under study. An interesting and readable report provides sufficient description to allow the reader to understand the basis for an interpretation, and sufficient interpretation to allow the reader to understand the description (Patton, 1990:430). 4

5 Nowadays, qualitative content analysis is usually supported by computer programs, such as Nvivo and Atlas-ti. The programs vary in their complexity and sophistication, but their common purpose is to assist researchers in organizing, managing, and coding data in a more efficient manner. The basic functions that usually supported by such programs include text editing, note and memo taking, text retrieval, and node manipulation. More and more qualitative data analysis software incorporate a visual presentation module into their programs, which allows researchers to see the relationships between categories more vividly. Some programs even record coding history to allow researchers to keep track of the evolvement of their interpretation. Trustworthiness Validity, reliability, and objectivity are criteria to evaluate the quality of research in the conventional paradigm. As a naturalist method, qualitative content analysis differs from the conventional tradition in its fundamental assumptions, research purposes, and inference processes, which makes the conventional criteria not suitable for judging research results produced by qualitative content analysis (Bradley, 1993). Recognizing this gap, Lincoln and Guba (1986) proposed four criteria to evaluate naturalist research work: credibility, transferability, dependability, and confirmability. They are naturalist equivalent to internal validity, external validity, reliability, and objectivity. Credibility refers to the adequate representation of the constructions of the social world under study (Bradley, 1993: 436). Lincoln and Guba (1986) recommended a set of activities that would help improve the credibility of research results: prolonged engagement in the field, persistent observation, triangulation, negative case analysis, checking interpretations against raw data, peer debriefing, and member check. To improve the credibility of qualitative content analysis, researchers not only need to design data collection strategies that are able to adequately solicit the representations, but also to design transparent coding and conclusion drawing processes that are able to make credible inferences from the raw data. Coders knowledge and experience have significant impact on the credibility of research results. It is necessary to provide coders precise coding definitions and clear coding procedures. It is also helpful to prepare coders a comprehensive training program (Weber, 1990). Transferability refers to the extent to which the researchers working hypothesis can be applied to another context. It is not the researchers task to provide an index of transferability; rather, they are responsible for providing data sets and enough descriptions to make the judgment of transferability possible on the part of other researchers. Dependability refers to the coherence of the internal process and the way the researcher accounts for changing conditions in the phenomena (Bradley, 1993:437). Confirmability refers to the extent to which the characteristics of the data, as posited by the researcher, can be confirmed by others who read or review the research results (Bradley, 1993: 437). The major technique for establishing dependability and confirmability is audit. The difference is that dependability is determined by checking the consistency of the process of the study and confirmability is determined by checking internal coherence of the research product, namely, data, findings, interpretations, and recommendations. The materials that could be used in auditing include raw data, field notes, theoretical notes, coding book, process notes, and so on. The audit process has five 5

6 stages: preentry, determinations of auditability, formal agreement, determination of trustworthiness (dependability and confirmability), and closure. A detailed list of activities and tasks at each stage can be found in Appendix B in Lincoln and Guba (1986). A letter of attestation is usually provided by the auditors after auditing (Lincoln and Guba, 1986). Example 1: Criteria for relevance judgment Schamber (2000) conducted an exploratory and naturalist inquiry into criteria, in particular criteria pertaining to information sources and presentation formats that occupational users of weather information employed to make relevance judgment on weather information. To get first-hand valid accounts from users, she used time-line interview method to collect data from thirty subjects, 10 each in construction, electronic power utilities, and aviation. These people are highly motivated and meanwhile specific in their needs for weather information. Complying with the naturalist approach, the interview responses must be interpreted in a way that does not compromise the original expression of the user. Inductive content analysis was chosen for its power to make such faithful inferences. The interviews were audio taped and transcribed. The transcripts served as primary sources of data for content analysis. Because of the purpose of the study is to identify and describe criteria used by people to make relevance judgment, Schamber defined a coding unit as a word or group of words that could be coded under one criterion category (Schamber, 2000: 739). Responses to each questionnaire item were unitized before they were coded. As Schamber pointed out, content analysis functions both as a secondary observational tool for identifying variables in text and an analytical tool for categorization. Content analysis started to be involved in this study at the pretest stage of developing questionnaire as a means for developing coding scheme as well as assessing the effectiveness of questionnaire items. The formal process of developing coding scheme began shortly after the first several actual interviews. The whole process was an arduous iteration of coding data sample, testing intercoder agreement, and revising schemes. When percentages of agreement did not reach acceptable levels, schemes were revised (Schamber, 1991). The author reported that: based on data from the first few respondents, the scheme was significantly revised eight times and tested by 14 coders until intercoder agreement reaches acceptable levels (Schamber, 2000: 738). The 14 coders were not involved in the coding at the same time; rather, they spread across three rounds of revision. The whole coding development process was an inductive analysis process and took a grounded theory approach. The author did not derive variables/categories from existing theories or previous related studies and she had no intention to verify existing theories; rather, she immersed herself into the interview transcripts and let the categories emerge on their own. Some categories in the coding scheme were straightforward and can be easily identified based on manifest content (or keywords); while others were harder to identify because they were partially based on latent content of the texts. The categories were expected to be mutually exclusive (distinct from each other) and exhaustive. The iterative coding process resulted in a coding scheme with eight first-level categories. 6

7 Credibility evaluates the closeness between researchers reconstruction and the real social reality. In the study, Schamber carefully designed and controlled the data collection and data analysis procedures to ensure the credibility of the research results. First of all, the time-line interview technique solicited respondent self account for relevance judgment on weather information in their real working environments instead of artificial test settings. Secondly, non-intrusive inductive content analysis was used to identify the themes emerged from the interview transcripts. The criteria were defined in respondents own language as appeared in the interviews. Furthermore, a peer debriefing process was involved in the coding development process, which ensures the credibility of the research by reducing the bias of a single researcher. As reported by Schamber (1991), a group of up to seven people, mostly graduate students including the researcher, met weekly for most of a semester and discussed possible criterion categories based on transcripts from four respondents (Schamber, 1991: 84-85). The credibility of the research results also was verified by the fact that most criteria were mentioned by more than one respondent and in more than one scenario. Theory saturation was achieved in light of the redundancy of criteria. Schamber did not claim transferability of the research results explicitly. But the transferability of the study was made possible by a detailed documentation of data processing in a Codebook. The first part of the Codebook explained procedures for handling all types of data (including quantitative). In the second part, the coding scheme was listed in the format of: identification numbers, category names, detailed category definitions, coding rules, and examples. This detailed record of data handling and the coding scheme makes it easier for future researchers to judge the transferability of the criteria to other user populations or other situational contexts. The transferability of the criteria was also verified by the fact that the criteria identified in this study were also widely documented in previous research works. Dependability of the research results in this study was established by transparent coding process and intercoder verification. The inherent ambiguity of word meanings, category definitions, and coding procedure threaten the coherent and consistency of coding practice, hence to affect the credibility of the results. To make sure that the distinctions between categories are clear to coders, a Codebook was developed specifically for the project. To ensure coding consistency, every coder used the same versions of scheme to code the raw interview data. Training and experience of coder is necessary for reliable coding (Neuendorf, 2002). In the study, intercoders were graduate students who had been involved the revision process of the coding scheme and thus were experienced at using the scheme (Schamber, 1991). The final coding scheme was tested for intercoder reliability with a first-time coder based on simple percent agreement: the number of agreements between two independent coders divided by the number of possible agreements. For the first-time coder, 90% agreement was achieved on all firstlevel categories except Criterion Type. For the Criterion Type category, 81.8% agreement was achieved at the summary level and 77.3% at the detail level. As suggested by Lincoln and Guba (1986), confirmability is primarily established by comfirmability audit. Schamber did not take processes similar to audit to address the comfirmability of the results. However, the overlap of the criteria identified in the study with other studies to a great extent indicates that the research results can be confirmed by 7

8 other researchers. Meanwhile, the detailed documentation of data handling also provides means for comfirmability checking. When reporting the trustworthiness of the research results, instead of using the naturalist terms credibility, transferability, dependability, and confirmability, Schamber used internal validity, external validity, reliability, and generalizability. It is worth pointing that there is no universal agreement on the terminology when reporting the quality of a qualitative inquiry. But it is recommended that the four criteria proposed by Lincoln and Guba (1986) should be used evaluate the trustworthiness of the research work conducted under the naturalist paradigm. Descriptive statistic like frequency of criteria was reported in the study. However, because the purpose of the study is to describe the range of the concept of relevance for the occupational use of weather information, the main result is a list of criteria that users employed to decide the degree of relevance of weather information in their work. The criteria were reported along with definitions, keywords, and examples. Quotations excerpted from interview transcripts were used to support identified criteria as well as illustrate the situational contexts. Example 2: Information seeking in interdisciplinary context Information seeking behavior in inter-disciplinary context was less covered in the literature than it is in single discipline. Foster (2005) presents an attempt to explore information seeking behaviors of scholars in interdisciplinary contexts. Two levels of inquiries were pursued: empirically, to identify the activities, strategies, contexts, and behaviors of interdisciplinary information seekers and the relationships between behavior and context; theoretically, to represent information seeking behavior of interdisciplinary researchers in an empirically grounded model. The study is a naturalist inquiry, using semi-structured interview technique to collect direct accounts of information seeking experiences from forty five interdisciplinary researchers. The respondents were drawn by purposive sampling along with snowball sampling. To enhance contextual richness and minimize fragmentation (Foster, 2004: 230), all participants were interviewed in their normal working places. In light of the explorative nature of the study, grounded theory approach was taken to guide the data analysis. Foster did not hold any specific expectations about the data before the analysis started. Rather, she/he expected that concepts and themes related to multi-disciplinary information seeking would emerge from the texts by inductive content analysis and constant comparison method. Coding took place in multiple interactions over time. The initial coding process was an open coding process. The author closely read and annotated each interview transcript. In this process, the texts were also unitized into concepts and concepts were highlighted and labeled. Based on the initial analysis, three stages of information seeking in interdisciplinary contexts, initial, middle, and final, along with activities involved in each stage were identified. Subsequent iterations of coding took place in the manner of constantly comparing the current transcript with previous ones to allow the emergence of categories and themes. As the coding proceeded, additional themes and activities emerged. The new categories were not covered by the initially identified three-stage model. Further analysis of emergent concepts, themes and their relationship to each other generated a two-dimensional model of information seeking behaviors in interdisciplinary 8

9 context: one dimension delineates three core processes of information seeking activities, opening, orientation, and consolidation. The three processes are not linear; the other dimension illustrates three levels of contextual interaction: cognitive approach, internal context, and external context. A computer program named Atlas-ti was employed to assist coding, which allows the researcher to retrieve text based on keywords, rename or emerge existing codes without messing up the rest of the codes, and generate visual views of emergent codes and their relationship to one another. Atlas-ti also maintains automatic logs of coding changes, which makes possible keeping track of the evolvement of analysis. As reported by Foster, coding consistency in this study was addressed by including three iterations of coding spread through a period of one year. However, the author did not report the three rounds of coding in detail, for example, how many coders were involved in the coding, how the coders were trained, how the coding rules were defined, and what strategies were used to ensure transparent coding. If all three rounds of coding were done by Foster, the indictor of coding consistency is a little weak compared to inter-coder agreement. But it is a legitimate practice for studies carried out by a single researcher. The issue of trustworthiness of the study was discussed in terms of the criteria suggested by Lincoln and Guba (1985): credibility, dependability, transferability, and confirmability. The credibility was establishing mainly through member checking and peer debriefing techniques. Member checking was utilized in four ways at various stages of data collection and data analysis: (1) at the pilot stage, the interviewer discussed interview questions with participants at the end of each interview; (2) during formal interviews, the interviewer fed ideas back to participants to refine, rephrase, and interpret; (3)in an informal post-interview session, each participant was given the chance to discuss findings; (4) an additional session was conducted with a sample of five participants willing to provide feedback on the transcripts of their own interview as well as evaluate the research results. Peer debriefing was used in the study to confirm interpretations and coding decisions including the development of categories (Foster, 2004: 231). No further details about who the debriefer was and how the peer debriefing was conducted were reported in the paper. The author made no claim for generalizability of the study results. But the transferability of the present study was ensured by rich description and reporting of the research process (Foster, 2004: 230). Future researchers can make transferability judgment based on the detailed description. The issues of dependability and confirmability were addressed through research notes, which recorded decisions, queries, working out, and the development results (Foster, 2004: 230). By referring to these materials, the researcher self could audit his/her inferences and interpretations and other interested researchers can review the research results. The content analysis results were reported by describing each component in the emerged model of information seeking behaviors in interdisciplinary contexts. Diagrams and tables were used to facilitate the description. A few quotations from participants were provided to reinforce the author s abstraction of three processes of interdisciplinary information seeking: opening, orientation, and consolidation. Finally, implications of the new model to the general exploration of information behaviors were laid out. 9

10 Reference Allen, B., & Reser, D. (1990). Content Analysis in Library and Information Science Research. Library & Information Science Research, 12(3), Berg, B. L. (2001). Qualitative Research Methods for the Social Sciences. Boston: Allyn and Bacon. Bradley, J. (1993). Methodological issues and practices in qualitative research. Library Quarterly, 63(4), De Wever, B., Schellens, T., Valcke, M., & Van Keer, H. (2006). Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review. Computer & Education, 46, Denzine, N. K. (1989). Interpretive Interactionism. Newbury Park, CA: Sage. Foster, A. (2004). A nonlinear model of information-seeking behavior. Journal of the American Society for Information Science and Technology, 55(3), Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: strategies for qualitative research. New York: Aldine Publishing Company. Hsieh, H.-F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), Krippendorff, K. (1980). Content Analysis: An Introduction to its Methodology. Beverly Hills, CA: Sage Publications. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic Inquiry. Beverly Hills, CA: SAGE Publications. Miles, M., & Huberman, A. M. (1994). Qualitative Data Analysis. Thousand Oaks, CA: Sage Publications. Minichiello, V., Aroni, R., Timewell, E., & Alexander, L. (1990). In-depth interviewing: Researching people. Hong Kong: Longman Cheshire Pty Limited. Neuendorf, K. A. (2002). The Content Analysis Guidebook. Thousand Oaks, CA: Sage Publications. Patton, M. Q. (1990). Qualitative Evaluation and Research Methods. NewBury Park, CA: SAGE Publications. Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts. International Journal of Artificial Intelligence in Education, 12, Schamber, L. (2000). Time-line interviews and inductive content analysis: Their effectivenss for exploring cognitive behaviors. Journal of the American Society for Information Science, 51(8), Schamber, L. (1991). Users' criteria for evaluation in multimedia information seeking and use situations. Unpublished doctoral dissertation, Syracuse University, Ann Arbor, MI. Schilling, J. (2006). On the pargmatics of qualitative assessment: Designing the process for content analysis. European Journal of Psychological Assessment, 22(1), Tesch, R. (1990). Qualitative Research: Analysis Types & Software Tools. Bristol, PA: The Falmer Press. Weber, R. P. (1990). Basic Content Analysis. Newbury Park, CA: Sage Publications. 10