Chat transcripts are fast becoming a standard tool both for assessing online reference. The Value of Chat Reference Services: A Pilot Study

Transcription

1 JoAnn Jacoby, David Ward, Susan Avery, and Emilia Marcyk 109 The Value of Chat Reference Services: A Pilot Study JoAnn Jacoby, David Ward, Susan Avery, and Emilia Marcyk abstract: This article explores student, instructor, and librarian perceptions of chat reference in the context of an introductory composition course. Participants in a mixed-method study responded to an anonymized chat transcript. While student respondents valued speed and efficiency, they were willing to receive instruction and open to questions that demonstrated interest or moved the research forward. Librarian and instructor comments focused on how these chats supported or could better support student learning, as well as on the interplay between classroom teaching, library instruction, and reference. The authors also identify implications that these findings have for the development of best practices in chat reference. Introduction Chat transcripts are fast becoming a standard tool both for assessing online reference services and for training reference staff. While many studies examine such areas as best practices for communication in chat 1 or for providing instruction in chat, 2 in nearly all cases only library researchers or experts evaluate the transcripts. Bringing the patron perspective to chat transcript analysis creates an important opportunity for the profession to see how the elements of chat reference that users value align with the advice and best practices advocated in expert studies. This article examines chat reference service from multiple viewpoints those of students, instructors, and librarians and focuses on a specific context: first-year undergraduate students completing a research assignment for an introductory composition course. As Alison J. Head found in the Project Information Literacy study, this is a formative time for undergraduate students. Head reports that, while most first-year students struggled to develop the research competencies expected in this new environment, they nevertheless found campus librarians (29%) and the English composition portal: Libraries and the Academy, Vol. 16, No. 1 (2016), pp Copyright 2015 by Johns Hopkins University Press, Baltimore, MD

2 110 The Value of Chat Reference Services: A Pilot Study instructors (29%) were the most helpful individuals on campus with guiding them through college-level research. 3 This study uses the framework of grounded theory, a method in which researchers develop categories or codes by first examining data for repeated patterns or concepts. As patterns emerge, they are summarized into more formal codes, which are then analyzed to develop hypothesis. Aiming to understand a setting or experience from the participants point of view, 4 the study focuses on identifying the elements of the chat reference interaction that students, instructors, and librarians found most significant. The study also examines how these reference interactions did and could better support student learning. The desired outcomes of this pilot study are twofold: (1) to identify issues associated with high performance or high perceived value of the chat interaction from the perspectives of students and class instructors; and (2) to investigate the implications of these findings to the ongoing development of best practices for chat reference. In addition, our discussion of the study s findings addresses how this new method for gathering evaluative user input builds and expands upon the research findings from more traditional evaluative methodologies. In particular, through the inclusion and analysis of focus-group comments and survey feedback, some unexpected crossover between student and librarian goals emerges. Literature Review The literature on chat reference is extensive, but the current study takes a novel approach that brings together and builds upon threads from previous research in three areas: (1) studies examining the user s perspective and motivations for using chat reference services; (2) the analysis and review of chat transcripts using various rubrics or scoring tools; and (3) chat reference used in conjunction with instruction, the use of instruction in chat, or both. Prior research into patron perspectives on chat reference has mainly focused on the quality of the transaction itself 5 or traditional measures of user satisfaction, 6 with little attention to the real-life context of the information need. Studies by Jeffrey Pomerantz and Lili Luo 7 and by Lynn Silipigni Connaway, Marie Radford, and Timothy Dickey 8 that examined users motivations for seeking reference assistance are notable exceptions. Both studies looked at a broad cross section of users with varied information needs. The current study, in contrast, approaches the patron s viewpoint as situated in a particular context, specifically focusing on student and instructor perspectives on chat reference as a resource for helping undergraduates develop research strategies for a specific assignment. The methodologies used in prior research also differ significantly from the current pilot study. Studies looking at user perspectives on reference services include Christina Desai and Stephanie Graves s examination of both chat transcripts and reference desk activity through unobtrusive observation. 9 Lynn Silipigni Connaway and Marie Radford s Seeking Synchronicity study utilized semi-structured interviews with users from various age groups to discover which factors users most valued. 10 These studies asked students to assess their experience and define service expectations, but none involved students looking at actual chat transcripts.

3 JoAnn Jacoby, David Ward, Susan Avery, and Emilia Marcyk 111 A number of authors focused on the role of existing professional standards and guidelines as tools for evaluating the role of instruction in chat reference. Studies by Fu Zhuo, Mark Love, Scott Norwood, and Karla Massia; 11 Wyoma van Duinkerken, Jane Stephens, and Karen I. MacDonald; 12 and Sarah Maximiek, Erin Rushton, and Elizabeth Brown 13 examined chat transcripts to determine how librarians adhered to the Reference and User Services Association (RUSA) Guidelines for Behavioral Performance of Reference and Information Service Providers. 14 Similarly, investigations by Lisa Ellis 15 and by Susan Avery and David Ward 16 focused on an examination of chat transcripts for evidence of instruction utilizing the Association of College and Research Libraries (ACRL) Information Literacy Competency Standards for Higher Education. Avery and Ward framed their study by highlighting the need to articulate, codify and assess educational outcomes for reference services. 17 This pilot study is unique in bringing both the student and instructor perspectives to bear on evaluation of transcripts, providing an opportunity to assess whether the aspects of chat reference that users most value align with the best practices articulated in professional standards and other expert studies. Seeking to discover a link between reference and instructional services is not new. James Elmborg wrote of the need to establish a reference desk pedagogy, noting the academic library s responsibility to align with the teaching mission of its parent institution. 18 Other studies have described the potential that chat reference offers as an instructional tool, examining both the measurable presence of instruction in chat, as well as best practices and methodologies to intentionally include it in online reference interviews. 19 Studies by Bonnie Swoger and Kimberly Hoffman, by Gillian Gremmels and Karen Lehmannn, and by JoAnn Jacoby and Nancy O Brien assessed student perceptions of what they had learned immediately after an in-person reference encounter. 20 Denise Agosto, Lily Rozaklis, Craig MacDonald, and Eileen Abels examined the evolving roles of reference services, ultimately concluding that librarians are now needed more than ever for their expertise in teaching users. 21 Methodology The methodology of this pilot study was shaped by a focus on examining chat reference in the context of undergraduates developing research strategies for an assignment, resulting in a more targeted recruitment strategy than used in prior research. The study limited participation to students assigned a specific research project as part of an introductory composition course. Within this frame, the researchers considered three distinct perspectives those of students, instructors, and librarians allowing the triangulation of commonalities and the consideration of differences among these groups. The investigators recruited student and instructor participants from Rhetoric 105 (RHET 105) courses. RHET 105 is one of the classes that fulfills the Composition 1 requirement at the University of Illinois at Urbana Champaign. RHET 105 comprises the largest Composition 1 course on campus, with approximately ninety sections each semester of mainly traditional students. More than half the sections taught during the academic year used a locally developed e-text, the University of Illinois. The e-text begins with students writing about their observations of some element of campus life. Then the students must locate primary sources relevant to the observation,

4 112 The Value of Chat Reference Services: A Pilot Study after which they seek secondary sources. While the university does not mandate library instruction, a majority of the sections do participate in library instruction during the secondary source assignment. Because of the consistency of assignments among RHET 105 sections using the e-text, this study targeted those sections. The investigators used a mixed-methods approach comprised of a survey, focus groups, and individual interviews administered in spring Participants included students currently enrolled in RHET 105, instructors of the course, and librarians and reference staff who provided chat reference service through the university s Ask-A- Librarian program at the time of the study. All participants responded to an anonymized transcript of a chat reference interaction with a RHET 105 student working on his or her research assignment. This approach represents a variation of the critical incident technique, a procedure used for collecting direct recollections of personal experiences, typically asking participants to recall the best or worst experiences, or both, they have had with the area being researched. 23 While participants were not explicitly asked to describe their own experience, as with classic critical incident technique, the timing of the survey and the focus groups ensured that the students had already begun working on their own research papers. Not surprisingly, many of them spontaneously related their own experiences as they responded to the chat transcripts. Likewise, the librarians and instructors often reflected on how something in a particular chat was similar to or different from interactions they had experienced when working with students on the RHET 105 research assignments. The research team pulled chat transcripts from the prior semester to help ensure the anonymity of the students involved. The team analyzed the transcripts for content and examined them for specific keywords, topic focus, and discussion, to identify those most likely to indicate a student working on a RHET 105 class assignment. From this pool of transcripts, the researchers selected the final transcripts based on a desire to have exemplars showing a high, medium, and low level of interaction between student and librarian. The reasoning behind this approach was to see whether participants in the study noticed differences in quality, and what behaviors (or lack thereof) they would attribute to these differences. Participants could choose to participate in two ways: (1) responding to a brief survey that included a mix of Likert-scale and open-ended questions (Appendix A) or (2) participating in a peer focus group (Appendix B). Because of the challenges of scheduling instructors for group sessions during a busy portion of the semester, the instructors participated in semi-structured individual interviews following a slightly modified version of the focus-group script. Rather than librarians running the focus groups, trained graduate and undergraduate students with prior experience as moderators guided the groups and interviews to encourage frank and open discussion. The research team invited fifty-seven reference providers (including professional librarians, preprofessional graduate assistants, and a small number of paraprofessional staff, referred to throughout simply as librarians ) 23 and thirty-six RHET 105 instructors (a group consisting of teaching assistants, lecturers, and instructors) to participate in the study. The researchers also asked instructors to forward invitations to students in their classes. Follow-up invitations were sent two weeks after the initial . All students and instructors who participated could enter a drawing to receive one of two $50 gift

5 JoAnn Jacoby, David Ward, Susan Avery, and Emilia Marcyk 113 Table 1. Survey respondents Focus group Total or interview respondents Survey by group Invited Response rate Librarians % Students (estimated*) 3% (estimated*) Instructors % *The estimated student response rate is based on the following assumptions: (1) every instructor contacted forwarded the invitation to his or her students; and (2) instructors taught a total of 52 sections with an average 18 students per section. These assumptions likely overestimate the number of students who actual received invitations and thus may underestimate the actual response rate. certificates. A total of twenty-five librarians, twenty-four students, and nine instructors participated. Table 1 shows a detailed breakdown of the participants and response rate. Analysis Qualitative analysis of the focus groups, interviews, and responses to the open-ended survey questions followed the constant comparative method, a process in which newly collected data are compared with data collected earlier in the same study, using the group (that is, students, instructors, or librarians) as the unit of analysis. 24 Two or more researchers independently coded each transcript, and the investigators reviewed the codes and compared them to ensure inter-coder reliability. The full research team reviewed the entire corpus to decide which closely related codes could be combined and which key themes would be the focus of further analysis. Some codes related directly to specific questions asked in the focus group or interview script, while others were emergent (Table 2). Both scripted and emergent themes uncovered similarities and differences among students, instructors, and librarians regarding expectations of chat reference service, as discussed below under the heading Findings. The small sample size precludes performing any statistical analysis of the survey responses, particularly the Likert questions (numbers 5 to 8 in Appendix A), and also limits the generalizability of the study findings. That said, this study s focus on a population with a shared frame of reference provides unusually rich insight into chat reference in support of undergraduate instruction and student learning. It also provides an opportunity to delve deeply into the perspectives offered by the participants in the

6 114 The Value of Chat Reference Services: A Pilot Study Table 2. Emergent themes versus questions asked Emergent themes Students experience as novice researchers Speed/Convenience Referrals Themes related to questions in script Question negotiation Focus-group question 4: Did the librarian understand the question? What questions could he or she have asked to get a better understanding of what the student needed? Instruction Focus-group question 5: Did the librarian explain how to use the sources they consulted to find the answer? Focus-group question 7: Do you think this chat helped the student learn more about how to do the research they needed for this assignment? Communication/Tone Focus-group question 1: Is the tone of the chat you just read welcoming and supportive? Do you think it s important that the tone be welcoming? study. The discussion of our findings therefore focuses on the key themes arising from the qualitative analysis of the focus groups or interviews, as well as from the responses to the open-ended survey questions. Students as Novice Researchers Findings After reviewing the chat transcripts, many students commented spontaneously on the challenges they and their peers faced as novice researchers. Some described what they imagined the student in the chat was experiencing, while others spoke directly to their own experience. Likewise, instructors remarked both on the specific situation faced by a student in the transcript, as well as how it related to the more general experience of students in their classes. Librarians, in contrast, often talked about specific strategies for helping students work through common points of confusion or frustration in the search

7 JoAnn Jacoby, David Ward, Susan Avery, and Emilia Marcyk 115 Students used such terms as process. As might be expected, participants tended to contextualize the transcripts within the larger scope of similar situations they had personally experienced. RUSA Behavioral Guideline One addresses the importance of making the patron feel comfortable in a situation that can be perceived as intimidating, confusing, or overwhelming. Our analysis focused on understanding the pain points the participants identified as common among this group of students, as well as which strategies seemed most effective to alleviate anxiety. Students used such terms as confused, frustrated, struggling, stuck, completely lost, and even scared to describe their experience working on the research assignment. They worried that they were doing the research incorrectly or that they were asking stupid questions, harking back to classic descriptions of library anxiety. 25 As one example of this theme, students often mentioned struggling to define their topic: It s a common issue that RHET 105 students have gone through, noted one student, and the librarian helped. Instructors observed that the students were frustrated and afraid to ask for help. They described the student s research questions in the chats as lacking context and kind of undefined and squishy. One instructor pointed out, Students sometimes get locked into particular terms and forget to be more flexible. Instructors also reflected more generally on some of the common challenges these students face while completing their research for the course: confused, frustrated, strug- I mean it sounds really like a RHET 105 student... it s way too vague. It s pretty typical. Some of the gling, stuck, completely lost, and even scared to describe their experience working on the research assignment. instructors went on to suggest specific approaches librarians could use to help students through some of these issues, such as starting with what the student had already done and making suggestions, paying attention to the process that they take on their own when they don t have a librarian s help, and steering them to something that would have research [published]... to the more researchable part of the question. Librarians also recognized when students were struggling or frustrated and short on sources. One librarian offered a typology of the issues observed in previous interactions with students: Everyone has different frustrations in their search process. Someone has more trouble finding the tool, the box to start their search. Some people [have] problem[s] understanding the multiple databases... Some people have difficulty understanding how to reframe their research questions and decide on the keywords for the search. Librarians usually quickly followed such observations with a mention of specific strategies they might use in such situations, for example: When someone comes in and you can tell they re already floundering and not clear on what they re doing that s when you need to put in more helpful phrases like oh yeah, lots of people have trouble doing that, but I think I can help you or let me see if I can make that a little easier. You rarely find what you need in one search. Sometimes it takes a few tries. That s an important concept to convey.

8 116 The Value of Chat Reference Services: A Pilot Study Question Negotiation One key area the research team was interested in was the perception of traditional reference interview techniques utilized by library staff during the initial question negotiation, when the student and librarian try to define and fine-tune the information sought and the search strategy. Parts of both the RUSA Behavioral Guidelines Three (Listening and Inquiring) and Four (Searching specifically formulating a search strategy) fall under this categorization. Our analysis focused on discovering which behaviors or techniques contributed to a perception of a positive engagement with each student, as well as which techniques helped bring about a successful, effective, and efficient transaction. There were some differences in factors cited between the different user groups analyzed, as well as differences within each group. Student responses tended to concentrate on how the librarian helped focus or refine the topic and did not comment extensively about formulating search strategies. Examples of student responses related to topic refinement include: It seems like if I am confused about my research I can turn to the Ask-A-Librarian service and they will definitely help clear up the confusion and help me gain a better understanding of my topic, maybe it could help me unravel ideas and my main topic/point/thesis. So I think... she did a really great job. Or he or she, this librarian, kinda narrowed down the thesis. During the focus groups, students also commented on the assistance librarians provided in selecting keywords for a topic. For example, one student said, I thought the librarian did a good job on that, with just giving the words. Cause normally those words can make all the difference as far as results. Librarian feedback on topic refinement was typically referred to in the larger context of performing a complete reference interview. Responses were both positive and negative in their analysis, for example: Good job of the reference interview, finding out what aspect of the topic the student is interested in, what the student has already done, and the kinds of information he/she is looking for. If he or she just did like I think a little more follow up interview would help to really focus on the question and find matching materials because really would have millions of stuff there. Librarians also commented more extensively than students on keyword selection. These observations typically revolved around both the need to select appropriate search terms, as well as the importance of selecting keywords prior to engaging in a search: Although they were persistent with the patron, the librarian moved on to searching before really having good keywords/a complete understanding of the question. Instructor feedback had some similarities to librarian feedback in points of emphasis, including keyword selection and choice of an appropriate research topic or thesis. As might be expected, the instructors comments tended to frame their analysis more in terms of pedagogical goals, as opposed to the librarians observations focusing on professional standards.

9 JoAnn Jacoby, David Ward, Susan Avery, and Emilia Marcyk 117 It s hard to tell whether the librarian and student have a shared understanding of what they mean by conformity. They started with what the student had already done and made suggestions to build from there. I help students all the time doing this sort of narrowing of topics so yeah I m thankful that the librarian was doing a little of that too. Like please pick something you can research. Open and Closed Questions A final finding in this area relates to the traditional use of open and closed questions in the reference interview. In the focus groups, the moderators specifically asked participants to provide feedback on the use of questions by librarians in the transcripts they analyzed. While participants typically welcomed the librarians questioning in the chat encounter, responses varied and expressed a full range of opinion on the value of questioning. For example, consider the following pair of student responses from the survey: I just like how the librarian... tries to understand everything the student needs. The librarian asked too many questions in the beginning. Instructor comments on questions tended to be the most in-depth and reflected back onto the classroom context: I would be asking more questions to kind of narrow it down to do with things like, you know, why this topic? Why is it important? Why should we be doing this now? My role as a teacher, not as a librarian, would be asking more questions. Kind of get the context of why they re asking the questions and what they want to find out. Instruction Numerous authors have acknowledged the importance of incorporating instruction into the provision of reference service. 26 Established standards, such as the ACRL Information Literacy Standards for Higher Education, provide guidelines for introducing instruction in reference. In particular, Standard One, which calls for defining an information need; The chat setting provides Standard Two, focused on accessing information; and Standard Three, about evaluating unique challenges, and the information, can offer guidance for reference strategies librarians employ librarians seeking to work with students in an educational setting. However, the chat setting provides unique challenges, and the strategies librarians employ can, and should, differ from in-person reference encounters. One of the goals of this study was to examine the chats for evidence of instruction; to note if the students, instructors, and librarians participating in the focus groups and surveys were conscious of instructional efforts on can, and should, differ from inperson reference encounters.

10 118 The Value of Chat Reference Services: A Pilot Study the part of the librarians providing the chat; and to probe whether they perceived such instruction as valuable. As noted earlier, the students participating in the survey and focus groups attended a library instruction session with their RHET 105 class. The impact of that instruction was evident in numerous comments the students shared. A pre-instruction activity in each RHET 105 class was the completion of a concept map that required students to write their research question and identify keywords and alternatives. The students bring the completed worksheet to the library instruction session and use it as they begin their database searches. In responses to the chat transcripts, several students spoke of the role of keywords in constructing a search strategy and the necessity of identifying keywords and alternative terminology. They also observed the need for Numerous students singled out teachable moments that were missed during the course of a chat transaction. distinct search terms to help broaden and narrow searches. One student commented on the need to start off broad and just keep narrowing it down to what you re trying to find. Likewise, the importance of recognizing the varied foci of individual databases, another element of the library instruction, was observed when a student spoke of the various perspectives found in databases. Evidence of student awareness of the librarian sharing strategies so that students could become more successful in their future research can be found in the following examples: The librarian... offered a new perspective and insight on the questions asked by the student. She could easily just send links, but, she also... gave some suggestions... So I think that s a very good service. Anyone can send links. Anyone can do this and that, but it s all about showing service and doing the best you can to help the student. She also helped... showing him how to use the database so that he could search for things himself. Numerous students singled out teachable moments that were missed during the course of a chat transaction. Several focus-group comments discussed the selection of search terms and the failure of the librarian to be more specific in explaining their recommendations to the student: She also could ve elaborated on more variations of the words and stuff. Like different synonyms. They didn t say like why the database prefers those words or why it would be more prevalent in the database. Students also exhibited awareness of a librarian failing to be attentive to student needs. For example: The librarian just searched by himself instead of exploring with students together. Similarly, it was obvious to students when the librarian discussed strategies unknown to the student, as in the case of a librarian who discussed the use of truncation.

11 JoAnn Jacoby, David Ward, Susan Avery, and Emilia Marcyk 119 Some threads evident in the librarian comments resembled those of the students who participated in the survey and focus groups. Reflecting on the degree of instruction provided to the students, librarians shared some positive observations in the focus groups: I like that... they were sort of up front about I m gonna teach you something now. I ll send you those articles, but I want you to know how to do it yourself the next time. Once we re both in the database, and we both tried some keywords, I d say Look, I found this one. Is this what you like? And then I ll tell them the keywords I used to get there. As with the students, not all of the comments of the librarians were favorable. Many remarks focused on the librarian either moving too quickly or failing to provide the student with an adequate explanation for one or more steps in the search process: The librarian was doing a lot of good things but trying to do too many of them at one time instead of step by step, which increased miscommunication with the student. We re talking about using the database or some search tool so it s important to be specific and do it step by step. The librarians were particularly attuned to instances where teachable moments were missed in the chat transcripts. Vigorous discussions often took place when such concerns were brought up in a focus group. Librarians librarian either moving too noted that those providing chat services failed to explain important steps and details in the search quickly or failing to provide process, missing what one called a big prompt. Another explained: Like I feel like there s a HUGE gap right there where they didn t talk at explanation for one or more all about something that is incredibly important steps in the search process: in searching and very basic in searching. The instructors brought a different perspective to the study, given their awareness of the assignment and expectations for the articles students would retrieve in their searches. The instructors were significantly less critical of the chat interactions than the librarians were. Observations of instruction, and a sense of appreciation for the service provided, were plentiful in survey comments. For example: Many remarks focused on the the student with an adequate The librarian... didn t simply give the student the answer. While the librarian provided a source that met the student s criteria, the librarian also explained to the student how to locate similar sources. I like that the librarian reminded the student to use the key words in both the article description and the side of the search page. Interviews with instructors yielded a greater sense of the perceived instructional value of the librarian-student interaction in the course of the chats. One observed that the chat would help them learn how to do it on their own. Another remarked, They kind of give them the path to get there again on their own. Instructors did recognize

12 120 The Value of Chat Reference Services: A Pilot Study when the librarian missed opportunities for instruction. These included instances of failures to explain information about sources, to describe the difference between a database (Academic Search Premier) and the vendor (EBSCOhost), or to encourage students to read article abstracts. Speed and Convenience There is often a tug of war in chat reference between providing high value services such as point-of-need instruction and scaffolding or providing support when students are learning new skills and individual patron expectations for speed and convenience. How to achieve this balance was not among our initial research questions, but it emerged as a key theme. Comments from participants demonstrate how observers, and student observers in particular, perceived this balance and articulated their specific expectations for the balancing point between getting a quick answer and engaging in a detailed discussion of research methods. Many students praised the time taken by the librarian to teach. Other students made suggestions for improving the interactions through instructional elements such as evaluation of sources, suggesting alternative databases, and searching for both journal articles and books. However, such actions did make the interaction longer, creating a tension between completeness and speed. Some of the student responses clearly attempted to weigh the time spent on the chat versus the information requested and the anticipated librarian response. A number of students made comments such as the Librarian asked too many questions and It was a little long. These concerns are perhaps most clearly articulated in two focus-group explanations: The instructors were significantly less critical of the chat interactions than the librarians were. Many students praised the time taken by the librarian to teach. Other students made suggestions for improving the interactions through instructional elements such as evaluation of sources, suggesting alternative databases, and searching for both journal articles and books. Ya know at the end of the day when you re a student your essay is due tomorrow, you call a librarian, all you want is for someone to kinda give you [what you re looking for], but at the end you just come out with someone understanding your thesis. And that s what you kinda feel you don t want in the end. So, I mean she basically wasted her entire time trying to ask all these questions. So... I mean that s just a thought... so minutes, half an hour is a long time. Students can do a lot in a half hour. Other students, however, made comments on the speed of the service, remarking that it was efficient and straight to the point, without taking too much time ; It actu-

13 JoAnn Jacoby, David Ward, Susan Avery, and Emilia Marcyk 121 ally turned out pretty well considering that you can do this from your dorm. It s pretty convenient ; and I like that a student that is writing a paper can have access to instant help instead of having to go to a library or being frustrated not being able to find what he or she is looking for. Students also expressed surprise at the quickness of the initial response, adding that the rapid response would make them more likely to consider using the service in the future: I also paid attention to the timing and (since I ve never used the Ask A Librarian) I was really surprised that the librarians do reply immediately. I always thought they would take a couple minutes at least to reply to the first message. Overall, the mixed student responses yielded no clear indicators of preference for speed and convenience. Rather, the variety of responses indicated that patrons may interpret these factors case-by-case, balancing a variety of factors, including time constraints, the perceived value of the librarian feedback, and the actual search results the interaction produced. Librarians and instructors approached speed and convenience from a different perspective than students did. They were much less likely to express concerns Factors that can contribute to pa- about the transactions taking too much time, and such comments as It seemed like the librarian was going fast at some parts were rare. Instructors, in particular, seemed to weigh the length of the tone, as well as the perception that the librarian sympathizes and cares chat transaction with the end result. For example, they said: The librarian about the outcome of the interview. gave the student some basic research techniques in a short amount of time and The student quickly obtains a useful article. Customer Service trons consciousness of higher quality service include friendly language and A common finding in studies of user satisfaction with reference service is the importance of good customer relations. Factors that can contribute to patrons consciousness of higher quality service include friendly language and tone, as well as the perception that the librarian sympathizes and cares about the outcome of the interview. 27 Our findings reiterated a number of these factors. Students commented on librarian behaviors, such as seeming understanding, helpful and caring, nice, encourage[ing], and respectful. The students also described the impact of these behaviors: I think it s good the librarian said, I ll be happy to help with that. So, it probably made the student feel comfortable when asking a question. For a student like you might be like scared to like get the courage to even ask and then like if they re welcoming then you would want to, for like future like papers and stuff, you could go back to ask. Instructor and librarian comments echoed these observations, describing positive behaviors using such terms as friendly, encouraging, reassuring, proactive without being judgmental, and super helpful. Focus-group feedback also clarified the benefits of com-

14 122 The Value of Chat Reference Services: A Pilot Study municating sincere interest, observing that the librarian in one transcript conveyed not just politeness, but actual encouragement. Librarian participants also remarked on the difficulty of expressing genuine involvement in the patrons questions through chat, observing: Especially since they can t like see you smile or like lean towards them or anything. You have to like exclusively state I m being friendly like You are welcome here. Participants also remarked on situations that seemed to indicate an unsatisfactory customer service encounter. Instructor and librarian comments described poor customer service as leading to conversations that were disjointed, not fluid, and potentially intimidating. Student observations in the survey tended to focus more on the positives and described the impact of poor customer service in more hypothetical terms: If the librarian s pretty rude or something, like the student s [sic] may feel like he or she asks something and their [sic] wrong and like they ll be a little bit scared of asking more questions. Like if the librarian was just like rude or like not so welcoming the, the student would probably feel like well, I didn t really feel like they cared, they re just there doing their job. Referrals Survey and focus-group participants, especially librarians and instructors, expected referrals to subject specialists, writing specialists, or even back to the instructors themselves as part of the research strategy suggested to patrons. Participants found chat interactions incomplete without referrals. As one librarian observes, if the librarian fails to make a referral when warranted, The patron will fairly quickly be back in the same situation they started using chat in frustrated and short on sources. One instructor also appreciated that the librarian made referrals outside the library to resources more appropriate to the current point in the student s writing process: Explaining why the Writer s Workshop or the instructor might actually be a better place to get help at this stage... because the topic is still quite vague and broad and you need to get it narrowed down and then come back to us. So that s doing a good job at explaining exactly what the library can be useful for and the kind of work they need to do before coming to get help with this. Some students also observed that the librarian should show resources beyond the ones the patron has already used: So the librarian, I guess, could ve gave like, like the list of systems to use because EBSCO isn t used for everything. Others expressed surprise at the extent of service the librarian was willing to offer, especially when the librarian offered to a subject specialist on the patron s behalf. One student focusgroup participant felt that it was very nice of the librarian to offer this service. Another student from another focus group was also enthusiastic: She was like, Do you want me to this person?... Like at 10:10, the education librarian. Like I can her and ask. I can go through all that stuff. I never would ve expected that. Like to go even beyond just databases. They really are just trying to help you.

15 JoAnn Jacoby, David Ward, Susan Avery, and Emilia Marcyk 123 Previous studies pointed out that chat interactions often lack a complete referral. 28 Although no questions in this study related to referrals (see Appendixes A and B), individuals in all three participant groups spontaneously pointed to a lack of referrals as incomplete service. While instructors and librarians indicated that referrals were an expected part of the chat interaction, some of the students were surprised by this level of service and responded positively. Although librarians may hesitate to make a referral for introductory composition students, the students in our study perceived referrals as exceptional customer service, suggesting this practice should be encouraged when there is sufficient time before the assignment is due. Discussion and Implications for Practice Individuals in all three participant groups spontaneously pointed to a lack of referrals as incomplete service. One of the key implications of this preliminary study was the willingness of students to receive instruction through chat. A common concern in academic libraries about providing chat reference is the delicate balance between providing a service that feels convenient and expedient, while at the same time advancing the library s pedagogical mission. Students who initially felt that the service should provide a few direct links to a handful of relevant articles as quickly as possible were, after further reflection, surprisingly open to the idea of a more sustained interaction with hands-on engagement in the research process. 29 Instructor feedback was mixed. Some instructors thought defining the topic was best done in the classroom with the instructor, not while working with a librarian. Others commented favorably on the similarities and potential synergy between the two. Regardless, many students come to the chat reference service with questions that were still squishy, suggesting the importance of aligning the teaching that happens in the classroom, in library instruction sessions, and on the desk so that these three types of instruction can function in concert. In that way, they can support students as they work through what really happens in the messy process of research, rather than what we think should happen. 30 The participants comments about defining a topic point to the iterative nature of the research process through which research questions are defined and redefined in the course of finding and evaluating sources (Figure 1). This lends further support to Elmborg s contention that reference interactions are a key teaching and learning opportunity. 31 His belief aligns with our finding that students are open to instruction at the crucial moment when they are fully engaged in searching, evaluating, and refining their topic in conversation with the sources they have discovered thus far. Findings also indicated that students could be receptive to appropriately phrased questions, particularly those that show interest in their topic and seem to develop the research question further or move the search process forward. Students appreciated questions that acknowledged the work they had already done and their contributions to the chat conversation. This aligns with Elmborg s observation that encouraging students to engage with search can help foster excitement about the research process. 32

16 124 The Value of Chat Reference Services: A Pilot Study Figure 1. Elements of the research process evident in chat reference interactions with RHET 105 students Students recognized the value of the instruction provided via the Ask-A-Librarian service in multiple comments in both the surveys and the focus groups. They acknowledged the usefulness of the service in several ways, including observations of the librarians insights and suggestions to improve individual student experiences. One unexpected outcome was the students recognition of the presence of teachable moments, both those observed in the transcripts and those missing from the interactions. Student awareness of missed teachable moments surprised the authors and further indicates the value of library instruction and the interconnectedness of reference, instruction, and other library services. Another unexpected result of the study was unsolicited feedback from students and instructors on their perception of the overall value of chat reference. Students frequently expressed surprise at the quality of the service, commenting, for example, I was actually kind of surprised how well it went, considering it was over chat, and They ll actu- Findings also indicated that students could be receptive to appropriately phrased questions, particularly those that show interest in their topic and seem to develop the research question further or move the search process forward.

17 JoAnn Jacoby, David Ward, Susan Avery, and Emilia Marcyk 125 ally help you with everything. Instructors expressed similar levels of satisfaction with what chat reference could offer as a supplement to their in-class work. One reported, I would definitely refer my students [to the chat service], because the student is so much better off than if they just kept struggling with the database where they had started. Another final area for improvement, which emerged organically from participants responses and was somewhat unexpected, was a call for better marketing of chat reference services. Participants, particularly students, were surprised at the extent of service. Through analysis of the chat, the participants seemed to arrive at a new recognition of its potential value, with one commenting: Why didn t anyone tell me about this before? Limitations and Future Directions surprise at the quality of the ser- Two primary concerns in this study were the small sample size and low response rate, which was exacerbated by analyzing multiple transcripts across groups. Because Students frequently expressed recruiting a large, committed pool of participants proved challenging, a follow-up study should focus on a single common transcript. vice, commenting, for example, Such a focus would help with gathering enough consistent feedback to support comparative analysis between the study groups: how well it went, considering it students, instructors, and librarians. A larger pool of responses would also aid in testing the specific themes that emerged from this study and help clarify the most important themes to focus on for training and ongoing program assessment. I was actually kind of surprised was over chat, and They ll actually help you with everything. JoAnn Jacoby is head of Research and Information Services at the University of Illinois at Urbana- Champaign; she may be reached by at: jacoby@illinois.edu. David Ward is reference services librarian in the Undergraduate Library at the University of Illinois at Urbana-Champaign; his address is: dh-ward@illinois.edu. Susan Avery is instructional services librarian in the Undergraduate Library at the University of Illinois at Urbana-Champaign; she may be reached by at: skavery@illinois.edu. Emilia Marcyk is instructional technology and information literacy librarian at Michigan State University in East Lansing; her address is: marcyk@msu.edu.

18 126 The Value of Chat Reference Services: A Pilot Study Appendix A Survey Instrument 1. What did you like about this chat? 2. What didn t you like about this chat? 3. How could it have been improved? 4. Other comments Likert scale for questions 5 to 8: Strongly disagree (1) Disagree (2) Neither agree nor disagree (3) Agree (4) Strongly agree (5). 5. The librarian was inviting and welcoming. 6. The librarian demonstrated a good understanding of the question by the end of the chat. 7. The librarian clearly explained what the person would need to know or do to continue searching on their own after the chat. 8. The student learned more about how to do the research needed for this assignment because of this chat. 9. (for students) Based on this interaction, would you be more likely to use the Ask-A- Librarian service for when you need help with your research? (for instructors) Based on this interaction, would you be more likely to refer your students to this service? (for librarians) Based on this interaction, do you think this student would be likely to use the service again next time they need this sort of help? 10. Why or why not? Appendix B Focus-Group Questions This transcript shows a chat with a student who is working on the secondary source research assignment for RHET 105 who contacted the Ask-A-Librarian chat service for assistance last semester. It has been made anonymous. Please read through the transcript of the chat session, focusing on the ways this chat did or did not help the student move forward with their research for the RHET 105 secondary source assignment. So look for things that stand out as working really well, as well as those that could have gone better. After everyone is done reading through the transcript, we are going to ask some questions about the chat. Your feedback and insights will help us better understand your needs and expectations so we can improve the chat service.