Application of Outcome- Based Evaluation to MINITEX Interlibrary Loan, Delivery, and Reference Services

Transcription

1 MINITEX Library Information Network Application of Outcome- Based Evaluation to MINITEX Interlibrary Loan, Delivery, and Reference Services August 2003 MANAGEMENT ANALYSIS DIVISION

2 Project team Mark Scipioni Roxanne Peterson Susan Senko Division director Judy Plante Assistant division director Bill Clausen Contact information Voice: Fax: Web site: Address: 203 Administration Building 50 Sherburne Avenue St. Paul, Minnesota Other formats To obtain these materials in an alternative format, call voice or TTY Copies of this report For more information or copies of this report, contact the MINITEX Library Information Network, University of Minnesota, 15 Andersen Library, st Avenue South, Minneapolis, MN Phone: or Management Analysis Division The Management Analysis Division is Minnesota government s in-house fee-for-service management consulting group. We are in our 18 th year of helping public managers increase their organization s effectiveness and efficiency. We provide quality management consulting services to local, regional, state, and federal government agencies, and public institutions.

3 CONTENTS Executive Summary 1 Introduction 3 MINITEX Interlibrary Loan, Delivery, and Reference Services 6 Outcome-Based Evaluation 17 Application of OBE to MINITEX Services 22 Conclusions and Recommendations 40 Appendix 1. Bibliography Persons interviewed MINITEX staff participants Survey pilot test participants Pilot test surveys Pilot test survey responses concerning value of services 76

4

5 Executive Summary MINITEX is a program of the Minnesota Higher Education Services Office and is located at the University of Minnesota Twin Cities Campus. MINITEX is a publicly supported network of participating academic, public, state government, school, and other types of libraries in Minnesota, North Dakota, and South Dakota. The mission of MINITEX is to enhance the efficiency and effectiveness of participating libraries. Services provided by MINITEX represent a major component of the infrastructure for resource sharing among libraries in the three-state region. Additionally, MINITEX has reciprocal service agreements with libraries in Wisconsin. Among the services provided are Interlibrary Loan and Delivery, Reference Referral, and Reference Training. The Institute of Museum and Library Services (IMLS), a federal agency, provides Library Services and Technology Act grants that, in part, fund MINITEX services. The IMLS has directed MINITEX and other library grantees to place greater emphasis on identifying and measuring the benefits to end-users of the services they provide. The IMLS defines the end-user as the library patron that is, persons who receive library services. In particular, the IMLS has directed the use of outcome-based evaluation (OBE) to measure the end-user benefits from funded services. Outcome-based evaluation presents the question, What has changed as a result of our work? The purpose of this project was to assist the MINITEX staff to: document interlibrary loan, delivery, and reference service components and expected changes in services based on foreseen trends and developments; identify end-user benefits and outcomes from the services; develop, pilot test, and refine survey instruments to capture and measure end-user outcomes; and recommend tools and methods to gather this evaluative information on an ongoing basis to address the directives of the funding agency. Using information from background research and interviews of local, state, and national authorities, the Management Analysis project team developed and pilot tested a survey instrument. The pilot test respondents were from the Minnesota library community and included a diverse mix of people with knowledge of library and MINITEX issues. Results from the pilot test provided significant new information for the study including refinements of end-user survey instruments for the next iteration of data gathering. The results of the pilot test are incorporated throughout the report. The report s recommendations section includes new survey instruments for gathering information from end-users (library patrons) and recommends how that would be done. 1

6 Two key conclusions from the study are: First, OBE applied to MINITEX services presents some challenges. They arise from the origins of OBE, which concerns the review of programs, 1 rather than services. 2 MINITEX ILL, Delivery, and Reference Referral and Training are services. In this instance, however, an approximation of OBE was used for the development of the pilot test surveys and the final proposed end-user surveys. The approach in these surveys has been to incorporate outcomes -type indicators along with customer satisfaction indicators. This combination of factors provides a reasonable and appropriate perspective for evaluation of the services. It is important to recognize, too, that the principal basis for performance evaluation that has historically been used by MINITEX for its services cost and efficiency measures will and should continue to play a central role in evaluation. Additionally, performance feedback information from libraries and library regional and multitype systems also comprises an important component of MINITEX performance evaluation. Second, MINITEX s visibility with library patrons must increase for outcome-based evaluation for end-users to succeed. Historically, MINITEX has been what might be called a silent partner of libraries in providing the four services. MINITEX has been built on the idea of supporting libraries, but not necessarily taking steps to make its role known to library patrons. For end-user OBE to work, that must change. Partnering with libraries and regional/multitype systems toward that objective will be important to achieving the result. This project has engaged members of the Minnesota library community as well as very knowledgeable people at the national level to identify how OBE for end-users can be applied to MINITEX ILL, Delivery, Reference Referral, and Reference Training services. The multi-purpose pilot test, including first-generation draft user surveys for evaluation, has provided the basis for recommendations of a new set of end-user surveys and methods to gather the information. These instruments, too, will be refined as returned data informs how best to measure the performance of these services. Finally, MINITEX efforts to increase community and state awareness and understanding of the contribution of these services to library systems may lead to identification and means to measure end-user outcomes at the levels of community and state. This would be a recognition that MINITEX services can be evaluated not only at the transaction level with end-users but also at the system levels of benefits to communities, regions, and the state. 1 The IMLS describes program as a series of services or activities with a clear beginning and distinct end leading toward a distinct and measurable outcome (meaningful change in skills, behaviors, conditions, knowledge, or attitudes). 2 The IMLS describes service as a component of a program or an activity within a program. 2

7 Introduction The MINITEX Library Information Network (MINITEX) is a program of the Minnesota Higher Education Services Office with a central office located on the Twin Cities Campus of the University of Minnesota. MINITEX is a publicly supported network of participating academic, public, state government, high school, and other types of libraries in Minnesota and affiliated libraries in North Dakota, South Dakota, and Wisconsin. MINITEX provides a range of statewide services to enhance the efficiency and effectiveness of participating libraries. These services represent a major component of the infrastructure for resource sharing among libraries throughout the state, enabled by interagency agreements with the state library agency (the Minnesota Department of Education, through its Library Development and Services unit). An essential feature of MINITEX services is providing access for all Minnesota libraries to the rich collections of the University of Minnesota s libraries. This service was an original purpose of MINITEX s creation. A complete service list and descriptions are shown at Among the services are the state s backbone interlibrary loan and delivery services and the reference referral (or backup reference ) and reference training services. These four service areas, and specifically the application of outcome-based evaluation to them, are the concern of this project. Here are a few basic definitions and descriptions: MINITEX Interlibrary Loan (also called Document Delivery) is the set of services through which books, journals, video and audio recordings, and other library materials are supplied to library patrons in response to requests received from libraries in the network. Interlibrary loan (ILL) is also a general term for all such services between libraries, whether or not MINITEX is a part of the service. In this report, ILL refers to the MINITEX ILL services. ILL services do not include the physical delivery (see Delivery below), except they do include electronic document delivery (MINITEX Electronic Document Delivery, or MEDD). MINITEX Delivery is the set of services through which library materials are delivered to libraries in the network and from there to library patrons. The delivery sites are academic, public, and state government libraries, depending upon the libraries local and regional resource sharing protocols. Contract commercial couriers make most deliveries. Other delivery methods are also used, including the U.S. Postal Service, United Parcel Service, fax, and digital transmission via Internet and . MINITEX Reference Referral is the reference service for reference librarians who have exhausted local resources in attempting to provide information to patrons. The librarian can forward reference questions to MINITEX Reference Services staff for additional research. The MINITEX response is returned to the referring librarian to be delivered to the patron. 3

8 MINITEX Reference Training is a statewide source of reference training for librarians. MINITEX Reference Services staff provide information and skills training in-person, over the Internet (Webinars), and through conferences and a newsletter. Training topics include the Electronic Library for Minnesota resources (ELM is a large collection of resources in electronic format that is available without cost to all Minnesotans), and others. Resource sharing among libraries and efficient, cost-effective services through which patrons can access information from beyond the local library are fundamental to the operation of libraries in the state. Notwithstanding the average person s increasing use of online information, the need for efficient access, cost-effective transport, and high quality reference services remain central functions of libraries and library systems. Funding and Outcome-Based Evaluation Funding for MINITEX services includes federal, state, and local sources. The Institute of Museum and Library Services (IMLS), a federal agency, provides library grants through the state library agency, the Minnesota Department of Education, which fund MINITEX services. Most prominent for the current project is the federal funding provided through Library Services and Technology Act (LSTA) grants. Additional funding comes from the Minnesota Legislature, through programs administered by the Higher Education Services Office. The funding agencies, notably the IMLS, have directed MINITEX and other library grantees to place greater emphasis on identifying and measuring the benefits to endusers of the services they provide. The IMLS defines the end-user as the library patron that is, persons who receive library services. In particular, the IMLS has in recent years directed the use of outcome-based evaluation (OBE) to measure the end-user benefits from funded services. Outcome-based evaluation presents the question, What has changed as a result of our work? as a substitute for the output-based question, What have we done to accomplish our goals? 3 OBE presents analytical challenges for interlibrary loan, delivery, and reference services. For example, although for most services provided by libraries the end-user is clearly and obviously the library patron, these MINITEX services have historically been considered directed to libraries and library staff members as the users. The libraries, in turn, serve their patrons. Consistent with that framework, MINITEX services have been highly visible to libraries but much less visible to library patrons. MINITEX s philosophy has been that it supports libraries to enhance their services to library patrons, largely leaving credit for services to local libraries that provide direct services to end-users. 3 Perspectives on Outcome Based Evaluation for Libraries and Museums, Institute for Museum and Library Services, p. 3. 4

9 Project purpose and approach The project conducted by Management Analysis had the purpose to assist the MINITEX staff to: document interlibrary loan, delivery, and reference service components and expected changes in services based on foreseen trends and developments, identify the types of end-user benefits and outcomes that would pertain to MINITEX interlibrary loan, delivery, and reference services, develop, test, and refine new tools to identify and measure end-user outcomes, and recommend tools and methods to gather this evaluative information on an ongoing basis to address the directives of funding agencies. The Management Analysis project team s approach was to: research current practices among similar agencies and libraries; interview and survey a sample of knowledgeable library staff around the state, MINITEX staff, and other experts at the national level; develop, test, and refine data-gathering instruments (including three instruments for end-users of services and four instruments for library staff users of MINITEX services) through interaction with selected MINITEX clients; and propose a plan for future data gathering (practices and reporting framework) in the mode of outcome-based evaluation for MINITEX interlibrary loan, delivery, reference referral and reference training services. The pilot test of a survey instrument developed during the study represented a significant source of new information for the study. Information developed from the pilot test is incorporated throughout the report. The pilot test survey and names of the pilot test group are contained in the Appendix. The proposed next-generation survey instruments for end-user outcomes information development are contained in the Conclusions and Recommendations section. Acknowledgements The MINITEX staff and many people from the library community contributed to this report. A list of those who participated through interviews and/or participation in the pilot test of questionnaires is contained in the Appendix. The assistance to this project from each of these people is greatly appreciated. 5

10 MINITEX Interlibrary Loan, Delivery, and Reference Services The MINITEX Library Information Network provides library support services for Minnesota libraries. Its geographical scope is shown in the map below a network with more than 2,000 4 participating academic, public, state government, high school, and other types of libraries in Minnesota and affiliated libraries in North Dakota, South Dakota, and Wisconsin. MINITEX Interlibrary Loan, Delivery, Reference Referral, and Reference Training services are being provided in the increasingly complex environment of modern library operations. MINITEX services and practices are evolving with innovations in support technologies while maintaining a fit within the variety of structures and practices of Minnesota s local libraries, consortia, 5 and regional library networks (which include regional public library systems 6 and multitype library systems 7 ). The types of libraries 4 MINITEX includes as participants 159 academic, 179 public, 98 special, and 1,731 K-12 school libraries in Minnesota, North Dakota, and South Dakota. 5 A library consortium is any local, regional, or statewide cooperative association of library entities that provides systems and coordination of resources for public, academic, school, and special libraries. 6 Regional public library system: defined in Minnesota Statutes as a multi-county public library service agency that provides free access to all residents of the region without discrimination, and is organized under the provisions of this chapter or chapter 317A, or section In Minnesota, there are six consolidated and six federated systems. A consolidated system has one centralized administrative unit and board. Federated systems were organized where public libraries already existed; each participating library maintains local and financial administrative autonomy. 7 Multitype library system: defined in Minnesota Statutes as a cooperative network composed of any combination of public libraries, regional public library systems, public school libraries, public or private 6

11 include public, 8 academic, 9 special, 10 and school/media center. 11 Major advances in computer technologies, including the Internet, have increased the capabilities of libraries to provide electronic services and changed how many basic services are delivered. Some of the same technologies have empowered people to access much more information themselves and simultaneously have led to higher expectations of what libraries should be able to provide. Improving access to information/resource sharing Resource sharing is an important element of effectiveness for individual libraries and library systems, with benefits ultimately flowing to library patrons. The services that support this resource sharing among libraries are in a mode of continual change. These ideas are referenced in the state library administrative agency s most recent five-year plan for federal funding: Library patrons in Minnesota highly value the ability to order books and utilize resources from other libraries and library services. To fill this need, Minnesota has been diligently working over the past years to facilitate resource sharing among libraries and reduce the costs of providing access to information for its residents. 12 From its beginnings as a pilot project in 1969, MINITEX has been providing services that promote resource sharing among libraries and access to information for library patrons. Many trends and developments that affect the future of library support services. In addition, for this study, MINITEX staff and other knowledgeable library personnel at the local, regional, and national levels were interviewed and surveyed to help identify trends and developments that will affect MINITEX Interlibrary Loan, Delivery, Reference Referral, and Reference Training services. Also for this study, the pilot test survey responders were asked to identify the trends and developments that would increase or decrease future use of the four MINITEX services. Overall, the expected trends and developments include new applications of technology to college or university libraries and any other libraries, which share services and resources within a multicounty area. 8 Public libraries are established under Minn. Stat. Ch The must provide free access to all residents of city or county without discrimination. They receive at least half of their financial support from public funds. They must have an organized collection of library materials, trained staff to meet the needs of their clientele, an established schedule of services, and funding and facilities to support the collection and services. They must be legally established by statute, resolution, or ordinance and be a member of a regional public library system. 9 Academic libraries are those which are integral parts of a college, university, or other academic institution for post-secondary education, organized to meet the learning and research needs of students, faculty, and staff of the institution. 10 Special libraries are those supported by government agencies, private corporations or businesses, associations, or other special interest groups to meet the information needs of the staff or members in pursuing the organizations goals. 11 School libraries/media centers are libraries or spaces in elementary, secondary, or combined schools where a collection consisting of a full range of media is available to support the curricular needs, with associated equipment and technology. School library/media center staff are available to students, teachers and school staff. 12 State of Minnesota, LSTA Five-Year Plan

12 enhance and even transform services, integration of service capabilities on a larger scale (including greater collaboration and resource sharing), and more unmediated services 13 all of which are redefining aspects of library services as well as MINITEX services. After presenting an overview of trends in relation to MINITEX services overall and recent innovations by MINITEX in response to those trends and developments, each of the four MINITEX services will be reviewed more fully in terms of current services, trends and developments, and some implications for the future. Some service changes may have implications for how services are evaluated in the future, including for the application of outcome-based evaluation to MINITEX services. Trends and developments The trends and developments that are changing libraries and library systems are also having impacts on MINITEX services. Conversely, the evolution of MINITEX services has important effects on the state s libraries and library systems. Among the major environmental changes that impact MINITEX services are technology innovations, the changing demographics of Minnesota s population, and changes in the availability of funding at the local, regional, and state levels. Technology. The overwhelming changes resulting from new and improved technologies available to library patrons and to libraries have revolutionized the way many library services are delivered. The Internet and World Wide Web have dramatically changed service delivery for libraries, making online transactions usual and providing direct access to a large and increasing range of online resources. Access from home to many library services has changed some of the fundamental dynamics of information access for interlibrary loan and for reference services, as examples. Library patrons expectations of services have coincidentally risen dramatically with the rise of common use of the new technologies. A complete description of technologies implications for library operations and for library support services is hardly possible, and the future implications are probably unknowable. One example is MnLINK. MnLINK is described as a statewide virtual library that electronically links Minnesota s major libraries. It provides a statewide virtual integration layer that enhances the current resource-sharing network (local libraries, systems for retrieval and delivery, electronic resources, and many more elements) of which MINITEX services are a major component. The MnLINK project consists of two major components: the Gateway (Fretwell-Downing software) and the Integrated Library System (Ex Libris software). The Gateway software will facilitate access to a large number and variety of library resources including library catalogs and the Electronic Library for Minnesota (ELM). The Integrated Library System (ILS) will be a shared library automation system for the Minnesota State Colleges and Universities system (replacing PALS), the University of Minnesota system (replacing NOTIS), state government agencies, several private 13 Unmediated services are those in which users help themselves for example, to initiate their own requests for interlibrary loan or to place their requests directly with the loaning library. In contrast, a mediated service occurs when a library staff member assists the patrons for example, to place or review the interlibrary loan request. 8

13 colleges, K-12 schools, public libraries, and special libraries. Similarly, North Dakota and South Dakota libraries that participate in MINITEX are moving to Ex Libris software (replacing PALS). As will be noted in detail below with respect to individual services, the MnLINK system ( will have an impact on MINITEX services. Service integration. MnLINK is also an example of the movements in process to integrate library service standards, protocols, and other mechanisms for information sharing and service provision that are the subjects of ongoing collaboration efforts of the library entities, including MINITEX. Self-help by library patrons. The phenomenon sometimes called self-help or unmediated services is working in some parts of library operations and support services. For example, library Websites commonly enable patrons to find the latest updates about their libraries, check if materials are available, reserve materials, renew materials, and even ask reference questions online. The innovations that bring direct services to patrons represent a strong and continuing trend. Demographics. Minnesota s population is growing in diversity and the population average age is increasing. The challenges of serving older, non-english speaking, and other disadvantaged library users reflected in these trends places increasing pressure on libraries to adapt to the specialized needs of underserved populations. Population distribution, ethnic diversity, and other population trends will affect how libraries and library support services such as those of MINITEX respond to the basic mission of free access to information without limitations due to library patrons location or special needs. Cost-effectiveness and accountability. In the same manner that other agencies of government are under pressures to reduce costs and improve efficiencies, libraries and the agencies that provide library support services are being directed to provide services more cost-effectively and to account for how money is being spent. The broad efforts, including this one introduced by the IMLS to identify and measure outcomes for endusers, reflect that those who provide funding must document that funds are well spent. MINITEX service developments In response to trends and developments in the library environment, MINITEX has introduced a number of significant service innovations that are related to and have implications for interlibrary loan, delivery, and reference services. They include: Electronic resources. MINITEX initiated and continues to maintain and develop the Electronic Library for Minnesota (ELM) 14 with funding from the Legislature and a federal LSTA grant. ELM is a large collection of electronic resources and books available without patron cost to all Minnesotans. Components include the Gale Periodical/ Reference Package, the OCLC Collection, netlibrary (MINITEX e-books), Gale s Consulta and Informe! (Spanish language databases), and ProQuest Newsstand Complete. 14 For more information, see 9

14 Cooperative purchasing agreements. MINITEX helps libraries in Minnesota, North Dakota, and South Dakota to acquire library products and electronic resources at largegroup pricing by negotiating cooperative purchasing agreements. Many of the costly electronic resources not included in ELM are available at less cost than would otherwise be the case for individual libraries or smaller library purchasing groups. Electronic delivery of articles. The MINITEX Electronic Document Delivery service (MEDD) was recently expanded to all MINITEX member libraries. MEDD makes serial articles available to patrons online, often within hours of their request. Articles are scanned into electronic format and an message is sent to the library patron, who can retrieve the article from a MINITEX server. After five retrievals or 14 days, whichever comes first, the item is deleted from the server. MINITEX is one of the pioneers in using this new technology. Delivery system enhancements. MINITEX Delivery service piloted an additional work shift to improve timeliness on overnight courier deliveries, and on a regular basis initiates other service innovations. MINITEX Delivery has added links to the courier route as the needs arise and improved efficient movements between libraries (expanding direct delivery). Reference services enhancements. Reference services were expanded to all Minnesota libraries, including K-12 school libraries. The monthly newsletter, Reference Notes, is now sent to all Minnesota academic, state government, and public libraries; K-12 libraries/media centers receive an message with link to Reference Notes on the MINITEX Website. Reference training (including Webinars) has expanded, particularly to enable library staff around the state to make use of the ELM collections. Additional information about the four MINITEX services the intended subjects of the outcomes-based evaluation will inform the types of information required for the effort. MINITEX Interlibrary Loan (Document Delivery) MINITEX Interlibrary Loan 15 (also called Document Delivery by MINITEX) 16 is the set of services through which books, journals, and other library materials are supplied to library patrons in response to requests received from libraries in the MINITEX network (Minnesota, North Dakota, and South Dakota). MINITEX staff members retrieve materials from University of Minnesota Twin Cities Campus libraries or refer the requests to other libraries in the network that own the item. Most materials loaned or borrowed through ILL are transported by the MINITEX Delivery service. Articles can be transmitted electronically directly to the patron's desktop or to the patron's library through electronic document delivery. 17 MINITEX ILL also provides related training and sponsors conferences. 15 In the generic use of the term, interlibrary loan is a library-to-library request for the loan or delivery of library materials by a library outside the local library s administrative unit. The purpose is to obtain library material not available in the local library. Interlibrary loan is the general term and includes all such services between libraries, including those through MINITEX ILL. 16 For more information, see 17 MINITEX Electronic Document Delivery (MEDD). 10

15 Trends and Developments As noted in a comprehensive study of the state interlibrary loan system, 18 the MnLINK system will likely have a significant impact on interlibrary loan across the state: Minnesota is replacing its two major online systems, PALS [Minnesota State Colleges and Universities system] and LUMINA [University of Minnesota system], with a comprehensive statewide system, MnLINK, which through a centralized file and designated gateways will connect library resources across the state. Planners expect MnLINK to have an enormous impact on interlibrary loan traffic and patterns. The changes resulting from MnLINK are envisioned as increasing the resource sharing potential among libraries and library systems. The changes will enhance and expand the capabilities of libraries to efficiently request materials from each other directly and of library patrons to initiate their own requests. The change for MINITEX Interlibrary Loan services may be mixed. There might be a reduction in some kinds of requests coming through the system but there might be an increase due to easier access to the University of Minnesota s Twin Cities campus collections. When the MnLINK components are in place, the degree of change for MINITEX will depend in part on the breadth and pace of adoption of MnLINK use. MINITEX Interlibrary Loan is also influenced by the availability of electronic resources that can be accessed directly. Currently, Minnesotans can obtain over 4,000 full-text journals online. Requests for photocopies of journal articles decline when readily accessible electronic resources provide the same information. In the survey sent to the pilot test group for this study, responders were asked to identify the trends and developments that they believed would increase or decrease future use of the four MINITEX services. For MINITEX Interlibrary Loan, they identified these factors: Factors tending to increase use Need for access to strong collections at the University Decreasing budgets for collections/materials Quantity of information available to people increasing Online databases are generally not full-text Collections coordination within a region Increased familiarity in high schools with service Online databases come and go depending on budget Electronic Library for Minnesota (ELM) access needed Electronic document delivery (MEDD) used more Distance education Factors tending to decrease use Online access to information The factors listed by responders denote the types of need that would support future ILL services. What is not indicated is whether the sum of the increase factors weighed against the decrease factor would result in more or less service need overall. In this snapshot of factors, the conclusion seems appropriate that more ILL resource sharing for University collections, funding uncertainties for local libraries, including the 18 Himmel and Wilson, Minnesota Interlibrary Loan Study (1999). 11

16 incorporation of electronic resources, limitations of current online databases, and other factors, are likely to support significant future ILL services. MINITEX Delivery MINITEX Delivery 19 is the set of services through which library materials are physically delivered to libraries and library patrons throughout the region (MN, ND, SD, WI). MINITEX staff and local library staff aggregate shipments into standard containers for movement among the delivery system s nodes. The delivery sites are academic, public libraries, and state government libraries, depending upon the libraries local and regional resource sharing protocols. Frequency of MINITEX delivery is five days per week; delivery frequency within local systems varies around the state. Contract commercial couriers make most deliveries. Other delivery methods include the U.S. Postal Service, United Parcel Service, fax, and digital transmission via Internet and . Trends and Developments The MnLINK system will provide access to the catalogs of nearly all Minnesota libraries. When fully implemented, it also will allow patrons of Minnesota libraries to request books or other materials owned by libraries throughout the state. The MINITEX Delivery service continues to be the primary method for moving requested items between major libraries throughout the state. As library patrons increasingly use the MnLINK Gateway to directly access library materials, overall traffic may increase and delivery requests would likewise increase. Some people interviewed for this study have concluded that the number of delivery requests could increase dramatically. Changes to local delivery that is, among libraries within a region or smaller geographic area will likely occur, both due to funding reductions and MnLINK changes. MINITEX Delivery, in consultation with local libraries, could initiate pilot projects to help improve local delivery. Responders to the pilot survey for this study identified trends and developments they believed would increase or decrease future use of the Delivery services. They identified these factors: Factors tending to increase use Online catalog/direct borrowing Aleph Gateway enhancements implemented More access to technology finding tools Online tools conflicts between authors and publishers Patron-initiated, unmediated requests using MnLINK Distance education Factors tending to decrease use Electronic document delivery Online (Web) access to information As previously noted, the comparative lengths of the increase and decrease lists do not signify which of the directions would have the stronger overall effect. However, many of the factors identified would lead to the conclusion that delivery services are likely to become more important in the foreseeable future. 19 For more information, see 12

17 MINITEX Reference Referral MINITEX Reference Referral 20 is the reference service for library staff that have exhausted local resources to provide information to patrons. When a reference librarian cannot answer a patron s information request because of limited local resources, insufficient specialized expertise, or access to special collections and online information, the librarian can forward the questions to MINITEX reference referral for additional research. The MINITEX response is returned to the referring librarian to be delivered to the patron. The types of questions forwarded to reference referral vary greatly but have included student research, genealogy, business, technical, high-level professional, and some esoteric topics. Requests are usually more difficult and/or time-consuming than usual library reference questions, rely on University of Minnesota resource materials, or require high-level reference research skills. Trends and Developments Reference referral is receiving fewer but more complex and obscure questions. This trend is occurring at least in part because patrons and local library staff can often resolve easier questions using the Internet. Among the available Internet resources are the statewide electronic resources (ELM) and databases licensed by MINITEX for use by all Minnesota residents. Previously, this type of access to great quantities of information was much more limited, as were the skills to search for it. In what might be a noteworthy but small countertrend, more researchers may find that the quality of information retrieved through unmediated online resources is unsatisfactory, with the result that they contact reference services more often. However, clearly the trend is for more users to access information using their own resources for an increasing number of questions. This trend will almost certainly continue or accelerate. A major development in reference services is virtual reference or digital reference. Virtual reference services are Internet-based question-and-answer services that connect users with experts and subject-matter expertise. The Internet connects people who have questions with experts who can answer them. The online expert also can help the person develop information-finding skills. The import of virtual reference is that it practically erases the barriers of time and distance to obtain reference services. Before the Internet, librarians essentially had no choice but to rely on mail, phone, and fax to help users who were unable to visit the library for reference assistance. Internet-based tools give librarians a greatly expanded set of communication options. Digital reference technologies may be asynchronous (where the user and librarian exchange messages) or synchronous (where the user and librarian use digital chat, simulating a live interaction). In the not too distant future, Minnesota librarians and information seekers could speak directly to each other or conduct a combination search and training session together using streaming video. Many limited-range (that is, available to patrons of specific libraries) digital reference services are in use including in some of the larger Minnesota libraries (Hennepin, 20 For more information, see 13

18 Ramsey, and Washington County Ask a Librarian services, for example). Many commercial services provide digital reference for a fee (including the Google Answers service, as one example of a service available to anyone with an Internet connection). The new reference technologies raise questions for reference librarians and for MINITEX reference referral. For example, what design for new collaborations will make this work on a statewide scale, especially with user expectations for around-the-clock ( 24/7 ) access to reference services? Who will develop such a collaborative network and its protocols and standards? What role does the MINITEX reference referral service have in the networked environment of virtual reference? Who will pay for this? These types of questions are being addressed in Minnesota s library community, with the active participation of MINITEX staff. MINITEX Reference Services has been involved in digital reference issues. On behalf of the Minnesota Collaborative Digital Reference Project (MCDRP), MINITEX in April 2003 hosted a web-based survey concerning virtual reference services. About 425 responses were received from staff from all types of libraries and school media centers across Minnesota, North Dakota, and South Dakota. Among the findings and conclusions were that many libraries view collaborative virtual reference services as a way to extend patron services, overcome limited staffing and budget reductions, and possibly reduce software costs. In the pilot test survey, responders identified trends and developments that they believed would increase or decrease future use of MINITEX Reference Referral services. They identified these factors: Factors tending to increase use Factors tending to decrease use Materials budget down so less access to local resources Internet access for staff and users Unable to subscribe locally to many online resources More use of online databases Richer resources usually available from the University Virtual/digital reference Virtual/digital reference Overall less use of reference services Reference technologies provide more options Review of the increase and decrease lists makes it difficult to foresee what effect these factors collectively will have on reference services and MINITEX Reference Referral. In particular, the effect of virtual reference services on the future of reference services is especially unclear. It seems apparent, however, that the level of expertise of reference librarians at the MINITEX level where the most specialized, difficult, and/or obscure questions are presented will have continued durability. The potential roles and level of effort appear to be somewhat open-ended questions. It may turn out that the most desirable future design of library-based digital reference services would expand the role for MINITEX Reference Referral in new roles involving collaboration and resource coordination. 14

19 MINITEX Reference Training MINITEX Reference Training 21 is a source of reference services training for librarians in the state. MINITEX staff members provide reference skills training in-person, over the Internet (Webinars), and through conferences and a newsletter. Training topics include the Electronic Library for Minnesota (ELM) resources, MN Opportunities for Reference Excellence, and many other reference skills topics. The MINITEX staff seeks to upgrade the knowledge of reference issues and the specific skills of those who take training or who receive the newsletters and attend reference-focused conferences. Trends and developments Reference training is becoming a larger component of MINITEX Reference Services due to the large increase in electronic services available to libraries and library patrons. 22 Library staff need to keep up with the skills needed to make best use of the new resources, including the Electronic Library for Minnesota and other electronic resources. The broad need to increase library staff familiarity with, and skills to use electronic resources are likely to be continuing trends. Innovative training tools that make use of the Internet to deliver training Webinars are well received and greatly leverage MINITEX staff resources. The innovative use of online tools to deliver training (Webinars) and broadly upgrade library staff skills (newsletter) is likely to be a continuing trend. A potential extension of some MINITEX Reference Training services might be to provide online training (such as abbreviated or targeted Webinars) to library patrons (the end-users of library services). Some of the topics that would be valuable to patrons and appropriate for them may already be in use. Leveraging already available MINITEX training resources to assist some library patrons (notably serious researchers), in addition to the more comprehensive and in-depth resources for librarians, may be a trend for the future. Whether the MINITEX mission and funding requirements would support such an expansion of training services would have to be explored. Pilot test survey responders identified trends and developments that they believed would either increase or decrease future use of MINITEX Reference Training services. They identified these factors: Factors tending to increase use Webinars (online/alternative format delivery) Fewer local resources for staff training Staff turnover due to retirements New electronic tools require training to use effectively Potential for end-user training options Factors tending to decrease use Fewer resources to pay for training Staff time and costs limit attendance The relative importance of increase and decrease list items are difficult to judge. However, it is clear that electronic resources will become more important to libraries and 21 For more information, see 22 MINITEX Reference Services does not charge a fee for ELM or reference training at this time. 15

20 beyond that (through library and other Websites) to end-users of information resources. Library staff (and end-users) will benefit greatly by having the knowledge and skills to make maximum use of online resources. Particularly when the online information resources are contracted for and maintained by MINITEX for use statewide, the rationale for providing good training services on a statewide basis is clear. The collections change often, the terms and limitations of use change, and the tools for access can be honed to improve speed and quality of results. As a result, the need seems to be clear for broadly available, high quality training to best use online information resources and for other reference skills and awareness training for library staff. Conclusions and implications for performance evaluation From this review, several conclusions may be made about expected directions for MINITEX services that may have implications for performance measurement: Innovating to make all services continually more cost effective Service cost effectiveness will be increasingly important. Measures that show improvements and the benefits of innovations will be needed. Providing more services that leverage the skills and time of library staff resources Support services to libraries will increasingly be seen as leveraging local resources. Measures or indicators that document this benefit will be useful. Ensuring that library patrons needs as well as library needs figure prominently in the design and redesign of services New technologies increasingly put end-users in the position of self-help. Measures that show how services support end-users directly as well as library staff services will be needed. Making best use of online resources for service delivery Electronic services can dramatically reduce transaction time and eliminate distance considerations in delivering some services (MEDD is an example). Measures that show expansion of these services and demonstrate how end-users benefit will be increasingly useful. Promoting integration, partnering, and collaboration among libraries and systems Service integration enables more efficient and low-cost access to library resources and resource sharing. Improving the mechanisms of collaboration and measures of their impacts will be increasingly important. The future direction of performance measurement can include both internal measures and external measures that monitor these dimensions of MINITEX service effectiveness. 16

21 Outcome-Based Evaluation Outcome based evaluation arose in the human services environment, where clientfocused outcome statements describe how clients lives would improve based on actions taken. The question asked is, What difference will this program/initiative make in the lives of those served? 23 The types of outcomes may be changes in circumstances, quality of life or functioning, attitude or behavior, knowledge, and skills. The Government Performance and Results Act of 1993 (P.L ) directs that federal agencies will examine the performance of their programs in terms of the outcomes achieved. The U.S. General Accounting Office provided guidelines to federal agencies to implement the GPRA: 24 The recommended steps are: Define mission and desired outcomes Involve stakeholders Assess the environment Align activities, core processes, and resources Measure performance Produce measures at each organizational level that: demonstrate results, are limited to the vital few, respond to multiple priorities, and link to responsible programs Collect data Use performance information Identify performance gaps Report information Use information Reinforce GPRA implementation Devolve decision making with accountability Create incentives Build expertise Integrate management reforms The Institute of Museum and Library Services (a chief funding agency for MINITEX ILL, Delivery, and Reference services) provides additional guidance for the use of outcome-based evaluation for recipients of its LSTA (Library Services and Technology Act) grants. The agency specifies that the end-user or library patron is the intended beneficiary of programs and services. Therefore, outcomes for end-users are the appropriate measures of performance. Libraries must measure changes or gains in individuals (end-users) skills, knowledge, attitudes, behaviors, life condition, or status that results from programs and services that are LSTA-funded. 23 Outcome Evaluation: Determining Project Outcomes, W.K. Kellogg Foundation Evaluation Handbook, p U.S. General Accounting Office, Executive Guide: Effectively Implementing the Government Performance and Results Act (Washington, D.C. 1996), p