DeGroote.indd Quantifying Cooperation: Collaborative Digital Reference Service in the Large Academic Library Sandra L. De Groote, Josephine L. Dorsch, Scott Collard, and Carol Scherrer The purpose of this study was to determine how successfully a large academic library with multiple reference departments and subject spe- cialties could combine virtually to create one digital reference service. Questions were coded to determine who the users of the service were, the types of questions being asked, and the subject expertise of the librarian answering the question. The study found that the majority of questions were submitted by persons affiliated with the university, that ready reference and directional questions predominated, and that the librarians were able to successfully share the duty of answering the general reference questions while ensuring that the questions requiring subject expertise were answered by the appropriate subject specialists. Analysis of the types of questions will inform future decisions regarding webpage redesign, online instruction needs, and more appropriate FAQs (frequently asked questions.) ver the past several years, digital reference has become an integral part of the services offered by academic librar- ies. The use of networked information systems to deliver asynchronous help to library patrons via e-mail has allowed librarians to reach their users at this new virtual “point of need,” and the intro- duction of chat technology has allowed an even greater level of interactivity by providing real-time, on-demand service. However, these new capabilities have ne- cessitated a certain amount of adjustment in academic libraries, and the need for this adjustment points up the importance of assessment and evaluation in the plan- ning, implementation, and provision of digital services. In March 2003, the University of Illi- nois at Chicago (UIC) library undertook an overhaul and reconfiguration of its public services activities, particularly those services offered online. This proj- Sandra L. De Groote is an Assistant Information Services Librarian, Library of the Health Sciences, Uni- versity of Illinois at Chicago; e-mail: sgroote@uic.edu. Josephine L. Dorsch is the Health Sciences Librarian at the Library of the Health Sciences – Peoria, University of Illinois at Chicago; e-mail: jod@uic.edu. Sco A. Collard is Librarian for Psychology, Education, and Linguistics, New York University; e-mail: sco . collard@nyu.edu. Carol Scherrer is the Information Services Librarian, Library of the Health Sciences, University of Illinois at Chicago; e-mail: cscherre@uic.edu. 436 mailto:cscherre@uic.edu mailto:collard@nyu.edu mailto:jod@uic.edu mailto:sgroote@uic.edu Quantifying Cooperation 437 ect examined the interrelationships among digital services librarywide, focusing especially on the provision of reference and information services in the digital environment. The task force charged with this examination was in- structed to investigate and implement synchronous online reference services (chat reference) for the entire library and to make necessary organizational adjustments to support this new ser- v i c e . T h o u g h i m p l e m e n t a t i o n o f a digital reference service option was not extraordinary, the UIC library’s situa- tion encompassed a number of specific challenges that quickly became apparent as the project developed. The first and most obvious challenge was that UIC is a highly subject- and geographically dispersed organization, composed of the Daley Library, which serves liberal arts and sciences, visual arts and architec- ture, and government documents; the Science Library, which serves the physi- cal sciences; and four Health Sciences Libraries (LHS) located in four differ- ent cities throughout Illinois (Chicago, Peoria, Rockford, and Urbana), which serve the medical school communities. In implementing the service, the task force would need to consider carefully the impact it would have on diverse patron bodies as well as the staff who would assert primary control over the services. Any solution would need to take into account the different levels of staffing, a diversity of reference styles, and a need for a high degree of flex- ibility and autonomy in the scheduling and provision of service, in addition to the diversity of subject disciplines and the physical dispersion of the library. The implementation team eventually selected OCLC’s QuestionPoint prod- uct because it satisfied many of these requirements, particularly in the flex- ibility it showed in creating self-defined collaborative units, the ability to “move” questions around the system easily and efficiently, and the question-logging ca- pabilities inherent in the product. Background and Purpose The investigation led the task force to conclude that provision of this new service would be best achieved through a significant reorientation of existing reference service. Traditionally, e-mail reference had been provided and handled by the individual reference departments. A Web form was available where the pa- tron would select the library/department expected to answer the question. (See figure 1.) Anecdotal evidence suggested that patrons were not always successful in submi ing their questions to the appro- priate library or department. Rather than maintain separate and distinct services at UIC’s numerous locations, the digital reference management system was in- troduced to centralize the location where questions were submi ed and viewed. This centralized digital reference man- agement system created a shi toward a collaborative, shared digital reference model that would take advantage of staff- ing and subject expertise available across the library. Further, instead of relying on patrons to determine where their inqui- ries should be sent, reference librarians would make the selection, thus providing one simple and seamless patron interface to digital reference services at UIC. Lastly, with a single digital reference system, personnel at each library site would be shared more efficiently across all sites. However, moving from the previous model to this integrated approach would prove difficult because the shi would be made simultaneously for all reference units of the library and would include thirty-five librarians at all library loca- tions. It would be particularly challeng- ing to create a means of assessment that would enable the implementation team to measure the success of the shi and that also would be quantifiable, statisti- cally relevant, and reproducible at future intervals. In essence, the task force wanted to explore whether a single service could replace multiple services that were previously disparate, highly discipline specific, and independently controlled 438 College & Research Libraries by dispersed library departments and sites. Further, the task force wanted to investigate the possibility of measuring the relative success of such a venture by quantifying data inherent in the questions September 2005 themselves, on the basis of user status, subject area, answering processes, and a host of other criteria. Moreover, this proj- ect would measure a multitude of other factors that could be taken into account FIGURE 1 Ask a Reference Question Quantifying Cooperation 439 TA B L E 1 St ud ie s E xa m in in g Ty pe s of Q ue st io ns A sk ed in D ig it al R ef er en ce S er vi ce s A ut ho rs Ti m e # of ? s Ty pe o f L ib ra ry C od in g R es ul ts D ia m on d & P ea se (2 00 1) 2. 5 yr s 45 0 e- m ai ls A ca de m ic : Si ng le li br ar y Q ue st io ns an sw er ed u s- in g st an da rd re fe re nc e re so ur ce s 22 % St ar tin g po in ts fo r te rm p ap er s/ as si gn m en ts 17 % Sp ec ifi c- fa ct ua l (n on re ad y re fe re nc e) 12 % C at al og lo ok -u p an d us e 11 % L ib ra ry po lic ie s, pr oc ed ur es , sc op e of c ol - le ct io n 10 % N av ig at io n O pa c 8% N on - lib ra ry qu es tio ns 6% In fo rm a- tio n lit er ac y (i ns tr uc tio n re qu ir ed ) 6 % C on ne c- tiv ity 5 % D at ab as e m ec ha ni cs 2% N on - qu es tio ns 2% K ib be e, W ar d, M a (2 00 2) 12 w ee ks 60 4 ch at s U nd er gr ad ac ad em ic : M ul - tip le l ib ra ri es Fi nd in g sp ec ifi c lib ra ry m at er ia ls 3 3. 3% In fo rm at io n ab ou t lib ra ry a nd s er vi ce s 30 .5 % Su bj ec t b as e re se ar ch 20 .2 % R ea dy re fe re nc e 9. 1% Te ch ni ca l p ro bl em 5 .3 % Q ue st io ns a bo ut c ha t se rv ic e 1. 7% Po w el l & B ra di ga n (2 00 1) 5. 5 yr s 21 2 e- m ai ls A ca de m ic : H ea lth s ci en ce s A cc es si ng on lin e se rv ic es 1 4% D at ab as e se ar ch in g 4% St at is tic al In fo 2 % Su bj ec t- ba se d re se ar ch 22 % L ib ra ry se rv ic es 15 % H ol d- in gs in fo 19 % D ir ec to ry in fo 6 % C on su m er he al th 15 % C ita tio n ve ri fi c at io n 4% M ar st el le r & N eu ha us (2 00 1) ch at s A ca de m ic : M ul tip le lib ra ri es Su bj ec t B re ak do w n “O th er ” B re ak do w n – 51 % Sc ie nc es e ng i- ne er 1 1% H um an iti es 6% So ci al s ci - en ce s 7% B us i- ne ss 6% G en er al 1 9% O th er 5 1% Te ch ni - ca l 6 4% O PA C 1 7% IL L 3% C ir cu la tio n 9% C ha t ab ou t c ha t 7% Pa rk er & Jo hn so n (2 00 3) 4 m on th s 58 ch at s 2 A ca de m ic : H ea lth s ci en ce In -d ep th 7 4% A cc es s 24 % L ib ra ry H ol di ng s 17 % Q & A 9 % L ib ra ry p ol ic y 9% Je ra nt & Fi re st ei n (2 00 3) 6 m on th s 10 5 ch at s 2 H ea lth s ci - en ce a ca de m ic & h os pi ta l R es ea rc h & re fe re nc e 49 % B ib lio gr ap hi c m an ag em en t ( e. g. , e nd - no te , r ef er en ce m an ag er ) 4 % G en er al li br ar y in fo rm at io n & p ol ic ie s 23 % L ib ra ry h ol di ng s an d ci ta tio n ve ri fic at io n 24 % C ic co ne & V an Sc oy (2 00 3) C ha t? A ca de m ic : M ul tip le lib ra ri es In st ru ct io n 27 % K no w n ite m 2 5% A bo ut th e lib ra ri es 1 2% E -r es ou rc es 9 % C ir cu la tio n8 % O th er 19 % M ar st el le r & M iz zy (2 00 3) 12 m on th s 86 5 ch at s A ca de m ic : M ul tip le lib ra ri es Te ch ni ca l p ro bl em s 32 % L ib ra ri an co m m un ic at io n 17 % D ir ec tio na l/p ol ic y 17 % K no w n ite m 1 4% Fa ct s/ re ad y re fe re nc e 10 % R ef er en ce 10 % B us ha llo w - W ilb ur e t a l (1 99 6) 18 m on th s 33 8 e- m ai ls A ca de m ic : M ul tip le lib ra ri es B as ic re fe re nc e 74 % L ib ra ry p ol ic y 12 % O PA C 7 % Pu rc ha se re qu es t 7 % 440 College & Research Libraries September 2005 as the service evolved and grew. Some of the areas of inquiry included: • What types of questions would be submi ed to the service and by whom (students, faculty, staff, the public)? • Would examination of questions provide information or feedback that would allow be er service to patrons? • Would the types of questions submi ed through chat differ from the questions submi ed through e-mail? Answers to these and other questions would allow the administrators of the system to fine-tune the functions of the system and steer a future course based on quantified data collected from actual interactions. This study presents the results of the data gathered and discusses their out- comes and meanings. It also documents the types of questions received through the service, the impact on the role of the reference librarian in the digital environ- ment, and the adequacy of the library’s Web services and resources based on frequently asked questions (FAQs). Literature Review The types of questions asked in the online environment are well documented in both the health sciences and social sci- ences library literature. Coding categories used by the various libraries examining their digital reference questions vary as much as the results of the coding. Some studies provided detailed categories for coding digital questions; others examined the questions more generally. Table 1 provides a summary of several studies examining the types of questions asked in the digital library environment.1–9 Several studies documenting the types of questions asked in the digital envi- ronment provide data that suggest one virtual reference desk may be a realistic option. For example, all the studies in the literature reviewed included a category dealing with holdings information. (See table 1.) Questions about library holdings ranged from 7 to 33 percent. Questions dealing with library policies also con- stitute a well-documented category in which any librarian, regardless of specific subject expertise, could probably field the question. Questions about library policies, procedures, and services ranged from 9 to 30 percent. The above findings suggest that the subject expertise of the librarian may not necessarily be required in all cases to answer all questions. Therefore, it would be quite likely that librarians of multiple backgrounds could conceivably share the workload at a combined virtual reference desk, leaving the questions re- quiring subject expertise to be answered by the subject experts. Wendy Diamond and Barbara G. Pease, noting a lack of detail in the literature about the types of questions that libraries were receiving in the digital environment, developed a detailed coding system to analyze the submi ed questions.10 Eleven question categories were created to code the types of questions received. (See table 1.) The questions also were examined for complexity: noncomplex (35%), broad or complex (35%), standard reference (22%), and referrals and nonquestions (8%). Dia- mond and Pease concluded that digital reference questions cover the full range of simple to complicated questions that are similar to questions asked in-person.11 Although this study was one of the first to provide a detailed look at the types of questions asked, it was conducted in a single library. Carol A. Powell and Paula S. Bra- digan also provided an in-depth look at the types of questions submi ed in the digital environment to an academic health sciences library. (See table 1.)12 Twenty-two percent of questions asked by faculty, students, or staff were related to information needed for assignments or patient care, which sometimes resulted in searching databases for citations and abstracts. Patrons requesting holdings information accounted for 19 percent of the questions. Requests for information about library services, such as the policies or procedures, accounted for 15 percent of the questions asked. Consumer health http:in-person.11 http:questions.10 Quantifying Cooperation 441 questions accounted for 15 percent of the questions. None of the studies in table 1 docu- mented the specifics of questions asked related to holdings information. Was the user asking about a book, a journal, or an online journal? Also lacking in the current literature is a direct comparison between questions asked in chat and questions asked through e-mail reference. Further, it should be noted that although many of the studies reviewed provide a thorough examination of the questions asked in the digital environment, they were predomi- nantly studies about questions submi ed to a specific library and not to a central location or where multiple libraries had a shared digital reference service, but the discipline orientation of the libraries was the same. Although two studies did report on shared chat reference services at universities with multiple libraries, the published findings of the studies did not really gauge the value or success of a shared service to serve multidisciplinary user groups. Josephine Z. Kibbee, David Henry Ward, and Wei Ma, who studied the questions submi ed to a shared chat reference service between two different discipline libraries, noted: “Collaboration between two library units in sharing pa- trons and questions added an additional challenge, but ultimately enabled us to extend hours and service.” 13 Ma hew R. Marsteller and Neuhaus also described the types of questions submitted to a shared chat reference service provided by several libraries at one university. 14 Although the study was not concerned primarily with the results of collaboration among multiple libraries, it was noted that librarians did not always have the needed subject background to answer a question. A system was in place for refer- ring questions to “appropriate libraries on-campus, to library departments (e.g., circulation), and to subject specialist librarians.” Also, neither of these shared chat service studies specifically examined FIGURE 2 Web Page for Digital Submission of Reference Questions 442 College & Research Libraries September 2005 the subject expertise of the librarian an- swering the question versus the type of question asked or the subject nature of the questions. Setting UIC is a large urban university with three regional medical campuses. Enrollment totals more than 25,000 students with ap- proximately 5,500 enrolled in the health sciences. The existing form for submi ing reference questions was removed (figure 1), and a Web page was developed for digital submission of reference questions either through e-mail or chat (figure 2). Chat was offered only between the hours of 11 a.m. and 6 p.m., Monday through Friday. No limits were placed on the types of questions that could be asked through the Ask a Librarian service. A link to Ask a Librarian was provided through a tem- plate on all library Web pages. Reference departments also provided a link to the Ask a Librarian page from all depart- mental reference pages. Departments such as circulation, interlibrary loan, special collections, and library systems continued to make the e-mail addresses of these departments available on their Web pages. Guidelines were developed for the provision of reference services in the digital environment prior to their official introduction. With multiple libraries and library departments sharing one service, it was important that the ques- tions continued to be answered by the librarians most qualified. Therefore, one librarian representing one of the four health sciences reference departments and one librarian representing the Daley library (arts and sciences/government documents) were assigned to do e-mail reference each day. (A librarian from the Science Library was available as backup for Daley reference.) The librarian most qualified would answer the question, and those questions that did not require subject expertise could be answered by either of the librarians on duty. If neces- sary, questions also could be assigned to the most appropriate librarian or depart- ment (e.g., science, special collections) if they were not best answered by one of the librarians on duty. With respect to the chat service, a limited number of available librarians meant that only one librarian at a time would monitor the chat service. Be- cause this would increase the likelihood that librarians might encounter a question outside their subject expertise, guidelines were developed to encourage librarians to answer the question as best they could in such instances and to assign the chat tran- script to the appropriate subject expert(s) for follow-up via e-mail. Methodology A coding system was developed through examining a range of sources. Other stud- ies that had coded reference transactions in both the traditional and digital environ- ments were examined to develop a basis for the coding system.15–24 In addition, questions asked of the UIC reference departments were examined to fill in any gaps related to typically asked questions and subject areas. The coding system that was subsequently developed was tested using questions submi ed to the digital reference service in June and July 2003, prior to the official sampling and coding of the questions. Problems and omissions were noted, and the coding system was adjusted as appropriate. The researchers also used the time to train themselves to use the coding system. Inter-rater reli- ability testing was done to ensure that all researchers were defining and coding the information accurately. An inter-rater reliability of 82 percent was achieved among the four coders before the official data collection began. From August 1, 2003, to March 31, 2004, random samplings of 120 ques- tions submi ed (either chat or e-mail) per month to the Ask a Librarian service were examined. The sampling of the questions was determined by using random number generator software. Coded information was entered into a spreadsheet to determine frequency. (See Quantifying Cooperation 443 TABLE 2 Total Questions Asked of the Ask a Librarian Service, April 2003 to March 2004 April ’03 May ’03 June ’03 July ’03 Aug ’03 Sept ’03 Oct ’03 Nov ’03 Dec ’03 Jan ’04 Feb ’04 Mar ’04 Total Chat 98 31 43 43 50 112 113 79 64 102 107 152 994 E-mail 119 68 98 80 103 163 122 102 43 106 139 148 1,291 Global Q 5 2 4 2 1 0 0 0 0 0 1 0 15 Total 222 101 145 125 154 275 235 181 107 208 247 300 2,300 the appendix.) General coding categories included month, subject area, user status, how the question was submi ed, what department answered the question, the type of question submi ed, and how the question was answered. Results Table 2 provides the number of questions submi ed to the Ask a Librarian service from April 2003 to March 2004. Questions were coded from August until March 2004. Because December 2003 only had 107, all 107 questions were coded. In total, 932 questions were coded. The status of the users of the service and the way in which questions were submi ed are presented in table 3. UIC affiliates were the largest users of the digital refer- ence service (63%), composed of graduate students (23.7%), undergraduates (23%), and faculty (15.9%). Visitors accounted for 28 percent of the questions submi ed. In 5.9 percent of the questions asked, it was not possible to determine the status of the user because his or her status was not provided and/or he or she did not provide a UIC e-mail address. Some questions (3.6%) were coded as “aborted” because a patron appeared, but a question was not asked. Aborted chats were due to technical problems, people asking the same question consecutively while they already had a chat session open, and the curious. E-mail was used more frequently for submi ing questions. Faculty (77.1%), graduates students (62.2%), and visitors (71.7%) were most likely to use e-mail; undergraduates were most likely to use chat (66.5%). Figure 3 presents the general cat- egories for the types of questions asked and the user group asking them. Ready reference questions (those that required only quick interactions, for example, known-item searches) were the most FIGURE 3 Types of Questions Asked and User Groups 444 College & Research Libraries September 2005 frequent, accounting for 37 percent of the questions. Other large categories were in-depth/mediated questions (17.9%), directional questions (17.5%), and ques- tions in which either the patron directly requested instruction or the librarian deemed it necessary to instruct the pa- tron in the use of resource(s) (11.8%). All groups asked a large number of ready reference questions (faculty, 42.1%; grad- uate students, 39.3%; undergraduates, 28.4%; visitors, 39.7%). Undergraduates were responsible for 25 percent of the instructional questions submi ed, and visitors asked 28 percent of the direc- tional questions. TABLE 3 User Status vs. How Submitted User Status How Question Was Submitted Total Questions Asked by User Group E-mail Chat Faculty/ Staff Count 108 32 140 % within user status 77.1% 22.9% 100.0% % within how submitted 22.1% 8.1% 15.9% % of total 12.2% 3.6% 15.9% Undergraduate Students Count 68 135 203 % within user status 33.5% 66.5% 100.0% % within how submitted 13.9% 34.2% 23.0% % of total 7.7% 15.3% 23.0% Graduate Students Count 130 79 209 % within user status 62.2% 37.8% 100.0% % within how submitted 26.6% 20.0% 23.7% % of total 14.7% 8.9% 23.7% Non-UIC Count 177 70 247 % within user status 71.7% 28.3% 100.0% % within how submitted 36.3% 17.7% 28.0% % of total 20.0% 7.9% 28.0% Unknown User Count 4 48 52 % within user status 7.7% 92.3% 100.0% % within how submitted .8% 12.2% 5.9% % of total .5% 5.4% 5.9% Aborted Count 1 31 32 % within user status 3.1% 96.9% 100.0% % within how submitted .2% 7.8% 3.6% % of total .1% 3.5% 3.6% Total Count 488 395 883 % within user status 55.3% 44.7% 100% % within how submitted 100% 100% 100% % of total 55.3% 44.7% 100% Quantifying Cooperation 445 TABLE 4 Question Asked versus How Submitted Question Asked How a Question was Submitted within Question Asked Total of Each Question AskedE-mail Chat Looking for articles on topic X Count 33 47 80 within question (%) 41.3 58.8 9.4 Research consultation Count 7 1 8 within question (%) 87.5 12.5 .9 Location of information on topic X Count 74 46 120 within question (%) 61.7 38.3 14.1 Database mechanics Count 10 11 21 within question (%) 47.6 52.4 2.5 Accessing e-resources Count 19 12 31 within question (%) 61.3 38.7 3.6 Accessing e-journals Count 37 15 52 within question (%) 71.2 28.8 6.1 Book holdings Count 41 40 81 within question (%) 50.6 49.4 9.5 Journal holdings Count 35 34 69 within question (%) 50.7 49.3 8.1 AV, special collections holdings Count 51 17 68 within question (%) 75.0 25.0 8.0 Factual Count 15 17 32 within question (%) 46.9 53.1 3.8 Statistical Count 6 4 10 within question (%) 60.0 40.0 1.2 Technical access Count 20 17 37 within question (%) 54.1 45.9 4.3 Library hours, location Count 9 10 19 within question (%) 47.4 52.6 2.2 Online catalog, My Account Count 17 10 27 within question (%) 63.0 37.0 3.2 Library policies and services Count 76 54 130 within question (%) 58.5 41.5 15.3 Citation verification Count 1 3 4 within question (%) 25.0 75.0 .5 Collection suggestions Count 6 0 6 within question (%) 100.0 .0 .7 446 College & Research Libraries September 2005 TABLE 4 Question Asked versus How Submitted How a Question was Submitted Total within Question of Each Asked Question Question Asked E-mail Chat Asked Notice of errors on Web Pages Count 0 1 1 within question (%) .0 100.0 .1 Complaints, courtesies Count 4 2 6 within question (%) 66.7 33.3 .7 Other Count 26 23 49 within question (%) 53.1 46.9 5.8 Total Count 487 364 851 within question (%) 57.2 42.8 100.0 To provide more detailed information about types of questions beyond general categories such as directional, ready ref- erence, and in-depth reference, a more exhaustive list of specific types of ques- tions also was coded. Table 4 presents the number of times specific types of ques- tions were asked and how questions were submi ed. Questions related to library services and policies represented the largest group of questions asked (15.3%). Where would information on a particular topic be found (14.1%) and looking for articles on a particular topic (9.4%) also represented a large number of questions asked. Other popular types of questions included: • Do you have this book (9.5%)? • Do you have this journal (8.1%)? • Do you have this audiovisual or special material (8%)? Total questions about materials owned by the library represented about 25.6 percent of the questions. Users appeared to favor one type of medium over another (e-mail versus chat) for asking specific types of questions. For example, users tended to use e-mail in categories such as research consultation, where to find information on a particular topic, accessing electronic resources (da- tabases and journals), audiovisual hold- ings, statistical information, the library catalog, library services and policies, and suggestions for the collection. Users tended to use chat to ask questions related to finding articles on a particular topic and citation verification. Table 5 presents the number of times a specific type of question was asked by user status. Faculty/staff were most likely to ask questions related to accessing e- journals (13.38%) and questions about library services and policies (15.45%). Of all the users of the service, undergraduate students were most likely to ask questions related to finding articles on a particular topic (18.9%) and asking for a general starting point on where to find informa- tion on a particular topic (18.5%). Gradu- ate students most frequently asked ques- tions related to journal holdings (14%) and library services and policies (15.4%). Visitors (non-UIC) were most likely to ask where information could be found on a general topic (17.4%), holdings related to audiovisuals and special collections (13.9%), and questions about library ser- vices and policies (19.7%). Each question was examined independently to deter- mine the specific user group most likely to ask each type of question. For example, undergraduate students asked questions related to finding articles on a particular Quantifying Cooperation 447 topic 48.1 percent of the time, whereas graduate students most frequently asked questions related to database mechanics (57.1%), accessing e-journals (41.5%), jour- nal holdings information (43%), technical and access questions (43.3%), and UIC- CAT My Account questions (40.64%). Fi y percent of research consultation requests were from faculty. Visitors were the most likely group to ask questions related to general information (36.2%), book-hold- ings information (31.7%), audiovisual and special collections (51.4%), factual infor- mation (60%), hours and location (68.4%), and library policies and services (38%). Figure 4 presents the number of ques- tions answered by each department and also the subject area of questions asked. The majority of questions asked were relat- ed to the liberal arts (32%), followed by the health sciences (23.5%). Science questions accounted for 5.5 percent and government document–related questions for 3 percent. A large number of questions asked were related to library services and/or policies (21.6%). The Daley reference department answered the majority of questions (45%), followed by the health sciences reference department (39.5%). Forty-four percent of all the questions answered by the health sciences reference librarians were related to the health sciences. Of all the health sciences questions asked, approximately 74 percent were answered by a health sciences librarian. Forty-three percent of the questions answered by the Daley TABLE 5 Type of Question Asked by User Status Faculty/ Staff Under- graduate Student Graduate Student Non-UIC User Unknown User Looking for articles on topic X 4 39 16 12 10 Research consultation 4 1 2 1 0 Location of info on topic X 10 38 19 45 10 Database mechanics 2 7 12 0 0 Accessing e-resources 9 8 10 4 1 Accessing e-journals 19 8 22 3 1 Book holdings 14 16 21 27 7 Journal holdings 8 20 31 9 4 AV, special collections 11 7 11 36 5 Factual 6 5 1 21 2 Statistical 2 6 1 1 0 Technical access 11 5 16 2 3 Library hours, location 1 3 2 13 0 Online catalog, My Account 5 7 11 2 2 Library policies and services 22 23 34 51 4 Citation verification 2 1 0 2 0 Collection suggestions 1 0 0 6 0 Notice of errors on WebPages 0 0 0 1 0 Complaints, courtesies 3 0 0 3 1 Other 8 12 11 19 1 448 College & Research Libraries September 2005 FIGURE 4 Questions Asked by Department and Subject Area reference department were related to the liberal arts and sciences. Of all the liberal arts and sciences questions asked, 60 percent were answered by the Daley reference librarians. The Daley reference department also answered 56.5 percent of all science questions. Both reference de- partments contributed relatively equally to answering questions related to library policies and services. The percentages of questions answered by the respective reference departments in both chat and e-mail also were ex- amined. For e-mail, the health sciences reference department answered most of the health sciences questions (81.1%) and the social sciences and humanities refer- ence department answered most of the social sciences and humanities questions (62.4%). The same pa ern held true for chat, although the percentage was not as great when compared with e-mail. The health sciences reference department answered the majority of health sciences questions (64%) and the social sciences and humanities reference department answered the majority of social sciences and humanities questions (57.3%). Figure 5 details the number of times specific questions were asked by various subject disciplines. Questions asked with a health sciences subject were most o en related to finding articles on a particular topic (17%) or journal-holdings informa- tion (19%). The most frequent type of question asked in the liberal arts was related to pointing to a general area to find information on a topic (21.4%). In the sciences, the most common types of questions were related to journal-hold- ings information (20.8%). Discussion This study examined the effectiveness of a one-service-point digital reference service in a complex academic library composed of multidisciplinary and geographically dispersed libraries. Second, the study sought to measure the relative success of this approach by quantifying data about the questions received, such as user status, subject area, and the answering process. The fear that the service would be in- undated by inappropriate questions from within and outside the university proved unfounded. Use, predictably, followed the academic calendar year with the highest use at the beginning of semesters and the lowest use during winter and summer Quantifying Cooperation 449 FIGURE 5 Frequency of Questions Asked by Subject Discipline F in di ng a rt ic le s0 R es ea rc h co ns ul ta ti on 0 In fo rm at io n on a t op ic 0 D at ab as e m ec ha ni cs 0 A cc es si ng e -r es ou rc es 0 A cc es si ng e -j ou rn al s0 B oo k ho ld in gs 0 Jo ur na l ho ld in gs 0 A V , S pe ci al C ol le ct io ns 0 F ac tu al 0 S ta ti st ic al 0 T ec hn ic al A cc es s0 L ib ra ry H ou rs , L oc at io n0 O nl in e ca ta lo g, M y A cc ou nt 0 L ib ra ry p ol ic ie s & s er vi ce s0 C it at io n ve ri fi ca ti on 0 C ol le ct io n su gg es ti on s0 N ot ic e of e rr or s0 C om pl ai nt s, c ou rt es ie s0 O th er 0 Health Science Liberal Arts Science Government holiday months, possibly indicating the need to reconsider staffing pa erns for the Ask a Librarian service. UIC affiliates accounted for nearly two-thirds of the activity. Surprisingly, no questions were referred outside UIC, even though Ques- tionPoint provided that capability. Examination of the types of questions asked and how they were submitted provides insight into user preferences and information needs. E-mail was used most frequently for submi ing questions. The breakdown within user groups shows that faculty used e-mail 77.1 percent of the time, visitors 71.7 percent of the time, and graduate students 62.2 percent of the time. Perhaps not surprisingly, un- dergraduates showed a clear preference for chat, submi ing 66.5 percent of their questions via this method. Preference for chat among undergrad- uates might be a ributed to a generational culture a uned to mobile communication and instant messaging. But the preference also may correlate with the information needs of undergraduates as reflected by the types of questions they asked. Analysis of questions showed that que- ries received via chat tended to relate to how to find articles on a particular topic, the type of question most o en asked by undergraduates in this study. This group of users also required the most guidance in selecting databases and pathways for finding information on specific topics. Based on these data, undergraduates not only require intervention in their research activities, but also tend to desire immediate assistance in completing their research tasks. This contrasts significantly with faculty and graduate students, who asked more ready reference and technical questions related to accessing electronic journals than any other user group in- cluding undergraduates. Interestingly, a combination of technological comfort and perceived time constraints seems to influence the chosen mode and content of submissions. Quantifying user communication preferences and types of questions asked by different groups of users informs reference and instructional programs. Obviously, these findings indicate that undergraduates need significant infor- mation literacy instruction, but the fact that they combine this need with a sense of urgency suggests that they may be receptive to “on-demand” methods of instruction such as online tutorials and context-sensitive help. Faculty and gradu- ate student questions reflect higher infor- 450 College & Research Libraries September 2005 mation literacy and less need for guidance in the selection and use of resources. For these more independent users, design changes and improved organization of library Web pages should meet their needs. Indeed, the large number of hold- ings and policy questions, from all users, point to this need as well. Frequently asked questions by visitors about library services identified a need to develop an FAQ page about library services for alumni and nonaffiliated users. Perhaps future undertakings that focus on increas- ing usability of library pages should take these findings into account and design methodologies that measure the dif- ferences in the ways these user groups interact with the pages. The majority of visitor queries were, appropriately, specific to UIC library holdings and services. Over a quarter of all queries were for holdings information and 15 percent related to library policies or services. This has several implications. First, given that 41 percent of all queries received required no subject expertise on the part of the librarians, it affirms that a one-stop digital reference service may be an efficient way to provide service, especially considering that when subject expertise was required, questions were generally assigned appropriately. Second, it suggests that reference collaboration across several institutions may not be feasible because 41 percent of questions were UIC-centric. Eventual shared ref- erence with other like institutions had been envisioned and indeed, was one of the primary reasons QuestionPoint was selected over other systems. These findings corroborate similar findings by other investigators that between 19 and 66 percent of questions were related to known item searches or library policies or services. (See table 1.) Kibbee, Ward, and Ma noted that the significant quantity of questions relating specifically to their institution’s library resources and services led them to question their ability to share a collaborative digital reference service with another institution.25 Although the results here suggest a similar finding, they also argue persuasively that intra- institutional collaboration in the service is indeed efficient, effective, and desirable. An analysis of the subject area/types of questions received showed some surpris- ing patterns. Although subject-related questions predominated, 22 percent of questions pertained to library services. It is unlikely that librarians were answer- ing as many of these questions before the implementation of this easily acces- sible digital reference service. The need to include staff from circulation and interlibrary loan became apparent as this pa ern emerged. The greatest number of questions in the subject breakdown came from the liberal arts (32%), followed by the health sciences (23.5%). It is noteworthy that such a large percentage came from the health sciences because health sciences students comprise only one-fi h of the student body. This finding may not be all that surprising given that one study found the ratio of remote users to in-house users of academic health science libraries is five to one, compared to 1.3 remote users for each in-house user at “main” academic libraries.26 Differences between the types of questions asked by the two main subject groups (health sciences, liberal arts) do reflect discrepancies in their needs. Not surprisingly, the reliance of the health sciences users on journal materials comes through. Of questions about journal hold- ings, 54.2 percent came from the health sciences and 42.3 percent of the questions about e-journals came from this same group. Other areas in which the health sciences predominated were research con- sultation and database mechanics, asking 57.1 and 38 percent of the questions in these categories respectively. The need for a wider variety of materials was appar- ent for the liberal arts users. These users were more likely to ask broader questions on where to seek information on a topic (50% of questions in this category). This group also asked the greatest percent- http:libraries.26 http:institution.25 Quantifying Cooperation 451 age of questions (39.4%) about accessing e-resources (that is, electronic databases and online textbooks), as opposed to e- journals. The preference for monographic information is seen again with this group whose members asked 42.4 percent of the book-holdings questions and 52.9 percent of the questions on audiovisual holdings and special collections. In total, 9.4 percent of questions were answered through reassignment, referral, or consultation. Although the librarian makes a choice to answer an e-mail ques- tion based on its content, he or she does not have a choice with chat, at least not upon initial contact. Despite this, chats oc- curred o en when the subject expert was available. Surprisingly, more questions asked through e-mail were referred to another librarian or answered in consul- tation with another librarian (9.1%) than those through chat (8.6%). The data show that the department with the relevant subject knowledge answered the majority of questions requiring subject expertise. The one-service-point model used by UIC demonstrates that subject specialists are not underutilized—and perhaps are utilized more effectively—in a dispersed library environment. Anecdotal evidence from librarians implies that working within this model has given them a bet- ter understanding of information issues across the university and a deeper ap- preciation of the specialized skills of their colleagues from other departments and library sites. The authors are investigating the librarians’ comfort with, and a itudes toward, a shared digital reference service; results will be reported separately. Conclusions Results of this study demonstrate that a centralized digital reference service is feasible even in a complex academic environment serving a diverse user popu- lation with widely divergent disciplines spread across five campuses. Having one contact point relieves the burden of choice for library users looking for real-time and expedient assistance and places the decision with the librarian, who is best equipped to triage the request to the most appropriate person. The success of the service has led to expansion beyond the reference departments to include special collections, circulation, interlibrary loan, and systems. This will mean new chal- lenges in accommodating the range of working pa erns and styles across these units. For staffing, it means personnel at each library site will more efficiently be shared across all sites. For users, it means a simple and seamless interface to digital services. Analysis of the questions asked, how they were asked, and by whom has pro- vided insight into ways the library can im- prove service to its patrons. A redesigned Web page could provide prominent links to information commonly requested, such as library privileges for alumni or pathways to specific subject resources grouped by user status. The need for instructional programs, both online and in the classroom, could be identified and developed to address those areas that generate the most confusion and frustra- tion for patrons. A compilation of FAQs or context-sensitive help screens could be assembled and made easily available to users. Though questions about the quality of digital reference services remain to be answered, the rich data provided by question logs present us with a chance to evaluate the accuracy and complete- ness of the service we give our patrons. As a complement to user surveys, chat and e-mail transcripts can offer another measure of user satisfaction. Unlike the fleeting moment of in-person or telephone reference encounters, these methods provide a record that can be analyzed to establish benchmarks for reference service that have heretofore proved elusive. The analysis of submit- ted questions gives librarians one more channel of information to utilize in decisions about library and Web page usability, instruction activities, and ref- erence services. But more important, it 452 College & Research Libraries September 2005 provides opportunities for librarians to impede their progress, and providing be proactive in fulfilling our users’ needs, be er and more valuable access to library removing stumbling blocks that so o en resources and services. Notes 1. Lara Bushallow-Wilbur, Gemma S. DeVinney, and Fritz Whitcomb, “Electronic Mail Refer- ence Service,” RQ 35 (spring 1996): 359–63+. 2. Wendy Diamond and Barbara G. Pease, “Digital Reference: A Case Study of Question Types in an Academic Library,” Reference Services Review 29, no. 3 (2001): 210–18. 3. Lisa Lo Jerant and Kenneth Firestein, “Not Virtual, But a Real, Live, Online, Interactive Reference Service,” Medical Reference Services Quarterly 22 (summer 2003): 57–68. 4. Karen Ciccone and Amy VanScoy, “Managing an Established Virtual Reference Service,” Internet Reference Services Quarterly 8, no. 1/2 (2003): 95–105. 5. Ma hew R Marsteller and Danianne Mizzy, “Exploring the Synchronous Digital Refer- ence Interaction for Query types, Question Negotiation, and Patron Response,” Internet Reference Services Quarterly 8, no. 1/2 (2003): 149–65. 6. Sandi K Parker and E. Diane Johnson, “The Region 4 Collaborative Virtual Reference Project,” Medical Reference Services Quarterly 22 (summer 2003): 29–39. 7. Carol A. Powell and Pamela S. Bradigan, “E-mail Reference Services: Characteristics and Effects on Overall Reference Services at an Academic Health Sciences Library,” Reference & User Services Quarterly 41 (winter 2001): 170–78. 8. Ma hew R Marsteller and P. Neuhaus, “The Chat Reference Experience at Carnegie Mellon University,” (June 2001). Available online at h p://www.contrib.andrew.cmu.edu/~ma hewm/ ALA_2001_chat.html. (Accessed 15 August 2004). 9. Josephine Z. Kibbee, David Henry Ward, and Wei Ma, “Virtual Service, Real Data: Results of a Pilot Study. Real-time Online Reference at the University of Illinois,” Reference Services Review 30, no. 1 (2002): 25–36. 10. Diamond and Pease, “Digital Reference,” 213–14. 11. Ibid., 217. 12. Powell and Bradigan, “E-mail Reference Services,” 174. 13. Kibbee, Ward, and Ma, “Virtual Service, Real Data,” 35. 14. Marsteller and Neuhaus, “The Chat Reference Experience at Carnegie Mellon University.” 15. Bushallow-Wilbur, DeVinney, and Whitcomb, “Electronic Mail Reference Service.” 16. Diamond and Pease, “Digital Reference.” 17. Jerant and Firestein, “Not virtual, But a Real, Live, Online, Interactive Reference Ser- vice.” 18. Ciccone and VanScoy, “Managing an Established Virtual Reference Service.” 19. Marsteller and Mizzy, “Exploring the Synchronous Digital Reference Interaction for Query types, Question Negotiation, and Patron Response.” 20. Parker and Johnson, “The Region 4 Collaborative Virtual Reference Project.” 21. Powell and Bradigan, “E-mail Reference Services.” 22. Marsteller and Neuhaus, “The Chat Reference Experience at Carnegie Mellon Univer- sity.” 23. Kibbee, Ward, and Ma, “Virtual Service, Real Data.” 24. Wendall Sullivan, Lisa A. Schoppmann, and Patricia M. Redman, “Analysis of the Use of Reference Services in an Academic Health Sciences Library,” Medical Reference Services Quarterly 13 (spring 1994): 35–55. 25. Kibbee, Ward, and Ma, “Virtual Service, Real Data,” 33. 26. B. Franklin and T. Plum, “Library Usage Pa erns in the Electronic Information Environ- ment,” Information Research 9, no. 4 (2004). Available online at h p://informationr.net/ir/9-4/pa- per187.html. (Accessed 15 September 2004). Quantifying Cooperation 453 Appendix Virtual Reference Coding System 1. What is the month the question was submitted? 2. Who was the user? • Faculty/staff • Undergraduate student • Graduate student • Non-UIC • Don’t know • Referral from other question point service • Aborted interaction 3. How was the question received/submitted? • Chat • E-mail 4. What was the Library/Department answering question? • LHS • Daley • Science • Gov Docs • Special Collections • Other 5. What was the subject area of question? (Choose best fit.) (Classify question about owned material by subject of the material [do you have…].) • Basic sciences: Biology/chemistry/engineering/math • Health sciences: Allied health, medicine, nursing, pharmacy, psychiatry, public health, health question asked by a consumer • Liberal arts: Arts/architecture, business, education, English composition, geog- raphy, history, humanities, psychology, social science, social work, sociology • Government question • Library (services, policies, etc.) • Other 6. What was the type of question asked (general category) ? • Directional questions: General questions regarding location of services, policies, col- lections (where are the books), and materials (where are the journals), hours, directions • Ready reference (factual): Question that can be answered quickly (do you have this book, do you have this journal), simple fact look-up • In-depth/mediated: Questions not easily answered (where can I find a copy of the Diagnostic Interview Schedule for Children, normal results of a liver enzyme test, how prevalent is TB in IL, relationship of crime to expansion of cities and suburbs, history of race in American political system) • Instructional: Questions requiring some form of instruction to answer (How do I use X database, doing an article on X and need journal articles, how do I use e-mail) • Technical: Difficulty accessing a resource, PDF problems, browsers problems, problem with net ID 454 College & Research Libraries September 2005 • Accounts, status: Status check on requested items, account probes (ILL, circula- tion, My Accounts) 7. What was the type of question asked (specific)? Please choose the most appropriate based on what was the main focus of the question. • Looking for articles on X: How do I find articles on X, help with search strategy or suggest terms (strategy based) • Research consultation: In-depth question where librarian creates search strategy and provides citations • Where would I find information about X (this would be a more general request than 1) • Database mechanics: How do I search/use this resource/navigate/instructional • Accessing e-resources (location of database, do we have, identify a database to use) • Accessing e-journals (do you have this e-journal, how do I get to this e-journal, why can’t I access this journal from 1985 online) • Do you have this book? • Do you have this journal? • Do you have this material (audio/visuals/sounds/slides/special collections (online or at UIC)? • Factual question (discrete answers, nonstatistical, definitions) (how many CCs, how many colors in rainbow) • Statistical information (how many people born in 1950) • Technical problem/access issues (browser won’t reload, Web page is missing, how do I get in from home, how do I get passwords or a net ID, is the server down) • Info about library hours, location • Questions about UICCAT, My Account • Questions about circulation, ILL services & policies (circ, ILL, Illinet, reserves) • Citation verification • Suggestions for collection • Notice of errors (journals, Web pages) • Complaints, courtesies (nonquestions) • Question about Blackboard • Other Association of College and Research Libraries A Division of the American Library Association 2006 Division Awards Call for Nominations Excellence in Academic Libraries Award $3,000 and plaque Sponsored by Blackwell’s Book Services Hugh C. Atkinson Memorial Award Cash award and plaque Sponsored by ACRL, ALCTS, LAMA, LITA Academic or Research Librarian of the Year Award $3,000 and citation Sponsored by YBP Library Services Samuel Lazerow Fellowship $1,000 and plaque Sponsored by Thomson Scientific Doctoral Dissertation Fellowship $1,500 and plaque Sponsored by Thomson Scientific Submit nominations for the 2006 Awards Program by December 2 to Megan Bielefeld ACRL Program Coordinator 50 East Huron Street, Chicago, IL 60611 Phone: (312) 280-2514 E-mail: mbielefeld@ala.org more details in the awards section of the acrl web site: www.ala.org/acrl mailto:mbielefeld@ala.org