College and Research Libraries The Effectiveness of an Information Desk Staffed by Graduate Students and Non professionals Beth S. Woodard The effectiveness of an information desk staffed by graduate students and nonprofessionals in an academic library was tested using unobtrusive methodology. Data on 164 transactions were collected during April and May 1987. While 83.5 percent of the questions were answerable with available sources, only 62.2 percent of these were correctly answered. Another 8. 5 percent were immediately referred to the reference desk. Relationships-weak, but statistically significant-exist between the level of the staff member and both the correctness of answers and the quality of referrals. Findings indicate that services would be improved by restructuring staffing patterns and strengthening training for particular kinds of questions. ibraries increasingly use non- professional staff at reference or information desks with the idea that a high proportion of questions do not require professional ex- pertise.1 Although nonprofessionals rarely have formal on-the-job training, their performance in answering these easy questions is rarely a focal point of re- search. 2 This study attempts to test the theory that nonprofessionals and stu- dents can be effective in reference service and make appropriate referrals. It is hoped that the results will help academic libraries in making decisions about estab- lishing and staffing information desks for answering simple questions and screen- ing reference questions. Additionally, the study presents data on current competen- cies and problem areas to further explora- tion of the effectiveness of particular train- ing methods for specific kinds of questions. BACKGROUND Since 1980, reference staff at the Univer- sity of Illinois in Urbana-Champaign have administered an information desk de- signed to handle directional questions; to answer some basic ready reference ques- tions using a small collection of reference titles; to screen incoming telephone refer- ence questions; and to instruct patrons in the use of the automated circulation sys- tem, the online catalog, and the card cata- log. Staff members attempt to answer most questions, but are encouraged tore- fer the question if the answer is not readily obtainable with the materials at hand. Staff members are instructed never to say that an item is not owned, but merely that they cannot locate it, and to refer the ques- tion to a librarian. At first the desk was staffed mainly by volunteers from various library depart- ments. Service is now provided by eight graduate library school students who hold Beth S. Woodard is Central Information Services Librarian and Assistant Professor of Library Administration at the University of fllinois, Urbana, fllinois. 455 456 College & Research Libraries quarter-time assistantships in the library, seven to eight student assistants (most of whom are also library school students) and two library technical assistants who spend about one-quarter to three-eighths of their time at the information desk. The reference staff and others in the uni- versity library believe that the information desk has been successful. A great deal of time and effort is devoted each year to the planning and coordination of orientation sessions and weekly training meetings for the staff. 3 Most graduate students re- sponding to an evaluation form each spring report a positive learning experi- ence. Use statistics for the information desk emphasize the number of directional questions that no longer have to be han- dled by professionals. The proportion of questions referred to the reference desk averages around 9 percent. This is less than the 14 percent cited by Beth J. Sha- piro as a significant achievement in her study. 4 Like Shapiro, the reference staff felt that nonprofessionals ''are able to pro- vide adequate assistance." 5 These positive indicators do not elimi- nate the need for a more formal evaluation of the service. First, the general assump- tion of success needs to be tested quantita- tively. Are correct answers being sup- plied? Are appropriate referrals being made? Are nonprofessionals attempting to do too much and giving misleading in- formation? Is the scope of the information desk too broad? Is its function appropri- ately defined? Charles McClure and Peter Hernon stress that ''perceptions and as- sumptions about what is thought to be oc- curring in the provision of library services should be validated by programs of ongo- ing evaluation research. " 6 The effectiveness of the training pro- gram also needs examination. Does the program need general improvement or are there only a few specific areas that need improvement? Can the same level of competency be achieved with a smaller in- vestment of time and energy? How can we measure improvement in training when all we have to go on are our qualitative perceptions about the program? McClure and Hernon stress that "without empiri- cal evidence-evaluation-describing the July 1989 level of quality provided, it is difficult to develop effective and meaningful mana- gerial strategies to improve reference ser- vices."7 PURPOSEOFTHESTUDY In an effort to gain some reliable infor- mation about the effectiveness of service provided at the information desk, a study was conducted. Information was collected on correctness of answers given, levels of patrons, types of questions, times when reference librarians staff the reference desk and two people staff the information desk, the level of the staff member, sources used to answer questions, appro- priateness of referrals, and patrons per- ceptions. ''While there are many ways to assess reference services, 'true evaluation deals with the results or outputs of an activity or program.' '' METHODOLOGY Rationale for the Data Collection Method While there are many ways to assess ref- erence services, 8 "true evaluation deals with the results or outputs of an activity or program. Applied to libraries, this means that evaluation must deal with the ser- vices users receive . " 9 Inputs (i.e., em- ployee selection, education, orientation, and training, and job aids, tools, and equipment) cannot be dramatically im- proved without careful analysis or with- out knowledge of the present effective- ness of outputs. Because actual on-the-job performance, not the potential for service, determines how effectively the information desk per- forms its routine functions, the methodol- ogy had to be conducted in an environ- ment as close to the actual as possible. Job performance may be evaluated through obtrusive methods such as observer dia- ries, self-diaries, interviews, question- naires, the critical incident method, pa- . tron satisfaction questionnaires, or correct The Effectiveness of an Information Desk 457 fill-rate estimates by library staff. 10' 11 These methods do not consider how the evalua- tion itself may change behaviors. 12 Terry Weech and Herbert Goldhor found a slight but statistically significant differ- ence in the complete and correct answers given in obtrusive versus unobtrusive tests. 13 Unobtrusive testing, in which the evalu- ation process is not known to those being evaluated and surrogates ask questions with predetermined answers, has been used in libraries with increasing regular- ity, primarily by educators rather than practitioners. Previous uses of this meth- odology are detailed in several reviews of the literature. 14 Although various unob- trusive studies have focused on academic libraries, only Janine Schmidt's and Peter Hernon and Charles McClure's studies have used in-person questions. 15 While Hernon and McClure's proxies were in- structed to seek out professionals, non- professionals may also have been in- cluded but are not identified separately. 16 Comparative studies of nonprofession..: als and professionals have been con- ducted by Charles Bunge, and by Egill Halldorsson and Marjorie Murfin, and by Murfin and Bunge. 1 Bunge found that nonprofessionals can accurately and quickly answer a wide range of factual ref- erence questions but generally take more time to do so than do professionals. 18 Halldorsson and Murfin focused on inter- viewing skills and found that nonprofes- sionals do about half as well as profession- als with indirect or faulty information questions that require more skill in inter- viewing.19 A recent study by Murfin and Bunge has shown that ''although para- professional staff can be used at the refer- ence desk in academic libraries with con- siderable success, the service provided by such staff may also result in significantly lower patron success and satisfaction than that achieved by professional librari- ans."20 The majority of unobtrusive studies have dealt exclusively with ready- reference or factual questions. Janine Schmidt's study is an exception, because general information rather than hard, identifiable fact questions was used. The present study is unique. It is an unobtru- sive test focused on nonprofessionals at an information desk in an academic li- brary. Test questions were asked in per- son, and included directional, procedural, bibliographic, and subject-oriented inqui- ries, as well as ready reference questions. Su"ogate Patrons One hundred surrogates were selected to ask two questions each during April and May 1987. An unpublished survey of information desk users conducted in fall 1982 by Ellistine Anita Boze found that 84 percent of the 250 people in the sample (eliminating visitors) who came to the In- formation Desk were students (graduate and undergraduates combined), 4% were faculty, and 12% were staff. The campus population in fall 1986 totalled 45,852, with 27,199 (59%) undergraduate stu- dents, 9,131 (20%) graduate students, 2,763 (6%) faculty, and 6,759 (15%) staff. 21 Because a representation of each user group was desirable, a proportional mix of 40%,30%,20%, and 10%, respectively, for undergraduates, graduate students, staff, and faculty was devised. It was necessary to increase the size of the smallest group (faculty) from the previous populations (4% and 6%) in order to have a large enough sample with which to work. After experiencing difficulty in identifying un- dergraduates who were willirig to partici- pate, 5 were replaced by graduate stu- dents, making the total 35 under- graduates, 35 graduate students (70% stu- dents), 20 staff members, and 10 faculty for the 100 surrogate patrons. Surrogates were selected on the basis of level and availability, Jasim Jirhees' crite- ria for proxy selection were also followed so that surrogates selected were not out of the ordinary, were reasonably articulate, and did not think the use of the technique was unethical. 22 Although several individ- uals were concerned about the ethics of the method, they agreed to participate af- ter being assured that it had been used elsewhere, that it was approved by the In- stitutional Review Board, and that the an- onymity of the individual staff members would be preserved in any discussion of the results. Slightly under half the surro- 458 College & Research Libraries gates participating were male; slightly more than half were female. Test Questions Questions used in the unobtrusive eval- uation were actual questions recorded during 1986. They were examined and grouped into five categories: (1) biblio- graphic (whether the library owned a par- ticular book or journal with a known title or author); (2) research guidance (how to find information on a particular subject); (3) procedural or instructional (request for information on, for example, how to use the online catalog; how to get photocopy charge cards; loan policies; or how to charge, renew, or reserve items on the on- line catalog; interpretation of online cata- log records was also included in this cate- gory); (4) ready reference (requests for brief, factual information such as tele- phone numbers, addresses, book prices, zip codes, and locations or phone num- bers of offices on campus); and (5) direc- tional (location of library facilities, collec- tions, or staff; requests for hours, pencils, staplers, change, phone, copiers, and sim- ilar inquiries). Two hundred questions were selected, 40 from each category. Because these were actual questions, citations were often in- adequate, subject requests were often confused and imprecise, and ready refer- ence questions were sometimes unan- swerable. These questions were used in- tentionally because it was important to test the accuracy of referrals. Questions were only asked in person and not over the telephone. Timing of Inquiries The information desk is staffed all the hours the library is open. Two staff mem- bers are used only at peak periods. The reference desk is staffed fewer hours. Be- cause staffing patterns might affect accu- racy of service, length of time taken to an- swer questions, and appropriateness of referrals, times were selected to reflect the following situations: (1) single staffed, li- brarian present; (2) single staffed, no li- brarian present; (3) double staffed, librar- ian present, and (4) double staffed, no librarian present. Because the last situa- tion only occurs on Sunday evenings, it July 1989 was eliminated as a category in final test- ing. Reporting of Results Report forms were prepared for each question. Each participant received an in- struction sheet and two report forms. Once the questions were recorded on the report forms, the questions were ran- domly paired with staffing levels and sur- rogate patron status and sex so that they were evenly distributed. For example, of the 40 bibliographic questions, 14 were as- signed to undergraduates, 14 to graduate students, 8 to staff and 4 to faculty. The 14 undergraduate questions were randomly assigned to the three staffing levels to be tested, so that no category of questions was asked predominantly by one status level of surrogate or during a time with a particular level of staffing. Report forms were also coded for the surrogates' status and sex, the type of question, and the staffing level of the information desk. In addition, the time to ask the question was written on the form. Sometimes surro- gates were given ranges of hours and days that questions could be asked within a staffing level in order to accommodate busy schedules. Questions were paired according to times and content. For example, a biblio- graphic question might be paired with a directional or a procedural question. Every effort was made to avoid pairing two research assistance questions. Some surrogates were able to make only one trip to the library, as both questions were to be asked during the same staffing level time, while others were required to ask their questions at two different times. If regular users of the information desk served as surrogates, an attempt was made to en- sure consistency between the user's level of sophistication and the level of sophisti- cation suggested by the question. The recording form asked surrogates to report whether the information desk staff asked questions to clarify the inquiry or re- quested further information before pro- viding an answer: the content of these ex- changes was recorded, and the answer to the test question recorded. Surrogates also noted if the staff gave the source of the answer or indicated how the informa- The Effectiveness of an Information Desk 459 tion was found. If the test question could not be answered, surrogates recorded whether they were referred elsewhere for further assistance and specified to what or to whom the referral was made. Surrogates were also asked to rank on a scale from one to five the staff's friendli- ness, quickness, politeness, helpfulness, interest, enthusiasm, and competence. Fi- nally, surrogates were asked to note the amount of time that elapsed between ask- ing the question and receiving the answer. After all forms were turned in, the su- pervisor of the information desk took all revised desk schedules and coded them for level of individual staff (graduate assis- tant, student assistant, and library techni- cal assistants). The research assistant then took coded schedules and compared them to report forms so that neither person in- volved in the project could associate a par- ticular question and answer with a specific individual. However, types of staff could be identified. FINDINGS Response rate Of the 200 question forms, 164 were re- turned and were usable, for a response rate of 82 percent. The 36 remaining forms could not be used for one of several rea- sons: the questions were asked at the wrong service desk, report forms were lost, report forms were not returned, or the surrogate recognized the staff member and did not ask the question. Accuracy of Answers Table 1 shows the number of correct re- sponses by the type of question. In the strict sense, 62.2% were answered cor- rectly. According to written guidelines for information desk personnel, however, re- ferrals are considered to be a correct re- sponse. Consequently, the broader con- cept of correct handling of the question may be a truer measure of accuracy than correct answering. When combined into two categories of acceptable responses, i.e., correct answers and immediate refer- rals to reference, and unacceptable re- sponses, i.e., no answer, or an incorrect or only partially correct answer, a different picture emerges. For instance, biblio- graphic questions were handled correctly 60% of the time, ready reference questions 67.6%, and subject questions 64% when this criterion of accuracy is used. While subject responses could use some im- provement (although partially correct an- swers were not paired with subsequent re- ferrals at this point), it is obvious that more attention needs to be given in the training program to bibliographic ques- tions and ready reference questions. Ability to Answer Of the 164 questions, 83.5% could be an- swered completely at the information desk using available sources, as detailed in table 2. The breakdown by type of ques- tion indicates that subject questions can be handled least well, although a large num- ber can be partially answered. Comparing the questions answered correctly as listed in column 1 of table 1 with the questions capable of being answered correctly as listed in column 1 of table 2, it is apparent TABLE 1 Correctness of Answer by Type of Questions Type of question Correct Partially Incorrect No answer Referred Row correct total Bibliographic 20 (57.1) 5 (14.3) 5 (14.3) 4 (11.4) 1 ( 2.9) 35 (19.6) (23.8) (27.8) (44.4) (1.7) (21 .3) Subject 9 (32.1) 6 (21 .4) 2 ( 7.1) 2 ( 7.1) 9 (32.1) 28 (8.8) (23.8) (11.1) (22.2) (64.3) (17.1) Procedural 24 (72.7) 4 (12.1) 5 (15.2) 33 (23.5) (19.0) (27.8) (20.1) Ready reference 20 (59.0) 4 (11.8) 5 (14.7) 2 ( 5.9) 3 ( 8.8) 34 (19.6) (19.0) (27.8) (22.2) (21.4) (20.7) Directional 29 (85.3) 2 ( 5.9) 1 ( 2.9) 1 ( 2.9) 1 ( 2.9) 34 (28.4) (9.5) (5.6) (11.1) (7.1) (20.7) Column totals 102 (62.2) 21 (12.8) 18 (11.0) 9 ( 5.5) 14 ( 8.5) 164 100.0 Figures in parentheses to the right of raw numbers represent row percentages. Figures in parentheses below raw numbers represent column percentages. 460 College & Research Libraries that by type, of those questions capable of being answered at the information desk, 63% of ready reference questions, 65% of bibliographic questions, 75% of subject questions, 83% of procedural questions, and 91% of directional questions were an- swered correctly. Table 1, then, can mis- lead without consideration of how many of those questions have the potential to be answered at the information desk. "Sources most heavily used were the online catalog, ready reference sources, and the rolodex (a file of lo- cal information and procedures). The card catalog was the most underuti- lized source available at the Informa- tion Desk.'' Use of Available Sources Sources most heavily used were the on- line catalog, ready reference sources, and the rolodex (a file of local information and procedures). The card catalog was the most underutilized source available at the information desk. A correct source was available, but not used, 16.5% of the time (27 questions). An examination of these questions showed that several depended on the use of ready reference sources, such as World Almanac or Commonwealth Universities Yearbook, but that several could have been answered at least partially by July 1989 use of the card catalog. The University of lllinois' online catalog contains entries for materials cataloged via OCLC since 1975. No retrospective con- version has been done, so subject access to older materials is exclusively through the card catalog. Quite often the information desk staff would search the online catalog for appropriate materials but would fail to search the card catalog, even when the topic (such as the Civil War) suggested that there should logically be many entries there. Responses of this type on the part of the staff were coded as partially correct. As Denis Grogan has pointed out, material-finding queries that require the presentation of a range of information on the topic have neither a single definitive answer nor a point at which the search can be described as complete.23 Unless the pa- tron specifically was instructed to say "something on" or "a book about," which indicated less than a comprehen- sive search, all answers that did not in- clude a referral to reference or an appro- priate subject library were coded as partially correct. Frequency and Accuracy of Referrals One hundred and two questions were correctly answered, so that the other 62 questions should have been referred. As table 3 shows, of the 62 questions needing referral, only 39 referrals were made, or 63%. A total of 50 referrals were made, in- cluding 11 cases in which staff members correctly answered the questions and also made appropriate referrals. If the 50 refer- TABLE2 Questions Answerable at the Information Desk by Type of Question Type of question Answerable a:s~f:le Not answerable Row total Bibliographic 31 (88.6) 1 ( 2.9) 3 ( 8.6) 35 Subject (22.6) (6.7) (25.0) (21 .3) 13 (46.4) 12 (42.9) 3 (10.7) 28 Procedural (9.5) (80.0) (25.0) (17.1) 29 (87.9) 2 ( 6.1) 2 ( 6.1) 33 Ready reference (21.2) (13.3) (16.7) (20.1) 32 (94.1) 2 ( 5.9) 34 Directional (23.4) (16.7) (20.7) 32 (94.1) 2 ( 5.9) 34 Column totals (23.4) (16.7) (20.7) 137 (83.5) 15 ( 8.1) 12 ( 7.3) 164 (100.0) 100.0 Figures in parentheses to the right of raw numbers are row percentages. Figures in parentheses below the raw numbers are column percentages. The Effectiveness of an Information Desk 461 rals are considered, 70% (35) of all referrals made were correct. However, when one considers that only 24 of the 39 referrals made in the 62 instances where referral was needed were correct referrals, 38.7% of needed referrals were correct. The present study found that nonprofession- als referred 62.9% of the questions they could not answer or could only partially answer. These findings compare favorably with those of Halldorsson and Murfin that non- professionals referred or consulted on only 28% of the questions they were un- able to answer. 24 Marcia Myers found that 27.6% of 282 respondents who failed to find acceptable answers to telephoned questions made referrals. 25 Similarly, in Hernon and McClure's study of docu- ments libraries, only 56 referrals were made for 213 unanswered or incorrectly or partially answered questions, for a 26% re- ferral rate. 26 Even though Myers, and Hernon and McClure found similar refer- ral rates (27.6% and 26%, respectively) to Halldorsson and Murfin's 28%, total com- parability with the present study is not . possible, since Myers used telephone questions and Hernon and McClure stud- ied specialized reference services. Referrals were most often absent for en- compassing subject requests where book materials in the online catalog were identi- fied but the patron was not informed that, with additional assistance, more informa- tion might be found in periodical articles. Because surrogates were not asked to rec- ord follow-up questions, no conclusions can be drawn as to whether requests for clarification increased referrals for parti- ally answered questions. 'Z7 Referrals made by graduate assistants were 83.3% correct, those made by stu- dent assistants were 74.3% correct, and those made by library technical assistants were 54.4% correct. Fifty-eight questions could not be coded for staff level because of double-staffing using combinations of the three staff levels. The assumption had always been that the library technical as- sistants, as long-term, full-time employ- ees, would know the library better than the other groups. While this assumption may be true, this experience may give them a false sense of security about their knowledge of the library. Because student assistants usually work fewer hours and have not participated in the more orga- nized formal training, their lower rate of correct referrals is understandable. Grad- uate assistants also work at the reference desk and thus have a better idea of what is contained in the reference collection. Probably the most important factor, though, is that library technical assistants staff the information desk during weekly meetings, so that student assistants and graduate assistants may all attend. This exclusion from weekly meetings may be a significant factor in the quality of referrals, because there is not as much formal op- portunity to ask questions, to receive feed- back, to hear about other units, or tore- ceive instruction in new procedures. TABLE3 Referrals Made for Types of Answers Types of answers Correct Correct 11 (11.0) (31.4) Partially correct 5 (24.0) (14.3) Incorrect 2(11.0) (5.7) No answer 7 (78.0) (20.0) Immediate referral 10 (71.0) (28.6) Column totals 35 (21.0) Total referrals (columns 1,2,3) = 50 Quality of Referrals Partially Incorrect correct 3(14.0) 1 ( 5.0) (43.0) (12.5) 4 (29.0) (57.0) 6 (33.0) (75.0) 1 (11.0) (12.5) 7 ( 4.0) 8 ( 5.0) No Total referrals answers 91 (89.0) 102 (79.8) (62.0) 12 (57.0) 21 (10.5) (13.0) 10 (56.0) 18 (8.8) (11.0) 1 (11.0) 9 (.9) (5.0) 14 (9.0) 114 (70.0) 164 (100) (100) Figures in parentheses to the right of the raw numbers are row percentages. Figures in parentheses below the raw numbers are column percentages. 462 College & Research Libraries Staffing Patterns The presence of a second person at the information desk or of a reference librar- ian at the reference desk affected both the accuracy of answers given and the appro- priateness of referrals. While single staffed, information desk personnal were 65% accurate in handling questions and 77% correct in making referrals, compared to 82% accuracy in handling questions and 88% appropriate referrals made while double staffed. When a reference librarian was present at the reference desk, infor- mation desk staff were 76% accurate in handling questions and 84% correct in making referrals. When a reference librar- ian was not present at the reference desk, 58% accuracy in question handling and 73% appropriateness in referrals were measured. Patron Perceptions Surrogate patron perceptions tended to be very positive. Several comments were made about staff asking follow-up ques- tions. Other comments ranged from" out- standing assistance" and "it would be dif- ficult to provide a better experience than the one I had today," to "moderately helpful; not up to taking creative steps to- ward fulfilling patron request." Time Needed to Answer The majority of questions-153 or 93.3%-were answered in five minutes or less. Twelve patrons made voluntary com- m~nts about waiting for five to twenty nunutes to be helped or about being inter- rupted by other patrons or phone calls. These situations most often occurred dur- ing single-staffing times. It is important to mention the high volume of business and the variety of activities engaged in by the information desk staff-instruct patrons in the use of the online and card catalogs, answer ready reference questions, screen telephone calls, refer in-depth questions, and handle procedural and directional queries. PROBLEMS The sample size was too small to ade- quately test all facets of interest. Further studies need to ~xpand the number of questions or focus on particular types of July 1989 que~tions. ~ther difficulties of the study are inherent m the unobtrusive methodol- ogy. The surrogate patrons were not infor- mation specialists or library school stu- dents and probably did not bring much bias to the study. However, because these questions were not their own, the surro- ~ates probably did not pursue them as dil- Igently nor as tenaciously as an actual pa- tron might. The sheer number of surrogates caused instructional and train- ing problems. Their recording of data was not as complete nor as thorough as would have been desirable. Their failure in this regard was no doubt due, at least in part, to the demands of the in-person format which required that they leave the desk t~ complete the forms. SUGGESTIONS FOR FURTHER STUDY In order to gain more information about ~ow. ~on professionals approach subject mqmnes, and thus how to train them to answer subject requests, it would be ad- v~tageous to have transaction logs from onlme catalog terminals for these ques- tions. This would allow the computer to record approaches to subject headings, such as keyword-in-title searches versus searches in the subject authority file, rather than expecting surrogates to re- member this complicated process. Surro- gates would only have to submit the titles of books identified, along with the time the search took place and the identifica- tion number of the terminal used. The training problems related to incom- plete retrospective conversion of the card catalog are of critical importance. How much the card catalog/online catalog split affects the answers to bibliographic and subject questions was not addressed but this needs further study. This situ~tion raises an additional question: if highly ~ained graduate students and nonprofes- Sionals do not use an important tool like the card catalog for questions that cannot fully be answered any other way, how can we expect our patrons to use it? The importance of training-not only in- troductory but also continual training and ~ee~bac~-ca~ot be overstated. Training m hstenmg skills, question negotiation, reviewing familiar reference sources and strategies for subject searching and biblio- The Effectiveness of an Information Desk 463 graphic verification, while included in the present training program, must be ana- lyzed for deficiencies, redeveloped, and redesigned. Follow-up studies are needed to determine if certain training methods are more effective in fostering particular reference skills. Similar treatment of staff in establishing a minimum number of desk hours, in re- quiring attendance at meetings, and in standardizing training is also important. While variation in effectiveness of differ- ent types of staff members (graduate assis- tant, student assistant, or library technical assistant) has been addressed to some de- gree, further exploration of the factors that influence the performance of these differ- ent types of staff is necessary. 110ne of the most important conclu- sions to be drawn from this study is that graduate students and nonpro- fessionals staffing an information desk can be 70 percent effective in an- swering certain kinds of reference questions with proper training.'' CONCLUSIONS One of the most important conclusions to be drawn from this study is that gradu- ate students and nonprofessionals staff- ing an information desk can be 70.7% ef- fective in answering certain kinds of reference questions with proper training. Of the 164 questions, 83.5% were capable of being answered at the information desk. Of all questions, 62.2% were an- swered correctly, with another 8.5% im- mediately referred, for a total of 70.7% handled correctly. Subsequent referrals, of which there were 14 appropriate ones, were not considered in this figure. How- ever, the study indicates that adequate staffing levels and professional backup are needed in order for the information desk staff to perform at an optimum level. Their success or accuracy in answering ques- tions increases when a reference librarian is present at the reference desk, when the information desk is double staffed, or if the staff member asked a question of the surrogate to clarify his or her request (see appendix for statistical analysis). This information desk model is least ef- fective in handling subject questions. As table 2 indicates, 53.6% of the subject questions were not completely answer- able at the information desk because of the limited number of bibliographical tools available there. Also, subject questions tend to be more complex and require greater skill in the reference interview to encourage the patron to clarify better his or her needs. Murfin and Bunge's study similarly found that paraprofessionals were ''significantly less successful than professionals . . . when a subject search in the library catalog was used to answer the question. ''28 Additional training in subject searching and making appropriate refer- rals is needed for staff to deal effectively with these questions. It is questionable whether the additional training needed to bring staff competen- cies to a higher level is justifiable. There is evidence that a different model, wherein the type of questions that information desk staff attempt to answer is more re- strictive, may be more effective or effi- cient. Murfin and Bunge's study suggests that paraprofessionals who handle less complex questions and consult other staff members more frequently are more effec- tive. 29 Their findings are supported by this study, which indicates that more accurate answers and referrals are provided when staff members are available for consulta- tion. However, until we know what train- ing methods work, as well as their cost ef- fectiveness, it is difficult to attempt to determine whether additional training or a change in the scope of the questions an- swered at the desk is more appropriate. Referrals are a very important part of the concept of an information desk that an- swers only certain types of questions. All questions not answered fully should be re- ferred to the reference desk. If 83.5% of the questions are completely answerable, then the other 16.5% should have been re- ferred. In this study 38 questions, or 23%, did not receive appropriate referrals. Writ- ten guidelines stressing referral to the ref- erence desk appear to be inadequate. Spe- cific examples and case studies must be used in training to highlight the impor- 464 College & Research Libraries tance of follow-up questions and referral. The analysis found that the quality of re- ferrals is significant for the number of staff at the desk, the presence of a librarian, the clarification of the question, and the level of the staff member at the desk (see analy- sis in: appendix). The findings of this study suggest that library technical assistants, who do not attend weekly meetings and have no opportunity to discuss questions in a methodical manner, are not getting the training they need to make appropri- ate referrals. Staff members are reluctant to refer patrons to a professional who may not be immediately available. It should be emphasized, however, that this step must be taken if the patron is to complete the transaction. Staff members need to realize that complete and accurate information is more important than the inconvenience that a delayed referral creates for the pa- tron. Supervisors of information desks need to establish communication channels with each other to exchange ideas and compare problems concerning turnover and re- July 1989 cruitment of personnel, staffing patterns, training for interpersonal skills as well as the knowledge of reference sources and tools, staff development, and standards of performance. The problems of nonprofes- sionals are sometimes unique and merit separate consideration from those of refer- ence professionals. One approach might be to create a discussion group on infor- mation desks or nonprofessionals in refer- ence service within ACRL or RASD. By sharing our successes and failures, per- haps we can collectively improve the per- formance of all staff at information desks. This study has shown that nonprofes- sionals at an information desk can provide effective service to library patrons asking particular kinds of questions. Now that base-line job performance data has been gathered, the training program can be changed and tested in future years to see if the cumulative effectiveness of the infor- mation desk can be improved. This appli- cation, toward ''improving the quality of reference service,'' is the ultimate goal of this study. 30 REFERENCES AND NOTES 1. Laura M. Boyer and William C. Theimer, Jr., quote a Canadian library administrator as saying that 85% of reference questions can be answered by nonprofessional staff in "The Use and Training of Nonprofessional Personnel at Reference Desks in Selected College and University Libraries,'' Col- lege & Research Libraries, 36:193 (May 1975); and Jeffry W. St. Clair and Rao Aluri, in "Staffing the Reference Desk: Professionals or Nonprofessionals," Journal of Academic Librarianship 3:149-53 Ouly 1977), analyzed questions asked at a reference desk and found that 62.1% were directional or instructional, and an additional17. 9% were ready reference, card catalog book checks, and simple movie or book review requests, which could also be handled by a "nonprofessional who has par- ticipated in a well-conducted reference orientation program ." 2. Boyer and Theimer, "The Use and Training of Nonprofessional Personnel," p.197, and Martin P. Courtois and Lori A. Goetsch, ''Use of Nonprofessionals at Reference Desks,'' College & Research Libraries 45:389 (Sept. 1984). 3. More information about the training program is available in Beth S. Woodard and Sharon J. Van Der Laan, "Training Preprofessionals for Reference Service," Reference Librarian 16:233-54 (Win- ter 1986/1987). 4. Beth J. Shapiro, "Trying to Fix What's Wrong With Reference," Jour)ial of Academic Librarianship 13:289 (Nov . 1987). \ 5. Shapiro, "Trying to Fix What's Wrong With Reference," p.289 . 6. Charles R. McClure and Peter Heron, "Unobtrusive Testing and the Role of Library Manage- ment," Reference Librarian 18:83 (Summer 1987). 7. McClure and Hernon, "Unobtrusive Testing," p.77. 8. See the following for reviews of reference service evaluation: Elizabeth Opal Stone, "Methods of Evaluating Reference Service," Library Journal67:296-98 (April1, 1942); Samuel Rothstein, 'Mea- surement and Evaluation of Reference Service," Library Trends 12:456-72 Oanuary 1964); Terry L. Weech, "Evaluation of Adult Reference Service," Library Trends 22:315-35 Oanuary 1974); F. Wilfred Lancaster, The Measurement and Evaluation of Library Services (Washington, D.C.: Informa- tion Resources Press, 1977), p .73-136; Rosemary Ruhig DuMont and Paul F. DuMont, "Measur- The Effectiveness of an Information Desk 465 ing Library Effectiveness, A Review and an Assessment," Advances in Librarianship 9:104-41 (1979); Ronald R. Powell, "Reference Effectiveness: A Review of Research," Library and Infonna- tion Science Research 6:3-19 (1984); Marilyn VonSeggern, "Assessment of Reference Services," RQ 26:487-96 (Summer 1987); and Frances Benham and Ronald R. Powell, Success in Answering Refer- ence Questions: Two Studies (Metuchen, N.J.: Scarecrow, 1987), p.26-34, 159-75. 1 9. Charles A. Bunge," Approaches to the Evaluation of Reference Services," in Evaluation and Scien- tific Measurement of Libraries and Infonnation Centres, ed. F. W. Lancaster and C. W. Cleverdon (Ley- den: Nordhoff, 1977), p.42. 10. A. C. Hamblin, Evaluation and Control of Training (London: McGraw-Hill, 1974), p.46-73, 111-63. 11. Marjorie Murfin and Charles Bunge, in "Evaluating Reference Service From the Patron Point of View: Some Interim National Survey Results," Reference Librarian 11:175-82 (Fall/Winter 1984), provide preliminary results of a survey in which patrons reported when they found exactly what they wanted, which was compared with the size of library, amount of time spent, the staff mem- ber's report of being busy, the practice of merely directing or actually helping to conduct the search, and the level of the staff member-professional or nonprofessional. 12. Hamblin, p.70. 13. Terry L. Weech and Herbert Goldhor, "Obtrusive Versus Unobtrusive Evaluation of Reference Service in Five Illinois Public Libraries: A Pilot Study," Library Quarterly 52:317 (Oct. 1982). 14. See the following for discussion of the use of unobtrusive methodology and its application in li- braries: Terry L. Weech, "Evaluation of Adult Reference Service," p.326-29; F. Wilfred Lancaster, The Measurement and Evaluation of Library Services (Washington, D.C.: Information Resources Press, 1977), p.91-109; Marcia J. Myers, "The Effectiveness of Telephone Reference/Information Ser- vices in Academic Libraries in the Southeast,'' (Ph.D. dissertation, Florida State University, 1979), p.10-28; Alvin M. Schrader, "Performance Standards for Accuracy in Reference and Information Services: The Impact of Unobtrusive Measurement Methodology," Reference Librarian 11:197-214 (Fall/Winter 1984); Charles R. McClure and Peter Hernon, Improving the Quality of Reference Service for Government Publications, ALA Studies in Librarianship no.10 (Chicago: American Library Assn., 1983), p.11-16; Peter Hernon and Charles R. McClure, Unobtrusive Testing and Library Refer- ence Services (Norwood, N.J.: Ablex, 1987), p.1-25. 15. Geraldine B. King and L. Rachel Berry, "Evaluation of the University of Minnesota Libraries Ref- erence Department Telephone Information Service, Pilot Study," (Minneapolis: University of Minnesota Library School; Arlington, Va.: ERIC Document Reproduction Service, ED 077 517, 1973); Marcia J. Myers, ''The Accuracy of Telephone Reference Services in the Southeast: A Case for Quantitative Standards,'' in Library Effectiveness: A State of the Art (Chicago: American Library Assn., 1980), p.220-31; Janine Schmidt, "Evaluation of Reference Service in College Libraries in New South Wales, Australia,'' in Library Effectiveness, A State of the Art (Chicago: American Library Assn., 1980), p.265-94; Janine Schmidt, "Reference Performance in College Libraries," Australian Academic and Research Libraries 11:87-95 Oune 1980); Jassim M. Jirjees, "The Accuracy of Selected Northeastern College Library Reference/Information Telephone Services in Responding to Fac- tual Inquiries,'' (Ph.D. dissertation, Rutgers University, 1981); the Myers and Jirjees studies are also reported in Marcia J. Myers and Jassim M. Jirjees, The Accuracy of Telephone Reference/Informa- tion Services in Academic Libraries: Two Studies (Metuchen, N.J.: Scarecrow, 1983); McClure and Hernon, Improving the Quality of Reference Service for Government Publications; Peter Hernon and Charles R. McClure, "Unobtrusive Testing: the 55 Percent Rule, Library ]ournal111:37-41 (April 15, 1986); Peter Hernon and Charles R. McClure, "The Quality of Academic and Public Library Reference Service Provided for NTIS Products and Services: Unobtrusive Test Results," Govern- ment Infonnation Quarterly 3:117-32 (May 1986); Charles R. McClure, Linking the U.S. National Tech- nical Infonnation Service with Academic and Public Libraries (Norwood, N.J.: Ablex, 1986); Hernon and McClure, Unobtrusive Testing and Library Reference Service. 16. Hernon and McClure, "Unobtrusive Reference Testing: The 55 Percent Rule," p.38. 17. Charles A. Bunge, "Professional Educ:;ation and Reference Efficiency," (Ph.D. dissertation: Uni- versity of Illinois, 1967); Egill A. Halldorsson and Marjorie E. Murfin, ''The Performance of Profes- sionals and Nonprofessionals in the Reference Interview, College & Research Libraries 38:385-95 (September 1977); and Marjorie E. Murfin and Charles A. Bunge, "Paraprofessionals at the Refer- ence Desk," Journal of Academic Librarianship 14:10-14 (March 1988). 18. Bunge, "Professional Education and Reference Efficiency," p.145. 19. Halldorsson and Murfin, p.394. 20. Murfin and Bunge, "Paraprofessionals at the Reference Desk," p.10. 21. The University of fllinois Pocket Facts 1987 (Urbana, ill.: University of illinois Office of Public Affairs, April1987) . . 466 College & Research Libraries July 1989 22. Myers and Jirjees, The Accuracy of Telephone Reference/Information Services in Academic Libraries, p.181. 23. Denis Grogan, Practical Reference Work. Outlines of Modern Librarianship (London: Clive Bingley, 1979), p.34. 24. Halldorsson and Murfin, p.393. 25. Myers, "The Effectiveness of Telephone Reference/Information," p.138. 26. Peter Hernon and Charles R. McClure, "Referral Services in U.S. Academic Depository Libraries: Findings, Implications and Research Needs," RQ 22:155 (Winter 1982). 27. Ralph Gers and Lille J. Seward, in "Improving Reference Performance: Results of a Statewide Study," Library ]ournal110:34 (November 1, 1985), suggest that asking follow-up questions is strongly associated with good reference performance. 28. Murfin and Bunge, "Paraprofessionals at the Reference Desk," p.13. 29. Ibid, p.14. 30. Peter Hernon and Charles R. McClure, ''Where Do We Go From Here? A Final Response?" Journal of Academic Librarianship 13:284 (Nov. 1987). . APPENDIX The Statistical Package for the Social Sciences (SPSS) was used to run cross-tabulations on several factors to test for relationships. The chi- square test, which tests for independence of the variables, not the strength or form of the associ- ation, was run, as well as a test for the linear re- lationship as expressed by the Pearson Correla- tion Coefficient. The correctness of answers was found to be significant in the chi-square test for the pres- ence of the reference librarian at the reference desk at .0398, for the number of staff at the in- formation desk at .0494, and by total staffing patterns at .0287, as shown in table 1. These sig- nificance levels are well below the commonly accepted level of .05, or 5 in 100, when the ob- served difference can be attributed to chance. In other words, there is a relationship between the staffing patterns and the effectiveness of the in- formation desk. Other variables that were not significant were the type of question, the level of the information desk staff member (graduate assistant, student assistant, or library technical ~ assistant), the amount of time spent with the question, or the type of patron. Other studies have suggested that the appearance and per- ceived importance of the patron may have a sig- nificant effect on the quality of reference service provided. 1 This finding is not supported in the present study (see table 1). In the Pearson correlation coefficients pre- sented in table 2, 0.00 represents ·a lack of rela- tionship between the two variables. The strong- est relationships would be represented by a positive or negative 1.00. The answer provided was more likely to be correct if a reference li- brarian was staffing the reference desk, if the information desk was double staffed, or if the staff member asked a question of the surrogate patron to clarify his or her request. Correct an- swers were also more likely to be given for cer- tain types of questions. Directional, proce- TABLE 1 Chi-Square Correlation Table Variables Correctness of answers correlated with variables: By presence of reference librarian By number of staff at desk By staffing levels (combinations) Perceptions: Friendliness by staffing levels Politeness by staffing levels Friendliness by presence of a reference librarian Politeness by presence of a reference liorarian Answerable: By type of question • After Yates Correction tLow numbers in several cells . The test may not be as powerful here. Value of chi-square Level of significance 4.22786 .0398* 3.86334 .0494* 7.10207 .0287 7.34400 .0254t 7.28642 .0262t 4.28542 .0384* 4.24629 .0393* 34.66797 .0000 The Effectiveness of an Information Desk 467 dural, ready reference, subject, and bibliographic inquiries were handled with de- creasing effectiveness. No significant relationships were found for factors involving the quality of referrals in the Chi-square test, but some linear relationships were significant. In table 2, the quality of ref- ferals were significant for the number of staff at the desk, the presence of a reference librarian, clarification of the question, and level of staff member (graduate assistant, student assistant, and library technical assistant) at the desk. While perceptions of friendliness, quickness, politeness, helpfulness, interest, enthusiasm, and competence by surrogate patrons tended to be positive, there were some significant differ- ences. The patrons' perceptions of friendliness varied by total staffing levels and by the pres- ence of a reference librarian at .0254 and .0384 levels of significance, respectively, in the chi- square test in table 1. The Pearson Correlation Coefficient in table 2 for friendliness (.21909) by the presence of a reference librarian was also significant at .0033. Politeness by staffing levels and by presence of a reference librarian were significant by .1262 and .0393, respectively, in the chi-square test. Politeness by presence of a reference librarian was also significant in the Pearson R test (.21407 at .0034 significance). The Pearson R test also showed a significant dif- ference in the perceptions of staff interest by to- tal staffing level of .14499 at . 0443 significance (see table 2). None of the variations in surrogate patrons' perceptions were significant for types of questions or for the presence of a second staff member at the desk. None of the relationships are particularly strong, but the evidence does suggest that graduate assistants and nonprofessionals in this setting are more effective when working with another person and when a reference li- brarian is present. Both referrals and the an- swering of questions are more likely to be cor- rect when the question is clarified. While the time spent in answering individual questions increased when reference librarians were not present, this additional time did not mean that correct answers were more forthcoming. TABLE2 Pearson Coefficient Correlation Table Variables Quality of referrals: By number of staff at desk By clarification question By presence of reference librarian By level of staff member at desk Correctness of answers: By presence of reference librarian By number of staff By clarification of question Bytimespentanswering By type ol question Time spent answering questions: By presence of reference librarian Perceptions of surrogate patrons: Uninterested by staffing levels Friendliness by presence of reference librarian Politeness by presence of reference librarian Answerable at the information desk: By type of question Pearson R value between -1.0 and + 1.0 -.12893 -.13030 .12290 .20341 .17529 -.16750 - .12578 .17654* -.18832 .19749* -.14499 .21909 .21407 -.20227 Level of significance .0380 .0482 .0585+ .0182 .0124 .0160 .0543+ .0123 .0079 .0059 .044m .0033 .00 .0047 *Although - and + signs usually indicate direction of relationships, all relationships are positive except those marked with an asterisk (*), which are negative. t Note: These are borderline significant. t Low frequencies in the cells may make this test Jess powerful. REFERENCE 1. McClure and Hernon, "Improving the Quality of Reference Service for Government Publica- tions," p.33.