College and Research Libraries Patron Online Catalog Success Rosemary Thome and Jo Bell Whitlatch This project explored the role of the online catalog in library user success. Three different studies were conducted to provide a more in-depth picture of online catalog use in three aspects of library research-as part of the reference process, as used by unassisted users, and as an intermediate step in obtaining actual documents. The methodology provides a model for surveying other library services or products. utomated products, espe- cially online catalogs, have become the norm in academic libraries. The online catalog is now a major resource utilized in the process of providing reference assis- tance to users and in providing assis- tance to library users operating in a self-service mode. Librarians and library users alike have adopted them enthusi- astically. Although user enthusiasm and satisfaction with online catalogs are known to be high, individual libraries have more limited information on the role of the library catalog in library user success. To obtain a more comprehen- sive understanding of the role of the on- line catalog in user success, individual library studies must be designed to study both patrons who use the online catalog with reference assistance and those who use the catalog unassisted. Also, such studies must employ a vari- ety of quantitative and qualitative data- collection methods and analyses and consider both librarian and patron per- spectives concerning success. The purpose of this project is to ex- plore how the online catalog contrib- utes, or does not contribute, to patron success. This project, in using a variety of methodologies, also seeks a multifac- eted view of the topic. Key questions that the project is designed to answer are: (1) Are students really getting ap- propriate assistance when using the online catalog? (2) What are the most common searching successes and fail- ures observed for people using the on- line catalog without staff assistance? (3) How effectively is the online cata- log used by librarians providing refer- ence assistance compared to other sources-reference books, electronic and printed indexes? and (4) Are students able to find the material they located in the online catalog on the shelves-if not, why not? The paper briefly reviews relevant studies of online catalogs and reference success, and describes the methodology and results of three different online cata- log studies: (1) the Reference Transac- tion Assessment Survey; (2) the online catalog study of unassisted users; and (3) the document availability study. Fi- nally, the paper concludes by discussing the implications for reference practice, Rosenu:~ry Thorne is Head of Reference and Instructional Services and Jo Bell Whitlatch is Reference Librarian and History Selector at San Jose State University, San Jose, California 95192-0028. The authors would like to thank the six librarians who participated in the Online Catalog Success Study: Edith Crowe, Mario Guzman, Linda Johnson, Mengxiong Liu, Judy Reynolds, and Lorene Sisson, as well as all members of the Reference Department and the Government Publications Department who participated in the RTAI study. They also thank Marjorie Murfin and Charles Bunge for their assistance in analyzing the RTAI results. 479 480 College & Research Libraries for the online catalog interface, and for study methodology. LITERATURE REVIEW Online Catalog Studies Many studies have been conducted concerning the role of online catalogs in providing assistance to users in a self- service mode. Jon Hufford observes that the most accurate studies of catalog use have been done through interviews with users when they are searching the cata- log. This technique requires more time to gather data, but is likely to yield more precise and reliable information. A seri- ous limitation of this method is that us- ers might not behave in the same way as when they are unobserved.1 In line with this focus on users, Walt Crawford notes that much of the research has asked the wrong question. The question is not whether users can identify a given piece of information, but whether they got the information wanted, how quickly they got the information, and whether the process was satisfactory.2 However, user satisfaction as a meas- ure is not without difficulties. User sat- isfaction is multidimensional, and there has been little recognition of the com- plex dimensions that underlie the meas- ure. Prudence Dalrymple and Douglas Zweizig found that users who spent a longer time searching the catalog were likely to assess the results obtained less positively than those who spent a shorter time searching. Conversely, us- ers who enjoyed their experience search- ing in the online catalog (easier to use, fun, and so forth) viewed the results ob- tained more positively than those who found the online catalog search less en- joyable.3 Past studies of online catalogs have identified subject rather than known- item (author/title) searches as the type of searches creating the most difficulties for users. Ray Larson has summarized the major problems with subject access in current online catalogs: users' lack of knowledge of Library of Congress Sub- ject Headings (LCSH); users' problems with mechanical and conceptual aspects of query formulation; searches that re- November 1994 trieve nothing; searches that retrieve too much; and searches that retrieve records that do not match what the user had in mind.4 Larson found that subject search- ing is the type of search most likely to fail-in a large university library, only 12 percent of subject searches retrieved be- tween one and twenty items.5 Other recent studies confirm Larson's observation. At North Carolina State, Hunter found that subject searching was the most used (at 52 percent) but the least success- ful search-62 percent of all subject searches resulted in zero hits.6 In No- vember 1992, 32 percent of searches on MELVYL, the University of California online catalog, yielded zero retrievals and the average retrieval set was ninety- eight titles. 7 User satisfaction is multidimensional, and there has been little recognition of the complex dimensions that underlie the measure. A few studies provide more detail on user behaviors when subject searching. Methods of uncontrolled vocabulary (i.e., keyword) access vary among online catalog systems, making it difficult to compare results of subject searches from different systems. In a system that did not provide keyword access to controlled vocabulary subject headings, Thomas Peters and Martin Kurth found that over 58 percent of users began with an uncontrolled attempt. Slightly less than half of subject search sessions also contained at least one title keyword search statement.8 In a study at the Uni- versity of Toronto Libraries, observers recorded protocols for one hundred on- line catalog search sessions. An analysis of 42 zero hit searches revealed that key- word subject, keyword title, or title searches using the original query from the user's zero hit subject search were more fruitful than new searches con- structed from cross-references provided by LCSH. Joan Cherry suggests that LCSH cross-references in online catalog or user training in use of LCSH will not solve the problems with the majority of zero hit subject searches. A more prom- ising approach may be to provide online catalog software that converts zero hit subject queries to other types of subject searches.9 Studies of Reference Success The role of the online catalog in pro- viding reference assistance to users has not been evaluated extensively. How- ever, Charles Bunge, Marjorie Murfin, and Gary Gugelchuk have developed the Reference Transaction Assessment Instrument (RTAI), which provides an opportunity to assess the effectiveness of the online catalog as a resource in providing reference assistance to users. The RTAI has been judged reliable and valid, has been utilized by many librar- ies evaluating reference services, and contains a section that provides informa- tion on whether the catalog was used in the process of answering a reference query.10 In an analysis of preliminary re- sults from the RTAI, two factors have emerged that appeared to undermine the results of reference service seriously. One factor was that the librarian was too busy and the other concerned the librar- ian's practice (either from habit or neces- sity) of directing or suggesting only rather than helping with the search.11 As a result of analyzing data combined from many individual libraries, this syn- drome of factors has been identified as characteristic of the least successful li- braries: significantly more directing rather than actually helping users with searches, significantly more transactions of two minutes or less, significantly more one-source transactions, signifi- cantly more reports from users that they did not receive enough time or help, sig- nificantly more communication difficul- ties with librarians reported by users, and significantly less agreement on pa- tron success by librarians and users- that is, lower sensitivity to user feelings by librarians. 12 METHODOLOGY The University Library at San Jose State University serves approximately Patron Online Catalog Success 481 thirty thousand students, a significant percentage of whom are members of di- verse social and ethnic groups, includ- ing persons with disabilities, African Americans, Hispanics, Asians, first-gen- eration college students, and older re- turning adult students. The campus has two library facilities: Clark Library, which houses current collections and most library services, and Wahlquist Li- brary, which holds older research mate- rials, reserves, and special collections. Library collections include more than 900,000 volumes and 3,127 periodical subscriptions. Located in urbanized, high-technology Silicon Valley, the uni- versity is a member of the twenty-cam- pus California State University system. The library introduced its first online catalog, an Innovative Interfaces prod- uct, to the public in April1991. Remote access to the catalog was made available through the campus computer cente.r. Access by title and subject keyword and permuted subject heading was available following an upgrade in fall1991. Prior to the introduction of the online catalog, patrons were able to use several public terminals to search the CLSI circulation system by author, title, and title algo- rithm. In January 1992 the library with- drew its card catalog, which had been closed in November 1990. For the project, three studies were undertaken. The first study used the Reference Transaction Assessment In- strument (RTAI). This was followed by the Online Catalog Success Study, which was conducted concurrently with the Document Availability Study. Reference 1Tansaction Assessment Instrument (RTAI) In spring 1992 the RTAI survey was used to collect information on success in reference and information seeking, as perceived by both the patron and the library staff member. The authors se- cured the cooperation of the Reference Department and the Government Publi- cations Department in carrying out the survey. Paired response forms for the patron and the staff member (librar- ian, support staff, or student assistant) , 482 College & Research Libraries permitted the surveyors to examine the response to the transaction from the viewpoint of both sides of the service desk. Separate standardized forms were provided for directional/ information queries and for reference questions. For a copy of the RTAI questions, readers should consult the article published in College & Research Libraries by Marjorie Murfin and Gary Gugelchuk.10 A useful feature of the RTAI was comparative data provided for other academic librar- ies that have participated in the RTAI. In our analysis, we will refer to the results for academic libraries of similar size for comparison purposes. The authors used a total of 300 forms, 150 each for direc- tional/ information and reference ques- tions. For RTAI sample size, Gary Gugelchuk recommends one hundred se- quential transactions. His study showed that with a 95 percent confidence level f9r a sample size of one hundred items per sample, the true success score for a library will lie within plus or minus ten percent- age points of the sample observation. 13 The twenty-four librarians and para- professionals in the Reference and - Government Publications departments participated in one-hour training ses- sions. These included a review of the logistics of the survey-schedule; where to place completed forms, etc.; review of definitions of terms; review of each questionnaire, question by question, us- ing sample reference questions as exam- ples; and clarification of any questions arising during the training. Each staff member was given a test set of forms, with the recommendation that they re- view them while at the service desk to further familiarize themselves with the forms and the process. Training for student assistant supervisors was very important, because these individuals trained the student assistants. The survey was conducted during the third week in April, a time of the semes- ter when library activity is high. Times were chosen to include at least one morning, one afternoon, one evening, and one weekend day. Signs informing patrons of "Survey in Progress" were posted at each service area. Each staff November 1994 member participating in the survey in- formed the library patrons of the survey, asked for their participation, and handed out the form with a #2 pencil. Clearly marked boxes for forms were pointed out; these boxes were located sufficient distance from the service desk that patrons could maintain a feeling of privacy in their responses. All surveys were completed during this period. Completed survey forms were mailed to the School of Library and Information Studies, University of Wis- consin-Madison. The most cost-effective method of utilizing the RTAI forms was to order copies from Charles Bunge at the School of Library and Information Studies, University of Wisconsin-Madi- son. Prices included data processing and printing results. Comparative data from other libraries were supplied as well. The computer-generated results, with comparisons with other libraries, were returned by mail. Online Catalog Study of Unassisted Users The purpose of the Online Catalog Study was to learn more about the suc- cesses and failures of people using the online catalog without staff assistance. A set of instruments was developed by the authors, including an interview I obser- vation section and a brief questionnaire for the patron. Also, a document avail- ability form developed by Paul Kantor was utilized as part of the study. 14 The three forms used in this survey were in numbered packets, which permitted comparison of the information on all three forms for the same observation. The librarian form asked observers to note the various online indexes (author, title, keyword, etc.) used by the patron, the order in which these were selected, and the level of difficulty the patron ap- peared to have in using them. Librarians were also asked to record the use of the various display options and limit com- mands, as well as other special features of the catalog, and starting and ending times for each patron search observed. The patron questionnaire asked the patrons to evaluate their experience, in- eluding success in locating what was needed, type of material sought, level of satisfaction, and perceived ease of use of the online catalog. A five-level ranking of success, similar to that used in the RTAI survey instrument, was used for appropriate questions. A final question asked for suggestions to improve the on- line catalog. The document availability form, "Did You Find It?" asked participants to use the sheet as "scratch paper" to note what they found in the catalog and wished to look for in the library. In addition to space for the author/title/journal name and call number, the form asked users to mark whether they found the item. Ad- ditional questions concerned the ur- gency of their need for the material, their perception of their success in finding materials, and the purpose for these ma- terials. A box was placed at the library exit for users to leave the forms. The survey was carried out by the authors and six other reference librari- ans who volunteered to participate. The volunteers reviewed draft survey in- struments and provided valuable feed- back in the development process. The instruments were then tested in January 1993 on several student assistants in the Reference Department. Each librarian was asked to choose four hours in which to participate in the nineteen-day survey period in March 1993. Librarians were given twelve pack- ets, each containing an observation form, a patron questionnaire, and a docu- ment availability form; they were asked to complete at least ten observation forms, for a total of eighty responses. A minimum of eighty observations per- mitted the library to survey a consider- able number of searches, yet was manageable within existing workloads. While greater sample sizes ensure that estimates are more precise and reliable, larger sample sizes require more expen- ditures for collecting data, especially when interviews are being carried out. A total of ninety-three completed forms was returned by the librarian-observers. Seventy-seven patron questionnaires and forty-three document availability Patron Online Catalog Success 483 patron forms were returned. The online catalog terminals located adjacent to the reference desk on the first floor of Clark Library were used for the survey. A ran- dom process was developed to guide the librarian in selecting a patron to ap- proach. Each librarian was given a set of small cards, each representing one of the eight clusters of three terminals. The li- brarian, prior to each observation pe- riod, shook up and drew the cards to establish a random approach to the ter- minals. Written sampling procedures and a group meeting to discuss the proc- ess prepared the observers. The librarian observer, using a recom- mended script complied with the uni- versity's Human Subjects in Research requirements, explained the project and procedure and asked for the individual's cooperation. The librarian stood directly behind the patron using the catalog and recorded information on the observation form. When the patrons had concluded the search, they were asked to complete the one-page questionnaires and return them to the observers, The observers were requested not to offer suggestions or advice to the user; however, they briefly could answer questions asked by the user. Data were analyzed using SPSS frequencies and cross tabulations. To relate the results of the small obser- vation study with overall online catalog usage, an analysis of the online catalog transaction logs was carried out. A ran- dom sample of three hours of the thirty hours of the online study (a 10 percent sample) was used; this was 1,560 searches. This random sample was used to check whether the searches observed were representative of the general type of searching done during this time pe- riod. These logs, arranged by index used (that is, author, title, keyword, subject heading, and call number), listed the terms entered by the patrons. A student assistant with extensive library experi- ence searched each term exactly as en- tered by patrons and recorded the number of hits found. For each index, a statistical summary was prepared to show how many searches had no hits, how many had one hit, two hits, etc. 484 College & Research Libraries Document Availability Study Concurrent with the Online Study, a Document Availability Study investi- gated the users~ success in locating ma- terials identified through the online catalog search. A sheet was provided to each subject in· the Online Study as de- scribed above. An experienced student assistant was hired to determine the location of each item patrons reported as not found. He checked stacks, sorting areas, circulation records, etc., to ascertain if possible the reason for failure in locating the items. This follow-up took place as soon as pos- sible after the patron search, to assure a minimum of material relocation. <:-, RESULTS AND DISCUSSION Survey Population This information is presented so that the reader may obtain an understanding of the nature of the student body. The demographic information was collected because the authors wanted to ensure that the different class levels and ethnic groups were included in the survey. The comparisons provide evidence that both survey samples are fairly representative of students enrolled in the university by class<1evel and ethnicity. Tables 1 and 2 record demographic information for students participating in both the Reference Transaction Analysis Instrument and the Online Catalog Study, relating it to the campus popula- tion of San Jose State University. Of the 125 persons asking reference questions in the RTAI study, 116 (94 percent) were students. Of the 77 persons completing questionnaires in the Online Catalog Study, 71 (92 percent) were students. The Reference Transaction Assessment Survey Because we wished to compare the results of the RTAI with the Online Cata- log Observation Study conducted by ref- erence staff at catalog terminals in the reference area, we have chosen to use only those results for the Reference De- partment, rather than include the results . for the Government Publications De- November 1994 partment. Among the questions on the library staff form, one asked for an iden- tification of types of sources used, recom- mended, or interpreted. The standard RTAI analysis does not include the analysis of all causes of failures for specific book questions because these questions are generally quite straight- forward. Including these questions would have greatly increased the cost of the analysis. Of the 125 completed forms, 22 involved questions concerning locating a specific book. They were not included in the RTAI analysis, which contained comparisons with other li- braries. Of the remaining 103 transaction forms, 87 had the type of source marked. Of the 87, 30 (34 percent) indicated that the online catalog was used. Of these 30 cases wJ:tere the online catalog was used, only eleven, or slightly over one-third, found exactly what was wanted. In the comparison group, libraries of similar size, just over half of patrons found ex- actly what was wanted when the catalog was used. Table 3 shows the percentage of patrons who found exactly what was wanted when specific types of sources were used for both San Jose State Uni- versity (SJSU) and the comparison group of large academic libraries. The demographic information was collected because the authors wanted to ensure that the different class levels and ethnic groups were included in the survey. The low rate of reference transaction success when the online catalog was a source may result from several different situations. The accepted philosophy of the Reference Department is to teach the patron how to use the tool, rather than to find the answer for the patron. As table 3 shows, within the three types of sources where a teaching approach is more likely to be used, periodical in- dexes, online catalog, and CD-ROM, the success rate is below 50 percent. The librarian will generally get the patron started in the use of these three types of Patron Online Catalog Success 485 TABLEt CLASS LEVEL OF STUDENT PARTICIPANTS, STUDENT BODY RTAI Online Catalog Study SJSU Student Body Status Number % Number % Number % Freshman 7 6.0 6 8.5 3,136 10.6 Sophomore 10 8.6 4 5.6 2,953 10.0 Junior 26 22.4 19 26.7 6,383 21.5 Senior 52 44.8 29 40.8 10,468 35.3 Graduate Student 21 18.1 13 18.3 6,686 22.6 Total 116 99.9 71 99.9 29,626 100.0 Note: Other participants included faculty I staff (RTAI 1, Online catalog 3), continuing education (Online catalog 1), and not affiliated with the University (RTAI6, Online catalog 2). Total of RTAI respondents is 125 (116 + 7 other + 2 with status unknown). TABLE2 ETHNIC BACKGROUND Online Catalog Success Study SJSU Student Body Background Number % Number % African American/Black 5 6.5 1,227 4.1 American Indian/ Alaskan 0 0.0 194 0.7 Asian 11 14.3 6,814 23.0 Chicano /Hispanic 12 15.6 2,962 10.0 Filipino 3 3.9 1,372 4.6 Pacific Islander 1 1.3 154 0.5 White 40 51.9 13,888 46.9 Unknown 5 6.5 3,015 10.2 Total 77 100.0 29,626 100.0 Note: Ethnic backgrounds for the Online Catalog Success Study and for the SJSU student body are self-re- ported. The ethnic background of participants is not available for the RTAI study. TABLE3 SUCCESS RATES FOR TYPES OF SOURCES Patrons Report Found Exactly Total and Completely Satisfied Number Number Comparison Type of Source Used SJSU SJSU %SJSU group% Reference books 33 20 60.6 56.3 Own knowledge 17 9 52.9 61.6 Consult with another staff member 2 1 50.0 33.6 Indexes to periodicals 29 11 37.9 52.3 Online catalog 30 11 36.7 51.9 Computer database, CD-ROM 25 9 36.0 51.0 Refer patron to another department 8 2 25.0 29.6 Note: Success levels based on 103 reference transactions. Comparison group is large academic libraries and is supplied as part of the RTAI analysis. 486 College & Research Libraries November 1994 TABLE4 PATRON REPORTS OF TIME, HELP, AND EXPLANATION Patrons report did not receive enough time Patrons report not enough help and explanation Librarians report patrons need extra help Librarians 'report patrons return frequently %SJSU 9.7 12.6 3.9 1.9 % Comparison Group 6.0 8.5 8.7 5.4 TABLE 5 SUCCESS LEVELS BY TYPES OF SOURCES AND BUSY CONDITIONS SJSU Comparison Large Acad. All Academic Libraries %Busy %Not Busy %Busy %Not Busy %Busy %Not Busy (N=58) (N = 45) (N= 659) (N = 1,287) (N = 1,783) (N=4672) Reference books Complete success 50.0 76.9 48.8 61.3 54.2 59.1 Adding mostly 60.0 84.6 58.7 72.1 65.3 70.7 Indexes to periodicals Complete success 35.0 44.4 43.0 56.2 51.5 53.6 Adding mostly 45.0 55.6 55:8 68.6 62.8 66.1 Online catalog Complete success 28.6 55.6 47.4 52.7 51.1 51.4 Adding mostly 33.3 66.7 62.3 64.7 64.0 63.9 Computer database, CD-ROM Complete success 31.6 50.0 48.6 58.7 51.2 57.5 Adding mostly 36.8 83.3 56.1 71.3 63.0 67.9 Note: "Complete success" means that patrons reported finding exactly what was wanted and being com- pletely satisfied. "Adding mostly" means that patrons reported finding exactly what was wanted or finding what they wanted within limits and being completely satisfied. sources and provide further assistance if it is needed. In the case of the online catalog, the librarian will usually show the patron the rudiments of its use and recommend an index to use (author, sub- ject, etc.) and an appropriate term with which to begin the search. The patron is then referred to online catalog terminals located at the far end of the room. It appears that this teaching approach, although frequently necessary because of the volume of business (the librarian reports "busy" or "very busy" in 56 per- cent of cases, in comparison to 29 per- cent of cases at libraries of similar size), may not assure patrons the high level of library success desired. Another prob- lem may be the librarian's assumption that the patron understands the instruc- tions well enough to carry on inde- pendently. The survey results tend to support these theories. In comparison with other large academic libraries, a higher proportion of SJSU patrons re- ported that they had not received enough time, help, and explanation (ta- ble 4). Also at SJSU, librarians reported that patrons needed extra help andre- turned less often than in the comparison group of libraries. In comparison with other libraries, SJSU patron success rates for questions involving use of the different types of answering sources were influenced dra- matically by the volume of demand for reference services. This is particularly true for the online catalog. When librari- ans report that they are not busy, success Patron Online Catalog Success 487 TABLE6 SUCCESS, TIME, AND HELP LEVELS FOR TYPES OF SOURCES AT SJSU % Periodical %Reference Indexes, %Online Books Printed %CD-ROMs Catalog (N=38) (N = 30) (N=25) (N = 38) Success level 68.4 46.7 40.0 39.5 Patron reports enough help 94.7 86.7 76.0 81.6 Patron reports enough time 94.7 93.3 84.0 86.8 Librarian reports helped 77.8 63.3 36.0 68.6 Patron reports librarian helped 91.9 66.7 52.2 51.4 NQte: Success levels (patrons reported finding exactly what was wanted and being completely satisfied) in this table are based on 125 reference transactions, including 22 specific book questions. TABLE7 ONE-SOURCE TRANSACTIONS AND SUCCESS BY TYPE OF SOURCE Periodical Online Catalog Reference Books Indexes CD-ROMS (N = 35) No./% involving 1 source 14 (40.0%) Percent successful 21.4% rates for questions involving the online catalog are consistent with success levels in comparison groups of libraries (table 5). Other libraries achieve comparable success rates for questions involving the use of the online catalog independent of the level of demand. However, the pa- tron success rate is much lower at San Jose under conditions of high demand for reference service. Questions involv- . ing the use of CD-ROMs at San Jose and other large academic libraries tend to follow this pattern. In contrast, for ques- tions involving reference books, success levels under busy conditions are compa- rable to other comparison groups under high demand, and much higher under conditions of low demand. For ques- tions involving use of periodical in- dexes, success rates for large academic libraries, including San Jose, are lower under high levels of demand. The library obtained the lowest suc- cess rate in relation to the comparison group (51 percent versus 65 percent) on questions librarians judged to be "easy" and higher success scores than compara- ble institutions (33 percent versus 27 percent) on questions librarians rated as "hard."-The low success rate for "easy" (N= 34) (N=27) (N = 19) 13 (38.2%) 9 (33.3%) 10 (52.6%) 76.9% 66.7% 60.0% questions relative to other large aca- demic libraries also suggests that librari- ans tended to overestimate the ability of patrons to locate information on their own. Librarians appeared to be particu- larly likely to underestimate the assis- tance and time patrons needed to utilize automated resources. For printed mate- rials, reference books, and periodical in- dexes, not only were patron success levels higher, but a much greater per- centage of patrons reported that they re- ceived sufficient time and assistance for questions involving the use of these sources than they reported for questions involving use of the online catalog and/ or CO-ROMs (table 6) Another finding sug- gests that the instructional philosophy may be related to the low success rate: the particularly low rate of success when the online catalog is the only source (table 7). When the online catalog was used alone, in comparison to other types of sources, only a very small percent were judged to provide the patron with success. The following comparisons all sug- gest that reliance on instruction rather than direct assistance, especially for easy questions, is related to the low suc- cess rate reported by users for reference 488 College & Research Libraries queries concerning the online catalog. In responding to queries, librarians were slightly more likely to consult only one source than those in comparison librar- ies (nearly 47 percent versus 42 percent); more likely to spend zero to two minutes per question (48 percent versus 37 per- cent); and less likely to spend over five minutes on questions (about 12 percent versus 21 percent). Librarians were also less likely to report that patrons had ac- tually found the information than li- brarians from the comparison group of institutions (54 percent versus nearly 65 percent), and more likely to report that they didn't know (33 percent versus 13.5 percent) on the same question. Agreement between librarian and patron on whether the information was found was also lower than in the comparison group (about 55 percent versus nearly 71 percent). Librarians appeared to be particularly likely to underestimate the assistance and time patrons needed to utilize automated resources. The nature of the student body at San Jose State University may play a role in these results. Because it is located in an area of great cultural diversity, San Jose State affirms its mission to be the educa- tion of students from this varied popula- tion. Campus demographic figures reflect the large number of nontraditional stu- dents attending the university. Fewer than half of all students are white, while three- quarters of first-time freshmen are co~­ prised of a rich variety of nonwhite minority groups (~ble 2). Many of these are the first generation in their families to attend college. A large number are immi- grants whose first language is not English. Compounding the educational challenges caused by the nature of the student body, the public schools in California have ex- perienced severe funding problems; very few public schools have librarians, including those that have libraries. Few students entering the university have basic library or research skills or much experience in using a large library. November 1994 Patrons were asked if the level of ma- terial was appropriate. It is interesting to note that about 7 percent wanted sim- pler material, while only about 6 percent wanted more comprehensive informa- tion. This is in contrast to the compari- son group where a smaller proportion (about 4 percent) wanted simpler data and a greater proportion (nearly 9 per- cent) wanted more thorough material. The Online Catalog Study As a complement to the Reference Transaction Survey, the purpose of the Online Catalog Study was to learn more about successes and failures of people using the online catalog without staff assistance. On the observation form, each librarian recorded the type(s) of access selected and the apparent ease with which people used the feature. The sample of searches conducted by the ninety-three people during the study ap- pears to be quite similar to the overall search pattern, as recorded in the online catalog transaction logs for the same time period. In the sample study, titles were searched by the people being ob- served somewhat less often than would be expected, based on the records of all public searching during these three weeks (table 8). In table 9, the type of access is listed in order of frequency of use, with subject the most frequent access type and OCLC number the least frequent. TABLES FREQUENCY OF USE OF ONLINE CATALOG OPTIONS % % $ample Transaction Study Log Study Catalog Option (N = 149) (N = 93,156) Subject 37.6 32.7 Title 21.5 31.3 Keyword 18.8 15.6 Author 16.1 16.1 Call number 3.4 3.1 Standard number 2.1 0.8 OCLCnumber 0.7 0.4 Total 100.1 100.0 Patron Online Catalog Success 489 TABLE9 ONLINE CATALOG ACCESS FEATURE USED Some Considerable %with Feature No Problems Difficulty Difficulty Difficulty Subject heading 18 25 7 64.0 Title 20 8 1 31.0 Keyword 14 11 1 46.2 Author 12 7 5 50.0 Call number 1 2 1 75.0 Reserve 0 4 0 100.0 Standard number 0 2 1 100.0 OCLCnumber 0 1 0 100.0 Note: To compile "% with Difficulty," we have combined numbers for "Some Difficulty" and "Considerable Difficulty." TABLElO NUMBERS LOCATED BY TYPE OF ONLINE CATALOG SEARCH Type of Search Numbers %Author %Title %Keyword %Subj. Head %Call No. %Other No. Located (N = 269) (N=495) (N = 258) (N = 497) (N = 28) (N = 13) No hits 31.6 39.8 16.7 49.3 42.9 38.5 1-20 43.9 46.7 35.7 25.4 53.6 7.7 21-40 4.1 4.2 9.3 6.6 0.0 0.0 41-100 7.8 1.8 10.9 2.4 3.6 0.0 101-1,000 9.3 6.5 17.8 7.2 0.0 7.7 1,001-5,000 3.0 0.6 8.9 8.5 0.0 46.2 Over 5,000 0.4 0.4 0.8 0.6 0.0 0.0 Total 100.1 100.0 100.1 100.0 100.1 100.1 Note: From transaction logs of March 4, 1993, Thursday 3-4 p .m .; March 16, 1993, Tuesday 10-11 a.m.; March 16, 1993, Tuesday 2-3 p.m. Of the 1,560 total transactions, 37.6% resulted in no hits; 37.4% resulted in 1-20 hits; 5.7%, in 21-40 hits; 4.6%, in 41-100 hits; 9.0%, in 101-1,000 hits; 5.3%, in 1,001-5,000 hits; and 0.5%, in over 5,000 hits. Subject, title, keyword, and author were by far the most frequently used features. Of these four, a greater percent of patrons had some or considerable dif- ficulty with subject heading searching, followed by author searching. Of those observed having considerable difficulty, while the largest number (seven) were in the category of subject headings, as a percentage of the total searches in a category, about 21 percent (five of twenty- four) of those attempting author searches were observed experiencing considerable difficulty. Fourteen percent (seven of fifty) of those attempting subject heading searches were observed experiencing considerable difficulty. Although ethnic- ity was not significantly associated with user reports of online catalog searching success, a greater proportion of whites (35 percent) used keyword searching than people from other backgrounds (about 19 percent). This difference was significant (Xl = 9.63, df = 3, p = .022). A study of three hours of transaction logs from March 1993 provides further evidence of the difficulties users experi- ence in searching by subject heading. As illustrated in table 10, subject heading searches were much more likely to result in no retrievals of titles than keyword searches (49 percent of all subject heading searches versus about 17 percent of all keyword searches). 490 College & Research Libraries November 1994 TABLE 11 ONLINE CATALOG LIMIT AND EXPAND FEATURES USED Transaction Log % Sample Study Sample Study % (N = 93,156 (N = 93) (N= 93 searches) Limit features Limit to words in subject 5 5.4 Limit to publisher 2 2.2 Limit-any 2.4 Expand features Show items nearby on shelf 6 6.5 2.5 Show items with same subject 4 4.3 1.4 Same search as word search 3 3.2 1.3 TABLE 12 LOCATING MATERIALS IN THE ONLINE CATALOG RfAI Locating Materials #of Users Yes, just what wanted 18 Yes, with limitations 8 Yes, not looked for but help- 3 ful Yes, but not really what 1 wanted Only partly 3 No 5 Total 38 Librarians also observed other op- tions that patrons used. At the bottom of the online catalog screen, users may se- lect, among other choices, options that will either narrow (limit) or expand their search. Slightly fewer than half ofthe participants found exactly what they wished to find. Only seven of the ninety-three per- sons made use of the powerful "limit" feature to narrow or refine their search (table 11). Analysis of the transaction logs for three weeks in March also re- vealed that the limit and expand fea- tures were very rarely used in searches. None of the patrons observed used Online Catalog Study % #of Users % 47.4 34 45.9 21.1 18 24.3 7.9 5 6.8 2.6 2 2.7 7.9 7 9.5 13.2 8 10.8 100.1 74 100.0 "Limit to Words in Title," which is par- ticularly useful in locating a relatively new term or phrase or one in common usage in a particular field but which is not reflected in the Library of Congress Subject Headings. Limit by words in author, year of publication, material type, and language was not observed being used by any users in this sample. A feature frequently used by SJSU li- brarians to locate periodical records among multiple hits, such as Life, is the Limit by Where Item Is Located, permit- ting limiting to periodical records. This feature was not used by the study group. Analysis of transaction logs from three hours in March revealed that of 1,560 total searches, 19 percent resulted in over forty hits (table 10); few people wish to browse that many titles. These results suggest that the limit feature has Patron Online Catalog Success 491 TABLE13 SATISFACTION WITH ONLINE CATALOG SEARCH Degree of Satisfaction #of Users Yes, completely satisfied 19 Yes, mostly satisfied Somewhat satisfied 16 No, mostly dissatisfied No, completely dissatisfied 2 Total 37 the potential to assist in many additional searches. Of the eight people who re- ported that they found nothing in the online catalog, none had used the limit or show features. The . number of search features used and the order in which they were used were observed by the librarians. A ma- jority of users (63 percent) used only one search strategy. When a single search strategy was used, subject headings were much more likely to be selected (48 percent), with title a distant second choice strategy (22 percent). A minority of users (about 38 percent ) employed multiple search features. Users did not appear to prefer one search strategy over others as a most frequent starting point. Half of the people employing multiple strategies used only two search strate- gies. Librarians observed the length of time users spent on their search and found catalog search times to be rela- tively brief. About half of the users spent five minutes or fewer. The mean or aver- age time is eight minutes; the most com- mon time is five minutes (eighteen users); the median is six minutes. Another question we sought to an- swer was whether people found what they wanted, too little or too much, and how they dealt with finding too little or too much. Information on user reports concerning whether they located just what they wanted is helpful in answer- ing this question. Slightly fewer than half of the participants found ex.actly what they wished to find. Roughly the same proportion of users found some material they could use, and a small mi- RfAI Online Catalog Stu9y % #of Users % 51.4 25 32.5 40 51.9 43.2 10 13.0 2 2.6 0 0.0 77 100.0 nority found nothing at all (table 12). In comparing patron reports of finding ma- terials in the online catalog with refer- ence assistance (RTAI) and without reference assistance (Online Catalog Study}, percentages of patrons finding just what was wanted and with limita- tions are remarkably similar. Patron satisfaction with the result of the search is also helpful in measuring success, although satisfaction may be a more general measure involving the pa- trons' general expectations of the library. Only about one-third were fully satis.: fled with their online search result, fewer than those who reported that they found just what they wanted. Very few were dissatisfied; no patrons reported that they were completely dissatisfied (table 13). In the case of satisfaction, a much greater proportion of patrons re- ported that they were completely satis- fied when they used the catalog with reference assistance (RTAI} than when they used the catalog unassisted (Online Catalog Study). However, this could be because the five-point scale used in the Online Catalog Study permits users to select an almost completely satisfied choice, whereas the three-point scale for the RTAI does not. A concept important in the under- standing of online catalog satisfaction is the relationship between a user's ex- pressed satisfaction and other variables relating to catalog use. Patrons who spent a longer time searching in the online catalog tended to be more satisfied with the online catalog (r = .23, p = .02). How- ever, longer searches were not related to 492 College & Research Libraries whether patrons found the materials they wanted in the online catalog. The association between satisfaction with the online catalog search and success in finding items in the catalog is a moder- ately strong relationship (r = .37, p = .001); however, much of the satisfaction rating cannot be explained by finding items in the catalog. Locating materials in the online catalog and length of the online catalog search accounted for 17 percent of the variance in the user satis- faction rating {R2 = .17, df = 2/71, F = 8.56, p = .0005). A more rigorous measure of total user success in the online catalog can be developed by combining thecate- gories of finding materials wanted in the online catalog with user satisfaction and counting only those who found exactly what they wanted and were completely satisfied and viewed the search as com- pletely successful. Of the seventy-four questionnaires for which this compari- son could be done, fifty-eight (78 per- cent) experienced reasonable levels of success in that they reported satisfaction and also indicated that they located at least something they could use. Total suc- cess, using this method, was 24 percent. In comparison with the findings of the RTAI, total success for the online catalog when reference assistance was provided was about 37 percent (exact figure is 36.7 per- cent; see table 3). The high proportion of failures due to user stack skills, ranging from one-third to over one-half of all failures over the years, is another indication of the need for more extensive assistance for students at SJSU. Perceived ease of use of the online catalog appears to be important for us- ers' general satisfaction with the library and its service. People rating ease of use of the catalog more highly also tended to be more satisfied (r = .27, p = .008). Slightly over one-quarter of the partici- pants in the study found the catalog ex- tremely easy to use, with 58 percent November 1994 finding it easy. No one found it difficult or extremely difficult to use. When asked for suggestions for improving the online catalog, a patron commented, "I like it very much. It's easy to use. Much easier than [another California university] system." Another patron commented: "This was the first time I've used the sys- tem. I'm quite impressed with it. My only complaint is that it is not quite as self-ex- planatory as I'd like for a first timer. I had to ask for assistance from a librarian a couple of times-no big deal though." When patrons report their success in locating materials, satisfaction with the search, and ease of online catalog use, patrons are reflecting on the results of their experience in related but slightly different ways. In table 14, the findings from these three questions on the patron questionnaire have been incorporated into one chart. For each of the questions, the results have been consolidated into three ranges-positive, neutral, and negative (table 14). Although the posi- tive "Finding Materials" percentage was somewhat lower than that of the "Satis- faction" and "Ease" percentages, the overall results of patron experience with the online catalog were very positive. Another broad question concerned those users looking for specific titles and/ or authors in contrastto those look- ing for anything/ everything on a sub- ject. Does the type of material wanted by the user affect the individual's success? The results in table 15 do seem to indi- cate a significant relationship between the material desired and success. Those seeking some materials on a topic were a little more likely to be successful than others. Those seeking everything on a topic were more likely to indicate that they had not had any success. Another concern of the authors of this study was whether people actually found the items in the library after locat- ing titles in the online catalog. A key to finding materials in the bookstacks ap- pears to be related to the user's viewing of the complete record in the catalog. When using the online catalog, the user has the option to stop the search at the summary screen, which provides a short Patron Online Catalog Success 493 TABLE 14 USER EXPERIENCE WITH ONLINE SEARCH Category Positive% Neutral% Negative% Total% Finding materials 70.2 18.9 10.8 99.9 Online catalog search satisfaction 84.4 13.0 2.6 100.0 Ease of online catalog use 84.4 15.6 0.0 100.0 TABLE 15 LOCATING MATERIALS BY SPECIFIC VERSUS GENERAL MATERIAL % Particular Author /Text Locating Materials (N = 30) Yes, exactly 40.0 Yes, with limits 30.0 Yes, not looked for but helpful 3.3 Yes, but not what really wanted 3.2 Only partly 13.3 No 10.0 Total 99.9 (Missing cases = 19; X2 = 36.49, df = 5, p < .0001.) title, brief library location symbol, and call number, which may or may not be complete depending on its length. The summary screen includes a maximum of eight titles in numbered order. By press- ing the number of the title desired, the user sees the complete record, including circulation information. The informa- tion in the item record may include more detailed location information, such as number of copies, the specific location of each copy, and the circulation status of each copy. The circulation information may inform the patron that the item is in circulation, missing, at the bindery, etc. This information may be crucial for pa- trons' success in locating, or under- standing why they could not locate the desired item. In the eighty-three instances where observer librarians marked this item on their observation sheets, fifty-five individuals (66 percent) did proceed to the detailed record, while twenty-seven (32.5 percent) stopped at the summary screen. One individual who searched retrieved a single hit and % Particular Item % Some Material % Everything on +Some on Topic (N = 36) Topic (N = 9) Material (N = 1) 50.0 42.9 100.0 22.2 14.3 0.0 11.1 0.0 0.0 2.8 0.0 0.0 5.6 14.3 0.0 8.3 28.6 0.0 100.0 100.1 100.0 thus received the detailed screen as the first display. By comparing the observer librarian's record with the "Did You Find It?" docu- ment availability questionnaire, dis- cussed further in the next section, it was possible to create a finding ratio for us- ers who proceeded to the detailed infor- mation screen and those who stopped at the summary screen (table 16). There is a significant and strong association (Cra- mer's V=.82) between the proportion of desired material found in the stacks and users who view the holdings detailed- level screen in the online catalog. Users looked for a total of ninety-three items in the stacks; sixty-one (about 66 percent) were found by users. Table 17 analyzes the reasons for failures and provides comparisons with earlier docu- ment availability studies done at SJSU. In this small sample of failures, items in circulation was the most frequent reason users were unable to obtain materials immediately. Circulation has often been the leading cause of failed retrieval in 494 College & Research Libraries TABLE16 SUMMARY SCREEN AND FINDING RATIO Summary Screen Action % %Stop at Finding Proceed to Summary Ratio Detailed Screen Screen %Found (N = 20) (N=9) 100.00 55.0 0.0 83.33 0.0 11.1 75.00 10.0 0.0 66.67 15.0 22.2 60.00 5.0 0.0 50.00 5.0 0.0 40.00 5.0 0.0 33.33 0.0 11.1 00.00 5.0 55.6 Total 100.0 100.0 (Missing Cases;= 2; X2 = 19.50, df = 8, p = .0124.) past studies also (table 17). Missing items and poor user stack skills also contributed significantly to failures. A number of users also were unaware of the need .to look carefully at the loca- November 1994 tion information and missed books shelved in the less heavily used building, Wahlquist Library stacks. The high pro- portion of failures due to user stack skills, ranging from one-third to over one-half of all failures over the years, is another indi- cation of the need for more extensive as- sistance for students at SJSU. CONCLUSION The online catalog success study re- sults at San Jose add to already existing knowledge concerning unassisted patron use of online catalogs. Subject heading searching is the type most likely to fail, as noted by librarian observers and con- firmed by analysis of transaction logs. Results for subject heading searching at San Jose, compared to other studies cited previously in the literature review, are favorable. While Ray R. Larson found that only 12 percent of subject searches yielded between one and twenty items, at San Jose over 25 percent did (table 10). Nonetheless, at San Jose, subject head- ings remain the most frequently used feature and the element with which pa- trons experience the most difficulty. TABLE 17 BOOK AVAILABILITY STUDIES* AT SAN JOSE STATE UNIVERSITY: 1976-1993 % April1976 % April1979 % April1983 % April1988 % April1993 (N = 1,365) (N = 2,033) (N = 350) (N=499) (N = 93) Items found 76.0 40.7 68.3 54.1 65.6 Items not found 24.0 59.3 31.7 45.9 34.4 Total 100.0 100.0 100.0 100.0 100.0 Reasons for failures Collections** 3.7 1.8 2.7 0.0 4.8 User skills*** 29.4 55.8 50.4 36.2 33.3 Circulation 43.7 37.6 23.4 45.4 38.1 Inventory**** 23.2 4.8 23.4 18.3 23.8 Total 100.0 100.0 99.9 99.9 100.0 .. All of these studies were based on a methodology originally developed by Paul Kantor. For details see Paul B. Kantor, "Availability Analysis," Journal of the American Society for Information Science 27 (1976): 316-18 . .... Books not owned by the San Jose University Library . ...... User stack skills are a measure of whether the user can find an item which is on the shelf in the proper location. By 1983, the collections had been split into two buildings; and in that study about two-thirds of the user stack failures were because users did not realize items were shelved in the older library building or other special locations. Even in 1993, over 40 percent of the failures were due to items being shelved in special locations . ........ Includes badly misshelved items and missing items. ., Patron online catalog searches can be characterized as quite brief, employing relatively few strategies, and rarely us- ing advanced searching options for lim- iting and expanding the search. Peters and Kurth's study cited in the literature review also indicates that many users do not employ multiple strategies in subject searching. In judging user success, satisfaction alone appears to be a very limited meas- ure of success with users reporting more positively on satisfaction with the search and ease of use than they do for finding materials (table 14). It appears that while one-fifth of patrons have negative expe- riences with finding materials in the on- line catalog, few respond negatively concerning satisfaction or ease of use. Therefore, user satisfaction is indeed a complex measure and not strongly associ- ated with finding materials in the online catalog. Finally, patrons' failure to access the detailed online catalog screens with circulation status and holdings informa- tion is strongly associated with user fail- ure in locating actual documents. One of the most interesting results of this study is that patrons reported simi- lar levels of success in finding just what was wanted in the online catalog whether they asked a librarian or used the online catalog as unassisted users (table 12). This suggests that at San Jose, asking a librarian does not seem to en- able patrons (who are perhaps less well prepared to search on their own) to be more successful in finding materials than catalog users who do not ask for reference assistance. Concerning reference assistance with the online catalog, an analysis of RTAI transactions involving the use of the online catalog indicates that these trans- actions resemble the syndrome of char- acteristics identified by Bunge and Murfin as typical of less successful refer- ence service in libraries. At San Jose, when librarians answer questions in- volving online catalog use, they are more likely to direct the user how to search, rather than actually doing the search with the user. Questions involving online catalog use at San Jose are also char- Patron Online Catalog Success 495 acterized by more transactions of two minutes or fewer, qne-source transac- tions, more users not receiving enough time and help, and low agreement on patron success by librarians and users. It would appear, in our multicultural, multilingual university with severe budget problems affecting our collec- tions and personnel resources, that pa- tron and librarian expectations of the library and the role of the librarian are frequently dissimilar. This occurs de- spite an active outreach program, with curriculum-based bibliographic instruc- tion, outreach programs directed toward minority groups, and special programs for international students. As previously mentioned, California's budget prob- lems have translated into public school funding problems. We are seeing the im- pact of inadequate libraries and library instruction in the elementary and secon- dary schools. We believe this study has implications that touch on reference practice, as well as the design of the online catalog. Implications for Reference Practice Changes in reference practice and phi- losophy should be considered in order to more adequately meet the needs of our academic community. If we consider "teaching" and "providing the informa- tion" as opposite ends of a spectrum, movement along that spectrum toward the "information providing" end may be an important emphasis. Developing a series of strategies that can provide ap- propriate reference behaviors for per- sons working at service points may be one of the most important actions. This could also be true at service desks, where refer- ence and/or instruction are not the pri- mary purposes. Staff at circulation, periodical service, and other similar desks, in libraries where directions and brief in- struction are the norm, also may not be meeting the needs of student patrons who require more assistance. Through their assignments and class- room presentations, teaching faculty play an important role in students' use of the library. Classroom teachers and librarians working together need to provide more 496 College & Research Libraries instruction and assistance in search strate- gies, particularly techniques for narrow- ing and expanding searches, and the appropriate use of keyword versus con- trolled vocabulary. The scope of the on- line catalog should be clarified. Informal examinations of the librarian observers indicated that many users do not under- stand the role of the catalog in locating information versus that of other bibli- ographic sources, such as periodical and newspaper indexes. Our findings need to be conveyed to faculty, and we need to create a plan for cooperatively work- ing toward solutions. Classroom teachers and librarians working together need to provide more instruction and assistance in search strategies, particularly techniques for narrowing and expanding searches, and the appropriate use of keyword versus controlled vocabulary. These proposed changes would seem to require that additional tim~ and effo~t be spent with patrons, accordmg to their needs. To effect these changes, there must be a strong degree of consensus at all levels of staffing and library struc- ture. Increased staffing levels, further staff training in appropriate service be- haviors, and expanded student instruc- tion, either in the classroom or in library-initiated and faculty-supported workshops may be necessary elements in a plan. Without adequate resource and staff support, reference librarians and others may need to make choices between quality and quantity. In scarce resource environments, painful choices must be made; either librarians help people more extensively and. pro~ide service to fewer people, or libranans help more people but not to the extent that they need. However, even when re- sources are not adequate, studies such as this project will help ensure that staff clearly understand the consequences ~f the choice in terms of success for their particular type of users. November 1994 Online Catalog Interface Implications Change in the present online catalog interface is an important element in im- proving user success. Changes in the in- troductory online catalog screen should be considered. Users might be better served by a less cluttered screen, listing only the four major access po~ts (Author, Title, Keyword, and SubJect Headings), plus Other (less heavily used features then listed on a second screen when this choice is selected) and Con- nect to Other Databases. Users attempt- ing to locate information on subjects would be better served by selecting key- word rather than subject heading as their first access strategy. Keyword is listed above Subject Heading on the screen but is less frequently searched according to transaction logs. A proposal is now be- ing considered which will more strongly encourage novice users to begin with key- word-keyword would still be listed above subject heading, but would bela- beled "Subject or Title Words." This should make keyword a more appealing choice to users, particularly nonnative speakers of English. The choice now la- beled "Subject Heading" would be changed to "Library of Congress Subject Heading." This should provide a clearer message to users that these headings are somewhat more esoteric and less desir- able as a first choice. Finally, the authors believe that online catalog software that converts zero hit searches to other forms of subject queries should be an impor- tant priority for vendors. The content of summary screen dis- plays should be reviewed to determine if changes should be made to encourage a greater proportion of users to view the detailed screen with the most accurate item location information. Because the screen summaries are a fairly standard format of Innovative Interfaces catalogs, vendor software changes may also be needed . . Implications for Study Methodology The methodology developed for this study provided a unique and highly use- ful multiperspective view of the role of the online catalog in library service. This methodology was very cost-effective. Although it required the cooperation and assistance of the entire Reference Department staff, the time each individ- ual contributed was relatively small. The group of eight librarians who carried out the online catalog study contributed an estimated five hours each, including plan- ning meetings and online catalog user in- terviews. The library provided the necessary funds (less than five hundred dollars) for the RTAI study, including the basic analysis. A small grant from the California Library Association's Califor- nia State University Librarians Group paid for student assistant help for the online catalog study. The SJSU Informa- Patron Online Catalog Success 497 tion Systems and Computing provided the resources for analysis using SPSS. We encourage other libraries to repli- cate the study, and the instruments have been submitted to the ERIC Document Reproduction Service (EDRS). As re- mote access to bibliographic systems, such as online catalogs and full-text elec- tronic databases, becomes increasingly common, one of the most important is- sues our profession must address is the role and nature of reference assistance for patrons conducting searches from re- mote sites. Studies comparing success of assisted and unassisted users will en- hance our knowledge and enable the profession to utilize scarce resources more effectively. REFERENCES AND NOTES 1. Jon R. Hufford, "Use Studies and OPACS," Technical Services Quarterly 9 (1991): 57-70. 2. Walt Crawford, "Starting Over: Current Issues in Online Catalog User Interface Design," Information Technology and Libraries 11 (1992): 62-76. 3. Prudence Ward Dalrymple and Douglas L. Zweizig, "User's Experience of Information Retrieval Systems: An Exploration of the Relationship between Search Experience and Affective Measures," Library and Information Science Research 14 (1992): 167-81. 4. Ray R. Larson, "Classification, Clustering, Probabilistic Information Retrieval and the Online Catalog," Library Quarterly 61 (1991): 133-93. 5. Ray R. Larson, "The Decline of Subject Searching: Long-Term Trends and Patterns of Index . Use in an Online Catalog," Journal of the American Society for Information Science 42 (1991): 197-215. 6. Rhonda N. Hunter, "Successes and Failures of Patrons Searching the Online Catalog at a Large Academic Library: A Transaction Log Analysis," RQ 30 (1991): 395-402. 7. Michael K. Buckland, Mark H. Butler, Barbara A. Norgard, and Christian Plaunt, "OASIS: A Front-End for Prototyping Catalog Enhancements," Library Hi Tech 10 (1992): 7-22. 8. Thomas A. Peters and Martin Kurth, "Controlled and Uncontrolled Vocabulary Subject Searching in an Academic Library Online Catalog," Information Technology and Libraries 10 (1991): 201-11. 9. Joan M. Cherry, "Improving Subject Access in OPACS: An Exploratory Study of Conversion of Users' Queries," Journal of Academic Librarianship 18 (1992): 95-99. 10. Marjorie E. Murfin and Gary M. Gugelchuk, " Development and Testing of a Reference Transaction Assessment Instrument," College & Research Libraries 48 (1987): 314-38. 11. Marjorie E. Murfin and Charles A. Bunge, "Evaluating Reference Service from the Patron Point of View: Some Interim National Survey Results," Reference Librarian 11 (1984): 175-82. 12. Charles A. Bunge and Marjorie E. Murfin, "Reference Questions-Data from the Field," RQ 27 (1987): 15-18. 13. Gary Gugelchuk, "Sample Size Determination," in A Cost Effectiveness Formula for Refer- ence Service in Academic Libraries, eds. Marjorie E. Murfin, Charles A. Bunge, Gary M. Gugelchuk, and Barbara Aronson (Washington, D.C.: Council on Library Resources, 1989), appendix2. 14. Paul B. Kantor, Objective Performance Measures for Academic and Research Libraries (Wash- ington, D.C.: Association of Research Libraries, 1984). l\Tew Titles from r~e Discovering Librarians: Profiles of a Profession Mary jane Sherdin, editor Results of national studies of vocational interests of library and information professionals. The librarian profile is presented from the ACT, the Strong Interest Inventories, the Meyers-Briggs Type Indicator, and the SIGI PLUS computer- aided career guidance tools, along with analysis of demographic data. $35.95,ACRL member $31.95, 220 p. 0-8389-7753-7' 1994 Managing Student Workers in College Libraries, CLIP Note #20 Michael Kathman, jane McGurn Kathman, comps. Comprehensive guidance for managing student employees. Includes examples of policies and procedures for employment, dismissal, orientation, training, supervision, and performance review. $29.95,ACRLmember$25.95, 140p. 0-8389-7752-9, 1994 ACRL University Library Statistics, 1992-93 Library Research Center, Graduate School of Library & Information Science, University of Illinois at Urbana-Champaign, comp. Data from over 109 participating libraries. Library categories include collections, personnel, expenditures, and interlibrary loan. Institutional categories include degrees offered, enrollment size, and faculty size. $69.95;ACRL member $39.95, BOp. 0-8389-7738-3, 1994 Formal Planning in College Libraries, CLIP Note # 19 Sarah Watstein, Pamela Wonsek, and Paula Matthews, comps. A survey of formal planning procedures at academic libraries. Includes examples of worksheets and checklists, vision statements, and formal. planning documents. $27.95;ACRL member $24.95, 120p. 0-8389-7743-X, 1994 To Order: Write: ALA Order Fulfillment 520 N. Dearborn St. Chicago, IL 60610 Call toll free: 1-800-545-2433 (press 7) Fax: 312-836-9958 AssociATION OF COLLEGE & RESEARCH LIBRARIES A DIVISION OF THE AMERICAN LIBRARY ASSOCIATION