College and Research Libraries Search Characteristics and the Effects of Experience on End Users of PaperChase Natalie Schoch King Transaction logs of 100 end users of PaperChase at the University of Michigan were examined in order to describe the use of various search features and to determine the effects of search experience on the use of search features. A large number of searchers used a variety of important MEDLINE search features. Although advanced searchers showed a significant increase in the use of some features as they gained experience, these increases were not large, and experi- ence seemed to have little effect on searcher utilization of most features. nterest in the searching of bib- liographic databases by non- librarians and noninforrnation specialists has soared in recent years. Currently, library users and other information seekers are being encour- aged to satisfy their own information needs directly through the use of a vari- ety of information retrieval systems. These end-user systems are available through online and optical disk technol- ogies and permit access to numerous in- formation resources. It has long been recognized in the field of medicine that rapid and convenient access to current information is critically important. In the 1950s and 1960s, the National Library of Medicine (NLM) de- veloped an automated information re- trieval system called MEDLARS (MEDical Literature Analysis and Retrieval Sys- tem) to facilitate information provision to health care professionals. Currently, MEDLARS contains more than 30 databases of medical and related litera- ture. The largest and most popular of the MEDLARS databases is MEDLINE, which contains more than six million ref- erences to the biomedical fiterature from more than 4,000 journals in English and foreign languages. The print Index Medi- cus is a subset of MEDLINE. With the rise in interest of end-user searching and the need for health care professionals to obtain timely informa- tion, it is not surprising that a variety of end-user searching systems for provid- ing access to MEDLARS databases has been developed. 1 One such system that provides access to MEDLINE is an online Natalie King is Assistant Director of Shiffman Medical Library, Wayne State University, 4325 Brush, Detroit, MI 48201. She performed this research while a student in the University Library Associates Program in the School of Information and Library Studies at the University of Michigan, Ann Arbor, Michigan 48109. The author is grateful to Jackie Kuhl, Edna S. Moody, and other PaperChase staff members who supplied the transaction logs and were extremely cooperative throughout this project. Gratitude is also extended to K£zren Cangialosi and Regina Baker for their statistical expertise. In addition, the author appreciates the comments on the manuscript of many colleagues at the Taubman Medical Library and the School of Information and Library Studies at the University of Michigan-especially those of Alexandra Dimitroff. information service called PaperChase. Developed by physicians at Beth Israel Hospital in Boston, PaperChase was de- signed to be used with minimal or no training. A menu interface guides users through searches, and the program pro- vides specific suggestions for improving the search. Descriptions and a review of PaperChase are available; however, studies of use, user behavior, and user satisfaction are scarce. 2.3 Gary L. Horo- witz and Howard L. Bleich report that PaperChase users at Beth Israel Hospit"' 1 were generally satisfied with the ~ j ..., • .::.m and that many staff members were repeat users.4 Other studies of PaperChase pri- marily have been comparisons between PaperChase and other user-friendly sys- tems providing access to MEDLINE.5 At the University of Michigan, MED- LINE searching using PaperChase has been available to all members of the univer- sity free of personal charges since January 1989. Use of the system, called UM-MED- LINE, has been tremendous, with thou- sands of staff and students registered for its use. This study was an attempt to examine UM-MEDLINE use and to provide further insights into end users and their searching behaviors. It seemed particularly interest- ing to study the users of this system because PaperChase is extremely user-friendly and was designed to require little or no training to use. Although this study was confined to users of a particular end-user system in a single environment, it was hoped that the findings might shed some light on end-user behavior in general. PURPOSE OF THE STUDY The purpose of this study was two- fold. The first purpose was to examine transaction logs of a selected group of UM-MEDLINE searchers and to de- scribe quantitatively some characteristics of their searches. The second purpose was to determine whether experience with the system causes changes in searching charac- teristics. Horowitz and Bleich postulated that users employ more sophisticated search techniques as they gained experience with the PaperChase system.6 It was, there- fore, of interest to know whether repeated Search Characteristics 361 use of the system resulted in increased use of various search features. END-USER STUDIES- A REVIEW OF METHODS Various methods have been used to study end users and the usefulness of end- user searching. Questionnaires and inter- views have been used extensively to determine user satisfaction and to glean descriptive data about how users operate the systems? However, growing evidence in the literature on end-user searching suggests that users often express satis- faction with searches even if they do not obtain particularly good results. 8 An- other technique employed in end user studies has been to ask users to compare results of searches performed by novice searchers with those of searches by expe- rienced librarian searchers. 9 Simply ob- serving the activities of end users has also been employed.10 The case study method has been used to study search- ing behaviors of librarian searchers;11 however, this method has not yet been applied to the study of end users. Trans- action log analysis is also used to study end users. A number of use studies of Online Pub- lic Access Catalogs (OPACs) have used the technique of transaction log analyses Y In addition, a few studies of users of online and CD-ROM bibliographic databases exist. For example, Naomi Miller and her colleagues examined search statements from Compact Cambridge MEDLINE (a CD-ROM product) and identified errors in the use of the system.13 Transaction logs of users of GRATEFUL MED, NLM's end-user system for access to the MEDLARS databases, were recently used to study search characteristics. 14 Winifred Sewell and Sandra Teitelbaum performed an exhaustive study of trans- action logs of online database users over eleven years Y This study used transac- tion log analysis, questionnaires, and follow-up interviews to investigate the searching behavior of pathologists and pharmacists using NLM databases. Both Thomas A. Peters and Mitchell A. Cahan discuss the advantages and dis- advantages of transaction log analysis as 362 College & Researt:h Libraries a method for studying end users. 16 Ad- vantages include that these studies are relatively cheap, they provide insights for librarians into patron problems so that better training can be provided, and they can supply information leading to better system design. In addition, users' attitudes do not affect the results as they can in questionnaires and interviews. The primary disadvantage of studying end users through transaction logs is that user intentions and satisfaction are not re- corded on the logs. In addition, database producers do not generally intend for transaction logs to be used for extensive research; thus, information is often miss- ing or incomplete. · METHODS Selection of Users and User Anonymity A random sample of 50 house officers (physicians in residency training pro- grams) and 50 medical students who had been searching UM-MEDLINE for ap- proximately six months (from mid-May to mid-November 1989) was chosen. These two groups were selected because they were identifiable on the PaperChase tapes and because they are among the groups for which PaperChase was designed. 17 The reason these individuals were se- lected was to determine whether experi- ence with the system improved effectiveness of searching. The author originally thought that searches performed in the first month could be compared to searches per- formed in the last month in order to determine the effect of experience on searching. However, as will be seen later, the number of searches performed proved a more useful measure of experi- ence than the length of time the users had been searching. To ensure anonymity, the author re- moved the subjects' names and social se- curity numbers from the printed logs and numbered the subjects sequentially. Subsequent analyses used only these numbers. This procedure was approved by the Human Subject Review Board of the School of Education at the University of Michigan. July 1991 Descriptive and Operational Definitions The following is a list of the terminol- ogy used to describe transaction logs and a description of how each variable was measured. An example of a typical transaction log can be found in table 1. New or Old Searches. One of the fea- tures available to PaperChase users is the option to return to an old search. PaperChase saves users' searches for up to six months. Old searches (i.e., searches displayed after the initial search was performed) were identifiable because all searches carry a unique identifier num- ber. Because little new information was available from old searches, only the first occurrence of a search (i.e., a "new" search) was analyzed. It has long been recognized in the field of medicine that rapid and con- venient access to current information is critically important. Statements. The number of state- ments for each search was counted as a crude estimation of search complexity. The search in table 1 has 19 statements. Displays. Values given for the number of records displayed represent the total number of items displayed in the new search. However, these figures do not necessarily represent the total number of . records displayed for that search be- cause additional records were often dis- played in subsequent returns to an old search. Only the records displayed in the first session of the search were counted, however, and no attempt was made to determine whether more records were displayed in old searches. Print. Although the transaction logs provide figures for the number of rec- ords selected to be printed, no analysis of this variable was performed for the following reasons. Throughout the time covered by this study, printers were often unavailable in the hospital; thus, a majority of house officers and students did not print any records. In addition, even when printers were available, search- ers commonly used the "print screen" op- Search Characteristics 363 TABLE 1 TRANSACTION LOG List Description (Search Number 701) A ARTHRITIS,RHEUMATOID B ARTHRITIS, ... c METHOTREXATE D ENGLISH E REVIEW F 1985 ... 90 G *ON A&C&D&E&F H ARTHRITIS,RHEUMATOID /MX METHOTREXATE /MX METHOTREXATE I AE K *ON'A&B L ANTI-INFLAMMATORY AGENTS, NON-STEROIDAL M *SUMCL N ANN INTERN MED 0 ABSTRACT ONLINE p CHILD, PRESCHOOL Q *ON D&E&F&H&I&N&O&P R *ONG&N s TREATMENT tion or simply sat with pencil and paper jotting down citations. Thus, the transac- tion log record of citations selected for printing underestimates the number of citations found to be relevant. Boolean Operators. The number of each Boolean operator-AND, OR, NOT -was counted for each search analyzed. The AND operation is accomplished by se- lecting an option from the PaperChase main options menu and is identifiable on the logs (see table 1, statements G, Q, and R). Several means of performing an OR operation exist in PaperChase. The searcher can select an option from the main options menu, and this can be iden- tified in the transaction logs (see table 1, statement M). The searcher can also pool search terms by selecting a number of items from a list, which produces a search statement followed by 11 • • • 11 (see table 1, statement B). It is important to Seconds REFS Display Print 0 28805 0 0 0 37069 0 0 0 12153 0 0 0 4375452 0 0 0 280096 0 0 ·o 1673113 0 0 0 40 9 6 0 20453 0 0 0 5101 0 0 0 1878 0 0 0 30000 0 0 0 3919 0 0 0 16072 0 0 0 11665 0 0 0 2270739 0 0 0 283255 0 0 0 0 0 0 0 2 2 2 0 239703 0 0 note that PaperChase truncates single subject terms that have more characters than can be displayed in a single line and that these truncated terms are also fol- lowed by 11 • •• II The investigator could not distinguish between truncated single sub- ject terms and lists of pooled terms; there- fore, all lists ending with 11 • • • 11 were counted as OR operations. Truncation of single terms appeared to occur infre- quently, but the OR category described in this study may be somewhat inflated. In addition, certain subject terms are fol- lowed by ALL, indicating that the use of that term will include a number of re- lated subject terms. All of these methods for pooling terms and references were counted in the OR category. The NOT operation was performed in only one search and was not analyzed further. The PaperChase options menu lacks a separate option for performing the NOT operation, although it can be 364 College & Research Libraries accomplished by selecting the AND op- tion and placing a minus sign (hyphen) between the letters of the lists to be com- bined (see table 1, statement K). MeSH and Title Words. It was possi- ble to determine which subject terms were Medical Subject Headings (MeSH), the controlled vocabulary terms used by NLM in producing MEDLINE (terms fol- lowed by ; see table 1, statements A and C), and which were title words (terms without trailing notation; see table 1, statement S). Only the occurrence of a MeSH term or a title word was counted. Collecting categorical data seemed rea- sonable given that the study was at- tempting to analyze the users' awareness of certain system features. Limit Categories. Searchers could re- duce the number of records obtained or "limit" a search by year of publication, review articles, human age categories, topical subheadings, language, articles from a particular journal, or articles with an abstract online. Table 1 displays the appearance of the various limit catego- ries on the transaction logs. A searcher can access these features in a variety of ways. The searcher can use the limit feature without any assistance from the system. For example, a user may use subheadings (part of the con- trolled vocabulary) by entering a MeSH term followed by a forward slash(/) and the two-letter abbreviation for the sub- heading. However, the user need not re- member these conventions because PaperChase prompts the user to select a limit feature in many cases. For instance, when displaying a long list from a single MeSH term search statement, PaperChase suggests appropriate subheadings to narrow the search. Similarly, displaying a long list will result in a prompt to select options such as limiting to review arti- cles only or articles with an online ab- stract only. In addition, simply typing the word limit at "LOOK FOR," the prompt to enter a query, provides a list of limiting features from which users can choose. PaperChase will also prompt the searcher to use a limit feature when a word entered at "LOOK FOR" is known to be related to a limit feature (e.g., a user July 1991 entering the word pediatric is asked to select an age category). When a limit feature appears on the transaction log, there is no way to tell which method was used to enter it. Was the user aware of the feature? Did the user know how to enter the term in the proper format, or did PaperChase prompt the user to limit the search? However, use of these features is important for effective MEDLINE searching, and the occurrence of these features in the searches indicates use of important searching techniques. Thus, as with MeSH terms and title words, the occurrence per search of these features was recorded. Descriptive Study This portion of the study provided de- scriptive data on the use of UM-MED- LINE by house officers and medical students and determined whether differ- ences in search performance existed be- tween the two groups. The results of this analysis assisted in determining the ap- propriateness of pooling user group data for subsequent analyses. A random sample of three new searches was selected from the logs of each of the 100 individual searchers (n=300). The num- ber of total, new, and old searches was determined for each searcher. In addition, transaction log analysis as described above was performed for each variable in each of the searches. The number of searches per- formed, statements created, records dis- played, and Boolean operators used was compared among user groups using a one- way analysis of variance (ANOVA). The categorical data (MeSH, title word, and limiters) were analyzed by calculating proportions and by performing a two-by- two contingency table analysis. A signifi- cance level of .05 was used throughout the study. Effect of Experience on Search Performance This portion of the study was performed to determine whether experience in- creases searchers' abilities to utilize any of the PaperChase searching features more effectively. Because of high vari- ability in their overall use of UM-MED- LINE, the 100 subjects were assigned to one of three experience groups: (1) be- ginner-those who had performed ten or fewer searches during the course of the study (n=38); (2) intermediate- those who had performed between 11 and 20 searches (n=41); and (3) ad- vanced-those who had performed more than 20 searches (n=21). In order to determine whether greater use of PaperChase affected search per- formance, first (experience level 1) and last (experience level 2) searches per- formed by users within each experience group were selected. Therefore, a check of experience could be performed within and between groups. For intermediate and advanced searchers, the first five searches and the last five searches for each individual were selected. For be- ginners, the number of searches per- formed was divided by two, and the quotient was the number of searches se- lected. In cases of an odd number of searches, the median search was elimi- nated. Using paired t-tests, the author corn- pared mean number of searches per- formed, statements created, records displayed, and Boolean operators used between experience levels within a group. Between-group differences were analyzed with a one-way ANOVA, fol- lowed by multiple range tests (Duncan, Tukey, and least square means). The cat- egorical data were analyzed using a log linear model for three-way categorical analysis. Additionally, a two-way AN OVA compared mean number of all limiters used between and within groups. Statisti- cal analyses were performed using SAS. RESULTS Descriptive Study Searching for articles according to subject was the most frequently used mode of searching. Seventy-five percent of searches performed by medical stu- dents and 84% of searches performed by house officers were searches by subject alone. An additional10% of medical stu- dent searches and 9.3% of house officer searches were performed using a cornbi- Search Characteristics 365 nation of subject and author. Searches for articles by author(s) were performed in 14.7% of medical student searches and 6.7% of house officer searches. Medical students performed an aver- age of 25.3±17.3 total and 14.1±10.8 new searches while house officers performed approximately 25.5±21.3 total and 17.4±14.7 new searches during the study. Thus, during the six months of the study, users in both groups searched UM-MED- LINE infrequently, averaging approxi- mately two new searches per month. The number of each type of search varied widely, and no significant differences were found between house officers and medical students in any category. Medical students performed searches with an average of nine statements and displayed an average of approximately 45 records. House officers averaged ap- proximately eight statements per search and displayed approximately 47 re- cords. The number of statements used varied widely, and the ANOVA showed no significant differences between med- ical students and house officers in this variable. The number of records dis- played also showed great variability ranging from zero to 449 records dis- played in one session. No significant dif- ference between medical students and house officers was detected in the num- ber of records displayed. The AND operator was used on aver- age 3.5±3.9 times per medical student search and 3.3±3.2 times per search per- formed by house officers. The OR oper- ator was used less frequently, 0.6±1.1 times per medical student search and 0.4±0.9 times per house officer search. AN OVA showed that the two groups did not differ significantly in the use of ei- ther opera tor. Table 2 compares the percentage of medical student searches and house of- ficer searches containing the remaining search features. A two by two contin- gency table analysis of the raw data showed no significant differences be- tween medical students and house offi- cers in any category. While searching by MeSH headings was common, it is note worthy that over one-third of the searches 366 College & Research Libraries July 1991 TABLE2 COMPARISON OF CHARACTERISTICS OF SEARCHES PERFORMED BY MEDICAL STUDENTS (N=150 SEARCHES) AND HOUSE OFFICERS (N=150 SEARCHES). Percentage of Searches* Search Feature Medical Student Searches House Officer Searches MeSH Title Word Year Review Age Subheadings English Journal Title Abstract Online 82.7 36.0 28.7 18.7 9.3 18.7 10.7 6.7 4.7 93.3 35.3 25.3 19.3 8.7 20.7 16.0 4.6 5.3 *Values represent the percentage of searches containing an occurrence of the variable. Two by two contingency table analyses indicate no significant differences between medical students and house officers in any category. 100 - r-- - - - D Medical Students {I) ... 80 Gl • House Officers .c u ... co Gl 60 Cl) r-- -eo.. 0 Gl ell 40 co .... = r--Gl u ... 20 r-Gl ~ [ 0 ....._ L..- ~ ....._ ....._ ....._ ....._ ....._ - ,__ AND OR NOT MeSH TW Year Rev Age Subs Eng Jml Abst Search Feature FIGURE 1 Percentage of searchers using various PaperChase search features for both groups used title words for sub- ject searching. Further, the limit features "year," "review," "subheadings," or "En- glish language" were each used in over 10% of the searches. Figure 1 shows the percentage of indi- vidual searchers in each group using the search features presented in table 2 and each of the Boolean operators. Every searcher used the AND operator and nearly 90% of the searchers in both groups used the OR operator. One hundred per- cent of the searchers in both groups used MeSH terms but, clearly, title words were also heavily used (by more than 90% of the users). Users seemed to be remarkably aware of several limit fea- tures, with more than 80% of both groups using the limit feature "year," nearly 56% of medical students and 68% of house officers using "review articles," 56% of medical students and 80% of •. Search Characteristics 367 TABLE3 WITHIN GROUP COMPARISONS OF SEARCH VARIABLES USED IN EARLY AND LATER SEARCHES* Group Variable t-value p Beginner statement 0.87 0.391 display 1.25 0.218 AND 0.99 0.327 OR -1.47 0.150 Intermediate statement -1.95 0.058 display 0.57 0.575 AND -2.06 0.046 OR -0.68 0.499 Advanced statement -0.68 0.505 display 0.47 0.643 AND 0.36 0.723 OR -0.71 0.484 *Paired t-test comparisons for variables used in early searches (experience level 1) compared to later searches (experience level2) within each group. Significant p values (p feature-are probably especially useful to users because nei- ther requires the user to seek consciously an OR operation. A unique feature of PaperChase is that the program monitors searches and sug- gests improvements to the user. The most frequent suggestion proposed to PaperChase users is that they conduct a search using a MeSH term rather than a title word. 34 It is commonly accepted that searching by MeSH terms usually results in greater success than searching by key- word; other MEDLINE studies suggest that failure to use MeSH terms is a com- mon search problem. 35 That over 80% of searches by medical students and house officers contained MeSH terms and that 100% of individuals in both groups searched by MeSH terms in at least one of the searches analyzed suggest that PaperChase is effectively leading end users to MeSH terms. However, approx- imately 35 % of the searches included statements containing title words and more than 90% of the study subjects searched by title word in at least one of the selected searches. Therefore, users seem to be taking advantage of the abil- ity to search for title words. To ascertain the true significance of title word search- ing in this study is difficult because the author did not perform an analysis of search content. Possibly, title words were used by necessity because no suit- able MeSH term existed, or perhaps a title word was used to limit a larger search. However, the transaction logs in- dicate that searchers used title words when an appropriate MeSH heading or subheading was available. A unique feature of PaperChase is that the program monitors searches and suggests improvements to the user. Interestingly, approximately one-fifth of all searches employed subheadings, and more than 50% of the UM-MED- LINE users in this study applied sub- headings in at least one search. Sewell and Teitelbaum noted that subheadings are extremely useful in MEDLINE searching but that end users have diffi- culty remembering and finding these subheadings when needed. 36 That such a large number of UM-MEDLINE searchers in this study used this feature indicates that PaperChase is probably helping searchers use this important MEDLINE search feature. No compara- ble figures for the use of subheadings by end users of other MEDLINE access sys- tems could be found, although Sewell and Teitelbaum reported that failure to use subheadings by NLM database end users may have been among the most costly of the errors made. 37 Additionally, the many reports of end users' failure to employ MeSH implies that the use of subheadings in these systems is also likely to be quite low. Search Characteristics 371 Similarly, PaperChase seems to be guiding users effectively toward other limiting features, such as "year of publi- cation," "review articles," and "articles written in English." The only compara- ble figures found for the use of these features in MEDLINE are from a study of GRATEFUL MED users, which found that approximately eight percent of 1,310 searches were limited to review articles, while approximately 23 % of these searches were limited to English. 38 UM-MEDLINE users in this study made greater use of the review article feature, but less frequent use of the ability to limit searches to English-language articles only. The less frequent use of limiting by En- glish may be attributable to PaperChase display techniques, which show first 118 English-language journals in Abridged Index Medicus, a subset of Index Medicus. This may decrease the need for users to limit by this feature. The current study did not analyze search content or at- tempt to determine the purpose of the search. Possibly, the simple nature of many searches precluded the need for advanced limiting features. Likewise, features such as age or year of publica- tion may have been unnecessary or un- desirable in a particular query. Effect of Experience on Search Performance Only the advanced group showed a significant increase in use of MeSH terms with experience and an increase in overall use of limiters. Thus, users tended to increase their use of MeSH terms and become more sophisticated with limiters as they gained experience. The present study seems to support the earlier suggestion that users employ more sophisticated search techniques as they work with PaperChase.39 However, these tendencies to increase the use of cer- tain features are not overwhelmingly large and, overall, experience seemed to have little effect on utilization of system fea- tures. Other studies of end users suggest that use of search features varies with search- ing experience. For example, Winifred Sewell and Alice Bevan observed that 372 College & Research Libraries relatively inexperienced end-user search:- ers of NLM databases tended to perform simple searches without utilizing more sophisticated search features. 4° Carol H. Fenichel performed an extensive study of the effect of experience in searching the ERIC databaseY She found that those who searched ERIC a great deal tended to use more commands and de- scriptors, to modify their searches more often, and to spend more time online. However, she also noted that there was no clear-cut pattern across the experi- ence groups (similar to the results of this study). CONCLUSIONS The descriptive segment of this study shows that the UM-MEDLINE users ob- served used a variety of features for per- forming effective MEDLINE searches (i.e., MeSH terms and limiters). The in- frequent use of these features in other MEDLINE end-user systems, coupled with the results of the present study, in- dicates that PaperChase and its user- friendly features are aiding searchers. Experience seemed to have little effect on searcher utilization of most features; however, the most experienced users tended to increase their use of a few search features. The present study does not analyze search content or search results. Casual observations of search contents indicate that searchers were making at least some July 1991 mistakes (e.g., searching for "bone mar- row" as a MeSH term and "transplanta- tion" as a title word rather than applying the "transplantation" subheading to the MeSH term). Similarly, conversations with users indicate that a lack of under- standing of PaperChase mechanics may also be impairing results. For example, explanations of MeSH tree structures and the inability to explode (a feature that allows the searcher to input a broad term and include narrower terms in the MeSH hierarchy) have usually led to user surprise and concern. These find- ings point to the importance of librarian searchers as trainers, consultants, and guides. The continued need for librari- ans to assist these end users is particu- larly noteworthy because PaperChase is considered to be extremely user- friendly. Likewise, this study did not measure user satisfaction with PaperChase; a fol- low-up study in this area would be inter- esting. End users may be satisfied with searches even if they do not retrieve all or even a significant portion of the refer- ences on a topic. On this point, librarians should consider the user and the use of the information. The information needs of medical students or house officers may be amenable to quick and possibly incomplete answers. The needs of other end users (e.g., researchers, grant seek- ers, textbook writers, etc.) may be sub- stantially greater. REFERENCES 1. For review, see James Shedlock, "End User Search Systems: An Overview," in End User Searching in the Health Sciences, eds. M. Sandra Wood, E. B. Horak, and Bonnie Snow (New York: Haworth, 1966), p.65-83. 2. Gary L. Horowitz and Howard L. Bleich, "PaperChase: A Computer Program to Search the Medical Literature," New England Journal of Medicine 305:924-30 (Oct. 15, 1981); Gary L. Horowitz, Jerome D. Jackson, and Howard L. Bleich, "PaperChase: Self-Service Bibliographic Retrieval," JAMA 250:2494-99 (Nov. 11, 1983); Howard L. Bleich, Jerome D. Jackson, and Harold A. Rosenberg, "PaperChase: A Program to Search the Medical Literature," M.D . Computing 2:54-58 (Feb. 1985). 3. Sue Stigleman, "PaperChase: For MEDLINE Searching," Online Review 12:67-76 (Feb. 1988). 4. Horowitz, Jackson, and Bleich, "PaperChase: Self-Service," p.2498. 5. Douglas Porter, and others, "Self-Service Computerized Bibliographic Retrieval: A Comparison of Colleague and PaperChase, Programs that Search the MEDLINE Data Base," Computers and Biomedical Research 21:488-501 (Oct. 1988); R. Brian Haynes, and Search Characteristics 373 others, "Computer Searching of the Medical Literature: An Evaluation of MEDLINE Searching Systems," Annals of Internal Medicine 103:812-16 (Nov. 1985). 6. Horowitz, Jackson, and Bleich, "PaperChase: Self-Service," p.2499. 7. For examples, see Natalie Schoch King, Sandra E. Goldstein, and Lesley A. Williams, "MEDLINE and PsycLIT on CD-ROM: A Survey of Users in an Academic Medical Library," Medical Reference Services Quarterly 9:43-58 (Spring 1990); Jane P. Kleiner, "InfoTrac: An Evaluation of System Use and Potential in Research Libraries," RQ 27:252-63 (Winter 1987). 8. For examples, see Renata Tagliacozzo, "Estimating the Satisfaction of Information Users," Bulletin of the Medical Library Association 65:243-49 (Apr. 1977); Berl Glitz, "Testing the New Technology: MEDLINE on CD-ROM in an Academic Health Sciences Library," Special Libraries 79:28-33 (Winter 1988); Ann B. Hubble, "MEDLINE Access through an Online Catalog: A Study of User Reactions," Proceedings of the ASIS Annual Meeting 25:137-42 (Dec. 1988); Naomi Miller, Martha Kirby, and Etheldra Templeton, "MEDLINE on CD-ROM: End User Searching in a Medical School Library," Medical Reference Services Quarterly 7:1-13 (Fall1988). 9. R. Brian Haynes, and others, "Online Access to MEDLINE in Clinical Settings: A Study of Use and Usefulness," Annals of Internal Medicine 112:78-84 (Jan. 1990); Martha Kirby and Naomi Miller, "MEDLINE Searching on Colleague: Reasons for Failure or Success of Untrained End Users," Medical Reference Services Quarterly 5:17-31 (Fall1986). 10. Glitz, "Testing the New Technology," p. 28-33; T.S. Plutchak, "On the Satisfied and Inept End User," Medical Reference Services Quarterly 8:45-48 (Spring 1989). 11. Raya Fidel, "Online Searching Styles: A Case-Study-Based Model of Searching Behav- ior," Journal of the American Society for Information Science 35:211-21 (1984). 12. For example, see Grace Agnew and others, "The Online Catalog and Patron Search Strategies at Georgia State University," Georgia Librarian 23:42-44 (May 1986); Margaret Henty, "The User at the Online Catalogue: A Record of Unsuccessful Keyword Searches," LASIE 17:47-52 (Sept./Oct. 1986); Thomas A. Peters, "When Smart People Fail: An Analysis of the Transaction Log of an Online Public Access Catalog," Journal of Academic Librarianship 15:267-73 (Nov. 1989). 13. Miller, Kirby, and Templeton, "MEDLINE on CD-ROM," p. 1-13. 14. Mitchell Aaron Cahan, "GRATEFUL MED: A Tool for Studying Searching Behavior," Medical Reference Services Quarterly 8:61-79 (Winter 1989). 15. Winifred Sewell and Sandra Teitelbaum, "Observations of End User Online Searching," Journal of the American Society for Information Science 37:234-45 (1986) . 16. Peters, "When Smart People Fail," p.271-72; Cahan, "GRATEFUL MED," p.61-79. 17. Horowitz and Bleich, "PaperChase: A Computer Program," p.924. This article identi- fied medical students, nurses, house officers, and practicing physicians as the primary users of PaperChase. 18. Sewell and Teitelbaum, "Observations of End User Online Searching," p.240. 19. Naomi C. Broering, "The miniMEDLINE System: A Library-based End User Search System," Bulletin of the Medical Library Association 73:144 (Apr. 1985). 20. Hubble, "MEDLINE Access through an Online Catalog," p.139. 21. J. D. Montgomery, "Training of End User Searchers: Use and Perceived Benefits of Online and Non-Online Information Resources," Canadian Journal of Information Science 12:109 (1987). 22. Ans Bleeker, Ingeborg A. S. Tijam, and Aloysius C. W. Volkers, "MEDLINE on CD- ROM: An Evaluation after Six Months' Use," Online Review 12:200 (1988). 23. Results of a questionnaire study performed by the author, Sandra Goldstein, and Lesley Williams at the University of Michigan, Fall 1988. For further information, contact Natalie King, Shiffman Medical Library, 4325 Brush, Detroit, MI 48201. 24. Haynes, and others, "Online Access to MEDLINE," p.78. 25. Horowitz, Jackson, and Bleich, "PaperChase: Self-Service," p .2498. 26. Horowitz and Bleich, "PaperChase: A Computer Program," p.927 noted that searchers created 7.7 lists per search on average. Horowitz, Jackson, and Bleich, "PaperChase: Self-Service," p .2498 noted 7.4 lists per search on average. 27. Broering, "The miniMEDLINE System," p.144. 28. Cahan, "GRATEFUL MED," p.71. 374 College & Research Libraries July 1991 29. Horowitz and Bleich, "PaperChase: A Computer Program," p .927 noted that searchers displayed 47 references per session on average. Horowitz, Jackson, and Bleich, "PaperChase: Self-Service," p.2498 noted 50.6 references displayed per session on average. 30. Stephen E. Wiberly and Robert Allen Daugherty, "Users' Persistence in Scanning Lists of References," College & Research Libraries 49:149-56 (Mar. 1988). 31. Sewell and Teitelbaum, "Observations of End User Online Searching," p .240. 32. ElaineTrzebiatowski, "End User Study in BRS/ After Dark," RQ 23:447 (Summer 1984). 33. Miller, Kirby, and Templeton, "MEDLINE on CD-ROM," p .9. 34. Horowitz and Bleich, "PaperChase: A Computer Program," p.929. 35. Bleeker, Tyam, and Volkers, "MEDLINE on CD-ROM," 197-204; Glitz, "Testing the New Technology," 28-33; King, Goldstein, and Williams, "MEDLINE and PsycLIT on CD-ROM," p.43-58; Kirby and Miller, "MEDLINE Searching on Colleague," p.17-31; Miller, Kirby, and Templeton, "MEDLINE on CD-ROM," p.1-13; Ellen H. Poisson, "End-User Searching in Medicine," Bulletin of the Medical Library Association 74:293-99 (Oct. 1986). 36. Sewell and Teitelbaum, "Observations of End User Online Searching," p.241. 37. Ibid. 38. Cahan, "GRATEFUL MED," p .72. 39. Horowitz, Jackson, and Bleich, "PaperChase: Self-Service," p.2499. 40. Winifred Sewell and Alice Bevan, "Nonmediated Use of MEDLINE and TOXLINE by Pathologists and Pharmacists," Bulletin of the Medical Library Association 64:382-91 (Oct. 1976). 41. Carol H. Fenichel, "Online Searching: Measures that Discriminate among Users with Different Types of Experiences," Journal of the American Society for Information Science 32:23-32 (1981) . IN FORTHCOMING ISSUES OF COLLEGE & RESEARCH LIBRARIES Research Productivity and Publication Output: An Interdisciplinary Analysis by Charles A. Schwartz The Role of Libraries in American Indian Tribal College Development by Cheryl Duran Evaluating Reference Service in a Large Academic Library by Cheryl Elzy, Alan Nourie, F.W. Lancaster, and Kurt M. Joseph With Feathers by Ann Okerson Humanists, Libraries, and Technological Change by Stephen Lehmann and Patricia Renfro Research Notes Paradigm Restrictions on Interdisciplinary Research into Librarianship by Jeffrey N. Gatten