College and Research Libraries Research Notes 251 The Lopez or Citation Technique of In-Depth Collection Evaluation Explicated Manuel D. Lopez Based upon the brief information sup- plied in the article, "A Guide for Begin- ning Bibliographers, " 1 the citation tech- nique was utilized, on an experimental basis, by the staff of the Elizabeth Dafoe Library, University of Manitoba, Canada. While the results indicated that "this tech- nique actually does measure the depth of the collection and consequently, would constitute a valid evaluation tool if the problems encountered could be over- come, " 2 that study also properly identi- fied some of the problems inherent in the use of the instrument. However, other dif- ficulties simply result from unidentified assumptions and inappropriate expecta- tions of the uses of the method. Unfortu- nately, some of the difficulties noted are derived from the fact that the original arti- cle provided only a description of the method, but did not delineate its imple- mentation. Perhaps some clarification of uses and limitations would be aided by a review of the citation technique of collec- tion evaluation procedures (CTCE). In 1979 the University Libraries of the State University of New York at Buffalo in- cluded approximately two million vol- umes to its shelves. These volumes served instructional and research needs primarily of graduate and professional educational programs in the schools of the humani- ties, social sciences, physical and health sciences (medicine, nursing, dentistry), education, and law. The University Li- braries have been, since 1964, a U.S. docu- ments depository, and a depository for New York State documents, supple- mented by a complete collection of UN publications, documents of the European Economic Community as well as extensive holdings of British and Canadian docu- ments. The microform holdings, some 600,000 items, include documents, news- papers, periodicals, dissertations, and books. The use of the citation technique of col- lection evaluation method has been lim- ited to the sociology, political science, and social work parts of the University Li- braries' collections. While the "stated in- terests" of each department or school are expressed in the collection development policy in terms of collecting intensity, pri- ority 1 (comprehensive level-regarded as the level necessary for creating or main- taining a ''special collection'') to priority 5 (minimal level-only highly selective pur- chases, "inadequate in quantity and vari- ety to support undergraduate instruc- tion"), the selection of materials also considers the needs of other departments, schools, and faculties that require, for ex- ample, publications with a sociological orientation that may not be included in the ''stated interests of the Sociology Depart- ment.'' For the purposes of the evaluation study, those "stated interests" -sociolog- ical theory, urban sociology, small group, medical sociology, etc., that are included in the categories of priority 2 (extensive re- search level) and priority 3 (teaching and research) were used to establish some of the study's limits. From the Choice list of 509 outstanding books of the year (March 1978 to February 1979) that "represented works that should be considered for prior- ity purchase," five titles approximating the'' stated interests'' of the Sociology De- partment were selected. From each, 20 ti- tles were taken from different and stag- gered sections of the bibliographies to generate 100 references, representing a variety of critical ~ibliographies, to books, Manuel D. Lopez is reference department librarian, Lockwood Library, State University of New York at Buffalo. 252 College & Research Libraries articles, documents, newspapers, disser- tations, papers, proceedings, and pam- phlets. This approach, hopefully, avoids the pitfall of the narrow bibliography com- piled by the specialist in one aspect of a discipline. The Nisonger study used, ini- tially, only 25 references from one bibliog- raphy, with another random sample of 25 titles from the same bibliography. One of the requirements of the Manitoba experi- ment was the search for ''a quick and effi- cient method of obtaining an objective evaluation of a particular subject area.' '3 The use of only 50 titles, at the University of Manitoba, may have been sufficient. However, use of CTCE at SUNY Buffalo always starts with 100 titles with a poten- tial of 400 titles to be checked. Nisonger notes that, "one of the most appealing features of this technique is that it can be carried out relatively quickly and without expending a great deal of effort." To in- crease the number of initial references would ''nullify this benefit,' ' 4 and the use of CTCE at SUNY Buffalo is based upon other premises and objectives. The procedures of CTCE are carried out for, if necessary, the duration of one se- mester. The study requires that the inves- tigator use no knowledge (such as search- ing acquisition files) or privileges that would not be available to the ordinary li- brary patron working on a graduate semi- nar paper or dissertation. The one excep- tion to this condition is that the investigator does not resort to interlibrary loan services but confines himself to in- house library resources. Specifically, each of the citations on the master bibliography is checked against the bibliographic records of the University Li- braries, and when possible, the reference is physically obtained. Then the center/ middle reference in the publication's bibli- ography, middle chapter's list of refer- ences, or the middle /center footnote is re- corded. That item is retrieved and the process repeated with the first citation in that reference's bibliography, chapter's notes, or footnotes. The third-level item, in turn, is located and the last citation in the bibliography, chapter notes or foot- note is obtained. Materials listed in the records but not found on the shelves are recalled from another patron, "traced" by May 1983 the circulation staff, or, if the item is not in circulation, on reserve, or being repaired, but theoretically on the shelf (SUNY at Buffalo has open stacks) that title is searched for on a weekly basis for the du- ration of the semester. Thus, simulation of the research process incorporates a factor of bibliographic access into the study, and the evaluation of the collection becomes, to a degree, more than simply an abstract bibliographic exercise. The lack of access to library materials may indicate the need of additional investigations concerning li- brary security, the need for an inventory, better internal records, or that the card cat- alog is partially obsolete. While a quick and minimum-effort ap- proach to collection evaluation is very de- sirable, the use of just four or five hundred citations out of possibly hundreds of thou- sands as a basis for evaluation could easily be challenged. EVALUATING THE RESULTS With the assumption, for Qualitative In- dex A, that (1) each succeeding level re- quires a cumulative expenditure of effort on the patron/researcher's part and, (2) the material would be more difficult to ob- tain due to age and/or more desirable, a geometric progression of values was as- signed to each level. (See table 1.) In terms of scoring, the interpretation of procedures was perhaps more rigid than if a researcher were actually engaged in a bibliographic search. Unless the exact ref- erence was available-not just listed in the library's bibliographic records-and met all the requirements of language, edition, editor, etc., it was rejected, and in terms of the study, considered as a negative result. The Manitoba study was more flexible in its approach, accepting an edition later than the one cited but not a previous one. This seems to be a reasonable solution-at least in the social sciences, but in the area of literary research with its emphasis upon primary sources and definitive editions, it is perhaps inappropriate. Calculating the "source" could be done in several ways. Index A could be the product of "actual" points (2, 760) in rela- tion to "possible" points, (3,'900) indicat- ing that 70.7 percent of the materials were TABLE 1 QUALITATIVE INDEX A First Level (Value of 5 x 100 = 500) Number of citations 100 Library lacked 34 Citation obtained 66 Research Notes 253 Possible level value (Value of 5 x 100 = 500) Actual level value (Value of 5 x 66 = 330) Second Level (Value of 10 x 100 = 1,000) Number of citations 62t Library lacked 19t Citation obtained 43 Possible level value (Value of 10 x 62 = 620) Actual level value (Value of 10 x 43 = 430) Third Level (Value of 20 x 100 = 2,000) Number of citations 43 Library lacked 10 Citation obtained 32 Possible level value (Value of 20 x 43 = 860) Actuallevel value (Value of 20 x 32 = 640) Fourth Level (Value of 40 x 100 = 4,000) Number of citations 24§ Library lacked 10 Citation obtained 14 Possible level value (Value of 40 x 24 = 960) Actuallevel value (Value of 40 x 14 = 560) Fifth Level (Value of 80 x 100 = 8,000) Number of citations 1i Library lacked Citation obtained 10 Possible level value (Value of 80 x 12 = 960) Actual level value (Value of 80 x 10 = 800) TOTALS-For All Levels Ideal Value 15,000 Possible 3,900 Actual 2,760 *Twenty-five titles were not owned by the libraries, another nine were listed and a weekly search for one semester failed to locate them . tFour of the references at first level had no footnotes, bibliography, etc. tFifteen titles were not owned by the libraries and four others listed were not located in spite of weekly searches for a semester. ~Eight of the third-level citations lacked footnotes, bibliographies, etc . Two of the fourth-level citations lacked footnotes, bibliographies, etc. available. Or, Index A could be derived from the computation of "actual" points (2,760) in comparison with ''ideal'' points (15,500), indicating that only 17.8 percent of the materials were available. The former score is more realistic, as the per- centage sc.ore has been adjusted to com- pensate for those citations lacking refer- ences for the next level. As Nisonger points out, the lack of citations and incom- plete references are characteristic of pri- mary sources and older titles published before the conventions of scholarship be- came standardized. Also, some disci- plines are less oriented toward such bib- liographic apparatus. Consequently, Ni- songer stated that for the adjusted per- centage, "this figure is probably the most significant statistical measurement (at least for our purposes) because it provides the most meaningful basis of comparison between the first and second times the ex- ercise was carried out as well as among the four subject areas." 5 The SUNY Buffalo experience substanti- ates that statement to a degree, but with this qualification: CTCE is used only every five years, and then it is only one part of the evaluation process. At the same time that this in-depth analysis is being made, four or five dissertation bibliographies that were completed at a university ranked by Gorman6 as being first in its field and approximating the ''stated inter- ests" of the relevant academic unit are checked against the library's holdings. No 254 College & Research Libraries attempt is made to obtain the cited items. Percentages of held recorded references are calculated and averaged. These figures are supplemented by the percentage of journals and serials owned by the library and included in the relevant indexes and abstracting services. Thus, three sources of figures are used every five years to pro- file a particular collection: critical bibliog- raphies, the citation technique, and jour- nal/serial availability. Ideally, each set of figures should approximate the other two, but it is reasonable to expect a higher suc- cess result from the dissertation bibliogra- phies as they are simply checked against bibliographic records and the "scores" are not adjusted for lack of availability. Nisonger points out that a lower than expected correlation between the results of the evaluation and the book selection policy could be'' due to any three possibil- ities or a combination thereof: (1) a defect in the evaluation technique; (2) the dis- tinctive nature of [farnpy therapy] as a dis- cipline; or (3) a failure of the University of Manitoba collection to correspond to the standard set in the selection policy. ''7 Per- haps a more thorough awareness of the limitations of CTCE will allow its retention May 1983 and even increase its value as a measure of quality with a more flexible use of its po- tential. The basic procedures of CTCE remain the same but the procedure for scoring is reversed, particularly for those disciplines that have only recently been developed such as public policy, medical sociology, sociobiology, black studies, or those sub- ject areas such as the sciences, social wel- fare, nursing, etc., where the emphasis is upon recent information and publica- tions. Thus the assumptions underlying Index Bare: (1) the most current research/ literature is of the greatest value to the re- searcher, (2) the information explosion of the last two decades has resulted in de- creasing probability that the library would own a particular needed item. (See table 2.) This approach produced an index of 77.9 percent, which is the result of "ac- tual" points (9,490) in the "possible" points (12,170). While both indexes A and B are calculated regardless of discipline, Index B appears to be more accurate when compared with the other facets of collec- tion evaluation. These indexes have not been used in the area of the humanities. TABLE2 QUALITATIVE INDEX B First Source (Value of 80 x 100 = 8,000) Number of citations 100 Library lacks 25 Citations obtained 75 Possible level value (Value of 80 x 100 = 8,000) Actual level value (Value of 60 x 100 = 6,000) Second Source (Value of 40 x 100 = 4,000) Number of citations 75 Library lacks 16 Citations obtained 59 Possible level value (Value of 68 x 40 = 2,720) Actual level value (Value of 59 x 40 = 2,360) Third Source (Value of 20 x 100 = 2,000) Number of citations 59 Library lacks 12 Citations obtained 42 Possible level value (Value of 54 x 20 = 1,080) Actual level value (Value of 42 x 20 = 840) Fourth Source (Value of 10 x 100 = 1,000) Number of citations 42 Library lacks 9 Citations obtained 29 Possible level value (Value of 37 x 10 = 370) Actual level value (Value of 29 x 10 = 290) TOTALS For All Levels Ideal Value 15,000 Possible 12,170 Actual 9,490 As it has already been noted, the needs of literary and fine arts investigation are ex- tremely compartmentalized resulting in highly restrictive bibliographies inappro- priate to use in CTCE situation. In con- trast, as social scientists are inclined to cite in one article a range of references from Plato to Pogo, an evaluation of one ~ocial science collection becomes an evaluAtion of the total resources of a library. SUMMARY The use of CTCE has been limited, in SUNY at Buffalo libraries, to the social sci- ence collections in conjunction with the more conventional methods of collection evaluations. Periodic, at five-year inter- Research Notes 255 vals, application has supplied useful com- parative data sufficiently accurate to reasonably predict the effects of modifica- tions in funding, collection policies, etc. Unfortunately, art rather than science re- mains the prime factor in CTCE. The choice of the initial bibliographies to use is subjective, and vulnerable to bias and er- ror. And the results remain a question of opinion. What is an acceptable percentage for Index A? Or should Index B be used? While the answer(s), according to each in- stitution's circumstances and objectives, will be different, however, the citation technique of collection evaluation will provide some indication of possible pa- tron satisfaction. REFERENCES 1. Manuel D. Lopez, ''A Guide for Beginning Bibliographers,'' Library Resources & Technical Services 13:462-70 (Fall1969) . 2. Thomas E. Nisonger, "An In-Depth Collection Evaluation at the University of Manitoba Library: A Test of the Lopez Method," Library Resources & Technical Services 24:337 (Fall1980). 3. Ibid., p .329. 4. Ibid., p.334. 5. Ibid ., p.331. 6. Jack Gorman, The Gorman Report: A Rating of Graduate and Professional Programs in American and Inter- national Universities (Los Angeles: National Educational Standards, 1980). 7. Nisonger, "An In-Depth Collection Evaluation," p .336. fr-; ~ I A . ·t· L.b . ~ ~ cqutst ton 1 rartans . • • t t I l : I I II J j t Now is the time to try the best source for: • any book in pri11t e ACCURATE INVOICII\G e meaningful reports • I r' ' • rUSh Order Ser\lCe • competitil·e discounts t e plus many other scn·ices I ! CALL TOLL-FREE TODAY I 1-800-248-1146 I I In Canada & Michigan I i CALL coLLECT (517) 849-2117 ACCt..:RACY IN BILLil\G & SIHP- PII\G..... to the spc<~ ifications ~· ou dc sin•. l11\·oiccs arc arranged ctlpha- betically b~· author or title, or numer- ically by purchase order. Separate billing anilable upon request. the BOOK HOUSE JOBBERS SERVING LIBRARIES WITH ANT BOOK IN PRINT SINCE 1M2 208 WEST CHICAGO STREET JONESVILLE . MICHIGAN 49250 SAN 169-3859 I · New from Noyes Available Spring 1983 ADHESIVES TECHNOLOGY-- Developments Since 1979 edited by Marcia Gutcho : ISBN 0-8155-0921-9 ; $48 . ALCOHOLS TOXICOLOGY hy William W. Wimer, John A. Russell and Harold L. Kaplan: ISBN 0-8155-0.948-0; $36. APPLIED GENETIC ENGINEERING-Future Trends and Problems by Morris A. Levin et al: ISBN 0-8155-0925 -1; $24. THE BIOLOGY AND MANAGEMENT OF AN EXTINCT SPECIES: PEREDA VID 'S DEER edited by Benjamin B. Beck and Christen Wemmer: ISBN 0-8155-0938-3; $26. CHEMICAL HAZARDS TO HUMAN REPRODUCTION by Ian C.T. Nisbet and Nathan J. Karch: ISBN 0-8155-0931-6 ; $28. CLINICAL TOXICOLOGY OF AGRICULTURAL CHEMICALS by Sheldon L. Wagner: ISBN 0-8155-0930 -8; $28. COAL PREPARATION PLANT AUTOMATION by J.C. Moyers et al: ISBN 0-8155-0942-1 ; $39. COGENERATION TECHNOLOGY ANO ECONOMICS FOR THE PROCESS INDUSTRIES edited by D.J . DeRenzo: ISBN 0-8155-0932-4; $42. DESTRUCTION AND DISPOSAL OF PCBs BY THERMAL AND NON-THERMAL METHODS by D.G. Ackerman et al: ISBN 0-8155-0934-0 ; $48. EDIBLE OILS AND FATS--Developments Since 1978 edited by S. Torrey: ISBN 0-8155-0923-5 ; $44. EMERGING TECHNOLOGIES FOR THE CONTROL OF HAZARDOUS WASTES by B.H . Edwards, J.N. Paullin and K. Coghlan.Jordan: ISBN 0-8155-0943-X; $24. ENZYMATIC HYDROLYSIS OF CELLULOSE-Theory and Applications by C.R. Wilke et al: ISBN 0-8155-0945-6 ; $24. FOOD ADDITIVES··-Recent Developments edited by J .C. Johnson: ISBN 0-8155-0935-9 ; $45 . FUGITIVE DUST CONTROL TECHNOLOGY by J.A. Orlemann et al and G.A. Jutze et al: ISBN 0-8155-0933-2; $48. GAZELLES AND THEffi RELATIVES·-A Study in Territorial Behavior by Fritz R . Walther, Elizabeth Cary Mungall and Gerald A. Grau: ISBN 0-8155-0928-6; $28. HANDBOOK OF INDUSTRIAL RESIDUES by Jon C. Dyer and Nicholas A. Mignone: ISBN 0-8155-0924-3; $54. HANDBOOK OF NUTRITION, HEALTH, AND AGING by Donald M. Watkin: ISBN 0-8155-0929-4 ; $32. HANDBOOK OF PLASTICS FLAMMABILITY AND COMBUSTION TOXICOLOGY by Arthur H. Landrock: ISBN 0-8155-0940-5 ; $36. HORSE BEHAVIOR by George H. Waring : ISBN 0-8155-0927-8 ; $35. INDUSTRIAL ENERGY CONSERVATION TECHNOLOGIES & RESEARCH OPPORTUNITIES edited by D.J. DeRenzo: ISBN 0-8155-0941-3 ; $64. INDUSTRIAL WATER TREATMENT CHEMICALS AND PROCESSES edited by M.J. Collie: ISBN 0-8155-0936-7 ; $42 . INSECTICIDE MANUFACTURING-Recent Processes and Applications edited by M.J. Satriana: ISBN 0-8155-0920-0 ; $45. LANDFILL METHANE RECOVERY edited by M.M. Schumacher: ISBN 0·8155-0946-4; $54 . LAND TREATMENT OF HAZARDOUS WASTES edited by James F. Parr, Paul B. Marsh and Joanne M. Kla: ISBN 0-8155-0926-X; $45. MICROBIOLOGICAL SYNTHESES-Recent Advances edited by S. Torrey: ISBN 0-8155-0937-5 ; $48. REMEDIAL ACTION TECHNOLOGY FOR WASTE DISPOSAL SITES by P. Rogoshewski, H. Bryson and K. Wagner : ISBN 0-8155-0947-2 ; $36. SPILL PREVENTION & FAIL.SAFE ENGINEERING FOR PETROLEUM & RELATED PRODUCTS by J.L. Goodier, R.J. Siclari and P A. Garrity: ISBN 0 -8155-0944-8; $36. TEXTILE WET PROCESSES--Vol. 1 : Preparation of Fibers and Fabrics by EdwardS. Olson: ISBN 0 -8155-0939 -1 ; $28. TREATMENT, RECOVERY, AND DISPOSAL PROCESSES FOR RADIOACTIVE WASTES edited by J.I. Duffy: ISBN 0-8155-0922 -7; $39. ndc NOYES DATA CORPORATION NOYES 120 Mill Road, Park Ridge, NJ 07656 np NOYES PUBLICATIONS