College and Research Libraries DAVID R. McDONALD, MARGARET -w. MAXFIELD, AND VIRGINIA G. F. FRIESNER Sequential Analysis: A Methodology for Monitoring Approval Plans Sequential analysis is a statistical method based on drawing sample items one at a time, with a decision at each stage whether further sampling is necessary to reach a conclusion . This methodology is shown to provide a convenient and appropriate method for investigating the performance of a library approval plan. The sequential approach minimizes the investment in staff time, by delaying, until a title is chosen for the sample, the determina- tion of its status in the approval plan . Application of the methodology at an academic library is reported in detail, and adaptations to other library situ- ations are explained. ALTHOUGH APPROVAL PLANS have become widely used in the last ten years as an im- portant acquisitions mechanism , doubts have been expressed regarding the reliabil- ity of such plans. McCullough, Posey, and Pickett note that looming largest among these [shortcomings] is the uncertainty factor . ... Except in a minority of cases, it is difficult to guarantee that a specific book will be produced by an approval plan. 1 Dudley has indicated that one of the charges against an approval plan, based on experience and the literature, is the un- certainty of knowing whether a specific book will arrive . . .. 2 Librarians using approval plans must con- tend with two major problems: (1) receipt of unwanted material and (2) nonreceipt of wanted material. Receipt of unwanted books David R . McDonald is associate librarian, Cubberley Library, Stanford University Librar- ies, Stanford, California. Margaret W. Maxfield is assistant professor, College of Business Admin- istration, Kansas State University, Manhattan . Virginia G. F. Friesner is director, Library De- velopment, Kansas State Library, Topeka. The authors express their thanks to Valera Lowe, Janet Brown, Ann Scott, and Carole Francq, all of the Kansas State University Library, for their comments on the study and the paper. is admittedly troublesome and entails some expense. However, failure to receive desired mate- rial poses, by far, the more serious conse- quences. To the degree that the approval plan breaks down and wanted books are not received, (1) the library staff must attempt to fill the gaps on a title-by-title basis, a difficult task when it is impossible to predict what will and will not arrive on approval; and (2) gaps discovered by patrons fre- quently result in complaints and poor evalu- ation of library service. Approval plans are beneficial only to the extent that staff time and paperwork asso- ciated with the selection and acquisition of material are reduced. Any attempt to monitor approval plans on a title-by-title basis in order to overcome the "uncertainty factor" defeats the purpose of approval plans as staff time devoted to selection and searching is not significantly reduced. Monitoring of approval plans is important, for much of the uncertainty surrounding the receipt of approval material appears to stem from differing expectations and interpreta- tions on the parts of vendors and of librar- ians. An objective monitoring plan, coupled with detailed analysis, can do much to rec- oncile differing interpretations. To date, however, most reports concerning the I 329 330 I College & Research Libraries • july 1979 monitoring of approval plans have been in- formal and descriptive. 3 •4 Librarians frequently retain a subjective impression of approval plan performance as a result of user complaints and time spent supplementing the plan; however, librarians seldom have facilities for collecting and analyzing quantitative data. Sequential analysis refers to a statistical method in which sample items are drawn one by one . . . and the results of the drawing at any stage decide whether sampling is to continue. The sample size is thus not fixed in advance but depends on the actual results and varies from one sample to another. The sampling terminates according to predetermined rules which are decided by the degree of precision re- quired.5 Sequential analysis offers several advan- tages: 1. It requires a minimun of staff time and essentially no other expense. 2. It can be used for a single evaluation or for periodic monitoring. 3. It yields quantitative data of value in explanations to patrons and to adminis- trators . 4. It should lead naturally to analysis and correction of any weaknesses found in the approval plan. In a recent study at Kansas State Univer- sity (KSU) Libraries a sequential analysis was performed to find how well the library's interpretation of an approval plan, particu- larly the profile, coincided with that of the vendor. Sequential analysis, in other words, was used to investigate whether the per- centage of "defects" (that is, books that the library expected to receive on approval but did not) was low enough to indicate good performance (Accept rate), or whether ad- justments were indicated (Reject rate). Sampling, rather than 100 percent screen- ing, was necessary as a practical matter be- cause of the number of titles involved and the limited amount of staff time that could be devoted to the project. The purpose of the sampling experiment was to allow statistical inference as to the degree of conformity between the interpre- tation of librarians and the interpretation of the vendor's book selectors. Each instance in which the library expected to receive a book on approval but did not was counted as a "defect," with an "Accept" result for a low percent defective and a "Reject" result for a high percent defective. Once a decision was made to conduct a sampling experiment, the next decision was whether to use a fixed-size sampling plan or a sequential plan-whether to choose the sample size in advance or to sample until enough evidence is accumulated to warrant an inference. A sequential approach was especially advantageous in this study be- cause of the difficulty of determining the necessary sample size for a fixed sampling plan in advance. When conclusions are generalized from a statistical sample, there are risks that an atypical sample (sampling "error") will lead to the wrong conclusion. It is especially difficult to decide in advance on some fixed number of titles to sample when the vari- able is "percent defective" as in the KSU study. The problem is that the sample size necessary to achieve reasonable risks de- pends on the variability from sample to sample, and the variability depends, in turn, on the unknown percent defective. Fortunately, this circularity can be avoided by use of a sequential plan whenever, as in the approval monitoring application , it is convenient and economical to draw and in- spect one item at a time . As noted above, in a sequential sampling plan the size of the sample is not deter- mined in advance of the experiment as it is in a fixed sampling plan. Sampling, in a sequential plan, simply continues until an "Accept" or "Reject" conclusion is supported by the evidence. So long as the evidence is inconclusive , the investigator continues to draw additional sample items. It should be pointed out that, as sug- gested by capitalization, the words "Accept" and "Reject" are used in a technical sense. "Accept" and "Reject" do not refer to accep- tance or rejection of the approval plan as a whole , and at no time was rejection of the entire approval plan under consideration by the KSU Libraries. Nor do the terms refer to acceptance or rejection of specific books. Rather, sequential analysis was used as a device to "tune" the approval plan. In a sequential analysis plan the sample size that will be necessary to reach a con- clusion is not known definitely in advance. However, the average sample number (ASN) can be calculated for various actual defect rates. If the actual defect rate is very high, a conclusion follows quickly, on the average. Similarly, the ASN for a very low actual defect rate is small. The largest ASN values occur at intermediate percentages . The saving in sample size (and therefore in time and money) in a sequential plan over a fixed-size plan often amounts to 50 percent. For that reason, sequential plans are in wide use in production quality control and accep- tance sampling. Details of sequential plans are available in several references. s, 7 ,s METHODOLOGY The choice of a particular plan amounts to the choice of an "Accept" defect rate p 1 with a related risk a, and a "Reject" defect rate P2 with a related risk {3. In the KSU study the "Accept" rate, that is , the accept- able level of conformity between the in- terpretations of iibrarians and vendors, was set at Pl = 2 percent defective, with a ("producer's," i.e ., vendor's) risk of a = 5 percent of an atypically bad sample leading to an error . The "Reject" rate was set at P2 = 10 percent defective, with a ("con- sumer's," i.e., librarian's) risk of {3 = 10 percent of an atypically good sample leading to an error. Other values, adapted to the needs of other libraries, can easily be sub- stituted for the ones used here . The sequential sampling chart shown in figure 1 was drawn according to speci- fications in standard references for the set of values p 1 = 2 percent, p 2 = 10 percent, a = 5 percent, and {3 = 10 percent, used in the KSU study . As explained below, critical line boundaries separate the two decision regions, "Accept" and "Reject," from the region of continued sampling. The 1977 edition of American Book Pub- lishing Record9 was used as a source of ti- tles for sampling. Five-digit sequences were drawn from a large random number table (uniform distribution), 10 entered randomly and read in a predetermined order to avoid bias. Since individual titles are not num- bered in the American Book Publishing Record, an estimate was made of the number of titles per page. Each random se- que-nce was decoded to represent a page number and then the number of a specific Sequential Analysis I 331 title on the page. For example, the number 12,305 would be read as page 123, the fifth title. The purpose of the study was to deter- mine the extent to which the li!">rary' s (i.e., subject bibliographers') interpretation of the approval plan profile coincided with the in- terpretation of the vendor (i.e., the vendor's book selectors). That is, did the approval plan supply the titles that the subject bib- liographers expected it to supply? Consequently when a title was drawn by this method, it was included in the sample only if it was expected to arrive on the ap- proval plan. Subject bibliographers · re- viewed each title drawn, in order to deter- mine whether the books were expected to arrive on approval. If the book was not ex- pected on approval, it was not included in the sample and a new random number wa drawn. If the book was expected, it was in- cluded in the sample and the cumulative sample size n was increased by one . A record trace was entered on the se- quential sampling chart (figure 1) as sam- pling continued. Whenever the cumulative sample size n increased (because the title drawn was an expected book) , the record trace was extended one unit to the right . If an expected book was not received on approval, it was counted as a defect, the cumulative defect number d was increased by one, and the record trace was extended one unit up. If the book had been received on approval, the cumulative defect number d was not changed and the record trace was not moved upward, but held at the same height on the chart. The record trace for the KS U study (bold step trace in figure 1) shows by upward steps that defects were found at items 10, 14, and 19. When the record trace entered the Reject region in figure 1, that is , when the third defect occurred at item 19, sampling was discontinued. Enough evidence had ac- cumulated to support a statistical "Reject" conclusion indicating need for a clearer understanding between the library and the vendor regarding the approval plan . The staff time involved was minimal-only five or six hours after preliminary conferences on goals and methods. It must be pointed out that questions concerning the status of individual books 332 I College & Research Libraries • july 1979 I v d, Cumulative Number / of Defects REJECT / 4 3 ~v / / CONTINUE v I TESTING I 2 RECORD I TRACE~ I I I / 1 I I v ~ I I I / ACCEPT I I I I I / I 0 I 0 10 30 40 50 14 19 n, Cumulative Sample Size Key: For a = 5% , {3 = 10%, PI = 2%, P2 = 10% Accept: d = -1.3285 + 0 .05025 n Reject: d = 1.7056 + 0.05025 n Fig. 1 Sequential Sampling Chart did arise during the monitoring program . In some cases communication with the vendor was necessary before it could be determined that a book was not expected and so should be dropped from the sample count n and the defect count d. In borderline cases the librarian , after gathering information from the vendor and elsewhere , should deter- mine status according to the reflection on approval plan performance. RESULTS AND RECOMMENDATIONS The second component of the study was the detailed analysis of individual defects in order to identify generic problems (see table . 1). The results verified and documented the impression of the librarians. The three de- fects identified resulted , as expected, from differing interpretation and understanding of the approval plan by the vendor and the KSU Libraries. Librarians verified that titles published simultaneously in the United States by a United States publisher and in the United Kingdom by a United Kingdom publisher were automatically excluded from the domestic approval plan. Some excluded ti- tles were supplied through a U.K. approval TABLE 1 ANALYSIS OF DEFECTS Classification Interface with foreign publications Interface between subject areas Previously published material Example Library expected to receive books published simultane- ously in the U.S. by a U.S. publisher and in the U.K . by a U.K. publisher. However, such titles are automatically excluded from the domestic approval plan and supplied through a U .K. plan Instructions from the library to the vendor to exclude medical materials resulted in exclusion of materials on the politics and sociology of medicine that were expected Instructions to exclude pre- viously published material led to the exclusion of books con- taining some previously pub- lished articles, along with new material plan; however, the U.K. plan was limited in coverage relative to the domestic plan, and as a result many titles were missed. In a similar vein the library had excluded medical material from its approval profile, since KSU does not include a medical school. Unfortunately, the deletion of medi- cal material also resulted in the exclusion of material concerned with the politics and sociology of medicine, topics of interest to the library. Similarly, instructions to exclude pre- viously published material from the profile also resulted in the exclusion of titles which included a mix of original and previously published articles. Clearly all these defects stemmed from inconsistencies in interpreta- tion or understanding on the part of the li- brary and vendor. With the inconsistencies identified and the necessary changes made, the defect rate can be expected to decrease, so that a second analysis in six to twelve months may well lead to an Accept conclu- sion. The methodology explained here is adapt- able to the needs of a wide range of librar- ies, since sequential plans for various Ac- cept and Reject rates are readily available. The choice of the Accept rate p 1 is usually based on practical considerations, since more samples, on the average, are required if p 1 is decreased. Sequential Analysis I 333 The choice of the Reject rate P2 affects sample cost, but it can also be chosen by a given library to reflect the perceived role of the approval plan. If the approval plan is viewed primarily as an acquisitions device rather than as a collection development tool, then a defect rate of 15 percent may be reasonable. However, if the collection development role is emphasized, a reason- able defect rate may be 10 percent or less. The role of the approval plan, therefore, in- fluences significantly the definition of what is and is not a reasonable defect rate. The use of sequential analysis as a monitoring methodology has important im- plications for collection development. Se- quential analysis involves only a small in- vestment of sl:aff time, may be repeated over a period of time, results in identifica- tion of specific problems and defects, and provides quantitative data of value in deal- ing with vendors and patrons. In short, sequential analysis enables col- lection development librarians to "fine tune" their approval plan. Collection de- velopment librarians can, in other words, ensure that approval plans function ·correctly without investing so inuch staff time that the purpose of the approval plan is de- feated. Sequential analysis leads to an Accept conclusion or to a Reject conclusion. If an Accept conclusion is reached, librarians and patrons can be assured that receipt of mate- rial is generally consistent with the profile. A Reject conclusion indicates some inconsis- tency of interpretation between librarians and veJidors, a situation that frequently re- sults in collection gaps . Resolution of such inconsistencies through further analysis can lead to improved performance of the ap- proval plan and of library service in general. REFERENCES 1. Kathleen McCullough, Edwin D . Posey, and Doyle C. Pickett, Approval Plans and Academic Libraries (Phoenix: Oryx Press, 1977), p.133. 2. Norman Dudley, "The Blanket Order," Li- brary Trends 18:318-27 Gan. 1970). 3 . G. Edward Evans and Claudia White Ar- gyres, "Approval Plans and Collection De- velopment in Academic Libraries, " Library Resources & Technical Services 18:35-50 (Winter 1974). 334 I College & Research Libraries • july 1979 4. Mary Lee DeVilbiss , "The Approval-Built Collection in the Medium-Sized Academic Library," College & Research Libraries 35:487-92 (Nov. 1975). 5. Maurice C. Kendall and William R. Buck- land, A Dictionary of Statistical Terms (Lon- don: Longman for the International Statistical Institute, 1971), p . 136. 6. Columbia University . Statistical Research Group, Sequential Analysis of Statistical Data: Applications (New York: Columbia Univ. Pr. , 1945). 7. Dudley J. Cowden, Statistical Methods in Quality Control (Englewood Cliffs: Prentice-Hall, 1957). · 8. Eugene L. Grant and Richard S. Leaven- worth, Statistical Quality Control (New York: McGraw-Hill, 1972). 9 . Amer.ican Book Publishing Record (New York: Bowker, 1977). 10. Charles E. Clark, Random Numbers in Uni- form and Normal Distribution: With Indices for Subsets (San Francisco: Chandler Publish- ing Co ., 1966). ABOUT COLLEGE & RESEARCH LIBRARIES College & Research Libraries (C&RL) is the official publication of the Association of College and Research Libraries, a division of the American Library Association. It is published seventeen times a year-six bimonthly journal issues and eleven monthly (combining July-August) news issues. As the official publication of ACRL, College & Research Libraries maintains a rec- ord of policy statements and actions taken by the association. In reporting activities of ACRL and its sections, it thus serves as the principal medium of communication among ACRL membership. The publication is also a medium for professional communication among academic and research librarians. The journal issues contain reports of research and thoughtful 1 articles on matters of current and continuing concern, communications from readers related to those articles, and reviews and announcements of important and relevant publications. The news issues report activities and programs of the association, information and news related to academic and research libraries and their personnel, and brief notices of relevant publications. General information on submission of manuscripts may be found on the masthead page in each issue. More detailed information on the journal and news issues is in- cluded in a statement in the April 1978 issue of College & Research Libraries News, pages 86 and 87.