blecic.p65 434 College & Research Libraries September 2001 The Measurement of Use of Web- based Information Resources: An Early Look at Vendor-supplied Data Deborah D. Blecic, Joan B. Fiscella, and Stephen E. Wiberley, Jr. To manage Web-based resources effectively, librarians need to evalu­ ate vendor-supplied data about their use. This article explores the types of data available, using as its starting point the elements defined by the International Coalition of Library Consortia’s (ICOLC) “Guidelines for Statistical Measures of Usage of Web-based Indexed, Abstracted, and Full-text Resources.” It discusses the problems and issues of comparing use data from different vendors. Then, illustrated with data from one library, the article addresses five measures that have implications for collection management: variability of ICOLC data elements over time, which demonstrated the need to examine data continually; ratios of que­ ries per session, which showed more stability over time than individual ICOLC elements; use by hour, which documented remote use but con­ firmed that most use occurred during regular library hours; use of elec­ tronic journal collections, which was more scattered than the classic 80/ 20 distribution; and use of Web-based resources in relation to a disci­ plinary population, which provided an index of value for assessing use of a particular resource. This study identifies aspects of data collection that librarians need to pay special attention to, recommends that ven­ dors report the maximum number of simultaneous users per day and data gaps in addition to ICOLC elements, and suggests per capita use as a comparative measure among libraries. he proliferation of Web-based resources has greatly increased the information that libraries can deliver to their users’ desk­ tops. These resources have great power and promise but come with substantial cost—the price of licensing them. Elec­ tronic versions of products often cost more than the print copy, and in most cases, the license allows for only a year of access, whereas the library purchases the print copy outright and can keep it Deborah D. Blecic is Bibliographer for the Life and Health Sciences and Associate Professor, Joan B. Fiscella is Bibliographer for Professional Studies and Associate Professor, and Stephen E. Wiberley, Jr. is Bibliographer for the Social Sciences and Professor at the University of Illinois at Chicago; e-mail: dblecic@uic.edu, jbf@uic.edu, and wiberley@uic.edu, respectively. The authors wish to thank John Cullars, Eric Novotny, and Ann Weller for their helpful reviews of this manuscript; Renee Schwartz for providing relevant background information; and Judy Luther and Tom Peters for their comments on measures of Web-based resource use. 434 mailto:wiberley@uic.edu mailto:jbf@uic.edu mailto:dblecic@uic.edu The Measurement of Use of Web-based Information Resources 435 indefinitely. Librarians working with fi­ nite funds must evaluate the use of elec­ tronic resources to maximize the impact of their expenditures. Fortunately, computers have remark­ able abilities to track the way people use Web-based resources through software that captures the transactions of patrons. Although librarians can approximate the number of log-ins to a resource by count­ ing the number of times patrons go To gather data, the investigators either drew on statistical reports sent regularly by vendors or retrieved data from password-protected Web sites provided by vendors. through the library’s Web gateway to that resource, this measure misses many other dimensions of use, such as number of queries, that only monitoring on the vendor ’s server can provide. Currently, the library community has proposed stan­ dards for reporting use statistics for elec­ tronic resources—the International Coa­ lition of Library Consortia’s “Guidelines for Statistical Measures of Usage of Web- based Indexed, Abstracted, and Full-text Resources” (hereafter called the ICOLC guidelines).1 A few vendors fully meet these standards, some partially comply, and others supply none of the ICOLC el­ ements. Although some vendors argue that they cannot afford to provide statis­ tics, Tom Peters has expressed skepticism at their claims of excessive costs.2 Judy Luther has taken an evolutionary view, reporting that “the industry is at the first stage of creating the capability to gather statistics, establish standards, and deliver comparable and reliable data.”3 The bot­ tom line is that for good management, li­ braries must have the kinds of statistics called for by ICOLC and the additional data recommended by the present article. Because Web-based resources are so new, the availability of data on their use is a recent phenomenon and there has been little time for data-intensive studies on use. The authors could find only a few studies that report analysis of data com­ parable to that discussed here. Of note is the investigation, reported in four articles, that Carol Tenopir and colleagues con­ ducted about database use patterns in ninety-six (ninety-three in one study) aca­ demic and ninety-nine public libraries.4–7 Charles T. Townley and Leigh Murray reported a case study on the use of net- worked (including CD-ROM, locally mounted, and Internet-based) resources at six academic libraries.8 Rather than summarize the findings and conclusions of these authors, this article refers to their work at relevant places below. The present authors examined data on the use of Web-based electronic resources in 1999 and in the first eight months of 2000 that vendors supplied to a library that serves a research university with 25,000 students, 1,750 teaching faculty; ninety-two bachelor ’s, eighty-five master ’s, and fifty-five doctoral pro­ grams; and a comprehensive health sci­ ences center. The library had an electronic resources budget of $360,000 in FY1999 and $475,000 in FY2000. What follows describes current conditions and major issues—what kinds of statistics are avail­ able, additional measures derived from the statistics, and what improvements will help librarians better serve their us­ ers. Because Web-based resources are a relatively recent development, much will change as time passes. Regardless, be­ cause of the importance of analysis and interpretation of data on the use of Web- based resources, it is essential for the li­ brary profession that there be early ex­ ploration of the subject. Study Design The present study began by determining the use data available for Web-based re­ sources from fifty-one vendors. The ven­ dors included publishers, aggregators, and consortia that distribute electronic resources. The resources examined fall into four categories: 1. indexing and abstracting databases; 2. collections of full-text e-journals; 3. directory or reference databases with full-text records; 436 College & Research Libraries September 2001 4. mixed databases with several seg­ ments, such as MD Consult, which in­ cludes full-text reference books, indexing and abstracts of journal articles, and full- text of selected journal articles. A key concern was how vendors ap­ plied the proposed ICOLC guidelines. Essentially, the guidelines call for five “elements,” or categories of data, that vendors should provide libraries about the use of a resource.9 ICOLC has identi­ fied and defined, or in some way de­ scribed, these elements as follows: • Queries (searches) are “unique in­ tellectual inquir[ies] … typically … re­ corded each time a search form is sent/ submitted to the server.” • Menu selections occur when “dis­ play of data is accomplished by brows­ ing (use of menus).” In such circum­ stances, “the number of alphabetic and subject menu selections should be tracked.” • Sessions (log-ins) “if relevant, must be provided as a measure of simultaneous use.” • Turnaways occur when “requests exceed simultaneous user limit.” • Items examined include data units “viewed, marked or selected, down­ loaded, emailed, printed [when this is recorded and] controlled by the server.” In the fall of 2000, of the fifty-one ven­ dors studied, three reported all ICOLC elements relevant to their resources, twenty-eight supplied selected elements, and twenty offered none. To gather data, the investigators either drew on statisti­ cal reports sent regularly by vendors or retrieved data from password-protected Web sites provided by vendors. They then entered the data into spreadsheets for analysis. The investigators did not study use of resources whose vendors only sup­ plied statistics on demand. With thirty- one vendors supplying data, often for multiple resources, the data-gathering process was labor-intensive. The differences among vendor-sup­ plied statistics were a central problem and will be the first one the present article addresses. From the data available, five measures emerged that have implications for collection management: 1. the variability of ICOLC data ele­ ments over time; 2. the ratios of queries per session for searchable databases; 3. hourly use; 4. uses of e-journal collections; 5. the ratio of uses of Web-based re­ sources per FTE in the disciplinary popu­ lation. The application of each measure is dis­ cussed later in this article. Comparing Use Across Vendors: Problems and Issues The ICOLC statistical categories have the virtues of being few in number and straightforward, but their simplicity belies the difficulty of applying them. Vendors or, in the case of locally loaded databases, the local systems administrators control what is reported. What they report de­ pends in part on the computer-monitor­ ing software they use and in part on how they label, define, and count activity. For example, in the present study, one vendor segmented its database into two parts and counted a single query twice when both segments were accessed. Other databases, even if similarly segmented, may count such searches as only one query. Thus, use statistics for the first database may appear higher than use statistics for the second database, even if they are arguably equal. Comparison of different vendors’ re­ sources also can be complicated by the ways that each statistical program counts repeated uses of a document within a ses­ sion and linking to new documents from a chosen document. Further, if vendors get new monitoring software, this may change the statistics reported and/or their mean­ ing, as was encountered twice in the present study. These examples demon­ strate that librarians must use updated documentation and explanatory materials to properly interpret current numbers and must retain older documentation to make comparisons over time. The authors rec­ ommend an annual review of the way each vendor counts the elements it reports. The Measurement of Use of Web-based Information Resources 437 Another factor contributing to the dif­ ficulty in comparing and interpreting use statistics is that basic session or log-in in­ formation may be compromised at public workstations if several different users can search a database without logging out in turn. Alternatively, a resource timing-out due to lack of interaction with it may mean that some users need to log in more than once per sitting (arguably a single session) to complete their work. Librarians would benefit from research that could ascertain whether length of time between interac­ tions with a resource while actively using it (e.g., reading a downloaded article onscreen) varies among different types of Web-based resources (e.g., longer gaps on humanities databases and shorter ones on medical databases). Such research would aid in setting time-out limits that minimize interruptions to users and maximize si­ multaneous use. The avenue of access to a resource also may influence use statistics. Libraries in a consortium, for example, may have the option of subscribing to resources from particular vendors either directly or through the consortium. In one notable instance in the present study, the library subscribed to resources from the same vendor in two different ways: some re­ sources were licensed directly, and oth­ ers were negotiated through a consor­ tium. The consortium loaded the latter resources on its computer and used moni­ toring software that was different from the vendor ’s software. As a result, the consortium reported only sessions, whereas the vendor offered complete ICOLC statistics. Finally, upon examination of daily data for several resources, the present study discovered that some days were missing, yet monthly and yearly summaries gave no indication of the gaps. To inform li­ brarians about the integrity of use statis­ tics, vendors should report when data have been lost or compromised. Measures of Use and Their Import Currently, the basic measures of Web-based resources are the ICOLC elements. The present study analyzed ICOLC-compliant data supplied by vendors. In the course of the analysis, the authors identified addi­ tional measures that could prove valuable for libraries. These included variability of data over time; queries per session; hourly use; the number of titles providing a given proportion of use in e-journal collections; and use in relation to disciplinary and in­ stitutional populations. Variability of Data over Time and Related Issues Electronic resource use data provide very helpful information, but their collection must be efficient. Thus, an important is­ sue is how often the library has to collect data. Most vendors included in this study provided monthly data. If use is stable from month to month, handling data twelve times a year is unnecessary and an annual report would do. In contrast, if use varies greatly from month to month, monthly data are necessary for a true un­ derstanding of how resources are used. In an academic library, one would expect some variability because of the changing need for information during the course of the academic year. To determine the variability of elements over time, the study team calculated the coefficient of variation, which is the ratio of the standard deviation to the mean. The coefficient of variation normalizes data and allows for comparison among resources with widely different numerical ranges of use. A low coefficient signifies little data scatter compared to the mean over the time periods studied; the coefficient of variation increases as data scatter relative to the mean increases. A coefficient of 0.2 is rela­ tively low, indicating that the data vary, on average, by 20 percent relative to the mean. A coefficient of 0.8 or above is relatively high, demonstrating that the data vary, on average, by a degree of 80 percent or more relative to the mean. The analysis of the variability of ICOLC elements and of que­ ries per session (see discussion of this ra­ tio in the next section) for the products studied can be found in table 1. There are some noteworthy patterns. 438 College & Research Libraries September 2001 TABLE 1 Variance of Monthly Freguencies by ICOLC Elements Resource Time Span Mean Std. Dev. Coer. Var. Sessions Linguistics & Language Behavior Abstracts 8/99-5/00 50.40 39.14 0.78 Art Abstracts 1999 228.83 176.88 0.77 Reader's Guide Abstracts 1999 470.17 359.76 0.77 Humanities Abstracts 1999 464.50 351.63 0.76 Social Work Abstracts 8/99-5/00 126.20 94.13 0.75 Predicasts PROMT 1999 62.33 46.12 0.74 America: History and Life 1999 136.67 95.42 0.70 PAIS International 1999 124.00 84.23 0.68 General Business File ASAP 1999 207.83 133.37 0.64 Historical Abstracts 1999 94.25 60.79 0.64 Social Sciences Abstracts 1999 1,065.58 680.59 0.64 PsycINFO 1999 3,080.92 1,817.11 0.59 Library Literature 1999 134.50 75.81 0.56 Biological and Agricultural Index 1999 238.67 129.77 0.54 General Science Abstracts 1999 294.58 156.39 0.53 Cambridge Scientific Abstracts 1999 174.75 84.27 0.48 ERIC 1999 1,168.08 520.19 0.45 Applied Science & Technology 1999 405.75 173.95 0.43 Ovid (health sciences databases)* 1999 2,372.25 887.37 0.38 Bowkers Books in Print 1999 143.83 45.90 0.32 Current Contents 1999 1,195.92 345.54 0.29 Ideal 12/98-11/99 1,026.64 289.43 0.28 Web of Science 1-6;8-12/99 1,358.36 367.82 0.27 Beilstein 1999 497.58 120.10 0.24 MDConsult 8/99-3/00 1,855.55 381.94 0.21 Queries Cambridge Scientific Abstracts 1999 370.58 329.48 0.89 Peterson's UndergradSearch 1999 24.67 21.56 0.87 ABI/INFORM 1999 1,034.75 848.15 0.82 Gale Literature Databases 1999 104.40 77.48 0.74 Econlit 1999 246.75 172.14 0.70 Periodical Abstracts 1999 385.25 268.34 0.70 America: History and Life 1999 204.83 141.04 0.69 Historical Abstracts 1999 119.00 76.93 0.65 Wilson Select 1999 953.00 619.64 0.65 Contemporary Women's Issues 1999 112.00 71.99 0.64 NetFirst 1999 122.83 78.62 0.64 Dissertation Abstracts 1999 333.86 209.47 0.63 Peterson's GradSearch 1999 25.58 15.75 0.61 Biography and Genealogical Master Index 1999 73.75 44.28 0.60 MLA International Bibliography 1999 735.25 420.27 0.57 Associations Unlimited 1999 45.08 25.13 0.56 Health Reference Center 1999 548.08 291.12 0.53 Research Centers & Services Directories 1999 6.75 3.25 0.48 World Almanac 1999 25.58 11.89 0.47 PapersFirst 1999 70.58 31.34 0.44 The Measurement of Use of Web-based Information Resources 439 TABLE 1 (CONT) Variance of Monthly Freguencies by ICOLC Elements Resource Timespan Mean Std Dev Coef Var Queries (cont.) GPO 1999 Proceedings First 1999 Medline (through OCLC) 1999 Ovid (health sciences databases)* 1999 ArticleFirst 1999 Contents First 1999 Union Lists 1999 Britannica Online 6/99-5/00 Web of Science 1-6;8-12/99 WorldCat 1999 71.08 32.17 363.58 11,278.65 1,811.25 147.67 43.75 1,431.75 6,049.09 3,886.25 30.20 13.35 133.92 4,118.84 649.47 51.93 13.45 401.19 1,443.60 653.48 0.43 0.42 0.37 0.37 0.35 0.35 0.31 0.28 0.24 0.17 Queries ger Session Cambridge Scientific Abstracts 1999 Historical Abstracts 1999 America: History and Life 1999 Ovid (health sciences databases)* 1999 Web of Science 1-6;8-12/99 2.05 1.32 1.55 4.80 4.50 1.13 0.30 0.26 0.36 0.37 0.55 0.22 0.17 0.08 0.08 Items Research Centers & Services Directories 1999 Historical Abstracts 1999 Peterson's UndergradSearch 1999 America: History and Life 1999 Predicasts PROMT 1999 ABI1INFORM 1999 Gale Literature Databases 1999 Peterson's GradSearch 1999 Wilson Select 1999 Periodical Abstracts 1999 General Business File ASAP 1999 Health Reference Center 1999 Ovid (health sciences databases)* 1999 Associations Unlimited 1999 Contemporary Women's Issues 1999 Biography and Genealogy Master Index 1999 Ideal 1199-11199 Britannica Online 6199-5100 2.25 175.00 16.67 282.00 220.08 1,115.70 59.42 52.42 1,022.50 397.00 856.50 545.00 6,888.12 30.08 140.33 80.92 764.00 1,779.25 3.44 246.92 21.01 334.30 214.61 1,040.27 54.83 40.72 728.24 260.17 549.57 332.87 4,232.02 17.94 79.98 45.48 243.00 427.30 1.53 1.41 1.26 1.19 0.98 0.93 0.92 0.78 0.71 0.66 0.64 0.61 0.61 0.60 0.57 0.56 0.32 0.24 Turnaways Harrison's Online 7/99-5/00 Web of Science 1-6;8-12/99 Ovid (health sciences databases)* 1999 MDConsult 7/99-3/00 95.91 10.55 248.58 0 272.45 16.81 252.72 0 2.84 1.59 1.02 0 * Weekly frequencies 440 College & Research Libraries September 2001 For sessions data, the range of coeffi­ cients of variance among databases was large, 0.21 to 0.78. The average coefficient of variation was 0.54; in other words, the average standard deviation was 54 per­ cent of the mean over the course of the year. Use of health sciences resources such as the Ovid health sciences database col­ lection and MDConsult was below aver­ age in variance and may show greater sta­ bility because the campus clinics and hos­ pital never close and health sciences stu­ dents attend classes throughout the year more so than students in other fields. All of the databases or database collections with a physical or biological science com­ ponent (Beilstein, Web of Science, Applied Science and Technology, Cambridge Sci­ entific Abstracts [among the databases offered by Cambridge Scientific Abstracts, the study library licensed Conference Papers Index, Environmental Sciences & Pollution Management, ERIC, Sociologi­ cal Abstracts, TOXLINE, Environmental RouteNet, and Water Resources RouteNet], and Biological and Agricul­ tural Index) were at or below the average coefficient of variance. Humanities (His­ torical Abstracts, America: History and Life, Humanities Abstracts, Art Abstracts) and social sciences databases (Social Sci­ ences Abstracts, General Business File ASAP, PAIS International, Predicasts PROMT, and Social Work Abstracts) were above average in variance, with the ex­ ception of ERIC. The authors speculate that these groupings reflect literature and database use patterns of disciplines. Sci­ entific disciplines have higher frequencies of publication, resulting in the need for scientists to check databases more often to keep up to date with the literature. Further, William C. Baum et al. offered evidence that the less paradigmatic (i.e., less scientific) a discipline, the longer its publications.10 This also may contribute to scholars in the social sciences—and even more in the humanities—being more episodic in searching databases than physical or biological scientists because it would take the first two longer than the last to read through publications identi­ fied by their searches. Speculation about reasons for variability aside, the amount of variability found in analyzing session patterns demonstrates the need to exam­ ine monthly statistics over the course of a year rather than for one or two selected months or only the annual totals. Queries showed a range of coefficients of variation as well, from 0.17 to 0.89, somewhat greater than for sessions. The average coefficient of variation was 0.54, the same average as sessions. Like data A count of items displayed may be the best measure of the value of a resource. for sessions, scientific resources are below average in variation, whereas social sci­ ences and humanities resources are above average, with the exception of Cambridge Scientific Abstracts which has a science component but showed the highest varia­ tion of all resources in the queries cat­ egory. However, Cambridge Scientific Abstracts’ mix of science and social sci­ ences databases may have contributed to results unlike other resources with only scientific components. As with sessions, the variation of queries is great enough that librarians need to analyze monthly rather than annual data. The items-examined category showed a higher degree of variability over time compared to sessions and queries. The range of the coefficients of variation was 0.24 to 1.53, with a mean of 0.81. Higher variability is understandable because searches may have vastly different results depending on topic; some retrieve hun­ dreds of items and others retrieve only a few. Once again, continual analysis en­ ables librarians to better understand the variation and patterns in the data, rather than relying on an annual summary. The item category as proposed by ICOLC does raise a significant question about its meaning. The ICOLC guidelines describe the category as “examined” items, but without tracking users’ eye movements, a librarian cannot be certain that users actually looked at the items. But http:publications.10 The Measurement of Use of Web-based Information Resources 441 computer-monitoring software can record what items users display (i.e., view, mark, select, download, e-mail, or print). In­ deed, the ICOLC guidelines enumerate these. This argues for renaming the cat­ egory “items displayed.” A count of items displayed may be the best measure of the value of a resource. Although one cannot be certain that searchers read what they display, they likely do read shorter entries such as ci­ tations and at least scan full text. Whether items are citations or full text, they are what a user ultimately seeks. There are occasions, usually at the start of a project, when scholars may want to determine that no one else has worked on their top­ ics. But, generally, users do not seek zero results. They want to find citations or full text that will tell them something about their topics. More than any other ICOLC element, the items element measures this. The number of turnaways demon­ strated the most variability, with a varia­ tion coefficient over 1 for three resources and zero for one resource that had low demand relative to the number of licenses purchased. The number of turnaways depends on the number of simultaneous users licensed. A library could reduce the number of turnaways to a very small number by licensing a large number of simultaneous users, but the variation co­ efficients indicate that use is widely vari­ able and that, for much of the time, licens­ ing a greater number of simultaneous users would be a waste of library money. Examining turnaways is crucial in de­ termining the number of simultaneous users needed for a product. When the number of turnaways is consistently zero, perhaps too many users have been li­ censed. In such cases, the number of si­ multaneous users needs to be examined to estimate demand. In a study at ninety- three academic libraries of the use of da­ tabases supplied by one vendor, Tenopir and Read found that simultaneous use was relatively uncommon: “providing access to only one user for a general re­ search database… would be satisfactory 82.8 percent of the time in research librar­ ies and 95.2 percent of the time in bacca­ laureate colleges”11 The present study did not examine simultaneous use the same way that Tenopir and Read did, who sampled the number of simultaneous us­ ers logged on once an hour, sixteen hours a day, for six months. Instead, the present study relied on daily reports from Web of Science and MDConsult of the maxi­ mum number of simultaneous users who logged on. The data for Web of Science show that simultaneous use occurred daily and increased. In February 1999, use reached the maximum of ten simulta­ neous users on only one day and in Feb­ ruary 2000, on seventeen days. For MDConsult, in the first year of the license, users never exceeded the simultaneous user limit and a broad range of simulta­ neous use occurred. With this informa­ tion, in the second year, the librarians at the present study’s library reduced the number of simultaneous users licensed. The second-year limit was chosen to maximize value while limiting turnaways. These cases illustrate how simulta­ neous user data are crucial in determin­ ing the number of licensed simultaneous users needed to meet most of demand. Therefore, the authors recommend that ICOLC add a sixth element—the maxi­ mum number of simultaneous users per day. As with turnaways, if vendors report only the highest number of simultaneous users in a month, librarians do not know whether the maximum was reached just once, every day, or somewhere in be­ tween. To preclude such ambiguity, ven­ dors should report turnaways and simul­ taneous users on a daily basis. Besides variability within a year dis­ cussed above, there is also variability from year to year. Data from January to August 2000 at the study library demonstrated various differences from the same period in 1999, as summarized in table 2. Beilstein, a database available since 1995, showed a 13.39 percent increase in ses­ sions from 1999 to 2000, whereas Web of Science, a database available since 1999, showed an increase in sessions of 71.80 TABLE 2 Comparison of 1999 and 2000 Use Frequencies by ICOLC Elements Resource VIC 1999-2000 1999 2000 Difference % �ther Subscription Months Data Data Change % Changes egan Studied Sessions Oueries Oueries Ite�s per Session Current Contents 1993 Jan-Aug 3,001 5,482 2,481 82.67 Web of Science 1999 Jan-May, Aug 6,914 11,878 4,964 71.80 61.74 -5.83 America: History and Life 1999 Jan-Aug 853 1,162 309 36.23 63.75 20.21 Cambridge Scientific Abstracts 1999 Jan-Aug 1,283 1,685 402 31.33 230.12 151.36 PAIS International 1999 Jan-Aug 879 1,035 156 17.75 Beilstein 1995 Jan-July 3,338 3,785 447 13.39 Predicasts PROMT 1999 Jan-Aug 385 419 34 8.83 32.53 Business Abstracts 1993 Jan-Aug 1,948 2,095 147 7.55 Bowkers Books in Print 1996 Jan-Aug 1,094 1,122 28 2.56 Social Sciences Abstracts 1993 Jan-Aug 7,437 7,540 103 1.38 Readers Guide Abstracts 1993 Jan-Aug 3,168 3,155 -13 -0.41 Humanities Abstracts 1993 Jan-Aug 3,617 3,542 -75 -2.07 Biological and Agricultural Index 1994 Jan-Aug 1,705 1,601 -104 -6.10 Historical Abstracts 1999 Jan-Aug 679 637 -42 -6.19 35.08 43.98 Library Literature 1994 Jan-Aug 1,058 982 -76 -7.18 General Science Abstracts 1993 Jan-Aug 2,306 2,093 -213 -9.24 PsycINFO 1994 Jan-Aug 22,277 18,688 -3,589 -16.11 General Business File ASAP 1999 Jan-Aug 1,309 1,054 -255 -19.48 -20.80 ERIC 1993 Jan-Aug 8,863 6,830 -2,033 -22.94 Art Abstracts 1995 Jan-Aug 1,704 1,229 -475 -27.88 Applied Science and Technology 1994 Jan-Aug 3,210 2,291 -919 -28.63 442 C ollege & R esearch L ib raries S ep tem b er 2001 TABLE 2 (CONT) Com[arison of 1999 and 2000 Use Freguencies by ICOLC Elements Resource VIC 1999-2000 1999 2000 Difference % �ther Subscription Months Data Data Change %�Changes egan Studied ECO 1998 April-Aug 5,131 20,519 15,388 299.90 Associations Unlimited 1996 Jan-Aug 360 1,267 907 251.94 61.02 Cambridge Scientific Abstracts 1999 Jan-Aug 1,690 5,579 3,889 230.12 31.33 151.36 Biography and Genealogy Master Index n.a. Jan-Aug 629 1,419 790 125.60 96.89 Econlit 1999 Jan-Aug 1,225 2,623 1,398 114.12 MEDLINE n.a. Jan-Aug 2,483 4,530 2,047 82.44 Gale Literary Databases n.a. Jan-Aug 837 1,410 573 68.46 134.83 NetFirst n.a. Jan-Aug 697 1,151 454 65.14 America: History and Life 1999 Jan-Aug 1,316 2,155 839 63.75 36.23 20.21 Web of Science 1999 Jan-May, Aug 32,033 51,809 19,776 61.74 71.8 -5.83 ABIIINFORM 1996 Jan-Aug 5,191 8,047 2,856 55.02 40.39 Historical Abstracts 1999 Jan-Aug 918 1,240 322 35.08 -6.19 43.98 MLA International Bibliography 1995 Jan-June 4,646 6,205 1,559 33.56 WilsonSelect n.a. Jan-Aug 6,141 8,065 1,924 31.33 24.62 Periodical Abstracts n.a. Jan-Aug 2,532 3,159 627 24.76 15.44 Health Reference Center n.a. Jan-Aug 3,399 3,874 475 13.97 7.82 Research Centers & Services Directories 2000 Jan-Aug 52 59 7 13.46 20.83 T h e M easu rem en t o f U se o f W eb -b ased In form ation R esou rces 443 Oueries Sessions Oueries Ite�s per�Session Resource TABLE 2 (CONT) Comparison of 1999 and 2000 Use Frequencies by ICOLC Elements VIC 1999-2000 1999 2000 Difference % Subscription Months Data Data Change egan Studied �ther % Changes Union Lists Contemporary Women's Issues PapersFirst WorldCat ArticleFirst Peterson's GradSearch Britannica Online Peterson's UndergradSearch WorldAlmanac Proceedings First ContentsFirst GPO n.a. 1999 n.a. n.a. n.a. 2000 1996 2000 n.a. n.a. n.a. n.a. Jan-Aug Jan-Aug Jan-Aug Jan-Aug Jan-Aug Jan-July Jan-Aug* Jan-July Jan-Aug Jan-Aug Jan-Aug Jan-Aug 364 908 581 30,379 12,907 169 12,253 184 217 306 1,188 539 Oueries (cont.) 409 45 999 91 626 45 32,282 1903 13,544 637 174 5 10,229 -2,024 142 -42 159 -58 216 -90 800 -388 256 -283 12.36 10.02 7.75 6.26 4.94 2.96 -16.52 -22.83 -26.73 -29.41 -32.66 -52.50 Sessions Oueries per Session Ite�s 10.85 0.27 -32.82 11.28 Cambridge Scientific Abstracts Historical Abstracts America: History and Life Web of Science 1999 1999 1999 1999 Jan-Aug Jan-Aug Jan-Aug Jan-May, Aug 1.32 1.35 1.54 4.63 Queries ger Session 3.31 1.99 1.95 0.6 1.85 0.31 4.36 -0.27 151.36 44.44 20.21 -5.83 Sessions 31.33 -6.19 36.23 71.80 Queries 230.12 35.08 63.75 61.74 ��e�s 444 C ollege & R esearch L ib raries S ep tem b er 2001 TABLE 2 (CONT) Com[arison of 1999 and 2000 Use Freguencies by ICOLC Elements Resource VIC 1999-2000 1999 2000 Difference % �ther Subscription Months Data Data Change % Changes egan Studied Items Sessions �ueries per Session �ueries Gale Literary Databases 2000 Jan-Aug 491 1,153 662 134.83 68.46 Biography & Genealogy Master Index n.a. Jan-Aug 676 1,331 655 96.89 125.60 Associations Unlimited 1996 Jan-Aug 236 380 144 61.02 251.94 ABI/INFORM 1996 Jan-Aug 5,311 7,456 2,145 40.39 55.02 Predicasts PROMT 1999 Jan-Aug 1,162 1,540 378 32.53 8.83 WilsonSelect n.a. Jan-Aug 6,638 8,272 1,634 24.62 31.33 Research Centers & Services Directories 2000 Jan-Aug 24 29 5 20.83 13.46 Periodical Abstracts n.a. Jan-Aug 2,533 2,924 391 15.44 24.76 Peterson's UndergradSearch 2000 Jan-July 133 148 15 11.28 -22.83 Contemporary Women's Issues 1999 Jan-Aug 1,023 1,134 111 10.85 10.02 Health Reference Center n.a. Jan-Aug 3,336 3,597 261 7.82 13.97 Peterson's GradSearch 2000 Jan-July 369 370 1 0.27 2.96 General Business File ASAP 1999 Jan-Aug 5,842 4,627 -1215 -20.80 -19.48 Britannica Online 1996 Jan-Aug* 16,672 11,201 -5,471 -32.82 -16.52 * Britannica Online did not capture remote data for 2/15-5/3/99 T h e M easu rem en t o f U se o f W eb -b ased In form ation R esou rces 445 446 College & Research Libraries September 2001 percent. Perhaps because Web of Science was newer, people had not discovered it or become accustomed to using it in 1999. As table 2 shows, other resources had varied patterns, with some showing a decrease in use and others an increase. As tables 1 and 2 show, the coefficient of variation in 1999 was not a predictor of the amount of change between 1999 and 2000 (January through August for most resources) for sessions, queries, and items. For example, for sessions, both Reader ’s Guide Abstracts and Art Ab­ stracts had coefficients of variation of 0.77 in 1999, but the absolute percentages of change were 0.41 percent and 27.88 per­ cent, respectively, from 1999 to 2000. Beilstein and Web of Science had coeffi­ cients of variation for sessions of 0.24 and 0.27, respectively, but one changed 13.39 percent and the other changed 71.80 per­ cent from 1999 to 2000. Although consistent trends in growth or decline would help librarians in plan­ ning, this is too much to expect in the early years of Web-based resources. Ann Peterson Bishop, in comparing the results from studies at several libraries, found that e-journal systems are not used in their first year of implementation by most of their target audiences, so perhaps use patterns take a long time to become es­ tablished.12 Townley and Murray found that twelve to eighteen months of access are needed before heavy use of a database will be observed. They also found that the number of alternative electronic informa­ tion resources available affects use.13 Even if the resource mix stays constant, perhaps at some point growth in use will cease. As years pass and research on the use of Web-based resources progresses, librar­ ians will learn more about what influ­ ences growth and decline in use. Queries per Session The two ICOLC elements most often re­ ported by vendors in the present study were sessions and queries. These are im­ portant measures in themselves, and also noteworthy is the ratio between them. Townley and Murray studied the use of networked databases at six academic li­ braries. Based on a formula developed by the Texshare consortium from experience with OCLC FirstSearch databases, they estimated that users made three queries per session when the computer-monitor­ ing software did not report queries.14 In the present study, the authors calculated queries per session for three individual databases and two database collections that supplied both elements. In 1999, none of these coincided with three queries per session, but their overall mean was close at 2.84. The three databases had the fol­ lowing ratios: Historical Abstracts, 1.32 queries per session; America: History and Life, 1.55; and Web of Science, 4.50. For the database collections, Ovid health sci­ ences databases had a ratio of 4.80 que­ ries per session and Cambridge Scientific Abstracts, 2.05 queries per session. A month-by-month examination of the data revealed that sessions alone can oc­ casionally provide a misleading indica­ tor of productive database use. Although the average for the year was greater than one, for some months the ratio of queries per session was less than one, indicating that some sessions did not result in use of the resource. For example, in the data for Historical Abstracts and Cambridge Scientific Abstracts, there were months during which users logged in more than they searched, suggesting that they either could not operate the search engine or decided not to conduct a search, or that a librarian was showing a patron how to access the database but nothing else. Despite some months with anomalous ratios, the most stable measure found in this study was queries per session. Al­ though the coefficients of variation for sessions ranged from 0.21 to 0.78, with an average of .54, and those for queries ranged from 0.17 to 0.89, also with an av­ erage of .54, the range for queries per ses­ sion was 0.08 to 0.55 for the five resources for which both sessions and queries were available. The average for the five was 0.22. For both Web of Science and the Ovid health sciences database collection, the coefficient of variation was 0.08. Humani­ http:queries.14 http:tablished.12 The Measurement of Use of Web-based Information Resources 447 ties databases showed slightly higher fig­ ures. America: History and Life had a co­ efficient of variation of 0.17, and Histori­ cal Abstracts had one of 0.22. The data­ base collection Cambridge Scientific Ab­ stracts had a coefficient of variation of 0.55. Except for Cambridge Scientific Ab­ stracts, the coefficients of variation of ra­ tios of queries to sessions were low, in two cases very low. Furthermore, tables 1 and 2 show that the coefficients of variation of queries per session, unlike coefficients of variation for sessions and queries, showed a positive correlation with the percent change between 1999 and 2000. A lower coefficient of variation corre­ sponded with a lower percent change; and as one rose, so did the other. In general, use was highest between 10 a.m. and 5 p.m., peaking some­ time in the afternoon. Finally, because the ratio of queries per session is more stable than the individual ICOLC elements, it has potential for be­ ing an indicator of changes in the way vendors gather data on use or of differ­ ences between resources in the same or similar disciplines. Thus, if a ratio sud­ denly increases, it may mean a vendor has segmented a resource and now, by previ­ ous standards, counts each search twice, as occurred in the present study. Or if one resource’s ratio of queries to sessions dif­ fers greatly from the ratios for similar re­ sources in the same or similar disciplines, the meaning of the data from the outlier may be suspect. Hourly Use Some vendors provide statistical informa­ tion on use by hour of day for selected ICOLC elements. The authors found that vendors may record use by time of day in the library’s local time, the server ’s lo­ cal time, or Greenwich Mean Time and may or may not adjust for daylight sav­ ings time. To compare hourly use patterns and to understand their meaning, librar­ ians must ascertain what time of day a resource’s monitoring software uses and then, if necessary, adjust the data to local time, as was done in the present study. In the present study, four vendors pro­ vided data on use by the hour. Infotrac and ABC-CLIO reported sessions per hour; Britannica Online reported the number of queries and documents per hour; and Ovid’s statistics module allows librarians to extract the number of ses­ sions for any given hour (although this takes so long that it limits data collection). Use by hour across these resources was very similar. In 1999, use of the two Infotrac databases, General Business File ASAP and Predicasts PROMT, was high­ est between 11 a.m. and 6 p.m., peaked between noon and 1 p.m., with evening use tapering off about midnight. For Janu­ ary through August 2000, use of the two ABC-CLIO databases, Historical Ab­ stracts and America: History and Life, was highest between 10 a.m. and 3 p.m., peaked between 1 p.m. and 2 p.m., and tapered off steadily throughout the after­ noon with few uses between midnight and 6 a.m. From June 1999 to May 2000, Britannica Online showed the heaviest use between 9 a.m. and 6 p.m., peaking between 4 p.m. and 5 p.m., with some use at every hour of the day. For Ovid’s health sciences databases, including full-text journals, six days were studied: November 9 and 10, 1999, and February 25, March 30, April 24, and July 11, 2000. These days were chosen because they occurred during a semester and rep­ resented the full spectrum of weekday use, Monday through Friday. After data collection, it was discovered that the data were reported in Greenwich Mean Time; and as use was shifted to local time, some of the use was actually for the prior cal­ endar day, but six twenty-four hour peri­ ods were examined. Use was highest be­ tween 9 a.m. and 5 p.m. and peaked be­ tween 1 p.m. and 2 p.m. There were 136 sessions between midnight and 6 a.m., or an average of 22.7 sessions per day dur­ ing that time period. Infotrac averaged 0.3 sessions per day between midnight and 6 a.m., and ABC-CLIO databases aver­ aged 0.08 sessions. Britannica Online did 448 College & Research Libraries not offer sessions but averaged 2.8 que­ ries per day between midnight and 6 a.m. As seen in figure 1, all the databases studied showed similar use patterns. The hour number on the x-axis represents the entire hour-long period, so that 5 p.m. equates to 5 p.m. to 6 p.m. Differing length of time studied and widely differ­ ent absolute amounts of use resulted in September 2001 differences in scale, but the hourly use patterns are very similar. In general, use was highest between 10 a.m. and 5 p.m., peaking sometime in the afternoon. Tenopir and Read found a similar pattern in their study of the number of simulta­ neous users of a set of online databases in academic libraries—highest use from 10 a.m. to 6 p.m., with a peak for several 0 5 0 10 0 15 0 20 0 25 0 30 0 35 0 12 am 1 am 2 am 3 am 4 am 5 am 6 am 7 am 8 am 9 am 10 am 11 am 12 pm 1 pm 2 pm 3 pm 4 pm 5 pm 6 pm 7 pm 8 pm 9 pm 10 pm 11 pm Se ss io n s � =o = 1 ��� o 0 10 0 20 0 30 0 40 0 50 0 60 0 70 0 80 0 90 0 12 am 1 am 2 am 3 am 4 am 5 am 6 am 7 am 8 am 9 am 10 am 11 am 12 pm 1 pm 2 pm 3 pm 4 pm 5 pm 6 pm 7 pm 8 pm 9 pm 10 pm 11 pm Se ss io n s 0 50 0 10 00 15 00 20 00 25 00 12 a m 1 a m 2 a m 3 a m 4 a m 5 a m 6 a m 7 a m 8 a m 9 a m 10 a m 11 a m 12 p m 1 p m 2 p m 3 p m 4 p m 5 p m 6 p m 7 p m 8 p m 9 p m 10 p m 11 p m Q ue rie s Ite m s 050 1 00 1 50 2 00 2 50 3 00 3 50 12 am 1 am 2 am 3 am 4 am 5 am 6 am 7 am 8 am 9 am 10 am 11 am 12 pm 1 pm 2 pm 3 pm 4 pm 5 pm 6 pm 7 pm 8 pm 9 pm 10 pm 11 pm Se ss io n s 0 § N § N §§ 0§ § § ��� : N 0; - r r - C � � �� §- N � The Measurement of Use of Web-based Information Resources 449 types of academic libraries around 2 p.m.15 Libraries have sought Web-based resources to increase access to informa­ tion outside the library building, and around-the-clock use shows this has hap­ pened, although use remains highest dur­ ing the hours the library is open. How­ ever, determining the percentage of use that occurs in the library and the percent­ age that originates outside the library is currently impossible given the statistics provided. It would be valuable for librar­ ies to work with vendors to obtain such data while preserving the privacy of in­ dividual users. Should statistics docu­ ment increased remote use over time, li­ braries will have reason to provide addi­ tional online help or perhaps a phone help line. Use of E-journal Collections E-journal collections are an important type of Web-based resource. When librarians can select and change the mix of journals they license within a collection, data on the use of individual titles are important. One approach to evaluating such data falls un­ der the rubric of the 80/20 rule. In the management literature in 1954, J. M. Juran discussed the phenomenon by which a small percentage of elements (the vital few) accounted for a large portion of an effect.16 Juran’s Vital Few principle was introduced to the library literature by Richard L. Trueswell in 1969, who demonstrated that often 80 percent of library use is satisfied by 20 percent of materials.17 The so-called 80/20 rule has been tested over the years in journal collections in various libraries. Tina E. Chrzastowski found that 26 per­ cent of the journal collection accounted for 80 percent of use in an academic chemis­ try library.18 Robert J. Veenstra found an almost perfect match with the 80/20 rule in an academic veterinary medical library: 80.1 percent of use was accounted for by 19.8 percent of journal titles held.19 How­ ever, not all studies have fit the 80/20 pat­ tern: the University of Minnesota Biomedi­ cal Library found that 47 percent of titles were needed to satisfy 77 percent of the total use.20 Methods of study of the use of print collections are unable to capture every use.21 Reshelving studies depend on pa­ tron compliance, whereas citation stud­ ies miss current-awareness uses and use for instructional and clinical purposes. Circulation studies miss in-house use. In contrast, with an e-journal collection, it is possible to record every time a journal article is accessed as long as the only route of access is through the software that monitors use of the supplying server. The study team gathered data for four e-journal collections to determine what percentage of the titles accounted for 80 percent of use. Three of the collections were studied from October 1999 to March 2000: American Chemical Society (ACS), Karger, and Project Muse. A fourth, Ovid, was studied from October 1999 to April 2000, with data unavailable for some dates. ACS supplies chemistry journals (no surprise); the Karger collection is com­ posed of health sciences journals; Project Muse is made up of humanities, social sciences, and mathematics journals; the Ovid e-journal collection contains health sciences journals. All four showed a ratio lower than 80/20; that is, more than 20 percent of the collection was needed to satisfy 80 percent of use. The ACS e-jour­ nal collection was the closest, with 28 per­ cent of titles accounting for 80 percent of use. In a collection of health sciences jour­ nals subscribed to through Ovid, the ra­ tio was 80/43. With the Karger collection, 44 percent of titles provided 80 percent of use. With Project Muse titles, the use data were reported in the categories of articles, images, table of contents, other (use that did not fall into one of the three previous categories), and total (all four combined). Mathematics journal articles that con­ tained many graphics were counted as images rather than articles for this data­ base. Each category resulted in a differ­ ent ratio. For articles, 80 percent of use was supplied by 38 percent of titles, im­ ages had a ratio of 80/29, and other had a ratio of 80/52. For the total category, 80 percent of use was supplied by 48 per­ cent of titles. http:library.18 http:materials.17 http:effect.16 450 College & Research Libraries September 2001 Noteworthy, then, are at least two ob­ servations. First, although the pure sci­ ence collection (ACS) is closest to the 80/ 20 distribution, the health sciences collec­ tions from OVID and Karger fit closely with what was obtained at the University of Minnesota Biomedical Library. This suggests that health sciences journal col­ lections have greater scatter of use. Sec­ ond, most of the measures of the use of Project Muse are equally scattered, except for images that largely describe use of mathematics journals. This and the data for use of ACS journals suggest that the 80/20 rule comes closest for paradigmatic science but does not apply elsewhere. When e-journal collections can be pur­ chased on a title-by-title basis, ratios of percentage of use to percentage of titles have collection development implica­ tions. If a library cannot afford to keep all titles, the question becomes, What per­ centage of use does the library want to meet? It may set 90 percent or 80 percent as its target goal. It then can ascertain the least expensive mix of titles that meets its goal and cancel the others. Onsite print collections or document delivery could supply articles from titles canceled in elec­ tronic format. During the present study, the authors found that for several e-jour­ nal collections, title-by-title use data were not available. Because such data are very helpful, libraries should require them when negotiating contracts with vendors. Use in Relation to Disciplinary and Institutional Populations Although the use of several resources in the same discipline can be compared in a somewhat straightforward manner, the size of the disciplinary population de­ serves consideration when comparing resources in different disciplines. To ex­ amine the relationship of the disciplinary population and a resource, the authors compared the use of resources that could be mapped to a particular program, de­ partment, or college to the population of that unit. The clearest indication of the likely disciplinary population for a re­ source is commonality of name of an aca­ demic unit and the resource (e.g., PsychINFO mapped to the psychology department, Social Work Abstracts to the school of social work). Per capita use, as discussed here, equals the frequency of an ICOLC element reported for a resource divided by the population of its corre­ sponding academic unit. Population equals full-time equivalency (FTE) faculty (including graduate assistants) plus FTE instructional enrollments—that is, the number of students enrolled in classes of a unit, calculated on the basis of course enrollment credit hours (fifteen per un­ dergraduate FTE, twelve per graduate).22 Table 3 lists the resources, use data for ICOLC elements, the academic units that correspond to the resources, the popula­ tion of those units in the fall of 1999, and the per capita use for the resource. It is possible to look at the data in table 3 in a variety of ways, but perhaps the most interesting result is which databases show high use per capita. Unquestionably, PsycINFO, ERIC, and Contemporary Women’s Issues are the most heavily used per capita. The absolute frequencies of use of PsycINFO and ERIC are also the high­ est, so obviously these are extraordinarily important sources. But Contemporary Women’s Issues has low absolute use, such that its cost may be questioned until one sees how high the use is per capita for such a small program. Of course, high use per capita also may be an indicator that a da­ tabase is of value to other disciplines be­ sides the primary population: in other words, use per capita may be an index of the multidisciplinarity of a database. For example, in addition to people from the psychology department, PsychINFO users may include people in medicine, nursing, education, social work, and public health. In contrast, business resources show lower use per capita, which may be attributed in part to a narrow focus that is not relevant to students and faculty in other fields. Moreover, relatively low per capita use for business resources may reflect that there are several of them to meet the demands of users, whereas PsychINFO is the only database licensed in psychology. http:graduate).22 The Measurement of Use of Web-based Information Resources 451 TABLE 3 Ratio of Use to Primary User Po(ulation, 1999 Database Totals Dept/College Population Per capita PsyclNFO ERlC Beilstein America: History & Life Social Work Abstracts* Historical Abstracts Business Abstracts General Business File ASAP Predicasts PROMT Contemporary Women's Issues Econlit ABI/INFORM America: History & Life MLA International Bibliography Historical Abstracts Contemporary Women's Issues ABI/INFORM General Business File ASAP Predicasts PROMT * 8/1999-7/2000 Sessions 36,971 14,017 5,971 1,640 1,358 1,131 3,568 2,494 748 Queries 1,344 2,961 12,417 2,458 8,823 1,428 Items 1,684 13,388 10,278 2,641 Calculations of per capita use provide an index of value. In a library where a specific subject fund may pay for a re­ source identified with that subject, high per capita use may suggest that the re­ source is supporting multiple subject ar­ eas and the subject fund deserves com­ pensation. If per capita use suggests that a database is multidisciplinary, it also may deserve greater consideration in library instruction than other databases, espe­ cially in discussions of relevant resources. Finally, the authors suggest that per capita use not by academic unit but, in­ stead, by educational institution should be considered for national statistical re- Psychology Education Chemistry History Social Work History Business Business Business Women's Studies Economics Business History English & Languages History Women's Studies Business Business Business 909.65 807.62 1,227.99 635.88 552.47 635.88 2,560.14 2,560.14 2,560.14 34.66 578.70 2,560.14 635.88 2,294.09 635.88 34.66 2,560.14 2,560.14 2,560.14 Sessions per Capita 40.64 17.36 4.86 2.58 2.46 1.79 1.39 0.97 0.29 Queries ler �alita 38.78 5.12 4.85 3.86 3.85 2.25 Items ler Calita 48.59 5.23 4.02 1.03 porting. To compare raw use data across libraries has limited meaning because of differences in institutional population sizes. It would be more telling to report use per capita, which normalizes the data. Although, ideally, all ICOLC elements should be normalized, in these early years of Web-based resource use, sessions are probably the easiest statistic for all ven­ dors to capture. As more vendors become ICOLC compliant, queries, items, menu selections, and turnaways should be added to sessions. It could be argued that in national reporting, a library should include statistics on the use of free, unli­ censed resources such as the National Li­ http:2,560.14 http:2,560.14 http:2,560.14 http:2,294.09 http:2,560.14 http:2,560.14 http:2,560.14 http:2,560.14 http:1,227.99 452 College & Research Libraries September 2001 brary of Medicine’s PubMed. The argu­ ment would be that when a library orga­ nizes and indexes its Web site to facilitate access and makes networked computers available to those who lack them, it should assess use of free resources reached through its site and equipment to have a full picture of the effectiveness of its efforts. Nevertheless, free resources normally do not offer statistics on use by a particular institution. A library can approximate the number of sessions for a resource by mea­ suring how often the Web gateway to a resource is accessed. But this would yield only a weak estimate of sessions because there is no way to be sure that a user logged onto the resource after looking at it. Thus, the best national measures are vendor-sup­ plied, ICOLC-compliant statistics, includ­ ing sessions, queries, items, menu selec­ tions, and turnaways, normalized by in­ stitutional population. Conclusions Electronic resource use exhibited a great deal of variance over time. This suggests that, in general, monthly data analyzed continually are necessary for an accurate picture of the scope of use. One or two selected months of statistics will not pro­ vide a true picture of use. Indeed, this study found that in-depth understanding is aided by some analysis of daily data, especially of turnaways and simultaneous use. Extreme highs on a few days can cre­ ate an appearance of heavy use in monthly or yearly summaries. A handful of extreme lows can be equally mislead­ ing. But an examination of a greater num­ ber of data points prevents misapprehen­ sions. Furthermore, over time, use of a resource is likely to change. Comparison of data from 1999 and 2000 suggests that changes do occur in varying degrees over time, perhaps influenced by the length of time a resource has been available and the ever-changing resource mix. It will take a careful analysis of statistics of use over many years and at many different librar­ ies before it is possible to make meaning­ ful generalizations about change in use of Web-based resources over time. As a result of this study, the authors make the following recommendations: For vendors: • Supply all relevant ICOLC ele­ ments. • Supply documentation explaining how the data are counted and reported, and notify librarians if there is a change in data reporting and the date the change goes into effect. • Report the maximum number of simultaneous users per day. • Indicate in summary data for a given period how many days are miss­ ing data. • For database or e-journal collec­ tions, supply use data for each title in the collection. For libraries: • Review the way vendors count the elements reported annually so that changes in definition may be noted and considered during interpretation. • Examine data at least monthly and occasionally daily to have a true under­ standing of variations in use. • Ascertain changes in use patterns from year to year. • Calculate queries per session for insight into the level of use and to moni­ tor stability of data. • Be aware that data may not be re­ ported in local time but may need to be converted from some other standard. • Examine use of individual titles in e-journal collections. • Evaluate use in terms of the pri­ mary user population of a Web-based re­ source for additional interpretation of value. For comparison among libraries, ini­ tially report the number of sessions per institutional population (students, fac­ ulty) for all Web-based resources. As more vendors become ICOLC compliant, que­ ries, items, menu selections, and turnaways should be added to per popu­ lation comparisons. Sound interpretation of data about use of Web-based resources is crucial to good library management. The data and issues surrounding them are new and complex, The Measurement of Use of Web-based Information Resources 453 and they require continued discussion in further investigation and look forward to the library literature. The authors of the the reports and reflections of their col- present study hope that it will stimulate leagues. Notes 1. International Coalition of Library Consortia, “Guidelines for Statistical Measures of Us­ age of Web-based Indexed, Abstracted, and Full-text Resources,” Nov. 1998 [Online]. Available from http://www.library.yale.edu/consortia/webstats.html. 2. Tom Peters, “Computerized Monitoring of Human–Computer Interaction,” July 8, 2000. [Online] Available from http://pallus.cic.uiuc.edu/cicLibraries/MtgNotes/LRRT.htm. 3. Judy Luther, White Paper on Electronic Journal Usage Statistics (Washington, D.C.: Council on Library and Information Resources, Oct. 2000) [Online]. Available from http://www.clir.org/ pubs/reports/pub94/pub94.pdf. 4. Carol Tenopir and Eleanor Read, “Patterns of Database Use in Academic Libraries,” Col­ lege & Research Libraries 61 (May 2000): 234–46. 5. Carol Tenopir and Danielle Green, “Patterns of Use and Usage Factors for Online Data­ bases in Academic and Public Libraries,” in Proceedings of the 62nd Annual Meeting of the American Society for Information Science, Washington, D.C., October 31– November 4, 1999 (Medford, N.J.: Information Today, 1999): 616–27. 6. Carol Tenopir, “Factors That Influence Database Use,” in Proceedings of the Ninth National Conference of the Association of College and Research Libraries, Detroit, April 8–11, 1999 (Chicago: ALA, 1999): 68–75. 7. Carol Tenopir and Danielle Green, “Simultaneous Usage of Online Databases in Academic and Public Libraries,” in Proceedings of the 20th Annual National Online Meeting, New York, May 18– 20, 1999 (Medford, N.J.: Information Today, 1999): 459–68. 8. Charles T. Townley and Leigh Murray, “Use-based Criteria for Selecting and Retaining Electronic Information: A Case Study,” Information Technology and Libraries 18 (Mar. 1999): 32–39. 9. International Coalition of Library Consortia, “Guidelines for Statistical Measures of Us­ age,” 1–2. 10. William C. Baum, G. N. Griffiths, Robert Mathews, and Daniel Scherruble, “American Political Science before the Mirror: What Our Journals Reveal about the Profession,” Journal of Politics 38 (Nov. 1976): 895–917. 11. Tenopir and Read, “Patterns of Database Use in Academic Libraries,” 239. 12. Ann Peterson Bishop, “Logins and Bailouts: Measuring Access, Use, and Success in Digi­ tal Libraries,” Journal of Electronic Publishing 4 (Dec. 1998) [Online]. Available from http:// www.press.umich.edu/jep/04-02/bishop.html. 13. Townley and Murray, “Use-based Criteria for Selecting and Retaining Electronic Informa­ tion,” 34, 38. 14. Ibid., 33. 15. Tenopir and Read, “Patterns of Database Use in Academic Libraries,” 238. 16. J. M. Juran, “Universals in Management Planning and Controlling,” Management Review 43 (Nov. 1954): 748–61. 17. Richard L. Trueswell, “Some Behavioral Patterns of Library Users: The 80/20 Rule,” Wil­ son Library Bulletin 43 (Jan. 1969): 458–61. 18. Tina E. Chrzastowski, “Journal Collection Cost-Effectiveness in an Academic Chemistry Library: Results of a Cost/Use Survey at the University of Illinois at Urbana-Champaign,” Col­ lection Management 14 (1991): 85–98. 19. Robert J. Veenstra, “A One-Year Journal Use Study in a Veterinary Medical Library,” Jour­ nal of the American Veterinary Medical Association 190 (Mar. 15, 1987): 623–26. 20. Pamela Tibbetts, “A Method for Estimating the In-house Use of the Periodical Collection in the University of Minnesota Biomedical Library,” Bulletin of the Medical Library Association 62 (Jan. 1974): 37–48. 21. Deborah D. Blecic, “Methods of Measurement of Journal Use,” in Encyclopedia of Library and Information Science, Suppl. In press. 22. Tom Ascher and Henry C. Young, UIC Resource Performance Measures: Annual Extract, Aca­ demic Units, FY1997–FY2000 (Chicago: Data Resources and Institutional Analysis, Univ. of Illi­ nois at Chicago, 1999). www.press.umich.edu/jep/04-02/bishop.html http:http://www.clir.org http://pallus.cic.uiuc.edu/cicLibraries/MtgNotes/LRRT.htm http://www.library.yale.edu/consortia/webstats.html