mcgillis.p65 Usability of the Academic Library Web Site 355 Usability of the Academic Library Web Site: Implications for Design Louise McGillis and Elaine G. Toms Today’s savvy library users are starting to equate the library Web site with the physical library. As they accomplish, virtually, many personal activities such as online shopping, banking, and news reading, they trans­ fer those experiences to other activities in their lives. This increases their expectations about the functionality of a library Web site and how one interacts with it. The purpose of this study was twofold: to assess the usability of an academic library Web site and to better understand how faculty and students complete typical tasks using one. Thirty-three typical users successfully completed 75 percent of a set of typical tasks in about two minutes per task and were satisfied with the clarity and organization of the site. Despite their success in completing the tasks, however, they experienced difficulties in knowing where to start and with the site’s information architecture—in particular, with interpreting the categories and their labels. The authors concluded that library Web sites fail to take into account how people approach the information problem and often reflect traditional library structures. library Web site is its virtual public face—the quasi equiva­ lent of the front door, signage, pathfinders, collections or sur­ rogates to the collections, services, and, to an extent, its people. Because library Web sites compete with a host of other Internet services and organizations for clientele, they must capture and motivate the user; provide useful, innovative, and interactive services and products; give the library some form of identity; and encourage re­ peat visitation while holding the interest of users and simplifying their tasks. They also must do so while supporting the mis­ sion of the library, which for an academic library typically is “to foster the search for knowledge and understanding in the Uni­ versity and the wider community.”1 This is a complex and complicated application for a Web site to support. Like consumer of e-commerce sites, li­ brary Web site users must be able to effec­ tively and efficiently access, use, and in­ teract with a library’s products and services and be personally satisfied with the expe­ rience. To date, a burgeoning literature, some formally published and distributed Louise McGillis is the Public Services Librarian in the Ferriss Hodgett Library at Sir Wilfred Grenfell College; e-mail: mcgillis@swgc.mun.ca. Elaine G. Toms is an Associate Professor for Faculty of Informa­ tion Studies at the University of Toronto; e-mail: toms@fis.utoronto.ca. This study was conducted at the School of Library and Information Studies, Dalhousie University, when McGillis was a researcher-in­ residence and Toms was on faculty as Associate Professor. The study was funded by a research grant from Memorial University of Newfoundland to McGillis. The authors wish to thank Elizabeth Behrens, Stephen Field, Dianne Taylor-Harding, and Shelagh Wotherspoon for conducting the study in the four remote locations, as well as research assistant Megan Butcher. 355 mailto:toms@fis.utoronto.ca mailto:mcgillis@swgc.mun.ca 356 College & Research Libraries July 2001 on the Web, has addressed Web site design and guidelines for common Web site ele­ ments such as page-loading speed, fonts, layout, color, graphics, animation, and in­ formation presentation can be extrapo­ lated.2–6 But these works do not address the unique requirements of specific applica­ tions, such as purchasing, making a reser­ vation, and requesting an interlibrary loan. A library Web site is (or will be) a complex application integrating access to and inter­ action with a diverse set of information products and services and with people. In this study, the authors accessed the usabil­ ity of an academic library Web site and examined how faculty and students com­ pleted typical library tasks. Related Work To date, there are few published assess­ ments of library Web sites. Christy Hightower, Julie Shih, and Adam Tilghman classified academic library Web pages as (1) home page: the initial page; (2) directional: hours, policies; (3) reference: collection guide, resource guides; and (4) combination.7 None of these categories embraces the wide range of services typi­ cally found in a physical library nor the range of innovative services made possible by technology, which may be indicative of the development of Web pages at the time of authorship. Mark Stover and Steven D. Zink undertook a feature-counting com­ parative exercise of forty academic library Web sites, and David L. King compared the formatting features of one hundred twenty ARL Web sites.8, 9 Laura B. Cohen and Julie M. Still examined the structure and purpose of one hundred academic li­ brary Web sites, noting that library Web sites serve four purposes: information, ref­ erence, research, and instruction.10 This result can be equated with the factors in a library’s mission statement. John D’Angelo and Sherry K. Little examined twenty Web sites to assess adherence to Web site de­ sign guidelines.11 All of these examinations of library Web sites have a developer or administrative focus and do not address the essential question: Can users accom­ plish tasks to meet their goals? To respond to this question, one must understand how usable a Web site is. In­ terestingly, usability, defined as “the extent to which a product can be used by speci­ fied users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use,” is a relatively new concept to libraries.12 Web site usabil­ ity has been a much-discussed topic over the past few years as developers place more attention on a user ’s ability to successfully negotiate a Web site rather than solely on the technology of site-building. Usability testing is a method or battery of tests that uses typical users of a system to evaluate “the degree to which a product meets specific usability criteria.”13 In usabil­ ity testing one measures three aspects: “1. Effectiveness: accuracy and completeness with which users achieve specified goals; 2. Efficiency: resources expended in rela­ tion to the accuracy and completeness with which users achieve goals; 3. Satisfaction: freedom from discomfort, and positive at­ titudes towards the use of the product” based on typical users executing typical tasks.14 Janet Chisman, Karen Diller, and Sharon Walbridge, who assessed an online public access catalog (OPAC) and elec­ tronic indexes interfaces, is one of the few usability studies conducted in a library; however, it does not assess the library Web site.15 Jerilyn Veldof, Michael J. Prasse, and Victoria A. Mills recently discussed a us­ ability study of the University of Arizona Library’s Web site, but it is impossible to assess the validity of that study because of insufficient detail in the published work (e.g., number of participants, data analy­ sis, and so on).16 In addition, several orga­ nizations have placed on the Web descrip­ tions of Web site usability testing.17, 18 Library Web Site Used for Testing The Memorial University of Newfoundland (MUN) Libraries Web site (www.mun.ca/ library) was chosen for this study because it is rich in content with good navigation and access, and reflects maturity in Web site development. The home page is divided into two key menu groups: Resources and Services & Contacts, as illustrated in figure http:www.mun.ca http:testing.17 http:tasks.14 http:libraries.12 http:guidelines.11 http:instruction.10 Usability of the Academic Library Web Site 357 FIGURE 1 Memorial University Libraries Homepage 1. Each group is subdivided into five main categories, with each category containing a brief annotation about the content of that category. Access to a search engine is pro­ vided via the phrase “Site Search” on the banner at the top of the page. On subsequent pages (see figure 2 for an example), the ban­ ner becomes a menu bar of navigational fea­ tures including an option called “MUN Li­ braries” for returning to the home page. The site was developed and is maintained by a Library Web Team. During the course of its development, user evaluation was received via an e-mail address on the home page that explicitly requested comments, questions, and concerns. Based on this feedback, the site was redesigned, information reorga­ nized, explanation of links on the main page inserted, and search aids, including a navi­ gation bar and a search engine, added. Methodology Participaoto Thirty-three individuals, sixteen females and seventeen males, participated in the study. There were seventeen undergradu­ ate students, two graduate students, and fourteen faculty members. Ages ranged from under twenty to over fifty-five, with 82 percent of the students under twenty- five and 81 percent of faculty over thirty (62% over thirty-five). Most participants were experienced computer and Web us­ ers: 97 percent used computers on a daily basis, and 97 percent used the Web at least once a week. Most participants were regu­ lar library users: 84 percent responded that they used the library at least once a week. However, only 42 percent used the library Web site once a week or more. Participants were recruited from each of the MUN libraries: six participants from the Queen Elizabeth II Library, six from the Health Sciences Library, seven from the Ferriss Hodgett Library, and eight from the Marine Institute Library. In addition, a group was recruited from off campus: six participants from the University of King’s College in Halifax. Notices were posted and individuals volunteered. Participants 358 College & Research Libraries July 2001 FIGURE 2 Memorial University Libraries Secondary Page Example who were randomly selected from the vol­ unteers were given $10.00 for their partici­ pation. Tasks Participants completed three tasks: 1. They completed a background/expe­ rience questionnaire that included both demographic information and questions about computer, library, and library Web site use. 2. They performed six tasks using the library Web site. (Tasks were developed by the MUN Library Web Team from the ba­ sic functions of a library Web site.) The tasks included: • finding a book; • finding a journal article; • finding an Internet resource; • finding help with a database; • finding out how to renew books: • asking a reference question via the Web site. 3. They completed a questionnaire that elicited perceptions of using the site and its features. The test contained seven Likert-scaled, one yes/no question and three open-ended questions. Materials Data were collected in print and by data logging. WinWhatWhere Investigator, a software program that logs link selection and time, was used to record each participant’s movement on the Web site and to record time. Test Administrators Test administrators were recruited to con­ duct the test at each of the five sites. As a result, steps were taken to ensure consis­ tency in the conduct of the test and the data collection at all sites. All test administra­ tors were given prepared test materials, including a script for administrating the test, a consent form, a letter of information for each participant, and a data sheet for recording observations. In addition, the primary investigator conducted a telecon­ ference meeting of the administrators and liaised with the administrators throughout the test period. Each local test administra­ Usability of the Academic Library Web Site 359 tor recruited participants, set up the test room, and conducted the test. Procedures Prior to the study, all procedures and test instruments were pretested using gradu­ ate student and faculty volunteers. The study was conducted in February and March 2000 at the five sites. Participants accessed the library Web site using Netscape 4.5 (or higher) and Internet Ex­ plorer 4.01 (or higher), depending on in­ dividual preference. Rather than artificially choosing one browser for test consistency purposes, the researchers allowed browser choice so that participants could work in a familiar environment; the only direct in­ teraction between participant and browser was in selecting hypertext links and using the browser’s ‘back’ button. All the partici­ pants used comparable computers with a fifteen-inch monitor set to 640 x 480 reso­ lution. The only people present in the room during the test were the participant and the test administrator. Participants first read the letter of infor­ mation that explained the purpose of the study and signed a consent form. To start, each participant completed the back­ ground/experience survey. Next, partici­ pants completed six tasks using the library Web site. The order of assignment of the six tasks was randomized to control for order and learning effects. After complet­ ing each task or after five minutes had elapsed, the participant responded in writ­ ing to a set of follow-up questions about the task just completed. Each task was re­ corded on a single sheet. Participants were not permitted to examine the next task until the previous one was completed. As the participant worked on the tasks and fol­ low-up questions, the test administrator sat within viewing distance, keeping track of time and recording user behavior and com­ ments. Little or no interaction took place between participant and administrator, except that which was essential to the con­ duct of the test. After completing the six tasks, the par­ ticipant responded to a short questionnaire giving his or her perceptions regarding the usability of the site. This was followed by a debriefing period in which the test ad­ ministrator asked for clarification or fur­ ther explanation on some of the approaches taken, as well as some general feedback about the site. The participant also was in­ vited to comment on any aspect of the site or his or her experience in using the site. Results Several types of data were collected to as­ sess user performance and user percep­ tions of negotiating the Web site, as follows: • Effectiveness was measured by the number of tasks successfully completed. • Efficiency was measured by the amount of time taken to complete each task. • Satisfaction was measured by a rat­ ing scale for several satisfaction elements. The researchers compared usability by faculty and student. In addition, they ex­ amined selected features of the Web site to determine their effectiveness. Tasks were deemed to be either com­ pleted or not completed. Because the li­ brary Web site acts as an intermediary be­ tween the user and a host of unique re­ sources and services, the researchers dif­ ferentiated between the ability to use the site and the ability to use a particular re­ source or service (each of which could and should be tested separately). The research­ ers did not want to confuse, for example, difficulties in searching the catalog with difficulties in negotiating the Web site. Thus, a task was considered “completed” when the participant found and accessed the correct resource or service. In essence, a participant who could find the catalog but did not know how to search it was still considered to have completed the task for the purposes of this study. Faculty were defined as those individu­ als who specified faculty status (n = 14) or graduate student status (n = 2). Students (n = 17) were undergraduate students who, on average, had 2.7 years in university. Although there are no significant differ­ ences between faculty and students in their use of the library Web site (F(1,29) = .718, p = ns), faculty and students do tend to dif­ 360 College & Research Libraries fer in their use of the library (F(1,31) = 4.6, p = .039). To do this latter check, the researchers cre­ ated aggregate measures for li­ brary and library Web site experi­ ence by summing the scores for each set of questions included on the background/experience ques­ tionnaire that pertained to each type of experience. Overall User Performance: Effectiveness and Efficiency Both faculty and students completed the same number (4.5) of tasks (F(1,31) = .004, p = ns). Although they took slightly differ­ ent amounts of time to complete each task, the differences were not statistically signifi­ cant (F(1,31) = .459, p = ns), nor was there an interaction among these variables (F(2,30) = .228, p = ns). Thus, neither uni­ versity experience nor the amount of li­ brary experience affected performance. The experience factor was dropped from further analysis. User performance by uni­ versity experience is shown in table 1. User Performance with Individual Tasks The performance for individual tasks is summarized in table 2. In addition to as­ sessing user performance, which appears in the second and third columns, the re­ searchers also sought user perspectives about the task completion, some of which are included in table 2. Overall, 67 to 85 percent of the participants completed a task July 2001 TABLE 1 User Performance by University Experience Number of Tasks Average Time Completed (in seconds) Taken (out of 6 tasks) to Complete Tasks Faculty 4.5 114.9 Students 4.5 124.9 Total 4.5 120.0 and needed from one and a half to two and a half minutes to do so. Task 1. Finding a Book To complete the first task, participants had to select “Unicorn,” which is the label as­ signed to the library catalog from the Re­ sources category (figure 1). Approximately 85 percent of participants found the answer in about two minutes. The site uses sev­ eral names for the library catalog, includ­ ing “catalogue,” “Unicorn,” and “Webcat.” About one-third of the participants indi­ cated that they did not know what “Webcat” meant. Participants commented that they had difficulties figuring out where to begin and deciding which option to select for the library catalog. Task 2. Finding a Journal Article To complete the second task, participants had to select “Databases” from the Re­ sources category and, subsequently, a spe­ cific discipline and then a database from the lists presented. Approximately 67 per- TABLE 2 Results of Task Completion Tasks Number Average % Who Did % Found it Completed Time in Seconds Task before Easy 1. Find library catalog 28 (84.8%) 113.5 24.2% 84.8% 2. Find library databases 22 (66.7%) 147.0 18.2% 69.7% 3. Find a reference source 24 (72.7%) 166.1 9.1% 66.7% 4. Ask a reference librarian remotely 25 (75.8%) 104.6 0% 75.8% 5. Use the help resources 23 (69.7%) 101.3 0% 75.8% 6. Use a library service- book renewals 26 (78.8%) 97.3 27.3% 72.7% Usability of the Academic Library Web Site 361 cent of the participants found a database. They had difficulties in choosing from the categories on the home page and, subse­ quently, in figuring out which database to select. Some participants also did not un­ derstand the concept of a “journal article.“ In summary, despite the completion rate, on average, only one-quarter of the participants per task said there was nothing difficult about complet­ ing the tasks. Task 3. Finding an Internet Resource To complete the third task, finding an Internet resource, participants had to se­ lect “Internet Resources” in the Resources category and then “Internet Reference Sites” from the list presented. Approxi­ mately 73 percent of the participants com­ pleted this task but took, on average, longer to complete it than the other five tasks. Similar to the previous tasks, participants had trouble selecting from the main and secondary levels. In particular, they could not differentiate easily between “Internet Resources” and “Databases” on the main page. On the “Internet Resources” page, users had trouble selecting from among “Internet Resources by Subject,” “Internet Search Engines,” “Electronic Books,” and “Internet Reference Sites.” Task 4. Asking a Reference Question via the Web Site To complete task 4, participants could se­ lect either “Research Help” or “Request Forms” from the Services & Contacts cat­ egory. Approximately 76 percent of the par­ ticipants completed this task. As with the previous tasks, they commented on differ­ entiating between “Research Help” and “Request Forms” on the home page. One participant said that “’Request Forms’ is vague and cryptic for a service option.” Task 5. Finding Help with a Database To complete the fifth task, participants could select “Research Help” under Ser­ vices & Contacts or “Databases” under Resources. The most direct route would be to select, first, “Research Help” and, then, “Guides to Databases.” If the participant selected “Databases,” he or she would then have to select “Education Databases” and scroll down to the ERIC database to find the guide. Approximately 70 percent of the participants found the guide. They expe­ rienced difficulties with initial category selection from the home page. Moreover, some were unfamiliar with ERIC and oth­ ers did not see the link to the guide on the “Database” page, whereas others searched the library catalog for a guide. Task 6. Finding Out How to Renew Books To complete task 6, participants had sev­ eral options: (1) select “Request Forms” under Services & Contacts and then select a MUN library; (2) select “Research Help,” also under Services & Contacts, and then select “Do-It-Yourself in Unicorn,” or (3) select “Unicorn” under Resources and then select “User Services.” Approximately 79 percent of the participants completed this task. When difficulties were experienced, they tended to be with category selection: which category contained the information they needed? In addition to the three routes mentioned, participants also tried “Poli­ cies.” When asked what was the most dif­ ficult part in completing this task, partici­ pants responded “once again, knowing where to look, I wasn’t sure as to where it would be” and “It was not immediately clear from the home page which way to go.” A few participants also noted that it was not obvious that renewals fell under “Do-It-Yourself in Unicorn.” In summary, despite the completion rate, on average, only one-quarter of the partici­ pants per task said there was nothing diffi­ cult about completing the tasks. For the re­ mainder, the key challenge was selecting a menu choice from the home page. The ter­ minology was not meaningful to partici­ pants. When asked what was the hardest part about completing the task, participants answered: “not knowing what heading to look under,” “I wasn’t sure where to start,” and “figuring out what the categories meant.” They also commented on what helped to complete the task: “the fact that certain items are listed under the main head­ 362 College & Research Libraries July 2001 TABLE 3 User Perceptions ings on the home page,” “the fact that reference questions were listed specifically under request forms.” Some participants sug­ gested it would be Low Neutral High No Response Satisfying 3 (9.1%) 4 (12.1%) 24 (72.7%) 2 (6.1%) Easy 3 (9.1%) 4 (12.1%) 26 (78.8%) o Clear 2 (6.1%) 7 (21.1%) 22 (66.7%) 2 (6.1%) more helpful if each category’s annotation contained all subsequent choices. User Satisfaction Data on user satisfaction were collected in several ways: response to a series of ques­ tions following each task and to the posttask questionnaire. Participants responded to three five- point Likert-scaled questions about their overall general impressions of the Web site. The data were aggregated as shown in table 3: low = 1, 2; neutral=3; and high = 4,5. Overall, 72.7 percent of the participants found using the site a highly satisfying ex­ perience, 78.8 percent found it easy to use, and 66.7 percent found it clear. In addition, the researchers assessed the differences among each of these measures by “success” to determine whether those who achieved high success tended to dif­ fer on the satisfaction measures. Success was defined as completion of two-thirds of the tasks. Thus, participants who com­ pleted five or six tasks were deemed highly successfully whereas those who completed four or fewer were deemed less success­ ful. Seventeen of the thirty-one participants were considered to be highly successful. The level of satisfaction (F(1.31) = 1.266, p = ns) and ease of use (F(1.31) = 1.911, p = ns) did not differ from group to group, but the least successful participants tended to find the Web site more confusing than those who met with high success (F(1.31) = 6.958, p = .013). Perceptions of the Effectiveness of Selected Web Site Features In addition to assessing overall user per­ formance and user perceptions, the re­ searchers assessed selected Web site fea­ tures. At the end of the session, participants responded to a series of seven-point Likert­ scaled questions about selected features of the Web site. The results are shown in table 4. More than 90 percent of the participants found the site to be well organized. They found the wording used to express site cat­ egories clear (90.9%), the annotations of those categories clear (75.8%), and the site relatively library-jargon free (78.8%). Only 33.3 percent found the search feature use­ ful, and 60.6 percent found the navigation bar useful. These values may be lower than the previous features because approxi­ mately 30 and 40 percent, respectively, did not use the search engine and navigation bar. However, administrators reported minimum use of these features in their field notes. Thus, in the participants’ own ad­ mission, the site is suitably organized and labeled, but the special features intended to enhance access are not being used or not being rated very highly by most partici­ pants. Analysis and Discussion On average, participants completed 75 per­ cent of the tasks, taking about two min­ utes per task, and, in general, were satis­ fied with the site’s clarity and organization and their ability to do the tasks. Surpris­ ingly, faculty and students completed the same number of tasks in about the same amount of time. The experience of faculty did not, in fact, influence their ability to interact with the Web site. Like most information products, using a Web site involves using a set of tools, in this case, a series of menus, access tools, and navigational aids. Results from the study show that the performance of par­ ticipants was affected by their ability to use these tools. In particular, they had difficul­ ties interpreting menu labels, understand­ Usability of the Academic Library Web Site 363 TABLE 4 User Assessment of Selected Web Site Features Feature Agreement with Statement No Neutral Yes Did Not Use Site is well organized. Search feature was helpful. Navigation bar was helpful. Terms in links are clear. Explanations to describe links are clear. Too much library jargon. 1 (3.0%) 7 (21.2%) 3 (9.1%) 3 (9.1%) 8 (24.2%) 26 (78.8%) 1 (3.0%) 2 (6.1%) 0 0 0 1 (3.0%) 31 (93.4%) 11 (33.3%) 20 (60.6%) 30 (90.9%) 25 (75.8%) 6 (18.2%) 0 13 (39.4%) 10 (30.3%) 0 0 0 ing the content of categories, and, finally, knowing where to start. These components of the interface provide functionality for a library Web site. Menus, Categories, and Labels Home pages typically start with a menu or series of menus that enable users to scan menu options until they recognize what they need. Conventions for these menus range from the minimalist approach with few selections on the home page to the in­ formation maxima with multiple groupings of multiple items. Menu research conducted mostly with early videotex systems has in­ formed menu design, although the results of that research are not always applied to menu design for the Web sites.19, 20 Recently, Kevin Larson and Mary Czerwinski added to that work, concluding that a medium condition of depth and breadth best suited Web architectures.21 Participants performed best with— and preferred—menu struc­ tures with sixteen and thirty-two items on the first and second levels, respectively, over a menu structure with eight items on three levels or one with thirty-two items on the first level and sixteen on the second. It seems that a menu structure that is not too deep or too overwhelming at the outset is recom­ mended. The menu style of the MUN home page partially fits this structure. This re­ duces the amount of effort that has to be expended; users are able to reach informa­ tion content with a minimum of two, and maximum of four, mouse clicks. Participants had trouble choosing from the list of menu options and differentiat­ ing from among possible choices. The ter­ minology used in the set of menus was not meaningful despite the fact that it is stan­ dard in libraries. Participants did “not [know] what heading to look under” and could not “figure out what the categories meant.” The classification of information and the labeling of resulting categories are perennial problems in information system design; Web sites are no exception. Shelley Gullikson and colleagues found that users of an academic Web site could not distin­ guish between the terms Academic and Administrative and interpreted Calendar dif­ ferently depending on context.22 Michael D. Corry, Theodore W. Frick, and Lisa Hansen found the formal name assigned to the university housing division, Halls of Residence, was confusing to users and that Housing was more clearly understood. Many usability studies conducted in dif­ ferent venues have similarly concluded that categories and their respective labels are unclear. In this study, participants had difficulties, for example, with “Resources” and “Databases” as well as with “Webcat” and “Unicorn.” The latter two are widely used by libraries to identify their online catalog: “Webcat” denotes the Web inter­ face to library online catalogs while “Uni­ corn” is the name of a vendor’s product. To overcome the problem of multiple interpretations of menu choices on the main page, provision was made for mul­ tiple routes to information in this acyclic network menu design. Even with this added feature, participants seemed hesi­ tant in “deciding whether to go to ‘Internet http:context.22 http:architectures.21 http:sites.19 364 College & Research Libraries July 2001 Resources’ or ‘Databases’” and “deciding between ‘Request Forms’ and ‘Research Help,’” despite the fact that either route would have led them to the correct place. Although this is also indicative of the ter­ minology problem, it also highlights the perceived cost of a click. The “click cost” phenomenon has been discussed, but not formally investigated.24 One discussant noted that “users are very reluctant to click a link unless they are fairly certain that they will discover what they are looking for.”25 Because of page load times, people seem reluctant to select an option if it is not al­ most guaranteed to meet their needs. The menu items on the MUN home page contain brief descriptions that serve to annotate the category, providing clues about the content expected at the next menu level. Participants thought “the fact that certain items are listed under the main headings on the home page” and “the fact that ‘Reference Questions’ was listed spe­ cifically under ‘Request forms’” provided valuable clues. Although these annotations were not comprehensive, a point noted by participants, they were the most effective aid in assisting participants to navigate the menus because they provided hints about what might be found on the next menu level. These were much more specific than the menu choice and able to add discrimi­ nating power. Access Tools and Navigation In this study, participants focused almost solely on the menu choices and not on the other tools for accessing information on this site. The home page contains a banner with the heading “Memorial University of Newfoundland Libraries,” while subse­ quent pages use a navigation bar that physically resembles the banner. In particu­ lar, the search engine and the navigation bar that were added to help users navigate the site were used very little over the course of the test. Some participants commented that they did not notice the search site or the navigation bar, whereas others did not realize that items on the bar could be clicked. Users may have experienced the documented Web phenomenon known as “banner blindness” in which large, obvi­ ous objects tend to be ignored.26 Jakob Nielsen, on the other hand, suggested that “users look straight at content and ignore the navigation areas.”27 Alternately, the navigation bar has no “affordances”; that is, there is no visual indication that the navigation bar has clickable space. A simi­ lar study of a banking Web site redesign observed participants clicking on a banner that visually had a 3-D effect that made the banner appear as a menu bar.28 A library Web site is much more than the compilation of HTML code and good visual design. The term MUN Libraries is used to rep­ resent “Home” on the navigation bar. Par­ ticipants did not associate the term with the home page. Using an organization’s logo as a clickable button to return to the home page is common (although it is not known how successful this convention re­ ally is). Tom Brinck and Darren Gergel also found problems with the logo convention and converted to a “Home” button on the navigation bar.29 Thus, clickable buttons must contain some clear indication that the object is clickable and banners must be eas­ ily distinguishable from menu bars. Library Web Site Design For the most part, the issues uncovered in the study were applicable to generic Web site design. The presentation of menu bars and other navigational aids and the struc­ ture of menus have universal application. Terminology has confounded many Web site developments, but the researchers found the problems with standard library terminology perplexing, particularly when that terminology is well established. Likely, however, it has never been user tested; we make too many assumptions about the extent of user knowledge. The significant issue that dominates the study’s findings, however, is that people did not know where to start, which the research­ ers believe contributes to the time taken to do the tasks. The tasks were highly focused, not difficult, and typical tasks for which http:ignored.26 http:investigated.24 Usability of the Academic Library Web Site 365 these users have used physical libraries in the past. The Web site is not so innovative that one could argue that the technology got in the way; the options are, according to par­ ticipants, clearly laid out and well orga­ nized. So why did they have so many diffi­ culties knowing how to begin? A library Web site is much more than the compilation of HTML code and good visual design. It is not just the presenta­ tion of content but, in fact, is a gateway to many types of content. To date, Web sites have not been structured to respond to in­ formation tasks. Ad hoc inventories have been created with myriad pathways to dif­ ferent formats. Web site architectures start as a list of things from which to choose in the form of static words and phrases. Of­ ten these menus are structured much like the physical library and read like the di­ rection board in a physical library (e.g., the catalog, the indexes, the journal collections, etc.), somewhat analogous to a building supply store. But when people approach a Web site with a specific problem, they of­ ten cannot fit the problem into one of the options and do not have access to a hu­ man being for guidance, direction, and problem negotiation. Typical Web site de­ sign is based on the premise that people enter a library or access a Web site to look for a particular tool, such as the catalog, or use a particular service such as interlibrary loan or a particular format such as a jour­ nal. But people have information problems and cannot extrapolate from the list which tool/service is best suited to help with them. The Web site must guide users in the resolution of their problems. How do we make a library Web site as self-servicable as some of the existing e-com­ merce sites, such as Amazon.com? Roy Tennant wrote provocatively about the fu­ ture of libraries and suggested the one-shot application much like the typical Web search engine, complete with a twenty-four-hour help line using videoconferencing technol­ ogy—an innovative approach.30 To accom­ plish a vision such as this, we must apply the disciplined rigor of the systems ap­ proach—from needs analysis and design, to development, evaluation, implementa­ tion, and maintenance—which we formerly applied to library automation projects. A Web site is, after all, another application or, more likely, a set of applications. We also must take a user-centered de­ sign approach to Web development. Even though we know that the interaction be­ tween users and designers during the de­ velopment stage highly correlates with product acceptability, we rarely consult users.31 Pat L. Ensor accurately described the real situation: “Too often we find our­ selves thinking users don’t really know what’s good for them and they don’t un­ derstand.”32 This statement mirrors the comments of one computer scientist who said, “Many computer scientists tend to underestimate the contributions that users could provide, and consider themselves ‘smart’ enough to anticipate them.”33 In a strange paradox, librarians share the bed with computer scientists who have been soundly criticized in the past for their lack of user understanding and who have only recently adopted a user-centered focus to research and development. Part of the problem is the confusion over what is meant by “user-focused” and “user-cen­ tered.” A case in point is that of a recent innovative development undertaken by one library that “conducted a user-focused needs analysis, including an ‘environmen­ tal scan’ of physical library settings … , ‘shadowing’ of librarians … , analysis of paper-based and HTML user guides, and structured interviews with reference librar­ ians.”34 Although the varied methods for data collection are rigorous and commend­ able, not one real user was consulted based on their published report. There is one added conundrum. How do we know when we get it right? In fact, how do we know when the site passes usability testing? In the case of this study, should we be satisfied with the outcome? How effec­ tive and efficient was this Web site? If, on average, 75 percent of the tasks can be com­ pleted successfully, does this rate as barely acceptable, successful, or highly successful? Is two minutes an acceptable amount of time to find the catalog or to find the form to re­ new a book? What is the minimum level of http:users.31 http:approach.30 http:Amazon.com 366 College & Research Libraries July 2001 acceptance that a Web site should reach be­ fore being publicly released? These ques­ tions remain unanswered. Although we have standards for the conduct of usability tests, we have no benchmark standards with which to compare the results. We do not know the values for acceptable efficiency, effectiveness, and satisfaction. Conclusion In this work, the researchers examined usability of an academic Web site. In the process of doing so, they learned about the problems that users have in interact­ ing with the library Web site. How infor­ mation is categorized and labeled for li­ brary Web sites seems to be not intuitive for users. As important, people did not know where to begin. The Web site pro­ vided a directory-style listing of options but failed to provide sufficient advice on how to interact with the tools, services, and products. Creating a Web site for a library cannot be equated with a Web site for an information brochure or an infor­ mation directory. Library Web sites are ap­ plications that support users in their in­ formation tasks. We must take a systematic, user-centered approach to their development. Notes 1. University of Toronto Library Mission Statement. Available online at: http:// www.library.utoronto.ca/aboutlibraries/index.html. 2. M. Harley, J. C. McCarthy, and R. K. Souza, “Why Most Web Sites Fail,” Interactive Technol­ ogy Series 3, no.7 (Forrester Research, 1998). 3. Gerald L. Lohse and Peter Spiller, “Quantifying the Effect of User Interface Design Features on Cyberstore Traffic and Sales,” in CHI ’98: Human Factors in Computing Systems, CHI 98 Conference Proceedings: Making the Impossible Possible, 18–23 April, Los Angeles (New York: Association of Com­ puting Machinery, 1998), 211–18. 4. Jakob Nielsen, Designing Web Usability (Indianapolis, Ind.: New Riders Publishing, 2000). 5. Jared M. Spool et al, Web Site Usability: A Designer’s Guide (San Francisco: Morgan Kaufman, 1999). 6. Shelley Gullikson, Ruth Blades, Marc Bragdon, et al., “The Impact of Information Architec­ ture on Academic Web Site Usability,” Electronic Library 17 (Oct. 1999): 293–304. 7. Christy Hightower, Julie Shih, and Adam Tilghman, “Recommendations for Benchmarking Web Site Usage among Academic Libraries,” College & Research Libraries 59 (Jan. 1998): 61–79. 8. Mark Stover and Steven D. Zink, “World Wide Web Home Page Design: Patterns and Anoma­ lies of Higher Education Library Home Pages,” Reference Services Review 24 (1996): 7–20. 9. David L. King, “Library Home Page Design: A Comparison of Page Layout for Front-Ends to ARL Library Web Sites,” College & Research Libraries 59 (Sept. 1998.): 458–65. 10. Laura B. Cohen and Julie M. Still, “A Comparison of Research University and Two-year College Library Web Sites: Content, Functionality, and Form,” College & Research Libraries 60 (May 1999): 275–89. 11. John D’Angelo and Sherry K. Little, “Successful Web Pages: What Are They and Do They Exist?” Information Technology and Libraries 17 (June 1998): 71–81. 12. ISO, “Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs), Part 11. Guidance on Usability.” ISO 9241-11-1998. 13. Jeffrey Rubin, Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests (New York: John Wiley, 1994). 14. ISO, “Ergonomic Requirements for Office Work with Visual Display Terminals.” 15. Janet Chisman, Karen Diller, and Sharon Walbridge, “Usability Testing: A Case Study,” Col­ lege & Research Libraries 60 (Nov. 1999): 552–69. 16. Jerilyn Veldof, Michael J. Prasse, and Victoria A. Mills, “Chauffeured by the User: Usability in the Electronic Library,” Journal of Library Administration 26 (1999): 115–40. 17. MIT Libraries. Available online at: http://macfadden.mit.edu:9500/webgroup/usability/ results/index.html. 18. Roger Williams University. Available online at: http://gamma.rwu.edu/users/smcmullen/ usable.html. 19. Kenneth R. Paap and Nancy J. Cooke, “Design of Menus,” in Handbook of Human–Computer Interaction, 2nd rev. ed., ed. Martin Helander, Thomas K. Landauer, and Prasad V. Prabhu (Amsterdam: Elsevier, 1997), 533–72. 20. Kent L. Norman, The Psychology of Menu Selection: Designing Cognitive Control of the Human/ Computer Interface (Norwood, N.J.: Ablex Publishing, 1991). http://gamma.rwu.edu/users/smcmullen http://macfadden.mit.edu:9500/webgroup/usability www.library.utoronto.ca/aboutlibraries/index.html Usability of the Academic Library Web Site 367 21. Kevin Larson and Mary Czerwinski, “Web Page Design: Implication of Memory, Structure, and Scent for Information Retrieval,” in CHI ’98: Human Factors in Computing Systems, CHI 98 Con­ ference Proceedings: Making the Impossible Possible, 18–23 April, Los Angeles (New York: Association for Computing Machinery, 1998), 25–32. 22. Gullikson et al, “The Impact of Information Architecture on Academic Web Site Usability.” 23. Michael D. Corry, Theodore W. Frick, and Lisa Hansen, “User-Centered Design and Usabil­ ity Testing of a Web Site: An Illustrative Case Study,” Educational Technology Research and Develop­ ment 45 (1997): 65–76. 24. CHI-WEB discussion group. Available online at: chi-web@acm.org. 25. cwodtke@SIRIUS.COM (9 Feb 2000), CHI-WEB discussion group. Available online at: chi­ web@acm.org. 26. Jan Panero Benway, “Banner Blindness: The Irony of Attention Grabbing on the World Wide Web,” in Proceedings of the Human Factors and Ergonomics Society: 42nd Annual Meeting, vol. 1 (Santa Monica, Calif.: Human Factors and Ergonomics Society, 1998), 463–67. Also available online at: http://www.oxy.edu/~panero/BannerBlindness.pdf 27. Neilsen, Designing Web Usability. 28. Tom Brinck and Darren Gergle, “The Design of Banking Websites: Lessons from Iterative Design,” in 3rd Asia Pacific Computer and Human Interaction: Proceedings, Shonan Village Center, Ja­ pan, July 15–17, 1998 (Los Alamitos, Calif.: IEEE Computer Society,1998), 386–91. 29. Brinck and Gergle, “The Design of Banking Websites.” 30. Roy Tennant, “Determining Our Digital Destiny,” American Libraries 31 (Jan. 2000): 54–58. 31. James D. McKeen, Tor Guimaraes, and James C. Wetherbe, “The Relationship between User Participation and User Satisfaction: An Investigation of Four Contingency Factors,” MIS Quarterly 18 (Dec. 1994): 427–51. 32. Pat L. Ensor, “What’s Wrong with Cool?” Library Journal. Supplement: netconnect (spring 2000): 11–14. 33. Tiziana Catarci, “What Happened When Database Researchers Met Usability,” Information Systems 25 (May 2000): 177–212. 34. Wayne Daniels and Kathy Scardellato, “Past into Future: Capturing Library Expertise in a Virtual Library,” Library Hi-Tech 17 (1999): 181–88. http://www.oxy.edu/~panero/BannerBlindness.pdf mailto:web@acm.org mailto:cwodtke@SIRIUS.COM mailto:chi-web@acm.org