Brantley.indd Usability Testing of a Customizable Library Web Portal Steve Brantley, Annie Armstrong, and Krystal M. Lewis The popularity of customizable Web sites in libraries has increased li- brarians’ interest in supplementing user services with portal technology. The open source-software MyLibrary gives the librarian control over the resources directed to users based on their interests. University of Illinois at Chicago librarians developed My Chicago Library as a way to stream- line user access to library resources. A usability study designed around common task categories tested the participants’ abilities to customize a personal library Web page, understand the resource categories as defined by librarians, and manage the discipline-specific content avail- able in the portal. ustomizable Web pages are now the rule rather that the exception on many library Web sites. Article databases and online library catalogs retain users’ searches or save lists in “book-bags.” Library catalogs give users control of their patron record and circulation data. Static library Web sites are complemented by database-driven sites, or information gateways, frequently referred to as por- tals. Portals are Web pages that permit us- ers to consolidate Web sites and resources pertaining to their individual needs and preferences. My Yahoo!™ is an example of a portal where users customize their page to display links and information of interest, such as business news, television listings, or gardening tips. Portals serve library patrons by highlighting high- quality online information sources and streamlining access to frequently used Web sites and library resources, thereby reducing the phenomenon of information overload that can distract and confuse library patrons. In July 2001, librarians at the Univer- sity of Illinois at Chicago (UIC) received a grant funded through the State of Illinois Library Services and Technology Act to create a customizable library Web portal for use by member libraries of the Chicago Library System, a consortium of public, school, academic, and special libraries (now called Metropolitan Library Sys- tem). A er investigating portal products, the team chose to adapt the MyLibrary portal to meet the grant requirements. Steve Brantley is an Assistant Reference Librarian and Assistant Professor in the Daley Library at the University of Illinois at Chicago; e-mail: jbrant1@uic.edu. Annie Armstrong is an Assistant Reference Librarian and Assistant Professor in the Daley Library at the University of Illinois at Chicago; e-mail: annie@uic.edu. Krystal M. Lewis is an Assistant Reference Librarian, Acting Assistant Special Collections Librarian, and Assistant Professor in the Daley Library at the University of Illinois at Chicago; e-mail: kmlewis1@uic.edu. 146 mailto:kmlewis1@uic.edu mailto:annie@uic.edu mailto:jbrant1@uic.edu Usability Testing of a Customizable Library Web Portal 147 MyLibrary was developed by a team at North Carolina State University through a grant from the National Science Founda- tion Digital Library Initiative. MyLibrary is an open-source project. Open-source so ware products are free or inexpensive programs which software developers may download to modify or enhance, providing they allow other developers access to their modifications. The MyLi- brary portal organizes content according to broad functional areas. MyLibrary creates a gateway to library resources, including subscription databases, online journals, selected Internet resources, and user’s bookmarks. When logged in, the user customizes the design and content of his or her page within parameters set by librarians. Librarians can design the MyLibrary portal to allow users a great deal of customization or very li le. Resources and services considered es- sential by librarians can be included in a users’ page in such a way that they cannot remove them. Moreover, librarians can choose to include or exclude any section of the portal deemed useful or beyond the scope of local collections or services. UIC’s team modified and enhanced MyLibrary, renaming it My Chicago Library. My Chicago Library contains four different versions of MyLibrary, indi- vidualized for school, special, public, and academic libraries. A er developing My Chicago Library, the UIC team speculated about the effectiveness of the portal and whether it was easy to customize and to use. New questions arose: • Are students interested in using a customizable resource portal to conduct research? • Are the portal features intuitive and easily personalized? • Does My Chicago Library assist in the research process? To help answer these questions, the investigators conducted a usability study on UIC’s academic version of the portal, populated with commercial resources li- censed by UIC and with information tools created by UIC librarians such as subject pathfinders. The study had four goals: to determine the clarity of the customization features and ease of navigation, to gather qualitative feedback on the layout and content of the portal, to identify areas needing improvement, and to measure students’ perceptions of the portal as a tool for research. This paper presents the results of that study. Review of the Literature At the time of writing, the body of re- search on measuring the effectiveness of customizable library Web portals through usability testing is limited. However, those implementing or evaluating cus- tomizable library Web portals can draw on studies pertaining to various aspects of open-source so ware, portals, usabil- ity testing, and the MyLibrary so ware. Some articles explore the usability of open-source so ware in general. In ad- dition, numerous usability tests measure the efficacy of use of library Web sites by different user populations. Multiple articles evaluate nonlibrary Web portals, and several articles evaluate portals built using the MyLibrary so ware developed by the team at North Carolina State Uni- versity (NCSU), either anecdotally or though evaluative methods other than usability testing. In “The Usability of Open Source So - ware,” David M. Nichols and Michael B. Twidale suggested that developers of open-source so ware have historically placed more emphasis on technical func- tionality of the so ware than on effective user-centered design.1 They claimed that, in general, open-source so ware develop- ers create applications without focusing on usability. Thus, consecutive releases of the so ware contain usability problems that could have been avoided if developers had given initial a ention to user-centered de- sign. Nichols and Twidale offered several approaches to improving the usability of open-source so ware, including partner- ing with corporate sponsors to increase funding and expertise and involving end users in usability testing.2 148 College & Research Libraries March 2006 Although few articles have presented usability tests of customizable library portals, numerous usability studies mea- suring the effectiveness of library Web sites provide templates that researchers can use as models in measuring the ease of use and functionality of library Web portals. Jeffry Rubin offered practical and comprehensive instructions for the usability testing process in the Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests.3 Elaina Norlin and CM! Winters also offered a practical approach to usability testing in a library se ing in Usability Testing for Library Web Sites: A Hands-on Guide.4 Susan Augus- tine and Courtney Greene supported the usability testing method as an effective means of gathering both quantitative and qualitative feedback about the design of a library Web site.5 Augustine and Greene measured the amount of time and number of clicks required to perform a given task against a benchmark “expert” value.6 In addition, they stressed the importance of recording the verbal feedback of users as they perform each task.7 Louise McGillis and Elaine G. Toms, Ruth Dickstein and Victoria A. Mills, Barbara J. Cockrell and Elaine Andreson Jayne, Brenda Ba leson, Austin Booth, and Jane Weintrop, and Janet K. Chisman, Karen R. Diller and Sharon L. Walbridge all also offered prac- tical models for assessing the usability of library Web sites and search tools.8–12 These studies concur that usability testing of a group of no more than eight to ten subjects is an effective and cost-efficient means of gathering data pointing to prob- lems in Web site functionality, design, and terminology. In 2000, Todd Zazelenchuk and James Lane released the results of a usability study of the OneStart Portal, a prototype of a campuswide information portal for Indiana University.13 Although OneStart was not a library Web portal, the study measured user satisfaction with custom- ization features unique to online portals, thus offering a practical model for the collection and compilation of usability study test data. In this study, usability test scores were divided into categories (e.g., overall flexibility, clarity of terms).14 As echoed in other Web site usability studies, “clarity of terms” proved to be an area in which users voiced greatest dis- satisfaction with the portal. Most usability studies present participants with a series of explicit tasks. In a departure from this method, Zazelenchuk and Lane presented nine users with a printout of an already customized portal page and asked them to manipulate an uncustomized portal page until it matched the printout.15 This method is limited to usability tests of customizable applications. The authors claimed that this method eliminates the level of coaching implicit in usability tests that outline specific tasks in detail.16 In 2001, Justin Dopke and Gary Mar- chionini published the results of a us- ability test of the North Carolina State Library StartSquad Web Portal for Chil- dren.17 This test gathered and synthesized feedback of eight test subjects ranging from preschool to middle-school age. Usability study tasks were designed to measure suitability and recognizability of interface graphics, top-level navigation functions, information retrieval functions, and overall satisfaction with the interface. Due to the intended age of the audience, the study has somewhat limited appli- cability to the testing of a college-level tool such as the academic version of My Chicago Library. However, the authors’ classification of task types can be easily applied to evaluative tests of academic library Web portals. Since the team at North Carolina State University conceived of and released the MyLibrary so ware, numerous case studies and anecdotal articles have been published that recount the experiences of libraries implementing the so ware. These articles offer insight into issues ranging from the initial workload re- quired for implementing MyLibrary to feedback from patrons about the useful- ness of having a customizable library Web portal at their disposal. These articles are http:detail.16 http:printout.15 http:terms).14 http:University.13 Usability Testing of a Customizable Library Web Portal 149 o en brief and occasionally anecdotal in tone, but they help explain the need for customization in the evolving online environment. The December 2000 is- sue of Information Technology in Libraries compiles several case studies, including some about the implementation of MyLi- brary at North Carolina State University, Virginia Commonwealth University, and Washington University.18 In this journal issue, Amos Lakos and Chris Gray argued that although portals will ultimately make libraries more customer centered and visible, librarians must realize the shi in work responsibilities and workload inherent in adopting and investing in a customizable library Web portal.19 In his 2003 article “Pu ing the ‘My’ in MyLi- brary,” Eric Lease Morgan contended that commercial sites such as Google.com or Amazon.com condition library users to expect automatic customization.20 He argued that MyLibrary supplies library users with the convenience they have learned to expect from the commercial sector.21 The Open University in the United Kingdom adapted the open-source MyLi- brary so ware to create a customizable library Web portal for university students and faculty called MyOpenLibrary.22 Jane Lunsford published results of a study of MyOpenLibrary using a task-oriented usability test on eight subjects. She iden- tified “a number of small problems with language and organization of the site.”23 Lunsford collected positive user feedback about the usefulness of the portal as a tool for organizing a “range of resources for a course.” To date, her usability study of MyOpenLibrary is the only published research to evaluate the ease of use of the MyLibrary open-source so ware. Methodology A usability test consisting of a series of nineteen tasks was administered to eight prescreened participants recruited from the UIC community. The tasks were de- signed to test a range of portal functions with an emphasis on customization. The time and actions taken to complete the tasks, as well as the participants’ verbal comments, were recorded and analyzed. At UIC, the Institutional Review Board (IRB) must approve studies involving hu- man subjects. The investigators submi ed a description of the study and all support- ing documents such as questionnaires, advertisements, and usability tasks to the IRB and received exemption from federal regulations for the protection of human subjects. The research protocol ensured the confidentiality of all participants and that there was no potential to harm participants. When IRB approval was obtained, the investigators began recruiting partici- pants for the study. Flyers were created and posted in buildings on campus, and an announcement was posted on the cam- puswide electronic announcement board. To interest participants, the investigators offered an incentive in the form of a $15 gi certificate redeemable at the campus bookstores, a provision of the original grant. Over the course of two weeks, the investigators received phone calls from respondents who were screened to de- termine their eligibility. The investigators wanted to recruit participants with a range of computer skills and experience. To this end, a brief screening survey was designed and ad- ministered to each respondent over the telephone. The survey contained ques- tions about experience using computers and familiarity with, and frequency of use of, library and Internet resources. (See appendix B.) Approximately fifteen people re- sponded to the recruiting advertisements and were given the screening survey. The investigators scheduled appointments with eleven respondents and conducted eight usability tests. The number of us- ability tests conducted was based on the literature. Jakob Neilsen posits in a column on his Web site, “Why You Only Need to Test with 5 Users” that the larger the test group, the more repetition will be http:MyOpenLibrary.22 http:sector.21 http:customization.20 http:Amazon.com http:Google.com http:portal.19 http:University.18 150 College & Research Libraries March 2006 revealed.25 According to the curve that il- lustrates this, 80 percent of the problems will be discovered with five users, eight users will uncover about 90 percent of the problems, and 100 percent of the problems will be revealed by fi een users. In this vein, in their usability study of an Online Public Access Catalog (OPAC), Chisman, Diller, and Wallbridge claimed that “eight participants would identify 80% of the problems users might have with the system.”26 The usability tests were conducted over a two-week period. One-hour ap- pointments were set with each user, and users were given a choice of a Windows-based PC or Macintosh computer to use for their test. Individual tests took place in the investigators’ of- fices with two investigators present. Participants were given a paper copy of the test to consult, and each task was read out loud by an investigator. Participants were encouraged to speak out loud while completing each task, giving explana- tions for their actions and commenting on their reactions to the content, appear- ance, and organization of the portal. One investigator recorded users’ paths, noting each action a participant took to complete a task. A second investigator read the tasks aloud to the participant and re- corded the amount of time spent on each task. It was unnecessary to take extensive notes detailing users’ commentary at the time of the test because each session was audiotaped. However, the investigators recorded salient points and particularly strong participant reactions at the time of the testing so that it would be easier to isolate problematic areas when analyz- ing the data. The investigators used the informal method of usability testing in which us- ers were instructed to complete a set of tasks using the portal while “thinking out loud,” voicing their thought process and comments, as suggested by Alison J. Head.27 Her recommendations of al- lowing users no more than four or five minutes to complete each task, for a total test time of no more than an hour, were followed as well.28 Before conducting the usability test on the participants, the investigators admin- istered a pretest with three volunteers to make sure that the wording of the tasks was straightforward and to identify any errors, omissions, or problems with the tasks. This ensured that a user’s perfor- mance would be based on the level of dif- ficulty of a given task, not the complexity of the wording. It also ensured that any problematic tasks could be corrected prior to actual testing. Two versions of the test were wri en, with the nineteen tasks ar- ranged in different orders, so that if user performance improved throughout the course of the test, it could not be a rib- uted to the order of the tasks. Appendix A represents the tasks in the order they were performed by one group. The tasks for the usability test were de- signed to test a wide range of portal func- tions. Though tasks varied, they fell into three categories the investigators named and defined: Administrative Functions, Category Recognition, and Content Ma- nipulation. Administrative Functions are tasks related to a user’s personal account or profile or the layout of the portal page, such as creating a new account, changing one’s discipline, or altering the colors of the page. Tasks that required the user to find a resource based on its category name fall under Category Recognition. These tasks were designed to determine the usefulness of the terms used to name each section of the portal, such as “E-Journals,” “Finding Articles,” “Other Resources,” and “Extras.” Asking a user to link to the UIC History Department is an example of this type of task. Content Manipulation describes tasks in which the user is asked to alter the content of the portal, such as adding an electronic journal from the film discipline or creating a link to the New York Times in the MyLinks section. Some of the Content Manipulation tasks inherently involve Category Recognition as well be- cause users must determine which section is to be customized. These task categories http:revealed.25 Usability Testing of a Customizable Library Web Portal 151 provided a framework for analyzing the qualitative and quantitative data accord- ing to the four goals of the study. In addition to creating task categories, the investigators established benchmark paths and times for each task based on the amount of time and the number of ac- tions it took the investigators to complete each task. These “expert paths” were the most direct route for completing a task. It was important to the validity of the test to determine beforehand what constituted an action. The investigators defined an action as any manipulation that changes the appearance of the portal page, such as pressing the “submit” bu on, resulting in a resource being added or removed, or click- ing on a link that takes a user to a different page. Field entry, meaning marking an item in a list or entering text in a text field of a Web form, was counted as one action. Navigation within the portal page, such as clicking on an HTML anchor or scrolling up and down, was not counted as an ac- tion. A time limit of three minutes was set for each of the nineteen tasks, which kept the test to just under an hour. (The list of tasks is shown in appendix A.) The My Chicago Library Portal My Chicago Library is arranged into six customizable resource categories, one current awareness category, and a sidebar that lists library catalogs, Chicago com- munity information, ready reference re- sources, and contact information. (See fig- ure 1.) Unlike the resource categories, the sidebar cannot be altered by the user. In addition, a logo in the top le corner links to the library homepage, which serves as a brand for the My Chicago Library home institution. The top of the page displays the My Chicago Library banner, a stylized silhoue e of the Chicago skyline. Beneath that is a navigation bar linking the user to all the administrative functions for managing a user account, customizing the color and layout of a page, and select- ing a discipline. The discipline, which is intended to correspond to the UIC user’s major and is chosen by the user, deter- mines the resources that are delivered to the user’s page. Directly beneath the navigation bar is a customizable title banner displaying the name of the portal, which can be personalized by the user. A navigation bar of HTML anchors linking to resource sections lies beneath the title, and, finally, the user’s name and the date are displayed. The content sections, labeled Services, Quick Searches, Finding Articles, Elec- tronic Journals, Other Resources, My links, and Extras, can be edited to include FIGURE 1 My Chicago Library* *Note, only two of the six resource categories are visible in the figure 152 College & Research Libraries March 2006 as much or as li le content as the indi- vidual prefers. An exception is Updates, which is reserved for current awareness information delivered by the librarian who manages the user’s discipline. The Services section links to interlibrary loan pages, circulation services, troubleshoot- ing, and other library services. Quick Searches is a shortcut to search engines, thesauri, and dictionaries. Finding Ar- ticles displays discipline-specific article indexes and full-text content aggregators, and Electronic Journals lists high-impact journals available electronically. Other Resources is a catchall term to organize se- lected links to nonsubscription Web sites and discipline-specific Web resources that do not fit the other categories. MyLinks and Extras are additional sections that display personal bookmarks, news, and games. Results of Task Category Analysis Each task category defined in the meth- odology—Administrative Functions, Category Recognition and Content Ma- nipulation—provided quantitative and qualitative data for accomplishing the first three goals of the study: to determine the clarity of the customization features and ease of navigation, to gather qualita- tive feedback on the layout and content of the portal, and to identify areas for im- provement. The fourth goal, to measure participants’ perceptions of the portal as a tool for research, was accomplished through analysis of qualitative data from user feedback. Administrative Functions Tasks Administrative Functions tasks had the greatest impact on evaluating the ease of customizing the portal. These tasks changed the background color, banner and text colors, and the layout of the portal. If Administrative Functions tasks were completed, the design of the cus- tomization features and navigation was considered to be effective. Screening survey data revealed that the participants were familiar and com- fortable with Internet applications. (See appendix B.) The majority of participants rated themselves at the highest level (5 on a scale from 1 to 5) in terms of their comfort and skill with computers and the Internet. All eight test participants recorded at least three years of computer use, five participants had used comput- ers for five to seven years, and two had at least ten years of experience. All the par- ticipants reported that they used the Inter- net every day. Although the participants self- reported a high level of comfort with computers and the Internet (daily Internet use might indicate competency), these are by no means standardized measures and are not intended to prove the participants’ skill levels, but only to indicate their own perceptions of their computer skills. Of the Administrative Functions tasks, all the participants completed tasks 1, 2, and 3. (See table 1 for a complete list of tasks and scores grouped by category.) Task 4 was completed by five participants, and task 5 was completed by four partici- pants. That 83 percent of Administrative Functions tasks were completed suggests that the customization features of My Chicago Library were conceptually clear and visually apparent. For this study, all participants were instructed to select history as their dis- cipline so that all resources would be identical for every test. Examination of the average time required for the Ad- ministrative Functions tasks shows that tasks 1, 4, and 5 required two or three times as long as the expert’s benchmark time. Of the Administrative Functions tasks, 4 and 5 were also the tasks that participants were most o en unable to complete. The 50 percent completion rate for task 5, “change your discipline,” may have occurred because the participants were instructed to select history as their discipline. Because this requirement of the study prevented them from selecting their own discipline, they may not have been as aware of the importance of mak- ing a personalized selection. If a given participant was a history major, he or she Usability Testing of a Customizable Library Web Portal 153 TABLE 1 Tasks And Scores Tasks Grouped by Category Ta sk c at eg or y* P er ce nt co m pl et ed E xp er t t im e (s ec on ds ) A ve ra ge ti m e (s ec on ds ) St an da rd de vi at io n E xp er t p at h ac ti on s A ve ra ge u se r ac ti on s D ev ia tio ns fr om ex pe rt p at h 1 Create a new account AF 100 50 105 32.08 4 4.6 2 2 Change the background color AF 100 14 23 15.90 3 3 0 3 Log out of My Chicago Library. Then log in again AF 100 22 30 14.15 5 5 0 4 Change the title to “My Strawberry” AF 62.5 26 70 50.67 4 5.8 11 5 Change your discipline to “psychology” AF 50 16 59 33.37 3 4.5 11 6 Link to the UIC History Department CR 100 9 70 58.36 2 4.9 13 7** Find the definition of the word neologism CR 100 17 21 8.41 3 2.5 2 8 Link to the library catalog you would use to get the call number for this book CR 87.5 8 42 58.24 1 1.9 3 9 Use the portal to recall a book CR 100 3 28 23.28 1 2.6 7 10 Find today’s Chicago Sun- Times headline CR 50 10 14 10.84 2 2 0 11** Remove the link to “Archives USA” CR/ CM 100 20 35 31.30 4 4.4 2 12 Remove the link to “Sports Scores” CR/ CM 100 11 24 15.26 3 4.2 5 13 Create a link to the New York Times Web site CM 100 30 45 21.75 4 4.9 1 14 Remove the link to the elec- tronic journal International Review of Social History CM 100 17 50 33.70 4 4.5 3 15 Add a resource to the “Other Resources” section from the film discipline CM 50 25 90 57.38 5 5 5 16 Add an electronic journal to the “Electronic Journals” section from the alphabeti- cal list CM 87.5 29 58 18.88 5 6 5 17** Add a link to the electronic journal Asian Art CM 62.5 31 96 64.52 5 5.4 7 154 College & Research Libraries March 2006 TABLE 1 Tasks And Scores Tasks Grouped by Category Ta sk c at eg or y* P er ce nt co m pl et ed E xp er t t im e (s ec on ds ) A ve ra ge ti m e (s ec on ds ) St an da rd de vi at io n E xp er t p at h ac ti on s A ve ra ge u se r ac ti on s D ev ia tio ns fr om ex pe rt p at h 18 Remove “Sage Family Stud- ies Abstracts” CM 100 13 27 16.83 4 4.8 4 19 Add two databases to the “Finding Articles” section from the sociology discipline CM 100 30 58 34.34 5 5.8 4 * AF= Administrative Functions, CM= Content Manipulation, CR=Category Recognition. CR/CM could be construed as both CR and CM tasks. ** In these cases the lowest number of actions is lower than the expert’s. In task 7 the participant im- mediately left the portal to perform the task, therefore no other actions could be counted as part of the task. In task 11, technical failure caused one user to complete the test using a backup account. In task 17, the participant selected an alternative route to completion which, while requiring fewer actions, took more time and was a less efficient route. might recognize the resources delivered to his or her page as relevant to history. Participants majoring in chemistry, for example, would be unlikely to recognize the resources delivered to their page and therefore might not make the connection between selecting a discipline in My Chicago Library and seeing the relevant resources on their page. Thus, being asked to change one’s discipline might not have the implicit importance that it would have had, had they first chosen a familiar area of study. Tasks 4 and 5 asked the users to make changes to the portal’s appearance and to their academic discipline or, as la- beled in the portal, their profile. To make these changes, users must navigate to an administrative editing page and make selections from a list of options. My Chi- cago Library displays two navigation bars along the top of the screen. (See figure 1.) The uppermost navigation, with links to “new account,” “layout,” “profile,” and “login” take the user out of the portal to the aforementioned administration page. The other navigation bar is a horizontal list of HTML anchors that are internal links to the resource sections further down the page. Because participants must use the upper navigation bar to complete Administrative Functions tasks, low completion rates indicate that they either could not find or did not understand the labels on the navigation bar. Category Recognition Tasks Completion of the second task type, Cat- egory Recognition, required participants to understand the type of electronic resource they were adding, removing, or linking to and the label of the section into which that resource was grouped. These tasks aided in the analysis of the portal’s ease of navigation. The results of Category Recognition tasks 6 and 8, “Link to the UIC History Department” and “link to the library cata- log you would use” illustrate important navigational issues. All the participants completed task 6, and only one par- ticipant did not complete task 8. Despite this high completion rate, on average, the tasks required a great deal of time to finish. Participants spent a lot of time reviewing the portal page for the correct choice or following incorrect links. The link to the UIC History Depart- ment is listed in the “Other Resources” section, and the library catalog is listed Usability Testing of a Customizable Library Web Portal 155 in the sidebar. “Other Resources” is a catchall category for resources such as academic departments or Web sites that do not fit into the categories of electronic journals, article databases, or library services. The participants recognized that the UIC History Department was not an article database or electronic journal and therefore was not listed in those sections, but they found the label “Other Resources” to be too vague. This may indicate that the users tried to find explicit, meaningful categories for the answers rather than catchall categories such as “other.” The UIC library catalog is within a section of the sidebar labeled “Library Catalogs” and is, in fact, the top choice in that list, a parameter controllable by the librarian administrator of My Chicago Library. It is possible that participants overlooked the sidebar because it can- not be customized and is not part of the central content of the portal page. For this reason, they might have spent the majority of the allo ed time scan- ning the portal page before finding the catalog link. There was a high number (13) of navi- gational deviations from the expert path for task 6. By examining the navigational deviations made in this task and in other tasks, salient examples were discovered pertaining to the problems encountered with Category Recognition tasks. Five participants made thirteen navigation actions more than were necessary before completing the task. Participants did not understand that the UIC History Department link would be included in their customized portal even a er they had been instructed to create a profile as a history major. One participant put it succinctly, saying, “I didn’t realize it was customized to someone in history. Like, if it was [customized] in biology, the UIC biology department would be there.” Before this task, some of the participants had not ascertained that by creating a profile in the portal, they were selecting discipline-specific resources. Fi y percent of the participants did not complete Category Recognition task 10, “Find today’s Chicago Sun-Times head- line.” Like the UIC catalog, the link to the newspaper is located in the sidebar, in an area populated with Chicago community information and ready reference sources. Completion of this task may indicate that users understood why the Chicago- oriented content was included and why it was located in the sidebar. Participants who did not complete the task overlooked the sidebar and spent time trying to find the newspaper in one of the library re- source categories. In the analysis of Category Recognition tasks, investigators recorded a total of 32 additional (and superfluous) actions made by the participants. These navi- gational actions, which deviate from the expert path, raised the average actions per task and suggest that the participants used trial and error rather than an under- standing of the portal organization. The screening survey reveals that the majority of participants had never received any library instruction. (See appendix B.) This may explain some of the participants’ navigational deviations because sections are labeled with library-specific terms that students might only be familiar with through library instruction. Augustine and Greene noted in their library Web site usability study that “especially note- worthy were the participants’ difficulty with library terminology and their lack of knowledge of library resources.”29 Even a Web research tool with a high level of visual and navigational clarity will be confusing if the end users do not under- stand the descriptive language used to identify categories of information. Content Manipulation Tasks The third task category, Content Manipu- lation, requires users to add or remove portal content and combines actions required by Category Recognition tasks and Administrative Functions tasks. Thus, Content Manipulation tasks were used to evaluate two of the primary 156 College & Research Libraries March 2006 goals of the study: determine the clarity of the customization features and ease of navigation. Of the three task categories, Content Manipulation tasks had the low- est completion rate. Content Manipulation tasks also had the highest average number of deviations from the expert path. When performing Content Manipulation tasks, users navigated to a customization page where content choices were listed in three groups. First, current or default selec- tions were listed next to check boxes. By adding or removing check marks, users can add or remove resources. Second, users can view an alphabetical list of all the resources in that section. Third, they can view a list of all resources designated for that discipline. The high incidence of deviations from the expert path indicated that participants were scanning the edit page without easily recognizing which type of listing to review. For example, one participant sought an answer by viewing resources listed by discipline, when searching for the resource from the alphabetical list would have been a more efficient choice. In addition to the deviations from the expert path, the time spent completing Content Manipulation tasks suggested difficulty in understanding the way the resources were organized in the portal. The average times recorded for tasks 15 and 17 (“Add a resource to the ‘Other Resources’ section” and “Add a link in the Electronic Journals section to the electronic journal Asian Art”) were three times those of the expert’s time. In task 15, participants may not have recognized that two actions were necessary to view the list of resources. In addition to selecting “cus- tomize” from the “Other Resources” title bar, they must select to view the resources either alphabetically or by discipline. The Content Manipulation task cat- egory is arguably the most difficult to per- form because it requires an understand- ing of the Administrative Functions and Category Recognition tasks, in addition to requiring the greatest number of actions to complete. A er making one or more changes to a section, users have to click on a “submit” bu on to complete the change. The changes are then displayed within the list of current resources. At this point, users can continue to customize or, if sat- isfied with the changes, can navigate back to their portal page through a “Return to MyLibrary” link. Participants said that this sequence was confusing. They were unsure that their edits had been success- ful and they wanted clearer confirmation of this. Some participants suggested that a pop-up dialog box stating the changes would be helpful. This desire for confir- mation stands in contrast to users’ com- ments about the account creation page. Those comments suggested that the portal provided too much explanation. Qualitative Feedback Participant commentary provided the in- vestigators with much useful information about navigation, layout, usability, and user comprehension of the design intent of the portal. Navigation problems were identified when participants commented that many pages were too long and re- quired too much scrolling. Participants made these comments when they were adding or removing resources from the customization page. Lists of resources are generated by the MyLibrary portal and can be very long depending on the number of resources in the database. My Chicago Library did not contain any navigational management tools for these lists. Participants suggested that the lists contain “back-to-top” links and HTML anchors linking to sections within the lists. Participants’ responses to the activities required for account creation revealed many usability issues. These comments were particularly important because they correlate with the first encounter a new user has with My Chicago Library. The form for creating a new account contains several text boxes and radio buttons, all of which require some action or text entry from the user. Next to each text Usability Testing of a Customizable Library Web Portal 157 entry field is a text-based help link, which provides an explanation of the informa- tion users need to enter. While creating a new account, one participant noted: “I don’t think the tab bu on is working. It is frustrating. I want to move from text box to text box and the tab bu on won’t automatically go there.” The user became frustrated when a empting to use the tab key instead of the mouse to place the cursor in the next field because the cur- sor advanced to the help link associated with that field before moving on to the next field. Participants also were critical of the login process. Following the login, but before a user is directed to their portal page, an intermediate page asks them if they want to place a cookie in the browser cache that would retain their login infor- mation. The rationale for incorporating this intermediate page arose from a concern that many users would be ac- cessing My Chicago Library from public workstations and that automatically sav- ing the user’s login information risked user privacy. Participants commented that this page was unnecessary and held too much explanatory information. Web design convention handles this issue by placing a small checkbox next to the login fields labeled “remember my user name” or “remember me on this computer.” Some participants complained that they did not automatically return to their portal page after submitting a change from the customization page. A er the “submit” bu on is clicked, the portal up- dates the customization page to include the most recent changes. Only when us- ers click on the “Return to My Chicago Library” link are they redirected to their portal page. Other critical comments related to the “Quick Search” function. The “Quick Search” section, located below the library services section, presents the user with a text entry field and a choice of Internet search resources such as Google.com, Dictionary.com, or Bartleby.com. The “Quick Search” is a shortcut to content. Contrary to the designer’s expectations, the participants expected the “Quick Search” to perform an internal search of the portal, not a search of the World Wide Web. They expressed surprise that the search box was positioned in a content section and said that a site search function should be located in a corner of the page. My Chicago Library does not provide a site search function in the way that users desired, in part because it is not composed of static HTML files containing textual content that can be searched. Unlike a stat- ic Web site, My Chicago Library generates each user’s portal page from content in a relational database. The users’ assumption that the “Quick Search” provided a site search and was therefore a navigational tool indicates their misunderstanding of the portal as a content tool. Participants commented that the side- bar was too clu ered, too lengthy, and a poor use of window space. Some told the investigators that they did not need the Chicago community information and would not use it, which indicated that the participants were not interested in the portal as a Chicago community-centered resource and suggests the importance of the participants’ orientation as UIC stu- dents above their orientation as citizens of Chicago. They also stated that menus presented as lists in the sidebar wasted page space. They suggested using pull- down menus instead. My Chicago Library for UIC was de- signed to complement the library’s Web site. The portal did not include a link to the University Web site in its tested incarnation. The UIC icon branding the portal went to the UIC library. To the surprise of the investigators, the par- ticipants suggested that a home bu on be included that would return them to the university homepage. They expected links to university services and student services. The designers and investigators possessed an orientation to the library whereas the test participants appeared to have a broader viewpoint. Comments such as “I would like to see headlines or 158 College & Research Libraries March 2006 UIC news” indicate that they did not dif- ferentiate between the university’s Web presence and the library’s Web presence in the same way that the portal designers and the investigators did. Participants misapprehended the scope of the portal because they expected a university-wide orientation and not a portal limited to library resources. The investigators received several posi- tive comments and a great deal of con- structive criticism from the participants about the proposed implementation of the portal. Participants responded enthusias- tically to the portal, despite their lack of a full understanding of its intended pur- pose. Comments about the service, such as “I would definitely use it if the library offered it,” “it is very intuitive,” and “it is very user-friendly,” suggested the partici- pants’ estimation of the portal as a useful research tool. Participants stated that they would require some time to “play” with the portal, familiarizing themselves with its content and functions, but that they would return to it when they had custom- ized it. Although these comments indicate that the portal requires a time investment to customize, other qualitative feedback suggests that the portal would be a useful library service. Redesign Suggestions This study confirmed the need for suc- cinct instructions. Participants indicated a preference for visual cues such as bu ons or icons rather than textual explanations. The text-based help links on the “account creation” page were an impediment to some users. Participants also commented on the lengthiness of instructions on the account creation page, the login page, and the editing pages. The majority of study participants reported a high level of computer skill; therefore, the help texts associated with making selections from lists or common Web activities such as completing and submi ing Web forms may be unnecessary. It is possible that this sample of highly skilled users may have suggested design changes that less experienced users would find difficult to navigate. However, it should be noted that the participants’ levels of computer skills were self-reported. The investigators did not perform any objective evaluation of skills as those skills might be defined by a standard measure. Without a standard by which to evaluate the participants, the self reported skill levels must be viewed as levels of confidence and comfort with computers and the Internet rather than objectively evaluated abilities. The par- ticipants’ scores in the study, whether they were close to the expert scores or not, do not in and of themselves sup- port the participants’ claims of expertise. Given this, design suggestions resulting from the study and from participant sug- gestions do not necessarily indicate that less experienced—or rather less comfort- able— users would have greater difficulty with the changes. Despite the ease with which the cus- tomizations were made, the participants wanted explicit confirmation that Content Manipulation changes had actually gone through. A small icon or textual expla- nation that appears whenever a form is submi ed could solve this issue. The difficulty that participants had with Category Recognition tasks indi- cates that the section labels need to be easier to understand. Given the well- established problems users have with library terminology, one possible remedy to this issue would be to use labels that identify the type of content listed rather than the type of resource listed. For ex- ample, rather than the label “Finding Articles,” which stresses the process of the search rather than the desired product, the section could be labeled “Journal Ar- ticles.” In addition, the label “Electronic Journals” could be changed to “History Journals” or “Film Journals,” depend- ing on the user’s discipline. Because so much periodical literature is available in full-text, the distinction that librar- ians draw between electronic journals and paper journals may no longer be as Usability Testing of a Customizable Library Web Portal 159 relevant as a few years ago when e-jour- nals were new and had limited content. The label “Magazines and Journals,” as general as that may sound to a librarian, might be a be er choice than “Electronic Journals.” The study identified many areas for improvement in the use of page space. Participants repeatedly suggested space- saving techniques and improvements to the layout of the sidebar, the services op- tions, and to the list of resource options on the customization pages. Participants suggested compact lists and menus, such as drop-down menus, or mouse-over menus to avoid long scrolling pages. Finally, participants wanted a bright and colorful portal. They approved of the Chicago skyline banner but felt that the default gray for the section labels was drab and uninviting. Even though the styles and colors were customizable by the user, the test group suggested that the default page of My Chicago Library needed a more professionally designed look. Conclusions The users in this study appeared to have difficulty understanding the discipline- specific nature of the My Chicago Library portal. Following instructions to create a history account rather than making personal selections may have hindered users from understanding the level of control they had over the portal content. In designing a study in which all users performed the same tasks with the same content, the investigators may have inadvertently prevented participants from learning and understanding the intended purpose of the portal. More- over, the participants suggested that the portal include university services and university headlines. This request and their presumption that the UIC logo be linked to the university homepage rather than to the library indicate a different orientation to the library than the design- ers expected. The participants considered the university to be the next logical par- ent site as opposed to the library’s main homepage. Study results indicate that even ex- perienced computer users struggle with customizing My Chicago Library. How- ever, results also suggest that when users overcome this learning curve, they appre- ciate the shortcuts offered by the portal. The individualized pages direct users to crucial resources and provide shortcuts through the universe of information avail- able from the library. My Chicago Library can be a useful alternative to traditional library Web sites in which users must navigate through many layers of pages to reach the resources they seek. Although appreciative of the stream- lined access the portal provides, some participants stated that they would use the “customize” tool to include all the resources from their discipline to ensure a comprehensive list of relevant resources. This comment and others suggested to the investigators that users want to select high-quality resources in their discipline, but they resist any other limitations on their options. Objections to the lengthy explana- tions, overly long lists, and ubiquitous help links showed the investigators that My Chicago Library navigation needs to conform to common Web features by including shorter pages or making page sections easily accessible through the use of anchors. This is especially important when managing potentially long lists. Designers should value clarity of visual layout and minimize textual explanations in future iterations of the portal. This study bolsters Augustine and Greene’s assertions that user misunder- standing of librarian-defined resource categories remains a serious stumbling block to library Web site usability.30 Li- brarians must label resources using com- mon language, even at the risk of incom- plete or slightly inaccurate descriptions. As demonstrated by participants in this study, users frequently misunderstand how libraries organize information if they have not had library instruction. http:usability.30 160 College & Research Libraries March 2006 Implications and Suggested Research Since this study was performed, the UIC library has implemented MyLibrary ver- sion 2.63, dubbing it MyLibrary@UIC. The portal now more closely resembles the UIC library Web site and incorporates the colors and the finished quality that test participants requested. Also, the list of disciplines has been expanded to in- clude all courses of study at UIC. An FAQ page explaining the portal’s purpose and functionality has been added. Currently, the library is investigating ways to incor- porate automated methods of uploading electronic resources into the MyLibrary database. In future studies, a screening survey revealing the participants’ experience with customizable Web products might be helpful. An exit survey designed to col- lect detailed perceptions of the portal also could provide valuable data, as would a standard measure of computer skills. The investigators anticipate conducting another usability study following the changes made to the My Chicago Library portal based on study findings and a er implementing new versions of the MyLi- brary so ware. By designing the study to measure the completion of specific tasks in My Chicago Library, the investigators were able to identify both problems and good design elements of MyLibrary as modi- fied for Chicago-area UIC students and faculty. By creating tasks that fall into three categories of use—Administrative Functions, Category Recognition, and Content Manipulation, the study helps the investigators understand the participants’ behavior in a conceptual framework. Results viewed in this framework allow librarians to apply data or lessons learned to other Web-based interactive research tools. As Web functionality expands and becomes more complex, interactions with Web-based research tools can be improved by usability testing that incorporates func- tional categories to the test design. This study and others like it may help libraries determine how to incorporate a portal into their electronic services or how to create a portal that meets the research needs of their user population. The investigators hope that this study will assist librarians in creating intuitive and user-friendly research tools, and provide a template for librarians conducting usability testing on library research tools. APPENDIX A List of Tasks in the Order in Which They Were Performed 1. Create a new account. Use “bu ercup” as your user name and “Illinois” as your password. Select “history” as your discipline. When you are logged in, select “Just show me my page.” 2. Create a link to the New York Times Web site (h p://www.nytimes.com/) in your “My Links” section. Then return to My Chicago Library. 3. Link to the UIC History Department. Then click the browser’s “Back” bu on to return to My Chicago Library. 4. Remove the link from the “Electronic Journals” to the electronic journal Interna- tional Review of Social History. Then return to My Chicago Library. 5. Change the background color of My Chicago Library to yellow. 6. Log out of My Chicago Library. Then log in again. When you are logged in, select “Just show me my page.” http:h�p://www.nytimes.com Usability Testing of a Customizable Library Web Portal 161 7. Add a resource to the “Other Resources” section from the film discipline. Then return to My Chicago Library. 8. Add an electronic journal to the “Electronic Journals” section from the alphabeti- cal list of electronic journals. Then return to My Chicago Library. 9. Use the portal to find the definition of the word neologism. 10. Remove the link to “Archives USA.” Then return to My Chicago Library. 11. Add a link in the “Electronic Journals” section to the electronic journal Asian Art. Then return to My Chicago Library. 12. The UIC library owns the novel A Day Late and a Dollar Short. Link to the library catalog you would use to get the call number for this book. Then click the browser’s “Back” bu on to return to My Chicago Library. 13. Remove the link to “Sports Scores.” 14. The novel A Day Late and a Dollar Short is checked out. Use the portal to recall it. 15. Change your discipline to “psychology.” Merge your presently customized selec- tions with the suggested selections of the new discipline. 16. Remove “Sage Family Studies Abstracts” from the “Finding Articles” section. Then return to My Chicago Library. 17. Change the title of “My Chicago Library” to “My Strawberry.” 18. Add two databases to the “Finding Articles” section from the sociology discipline. Then return to My Chicago Library. 19. Find today’s Chicago Sun-Times headline. Then click the browser’s “Back” bu on to return to My Chicago Library. APPENDIX B The Participant Screening Survey How long have you been using a computer? 0–1 years 1–3 years 3–5 years 5–7 years More than 10 years 0 0 1 5 2 What kinds of things do you use the computer to do? Word processing Internet searching Online chat E-mail other 8 8 7 8 8 What is your status at the University of Illinois at Chicago? undergraduate graduate faculty staff 0 7 0 1 162 College & Research Libraries March 2006 How much experience do you have using the UIC library? Never used Occasional use (once a month) Frequent use (once a week) 0 3 5 Have you had a library instruction session? yes no 2 6 Which of the following UIC library resources have you used on the World Wide Web? UICCAT Article databases Electronic journals Electronic reference materials 8 3 5 4 How often do you use the Internet? Never Rarely (once a month) Occasionally (once a week) Frequently (once a day) 0 0 0 8 On a scale from 1 to 5 with 1 being low and 5 being high, how would you rate your comfort with using computers in general? 1 2 3 4 5 0 0 1 1 6 On a scale from 1 to 5 with 1 being low and 5 being high, how would you rate your comfort with using Internet resources? 1 2 3 4 5 0 0 2 1 5 On a scale from 1 to 5 with 1 being low and 5 being high, how would you rate your skill at using computers in general? 1 2 3 4 5 0 1 0 3 4 On a scale from 1 to 5 with 1 being low and 5 being high, how would you rate your skill at using Internet resources? 1 2 3 4 5 0 11 1 2 4 Are you more comfortable using a PC or a Mac? PC Mac 8 0 Usability Testing of a Customizable Library Web Portal 163 Notes 1. David M. Nichols and Michael B. Twidale, “The Usability of Open Source So ware,” First Monday 8, no. 1 (Jan. 2003). Available online at h p://firstmonday.org/issues/issue8_1/ nichols/index.html. [Accessed 14 February 2005]. 2. Ibid. 3. Jeffrey Rubin, Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests (New York City: Wiley, 2002). 4. Elaina Norlin and CM! Winters, Usability Testing for Library Websites: A Hands-on Guide (Chicago: ALA, 2002). 5. Susan Augustine and Courtney Greene, “Discovering How Students Search a Library Web Site: A Usability Case Study,” College & Research Libraries 63, no. 4 (July 2002): 354–65. 6. Ibid., 357. 7. Ibid., 358. 8. Louise McGillis and Elaine G. Toms, “Usability of the Academic Library Web Site: Implica- tions for Design. Study at the Memorial University of Newfoundland,” College & Research Libraries 62, no. 4 (July 2001): 355–67. 9. Ruth Dickstein and Victoria A. Mills, “Usability Testing at the University of Arizona Library: How to Let the Users in on the Design,” Information Technology and Libraries 19, no. 3 (Sept. 2000): 144–51. 10. Barbara J. Cockrell and Elaine Anderson Jayne, “How Do I Find an Article? Insights from a Web Usability Study at Western Michigan University,” Journal of Academic Librarianship 28, no. 3 (May 2002): 122–32. 11. Brenda Ba leson, Austin Booth, and Jane Weintrop, “Usability Testing of an Academic Library Web Site: A Case Study at SUNY Buffalo,” Journal of Academic Librarianship 27, no. 3 (May 2001): 188–98. 12. Janet K. Chisman, Karen R. Diller, and Sharon L. Walbridge, “Usability Testing: A Case Study at Washington State University,” College & Research Libraries 60, no. 6 (Nov. 1999): 552–69. 13. Todd Zazelenchuk and James Lane, “OneStart Portal Usability Report: Electronic Prototype Test Results and Recommendations, Tested November 2000.” Available online at h p://www. indiana.edu/~usable/reports/test2_report.pdf. [Accessed 18 February 2005]. 14. Ibid., 7. 15. Ibid., 5. 16. Ibid. 17. Justin Dopke and Gary Marchionini, “Report of the Results of a Usability Test of the North Carolina State Library StartSquad Web Portal for Children, University of North Carolina- Chapel Hill, 2001.” Available online at h p://ils.unc.edu/idl/Kids_portal_final_report.pdf. [Accessed 14 February 2005]. 18. Amos Lakos and Chris Gray, “Personalized Library Portals as an Organizational Culture Change Agent,” Information Technology and Libraries 19, no.4 (Dec. 2000): 169–74. 19. Ibid., 172. 20. Eric Lease Morgan, “Pu ing the ‘My’ in MyLibrary,” Library Journal 128, no. 17 (fall 2003): 24–27. 21. Ibid., 26. 22. Jane Lunsford, “MyOpenLibrary: Personalized Library Resources. Report on a Usability Evaluation, May 12th, 13th, 14th, 2003.” Available online at h p://library.open.ac.uk/aboutus/myo- lib/reports.htm. [Accessed 5 January 2004]. 23. Ibid. 24. Ibid. 25. Jakob Nielsen, “Why You Only Need to Test with 5 Users: Jakob Nielsen’s Alertbox.” Available online at h p://www.useit.com/alertbox/20000319.html. [Accessed 14 February 2005]. 26. Chisman, Diller, and Walbridge, “Usability Testing,” 553. 27. Alison J. Head, “Web Redemption and the Promise of Usability,” Online 23, no. 6, (Nov./Dec. 1999): 20–28. 28. Ibid. 29. Augustine and Greene, “Discovering How Students Search a Library Web Site,” 364. 30. Ibid., 358.