Library Discovery Products: Discovering User Expectations through Failure Analysis Irina Trapido INFORMATION TECHNOLOGY AND LIBRARIES | SEPTEMBER 2016 9 ABSTRACT As the new generation of discovery systems evolve and gain maturity, it is important to continually focus on how users interact with these tools and what areas they find problematic. This study looks at user interactions within SearchWorks, a discovery system developed by Stanford University Libraries, with an emphasis on identifying and analyzing problematic and failed searches. Our findings indicate that users still experience difficulties conducting author and subject searches, could benefit from enhanced support for browsing, and expect their overall search experience to be more closely aligned with that on popular web destinations. The article also offers practical recommendations pertaining to metadata, functionality, and scope of the search system that could help address some of the most common problems encountered by the users. INTRODUCTION In recent years, rapid modernization of online catalogs has brought library discovery to the forefront of research efforts in the library community, giving libraries an opportunity to take a fresh look at such important issues as the scope of the library catalog, metadata creation practices, and the future of library discovery in general. While there is an abundance of studies looking at various aspects of planning, implementation, use, and acceptance of these new discovery environments, surprisingly little research focuses specifically on user failure. The present study aims to address this gap by identifying and analyzing potentially problematic or failed searches. It is hoped that focusing on common error patterns will help us gain a better understanding of users’ mental models, needs, and expectations that should be considered when designing discovery systems, creating metadata, and interacting with library patrons. TERMINOLOGY In this paper, we adopt a broad definition of discovery products as “tools and interfaces that a library implements to provide patrons the ability to search its collections and gain access to materials.”1 These products can be further subdivided into the following categories: Irina Trapido (itrapido@stanford.edu) is Electronic Resources Librarian at Stanford University Libraries, Stanford, California. mailto:itrapido@stanford.edu LIBRARY DISCOVERY PRODUCTS: DISCOVERING USER EXPECTATIONS THROUGH FAILURE ANALYSIS |IRINA TRAPIDO |doi:10.6017/ital.v35i2.9190 10 • Online catalogs (OPACs)—patron-facing modules of an integrated library system. • Discovery layers (also referred to as “discovery interfaces” or “next-generation library catalogs”)—new catalog interfaces, decoupled from the integrated library system and offering enhanced functionality, such as faceted navigation, relevance-ranked results, as well as the ability to incorporate content from institutional repositories and digital libraries. • Web-scale discovery tools, which in addition to providing all interface features and functionality of next generation catalogs, broaden the scope of discovery by systematically aggregating content from library catalogs, subscription databases, and institutional digital repositories into a central index. LITERATURE REVIEW To identify and investigate problems that end users experience in the course of their regular searching activities, we analyzed digital traces of user interactions with the system recorded in the system’s log files. This method, commonly referred to as transaction log analysis, has been a popular way of studying information-seeking in a digital environment since the first online search systems came into existence, allowing researchers to monitor system use and gain insight into the users’ search process. Server logs have been used extensively to examine user interactions with web search engines, consistently showing that web searchers tend to engage in short search sessions, enter brief search statements, do not browse the results beyond the first page, and rarely resort to advanced searching.2 A similar picture has emerged from transaction log studies of library catalogs. Researchers have found that library users employ the same surface strategies: queries within library discovery tools are equally short and simply constructed; 3 the majority of search sessions consist of only one or two actions.4 Patrons commonly accept the system’s default search settings and rarely take advantage of a rich set of search features traditionally offered by online catalogs, such as Boolean searching, index browsing, term truncation, and fielded searching.5 Although advanced searching in library discovery layers is uncommon, faceted navigation, a new feature introduced into library catalogs in the mid-2000s, quickly became an integral part of the users’ search process. Research has shown that facets in library discovery interfaces are used both in conjunction with text searching, as a search refinement tool, and as a way to browse the collection with no search term entered.6 A recent study that analyzed interaction patterns in a faceted library interface at the North Carolina State University using log data and user experiments demonstrated that users of faceted interfaces tend to issue shorter queries, go through fewer iterations of query reformulation, and scan deeper along the result list than those who use nonfaceted search systems. The authors also concluded that facets increase search accuracy, especially for complex and open-ended tasks, and improve user satisfaction.7 INFORMATION TECHNOLOGY AND LIBRARIES | SEPTEMBER 2016 11 Another traditional use of transaction logs has been to gauge the performance of library catalogs, mostly through measuring success and failure rates. While the exact percentage of failed searches varied dramatically depending on the system’s search capabilities, interface design, the size of the underlying database, and, most importantly, on the researchers’ definition of an unsuccessful search, the conclusion was the same: the incidence of failure in library OPACs was extremely high.8 In addition to reporting error rates, these studies also looked at the distribution of errors by search type (title, author, or subject search) and categorized sources of searching failure. Most researchers agreed that typing errors and misspellings accounted for a significant portion of failed searches and were common across all search types.9 Subject searching, which remained the most problematic area, often failed because of a mismatch between the search terms chosen by the user and the controlled vocabulary contained in the library records, suggesting that users experienced considerable difficulties in formulating subject queries with Library of Congress Subject Headings.10 Other errors reported by researchers, such as the selection of the wrong search index or the inclusion of the initial article for title searches, were also caused by users’ lack of conceptual understanding of the search process and the system’s functions.11 These research findings were reinforced by multiple observational studies and user interviews, which showed that patrons found library catalogs “illogical,” “counter-intuitive,” and “intimidating,”12 and that patrons were unwilling to learn the intricacies of catalog searching.13 Instead, users expected simple, fast, and easy searching across the entire range of library collections, relevance-ranked results that exactly matched what users expected to find, and convenient and seamless transition from discovery to access.14 Today’s library discovery systems have come a long way: they offer one-stop search for a wide array of library resources, intuitive interfaces that require minimal training to be searched effectively, facets to help users narrow down the result set, and much more.15 But are today’s patrons always successful in their searches? Usability studies of next-generation catalogs and, more recently, of web-scale discovery systems have pointed to patron difficulties associated with the use of certain facets, mostly because of terminological issues and inconsistencies in the underlying metadata.16 Researchers also reported that users had trouble interpreting and evaluating the results of their search;17 users also were confused as to what resources were covered by the search tool.18 Our study builds on this line of research by systematically analyzing real-life problematic searches as reported by library users and recorded in transaction logs. BACKGROUND Stanford University is a private, four-year or above research university offering undergraduate and graduate degrees in a wide range of disciplines to about sixteen thousand students. The study analyzed the use of SearchWorks, a discovery platform developed by Stanford University Libraries. SearchWorks features a single search box with a link to advanced search on every page, relevance- ranked results, faceted navigation, enhanced textual and visual content (summaries, tables of LIBRARY DISCOVERY PRODUCTS: DISCOVERING USER EXPECTATIONS THROUGH FAILURE ANALYSIS |IRINA TRAPIDO |doi:10.6017/ital.v35i2.9190 12 content, book cover images, etc.), as well as “browse shelf” functionality. SearchWorks offers searching and browsing of catalog records and digital repository objects in a single interface; however, it does not allow article-level searching. SearchWorks was developed on the basis of Blacklight (projectblacklight.org), an open-source application for searching and interacting with collections of digital objects.19 Thanks to Blacklight’s flexibility and extensibility, SearchWorks enables discovery across an increasingly diverse range of collections (MARC catalog records, archival materials, sound recordings, images, geospatial data, etc.) and allows to continuously add new features and improvements (e.g., https://library.stanford.edu/blogs/stanford-libraries-blog/2014/09/searchworks-30-released). STUDY OBJECTIVES The goal of the present study was two-fold. First, we sought to determine how patrons interact with the discovery systems, which features they use and with what frequency. Second, this study aimed to identify and analyze problems that users encounter in their search process. METHOD This study used data comprising four years of SearchWorks use, which was recorded in Apache Solr logs. The analysis was performed at the aggregate level; no attempts were made to identify individual searchers from the logs. At the preprocessing stage, we created and used a series of Perl scripts to clean and parse the data and extract only those transactions where the user entered a search query and/or selected at least one facet value. Page views of individual records were excluded from the analysis. The resulting output file contained the following parameters for each transaction: a time stamp, search mode used (basic or advanced), query terms, search index (“all fields,” “author,” “title,” “subject,” etc.), facets selected, and the number of results returned. The query stream was subsequently partitioned into task-based search sessions using a combination of syntactic features (word co- occurrence across multiple transactions) and temporal features (session time-outs: we used fifteen minutes of inactivity as a boundary between search sessions). The analysis was conducted over the following datasets: Dataset 1. Aggregate data of approximately 6 million search transactions conducted between February 13, 2011, and December 31, 2014. We performed quantitative analysis of this set to identify general patterns of system use. Dataset 2. A sample of 5,101 search sessions containing 11,478 failed or potentially problematic interactions performed in the basic search mode and 2,719 sessions containing 3,600 advanced searches, annotated with query intent and potential cause of the problem. The searches were performed during eleven twenty-four-hour periods, representing different years, academic http://projectblacklight.org/ https://library.stanford.edu/blogs/stanford-libraries-blog/2014/09/searchworks-30-released INFORMATION TECHNOLOGY AND LIBRARIES | SEPTEMBER 2016 13 quarters, times of the school year (beginning of the quarter, midterms, finals, breaks), and days of the week. This dataset was analyzed to identify common sources of user failure. Dataset 3. User feedback messages submitted to SearchWorks between January 2011 and December 2014 through the “Feedback” link, which appears on every SearchWorks page. While the majority of feedback messages were error and bug reports, this dataset also contained valuable information about how users employed various features of the discovery layer, what problems they encountered, and what features they felt would improve their search experience. For the manual analysis of dataset 2, all searches within a search session were reconstructed in SearchWorks and, in some cases, also in external sources such as WorldCat, Google Scholar, and Google. They were subsequently assigned to one of the following categories: known-item searches (searches for a specific resource by title, combination of title and author, a standard number such as ISSN or ISBN, or a call number), author searches (queries for a specific person or organization responsible for or contributing to a resource), topical searches, browse searches (searches for a subset of the library collection, e.g., “rock operas,” “graphic novels,” “DVDs,” etc.), invalid queries, and queries where the search intent could not be established. To identify potentially problematic transactions, the following heuristic was employed: we selected all search sessions where at least one transaction failed to retrieve any records, as well as sessions consisting predominantly of known-item or author searches, where the user repeated or reformulated the query three or more times within a five-minute time frame. We hypothesized that this search pattern could be part of the normal query formulation process for topical searches, but it could serve as an indicator of the user’s dissatisfaction with the results of the initial query for known-item and author searches. We identified seventeen distinct types of problems, which we further aggregated into the following five groups: input errors, absence of the resource from the collection, queries at the wrong level of granularity, erroneous or too restrictive use of limiters, and mismatch between the search terms entered and the library metadata. Each search transaction in dataset 2 was manually reviewed and assigned to one or more of these error categories. FINDINGS Usage Patterns Our analysis of the aggregate data suggests that keyword searching remains the primary interaction paradigm with the library discovery system, accounting for 76 percent of all searches. However, users also increasingly take advantage of facets both for browsing and refining their searches: the use of facets grew from 25 percent in 2011 to 41 percent in 2014. LIBRARY DISCOVERY PRODUCTS: DISCOVERING USER EXPECTATIONS THROUGH FAILURE ANALYSIS |IRINA TRAPIDO |doi:10.6017/ital.v35i2.9190 14 Although both the basic and the advanced search modes allow for “fielded” searches, where the user can specify which element of the record to search (author, title, subject, etc.), searchers rarely made use of this feature, relying mostly on the system’s defaults (the “all fields” search option in the basic search mode): users selected a specific search index in less than 25 percent of all basic searches. Advanced searching was infrequent and declining (from 11 percent in 2011 to 4 percent in 2014). Typically, users engaged in short sessions with a mean session length of 1.5 queries. Search queries were brief: 2.9 terms per query on average. Single terms made up 23 percent of queries; 26 percent had two terms, and 19 percent had three terms. Error Patterns The breakdown of errors by category and search mode is shown in figure 1. In the following sections, we describe and analyze different types of errors. Figure 1. Breakdown of errors by category and search mode Input Errors Input errors accounted for the largest proportion of problematic searches in the basic search mode (29 percent) and for 5 percent of problems in the advanced search. While the majority of such errors occurred at the level of individual words (misspellings or typographical errors), entire search statements were also imprecise and erroneous (e.g., “Diary of an Economic Hit Man” instead of “Confessions of an Economic Hit Man” and “Dostoevsky War and Peace” instead of “Tolstoy War and Peace”). It is noteworthy that in 46 percent of all search sessions containing INFORMATION TECHNOLOGY AND LIBRARIES | SEPTEMBER 2016 15 problems of this type, users subsequently entered a corrected query. However, if such errors occurred in a personal name, they were almost half as likely to be corrected. Absence of the Item Sought from the Collection Queries for materials that were not in the library’s collection accounted for about a quarter of all potentially problematic searches. In the advanced search modality, where the query is matched against a specific search field, such queries typically resulted in zero hits and can hardly be considered failures per se. However, in the default cross-field search, users were often faced with the problem of false hits and had to issue multiple progressively more specific queries to ascertain that the desired resource was absent from the collection. Queries at the Wrong Level of Granularity A substantial number of user queries failed because they were posed at the level of specificity not supported by the catalog. Such queries accounted for the largest percentage of problematic advanced searches (63 percent), where they consisted almost exclusively of article-level searching: users either tried to locate a specific article (often by copying the entire citation or its part from external sources) or conducted highly specific topical searches more suitable for a full- text database. In the basic search mode, the proportion of searches at the wrong granularity level was much lower, but still substantial (20 percent). In addition to searches for articles and narrowly defined subject searches, users also attempted to search for other types of more granular content, such as book chapters, individual papers in conference proceedings, poems, songs, etc. Erroneous or Too Restrictive Use of Limiters Another common source of failure was the selection of the wrong search index or a facet that was too restrictive to yield any results. The majority of these errors were purely mechanical: users failed to clear out search refinements from their previous search or entered query terms into the wrong search field. However, our analysis also revealed several conceptual errors, typically stemming from a misunderstanding of the meaning and purpose of certain limiters. For example, “Online,” “Database,” and “Journal/Periodical” facets were often perceived by the user as a possible route to article-level content. Even seemingly straightforward limiters such as “Date” caused confusion, especially when applied to serial publications: users attempted to employ this facet to drill down to the desired journal issue or article, most likely acting on the assumption that the system included article-level metadata. Lack of Correspondence between the Users’ Search Terms and the Library Metadata A significant number of problems in this group involved searches for non-English materials. When performed in their English transliteration, such queries often failed because of users’ lack of LIBRARY DISCOVERY PRODUCTS: DISCOVERING USER EXPECTATIONS THROUGH FAILURE ANALYSIS |IRINA TRAPIDO |doi:10.6017/ital.v35i2.9190 16 familiarity with the transliteration rules established by the library community, whereas searches in the vernacular scripts tended to produce incomplete or no results because not all bibliographic records in the database contained parallel non-Roman script fields. Author and title searches often failed because of the users’ tendency to enter abbreviated queries. For example, personal name searches where the user truncated the author’s first or middle name to an initial while the bibliographic records only contained this name in its full form were extremely likely to fail. Abbreviations were also used in searches for journals, conference proceedings, and occasionally even for book titles (e.g., “AI: a modern approach” instead of “Artificial intelligence: a modern approach”). Such queries were successful only if the abbreviation used by the searcher was included in the bibliographic records as a variant title. A somewhat related problem occurred when the title of a resource contained a numeral in its spelled out form but was entered as a digit by the user. Because these title variations are not always recorded as additional access points in the bibliographic records, the desired item either did not appear in the result set or was buried too deep to be discovered. Topical searches within the subject index were also prone to failure, mostly because patrons were unaware that such searches require the use of precise terms from controlled vocabularies and resorted to natural language searching instead. User Feedback Our analysis of user feedback revealed substantial differences in how various user groups approach the search system and which areas of it they find problematic. Students were often frustrated by the absence of spelling suggestions, which, as one user put it, “left the users wander [to?] in the dark” as to the cause of searching failure. This user group also found certain social features desirable: for example, one user suggested that having ratings for books would be helpful in his choice of a good programming book. By contrast, faculty and researchers were more concerned about the lack of the more advanced features, such as cross-reference searching and left-anchored browsing of the title, subject, and author indexes. However, there were several areas that both groups found problematic: students and faculty alike saw the system’s inability to assist in the selection of the correct form of the author’s name as a major barrier to effective author searching and also converged on the need for more granular access to formats of audiovisual materials. DISCUSSION Scope of the Discovery System The results of our analysis point to users’ lack of understanding of what is covered by the discovery layer. Users are often unaware of the existence of separate specialized search interfaces for different categories of materials and assume that the library discovery layer offers Google-like INFORMATION TECHNOLOGY AND LIBRARIES | SEPTEMBER 2016 17 searching across the entire range of library resource types. Moreover, they are confused by the multiple search modalities offered by the discovery layer: one of the common misconceptions in SearchWorks is that the advanced search will allow the user to access additional content rather than offer a different way of searching the same catalog data. In addition to the expanded scope of the discovery tools, there is also a growing expectation of greater depth of coverage. According to our data, searching in a discovery layer occurs at several levels: the entire resource (book, journal title, music recording), its smaller integral units (book chapters, journal articles, individual musical compositions, etc.), and full text. User Search Strategies The search strategies employed by SearchWorks users are heavily influenced by their experiences with web search engines. Users tend to engage in brief search sessions and use short queries, which is consistent with the general patterns of web searching. They rely on relevance ranking and are often reluctant to examine search results in any depth: if the desired item does not appear within the first few hits, users tend to rework their initial search statement (often with only a minimal change to the search terms) rather than scrolling down to the bottom of the results screen or looking beyond the first page of results. Given these search patterns, it is crucial to fine-tune relevance-ranking algorithms to the extent that the most relevant results are displayed not just on the first page but are included in the first few hits. While this is typically the case for unique and specific queries, more general searches could benefit from a relevance-ranking algorithm that would leverage the popularity of a resource as measured by its circulation statistics. Adding this dimension to relevance determination would help users make sense of large result sets generated by broad topical queries (e.g., “quantum mechanics,” “linear algebra,” “microeconomics”) by ranking more popular or introductory materials higher than more specialized ones. It could also provide some guidance to the user trying to choose between different editions of the same resource and improve the quality of results of author searches by ranking works created by the author before critical and biographical materials. Users’ query formulation strategies are also modeled by Google, where making search terms as specific as possible is often the only way to increase the precision of a search. Faceted search systems, however, require a different approach: the user is expected to conduct a broad search and subsequently focus it by superimposing facets on the results. Qualifying the search upfront through keywords rather than facets is not only ineffective, but may actually lead to failure. For example, a common search pattern is to add the format of a resource as a search term (e.g., “Fortune magazine,” “Science journal,” “GRE e-book,” “Nicole Lopez dissertation,” “Woody Allen movies”), and because the format information is coded rather than spelled out in the bibliographic records, such queries either result in zero hits or produce irrelevant results. In a similar vein, making the query overly restrictive by including the year of publication, publisher, or edition LIBRARY DISCOVERY PRODUCTS: DISCOVERING USER EXPECTATIONS THROUGH FAILURE ANALYSIS |IRINA TRAPIDO |doi:10.6017/ital.v35i2.9190 18 information often causes empty retrievals because the library might not have the edition specified by the user or because the query does not match the data in the bibliographic record. Thus our study lends further weight to claims that even in today’s reality of sophisticated discovery environments and unmediated searching, library users can still benefit from learning the best search techniques that are specifically tailored to faceted interfaces.20 Error Tolerance Input errors remain one of the major sources of failure in library discovery layers. Users have become increasingly reliant on error recovery features that they find elsewhere on the web, such as “Did you mean . . . ” suggestions, automatic spelling corrections, and helpful suggestions on how to proceed in situations where the initial search resulted in no hits. But perhaps even more crucial are error-prevention mechanisms, such as query autocomplete, which helps users avoid spelling and typographical errors and provides interactive search assistance and instant feedback during the query formulation process. Our visual analysis of the logs from the most recent years revealed an interesting search pattern, where the user enters only the beginning of the search query and then increments it by one or two letters: pr pro proq proque proques proquest Such search patterns indicate that users expect the system to offer query expansion options and show the extent to which the query autocomplete feature (currently missing from SearchWorks) has become an organic part of the users’ search process. Topical Searching While next-generation discovery systems represent a significant step toward enabling more sophisticated topical discovery, a number of challenges still remain. Apart from mechanical errors, such as misspellings and wrong search index selections, the majority of zero-hit topical searches were caused by a mismatch between the user’s query and the vocabulary in the system’s index. In many cases such queries were formulated too narrowly, reflecting the users’ underlying belief that the discovery layer offers full-text searching across all of the library’s resources. In addition to keyword searching, libraries have traditionally offered a more sophisticated and precise way of accessing subject information in the form of Library of Congress Subject Headings (LCSH). However, our results indicate that these tools remain largely underused: users took advantage of this feature in only 21 percent of all subject searches in our sample. We also found INFORMATION TECHNOLOGY AND LIBRARIES | SEPTEMBER 2016 19 that 95 percent of LCSH usage came from clicks on subject heading links within individual bibliographic records rather than from “Subject” facets, corroborating the results of earlier studies.21 There is a whole range of measures that could help patrons leverage the power of controlled vocabulary searching. They include raising the level of patron familiarity with the LCSHs, integrating cross-references for authorized subject terms, enabling more sophisticated facet- based access to subject information by allowing users to manipulate facets independently, and exposing hierarchical and associative relationships among LCSHs. Ideally, once the user has identified a helpful controlled vocabulary term, it should be possible to expand, refine, or change the focus of a search through broader, narrower, and related terms in the LCSH’s hierarchy as well as to discover various aspects of a topic through browse lists of topical subdivisions or via facets. Known-Item Searching Important as it is for the discovery layer to facilitate topical exploration, our data suggests that SearchWorks remains, first and foremost, a known-item lookup tool. While a typical SearchWorks user rarely has problems with known-work searches, our analysis of clusters of closely related searches has revealed several situations where users’ known-item search experience could be improved. For example, when the desired resource is not in the library’s collection, the user is rarely left with empty result sets because of automatic word-stemming and cross-field searching. While this is a boon for exploratory searching, it becomes a problem when the user needs to ensure that the item sought is not included in the library’s collection. Another common scenario arises when the query is too generic, imprecise, or simply erroneous, or when the search string entered by the user does not match the metadata in the bibliographic record, causing the most relevant resources to be pushed too far down the results list to be discoverable. Providing helpful “Did you mean . . . ” suggestions could potentially help the user distinguish between these two scenarios. Another feature that would substantially benefit the user struggling with the problem of noisy retrievals is highlighting the user’s search terms in retrieved records. Displaying search matches could alleviate some of the concerns over lack of transparency as to why seemingly irrelevant results are retrieved, repeatedly expressed in user feedback, as well as expedite the process of relevance assessment. Author Searching Author searching remains problematic because of a convergence of factors: a. Misspellings. According to our data, typographical errors and misspellings are by far the most common problem in author searching. When such errors occur in personal names, they are much more difficult to identify than errors in the title, and in the absence of LIBRARY DISCOVERY PRODUCTS: DISCOVERING USER EXPECTATIONS THROUGH FAILURE ANALYSIS |IRINA TRAPIDO |doi:10.6017/ital.v35i2.9190 20 index-based spell-checking mechanisms, often require the use of external sources to be corrected. b. Mismatch between the form and fullness of the name entered by the user and the form of the name in the bibliographic record. For example, a user’s search for “D. Reynolds” will retrieve records where “D” and “Reynolds” appear anywhere in the record (or anywhere in the author fields, if the user opts for a more focused “author” search), but will not bring up records where the author’s name is recorded as “Reynolds, David.” c. Lack of cross-reference searching of the LC Name Authority file. If the user searches for a variant name represented by a cross-reference on an authority record, she might not be directed to the authorized form of the name. d. Lack of name disambiguation, which is especially problematic when the search is for a common name. While the process of name authority control ensures the uniqueness of name headings, it does not necessarily provide information that would help users distinguish between authors. For instance, the user often has to know the author’s middle name or date of birth to choose the correct entry, as exemplified by the following choices in the “Author” facet resulting from the query “David Kelly”: Kelly, David Kelly, David (David D.) Kelly, David (David Francis) Kelly, David F. Kelly, David H. Kelly, David Patrick Kelly, David St. Leger Kelly, David T. Kelly, David, 1929 July 11– Kelly, David, 1929– Kelly, David, 1929–2012 Kelly, David, 1938– Kelly, David, 1948– Kelly, David, 1950– Kelly, David, 1959– e. Errors and inaccuracies in the bibliographic records. Given the past practice of creating undifferentiated personal-name authority records, it is not uncommon to have one name heading for different authors or contributors. Conversely, situations where a single person is identified by multiple headings (largely because some records still contain obsolete or variant forms of a personal name) are also prevalent and may INFORMATION TECHNOLOGY AND LIBRARIES | SEPTEMBER 2016 21 become a significant barrier to effective retrieval as they create multiple facet values for the same author or contributor. f. Inability to perform an exhaustive search on the author’s name. A fielded “Author” search will miss the records where the name does not appear in the “Author” fields but appears elsewhere in the bibliographic record. g. Relevance ranking. Because search terms occurring in the title have more weight than search terms in the “Author” fields, works about an author are ranked higher than works of the author. Browsing Like many other next-generation discovery systems, SearchWorks features faceted navigation, which facilitates both general-purpose browsing and more targeted search. In SearchWorks, facets are displayed from the outset, providing a high-level overview of the collection and jumping-off points for further exploration. Rather than having to guess the entry vocabulary, the searcher may just choose from the available facets and explore the entire collection along a specific dimension. However, findings from our manual analysis of the query stream suggest that facets as a browsing tool might not be used to their fullest potential: users often resort to keyword searching when faceted browsing would have been a more optimal strategy. There are at least two factors that contribute to this trend. The first is users’ lack of awareness of this interface feature: it is common for SearchWorks users to issue queries such as “dissertations,” “theses,” and “newspapers” instead of selecting the appropriate value of the “Format” facet. Second, many of the facets that could be useful in the discovery process are not available as top-level browsing categories. For example, users expect more granular faceting of audiovisual resources, which would include the ability to browse by content type (“computer games,” “video games”) and genre (“feature films,” “documentaries,” “TV series,” “romantic comedies”). Another category of resources commonly accessed by browsing is theses and dissertations. Users frequently try to browse dissertations by field or discipline (issuing searches such as “linguistics thesis,” “dissertations aeronautics,” “PhD thesis economics,” “biophysics thesis”), by program or department and by the level of study (undergraduate, master’s, doctoral), and could benefit from a set of facets dedicated to these categories. Browsing for books could be enhanced by additional faceting related to intellectual content, such as genre and literary form (e.g., “fantasy,” “graphic novels,” “autobiography,” “poetry”) and audience (e.g., “children’s books”). Users also want to be able to browse for specific subsets of materials on the basis of their location (e.g., permanent reserves at the engineering library). Browsing for new acquisitions with the option of limiting to a specific topic is also a highly desirable feature. LIBRARY DISCOVERY PRODUCTS: DISCOVERING USER EXPECTATIONS THROUGH FAILURE ANALYSIS |IRINA TRAPIDO |doi:10.6017/ital.v35i2.9190 22 While some browsing categories are common across all types of resources, others only apply to specific types of materials (e.g., music, cartographic/geospatial materials, audiovisual resources, etc.). For example, there is a strong demand among music searchers for systematic browsing by specific musical instruments and their combinations. Ideally, the system should offer both an optimal set of initial browse options and intuitive context-specific ways to progressively limit or expand the search. Offering such browsing tools may require improvements in system design as well as significant data remediation and enhancement because much of the metadata that could be used to create these browsing categories is often scattered across multiple fixed and variable fields in the bibliographic records, inconsistently recorded, or not present at all. One of the hallmarks of modern discovery tools has been their increased focus on developing tools that would facilitate serendipitous browsing. SearchWorks was one of the pioneers to offer virtual “browse shelf” feature, which is aimed at emulating browsing the shelves in a physical library. However, because this functionality relies on the classification number, it does not allow browsing of many other important groups of materials, such as multimedia resources, rare books, or archival resources. Call-number proximity is only one of the many dimensions that could be leveraged to create more opportunities for serendipitous discoveries. Other methods of associating related content might include recommendations based on subject similarity, authorship, keyword associations, forward and backward citations, and use. Implications for Practice Addressing the issues that we identified would involve improvements in several areas: • Scope. Our findings indicate that library users increasingly perceive the discovery interface as a portal to all of the library’s resources. Meeting this need goes far beyond offering the ability to search multiple content sources from a single search box: it is just as important to help users make sense of the results of their search and to provide easy and convenient ways to access the resources that they have discovered. And whatever the scope of the library discovery layer is, it needs to be communicated to the user with maximum clarity. • Functionality. Users expect a robust and fault-tolerant search system with a rich suite of search-assistance features, such as index-based alternative spelling suggestions, result screens displaying keywords in context, and query auto-completion mechanisms. These features, many of which have become deeply embedded into user search processes elsewhere on the web, could prevent or alleviate a substantial number of issues related to problematic user queries (misspellings, typographical errors, imprecise queries, etc.), enable more efficient recovery from errors by guiding users to improved results, and facilitate discovery of foreign-language materials. Equally important is the continued focus on relevance ranking algorithms, which ideally should move beyond simple keyword- INFORMATION TECHNOLOGY AND LIBRARIES | SEPTEMBER 2016 23 matching techniques toward incorporating social data as well as leveraging the semantics of the query itself and offering more intelligent and possibly more personalized results depending on the context of the search. • Metadata. The quality of the user experience in the discovery environments depends as much on the metadata as it does on the functionality of the discovery layer. Thus it remains extremely important to ensure consistency, granularity, and uniformity of metadata, especially as libraries are increasingly faced with the problem of integrating heterogeneous pools of metadata into a single discovery tool. CONCLUSIONS AND FUTURE DIRECTIONS The analysis of the transaction log data and user feedback has helped us identify several common patterns of search failure, which in turn can reveal important assumptions and expectations that users bring to the library discovery. These expectations pertain primarily to the system’s functionality: in addition to simple, intuitive, and visually appealing interfaces and relevance- ranked results, users expect a sophisticated search system that would consistently produce relevant results even for incomplete, inaccurate, or erroneous queries. Users also expect a more centralized, comprehensive, and inclusive search environment that would enable more in-depth discovery by offering article-level, chapter-level, and full-text searching. Finally, the results of this study have underscored the continued need for a more flexible and adaptive system that would be easy to use for novices while offering advanced functionality and more control over the search process for the “power” users, a system that would provide targeted support for the different types of information behavior (known-item look-up, author searching, topical exploration, browsing) and would facilitate both general inquiry and very specialized searches (e.g., searches for music, cartographic and geospatial materials, digital collections of images, etc.). Just like discovery itself, building discovery tools is a dynamic, complex, iterative process that requires intimate knowledge of ever-changing and evolving user needs and expectations. It is hoped that ongoing focus on user problems and frustrations in the new discovery environments can complement other assessment methods by identifying unmet user needs, thus helping create a more holistic and nuanced picture of users’ search and discovery behaviors. REFERENCES 1. Marshall Breeding, “Library Resource Discovery Products: Context, Library Perspectives, and Vendor Positions,” Library Technology Reports 50, no. 1 (2014): 5–58. 2. Craig Silverstein et al., “Analysis of a Very Large Web Search Engine Query Log,” SIGIR Forum 33, no. 1 (1999): 6–12; Bernard J. Jansen, Amanda Spink, and Tefko Saracevic, “Real Life, Real Users, and Real Needs: A Study and Analysis of User Queries on the Web,” Information LIBRARY DISCOVERY PRODUCTS: DISCOVERING USER EXPECTATIONS THROUGH FAILURE ANALYSIS |IRINA TRAPIDO |doi:10.6017/ital.v35i2.9190 24 Processing & Management 36, no. 2 (2000): 207–27, http://dx.doi.org/10.1016/S0306- 4573(99)00056-4; Amanda Spink, Bernard J. Jansen, and H. Cenk Ozmultu, “Use of Query Reformulation and Relevance Feedback by Excite Users,” Internet Research 10, no. 4 (2000): 317–28; Amanda Spink et al., “Searching the Web: The Public and Their Queries,” Journal of the American Society for Information Science & Technology 52, no. 3 (2001): 226–34; Bernard J. Jansen and Amanda Spink, “An Analysis of Web Searching by European AllteWeb.com Users,” Information Processing & Management 41, no. 2 (2005): 361–81, http://dx.doi.org/10.1016/S0306-4573(03)00067-0. 3. Cory Lown and Bradley Hemminger, “Extracting User Interaction Information from the Transaction Logs of a Faceted Navigation OPAC,” code4lib 7, June 26, 2009, http://journal.code4lib.org/articles/1633; Eng Pwey Lau and Dion Ho-Lian Goh, “In Search of Query Patterns: A Case Study of a University OPAC,” Information Processing & Management 42, no. 5 (2006): 1316–29, http://dx.doi.org/10.1016/j.ipm.2006.02.003; Heather Moulaison, “OPAC Queries at a Medium-Sized Academic Library: A Transaction Log Analysis,” Library Resources & Technical Services 52, no. 4 (2008): 230–37. 4. William H. Mischo et al., “User Search Activities within an Academic Library Gateway: Implications for Web-Scale Discovery Systems,” in Planning and Implementing Resource Discovery Tools in Academic Libraries, edited by Mary Pagliero Popp and Diane Dallis, 153–73 (Hershey, : Information Science Reference, 2012); Xi Niu, Tao Zhang, and Hsin-liang Chen, “Study of User Search Activities with Two Discovery Tools at an Academic Library,” International Journal of Human-Computer Interaction 30, no. 5 (2014): 422–33, http://dx.doi.org/10.1080/10447318.2013.873281. 5. Eng Pwey Lau and Dion Ho-Lian Goh, “In Search of Query Patterns”; Niu, Zhang, and Chen, “Study of User Search Activities with Two Discovery Tools at an Academic Library.”. 6. Lown and Hemminger, “Extracting User Interaction; Kristin Antelman, Emily Lynema, and Andrew K. Pace, “Toward a Twenty-First Century Library Catalog,” Information Technology & Libraries 25, no. 3 (2006): 128–39; Niu, Zhang, and Chen, “Study of User Search Activities with Two Discovery Tools at an Academic Library.” 7. Xi Niu and Bradley Hemminger, “Analyzing the Interaction Patterns in a Faceted Search Interface,” Journal of the Association for Information Science & Technology 66, no. 5 (2015): 1030–47, http://dx.doi.org/10.1002/asi.23227. 8. Steven D. Zink, “Monitoring User Search Success through Transaction Log Analysis: The WolfPAC Example,” Reference Services Review 19, no. 1 (1991): 49–56; Deborah D. Blecic et al., “Using Transaction Log Analysis to Improve OPAC Retrieval Results,” College & Research Libraries 59, no. 1 (1998): 39–50; Holly Yu and Margo Young, “The Impact of Web Search http://dx.doi.org/10.1016/S0306-4573(99)00056-4 http://dx.doi.org/10.1016/S0306-4573(99)00056-4 http://dx.doi.org/10.1016/S0306-4573(03)00067-0 http://journal.code4lib.org/articles/1633 http://dx.doi.org/10.1016/j.ipm.2006.02.003 http://dx.doi.org/10.1080/10447318.2013.873281 http://dx.doi.org/10.1080/10447318.2013.873281 INFORMATION TECHNOLOGY AND LIBRARIES | SEPTEMBER 2016 25 Engines on Subject Searching in OPAC,” Information Technology & Libraries 23, no. 4 (2004): 168–80; Moulaison, “OPAC Queries at a Medium-Sized Academic Library.” 9. Thomas Peters, “When Smart People Fail,” Journal of Academic Librarianship 15, no. 5 (1989): 267–73; Zink, “Monitoring User Search Success through Transaction Log Analysis”; Rhonda H. Hunter, “Successes and Failures of Patrons Searching the Online Catalog at a Large Academic Library: A Transaction Log Analysis,” Reference Quarterly (Spring 1991): 395–402. 10. Karen Antell and Jie Huang, “Subject Searching Success: Transaction Logs, Patron Perceptions, and Implications for Library Instruction,” Reference & User Services Quarterly 48, no. 1 (2008): 68–76; Hunter, “Successes and Failures of Patrons Searching the Online Catalog at a Large Academic Library”; Peters, “When Smart People Fail.” 11. Peters, “When Smart People Fail.”; Moulaison, “OPAC Queries at a Medium-Sized Academic Library”; Blecic et al., “Using Transaction Log Analysis to Improve OPAC Retrieval Results.” 12. Lynn Silipigni Connaway, Debra Wilcox Johnson, and Susan E. Searing, “Online Catalogs from the Users’ Perspective: The Use of Focus Group Interviews,” College & Research Libraries 58, no. 5 (1997): 403–20, http://dx.doi.org/10.5860/crl.58.5.403. 13. Karl V. Fast and D. Grant Campbell, “‘I Still Like Google’: University Student Perceptions of Searching OPACs and the Web,” ASIST Proceedings 41 (2004): 138–46; Eric Novotny, “I Don’t Think I Click: A Protocol Analysis Study of Use of a Library Online Catalog in the Internet Age,” College & Research Libraries 65, no. 6 (2004): 525–37, http://dx.doi.org/10.5860/crl.65.6.525. 14. Xi Niu et al., “National Study of Information Seeking Behavior of Academic Researchers in the United States,” Journal of the American Society for Information Science & Technology 61, no. 5 (2010): 869–90, http://dx.doi.org/10.1002/asi.21307; Lynn Sillipigni Connaway, Timothy J. Dikey, and Marie L. Radford, “If It Is Too Inconvenient I’m Not Going after It: Convenience as a Critical Factor in Information-Seeking Behaviors,” Library & Information Science Research 33, no. 3 (2011): 179–90; Karen Calhoun, Joanne Cantrell, Peggy Gallagher and Janet Hawk, Online Catalogs: What Users and Librarians Want: An OCLC Report (Dublin, OH: OCLC Online Computer Library Center, 2009). 15. F. William Chickering and Sharon Q. Young, “Evaluation and Comparison of Discovery Tools: An Update,” Information Technology & Libraries 33, no.2 (2014): 5–30, http://dx.doi.org/10.6017/ital.v33i2.3471. 16. William Denton and Sarah J. Coysh, “Usability Testing of VuFind at an Academic Library,” Library Hi Tech 29, no. 2 (2011): 301–19, http://dx.doi.org/10.1108/07378831111138189; Jennifer Emanuel, “Usability of the VuFind Next-Generation Online Catalog,” Information Technology & Libraries 30, no. 1 (2011): 44–52; Erin Dorris Cassidy et al., “Student Searching http://dx.doi.org/10.5860/crl.58.5.403 http://dx.doi.org/10.5860/crl.65.6.525 http://dx.doi.org/10.1002/asi.21307 http://dx.doi.org/10.6017/ital.v33i2.3471 http://dx.doi.org/10.1108/07378831111138189 LIBRARY DISCOVERY PRODUCTS: DISCOVERING USER EXPECTATIONS THROUGH FAILURE ANALYSIS |IRINA TRAPIDO |doi:10.6017/ital.v35i2.9190 26 with EBSCO Discovery: A Usability Study,” Journal of Electronic Resources Librarianship 26, no. 1 (2014): 17–35, http://dx.doi.org/10.1080/1941126X.2014.877331. 17. Sarah C. Williams and Anita K. Foster, “Promise Fulfilled? An EBSCO Discovery Service Usability Study,” Journal of Web Librarianship 5, no. 3 (2011): 179–98, http://dx.doi.org/10.1080/19322909.2011.597590; Rice Majors, “Comparative User Experiences of Next-Generation Catalogue Interfaces,” Library Trends 61, no. 1 (2012): 186– 207; Andrew D. Asher, Lynda M. Duke, and Suzanne Wilson, “Paths of Discovery: Comparing the Search Effectiveness of EBSCO Discovery Service, Summon, Google Scholar, and Conventional Library Resources,” College & Research Libraries 74, no. 5 (2013): 464–88. 18. Jody Condit Fagan et al., “Usability Test Results for a Discovery Tool in an Academic Library,” Information Technology & Libraries 31, no. 1 (2012): 83–112; Megan Johnson, “Usability Test Results for Encore in an Academic Library,” Information Technology & Libraries 32, no. 3 (2013): 59–85. 19. Elizabeth (Bess) Sadler, “Project Blacklight: A Next Generation Library Catalog at a First Generation University,” Library Hi Tech 27, no. 1 (2009): 57–67, http://dx.doi.org/10.1108/07378830910942919; Bess Sadler, “Stanford's SearchWorks: Unified Discovery for Collections?” in More Library Mashups: Exploring New Ways to Deliver Library Data, edited by Nicole C. Engard, 247–260 (London: Facet, 2015). 20. Andrew D. Asher, Lynda M. Duke and Suzanne Wilson, “Paths of Discovery: Comparing the Search Effectiveness of EBSCO Discovery Service, Summon, Google Scholar, and Conventional Library Resources,” College & Research Libraries 74, no. 5 (2013): 464–88; Kelly Meadow and James Meadow, “Search Query Quality and Web-Scale Discovery: A Qualitative and Quantitative Analysis,” College & Undergraduate Libraries 19, no. 2–4 (2012): 163–75, http://dx.doi.org/10.1080/10691316.2012.693434. 21. Sarah C. Williams and Anita K. Foster, “Promise Fulfilled? An EBSCO Discovery Service Usability Study,” Journal of Web Librarianship 5, no. 3 (2011): 179–98, http://dx.doi.org/10.1080/19322909.2011.597590; Kathleen Bauer and Alice Peterson-Hart, “Does Faceted Display in a Library Catalog Increase Use of Subject Headings?” Library Hi Tech 30, no. 2 (2012), 347–58, http://dx.doi.org/10.1108/07378831211240003. http://dx.doi.org/10.1080/1941126X.2014.877331 http://dx.doi.org/10.1080/19322909.2011.597590 http://dx.doi.org/10.1108/07378830910942919 http://dx.doi.org/10.1080/10691316.2012.693434 http://dx.doi.org/10.1080/19322909.2011.597590 http://dx.doi.org/10.1108/07378831211240003 ABSTRACT INTRODUCTION REFERENCES