111 Integrating Library Resources in a Learning Management System: Exploring Instructor Obstacles and Motivations Kenneth Haggerty, Caitlin Harrington, and Rachel E. Scott* This usability study investigates how instructors in a university setting integrate re- sources into their Learning Management System (LMS) course shells and explores bar- riers and incentives for doing so. Findings suggest that obstacles encountered while appropriately incorporating licensed library resources in a D2L LMS are substantial. Although many instructors articulate a desire to adhere to copyright guidelines and support the work of librarian colleagues, their methods for providing students ac- cess to assigned and recommended readings are influenced by convenience. These findings highlight the need for clear and easily implemented guidelines for providing access to licensed resources, as well as online platforms that are easily navigated for both students and instructors. Introduction Library personnel are invested in the authorized use of licensed resources. In academic settings, librarians acquire resources needed to support the curricular and research needs of their institu- tion’s faculty, students, and researchers. When instructors do not link to or appropriately inte- grate online library resources into their Learning Management System (LMS) classes, resource usage data will not reflect this usage and librarians will not know that the acquired resources have been assigned and represent a curricular need. Librarians leverage usage and turnaway data to inform library subscription, renewal, and licensing decisions, sometimes deciding to discontinue resources that have little or no usage. Librarians frequently make acquisitions decisions with input from institutional users; it would not, however, be practical for instruc- tors to notify a librarian each time a licensed resource has been assigned and seek guidance on how to integrate it into the LMS appropriately. This study investigates how instructors in a university setting integrate resources into their course shell and describes their obstacles to and motivations for doing so. There are many ways to integrate licensed content in an LMS; these vary according to the LMS used, the preferences of individual instructors, institutional policies, existing library * Kenneth Haggerty is Digital Initiatives Coordinator, Assistant Professor at the University of Memphis; email: khggerty@memphis.edu. Caitlin Harrington is Head of Technical Services and Electronic Resources Librarian at the University of Memphis; email: chrrngt4@memphis.edu. Rachel E. Scott is Associate Dean for Information Assets at Illinois State University; email: rescot2@ilstu.edu.©2022 Kenneth Haggerty, Caitlin Harrington, and Rachel E. Scott, Attribution-NonCommercial (https://creativecommons.org/licenses/by-nc/4.0/) CC BY-NC. 112 College & Research Libraries January 2022 reserves programs, and other criteria. This study employs usability methods to evaluate how faculty members, nonfaculty instructors, and graduate teaching assistants—inclusively referred to as instructors—navigate the process of integrating licensed library content into an LMS course environment. The University of Memphis uses D2L, locally branded eCourseware as the enterprise LMS.1 Testing was conducted using Morae usability software, and additional observations were captured using think-aloud techniques.2 The goal of this research is to re- veal instructor practices and usability concerns related to the integration of licensed library resources into an LMS and to investigate Curriculum Builder, an LMS plugin, as a potential solution to this challenge. Research Questions 1. How do instructors integrate licensed library resources in the LMS? 2. What are instructors’ perceived obstacles to appropriately integrating library resources in the LMS? 3. What motivates instructors to appropriately integrate library resources in the LMS? Literature Review Academic libraries have a long history of supporting course-specific assignments. Reference, general circulating, and specialized library collections support broad and diverse learning opportunities, but course reserves and related collections have long functioned to support the course-specific needs of students and their instructors. Arthur Hamlin traces course reserves in American libraries to the late nineteenth century.3 Physical reserves remained important services in academic libraries through the end of the twentieth century and continue to co-exist with electronic resources in some libraries; Brice Austin highlights early adopters of electronic resources in the early 1990s.4 As use of physical library course reserves waned, electronic reserve use climbed.5 Many authors documented the shift from brick-and-mortar reserve rooms to electronic reserves in the 1990s. For example, Don Bosseau discusses technical, financial, and copyright considerations in the creation of San Diego State University’s Electronic Reserve Book Room.6 Brett Butler considers electronic course reserves in the context of larger shifts toward digital library collections.7 Cindy Kristof documents ARL libraries’ electronic reserves guidelines, policies, and procedures in a 1999 Spec Kit.8 Kymberly Anne Goodson and Linda Fredericksen conducted an environmental scan on electronic reserve models by informally polling both Association of Research Libraries (ARL) and non-ARL institutions. They identified several trends, the most influential of which was LMS—also and previously referred to as CMS (course management system)—integration of licensed library materials: “Perhaps the most significant driver of change, however, has been the rapid adoption by academic institutions across the country of course management systems. CMSs allow for and support a self-service model for creating supplementary course materi- als.”9 The authors acknowledge that some institutions had opted to move electronic course reserve from the purview of the library to self-service models in which instructors would be responsible for integrating licensed content in their own course shell. Several librarians have documented their experiences integrating library content into the LMS, but most of these focus on librarian-generated content such as LibGuide or other subject guides.10 There are also several studies that demonstrate the benefits of embedding librarians in the LMS.11 Steven Shapiro discusses the LMS as a potential marketing tool for Integrating Library Resources in a Learning Management System 113 academic librarians.12 Fewer studies, however, detail the integration of course reserves or licensed library content in the LMS; of these, most discuss the implementation of a vendor- supplied integration tool. Library vendors have created products to support integrating licensed library materials in the LMS. Sharon Ince and John Irwin discuss their implementation of the LibGuides CMS eReserves module in a Blackboard LMS environment at Seton Hall University and indicate that the implementation led to a 142 percent increase in the usage of reserves.13 Ex Libris of- fers Leganto, with which “instructors can create course resource lists [comprising] all types of materials, including print and electronic items from their library’s collections, Open Educa- tional Resources, and other resources available on the web.”14 Blake Galbreath is one of several librarians to have presented on Leganto implementation.15 Olivia Walsby recently published a case study on the University of Manchester’s implementation of Leganto.16 Additionally, EBSCO developed Curriculum Builder, a subscription-based LMS plugin that allows instructors to integrate library resources into their courses. To date, no articles have studied instructor integration of licensed resources into the LMS using Curriculum Builder. Although Lucy Rosenbloom and Jennifer Murray separately discuss the plugin, neither studied teaching faculty engagement with it.17 Although somewhat cumbersome to use for instructors—they must navigate to external learning tools and click multiple times to create a completely discrete reading list—students can somewhat seamlessly access assigned readings from within the course shell. A clear advantage to employing a library product to integrate licensed content into the LMS is that usage data is ensured; if instructors do not know how to generate persistent links and upload PDFs instead, the actual use of the resource will not be accurately reflected in usage statistics. EBSCO recently launched Faculty Select, which like Ex Libris Leganto, “empowers academic libraries to directly support textbook affordability efforts” through a paid service.18 Although LMS plugins are increasingly used for class content delivery, they are not an instructor’s only method for disseminating content. Some instructors, especially those teach- ing in-person classes, opt to refer their students to assigned or recommended content in other ways, which may or may not involve the institutional library. Accordingly, some librarians have taken proactive approaches to identify and acquire the materials instructors assign in their courses. Patrick L. Carr, James D. Cardin, and Daniel L. Shouse used their university’s adopted text list to acquire books that were not actually textbooks.19 Steve Rokusek and Rachel Cooke similarly purchased multiuser e-books in their institution’s list of required social sci- ence texts.20 Some librarians look to interlibrary loan requests to identify titles at the point of need; Emily Riha and Danika LeMay looked to interlibrary loan requests and found evidence that academic libraries best serve students by purchasing required materials as e-books with multiple user licenses.21 The library acquisition of digital textbooks for CMS integration is not a topic that has been extensively covered in the literature, though some libraries have begun to license digital textbooks.22 Several vendors offer digital textbooks subscription products, such as McGraw-Hill’s Access Engineering platform.23 Once an academic library acquires digital content, it can be embedded in the LMS. That does not mean, however, that instructors will choose to do so or that the content will integrate with a given LMS. Several recent studies discuss this in relation to assigned e-books. Cindy Pierard, Vanessa Lynn Svihla, Susanne K. Clement, and Bing-Shan Fazio discuss some concerns surrounding using library-licensed e-books to support course-assigned readings.24 Students 114 College & Research Libraries January 2022 in their two-semester study noted more barriers than affordances to learning with e-books in courses; most frequently cited difficulties included navigation, printing and downloading dif- ficulties, and interface design. Students in Mara Rojeski’s study of e-books integrated into the LMS reported similar challenges.25 Sara Samuel, Paul Grochowski, Natsuko Nicholls, Leena Lalwani, and David Carter noted that the perceived ease of use of e-books varies considerably depending on the platform in which content is provided.26 Nicole Johnston and Neil Ferguson employed usability methods to discover that students did not think that digital features of eTextbooks helped them learn.27 Learning Tools Interoperability (LTI) has facilitated integra- tion of external content in the LMS, but embedding various types of content in the LMS can still cause a variety of problems for a range of reasons. One might question whether course reserves are still necessary and helpful in an age of web-scale discovery. Would students be better served by being provided with a citation and told to find the resource themselves? Should students be pointed to the library’s discovery layer or catalogs to find and vet their own sources, as they will in the “real world”? Instructional design emphasizes the importance of keeping all content within the LMS so students will have a more personalized and seamless experience.28 Scholars have also demonstrated how embedded LMS content enables problematic tracking of more student data.29 The integration of external resources into an LMS has been enhanced by LTI, which facilitates the embedding of a host of resources, including library resources like LibGuides, in the LMS. The authors have found that, although an application using LTI protocol may embed external content in the LMS, it does not necessarily yield a user-friendly experience. This study will investigate the usability of the Curriculum Builder plugin in a D2L environment. Methods Usability testing is a valuable tool to understand the experiences of library patrons. Academic librarians have employed usability to discover, for example, how users search library platforms and what obstacles they encounter in doing so.30 According to the Digital Communications Division of the US Department of Health and Human Services, “usability refers to the qual- ity of a user’s experience when interacting with products or systems, including websites, software, devices, or applications.”31 Usability studies typically aim to measure intuitive de- sign, ease of learning, efficiency of use, memorability, error frequency/severity, and subject satisfaction and employ methods including interviews, surveys, card sorting, focus groups, and task analysis. For the study at hand, interviews and task analysis were both deployed to investigate how instructors integrate library resources into LMS classes and their obstacles to and motivations for doing so. The University of Memphis is an urban, public research university with a Spring 2020 enrollment of 20,245 and a Carnegie classification of Doctoral Universities: Higher Research Activity. The authors—at that time serving as User Interfaces Librarian, Electronic Resources Librarian, and Integrated Library Systems Librarian—met on several occasions before the Spring 2020 semester to plan the study. Planning involved designing tasks, questions, and prompts. Incentives for and recruitment of participants were determined in advance. The User Interfaces Librarian tested the instrument with several library staff and instructors to pilot-test the study. In January 2020, the authors and library liaisons emailed departmental library represen- tatives and personal contacts to recruit University of Memphis affiliates who used eCourse- Integrating Library Resources in a Learning Management System 115 ware as instructors; participants were offered a $25 Starbucks™ gift card. Twenty instructors from various departments responded to the call. Data collection began in January 2020 and concluded in March 2020. The University of Memphis Institutional Review Board indicated that this study did not meet the Office of Human Subjects Research Protections’ definition of human subjects research. Once recruited, the twenty participants read and signed a consent form describing the study and indicating to them that the study involved collecting screen and audio recordings. The authors used Morae Usability Software to collect qualitative and quantitative data as participants went through prescribed tasks. The software was programmed to list the task and specify the starting page for each. As they completed tasks, participants were encouraged to think aloud to share their impressions of the functionality and opportunities for improve- ment as they integrated resources into eCourseware. The researcher was with the participants as they completed tasks, and there were occasions in which the researcher asked follow-up questions pertaining to the participants’ opinions of the experience. The complete survey instrument is provided in appendix A and consists of a pre-survey, six tasks, five follow-up questions, and a post-task survey. The pre-survey established the par- ticipant’s department, title, familiarity with eCourseware, and current practices for providing students access to course readings—for example, by uploading a file to eCourseware, linking out, emailing or sharing files directly, providing citation information, or other means. The six tasks required participants to integrate licensed library content into eCourseware by various means: creating and embedding a persistent link, using the Curriculum Builder plugin, or uploading a PDF file. After each of these tasks, participants were asked to rate its difficulty on a scale of 1 to 5. The researcher asked open-ended follow-up questions to encourage the participants to share their thoughts on the Curriculum Builder interface and its benefits. The questions also prompted participants to discuss their obstacles and motivations to appro- priately integrating library resources in the LMS, as well as the implications of integrating licensed content in eCourseware for library subscriptions and copyright law. After answering follow-up questions, participants completed a System Usability Scale (SUS), a ten item survey that has become an industry standard for measuring usability.32 To enhance the accuracy of the analysis and findings, the authors sought out and incorpo- rated best practices for qualitative data.33 After the completion of data collection, participants were invited to read and answer questions about the accuracy and completeness of a draft to promote participant “collaboration” and “member checking.”34 The authors invited col- leagues inside and outside the library to critique their methods, interpretations, and reporting. The authors also presented their findings to EBSCO product managers for their input with the hope of identifying and resolving some of the problems. The authors then incorporated participant, peer, and vendor feedback back into the manuscript. Results Pre-survey Participants represented a diverse group of disciplines that included STEM, Social Sciences, Humanities, and Arts. To protect participants’ anonymity, their departmental affiliations and titles are presented alphabetically and grouped as appropriate in tables 1 and 2. The twenty study participants represented thirteen different campus departments, with three participants from the History, Communication and Counseling, Edu cational Psychology and Research 116 College & Research Libraries January 2022 departments. While all participants provided instruction using the LMS, their titles varied significantly. A plurality of participants stated “Instructor” as their title, but other position titles included Assistant Professor, Associate Professor, Graduate Instructor, and Career Specialist. Participants were then asked to use a scale of 1 to 5, with 1 being the lowest and 5 be- ing the highest, to rank their familiarity with eCourseware and the tool Curriculum Builder. Overall, eight participants were very familiar (5), six participants were familiar (4), and six participants were somewhat familiar (3) with D2L/eCourseware. Only one participant had heard of Curriculum Builder; nineteen out of 20 participants had no prior knowledge of Cur- riculum Builder. A subsequent study will detail our investigation into external learning tools in the LMS, but initial results indicate that these plugins are often unfamiliar to instructors.35 TABLE 1 Participant Departmental Affiliation Business Information Technology Career Services Communication (3) Counseling, Educational Psychology and Research (3) English (2) History (3) Institute for Intelligent Systems Instruction Curriculum Leadership Physics and Materials Science School of Music Social Work Student Success Programs Theater & Dance TABLE 2 Participant Title Affiliation Assistant Director Assistant Professor (3) Associate Professor (3) Career Specialist Chair & Professor Graduate Teaching Assistant (3) Instructor (5) Post-Doctoral Fellow & Adjunct Professor Professor Research Assistant Professor FIGURE 1 Pre-Task Survey eCourseware (N = 20) Integrating Library Resources in a Learning Management System 117 Finally, participants were asked to share how they currently provide access to course readings in eCourseware. Participants were directed to select all access methods that applied (see figure 1). All twenty participants selected “Upload PDF/other file to eCourseware,” fifteen participants selected “Link to readings in eCourseware,” six participants selected “Email file or share via OneDrive/Dropbox/Google Drive” and “Provide citation information only,” and four participants selected “Email or otherwise distribute a link to the file.” No participants described a method of providing access outside the five included in the survey. Six Tasks The first two tasks asked participants to add persistent links to a predetermined article (Task 1) and e-book (Task 2) to a module in eCourseware, and the third task asked the participants to upload a predetermined article PDF to a module in eCourseware. To complete these tasks, participants had to use the University Libraries’ discovery layer, EBSCO Discovery Services, to locate the articles and e-book. Participants struggled to correctly identify persistent links to complete Task 1. Common issues included copying the digital object identifier (DOI) (three participants) or browser URL (three participants) instead of the persistent link or uploading a PDF (one participant) instead of using a persistent link as directed. Once familiar with persistent links—either after experimentation or after the researcher pointed them out—participants easily identi- fied where and how to generate them in Task 2, which also required embedding a persistent link. Overall, participants had few issues completing Task 3, which entailed finding, sav- ing, and uploading a predetermined article PDF to an eCourseware module. This method of content delivery was selected by all participants in the pre-survey and is apparently a familiar process. The final three tasks required participants to use Curriculum Builder. Screenshots of Curriculum Builder are provided in appendix B. Participants were directed to add a prede- termined article (Task 4), e-book (Task 5), and any article or e-book of their choosing (Task 6) to an eCourseware module using Curriculum Builder. For Tasks 4–6, the Curriculum Builder LTI provided access to the library resources required to complete the tasks, and therefore participants did not use the University Libraries’ discovery interface for these tasks. For all but one participant, who stated they had some familiarity with Curriculum Build- er, Task 4 was the first time participants used the product. As with Tasks 1–3, participants struggled with Task 4 but recovered and had fewer issues with Tasks 5 and 6. Participants experienced various navigation and procedural challenges during Task 4, as they became accustomed to the tool and its interface. For example, since most participants had never incorporated plugins into their courses, many experienced difficulties locating Curriculum Builder despite having been provided with instructions. In addition, the long load times often caused participants to question whether they did something wrong. While completing Tasks 4–6, participants posed insightful questions about the purpose of Curriculum Builder: Is it a tool for assembling bibliographies, such as Zotero/End Note? Is it connected to interlibrary loan? Many participants expressed confusion regarding the Curriculum Builder reading list. They struggled to identify the “Add to Reading List” button within Curriculum Builder and lacked confidence in whether they had completed the task. Because Curriculum Builder is a plugin within eCourseware, some participants were unsure whether they were situated within Curriculum Builder or eCourseware after they completed Tasks 4–6. 118 College & Research Libraries January 2022 Curriculum Builder’s load time was a source of frustration noted by all but two par- ticipants. Regardless of time of day or location of the study, Curriculum Builder’s load time within eCourseware lasted multiple minutes. Participants identified the location of Curricu- lum Builder within eCourseware and its name as two other common complaints. To access Curriculum Builder for the study, participants had to execute a number of steps that are not commonly a part of their activities within eCourseware. That Curriculum Builder is only accessible within External Learning Tools caused confusion for several participants who felt this location, or its label, was misleading or had not previously used External Learning Tools. Follow-up Questions Following tasks 1–5, participants were asked to use a scale of 1–5 to rate how easy it was to complete the preceding task. For task 1, users were asked to embed a persistent link into module 1 and rate how easy it was to complete the task on a scale from 1 (very difficult) to 5 (very easy). Ten participants thought the task was very easy (5). One rated the task very dif- ficult. The average score among the twenty participants was 4.1. Task 2 asked users to embed an e-book link into module 2. It was very similar to task 1; but, overall, users felt it was much easier after conducting task 1. The average score for this task was 4.5. For task 3, users were asked to upload a PDF, which the vast majority of users had done before. The average score was 4.35. Task 4 was the first task in which users were asked to add an article to a reading list using Curriculum Builder. Ratings were expectedly lower with an average of 2.75. Finally, as users learned how to use Curriculum Builder, it became much easier; for task 5 the average score was 3.7. Ratings and averages for all tasks are presented in table 3. Overall, tasks that requested users to embed persistent links and upload PDFs were rated as easier to complete than tasks involving Curriculum Builder. This finding is supported by the higher percentage of participants rating tasks 1, 2, and 3 “very easy” (55%, 65%, 60%, respectively) compared to tasks 4 and 5, where only 5% and 20% rated the tasks “very easy” (see figure 2). After completing task 6, in which users choose an e-book or article to add to a module using Curriculum Builder, participants were asked to verbally respond to the following questions: • What are your thoughts on the Curriculum Builder interface? Under what circumstances do you see yourself using Curriculum Builder? • What benefits do you see with using Curriculum Builder? • Are there any improvements you would make to the plugin? TABLE 3 Task Ratings Task # P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 P16 P17 P18 P19 P20 Average Task 1 5 3 5 5 4 5 5 4 4 5 5 3 5 5 5 4 3 1 2 4 4.1 Task 2 5 5 5 5 4 5 5 5 4 2 5 4 5 5 5 5 4 3 4 5 4.5 Task 3 4 5 5 3 4 5 5 5 4 5 5 5 5 5 5 5 4 1 3 4 4.35 Task 4 2 1 4 3 2 3 4 3 3 3 1 2 5 3 4 2 3 2 2 3 2.75 Task 5 4 3 4 3 3 4 5 5 3 5 3 4 5 4 4 2 4 3 3 3 3.7 Integrating Library Resources in a Learning Management System 119 • Unless instructors link to or integrate content via Curriculum Builder, the library does not know that a particular resource is being used by a course; without usage data, we may not be able to justify renewals or purchases. Does this influence how you might integrate library resources into eCourseware in the future? • Additionally, PDFs posted directly to eCourseware may not be in compliance with copy- right law. Does having this information influence how you might integrate resources into eCourseware in the future? Despite issues with load times and branding issues, participants did not dismiss the potential utility of Curriculum Builder. Many participants stated that they understood the potential value of Curriculum Builder but indicated that they would only use it if certain is- sues were addressed. Participants identified the following as the most critical problems to address: improved organization and ease of use for students; renaming the tool to explicitly reference the library; fewer steps, consistency of the “Add to Reading List” button; and more robust training materials. A majority of participants did not anticipate that they would ever use Curriculum Builder in their eCourseware courses for some of the following reasons: disin- terest in learning new tools due to time constraints, lack of need, opposition to the premise of accessing library resources outside of the libraries’ interfaces, and belief that students would refuse to use Curriculum Builder. The majority of participants stated that copyright compliance and helping the library with usage statistics would positively influence their prospective use of Curriculum Builder in the future. Participants who were not influenced by these benefits stated that they did not go to or use the library very often, that those same usage statistics could be collected by directing students to locate assigned readings via the libraries’ interface, and because it was more im- portant that a tool be user friendly for students than helpful to the libraries. Two participants indicated that they were unconcerned with copyright compliance because they believed authors care more about their work being read than copyright laws. One indicated that they only assign readings that exist in the public domain, and another indicated that, as all course materials are accessed through the libraries’ resources, they should be copyright compliant. FIGURE 2 Ease by Task: Percentage Ratings 1 (very difficult) –5 (very easy) 120 College & Research Libraries January 2022 Post-task Survey A modified System Usability Scale was used in the post-task survey. Participants were asked to score the following 10 items ranging from Strongly Disagree (1) to Strongly Agree (5); re- sults are presented in table 4. The SUS allows researchers to gain a quick but reliable idea of a product or services usability. Since the late 1980s, it has become “an industry standard” and is a default survey when using Morae Usability Software. Morae Manager automatically calculates the SUS score when the SUS is included as part of a usability study. According to usability.gov, a score of above 68 is considered above average.36 For this study, Curriculum Builder achieved a SUS score of 72.63 out of 100 as calculated by Morae Usability Software. Discussion Persistent Links versus Uploading PDFs As indicated by the presurvey results, the most popular method for sharing course read- ings—selected by all study participants—was to upload a PDF or other file into eCourseware. Despite linking to readings being the second most popular choice, many participants struggled to correctly identify the persistent link for the predetermined article in order to complete task 1. Two participants attempted to copy/paste the URL from their browser window, while another participant initially selected the article digital object identifier (DOI) before locating the persistent link. While the DOI is a stable link, it does not include the proxy information required for access outside of University of Memphis’s IP range. Some participants self- reported while completing tasks 1 and 2 that they did not often use EBSCO record tools that include persistent links, or that it had been many years since they used persistent links, and yet they had no issues correctly identifying a persistent link. Their ability to quickly find and use the persistent link generator tool perhaps speaks to an advanced conceptual understand- ing of library tools and resources, or a high degree of comfort with online library resources. TABLE 4 System Usability Scale for Curriculum Builder Item Average Score (1 Strongly Disagree/ 5 Strongly Agree) I think that I would like to use Curriculum Builder frequently 3.5 I found Curriculum Builder unnecessarily complex 1.9 I thought that Curriculum Builder was easy to use 3.75 I think that I would need the support of a technical person to be able to use Curriculum Builder 2.3 I found the various functions in Curriculum Builder were well integrated 3.65 I thought there was too much inconsistency in Curriculum Builder 2 I would imagine that most people would learn to use Curriculum Builder very quickly 4.15 I found Curriculum Builder very cumbersome to use 1.9 I felt very confident using Curriculum Builder 3.85 I needed to learn a lot of things before I could get going with Curriculum Builder 1.95 Integrating Library Resources in a Learning Management System 121 Embedding persistent links from the libraries’ resources provides reliable access to as- signed readings while also making it possible for the library to account for use. Tasks 1 and 2 indicated a disparity between the participants’ behavior reported in the presurvey of linking to course readings, and the evidence of their behavior when asked to add a persistent link to the eCourseware module to complete tasks 1 and 2. In the presurvey, fifteen out of twenty participants stated that they linked to course readings in the LMS. However, the results from tasks 1 and 2 reflect that eleven out of twenty participants struggled to locate the persistent link. The presurvey did not ask participants how they link to course readings, or from where, so it is possible that they are not linking from library resources, but from the World Wide Web. Another reason librarians at the University of Memphis advocate for use of persistent links over uploading files to the LMS is concerns around the accessibility of uploaded content. Vendors are required to vouch for the accessibility of their product, and many of our article database providers provide enhanced accessibility features such as read-aloud and text/file formats designed to work with browser readers.37 Participants in this study were not asked about the accessibility of assigned content and whether that is a consideration in the content delivery methods they choose. How accessibility interacts with instructors’ motivations for and obstacles to integrating content in their courses requires additional research. Helping the Library Of the twenty participants, seventeen (85%) agreed that helping the library with usage statistics would influence them to consider using Curriculum Builder in the future. One participant, however, was not unconcerned with the libraries’ ability to track resources but correctly noted there were other ways for libraries to track student use of resources than using Curriculum Builder. Providing persistent links to assigned texts or providing only citation information are two methods participants are already using to provide access to content in the LMS, while also supporting the libraries’ efforts to track usage statistics. The participants’ interest in supporting the libraries’ goal to improve the documentation of assigned texts through usage statistics indicates an opportunity for instructor education about ways to integrate library resources in the LMS. The most commonly used method for assigning course readings, uploading a PDF/other file to the LMS, does not help the library track usage statistics. While a benefit of Curriculum Builder is its ability to track usage of library resources assigned in the LMS, it cannot be presented as the only, or best, option. In- structors should receive regular reminders about the variety of ways they support tracking usage statistics through their assigned readings with plenty of assistance from the libraries. A limitation of this study was not asking participants to rank the importance of helping the library with usage statistics among other concerns, such as ease of utility for the instruc- tor, perceived ease of use by students, and other considerations. Participants were not asked to elaborate on or qualify the degree to which helping the library would influence their use of Curriculum Builder in the future. Furthermore, the researcher who conducted the study is a librarian, and this may have created an element of social pressure or professional courtesy to express interest in helping the library. Copyright Compliance Most participants were sympathetic toward the protection of works of authorship and ex- pressed a willingness to use Curriculum Builder for the purposes of complying with copyright 122 College & Research Libraries January 2022 law. In many cases, instructors stated that copyright compliance is something they strongly consider when including resources in their classes before learning about Curriculum Builder. However, as displayed in the presurvey, all twenty participants had included resources in their courses by uploading PDFs, which is a method that does not guarantee copyright compliance. There were a few participants who stated that, although they were sympathetic to the author’s rights, they would not be willing to use Curriculum Builder because it was not user-friendly and ease of access for students was more important to them than complying with copyright. Curriculum Builder Load Time, Branding, and Location The post-task SUS survey indicates that users identified potential benefits of the Curriculum Builder product. Participants also expressed considerable frustration with locating the plugin within the LMS and waiting for Curriculum Builder to load. Although Curriculum Builder achieved an SUS score that was above average, its continued use should not be assumed. In order for the University of Memphis libraries to consider next steps with Curriculum Builder, such as creating training materials and robust campuswide promotion, the issues related to excessive load time and branding and location of Curriculum Builder would need to be ad- dressed. These are issues that must be resolved in conversations with the product vendor, EBSCO, and the University of Memphis’s Information Technology Services (ITS) department, who is responsible for the maintenance of the LMS and integration of products within it. The name “Curriculum Builder” belies the purpose of the tool for the end users: instructors and students. Because this product is sold to libraries for the improved use of their resources, the name may have been selected to suit the library audience instead of the end user. However, the purpose of Curriculum Builder must be easily intuited by all relevant parties, librarians, instructors, and students, for widespread adoption. The authors were pleased to learn from EBSCO that Curriculum Builder can be rebranded locally. The location of Curriculum Builder within eCourseware came up throughout the study. Participants referenced a need for fewer steps and a disinterest in using a plugin generally. Within eCourseware, the Curriculum Builder plugin is located under an area labelled “Add Existing Activities.” Four participants reported that they would never check “Add Existing Activities” for a tool related to library resources and assumed only functions related to class activities, such as discussions, lived in that section of eCourseware. This is an issue to be ad- dressed with the LMS vendor, D2L, via the ITS department at University of Memphis. While it is in EBSCO’s best interest to situate Curriculum Builder intuitively within all learning management systems, in reality it is not yet intuitive or well-integrated. Conclusion As the first study to employ usability methods to investigate instructor obstacles to and mo- tivations for integrating licensed library resources in the LMS, this study provides insight into how and why instructors provide access to licensed content in online courses. Providing access to licensed resources through the LMS is a perennial challenge, one in which academic librarians have a vested interest. This study not only investigates the relative utility of the Curriculum Builder tool but also provides rich context surrounding the options instructors have and the choices they make when integrating resources in the LMS. Findings suggest that instructors are interested in supporting their librarian colleagues and providing their students access to required texts in a way that is compliant with copyright laws, even if they may not Integrating Library Resources in a Learning Management System 123 be sufficiently committed to do so in practice. The convenience of providing content directly to students appeals to instructors and often trumps other considerations. This study is also the first investigation of the perceived usability of Curriculum Builder, a product devised to facilitate the integration of licensed library content in the LMS. For a va- riety of reasons—frustration with the load time, branding, and location of the plugin, among others—using this ready-made solution has not been embraced at the University of Mem- phis. Because the library is financially invested in the authorized use of licensed resources, and librarians are charged to be good stewards of these resources, a solution—or at least an improvement—is required. The authors will continue to explore the complex motivations and obstacles to the appro- priate integration of licensed content in the LMS and endeavor to make this process as stream- lined as possible. Most participants were quick to understand how to create a persistent link and the importance of doing so; the authors will pursue education surrounding this process to facilitate increased linking to proxied resources as opposed to sharing PDFs. Conducting this usability study has not only provided insight into instructor practices and usability concerns but has also suggested a potentially viable path forward: creating and embedding persistent links. This process supports copyright compliance, accessibility for users with diverse abilities, ease of access for instructors and students, and resource usage data for librarians. 124 College & Research Libraries January 2022 APPENDIX A Survey Instrument Pre-Task Survey 1. What department are you in? 2. What is your title? 3. On a scale from 1 to 5, how familiar are you with D2L/eCourseware? 4. How do you provide access to course readings? (select all that apply) a. Upload PDF/other file to eCourseware b. Link to readings in eCourseware c. Email file or share via OneDrive/Dropbox/Google Drive d. Provide citation information only e. Other, please describe User begins at home page of the course “Library Curriculum Builder Course” Tasks 1. Add the article “A Qualitative Examination of User Perceptions of User-Driven and App- Controlled Hearing Technologies” into Module 1 by embedding a persistent link to the article. Task Survey 1: On a scale from 1 to 5, how difficult was it to complete this task? 2. Add the e-book Chemistry: The Impure Science into Module 2 by embedding a persistent link to the e-book. Task Survey 2: On a scale from 1 to 5, how difficult was it to complete this task? 3. Add the article “The Implicit Morality of the Market and Joseph Heath’s Market Failures Approach to Business Ethics” into Module 3 by uploading a PDF file. Task Survey 3: On a scale from 1 to 5, how difficult was it to complete this task? 4. Add the article “Big Music Data, Musicology, and the Study of Recorded Music: Three Case Studies” into Module 4 by using Curriculum Builder. Task Survey 4: On a scale from 1 to 5, how difficult was it to complete this task? 5. Add the e-book Film, Theory, and Philosophy: The Key Thinkers into Module 5 by using Cur- riculum Builder. Task Survey 5: On a scale from 1 to 5, how difficult was it to complete this task? 6. Add an article or e-book of your choice into Module 6 using Curriculum Builder during which feel free to share your thoughts and opinions of the plugin. Follow-up questions: i. What are your thoughts on the Curriculum Builder interface? Under what circumstances do you see yourself using Curriculum Builder? ii. What benefits do you see with using Curriculum Builder? iii. Are there any improvements you would make to the plugin? iv. Unless instructors link to or integrate content via Curriculum Builder, Integrating Library Resources in a Learning Management System 125 the library does not know that a particular resource is being used by a course; without usage data, we may not be able to justify renewals or purchases. Does this influence how you might integrate library re- sources into eCourseware in the future? v. Additionally, PDFs posted directly to eCourseware may not be in com- pliance with copyright law. Does having this information influence how you might integrate resources into eCourseware in the future? 126 College & Research Libraries January 2022 APPENDIX B Curriculum Builder Screenshots Integrating Library Resources in a Learning Management System 127 Notes 1. “Brightspace for Higher Education,” D2L, https://www.d2l.com/. 2. “Morae 3,” Techsmith, http://www.techsmith.com/morae. 3. Arthur T. Hamlin, The University Library in the United States: Its Origins and Development (Philadelphia, PA: University of Pennsylvania Press, 1981). 4. Brice Austin, Reserves, Electronic Reserves, and Copyright: The Past and the Future (Binghamton, NY: Haworth Information Press, 2004). 5. Charles Martell, “The Absent User: Physical Use of Academic Library Collections and Services Continues to Decline 1995–2006,” Journal of Academic Librarianship 34, no. 5 (2008): 400–07. 6. Don L. Bosseau, “Anatomy of a Small Step Forward: The Electronic Reserve Book Room at San Diego State University,” Journal of Academic Librarianship 18, no. 6 (1993): 366–68. 7. Brett Butler, “Electronic Course Reserves and Digital Libraries: Progenitor and Prognosis,” Journal of Academic Librarianship 22, no. 2 (1996): 124–27. 8. Cindy Kristof, Electronic Reserves Operations in ARL Libraries A SPEC Kit, SPEC No. 245 (Washington, DC: Association of Research Libraries, Office of Leadership and Management Services, 1999). 9. Kymberly Anne Goodson and Linda Frederiksen, “E-reserves in Transition: Exploring New Possibilities in E-reserves Service Delivery,” Journal of Interlibrary Loan, Document Delivery & Electronic Reserve 21, no. 1/2 (2011): 54. 10. For example, see Aaron Bowen, “A LibGuides Presence in a Blackboard Environment,” Reference Services Review 40, no. 3 (July 2012): 449–68, https://doi.org/10.1108/00907321211254698; Emily Daly, “Embedding Library Resources into Learning Management Systems: A Way to Reach Duke Undergrads at Their Points of Need,” College & Research Libraries News 71, no. 4 (April 1, 2010): 208–12; Britt Fagerheim et al., “Extending Our Reach: Automatic Integration of Course and Subject Guides,” Reference & User Services Quarterly 56, no. 3 (2017), 180–88, http://dx.doi.org/10.5860/rusq.56n3.180; Pamela Alexondra Jackson, “Integrating Information Literacy into Black- board: Building Campus Partnerships for Successful Student Learning,” Journal of Academic Librarianship 33, no. 4 (2007): 454–61. 11. For example, John J. Burke and Beth E. Tumbleson, “LMS Embedded Librarianship and the Educational Role of Librarians,” Library Technology Reports 52, no. 2 (2016): 5–9; Cassandra Kvenild et al., “Embedded Librari- anship: Questions and Answers from Librarians in the Trenches,” Library Hi Tech News 33, no. 2 (2016): 8–11; Stephanie J. Schulte, “Embedded Academic Librarianship: A Review of the Literature,” Evidence Based Library and Information Practice 7, no. 4 (2012): 122–38, https://doi.org/10.18438/B8M60D. 12. Steven Shapiro, “Marketing the Library with Content Management Systems: A Case Study of Blackboard,” Library Hi Tech News 29, no. 3 (April 2012): 10–11, https://doi.org/10.1108/07419051211241859. 13. Sharon Ince and John Irwin, “LibGuides CMS eReserves: Simplify Delivering Course Reserves through Blackboard,” Interlending & Document Supply 43, no. 2 (2015), https://doi.org/10.1108/ILDS-05-2015-0014. 14. “Supporting the Use of Open Educational Resources,” Ex Libris, https://www.exlibrisgroup.com/products/ leganto-reading-list-management-system/open-educational-resources-oer/. 15. Blake Galbreath, “Leganto One Year Later: Successes, Challenges, Next Steps,” presented at ELUNA 2018 Annual Meeting, May 1–4, 2018, Spokane, Washington, http://documents.el-una.org/1697/. 16. Olivia Walsby, “Implementing a Reading List Strategy at The University of Manchester–Determination, Collaboration and Innovation,” Insights 33, no. 1 (2020), https://insights.uksg.org/articles/10.1629/uksg.494/print/. 17. Lucy Rosenbloom, “Curriculum Builder,” Charleston Advisor 19, no. 4 (2018): 5–7, https://doi.org/10.5260/ chara.19.4.5; Jennifer L. Murray, “All About Access: Technical & Public Services Collaboration,” presentation made at the 2019 Florida Library Association Annual Conference Wednesday, May 15, 2019, https://digitalcom- mons.unf.edu/library_facpub/75. 18. “Faculty Select,” EBSCO, https://www.ebsco.com/products/ebsco-faculty-select. 19. Patrick L. Carr, James D. Cardin, and Daniel L. Shouse, “Aligning Collections with Student Needs: East Carolina University’s Project to Acquire and Promote Online Access to Course-Adopted Texts,” Serials Review 42, no. 1 (2016): 1–9. 20. Steve Rokusek and Rachel Cooke, “Will Library E-Books Help Solve the Textbook Affordability Issue? Using Textbook Adoption Lists to Target Collection Development,” Reference Librarian 60, no. 3 (2019): 169–81. 21. Emily C. Riha and Danika LeMay, “Saving Students Money with ebooks: A Cross-Departmental Col- laboration between Interlibrary Loan and Course Reserve,” Technical Services Quarterly 33, no. 4 (2016): 386–408. 22. Dominic Broadhurst, “The Direct Library Supply of Individual Textbooks to Students: Examining the Value Proposition,” Information and Learning Sciences 118, no. 11/12 (2017): 629–41, https://doi.org/10.1108/ILS-07- 2017-0072; Olivia Walsby and Flora Bourne, “Putting Our Students at the Heart of What We Do: eTextbooks at 128 College & Research Libraries January 2022 The University of Manchester,” Against the Grain 31, no. 3 (2019): 16–19, https://against-the-grain.com/2019/06/ v31-3-putting-our-students-at-the-heart-of-what-we-do-etextbooks-at-the-university-of-manchester. 23. “Access Engineering,” McGraw Hill, https://www.accessengineeringlibrary.com/front?implicit-login=true. 24. Cindy Pierard et al., “Undesirable Difficulties: Investigating Barriers to Students’ Learning with Ebooks in a Semester-length Course,” College & Research Libraries 81, no. 2 (2020), https://crl.acrl.org/index.php/crl/article/ view/24330/32161. 25. Mara Rojeski, “User Perceptions of ebooks Versus Print Books for Class Reserves in an Academic Library,” Reference Services Review 40, no. 2 (2012): 228–41, https://doi.org/10.1108/00907321211228291. 26. Sara Samuel et al., “Students, Vendor Platforms, and E-textbooks: Using E-books as E-textbooks,” American Society for Engineering Education 120th Conference (2013), https://www.asee.org/public/conferences/20/papers/7427/ download. 27. Nicole Johnston and Neil Ferguson, “University Students’ Engagement with Textbooks in Print and E- book Formats,” Technical Services Quarterly 37, no. 1 (2020): 24–43, https://doi.org/10.1080/07317131.2019.1691760. 28. Eden Dahlstrom, D. Christopher Brooks, and Jacqueline Bichsel, The Current Ecosystem of Learning Manage- ment Systems in Higher Education: Student, Faculty, and IT Perspectives (Louisville, CO: Educase Center for Analysis and Research, September 2014), http://www.educause.edu/ecar. 29. Kyle Jones et al., “A Comprehensive Primer to Library Learning Analytics Practices, Initiatives, and Pri- vacy Issues,” College & Research Libraries 81, no. 3 (2020): 570–91, https://doi.org/10.5860/crl.81.3.570. 30. Kenneth Haggerty and Rachel E. Scott, “Do, or Do Not, Make Them Think? A Usability Study of an Academic Library Search Box,” Journal of Web Librarianship 13, no. 4 (2019): 296–310, https://doi.org/10.1080/19322 909.2019.1684223. 31. “Usability Evaluation Basics,” US Department of Health and Human Services (2017), https://www.usability. gov/what-and-why/usability-evaluation.html. 32. Aaron Bangor, Philip T. Kortum, and James T. Miller, “An Empirical Evaluation of the System Usability Scale,” International Journal of Human–Computer Interaction 24, no. 6 (2008): 574–94. 33. John W. Creswell and Dana L. Miller, “Determining Validity in Qualitative Inquiry,” Theory into Practice 39, no. 3 (2000): 124–30. 34. Creswell and Miller, “Determining Validity in Qualitative Inquiry.” 35. Kenneth Haggerty, Caitlin Harrington, and Rachel E. Scott, “‘You Have to Burrow Pretty Far Down’: A Usability Study of an External Learning Tool in D2L” (under review). 36. “System Usability Scale (SUS),” US Department of Health and Human Services, https://www.usability. gov/how-to-and-tools/methods/system-usability-scale.html. 37. The University of Memphis policy states: “Contractor warrants and represents that the service and software, including any updates, provided to the Institution will meet the accessibility standards set forth in WCAG 2.0 AA (also known as ISO standard, ISO/IEC 40500:2012) and will be compliant with Section 508 of the Americans with Disabilities Act (ADA),” https://bf.memphis.edu/forms/procurement/agreement.html.