32 Authentic Assessment of Student Learning in an Online Class: Implications for Embedded Practice Jessica Alverson, Jennifer Schwartz, and Sue Shultz* Growth in online course delivery has created challenges and opportunities for aca- demic librarians. Working with an online undergraduate course for adult students, three academic librarians revised their model for embedded practice, focusing on high touch and quality contact with the students. Using a citation analysis of final papers collected over several quarters, we compared sufficiency of sources, quality of sources, selection of sources, and diversity of perspectives in bibliographies created before and after our revised curriculum. Findings show that our added attention to this cohort resulted in improvements across all of these metrics. Introduction According to the 2017 Distance Education Enrollment Report, the number of students enrolled in online courses has been increasing year after year, with more than 6 million students taking at least one online course in 2015.1 The students who are enrolled in online classes, however, aren’t always prepared for the work expected of them. One survey conducted in 2016 found that nearly 60 percent of faculty who engage in online teaching strongly agree that their un- dergraduate students have poor research skills, especially related to finding and evaluating scholarly information.2 In response, teaching faculty have become more reliant on librarians as experts to help students master these skills.3 Embedded librarianship, defined as having professional librarians interact with students directly through the learning management sys- tem (LMS), has been employed as one strategy for meeting the needs of these online students. Instruction librarians at DePaul University have been actively working with students in online courses since 2008. These embedded librarians have noticed firsthand the struggles that many online students experience. With the desire to improve student outcomes, the embedded librarians redesigned their online curriculum in 2013, focusing on a high-touch model that takes inspiration from the Community of Inquiry paradigm in online education.4 As the university has continued to increase online course offerings each year, the library has expanded its embedded program to support additional courses. The increased amount of time that instruction librarians have spent working with online students has sparked conversa- tion about the sustainability of this level of librarian involvement. Feedback from both students Jessica Alverson is Implementation Consultant, ExLibris, Jennifer Schwartz is Humanities and Social Sciences Li- brarian, and Sue Shultz is Business and Social Sciences Librarian, all at DePaul University Libraries; email: Jessica. Alverson@exlibrisgroup.com, jennifer.schwartz@depaul.edu, sshultz@depaul.edu. ©2019 Jessica Alverson, Jennifer Schwartz, and Sue Shultz, Attribution-NonCommercial (http://creativecommons.org/licenses/by-nc/4.0/) CC BY-NC. mailto:Jessica.Alverson@exlibrisgroup.com mailto:Jessica.Alverson@exlibrisgroup.com mailto:jennifer.schwartz@depaul.edu mailto:sshultz@depaul.edu http://creativecommons.org/licenses/by-nc/4.0/ Authentic Assessment of Student Learning in an Online Class 33 and instructors, however, suggests that the redesigned high-touch model is successful and has a positive impact on student outcomes. To more systematically determine whether or not the additional time spent with these students is worth the time lost to other activities, the embedded librarians undertook a formal evaluation of its effectiveness. This study examines how students fared in one online course over the span of several quar- ters. The investigators evaluated whether changes made to the embedded model implemented in Fall 2013 impacted students’ research skills by comparing authentic student work, in the form of bibliographies, from quarters prior to Fall 2013 with those of Fall 2013 through Spring 2014. The course under investigation, Research Seminar, is a required course in the School for New Learning (SNL), the university’s college that serves adult students. The course introduces students to systematic inquiry, research design, and research methods. The final student paper for the course is a proposal for original research, complete with a rigorous research question, literature review, and bibliography. This bibliography, submitted with the final project, is the student artifact under investigation. From 2008 to 2013, librarians supported online Research Seminar students in the follow- ing ways: A librarian was assigned to a specific Research Seminar class in the online learning management system, the librarian monitored and responded to questions that students posted to a discussion board in the course site (labeled “Ask a Librarian”), and the librarians reviewed and graded two research strategy homework assignments. This embedded model provided Research Seminar students with dedicated librarian support during the 10-week course, and personal feedback from the librarian through the discussion board and assignments. In 2013, the instruction librarians revised the model by incorporating even more robust librarian support for students. The following elements were added into the model: posting an introductory video of the librarian, requiring an individual research consultation with each student, and sending weekly emails to the class with instruction tips and messages of encouragement. The two assignments were condensed into one. The research consultation was the most time-consuming new element in the curriculum. Each student met—over the phone, in person, or using the library’s chat reference service—for up to 30 minutes with the librarian. The content of the consultation included discussion of the student’s completed assignment, how to properly scope a research question, how to access the library’s resources, develop an appropriate search strategy, identify scholarly sources and know their value, and find the full text of the information sought. The amount of time cover- ing the different topics was driven by the needs and interests of the student. The investigators analyzed the bibliographies submitted as part of the students’ final pa- pers over 6 quarters, comparing the quality of citations before and after the implementation of the revised curriculum, using an original rubric. The intention was to determine whether the quality of sources used by the students for their research was substantively better after incorporating the high-touch elements of weekly emails, personal video, and the time-intensive research consultation into the curriculum. These activities provided more personalized atten- tion to students who may be ill-prepared with research skills and allow the librarian to create a teaching presence in the online community, an essential part of the Community of Inquiry Model that has been shown to positively impact student learning.5 Engaging in this authentic assessment of student work not only determines if librar- ian interventions can successfully influence student outcomes, it also draws attention to the somewhat invisible work that librarians do in support of this growing demographic of online 34 College & Research Libraries January 2019 students. The evidence from this project should inform embedded programs in other institu- tions and help guide strategic decision-making about library instruction, determining how to allocate staff resources to those activities that are proven helpful for students. Literature Review This research explores not only the most effective role of the embedded librarian in an online course, including the role of the librarian in contributing to teaching presence, but it also assesses learning through citation analysis of student bibliographies. This study, therefore, is situated in the context of two bodies of research. First, it builds upon work detailing best practices for librarians embedded in online courses, including best practices for teaching in the Community of Inquiry model. Second, it is dependent upon research that establishes the value of citation analysis for evaluating the impact of library instruction on student learning. Since the term “embedded” first appeared in the library literature around 2004,6 research- ers have defined its multiple meanings in different contexts. Schulte categorized various defi- nitions of the “embedded” librarian found in the literature, including embedding through the use of social media, with an academic department, or simply providing in-depth research to support a student.7 Embedding in a learning management system, with associated activities like participating in a discussion board, providing links to library resources, and maintaining relevant online subject guides, has become the dominant definition in the scholarship during the past ten years.8 After limiting the meaning of “embedded” to describe only activities in an online class, it can still be difficult to sufficiently explain what the embedded librarian is or does. Hoffman surveyed librarians in six different institutions about their embedding practices and found they consistently struggled to define their duties in an online course. The activities that the librarian was expected to perform were murky, and the teaching faculty at times had different expectations from those of the librarian.9 Drawing from different research studies, Hoffman and Ramin establish a set of best practices for interacting with online students: creating a library module open to each online student, posting the librarian’s contact information in the course, posting in a single library- specific discussion board, posting information proactively, and including visuals in discus- sion board posts.10 Kvenild and colleagues surveyed a range of librarians at an open forum of two professional conferences about the most effective communication habits of librarians in embedded courses and found similar anecdotal advice. Librarians indicated that, when embedded, being proactive and approachable were key. Adding a picture or an introductory video or posting contact information make it more likely for students to view the librarian as a trusted research companion whom one can ask for help.11 Cuthbertson and Falcone stress the importance of including some type of synchronous communication among the students and the instructors to create a sense of community.12 Research that recommends adding visu- als to the course, emphasizing a personal touch, and being proactive have greatly informed this study’s approach to working with this cohort of online adult students. These recommendations echo the larger body of scholarship concerning the Community of Inquiry (CoI) model in online education that recognizes the benefits of establishing a robust teaching presence for online courses. The CoI model, established by Garrison, Anderson, and Archer, argues that a meaningful educational experience is best achieved through collabora- tive discovery, critical discourse, and creating knowledge together.13 The CoI model is based Authentic Assessment of Student Learning in an Online Class 35 on three components: teaching presence, established through a strong teacher-student rela- tionship; social presence, established through the students’ relationships with other students; and cognitive presence, established with the relationship between the student and the class content. Teaching presence is created through good instructional course design, facilitation of student discussion and discourse, as well as direct instruction.14 A study of 2,036 students across thirty-two different colleges supports the importance of teaching presence to establish a sense of community in an online environment, which in turn affects the quality of the online instruction.15 To evaluate the impact on student learning, investigators used citation analysis, which enabled the assessment of authentic student work. Rather than relying on satisfaction surveys or librarian impressions of student learning, evaluating actual student artifacts allows researchers a better view of student performance. A recent example of this type of authentic assessment is the Multi-State Collaborative to Advance Quality Student Learning, a large-scale project involving 80 public two- and four-year institutions across 13 states.16 These researchers use student papers and homework to assess what students are learning, rather than accepting self-reported study habits or standardized tests that are not tied to the curriculum. For librarians, the use of citation analysis to evaluate information literacy instruction has a proven record in the scholarly literature.17 In 1971, Kirk may have been the first librarian to specifically use citation analysis to measure the effectiveness of information literacy, rather than using it as a tool to measure collection usage or for collection development.18 With this type of tool, scholars can determine if students use materials presented in class19 or if they integrate other sources into their research.20 Especially useful for the work here, Kohl and Wilson analyzed student bibliographies to determine what pedagogical approaches are most effective.21 Carbery and Leahy note that citation analysis can point out areas of instruction that require additional attention.22 Library instruction alone, however, doesn’t always improve the quality of a student’s bibliography or promote the use of scholarly journals.23 Evaluating resources, distinguishing scholarly from popular, is often difficult for students, even after instruction from the librar- ian.24 Students’ higher-level thinking skills, like distinguishing between popular and academic sources, were weak in the study by Choinski, Mark, and Murphy.25 Only when the primary instructor ties the use of certain library materials to a grade does the use of scholarly journals increase.26 Sokoloff and Simmons, as well as Reinsfelder, find that incorporating a research consulta- tion into the instruction can improve bibliographies in a significant way.27 These two studies, however, do not involve online students, which is the audience for this research study. By examining the effect of requiring additional personal attention from the librarian in an online class, this project fills a gap in the research landscape. Many libraries rely solely on online tools embedded in the LMS, like tutorials and static webpages, for reaching online students. Clark and Chinburg try to determine whether face- to-face instruction or online learning tools have a stronger impact on student learning. They find essentially no difference.28 York and Vance caution against overextending the librarian’s time commitments for online classes, noting that it is a very labor-intensive endeavor.29 Ac- cording to their survey, most librarians feel they can commit themselves to involvement in no more than five classes.30 Kvenild also finds that librarians are worried about their ability to 36 College & Research Libraries January 2019 meet the increased demand for online instruction.31 Yet another study presents how embed- ding can drive reference interactions up much higher, leading to additional concerns about increased workload.32 It is possible that the personal, time-intensive solution of offering a research consulta- tion to so many students in an online class is not a scalable solution. If this model or another service makes a meaningful impact on student learning, it may be necessary to adjust the focus of the librarians’ time and attention. Services that have declining rates of usage, like the reference desk or others that don’t clearly contribute to student achievement (or other desired outcomes), might need to be let go in favor of services that have been vetted and are proven to be successful. Methodology To assess the impact of the redesigned curriculum on student learning, the investigators analyzed 177 bibliographies from students who completed Research Seminar during six quarters from Spring 2013 through Summer 2014. After obtaining institutional review board approval, students’ final projects were downloaded. The investigators examined the final bibliographies, removed any identifying information, and assigned each a random number. Only students who submitted a final paper were included in the study; those who did not complete the course were not included in the final data set. While the citation dates in the bibliographies were not removed, given the narrow time frame in which all of the classes in the investigation occurred, the investigators are confident that they were not biased by these dates in their analyses of pre- and post-revision. The investigators developed an original rubric to score each bibliography on the follow- ing defined criteria: 1. Sufficiency of sources. Did the students have enough sources according to the re- quirements in the course syllabus and assignment? 2. Quality of sources. Did the bibliographies include enough scholarly material? 3. Selection of sources. Were the sources chosen using library databases or from library- owned magazines and journals? The investigators conducted three separate norming sessions with a sample of the bib- liographies. At the first meeting, the investigators determined they would count the number of citations that adhered to each criteria in the rubric. First, the total number of citations in the bibliography were counted, then the number of citations that were considered scholarly sources, and, finally, the number of citations that were from databases owned by the library. The three investigators completed their scoring/counting independently before working through the examples as a group. Each time there was a disagreement or question (would books be considered scholarly, would the course textbook count toward the total number of citations in the bibliography, if an article might appear both in a library database and freely online, would that count toward the number of library sources, and so on), the investigators worked through the issue until there was an agreement, and detailed notes were added to the rubric to explain the decision for future reference. After the third norming session, inves- tigators felt that they had addressed all of the potential discrepancies through several sample bibliographies and had come to a general agreement about how to categorize each citation. Investigators then randomly divided the bibliographies into three groups, with each librarian totaling the citations according to the rubric categories and scoring the assignment overall. Authentic Assessment of Student Learning in an Online Class 37 Based on knowledge of professors’ expectations, the teaching faculty’s assignment rubric, and by calculating the mode for each criteria, the investigators established thresholds for success for the three criteria. All scores along with their corresponding identifying number were recorded on one master spreadsheet. It was also recorded whether or not the individual associated with the bibliography completed the library assignments, and if they passed or failed. Raw numbers of citations were converted to rubric scores, using a scale from 1 to 5: 1 being the lowest and 5 being the highest. Using spreadsheet software, the investigators compared the pre- and post-curriculum change in students’ scores, and tested for other variables such as whether or not students had completed the library assignments and if that had an impact on scores. Results Investigators examined the data to determine how the curricular changes affected students’ research abilities as recorded in their bibliographies. The 65 bibliographies submitted before Fall 2013 were compared with the 112 bibliographies submitted in the four subsequent quar- ters. In each category, the number of students meeting the established threshold for success increased from pre- to post-curricular changes, both in number and as a percentage of the whole. After converting total numbers to rubric scores, the investigators compared the scores before and after the curricular changes and calculated averages. TABLE 1 Students Meeting Thresholds for Success, Pre- and Post-Curricular Change Rubric Criteria Pre-Curricular Change: Before Fall 2013 (n=65) Post-Curricular Change: Fall 2013 and After (n=112) All Students (n=177) Number % Number % Number % Sufficiency of sources: 15 or more sources 44 68% 87 78% 131 74% Quality of sources: 11 or more scholarly 23 35% 62 55% 85 48% Selection of sources: 11 or more from databases 48 74% 95 85% 143 81% TABLE 2 Pre- vs. Post-Average Scores, by Criteria Criteria Pre-Average Score Post-Average Score Percentage Change Sufficiency of sources 2.9 3.2 +10.4% Quality of sources 2.7 3.2 +18.5% Selection of sources 3.9 4.1 +5.1% Cumulative score 9.5 10.5 +10.5% 38 College & Research Libraries January 2019 For each criterion, there was an improvement in performance post-curricular changes. Overall, average total scores increased by 10.5 percent, post-changes. A t-test was conducted to compare the scores before and after the changes. Results of the t-tests indicated that there were statistically significant differences in scores for students before and after the changes to the curriculum in all areas measured by the rubric. Investigators also looked at the distribution of the scores for each measure by determin- ing how many students scored a 1 or greater, 2 or greater, etc. Each category saw increases in the number of students scoring at the higher end of the scale. FIGURE 1 Score Distribution for Sufficiency of Sources FIGURE 2 Score Distribution for Quality of Sources Authentic Assessment of Student Learning in an Online Class 39 Finally, all scores (pre- and post-) were examined based on whether or not students had completed the library assignments, those who had passed both assignments, those who had passed one assignment, and those who did not complete a research consultation in the revised model. As seen in table 3, students who completed both assignments successfully had higher scores on all of the criteria. Of the 16 students who failed at least one assignment, seven of those didn’t have a re- search consultation with the librarian. Comparing the average rubric score for those seven students without a research consultation to all of the students who had a research consulta- tion (post-revision), the completion of a research consultation correlates to a higher average rubric score (9.9 without a consultation and 10.5 with a consultation). The improvement is most noticeable, again, in the quality of sources category, which improved from 2.7 to 3.2 for the group who had a research consultation. The other criteria had similar scores, or negligible changes. Although the sample size of the group without research consultations is too small to make any sweeping observations, the data begin to suggest that a research consultation does help students adequately evaluate sources. FIGURE 3 Score Distribution for Selection of Sources TABLE 3 Passed vs. Failed Assignments Cumulative Scores Criteria Passed One Assignment, Failed One Assignment (n=16) Passed Both Assignments (n=161) Percentage Change Sufficiency of sources 2.8 3.1 +10.7 Quality of sources 2.6 3.0 +15.4 Selection of sources 3.9 4.1 +5.1 Total Average 9.3 10.2 +9.7 40 College & Research Libraries January 2019 This study only looked at the final projects of students, and only those students who were set to successfully complete the course submitted these projects. Historically, this course has had a high attrition rate, and some students inevitably do not complete the course. It is pos- sible that students who do not complete the course are also the students who do not complete the library assignments successfully, but these students were not part of the study. Discussion Results indicate that curricular changes were associated with better citation choices for each of the rubric measures. The average of the rubric scores increased more than 10 percent, and the quality of sources score, measuring the number of scholarly sources included in the bib- liography, increased by more than 18 percent. This correlation, which associates additional attention to students and improved librarian teaching presence in the online environment with improved research, begins to justify the added expense of time and resources dedicated to this cohort of students. The scores for the selection of sources did not increase as dramatically as the measures associated with quality and sufficiency. Examining established thresholds for success, the investigators speculate that students were already using the library databases to find the requisite number of sources. Efforts, then, were more useful in helping students evaluate the materials they found, rather than pointing them in the direction of the library to retrieve them in the first place. Evaluating sources and distinguishing scholarly from popular can be one of the most difficult skills for students to master. Unfortunately, many students still struggle with recognizing scholarly sources and un- derstanding their value, reflecting the findings in the literature. Even after these curricular changes, approximately half of these students still did not include enough scholarly material in their bibliographies. With the course expectations set to require 11 or more scholarly sources, only 55 percent of students met this threshold for success even after librarian interventions. These were students who were successfully using the library databases, as shown through their scores on the selection of sources, but were still unable to incorporate an appropriate amount of scholarly material in their final projects. Limitations There are limitations of the study, as well as factors that may have influenced the data that were not in the investigators’ control. First, the librarians were embedded in courses with more than five different primary instructors rotating through the classes each quarter. As participants in the class, the librarians were able to observe how often feedback was provided to students on assignments and how much teaching presence the instructor established, and TABLE 4 Rubric Scores Post-Revisions, with and without a Research Consultation Criteria No Research Consultation (n=7) Completed Research Consultation (n=105) Sufficiency of sources 3.1 3.2 Quality of sources 2.7 3.2 Selection of sources 4.1 4.1 Total Average 9.9 10.5 Authentic Assessment of Student Learning in an Online Class 41 found disparities. The quality of the instructor-student interactions, and how that may enhance or impede student success, cannot be adequately measured in this study. In addition, some instructors may have contacted students individually, outside of the course documentation. Faculty instructors may have allowed certain students to include fewer scholarly sources in the bibliography, or fewer sources overall, based on students’ specific topics, without the librarian’s knowledge. However, since the same instructors were teaching both before and after the curricular changes, discrepancies based on their individual personalities should be found evenly throughout the study. Second, since the librarians made several changes at once to the course, it cannot be de- termined with certainty which changes were the most successful. The time-intensive research consultations would seem to be the main driver of student improvement, but this is one of a constellation of services provided to these students. The overall improved teaching presence of the librarian in the online course, which includes the research consultation, must be con- sidered as a whole. Future research might look to just the research consultation to determine its importance as a contributing factor for improved searching and evaluation skills. Third, the methodology employed here entails a quantitative citation analysis; the papers themselves were not examined. The study, then, does not evaluate how well the students may have understood the sources, or judge whether or not they created a coherent thesis statement using these resources. Instead, the investigators were primarily interested in whether the stu- dents had benefited from the librarians’ instruction. The librarians were focused on teaching the students to navigate through the library’s resources and evaluate the articles found for use in a college-level paper. This study looked only at those measurable skills. Conclusions and Implications for Practice While the literature on embedded librarianship has increased in recent years, additional research that assesses the relationship between the role of librarians in online courses and student learning outcomes is needed. This study contributes to the field of academic librarian- ship by addressing this gap using a rubric-based citation analysis of student bibliographies in an online course. This type of assessment is vital for ongoing efforts to strengthen librarian support of students’ academic success. This citation analysis reveals that the redesigned embedded model in this course correlates with increased student success. Students who completed the course post-redesign performed better on all three criteria in the rubric: sufficiency of sources, quality of sources, and selection of sources. While the redesigned model incorporated several changes, investigators speculate that the addition of the required research consultation had the greatest impact. The research consultations occur in weeks 3 through 5 of the course, after students have completed library assignments designed to teach students the necessary skills for locating articles and resources through the library’s website. The research consultations reinforced database selection and search strategies, especially evaluating scholarly sources. This study also contributes to the scholarship on teaching presence by extending the discussion to include librarians as participants in the CoI model developed by Garrison, Anderson, and Archer. While these authors consider teaching presence, course design, fa- cilitation, and direct instruction to be the primary domain of instructors, their model allows for other CoI participants to collaborate in these functions. A high-touch embedded model, which focuses on personalized interactions with students, is a natural vehicle for creating 42 College & Research Libraries January 2019 community; therefore the investigators propose that librarians are participants in the CoI in the online Research Seminar course. The research consultations provide an opportunity for librarians to engage directly with students, surface difficulties they encounter in their research, provide suggestions, and easily respond to any questions they may have. Through these conversations, students became more aware of the scope of the librarian’s role in the course. In other words, students understand librarians as valuable partners in their academic success. In many cases, librarians found that the research consultation was a launching pad for continuous help throughout the course, and beyond. One important implication for practice is that continued growth and sustainability of embedded instruction in online courses depends upon support from library administration. Strategic planning based on authentic assessment of services is important for libraries to suc- cessfully contribute to institutional goals. An embedded program should be supported by library administration because of its contribution to the mission of the university. The model presented here is resource-intensive; because of the significant investment in librarian time, scalability of the model may be limited. This does not mean that the model should not be pursued at other institutions, but rather that libraries should be selective about the programs where the embedded model would have the most impact on student outcomes. Strategically aligning embedded instruction with institutional goals would be more likely to gain the es- sential support from library administration. As more courses and programs shift to online, face-to-face instruction will decrease and librarians will be faced with developing instructional presence for online courses in a meaningful and sustainable way. Librarians must find a way to translate their important work, which is recognizable in face-to-face classes, to the online environment. This study assesses one model of embedded librarians in an online class. Given the variety of embedded models in online courses offered in academic libraries, further research inves- tigating the impact of other models on student learning outcomes is necessary to continue to address the gap in the professional literature. Additionally, based on supposition that re- search consultations have a significant impact on successful student learning outcomes, future research exploring the role of research consultations in student learning in online courses is encouraged. Notes 1. “Distance Education Enrollment Report 2017,” Digital Learning Compass, 2017, available online at https:// onlinelearningsurvey.com/reports/digtiallearningcompassenrollment2017.pdf [accessed July 25, 2017]. Also available: https://onlinelearningsurvey.com/highered.html. 2. Christine Wolff, “Canadian Association of Research Libraries Faculty Survey: Executive Summary of Findings” (Oct. 4, 2016), available online at www.sr.ithaka.org/wp-content/uploads/2016/10/SR_Report_CARL_ Faculty_Survey_20161004.pdf [accessed March 15, 2017]. 3. Wolff, “Canadian Association of Research Libraries Faculty survey”; Christine Wolff-Eisenberg, Alisa B. Rod, and Roger C. Schonfeld, “Ithaka S+R US Faculty Survey 2015” (April 4, 2016), available online at www. sr.ithaka.org/publications/ithaka-sr-us-faculty-survey-2015/ [accessed March 15, 2017]. 4. D. Randy Garrison, Terry Anderson, and Walter Archer, “Critical Inquiry in a Text-Based Environment: Computer Conferencing in Higher Education,” The Internet and Higher Education 2, no. 2/3 (2000): 87–105. 5. Peter Shea, Chun S. Li, Karen Swan, and Alexandra Pickett, “Developing Learning Community in Online Asynchronous College Courses: The Role of Teaching Presence,” Journal of Asynchronous Learning Networks 9, no. 4 (2005): 59–82. 6. Barbara I. Dewey, “The Embedded Librarian: Strategic Campus Collaborations,” Resource Sharing & In- https://onlinelearningsurvey.com/reports/digtiallearningcompassenrollment2017.pdf https://onlinelearningsurvey.com/reports/digtiallearningcompassenrollment2017.pdf https://onlinelearningsurvey.com/highered.html http://www.sr.ithaka.org/wp-content/uploads/2016/10/SR_Report_CARL_Faculty_Survey_20161004.pdf http://www.sr.ithaka.org/wp-content/uploads/2016/10/SR_Report_CARL_Faculty_Survey_20161004.pdf http://www.sr.ithaka.org/publications/ithaka-sr-us-faculty-survey-2015/ http://www.sr.ithaka.org/publications/ithaka-sr-us-faculty-survey-2015/ Authentic Assessment of Student Learning in an Online Class 43 formation Networks 17, no. 1/2 (2004): 5–17, doi:10.1300/J121v17n01_02. 7. Stephanie J. Schulte, “Embedded Academic Librarianship: A Review of the Literature,” Evidence Based Library & Information Practice 7, no. 4 (2012): 122–38. 8. Ibid. 9. Starr Hoffman, “Embedded Academic Librarian Experiences in Online Courses: Roles, Faculty Collabora- tion, and Opinion,” Library Management 32, no. 6/7 (2011): 444–56. 10. Starr Hoffman and Lilly Ramin, “Best Practices for Librarians Embedded in Online Courses,” Public Services Quarterly 6, no. 2/3 (2010): 292–305. 11. Cassandra Kvenild et al., “Embedded Librarianship: Questions and Answers from Librarians in the Trenches,” Library Hi Tech 34, no. 2 (Mar. 2016): 8–11. 12. William Cuthbertson and Andrea Falcone, “Elevating Engagement and Community in Online Courses,” Journal of Library & Information Services in Distance Learning 8, no. 3/4 (2014): 216–24. 13. Garrison et al., “Critical Inquiry in a Text-Based Environment,” 87–105. 14. Ibid., 97. 15. Shea et al., “Developing Learning Community in Online Asynchronous College Courses,” 59–82. 16. Dan Berrett, “The Next Great Hope for Measuring Learning.” Chronicle of Higher Education 63, no. 8 (Oct. 2016): A30-33. 17. See especially: Virginia E. Young and Linda G. Ackerson, “Evaluation of Student Research Paper Bibliog- raphies: Refining Evaluation Criteria,” Research Strategies 13, no. 2 (1995): 80–93; this article provides a thorough review of the literature. 18. Thomas Kirk, “A Comparison of Two Methods of Library Instruction for Students in Introductory Biol- ogy,” College and Research Libraries 32, no. 6 (1971): 435–74. 19. Lara Ursin, Elizabeth B. Lindsay, and Corey M. Johnson, “Assessing Library Instruction in the Freshman Seminar: A Citation Analysis Study,” Reference Services Review 32, no. 3 (2004): 284–92. 20. Lisa Hinchliffe et al., “What Students Really Cite: Findings from a Content Analysis of First-Year Student Bibliographies,” in Integrating Information Literacy into the College Experience: Papers Presented at the Thirtieth LOEX Library Instruction Conference, eds. Julia K. Nims et al. (Ann Arbor, MI: Pierian Press, 2003): 69–74. 21. David F. Kohl and Lizabeth A. Wilson (1986), “Effectiveness of Course-Integrated Bibliographic Instruc- tion in Improving Coursework,” RQ 26, no. 2 (1986): 206–11. 22. Alan Carbery and Sean Leahy, “Evidence-based Instruction: Assessing Student Work Using Rubrics and Citation Analysis to Inform Instructional Design,” Journal of Information Literacy 9, no. 1 (2015): 74–90. 23. Kristina Howard, Thomas Nicholas, Tish Hayes, and Christopher W. Appelt, “Evaluating One-Shot Library Sessions: Impact on the Quality and Diversity of Student Source Use,” Community & Junior College Libraries 20, no. 1/2 (2014): 27–38; Ursin, Lindsay, and Johnson, “Assessing Library Instruction in the Freshman Seminar,” 284–92. 24. Marcia E. Rapchak, Leslie A. Lewis, Julie K. Motyka, and Margaret Balmert, “Information Literacy and Adult Learners Using Authentic Assessment to Determine Skill Gaps,” Adult Learning 26, no. 4 (2015): 135–42; Lorrie A. Knight, “Using Rubrics to Assess Information Literacy,” Reference Services Review 34, no. 1 (2006): 43–55. 25. Elizabeth Choinski, Amy E. Mark, and Missy Murphey, “Assessment with Rubrics: An Efficient and Objective Means of Assessing Student Outcomes in an Information Resources Class,” portal: Libraries and the Academy 3, no. 4 (2003): 563–75. 26. Howard, Nicholas, Hayes, and Appelt, “Evaluating One-Shot Library Sessions,” 27–38; Andrew M. Rob- inson and Karen Schleg, “Student Bibliographies Improve When Professors Provide Enforceable Guidelines for Citations,” portal: Libraries and the Academy 4, no. 2 (2004): 275–90. 27. Jason Sokoloff and Rebecca Simmons, “Evaluating Citation Analysis as a Measurement of Business Librarian Consultation Impact,” Journal of Business & Finance Librarianship 20, no. 3 (2015): 159–71; Thomas L. Reinsfelder, “Citation Analysis as a Tool to Measure the Impact of Individual Research Consultations,” College and Research Libraries 73, no.3 (2012): 263–77. 28. Sarah Clark and Susan Chinburg, “Research Performance in Undergraduates Receiving Face to Face Ver- sus Online Library Instruction: A Citation Analysis,” Journal of Library Administration 50, no. 5/6 (2010): 530–42. 29. Amy C. York and Jason M. Vance, “Taking Library Instruction into the Online Classroom: Best Practices for Embedded Librarians,” Journal of Library Administration 49, no. 1/2 (2009): 197–209. 30. Ibid. 31. Kvenild et al., “Embedded Librarianship,” 8–11. 32. Erika Bennett and Jennie Simning, “Embedded Librarians and Reference Traffic: A Quantitative Analysis,” Journal of Library Administration 50, no. 5/6 (July 2010): 443–57. https://doi.org/10.1300/J121v17n01_02 _gjdgxs