Assessing the Value of Course-Embedded Information Literacy on Student Learning and Achievement 164 Assessing the Value of Course- Embedded Information Literacy on Student Learning and Achievement Sara Davidson Squibb and Susan Mikkelsen Sara Davidson Squibb is Head, User Communication & Instruction, University of California, Merced Library, e-mail: sdavidson2@ucmerced.edu; Susan Mikkelsen is Teacher Librarian, Merced High, (formerly, Instruction & Scholarly Communications Librarian, University of California, Merced Library), e-mail: smikkelsen@muhsd.org. © 2016 Sara Davidson Squibb and Susan Mikkelsen, Attribution-NonCommercial 3.0 (http://creativecommons.org/licenses/by-nc/3.0/) CC BY-NC 3.0. A team at the University of California, Merced, collaborated to evaluate the value of integrating information literacy into introductory composition courses through a curriculum developed by librarians and writing faculty. Using a mixed-methods approach, the team investigated the impact of the curriculum on students’ learning and achievement at the end of their first semester of college. Students participating in the curriculum demonstrated greater gains than their peers in using suitable sources and presenting arguments and multiple viewpoints with evidence. This learning did not translate to higher student achievement as represented by course grades and grade point average. Introduction As educators and librarians at a research university, we are eager to contribute to student learning, especially to students’ information literacy and critical-thinking skills. Often those contributions have been limited to one-shot instruction sessions introducing students to basic research skills and strategies. An exciting collaboration between University of California (UC), Merced, librarians and faculty from the Mer- ritt Writing Program (MWP) focused on a course-embedded approach to information literacy in introductory composition curriculum. This project and curriculum, called TRAIL (Teaching Research and Information Literacy),1 meant that writing faculty in- troduced students to content about the research process and information literacy via activities, readings, tutorials, and reflections before students had in-person instruction with a librarian. Five writing faculty members piloted the first sections of TRAIL in spring 2014, and librarians, with a desire to evaluate the effectiveness of this course- embedded model, applied for the Association of College & Research Libraries’ (ACRL) Assessment in Action (AiA) program.2 Upon acceptance, they formed a campus AiA team to assess the impact of this course-integrated writing and research curriculum on student learning and achievement. Team members included the MWP co-director, an MWP faculty member, and the Director of Institutional Assessment. In addition, the campus’s Principal Research Analyst from Institutional Research and Decision Support doi:10.5860/crl.77.2.164 Assessing the Value of Course-Embedded Information Literacy 165 consulted with the team. The assessment project exhibited some of action research’s key characteristics, including collaboration with others, systematic planning with on- going reflection, a focus on producing change, and a flexible and adaptable practice.3 Research Questions As the team considered how it might design an assessment that could determine the impact of this course-integrated model of library instruction (TRAIL) on students’ learning and achievement, team members articulated research questions. Did TRAIL students who participated in Writing 10 sections with curriculum crafted around in- tegrating the research process with the writing process express the knowledge, skills, and attitudes representative of developing student researchers? Did they demonstrate more developed information literacy skills in research writing than their non-TRAIL Writing 10 peers? Did they demonstrate higher levels of academic achievement, as indicated by writing course grades and overall grade point average (GPA) than their non-TRAIL peers? Literature Review Since the assessment intended to evaluate the impact of the curriculum on student learning and student achievement, librarians on the team consulted the literature discussing the use of rubrics to assess student work and the impact of library instruc- tion on student GPA. Rubrics to Evaluate Information Literacy Proficiencies The literature offers many examples of librarians using rubrics to assess students’ infor- mation literacy proficiency levels4 and evaluating the impact of library instruction on student learning.5 Rubrics have been used to evaluate authentic tasks and are flexible enough to be applied to student work resulting from a variety of assignments includ- ing annotated bibliographies, reflective writing, student responses to a prompt, and papers.6 Librarians often work with faculty and instructors to apply rubrics to student work,7 and these partnerships regularly result in ongoing collaborations with campus partners to improve students’ information literacy skills and ongoing discussions with colleagues to redesign library instruction.8 Library Instruction and Student Achievement Librarians have not only been interested in assessing the impact of library instruc- tion on student learning but also have turned their attention to the impact of library resources and services, including instruction, on student achievement represented by GPA and course grades.9 Articles examining this correlation at or near the end of students’ undergraduate degree have noted a positive correlation between GPA and library instruction.10 Literature is also available investigating similar correlations at the end of students’ first year or first semester.11 While some findings point to positive relationships between students’ exposure to library interactions, including library instruction, and GPA, differences tend to be small or mixed.12 Design and Methods With research questions articulated, literature consulted, and local expertise available, the AiA team discussed what evidence to collect. By August 2014 the team had outlined a mixed-methods approach to the assessment, using both qualitative and quantitative data representing indirect and direct evidence. In keeping with action research practice, the team used techniques from different types of research and included a significant qualitative piece. The main pieces of evidence collected included student reflections 166 College & Research Libraries March 2016 (TRAIL only), faculty debriefs (TRAIL faculty), final papers (TRAIL and non-TRAIL), final Writing 10 course grades (TRAIL and non-TRAIL), and GPA at the end of the first semester (TRAIL and non-TRAIL). Lines of Evidence, Instruments, and Hypotheses To answer the research questions, the team identified lines of direct and indirect evi- dence to collect, prepared instruments to evaluate the evidence, and formed hypoth- eses. To determine if students were expressing the knowledge, skills, and attitudes representative of developing student researchers, as expressed in the first research question, team members chose to evaluate both TRAIL students’ final reflections (ap- pendix A) and their final papers, along with TRAIL faculty observations of student learning. The indirect evidence of TRAIL students’ reflections would provide insight into their attitudes and knowledge about the research process, while final papers, a form of direct evidence, would show the extent to which TRAIL students used suitable sources and developed arguments and counterarguments supported with evidence. The TRAIL faculty debrief would also serve as a piece of indirect evidence, giving their perspective on the impact of the curriculum on student learning. To evaluate final student reflection responses, the team librarians crafted a rubric (see appendix B). Due to the unique nature of the final reflection questions, the librar- ians did not rely on existing rubrics other than exploring the language used to identify performance levels (e.g., advanced, satisfactory, etc.). Each sub-scale of the rubric aligned with one of the six questions in the reflection. Students could score up to 4 points on each question for a maximum score of 24. Team librarians also created a second rubric (see appendix C) to evaluate student papers after examining existing rubrics.13 They identified four criteria most closely aligned with the information literacy emphasis of the TRAIL curriculum and the Writing 10 course outcome of being able to work with evidence. They created descriptors that could be applied to final papers that did not have the same assignment prompt. The initial rubric draft underwent significant revi- sions based on feedback from MWP assessment subcommittee members. This article focuses on examining the two criteria most closely tied to the TRAIL curriculum: source suitability, and argument and evidence. To obtain TRAIL MWP faculty feedback about the effectiveness of this curriculum in developing students’ research skills, knowledge, and attitudes, team librarians created a short debrief survey. The team set high targets, expecting that 80 percent of the TRAIL students would score at Developing or Advanced levels on all six criteria outlined in the reflection rubric and would score at Developing or Advanced levels in the final paper rubric for Source Suit- ability and Argument and Evidence criteria. The team also anticipated that all participating writing faculty would observe that TRAIL students were more likely to demonstrate the characteristics of emerging researchers than students they had taught in the past. In order to address the research question that asked whether or not TRAIL students had more developed information literacy skills in research writing than their non- TRAIL peers, the team chose to examine final papers from both TRAIL and non-TRAIL students. In addition to the 40 papers selected from freshman TRAIL students taking Writing 10 for the first time, another 40 papers would be collected from each of two other groups: non-TRAIL students who had received library instruction in Writing 10 and non-TRAIL students who had not received library instruction in their Writing 10 course. This would provide a total of 120 papers and a comparison between three groups. The team also asked whether TRAIL’s focus on research skills would result in higher levels of performance in writing course grades and GPAs in contrast to Writ- ing 10 students who had not received formal library instruction and those who had Assessing the Value of Course-Embedded Information Literacy 167 received research support through one-shot instruction sessions. To investigate this question, Writing 10 course grades and GPAs would be acquired and compared for the same 120 students from whom final papers were collected. The team anticipated that TRAIL students would perform better in Writing 10 than their non-TRAIL peers since they would be more adept at finding, evaluating, and using information, which in turn would significantly improve their research writing performance. The team predicted that students in non-TRAIL sections with library instruction would perform nominally better than non-TRAIL students who had no formal library instruction in Writing 10 since some exposure to library instruction about the research process and available resources would translate into stronger papers and grades. Lastly, the team expected TRAIL students to have higher GPAs at the end of the fall semester than their non-TRAIL peers. Though studies, mentioned in the literature review, that investigated the impact of library services and resources on freshman student GPA found statistically small positive relationships or mixed results requiring more examination,14 their models of library instruction were not as extensive as the TRAIL curriculum. The embedded nature of the TRAIL cur- riculum suggested greater promise of positively influencing student achievement as represented by GPA. Data Collection and Analysis In anticipation of data collection and analysis, team members outlined in the protocol how they would collect, code, and store data in order to ensure confidentiality. They created a crosswalk document from class lists and assigned each student a unique identifier. When obtaining reflections or final papers, librarians removed identifying information, assigned a unique ID, and uploaded redacted documents to a restricted- access folder. Writing 10 course grades and GPAs for fall 2014 from the randomized sample of 120 students were sent directly from the Registrar to the Principal Research Analyst for analysis. Evaluation of evidence started in January 2015, with three librarians applying a rubric to 157 TRAIL student reflections. Librarians assessed all student reflections and did not limit their analysis to reflections from 40 TRAIL students whose final pa- pers were selected. Following a norming process, two librarians read each reflection and completed a rubric evaluation independently before completing a rubric based on consensus. While the six questions of the reflection acted as themes, such as the transferability of research skills, more specifics about these themes emerged. Apply- ing a rubric translated students’ written reflections into an overall score, yet it did not reveal or quantify specific student experiences, such as how often they used their newly acquired research skills in other courses. Though coding was not part of the original analysis plan, one librarian coded responses from three of the six questions from each final reflection to expose this additional content. The librarian informally started to build codes based on themes recognized during the initial rubric assessment but confirmed these codes by rereading some student responses. Once the codes were confirmed, the librarian applied them to all student reflections and then reread the responses to check for code accuracy and consistency; this work was not verified by a second reader. Though coding was influenced by the predetermined themes of the reflection questions, it relied on an inductive approach as more details emerged in student responses. While librarians evaluated the students’ reflections, writing faculty evaluated students’ papers. The co-director of the MWP, an AiA team member, offered to make the evaluation of final papers part of that year’s MWP assessment project. While the co-director of the MWP coordinated the participants for each session, the team 168 College & Research Libraries March 2016 librarians organized the materials and led the norming process for all eight sessions in spring 2015. Once the norming process was completed, MWP faculty participants formed pairs. Each pair received the same four redacted papers. They were instructed to read papers separately and to score individually. Following this individual review, they were to discuss their scores with each other, resolve any differences, and fill in a consensus rubric. In spite of directives to follow this protocol, the librarians did notice that some pairs may not have firmly locked in individual decisions before discussing with a colleague. While involving over 30 individuals in assessment presented some norming and protocol challenges, the team highly valued MWP faculty’s expertise in the evaluation of students papers, and their participation made it possible to have 120 papers double-read. Librarians provided paper scores for each criterion to the Principal Research Analyst. He used this information, along with student grades for Writing 10 and GPAs for fall 2014, to conduct a one-way Analysis of Variance (ANOVA), an analysis allowing the comparison of more than two groups. He calculated the means for each group for the rubric criteria, course grades, and GPAs and then tested the differences between these means for any statistical significance. The process for evaluating results from the faculty debrief survey was much simpler. The survey tool had a reporting feature that allowed the librarian team lead to easily generate and edit a report of the survey results. Evolving Quantitative Design Though the data collection and analysis looked linear and well-developed from the outset, the quantitative portion of the design was undergoing refinement as the action research project progressed. In this way, the action research process was emergent. These refinements were largely prompted by a conversation with the campus’s Principal Research Analyst about the best course of action for randomly selecting students whose papers would then be evaluated. Since statistical analysis was not an area of strength for the team librarians, they appreciated the advice and direction of the Principal Research Analyst and relied on his expertise. Through his participation, librarian team members developed additional knowledge about research design. He worked with them to outline a strategy that took into consid- eration some of the constraints under which they were working, such as knowing that only 120 final papers, 40 from each group, could be evaluated. With a limited number of papers to be evaluated, how might noise in the data be minimized? The Principal Research Analyst suggested narrowing the group of students from whom final papers would be sampled to those who were taking Writing 10 for the first time in fall 2014. This approach would focus on a higher functioning group of freshmen by removing students who had taken Writing 1, a preparatory course for Writing 10, and students who were retaking Writing 10. Results Student Learning—Student Reflections For four of the six criteria in the final reflection, over 50 percent of TRAIL students (n = 157) scored at Advanced or Developing. TRAIL students performed highest (59.9 percent) when answering a question about transferability and lowest (15.3 percent) when addressing the question regarding research challenges (see figure 1). Student scores did not reach the target outlined in the hypothesis, which anticipated 80 percent of the students scoring at Advanced or Developing for all six criteria. For total reflection rubric scores, 17.2 percent of students scored at Advanced (19–24), with the greatest number of students, over 53.5 percent, scoring at Developing (13–18). Assessing the Value of Course-Embedded Information Literacy 169 Another portion of students (27.4 percent) scored at Emerging (7–12), with a much smaller number (1.9 percent) scoring under 6 points (see figure 2). Minimal coding of some prompts provided additional findings. When librarians examined students’ responses about changes in their confidence levels, the majority of students (88.5 percent) indicated that they were more confident in their abilities to do research after taking Writing 10 (TRAIL) as represented by a student who wrote, “Not only do I feel more confident now but also I believe I will enjoy writing another research paper.” Of that group, 32.5 percent indicated that there was a big change in their confidence level in doing research, as expressed by a student who stated, “My attitudes, perceptions, and confidence level about doing research have definitely changed over the semester…. Now I have confidence to engage and apply the entire research process.” For the question about transferability, 45.2 percent of the students wrote that they were able to use information they learned about the research process from Writing 10 in at least one other class they were taking during the same semester. Yet 42.0 percent of FIGURE 1 Scoring of Student Reflections FIGURE 2 Distribution of Scoring TRAIL Student Reflections 170 College & Research Libraries March 2016 TRAIL students noted that they did not use the information about the research process outside of their writing class during the semester, and of those, some explicitly stated that they did not have other assignments with research requirements. The remaining percentage (12.7 percent) did not clearly indicate whether or not they were able to use the information they had learned in other courses. Almost 80 percent of students anticipated using what they learned about the research process in future classes, with just over 20 percent of students either uncertain or not stating if they anticipated using what they learned about the research process in future classes. Student Learning—Faculty Observations TRAIL MWP faculty, five in total, who incorporated the research process with the writing process in fall 2014 completed a debrief survey at the end of the semester. When asked to reflect on the work of TRAIL students, four of the five MWP faculty responded affirmatively that students in the TRAIL sections thought and wrote more like emerging researchers than students in previous non-TRAIL classes. One of the five MWP faculty responded with “unsure.” The team had hypothesized that all writing faculty would respond affirmatively. More specifically, MWP faculty were asked if TRAIL students demonstrated certain attitudes or behaviors to a greater extent than students they had previously taught in introductory writing classes. All five MWP faculty responded that students in TRAIL classes were more able to engage with research as an ongoing process than previous students. When asked to reflect on students’ ability to demonstrate persistence in information finding, write strong research questions, and select suitable resources for their assignments, four of the five writing faculty observed that TRAIL students did so more successfully than previous students, while one writing faculty member ob- served no discernable differences. Concerning students’ ability to incorporate multiple viewpoints, three of the five writing faculty indicated that their TRAIL students did incorporate multiple viewpoints in their work more than previous students (see table 1). TABLE 1 Faculty Observations about TRAIL Student Learning Based on your observations, did your students… Yes, More So Than Previous Students No Discernible Differences No, Less So Than Previous Students Engage with research as an ongoing process? 5 0 0 Demonstrate persistence in information-finding? 4 1 0 Write strong research questions? 4 1 0 Select suitable resources for their assignments? 4 1 0 Incorporate evidence from multiple viewpoints? 3 2 0 Assessing the Value of Course-Embedded Information Literacy 171 Student Learning—Final Papers While faculty observations provided a professional practitioner perspective on student learning and student reflections gave valuable insight into student thinking about the research process, papers provided direct evidence of what students were able to do (see figures 3–6). In the area of source suitability, 45.0 percent of TRAIL students performed at the Advanced level, with another 27.5 percent performing at the Developing level. Overall, 72.5 percent of TRAIL students scored at Advanced or Developing, almost meeting the hypothesis that 80.0 percent of TRAIL students would score at the upper two perfor- mance levels. The remainder of the TRAIL students scored at Emerging (10.0 percent) or Marginal (15.0 percent) or did not qualify for the lowest performance level (2.5 percent). FIGURE 3 Source Suitability—Final Paper Scores FIGURE 4 Source Suitability—Final Paper Scores Combined 172 College & Research Libraries March 2016 For the criterion of argument and evidence, TRAIL students did not perform at the levels targeted. A small percentage of TRAIL students scored Advanced (7.5 percent), and another 27.5 percent scored Developing, for a total of 35.0 percent scoring in the upper two performance levels. The majority of the students scored in the bottom two levels (65.0 percent), with 35.0 percent at Emerging and 30.0 percent at Marginal. Student Learning—Comparing Final Paper Results To compare the performance of TRAIL students to their peers who had not received TRAIL instruction, additional papers from non-TRAIL students with library instruc- tion and non-TRAIL students without library instruction were assessed using the same rubric applied to TRAIL student papers. FIGURE 5 Argument and Evidence—Final Paper Scores FIGURE 6 Argument and Evidence—Final Paper Scores Combined Assessing the Value of Course-Embedded Information Literacy 173 An ANOVA allowed for a comparison of the means for two criteria between the three groups (see figure 7). Before the ANOVA was conducted, the campus’s Principal Research Analyst ensured that the three groups of first-time freshmen had baseline equivalence. For both criteria, mean student scores differed significantly among the three groups with a p-value of 5 percent (0.05). In the area of source suitability, the average score for TRAIL students was 3.03 versus 2.83 for the non-TRAIL with library instruction group and 1.98 the non-TRAIL without library instruction group (p = .000). A similar pattern, but with lower mean scores, was observed for the argument and evidence criterion; the average score for TRAIL students was 2.15, compared to 1.80 for the non-TRAIL with library instruction group and 1.35 for the non-TRAIL without library instruction group (p = .000). Student Achievement—Writing 10 Course Grades and GPA The ANOVA was also used to compare the means of Writing 10 course grades and overall GPA in fall 2014 for each of the three groups. Between the three groups there was no statistically significant difference (p = 0.961) in the mean grades students received in their introductory writing course. Writing course grade means ranged from a high of 2.97 for the non-TRAIL library instruction group, through 2.96 for the non-TRAIL group with no library instruction, to the lowest—2.94—for the TRAIL group. There was a statistically significant difference (p = 0.042) between the GPA students achieved during the semester, with the TRAIL students performing at the lowest level of the three groups at 2.57, the non-TRAIL library instruction group scoring the highest at 2.79, and the non-TRAIL group with no library instruction scoring in the middle with 2.65. Discussion, Limitations, and Next Steps The evaluation of student reflections, final papers, and faculty observations point to a positive relationship between the TRAIL curriculum and student learning even though the findings did not reach the targets outlined in the hypotheses. These targets would have benefited from more reflection and refinement in the initial design. In retrospect, they look overly ambitious, especially for a freshman population who are novice learn- ers of information literacy.15 FIGURE 7 Means—Final Paper Rubric Sub-scales 174 College & Research Libraries March 2016 Student reflections provided the team with insight into students’ struggles and successes during Writing 10 as they strove to develop as researchers who could suc- cessfully work with evidence. Though it was not the case that 80 percent of TRAIL students scored at Advanced or Developing for all six questions in the reflection, over 50 percent of students did score at Advanced or Developing for four of the six question prompts. While student scores were drastically lower for the question addressing re- search challenges, with only 15.3 percent at Advanced or Developing, librarians on the team believed this was due to the rubric score wordings. The rubric required students to have reflected on a minimum of two research challenges and how they overcame them in order to receive a score of 3 (Developing) or 4 (Advanced). While many students eloquently expounded on a single research challenge and their strategies for overcom- ing it, they were not eligible for a score of more than 2 because they only reflected on one challenge. The team found limitations in its original decision to score student reflections using a rubric. Though this approach was useful, to some extent it tended to mask the rich- ness of the evidence—what students were thinking, attempting, and changing as they researched. Comments from students scoring in the Advanced range (19–24) expressed the type of thinking and research practices team members hoped to see as a result of the TRAIL curriculum. But at the same time, students who scored in the Developing range (13–18) still wrote statements indicating that they too had made progress in de- veloping the knowledge, skills, and attitudes of student researchers. In future projects with reflections as a form of evidence, librarians from the team would plan to code the students’ text as a preferred form of analysis over a rubric. It was encouraging to see the majority of students score in the upper two performance levels for source suitability on their papers. Though some students were Emerging (10.0 percent) in their use of suitable sources, other students struggled with this skill, as indicated by scores of 1 and 0 (17.5 percent). These students may have had difficul- ties in the information-finding process or the evaluation of information, or they may have lacked the desire to seek out appropriate information sources. TRAIL students found it much more difficult to support their arguments and counterarguments with evidence than to use suitable sources. While 35.0 percent of them did score in the upper two performance levels for argument and evidence, the majority of the students (65.0 percent) performed in the lower two levels. The results suggest that first-year college students in introductory composition courses still need much ongoing preparation and practice in the areas of selecting suitable sources and, particularly, in using evidence in argument. Ideally, opportunities to hone these critical-thinking skills would extend beyond writing courses. Giving voice to MWP faculty expertise was an important piece of this action research project due to their involvement in co-creating the curriculum and, most importantly, integrating the readings, activities, and reflections about the research process into their classes. TRAIL MWP faculty also observed positive outcomes of the TRAIL curriculum on student learning. The majority of them (four of five) thought that TRAIL students were thinking and writing more like researchers than students they had taught in previous introductory composition courses. In terms of specific observations, two writing faculty did not find that TRAIL students were better able to incorporate evidence from multiple viewpoints than composition students they had taught in the past (see table 1). Their observations suggest that TRAIL students found this outcome challenging to meet, and final paper findings confirm this. Taken together, the evidence implies that student competencies related to incorporating evidence from multiple viewpoints requires more instructional time and attention. Assessing the Value of Course-Embedded Information Literacy 175 Not only did the team investigate the extent to which TRAIL students were able to meet performance standards outlined in the final paper rubric, but they also examined how TRAIL students performed in relationship to other freshman students taking Writ- ing 10 for the first time. A comparison of TRAIL and non-TRAIL students revealed that TRAIL students were more adept than their non-TRAIL peers at using suitable sources and presenting argument and multiple viewpoints with supporting evidence. They had greater mean scores in those areas than their peers who had received in-person library instruction in their Writing 10 class and to a much greater extent than their peers who did not have any type of instruction from librarians during their Writing 10 course. Students in all three groups were more successful in using suitable sources than they were in presenting arguments with evidence, a much more complex skill. Though this assessment focused primarily on evaluating the performance of the TRAIL cohort, it was interesting to note that non-TRAIL students who received library instruction were more likely to score at higher performance levels for suitable sources and for argument and evidence criteria than their non-TRAIL peers who had not received library instruction. This points to a couple of possible interpretations. One interpretation may be that both forms of instruction, whether the embedded TRAIL curriculum or one-shot sessions, have value, though the embedded information literacy curriculum points to greater gains. It may also suggest that faculty who involve librar- ians, at varying levels, in their curriculum place more emphasis on research skills and process than those who do not. While findings pointed to the benefits of the TRAIL curriculum on student learn- ing, they did not show evidence of a positive relationship with student achievement as represented by writing course grades or GPA. The minimal differences between Writing 10 course grade means between the three groups (0.034) were not statistically significant. Librarians may have had unrealistic expectations about the extent to which research knowledge, skills, and attitudes are valued, included, and graded in both writing courses and other introductory freshman courses. Though TRAIL students were working on both the research and writing process during Writing 10, the grading may not have reflected the gains in research skills as much as students’ performance in writing. While TRAIL students scored better than their peers on two criteria on the final paper rubric, students would have been evaluated on more criteria than source suitability and argument and evidence when their final papers were scored. This aligns with a finding from librarians from the New Jersey Institute of Technology who participated with humanities faculty in a joint assessment of final papers from senior capstone courses. While they found that both information literacy rubric scores and writing rubric scores had positive relationships with both course grade and GPA, the writing score relationship exhibited a stronger correlation, which suggested to them that “the concept of information literacy is not yet a significant factor used in grading by individual writing instructors.”16 Though differences between the means for the three groups in the area of Writing 10 course grades were not statistically significant, the differences in means for GPA were statistically significant, with the TRAIL group having the lowest mean of the three groups. Though this shows relationship, it does not show causation. Since the three cohorts included only first-time freshmen, the GPA mean for each was based on one semester of study at the university. Again, the extent to which research skills were evaluated and emphasized may be at play in overall GPA. As noted in the student re- flections, many TRAIL students did not have the opportunity to use their knowledge of the research process and resources in other coursework. Specifically, 57.1 percent of TRAIL students who turned in the final reflection and whose papers were scored (n = 35) wrote that they did not use information about the research process from Writing 10 176 College & Research Libraries March 2016 in any other classes in the fall semester. One might extrapolate that students’ research skills would not have been regularly evaluated in other courses and, therefore, may be minimally represented in student GPA. It bears pointing out that this group of students who were low-performing in terms of GPA still outperformed their peers in the areas of source suitability and argument and evidence in their Writing 10 papers. Future Research The team has been interested in the long-term impact of the curriculum and to what extent TRAIL students might transfer their knowledge, attitudes, and skills about the research process to other information-finding needs and research writing experiences. Librarians on the team may pursue an additional amendment to the protocol in order to interview a selected number of TRAIL students or to review upper division writing samples in the next two years to discover more about their ongoing research practices. This would also be an opportunity to determine if their research competencies are continuing to be developed through information literacy instruction and practice in other classes. Action Implementation of the TRAIL curriculum has resulted in presentations to MWP faculty and the campus community through an assessment symposium. Librarians on the team had the opportunity to present both the curriculum and the initial assessment findings to the Undergraduate Council (UGC), composed of faculty and administrators from across the university. This meeting prompted a rich discussion about the need for information literacy curriculum that focuses on critical thinking and not merely on the mechanics of searching. Audiences such as the UGC have seemed most interested in this assessment when we have highlighted student statements from their reflective writing that suggest growth in students’ research attitudes and behaviors. The TRAIL curriculum has continued to be used, with modifications, by Writing 10 faculty, who found value in the added information literacy components. Promising assessment results have garnered support from library and writing program leaders, who have provided opportunities for additional faculty training sessions. Two two-hour workshops for faculty were conducted during the spring 2015 semester to acquaint faculty with selected TRAIL lessons and assignments.17 In summer 2015, additional writing faculty were offered a stipend to attend a two-day intensive training workshop that covered the entire TRAIL curriculum. This training occurred in August, and there is a commitment to offer the same two-day training in January 2016 for writing faculty interested in using the curriculum with future Writing 10 sections. Ultimately, the cur- riculum could be formally adopted for Writing 10. Conclusion This assessment project has been characterized by action research’s focus on collabora- tion with others, systematic planning, a focus on producing changes, and a flexible, adaptable practice. A team of five along with many campus constituents—including numerous writing program faculty and the campus Principal Research Analyst—were involved in designing the study, assessing evidence, and analyzing data. This involved significant planning to try to ensure that all parts of the assessment were completed in accordance with the protocol. Adaptations were made during the process in response to feedback and additional information about how to best pursue the qualitative analysis. The primary goal has been to determine the impact of the TRAIL curriculum in order to encourage adoption of a curriculum that integrates the writing and research process into introductory composition classes. Assessing the Value of Course-Embedded Information Literacy 177 The assessment has included lines of indirect evidence that value student and faculty voices as well as direct evidence of authentic student work to reveal student successes and struggles. While the lines of evidence point to student learning, they also confirm the need for repeated and intentional efforts to teach students higher-level critical- thinking skills, such as the ability to support arguments with evidence. A comparison of final papers suggests that students benefited from the TRAIL curriculum in that they demonstrated more developed information literacy skills in research writing than their non-TRAIL Writing 10 peers—as indicated by their use of suitable sources and presenta- tion of arguments and multiple viewpoints supported with evidence. However, TRAIL students did not demonstrate higher academic achievement, as represented by their Writing 10 course grades or their fall semester GPAs, than their non-TRAIL peers. Yet for the two criteria assessed in final papers, TRAIL students scored higher than their peers even though they were the lowest performing group in terms of overall GPA. Since the findings indicate that the TRAIL curriculum did positively impact student learning, the library continues to invest in this model. Of particular interest is how this learning may transfer into upper division writing experiences. Overall, the action research assessment has been a valuable experience and, in particular, has provided insight into students’ research practices, generated conversations at the campus level about students’ information literacy proficiencies, and secured momentum for ongo- ing collaboration with the writing program to integrate instruction about the research process with the writing process. Acknowledgements Thanks to Assessment in Action (AiA) campus team members Laura Martin, Matt Moberly, and Anne Zanzucchi; TRAIL faculty from fall 2014—Heather Devrick, Kath- erine Lee, Rex Krueger, Matt Moberly, and Grace Rocha; and many other individuals in the Merritt Writing Program who provided rubric feedback or assessment of student papers. In addition we would like to thank our colleague Elizabeth McMunn-Tetangco for evaluating student reflections and Principal Research Analyst Mike Roona for shar- ing his expertise and running the statistical analysis. Appendix A. TRAIL Final Research Reflection Be sure that your answers are thoughtful and complete. Think about the research you’ve done this semester in Writing 10 and answer each of the following questions: 1. How has your process for doing academic research changed since the beginning of the semester? Be specific. Give examples. 2. Describe your process for evaluating and selecting sources for your research as- signments in Writing 10. How did you decide which sources to use and which not to use? Did you add or change sources for your final assignment after turning in your annotated bibliography? If so, please explain why. 3. What challenges did you encounter when doing research for your assignments in this class? What strategies did you use to overcome them? Be specific. Give examples. 4. Have your attitudes and perceptions (confidence level) about doing research changed over the course of the semester? Be specific. Give examples. 5. Did learning more about the research process in this class help you in other classes this semester? Do you think it will help you in future classes? Be specific. Give examples. 6. Think about the research you’ve done this semester, and describe what you think it means to “Think Like a Researcher?” 178 C ollege & R esearch Lib raries M arch 2016 Appendix B. Rubric—Student Reflections: TRAIL WRI 10 *If students do not meet the marginal level, please score as 0. Student ID: Evaluator Initials: Concept Guiding Question 4 Advanced 3 Developing 2 Emerging 1 Marginal Score Academic Research Changes To what extent has a student incorporated new practices into the academic research process? Clearly outlines a minimum of two new or increasingly sophisticated research practices. Demonstrates maturity in the research process. Clearly outlines a minimum of two new or increasingly sophisticated research practices. Demonstrates strong progress in the research process. Outlines a minimum of one new or increasingly sophisticated research practice. Demonstrates some progress in the research process. May or may not outline a minimum of one new or increasingly sophisticated research practice. Demonstrates minimal progress in the research process. Source Selection Is the student using good judgment to select appropriate sources? Provides a thoughtful rationale for determining the selection of sources. Refers to a minimum of three appropriate criterion in source selection considerations. Provides a thoughtful rationale for determining the selection of sources. Refers to a minimum of two appropriate criterion in source selection considerations. Provides an adequate rationale for determining the selection of sources. Refers to a minimum of one appropriate criterion in source selection considerations. Provides a limited, incomplete, or superficial rationale for determining the selection of sources. May or may not refer to using appropriate criteria in source selection considerations. Challenges Is the student overcoming research challenges with useful strategies? Clearly identifies a minimum of two research challenges and provides highly useful strategies for overcoming them. Clearly identifies a minimum of two research challenges and provides some useful strategies for overcoming them. Identifies a minimum of one research challenge and provides at least one useful strategy for overcoming it. Identifies one or more challenges but lacks useful strategies for overcoming challenges. A ssessin g th e V alu e of C ou rse-Em b ed d ed In form ation Literacy 179 Concept Guiding Question 4 Advanced 3 Developing 2 Emerging 1 Marginal Score Attitude Does the student convey the attitudes required of a researcher? Tone and text clearly convey a positive attitude about the research process. Changes in research practices clearly indicate growth. Tone and text usually convey a positive attitude about the research process. Changes in research practices clearly indicate growth. Tone and text convey, at minimum, some positive attitudes about the research process. Changes in research practices indicate some growth. Tone and text convey some or limited positive attitudes about the research process. Changes in research practices indicate limited growth. Transferability Is the student applying research to other academic research needs? Demonstrates a strong understanding of the applicability of the research process to other academic needs. Clearly identifies and illustrates with a minimum of two specific examples. Demonstrates a solid understanding of the applicability of the research process to other academic needs. Clearly identifies and illustrates with a minimum of one specific example. Demonstrates some understanding of the applicability of the research process to other academic needs. May or may not illustrate with one or more examples. Makes a limited connection between the applicability of the research process to meeting other academic needs. May or may not illustrate with one or more relevant examples. Think Like a Researcher Does the student understand the thinking required of a researcher? Illustrates a strong understanding of the researcher mind-set. Refers to a minimum of three higher-level thinking characteristics required of researchers. Illustrates a solid understanding of the researcher mind-set. Refers to a minimum of two higher-level thinking characteristics required of researchers. Illustrates some understanding of the researcher mind-set. Refers to a minimum of one higher-level thinking characteristic required of researchers. Illustrates a limited understanding of the researcher mind-set. May tangentially refer to higher-level thinking characteristics required of researchers. Comments/Observations/Student Quotes: Total: 180 C ollege & R esearch Lib raries M arch 2016 Appendix C. Rubric—Evaluating Final WRI 10 Student Papers *If students do not meet the marginal level, please score as 0. Student Code: Evaluator Initials Concept Guiding Question(s) 4 Advanced 3 Developing 2 Emerging 1 Marginal Score Source Suitability To what extent have students use suitable sources (credible, relevant) in their papers for evidence? All sources are suitable. Most sources are suitable. Some sources are suitable. Few sources are suitable. Argument & Evidence Are students presenting multiple viewpoints? Do they support their arguments and counter- arguments with evidence? Multiple viewpoints are presented. Arguments and counter-arguments are strongly supported with evidence. Multiple viewpoints are presented. Arguments and counter-arguments are adequately supported with evidence. Multiple viewpoints are presented. Arguments and counter-arguments are minimally supported with evidence. A single viewpoint is presented and/or arguments generally lack evidence. Style Do students cite sources accurately? Do they successfully produce proper in-text citations and bibliographies? All in-text citations and bibliography references are consistently formatted in a standard citation style with minor or no errors. Citations include information needed for readers to locate the resource. In-text citations and bibliography references are consistently formatted in a standard citation style with few errors. Citations include most, if not all, information needed for readers to locate the resources. In-text citations and bibliography references are formatted in a standard citation style with some errors. Citations may be missing information needed for readers to locate the resources. In-text citations and bibliography references are often inconsistently formatted, incomplete, and/or missing. Citations are missing some information needed for readers to locate the resources. A ssessin g th e V alu e of C ou rse-Em b ed d ed In form ation Literacy 181 Concept Guiding Question(s) 4 Advanced 3 Developing 2 Emerging 1 Marginal Score Integration Do students successfully incorporate sources in their papers? As needed, do they use introductory phrases and transitions in order to smoothly integrate text? Sources receive attribution. Content is expertly incorporated into the text. Sources receive attribution. Content is satisfactorily incorporated into the text. Sources receive attribution. Some content is satisfactorily incorporated into the text. Sources may not always receive attribution. Some or little content is satisfactorily incorporated into the text. Comments/Observations: Total: Source Selection Observation: (not formally part of the rubric) □ All or most sources are selected from library collections or databases rather than the free web. □ Many sources are selected from library collections or databases rather than the free web. □ Few sources are selected from library collections or databases. 182 College & Research Libraries March 2016 Notes 1. “Tool for Real-Time Assessment of Information Literacy Skills (TRAILS),” Kent State University Libraries, accessed October 28, 2015, http://www.trails-9.org/. UC Merced’s TRAIL project is not connected to the TRAILS assessment tool of information literacy skills. 2. “Assessment in Action: Academic Libraries and Student Success,” Association of College & Research Libraries, accessed August 28, 2015, http://www.ala.org/acrl/AiA. 3. Marguerite G. Lodico, Dean T. Spaulding, and Katherine H. Voegtle, “Action Research,” in Methods in Educational Research from Theory to Practice (San Francisco: Jossey-Bass, 2010), 334–36. 4. Alan Carbery and Sean Leahy, “Evidence-Based Instruction: Assessing Student Work Using Rubrics and Citation Analysis to Inform Instructional Design,” Journal of Information Literacy 9, no. 1 (2015): 74–90; Davida Scharf, Norbert Elliot, Heather A. Huey, Vladimir Briller, and Kamal Joshi, “Direct Assessment of Information Literacy Using Writing Portfolios,” Journal of Academic Librarianship 33, no. 4 (2007): 462–77; Wendy Holliday, Betty Dance, Erin Davis, Britt Fagerheim, Anne Hedrich, Kacy Lundstrom, and Pamela Martin, “An Information Literacy Snapshot: Au- thentic Assessment across the Curriculum,” College & Research Libraries 76, no. 2 (2015): 170–87. 5. Mark Emmons and Wanda Martin, “Engaging Conversation: Evaluating the Contribution of Library Instruction to the Quality of Student Research,” College & Research Libraries 63, no. 6 (2002): 545–60; Debra Hoffmann and Kristen LaBonte, “Meeting Information Literacy Outcomes: Partnering with Faculty to Create Effective Information Literacy Assessment,” Journal of Informa- tion Literacy 6, no. 2 (2012): 70–85. 6. Barbara E. Walvoord, Assessment Clear and Simple: A Practical Guide for Institutions, De- partments, and General Education, 2nd ed. (San Francisco: Jossey-Bass, 2010), 18. For annotated bibliographies, see Carbery and Leahy, “Evidence-Based Instruction.” For reflective writing, see Hoffmann and LaBonte, “Meeting Information Literacy Outcomes.” For responses to a prompt, see Megan Oakleaf, “The Information Literacy Instruction Assessment Cycle: A Guide for Increas- ing Student Learning and Improving Librarian Instructional Skills,” Journal of Documentation 65, no. 4 (2009): 548. For papers, see Emmons and Martin, “Engaging Conversation”; Hoffmann and LaBonte, “Meeting Information Literacy Outcomes”; Holliday et al., “An Information Literacy Snapshot”; Scharf et al., “Direct Assessment of Information Literacy.” 7. Emmons and Martin, “Engaging Conversation”; Hoffmann and LaBonte, “Meeting Infor- mation Literacy Outcomes”; Scharf et al., “Direct Assessment of Information Literacy.” 8. For improving information literacy skills, see Emmons and Martin, “Engaging Conversa- tion”; Christina Hoffman Gola, Irene Ke, Kerry M. Creelman, and Shawn P. Vaillancourt, “Devel- oping an Information Literacy Assessment Rubric: A Case Study of Collaboration, Process, and Outcomes,” Communications in Information Literacy 8, no. 1 (2014): 131–44, http://www.comminfolit. org/index.php?journal=cil&page=article&op=view&path%5B%5D=v8i1p131; Scharf et al., “Direct Assessment of Information Literacy.” For redesigning library instruction, see Carbery and Leahy, “Evidence-Based Instruction”; Gola et al., “Developing an Information Literacy Assessment Rubric”; Holliday et al., “An Information Literacy Snapshot.” 9. Megan Oakleaf, The Value of Academic Libraries: A Comprehensive Research Review and Re- port (Chicago: Association of College & Research Libraries, 2010), 19, http://www.acrl.ala.org/ value/?page_id=21. Oakleaf identified student GPA and course grades as surrogates for student achievement. 10. Melissa Bowles-Terry, “Library Instruction and Academic Success: A Mixed-Methods As- sessment of a Library Instruction Program,” Evidence Based Library & Information Practice 7, no. 1 (2012): 82–95; John Cornell Selegean, Martha Lou Thomas, and Marie Louise Richman, “Long- Range Effectiveness of Library Use Instruction,” College & Research Libraries 44, no. 6 (November 1983): 476–80; Shun Han Rebekah Wong and Dianne Cmor, “Measuring Association between Library Instruction and Graduation GPA,” College & Research Libraries 72, no. 5 (2011): 464–73. Each study offered instruction through a different venue: library instruction in upper division classes; library credit course; and library workshops. 11. For first year, see Jason M. Vance, Rachel Kirk, and Justin G. Gardner, “Measuring the Im- pact of Library Instruction on Freshman Success and Persistence,” Communications in Information Literacy 6, no. 1 (2012): 56. For first semester, see Felly Chiteng Kot and Jennifer L. Jones, “The Impact of Library Resource Utilization on Undergraduate Students’ Academic Performance: A Propensity Score Matching Design,” College & Research Libraries 76, no. 5 (2015): 566–86; Krista M. Soria, Jan Fransen, and Shane Nackerud, “Library Use and Undergraduate Student Outcomes: New Evidence for Students’ Retention and Academic Success,” portal: Libraries and the Academy 13, no. 2 (2013): 147–64, doi:10.1353/pla.2013.0010. 12. Kot and Jones, “Impact of Library Resource Utilization”; Soria, Fransen, and Nackerud, “Library Use and Undergraduate Student Outcomes”; Vance, Kirk, and Gardner, “Measuring Assessing the Value of Course-Embedded Information Literacy 183 the Impact of Library Instruction.” 13. “Information Literacy VALUE Rubric,” Association of American Colleges & Universities, accessed October 28, 2015, https://www.aacu.org/value/rubrics/information-literacy; “Rubrics,” Rubric Assessment of Information Literacy Skills (RAILS), accessed October 28, 2015, http:// railsontrack.info/rubrics.aspx. All articles listed in the literature review section on rubrics have appended rubrics. Team librarians consulted these and found them to be valuable. 14. For small positive relationships, see Kot and Jones, “Impact of Library Resource Utilization”; Vance, Kirk, and Gardner, “Measuring the Impact of Library Instruction.” For mixed results, see Soria, Fransen, and Nackerud, “Library Use and Undergraduate Student Outcomes.” 15. “Framework for Information Literacy for Higher Education,” Association of College & Research Libraries, accessed November 11, 2015, http://www.ala.org/acrl/standards/ilframework. The framework refers to novice and expert researchers as well as knowledge practices and dis- positions for each of the frames. 16. Scharf et al., “Direct Assessment of Information Literacy,” 469. 17. “Think Like a Researcher: Instruction Resources,” UC Merced Library, accessed November 13, 2015, http://libguides.ucmerced.edu/think_like_a_researcher.