Mixed or Complementary Messages: Making the Most of Unexpected Assessment Results 197 Mixed or Complementary Messages: Making the Most of Unexpected Assessment Results Phil Jones, Julia Bauder, and Kevin Engel Phil Jones is Humanities Librarian and Coordinator of Research Services, Julia Bauder is Social Studies and Data Services Librarian and Interim Librarian of the College, and Kevin Engel is Science Librarian, all at Grinnell College Libraries; e-mail: jonesphi@grinnell.edu, bauderj@grinnell.edu and engelk@grin- nell.edu. Acknowledgements: The authors would like to thank our project team of Henry Rietz and Carlie Van Willigen and our faculty partners Stella Chan, Laura Sinnett and Carmen Valentín. We dedicate this article to Richard Fyffe, whose vision for teaching and learning it reflects. Attribution: ©2016 Phil Jones, Julia Bauder, and Kevin Engel, Attribution-NonCommercial 3.0 (http://creativecommons.org/licenses/ by-nc/3.0/) CC BY-NC 3.0. Grinnell College participated in ACRL’s first cohort of Assessment in Ac- tion (AiA), undertaking a mixed-methods action research project to assess the effectiveness of librarian-led research literacy sessions in improving students’ research skills. The quantitative data showed that the quality of students’ sources did not markedly improve following a research literacy session, while the qualitative data indicated that many students were able to state and describe important research concepts they learned. This article profiles the development of Grinnell’s AiA project and discusses how Grinnell’s librarians responded when the initial results led to more questions rather than to satisfactory answers. Background Grinnell College, a highly selective residential liberal arts college that enrolls 1,600 full- time undergraduate students, is located in central Iowa. Grinnell’s AiA team included members from across campus, including three librarians, a religious studies faculty member with assessment experience, and a professional from the Office of Analytic Support and Institutional Research. In abidance with AiA guidelines, this team con- ducted action research, a methodology concisely defined by McMillan and Wergin as research undertaken in a practice setting for the purpose of better understanding professional work and how to improve it.1 Students’ academic research is a deeply rooted priority at Grinnell. The college’s mission statement and core values emphasize that students learn to acquire and to critically consider diverse forms of knowledge, all “for the common good.”2 Recent faculty discussions have focused on clarifying the phrase student research and on de- veloping institutional learning goals. Though still in draft form, two of these campus- wide goals, listed here, align well with the concepts and skills Grinnell’s librarians cover during research literacy sessions: that students (1) “develop creative and criti- cal thinking skills that allow them to analyze the work of others, formulate relevant questions, and respond to those questions in a substantive way using quantitative doi:10.5860/crl.77.2.197 198 College & Research Libraries March 2016 and qualitative evidence,” and (2) “develop the ability to approach a question from multiple perspectives, representing a diversity of ideas and experiences.”3 Addition- ally, Grinnell’s faculty members are working to articulate or to update learning goals for each of Grinnell’s 39 academic majors and concentrations, and many academic departments have included goals tied to research literacy. It was this significant, campus-wide interest in student research that shaped the guiding question for the present study: whether and how do typical in-class research literacy sessions help students to learn and to grow as researchers.4 To address this question, the team de- cided to conduct a mixed-methods study, including both a quantitative performance assessment using citation analysis and a qualitative element in which students were asked to reflect on their research process and learning. Literature Review Neither faculty nor students had suggested that Grinnell’s librarians change their teaching methods, but their pedagogy, while incorporating some best practices such as scaffolded research assignments and an effort to keep sessions active, was largely traditional, not drawing on ideas such as threshold concepts or popular instructional strategies like flipped classrooms. While a few studies strive to show the limits of one- shot sessions,5 the literature includes a wide range of studies contending that, since the single session format will remain the preferred and most practical setting for the teaching of information literacy skills and concepts, librarians should structure these sessions in ways to best help their students learn.6 Citation analysis studies are a long-standing strategy for assessing the impact of library instruction. Typically, these studies have focused on documenting any changes in students’ research skills following an instruction session. Citation analysis has also been used to assess a range of library services; recent studies focus on effects of library collection use and research consultations.7 Rather than provide an exhaustive review of this considerable literature, only work relevant to our study is discussed here. For a more detailed overview of citation analysis, including its use to explore the effective- ness of library services, please see Long and Shrikhande’s chapter discussed below.8 Dykeman and King’s early study and Long and Shrikhande’s more recent one are representative, as both find that library instruction sessions improve the quality of students’ research bibliographies.9 Another topic of consistent study in the citation analysis literature is methodology for the evaluation for students’ research bibliogra- phies. Gratch’s early work identifies criteria and processes for rating student’s sources.10 Thereafter, standard criteria for citation analysis studies include number, authority, variety, availability, and citing of sources, with librarians and less often faculty members rating students’ selections.11 Mixed-Methods Action Research This study attempts to fill two gaps in the wide-ranging literature of citation analy- sis and library services. First, the student bibliographies in this study were rated by both faculty and students, but not by librarians. Although librarians often rate sources in citation analysis studies, Grinnell’s AiA team wanted faculty to be the raters because they have deep, expert knowledge of the scholarly literature their students explored. The three classes taking part in this study included only second-, third- and fourth-year students, all of whom had met with a librarian previously. The AiA team expected that these students would all select sources that were, at minimum, appropriate for college-level work, and hence the team wanted raters with the best possible ability to distinguish degrees of authority and relevance within the literature of each discipline. Making the Most of Unexpected Assessment Results 199 Secondly, few if any studies have asked students to rate the quality of their sources or to reflect on their selection of sources in their own words. While, in the typical citation analysis study, raters are given the opportunity to distinguish de- grees of quality among the sources students included, the students’ evaluations of each source can be captured only as a binary decision to include or not include it. Since students’ ability to distinguish levels of quality among appropriate sources is a potential outcome for information literacy instruction, Grinnell’s AiA team wanted to explore whether students’ judgments of the relative strength of their sources matched the faculty’s judgments. The AiA team also compiled the number of sources each student omitted from their revised bibliography as well as the number of sources they added during revision, as we consider these changes to represent the number of decisions students made during the process of revision. As complementary qualitative data, the team gathered students’ written responses to three open-ended questions (see appendix A). Grinnell’s AiA team recruited a faculty partner from each of Grinnell’s three academic divisions—humanities, social studies and the sciences—to participate in the study. Each of our faculty partners brought deep disciplinary knowledge and years of experience working with librarians to integrate research literacy sessions into their classes. While the research literacy sessions under study here were developed for specific classes, all three sessions shared elements like brief overviews of databases and websites, discussion of searching techniques, and consideration of the types of sources most appropriate for the respective assignments. Other than ensuring that each student completed rough and revised bibliographies, the AiA team did not closely control for instructional method, as we wanted our study to reflect the day-to-day practice of academic librarians working with a range of teaching styles, academic disciplines, and lower- as well as upper-division courses. Participating Classes and Their Research Assignments Spanish 343, The Art of Language, is a required, upper-division class for majors that focuses on Spanish grammar and language use. Students draft and revise a five-page paper drawing upon up to five varied sources. One of the goals of the assignment is for students to find, read, and incorporate sources from throughout the Spanish-speaking world—Spain, South and Central America, the Caribbean, and different parts of the United States—into brief research essays that first describe and then argue a particular point of view on topical issues. Recent examples include methods for treating obesity, bullfight- ing, the Catholic Church, and bilingual education. Librarian Phil Jones collaborated with faculty member Carmen Valentín to plan and teach this course’s research literacy session. Psychology 225, Research Methods, is a required course for majors that describes experimental designs and attendant statistical techniques. Students learn to use quan- titative methods to pose meaningful questions to data. Topics include between-group and within-group designs, analysis of variance for main effects and interactions, the adaptation of statistical inquiry to less than optimal situations, and critical thinking about research methods. Librarian Kevin Engel collaborated with faculty member Laura Sinnett to plan and teach this course’s research literacy session. Economics 380, Seminar in Monetary Economics, is an elective, upper-division seminar that focuses on the roles of national central banks and similar international institutions in the economy. For their final project, students produce substantial research papers on a topic of their choosing related to monetary economics, such as the effects of the Federal Reserve’s quantitative easing program on the global economy or the fac- tors that influence a currency’s exchange rate. Librarian Julia Bauder collaborated with faculty member Stella Chan to plan and teach this course’s research literacy session. 200 College & Research Libraries March 2016 After receiving approval for this study from Grinnell’s Institutional Review Board, librarians or faculty partners explained to each of the three classes that the survey was part of a formal research project. Students willing to participate signed a consent form; the few students who declined to participate or were under 18 years of age still completed their class’s research assignments, but their work was not included in our study. Fifteen students agreed to participate from the Spanish class, 14 from the psy- chology class, and 11 from the economics class. Soon after, the students searched for sources they deemed relevant and promising for an upcoming course assignment and then submitted a draft bibliography to their professor and librarian. After reviewing the students’ draft bibliographies, each librarian then worked with his or her faculty colleague to plan a research literacy session that recognized and extended the students’ understanding of research resources and strategies. Librarians came to the subsequent research literacy session with specific points drawn from the draft bibliographies and, of course, took questions from students during and following the class. Students then revised their research processes and lists of potential sources. Soon after the final research assignments were due, librarians distributed to each class the citation surveys, which asked students and faculty to rate each source numerically on a scale of 1 to 5 for three criteria: relevance, timeliness, and authority (see table 1). Prior to distributing the surveys, the librarians provided the students and faculty partners a concise explanation of our project’s rating criteria. The librarians did not offer training to students on how to rate their sources, beyond any discussion about evaluating sources that may have occurred during their research literacy sessions. While faculty were asked to analyze citations on both draft and revised bibliographies, students rated only their revised, final citation lists. Each faculty member rated only the bibliographies from her own class, so each item was rated by only two people, one student and one faculty member. And a written portion of the survey asked students for further narrative discussion of the process of constructing their bibliographies (see appendix A). Results Generally, faculty believe that their students are citing relevant, timely, and authoritative sources. The average rating for all three criteria on both the draft and final bibliogra- phies was quite high, as can be seen in table 2. TABLE 1 Citation Survey This item is… Strongly Agree (5) (4) (3) (2) Strongly Disagree (1) Very relevant to my topic Timely Authoritative Making the Most of Unexpected Assessment Results 201 Students revised their bibliographies, often extensively, following the research lit- eracy session. All of the final bibliographies contained between four and six sources, with the vast majority containing exactly five sources. From reviewing student citation surveys, it is clear that some students followed a one-for-one revision process—that is, omitting a source when choosing to add another in its place—but many students also simply reduced or increased the number of sources on their revised bibliographies following the instruction session with a librarian (see table 3). The research assignment for Spanish students allowed for the greatest range of sources—Spanish-language magazines, newspaper articles and editorials, scholarship, websites, government publications, and videos, for instance—and so these students appear to have explored these options, most frequently opting for new sources. Both the economics and psychology classes were required to use scholarly materials for their research projects; on both draft and revised bibliographies, economics students cited a range of scholarly sources—journal articles, working papers, and conference proceedings—while the psychology students cited only scholarly articles on both ver- sions of their bibliographies, in abidance with their assignment guidelines. It should also be noted that the psychology students had a shorter timeframe for revision, with only one week between handing in rough and final versions of their bibliographies, while Spanish and economics students had three weeks. However, despite revising their bibliographies, the students did not, on average, greatly improve the quality of the sources cited; again, only faculty analyzed citations on both draft and revised bibliographies. For the 34 students for whom draft and final bibliographies could be compared TABLE 3 Average Number of Sources Changed by Class Average Number of: SPN 343 ECN 380 PSY 225 All Three Classes Omitted Sources 3.29 1.55 1.31 2.05 Added Sources 2.5 3.09 1.23 2.27 Combined Changes 5.79 4.64 2.54 4.32 TABLE 2 Average Faculty and Student Citation Rating by Criteria Average Rating: Scale 1 (strongly disagree) to 5 (strongly agree)—All Classes Draft Bibliography— Professor Rating Final Bibliography— Professor Rating Final Bibliography— Student Rating Criterion: This item is very relevant to the topic. 4.25 4.3 4.43 Criterion: This item is timely. 4.32 4.38 4.09 Criterion: This item is authoritative. 4.33 4.22 3.99 Average Items Changed: 4.32 202 College & Research Libraries March 2016 • 5 improved on one or more criteria and held steady on the rest, • 8 worsened on one or more criteria and held steady on the rest, and • 21 improved on some criteria and worsened on others.12 On average, for these 34 students, the improvement across all three criteria was a meager .22 points on the 15-point total rating scale: .09 average improvement on the 5-point relevance scale, .11 average improvement on the 5-point timeliness scale, and .03 average improvement on the 5-point authority scale. (The totals do not add to .22 because of rounding.) None of these differences, either on the single-criterion scales or on the combined scale, is statistically significant (all p > .05). Grinnell’s AiA team also asked students to rate their own sources, as their point of view has rarely been included in citation analysis studies. The librarians on the team compared students’ average ratings for the items on their revised bibliographies for each of the three criteria to the faculty’s average ratings for the same items to gauge if students have learned what their professors (and librarians) intended and, therefore, rated sources comparably. As can be seen in table 2, students rated the relevance of sources on their final bibliographies slightly higher than faculty did (although the difference was not statistically significant), but they rated the timeliness and authority of their sources sig- nificantly lower than the faculty: .29 points (p = .001) and .23 points (p < .05), respectively. What Students Have to Say: Using Nvivo to Explore Qualitative Data As the three authors moved from analyzing the quantitative data to exploring the qualitative data, there were two major questions we hoped to be able to answer: • Did students learn skills and concepts in these research literacy sessions other than the three measured on the survey? • Why did students believe that the sources they chose were less timely and authoritative than their professors did? After reviewing the student’s responses, the authors developed a list of 19 catego- ries of student responses to use as we began coding the qualitative data. As two of the authors then separately coded all the students’ responses, 13 more codes emerged. Of these 32 codes, 23 proved most relevant and appear in table 4 in percentage breakdowns of students’ responses to two of our three open-ended survey questions. Again, as two of the three classes in our study required that students include only scholarly sources in their bibliographies, it is not surprising that two-thirds of students cited journal articles as influential sources. The range of sources students in Spanish TABLE 4 Student Responses on Identifying Influential Source Question 2: Influence: “Which single source has most influenced your thinking on the topic? Why did you find that particular source so influential?” Influential Source Type 81 coded responses from 39 students Journal Article 66% 26 coded responses Website 10% 4 Video 8% 3 Working Paper 5% 2 Conference Paper 3% 1 Newspaper Article 3% 1 Thesis 3% 1 Not Known 3% 1 Making the Most of Unexpected Assessment Results 203 343 listed as influential—websites, videos, a newspaper article, and a thesis—shows, again, that students will use a variety of materials if their professor and the assignment at hand allow for it. These results are consistent with Middleton’s finding that it is the nature of an academic assignment that most heavily influences the type of sources students select (see table 4).13 We asked students to identify an influential source as a way to prompt their en- gagement with sources and reflection upon research processes. The faculty member on Grinnell’s AiA team requested that we include this question on the citation survey as students do not usually consider a source’s influence upon them; he stressed, too, that source influence is a topic other faculty members would want to know about. The authors conclude from this list that when asked, students will provide on-target, insightful replies regarding source influence, touching on but going well beyond the practical, perhaps expected responses that a source is frequently cited or includes a helpful list of works cited (see table 5). Representative responses follow: The paper by [name], et al was the most influential for my thinking on the topic because it changed my preconceived notions about [this problem]. Originally, I believed that… techniques would be particularly useful for handling [this prob- lem], but after reading [this] paper, I realize that this belief is inaccurate. I would have to say that the YouTube documentary is the source that has most influenced my thinking because it contains the point of view of many scientists from different parts of the world. In fact, some of those scientists used to work for the [name of organization], which… is mainly responsible for the public disagreement on [topic]. Thus, by listening to the point of view of those former scientists of the [organization], I have a better understanding of how the [orga- nization] comes up with the [topic] report that the UN releases every few years. Recall that librarians and faculty members planned research literacy sessions that would be both effective and representative of our work at Grinnell. Most of the top- ics included in the student responses regarding learning were in fact covered, either implicitly or explicitly, in each research literacy session: evaluating sources, search techniques, and identifying and using new databases, for instance. The assignment for one class (Spanish 343) stressed source variety and point of view, and so those students’ replies tend to touch on these topics. Of interest here is the number of stu- dents’ responses to question 3, which asked them to reflect on their learning: 230, or nearly three times the total responses students made regarding source influence (see TABLE 5 Student Responses on Why a Source Was Influential Why was a source influential? 81 coded responses from 39 students Gives me new ideas 25% 20 coded responses Serves as a model 21% 17 Strong, relevant source 20% 16 Provides good overview of topic 16% 13 Confirms my hypothesis 14% 11 Is important or frequently cited 2% 2 Lists potential sources 2% 2 204 College & Research Libraries March 2016 table 6). Students are likely more accustomed to answering questions on learning than influence, but the volume and variety of their responses related to learning shows that they clearly believed they had learned a range of research skills and concepts during one of the research literacy sessions under study. Again, representative responses follow below: I learned about the importance of precision of language while creating search terms, including some specific tools for smarter searching like truncation and manipulating the types of results found, like restricting dates for timeliness. I also learned how to determine the quality of a source beyond whether or not it is peer-reviewed, like evaluating bias and looking at the purpose of research. I learned about the importance of diversity in sources. Academic articles are wonderful sources of information and usually provide the most [comprehensive] look at a topic, but newspaper and magazine articles may better capture the con- text in a way a journal article cannot. Also, I learned advanced search techniques which I had not been exposed to previously. Using the advanced searches and clarifying the search can lead to a narrower and more appropriate list of results. These two responses indicate that some students will engage in recursive, critical research practices when encouraged to do so. But is there a connection between what the students say they learned and how the faculty and students themselves rated sources on the bibliographies? This question is explored below. Words Talking to Numbers: Comparing Qualitative and Quantitative Data So far, the authors have found that the three faculty members agree that their students are, generally, finding acceptable sources for research bibliographies; that students are changing sources between draft and revised bibliographies; and that students tell us in their written responses in articulate, detailed ways that they learned valu- able skills and concepts in three representative research literacy sessions. But how do these findings compare to faculty ratings for our study’s three criteria of source relevance, timeliness, and authority? To explore these questions, two of the authors used Nvivo to run cross-tabulations between a normalized average score for each TABLE 6 Student Responses on What They Learned from the Research Literacy Session Question 3: Learning: “What are the two or three main points you learned from the research literacy session a librarian held with this class earlier in the semester? 230 coded responses from 39 students Evaluating sources 25% 58 coded responses Revision of research bibliography or paper 16% 36 Search techniques 15% 34 Reflection upon research skills or concepts 13% 31 New databases or tools 10% 23 Source variety (audience, format, etc.) 9% 20 Source point of view 8% 18 Assignment guidelines 4% 10 Making the Most of Unexpected Assessment Results 205 of the students’ bibliographies and various other factors. Normalizing the scores was necessary because there were substantial differences in the average scores for each class. (The authors believe that this reflects the fact that the faculty raters did not have the opportunity to discuss the rating scales with each other, rather than any intrinsic difference in the quality of sources used by the students in each class.) The averages of each of the three faculty members’ ratings for relevance on the final bibliographies were 3.72, 4.2, and 4.86; for timeliness, 4.26, 4.28, and 4.59; and for authority, 3.87, 4.13, and 4.59. To normalize the scores, the average ratings for each student were subtracted from the average ratings for the entire class for each of the three criteria, showing how far above or below average that student was compared to the rest of their class. These normalized scores were then used to divide the bibliographies into three equally-sized “high,” “medium,” and “low” groups for each criterion. When comparing what students reported learning from a research literacy ses- sion to how faculty members rated sources appearing on revised bibliographies, the authors found just one instance of a positive relationship between student comments and faculty ratings. However, given that this is the sole positive relationship found between students’ comments and faculty members’ citation ratings from the over 30 cross-tabulations run for this study, it is likely that this finding is the result of random chance and that it would not be consistently replicated in future studies. Discussion This study appears to have produced conflicting results. The qualitative data—students’ responses to three survey questions—suggest that students learned more than the authors realized we were teaching them and that they can readily identify and reflect upon sources they find influential as well as upon their research processes. And part of the quantitative data is at first glance encouraging—the fact that students omitted and added sources to their revised bibliographies. Even though librarians and faculty members demonstrated and encouraged students to find and consider new sources to strengthen or extend their research projects, the authors were surprised to learn that these new sources did not substantially improve the overall quality of the students’ bibliographies. The authors are also concerned that we can find so few clear connections between what students say they have learned during a research literacy session and how faculty, or the students themselves, rated bibliography items. The goal for this assessment project has been to investigate if and what students are learning in the types of research literacy sessions Grinnell’s librarians tend to plan and teach with faculty members. Our conclusion is that students were doing an adequate job of finding sources prior to research literacy instruction and that their ability to evaluate sources, in terms of timeliness and authority in particular, did not improve markedly following such a session. Another Look at Scholarly Literature: Classic and Recent, 1996–2013 To consider the implications of our findings, the authors turn to two classic, often- cited studies in the literature of library science and to two more recent studies. The phenomenon of students “desperately seeking citations” has been documented by Leckie and recently revisited by Rose-Wiles and Hofmann.14 To calm and to bring focus to students’ search for sources, Leckie’s 1996 article called for faculty to scaffold research papers and projects and for librarians to help prepare faculty members to teach information literacy concepts themselves, consistently and when most appropriate in their classes. In updating Leckie’s work, Rose-Wiles and Hofmann concluded that 206 College & Research Libraries March 2016 web-based discovery systems, while likely too sophisticated for most faculty members to effectively demonstrate to students, can free librarians to teach one interface, not multiple databases, allowing more time for faculty and librarians to teach students active and reflective research practices. Our findings indicate that even the sequenced, collaborative research assignments Leckie called for and the focused research literacy sessions introducing a major database, search strategies, and research tips that Rose- Wiles and Hofmann suggested can still send students scurrying after new sources that may or may not improve the overall quality of their work. So where to turn? Again, the authors look to a classic, but more recent article. In 2006, Elmborg shot a flare into the scholarship of information literacy, a literature of staggering size, burdened by idiosyncratic and repetitive studies, by calling for “a critical practice of librarianship,” in which librarians become specialists in coaching intellectual growth and critical development. Learning becomes the essentially humanistic process of engaging and solving significant problems in the world, a process central to both teaching and learning. Information can then be redefined as the raw material students use to solve these problems and to create their own understandings and identities, rather than as something out there to be accessed efficiently, either in the library or in the world.15 For the next decade, the topic of critical information literacy was explored with increasing frequency and focus in conference presentations, book chapters, scholarly articles, and blog postings, refining Elmborg’s philosophical charge into exercises and strategies apt for librarians’ day-to-day use.16 A recent response to Elmborg’s article is an observational study of a writing class by Holliday and Roberts that makes a recom- mendation supported by our study’s findings: librarians should be spending less time on demonstrations, tutorials and lectures that focus only on searching for information or evaluating sources using external proxies for qual- ity. Instead, we need to develop learning activities to help students read and interrogate sources, follow ideas through the practice of citation chaining and summarizing their understanding of sources in their totality.17 Holliday and Roberts also stress that as librarians shift the emphasis of their teaching from skill-based outcomes, such as finding sources, we should also rethink our use of “library-centric discourse” and strive to use language with students and faculty mem- bers stressing research as a practice of learning.18 This way, both librarians’ pedagogy and speech will help convey the message that their professional responsibility is not met once students have a few promising sources in hand. Recommendations and Conclusion So what do the authors make of this study’s findings? The answer: this action research project has yielded complementary rather than mixed results. As librarians, we need to reflect on the finding that the majority of our students in 200+ level courses (usually sophomores, juniors, and seniors) are able to produce strong bibliographies and do not need to be taught standard database search methods. Though students may not be as efficient and effective searchers as librarians may wish, they have sufficient experience and understanding to identify resources that their teachers judged to be relevant and authoritative. This finding is at odds with the library instruction that faculty usually ask for—that is, sessions with an emphasis on introducing databases and the mechan- ics of searching. And since students can state with clarity and insight what they have Making the Most of Unexpected Assessment Results 207 learned from research literacy sessions, librarians should prioritize class discussions and include open-ended questions on session feedback forms. Librarians at Grinnell College were already moving toward the pedagogy suggested by this study, but these findings can guide us as we reconsider what and how we are teaching our students. The authors welcome ACRL’s recently-adopted Framework for Information Literacy for Higher Education as a more conceptual, open-ended guide to the instruction work recommended both by the studies cited in this article and by the findings of our action research project. Accordingly, here is an overview of the steps Grinnell’s librarians have already taken as well as those we are planning. What we have done to date: • Developed a tip sheet drawn from our study and the new Framework to encour- age Grinnell’s librarians to experiment during their research literacy sessions. See appendix B. • Hosted a catered lunch to thank Grinnell’s AiA team and faculty partners for their contributions over the 2.5 years we have been working on this study. We also discussed steps for sharing our study’s findings across campus and consid- ered how our action research might impact research literacy work at Grinnell. • Held two discussions among Grinnell’s library faculty members to consider our successes and challenges in integrating concepts from our AiA project and ACRL’s Framework into research literacy sessions during the fall 2015 semester. • Agreed that in-depth introduction of databases and the mechanics and strate- gies of searching are still appropriate and valuable for courses predominantly involving first-year students; at Grinnell College, for example, the first semester Tutorial and 100-level courses. Beyond those courses, introducing databases and searching mechanics will have a much lower priority. • Agreed that for 200+ level courses, a targeted, limited time is appropriate at the end of a class session to introduce students to relevant databases, advanced search techniques, bibliography managers, library catalogs, etc. • Flipped the classroom, by having students explore a database or tool that a faculty member particularly wanted them to use as a way to prepare for an upcoming research literacy session. Doing so allowed the librarian to focus a class session on higher-level topics such as judging the quality, authority, and appropriateness of sources. • Expanded introductory research literacy sessions from a one-shot to a two- session sequence. Session one focused on search techniques and results and library services; session two, on source evaluation and use of potential sources in students’ academic work. • Prioritized class discussion of potential sources during research literacy ses- sions. What is the source’s main point, and how might it be used in advancing the student’s own argument or project? What we will do in the future: • Our campus is beginning the search for a new library director, and as part of this process librarians will meet with groups of faculty and students to discuss a range of questions related to library services, such as, “What should be the role of Grinnell’s libraries in supporting, for instance, student research, study spaces, events, and employment? In supporting faculty teaching and research?” These discussions may allow librarians to share some of the findings from this study and from our experiences teaching research literacy in light of our AiA project and the new ACRL Framework. • Grinnell’s librarians will host a Faculty Friday lunch discussion on research literacy; this campus luncheon series is co-sponsored by Grinnell’s Dean’s Office 208 College & Research Libraries March 2016 and Center for Teaching, Learning, and Assessment as a forum for topics and discussion relevant to faculty members’ teaching and research. We’ll share an overview of this study’s findings and provide examples of how we’re changing our instructional practice accordingly. • Focus our planning and discussion of research literacy with faculty, students, and staff in terms of existing, relevant institutional and departmental learning goals. Many such goals already include or refer to research literacy. This way, librarians can collaborate with faculty in using a common, agreed-upon campus structure rather trying to create our own set of library-specific learning goals and having to cultivate interest. This article documents the progression of an action research project and the de- velopment of academic librarians into researchers working with complementary quantitative and qualitative data to explore questions of local, practical use as well as professional and scholarly importance. Professional resources have already begun to emerge to help librarians adapt both ACRL’s Framework and, by extension, this study’s recommendations to their teaching. Recent articles by Oakleaf and by Bauder and Rod are good starting points.19 Appendix A Student Citation Survey Student’s name: Assessment in Action Project Professor’s name: Fall 2013 Department and Course Number: Written Responses: For the three questions below, please provide concise responses with helpful detail. 1. Did you omit any sources appearing in your draft bibliography from your revised bibliography? If so, why did you decide not to include those sources in your re- vised bibliography? Please note: if you changed your research topic between the draft and revised bibliographies, please explain why you did so rather than answer the first part of this question. 2. Of all of the sources in your bibliography, which single source has most influenced your thinking on the topic? Why did you find that particular source so influential? 3. What are two or three main points you learned from the research literacy session a librarian held with this class earlier in the semester? Making the Most of Unexpected Assessment Results 209 Appendix B Moving Closer: Grinnell’s AiA Project and the New ACRL Framework for Information Literacy During the past few months, we have discussed both Grinnell’s Assessment in Action Project and Association for College and Research Libraries (ACRL) recently-unveiled Framework for Information Literacy for Higher Education. It’s become clear through these conversations that both our AiA study and the new Framework are pointing us in the same direction: the development of research literacy as a higher-order intellectual process rather than a mastery of a list of library skills. Both our own action research and the ACRL document accurately reflect the work that we, as academic librarians, are already doing. To move our teaching even closer to the practice of students’ research and writing, ideas for research literacy session planning and activities are provided below to try in partnership with our faculty colleagues. For Tutorial and Appropriate 100-level Courses During research literacy sessions, librarians continue to cover a range of topics that introduce new students to Grinnell’s libraries: • profile the libraries’ website • discuss evaluating and citing sources • demonstrate access and use of the library catalog and 3Search • demonstrate access and use of subject guides • identify and profile relevant databases • demonstrate effective search techniques • interpret result lists • demonstrate access to full text sources, including ILL Librarians and Tutors can consider a second research literacy session later in the semes- ter, perhaps asking students to research on their own to find promising sources before- hand, in an inverted classroom approach. Possible emphases for the second session: • center on evaluation of the sources students identified ◊ What is this source’s main point, and how will it contribute to your argu- ment? ◊ Is this source credible, and why? ◊ Is this source appropriate for your subject and for your assignment? • include a student-led component during which research experiences—both positive and negative—are shared with their peers, Tutor, and librarian in a supportive environment. For Courses beyond the Introductory Level For 200+ level courses such as research methods and seminars, librarians and their faculty colleagues can try inverted classroom techniques and activities, asking students to conduct research on their own beforehand. Use the research literacy session to: • discuss what students found, where they looked, and what they did not find but would like to have • help students identify resources for further research • discuss evaluation of sources ◊ Is the source credible and appropriate and why? ◊ How will the source contribute to your argument? • cover the use of databases and a review of searching mechanics as needed toward the end of the session 210 College & Research Libraries March 2016 Librarians can meet with a class for the second time during a semester after students have, for instance, handed in an annotated bibliography. During the second session, students can: • revise their research processes and source selections • discuss and share how any new sources help to change or to refine their research question or topic. Assessment Librarians and faculty colleagues can try implementing new ideas inspired by the Framework and carefully assess the ideas’ effectiveness. • Librarians can make use of the updated Research Literacy Feedback Form and consider asking students additional questions that are important for a particular session. • Librarians can track student questions, comments, suggestions, and concerns. • Faculty partners can assess the quality of student assignments in connection to changes in research literacy sessions and discuss results with librarians. For Further Reading Bauder, Julia, and Catherine Rod. “Crossing Thresholds: Critical Information Literacy Pedagogy and the ACRL Framework.” College & Undergraduate Libraries. Forthcom- ing. Association of College and Research Libraries. “Framework for Information Literacy for Higher Education.” February 2, 2015. http://www.ala.org/acrl/standards/il- framework Jones, Phil, Julia Bauder, and Kevin Engel. “Mixed or Complementary Messages: Making the Most of Unexpected Assessment Results.” College & Research Libraries 77, no. 2 (2016). Notes 1. James McMillan and Jon Wergin, Understanding and Evaluating Educational Research, 3rd ed. (Upper Sale River, NJ: Pearson, 2006): 164. 2. Grinnell College, “Mission and Values,” accessed November 11, 2015, https://www.grin- nell.edu/about/si/mission. 3. Grinnell College, “College-Wide Learning Outcomes for Students,” unpublished document, 2015. 4. Over the past few years, librarians at Grinnell College have started using the term research literacy rather than information literacy, since we find that the former term resonates more easily with faculty and students. 5. Yvonne Mery, Jill Newby, and Ke Peng, “Performance-Based Assessment in an Online Course: Comparing Different Types of Information Literacy Instruction,” portal: Libraries and the Academy 12, no. 3 (2012): 283–98. 6. Elizabeth Spievak and Pamela Hayes-Bohanan, “Just Enough of a Good Thing: Indications of Long-Term Efficacy in One-Shot Library Instruction,” Journal of Academic Librarianship 39, no. 6 (2013): 488–99; Shevaun E. Watson, Cathy Rex, Jill Markgraf, Hans Kishel, Eric Jennings, and Kate Hinnant, “Revising the ‘One-Shot’ through Lesson Study: Collaborating with Writing Faculty to Rebuild a Library Instruction Session,” College & Research Libraries 74, no. 4 (July 2013): 381–98; William Badke, “Ramping Up the One-Shot,” Online 33, no. 2 (March 2009): 47–49; Barbara Ferrer Kenney, “Revitalizing the One-Shot Instruction Session using Problem-Based Learning,” Reference & User Services Quarterly 47, no. 4 (Summer 2008): 386–91. 7. Stacey Knight-Davis and Jan S. Sung, “Analysis of Citations in Undergraduate Papers,” College & Research Libraries 69, no. 5 (September 2008): 447–58; Thomas L. Reinsfelder, “Citation Analysis as a Tool to Measure the Impact of Individual Research Consultations,” College & Research Libraries 73, no. 3 (May 2012): 263–77. 8. Casey M. Long and Milind M. Shrikhande, “Using Citation Analysis to Evaluate and Im- Making the Most of Unexpected Assessment Results 211 prove Information Literacy Instruction,” in Collaborative Information Literacy Assessments: Strategies for Evaluating Teaching and Learning, ed. Thomas P. Mackey and Trudi E. Jacobson, 5–24 (New York: Neal-Schuman, 2010). 9. Amy Dykeman and Barbara King, “Term Paper Analysis: A Proposal for Evaluating Bib- liographic Instruction,” Research Strategies 1, no 1 (1983): 14–21; Long and Shrikhande, “Using Citation Analysis,” 6–8. 10. Bonnie Gratch, “Toward a Methodology for Evaluating Research Paper Bibliographies,” Research Strategies 3, no. 4 (1985): 170–77. 11. Virginia E. Young and Linda G. Ackerson, “Evaluation of Student Research Paper Bibli- ographies: Refining Evaluation Criteria,” Research Strategies 13, no. 2 (1995): 80–93; Karen Hovde, “Check the Citation: Library Instruction and Student Paper Bibliographies,” Research Strategies 17, no. 1 (2000): 3–9. 12. Two students (one in the psychology class and one in the Spanish class) did not submit a draft bibliography, and four students (three in the psychology class and one in the Spanish class) did not make any revisions to their draft bibliography. 13. Anne Middleton, “An Attempt to Quantify the Quality of Student Bibliographies,” Perfor- mance Measurement and Metrics 6, no. 1 (2005): 7–18. 14. Gloria J. Leckie, “Desperately Seeking Citations: Uncovering Faculty Assumptions about the Undergraduate Research Process,” Journal of Academic Librarianship 22, no. 3 (1996): 201–8; Lisa M. Rose-Wiles and Melissa A. Hofmann, “Still Desperately Seeking Citations: Undergraduate Research in the Age of Web-Scale Discovery,” Journal of Library Administration 53 (2013): 147–66. 15. James Elmborg, “Critical Information Literacy: Implications for Instructional Practice,” Journal of Academic Librarianship 32, no. 2 (2006): 198. 16. For a recent example, see Maria T. Accardi, Emily Drabinski, and Alana Kumbier, Critical Library Instruction: Theories and Methods (Duluth, MN: Library Juice Press, 2010). 17. Wendy Holliday and Jim Rogers, “Talking about Information Literacy: The Mediating Role of Discourse in a College Writing Classroom,” portal: Libraries and the Academy 13, no. 3 (2013): 269. 18. Ibid., 268. 19. Megan Oakleaf, “A Roadmap for Assessing Student Learning Using the New Framework for Information Literacy for Higher Education,” Journal of Academic Librarianship 40, no. 5 (2014): 510–14; Julia Bauder and Catherine Rod, “Crossing Thresholds: Critical Information Literacy Pedagogy and the ACRL Framework,” College and Undergraduate Libraries, forthcoming.