College & Research Libraries News vol. 79, no. 4 (April 2018) C&RL News April 2018 186 Inspired by the ACRL Framework for In-formation Literacy for Higher Education, librarians at the Carl B. Ylvisaker Library at Concordia College-Moorhead decided to scrap our old information literacy assess- ment plan and start from scratch. We began by digging into the library’s past assessment plans in an attempt to gain insight into how the organization’s priorities have changed over time. The earliest assessment notes we uncovered were from 2005 and featured a new student orientation pre- and post-test, administered to a random sample of library users. The focus then was on incoming stu- dents’ information literacy abilities and how much they learned during library orientation. A sketch of an abandoned assessment plan from 2010 marked a turn to assessing the library’s instruction program and featured the use of focus groups along with a survey in which alumni were asked how the instruc- tion program may have impacted their life after college. With this pivot towards instruc- tion, the library developed a list of learning outcomes and tracked them to the ACRL Information Literacy Competency Standards for Higher Education. This list was used in- house for designing assignments and linked on our library web page to provide faculty with an overview of the program. In 2011, the library began a four-year trial with the Standardized Assessment of Informa- tion Literacy Skills (SAILS) test, which was based directly on the ACRL Information Lit- eracy Competency Standards for Higher Edu- cation. Concordia students consistently tested higher than both the Baccalaureate-Liberal Arts group and across all institutions, and the two cohorts tested showed improvement in all categories of the standards during each year of the testing. While this investigation gave us some useful information, it did not provide a clear way forward for measuring student learning in the library. Goals In reimagining our student assessment, we found that several things needed changing. Our main assessment goal is to understand what our students are learning, what skills and dispositions do they come into college with, and how can we build on those to further their information literacy? We want specific information that will allow us to actively change our teaching in a timely and supportive manner. Secondly, we need to adapt to the Framework and make it a seamless part of our instruction and as- sessment of student learning. We value the Framework as a flexible means of respond- ing to campus initiatives, and as an inspira- tion to look at what we do from multiple perspectives. Kevin Baggett is instruction librarian, email: kbaggett@ cord.edu, Virginia Connell is coordinator of library instruction, email: vconnell@cord.edu, and Allie Thome is systems and web librarian at Concordia College, email: athome@cord.edu © 2018 Kevin Baggett, Virginia Connell, and Allie Thome Kevin Baggett, Virginia Connell, and Allie Thome Frame by frame Using the ACRL Framework for Information Literacy to create a library assessment plan Perspectives on the Framework mailto:kbaggett%40cord.edu?subject= mailto:kbaggett%40cord.edu?subject= mailto:vconnell%40cord.edu?subject= mailto:athome%40cord.edu?subject= April 2018 187 C&RL News Methodology Given that our old assessments were not pro- viding us the information we needed and were ill-suited for the new ACRL Framework, we began our new approach to assessment planning by discussing the implications of the Framework for our teaching. This conversation coincided with the campus-wide adoption of an integrative learning model that requires all students to complete two PEAK (Pivotal Expe- rience in Applied Knowledge) projects before graduation. Examples of PEAK projects include creating community digital assets, lesson plans for K–12 students, presentations, shareable re- ports and reflections, and short films screened at the local theater. We felt it imperative to in- clude this campus initiative in building a new assessment plan, and the ACRL Framework made envisioning research assignments within the PEAK environment easier. One of our first library discussions resulted in a set of “decon- structed” assignments emphasizing research skills and research applications to be used in classes without a traditional research paper. Li- brarians tracked these research assignment op- tions to the Framework, to Concordia’s Goals for Liberal Learning, and to PEAK goals and made them available for faculty in campus workshops and online. In part, this idea of deconstructing, or break- ing the whole into more manageable parts, grew from our history of scaffolding library instruction and informed our approach to the Framework. Deciding to start with one frame, we then un- packed the learning practices and dispositions within that frame to build lessons and develop assessments. This seemed a natural approach within the context of our teaching habits. To start, we decided to do more substantive cur- riculum mapping (see the description of our new statistics forms in the next section) and to build the Authority Frame into our required first- year library instruction, the Library Launch. The Launch involves hands-on learning in a face-to- face, librarian-led 70-minute session scheduled by every First Year Inquiry instructor. Activities include brainstorming keywords, class-topic relevant searching, physically locating books, and participating in a reference interview. Li- brarians have curricular control of this session, so we can build a consistent information literacy foundation for the entire first-year class. Curriculum mapping all of our instruction sessions will allow us to expand our assessment methods to the other five frames and across all four years of a student’s time at Concordia. Beyond the Library Launch, mapping our cur- riculum will help us set priorities in first-year written and oral communication classes, so that we may introduce other frames before students move beyond core classes. We will also be learning what frames our librarians teach in upper-level classes by gathering this informa- tion using a new instruction statistics form. This should allow us to better scaffold students’ grasp of each frame and to grow those dispositional habits of mind for lifelong learning. In addition, we are widening our under- standing of “instruction,” finding more ways to emphasize the importance of information literacy across campus. Highlighting aspects of the research process in deconstructed ways has already strengthened our partnerships across campus. Examples include: • providing workshops for Residence Life student employees on finding copyright- permitted images, fonts, and color palettes for the required posters and bulletin boards they build in the dorms; • working with the Student Government Association to provide campus access to the New York Times; and • planning opportunities for paid or for- credit internships that fulfill PEAK requirements. While we have yet to decide the best as- sessment for these partnership instruction op- portunities, if what we are experiencing in our traditional instruction holds true, we are likely to expand our ideas of what assessment means in these contexts. Beyond the classroom We are taking a holistic view of our assess- ment of student learning in the library, which means going beyond the classroom. One of the most common places for student learning to happen in the library is at the reference desk. We have redesigned our statistics form C&RL News April 2018 188 for reference transactions to include a ques- tion that asks whether the reference interac- tion addressed one of the frames. If answered in the affirmative, the reference librarian then selects one or more of the frames that the reference interaction addressed. How we un- derstand and interpret the framework in de- termining if a question falls under one of the frames is an ongoing discussion at reference and instruction meetings. Another place that we have begun to as- sess our incorporation of the Framework in the library is in our Technical Services and Circulation/ILL departments. Our library staff in those departments have extensive interac- tion with students, whether via a service desk or supervising student workers. We have held discussions on the Framework with these supervisors to help us understand how they may indirectly teach the frames to our student workers. We are currently working on a way to track these interactions. We have also begun to assess how students apply the Framework in their research papers. Each year, the library awards the Exemplary Research Award, a prize for the best research paper written by a junior or senior. We re- ceive 10 to 15 submissions each year, which are judged by a campus committee com- prised of three librarians and two teaching faculty using a rubric to grade the student’s research skills. Because this award attracts some of the best upper-level undergraduate research work completed at Concordia, we want to measure how well students have applied and exhibited the concepts of the Framework in their papers. The rubric has always measured student’s research skills, but we have recently made changes to in- clude the language of the Framework for spring 2018. Current status and next steps Our new forms for recording reference and instruction statistics have provided us with an excellent snapshot of how our librar- ians apply the ACRL Framework in their work. Of the 880 reference questions we Figure 1. April 2018 189 C&RL News answered between September 1 and No- vember 27, 2017, our data shows that one third of those addressed at least one of the frames (34.43%, 303 questions). If we break this down further, we find that the majority of those 303 questions specifically address “Searching is Strategic” (Figure 1). We think this makes sense, and that the dispositions and knowledge practices associated with “Searching is Strategic” match best with the kinds of questions we typically answer at the reference desk. In the future, we will continue to discuss how we can make the other frames more transparent during the reference interview. Our instruction statistics show a slightly more balanced application of the frames, though “Information Creation as a Process” and “Scholarship as Conversation” were covered less frequently than the other frames (Figure 2). When we reviewed the knowl- edge practices and dispositions associated with these two, we found that we teach these concepts more in our upper-division classes than in our First Year-level courses. One of our library’s long-term strategic goals is to increase our instruction for upper-level classes, so hopefully we will teach these frames more as we achieve that goal. Based on this data, one of our new long-term goals will be to better incorporate “Information Creation as a Process” and “Scholarship as Conversation” into other required first-year courses on written and oral communication skills, and then to implement assessments in those library sessions. Because we focused this year’s assessment on “Authority is Constructed and Contextual,” we included two additional questions in our instruction assessment form. If librarians indicated they taught the Authority frame, they were asked which dispositions and Figure 2. (continues on page 200) C&RL News April 2018 200 Enabling Programs and Services: Operations • Confirmed the virtual votes approv- ing the minutes of the ACRL Board meet- ings at ALA Annual Conference 2017 and the minutes of the virtual ACRL Board of Directors Fall Meeting held on October 26, 2017. • Confirmed the virtual vote approv- ing to postpone indefinitely the proposed name change of the Value of Academic Libraries Committee. • Amended the policy that the Execu- tive Director is authorized to approve indi- vidual unbudgeted expenditures increasing the amount from $5,000 to $10,000 not to exceed 1% of the total ACRL and CHOICE budgeted expenses in a fiscal year. The Executive Director will notify the ACRL President of such expenditures. • Approved up to $25,000 to co-fund, with the ALA Development Office and other ALA divisions, a six-month prospect researcher. • Approved a Statement on Vendor Relations that articulates ACRL’s aim to engage with all stakeholders in establishing best practices that preserve the integrity of the association while benefiting members and stakeholders. knowledge practices they taught during the session. We found this level of granularity manageable to track for one frame, but anticipate that it will be difficult to assess for all the frames because the form will be- come too long. Because of this, our assess- ment committee is currently amending the library’s assessment plan to make it simpler and less granular. Once this is finished, each frame will have two measurable learning outcomes we can use in our curriculum mapping, instead of trying to measure all the knowledge dispositions and practices. During our Library Launch sessions, we had planned to test students on their use of the “peer-reviewed” check box in our discovery layer to check their understand- ing of the Authority frame. During instruc- tion, students watched a video that defined peer-review and listed some of the markers of authority. In the class activity, we asked students to perform a search, limit to articles, then limit to “peer-reviewed” by checking a box. Most students in the Library Launch successfully limited to peer-reviewed jour- nals, but then applied the concept inexpertly when choosing an article from the list of results. Our data shows us that some students picked a nonscholarly article or review from within a peer-reviewed publication. In the future, we will walk students through an example of what a peer-reviewed article should look like, then ask them a question that more directly assesses their learning of “Authority is Constructed and Contextual.” Through this summative assessment, we will be able to track student learning from year to year. Conclusion Our assessment efforts have yielded in- teresting and fruitful information on how we use the ACRL Framework at the Carl B. Ylvisaker Library. This has helped us in our plans to refine our forms for Reference and Instruction with the aim of collecting less granular data. We also plan to collect more data on how well students are applying the frames in library sessions by including an assessment on “Authority is Constructed and Contextual” in our Library Launch ses- sion. In spring of 2018 we will grade our first batch of Exemplary Research Award papers using an updated rubric, which will provide data on how well our best upper- level students apply the ACRL Framework in their research and writing. Our long- term goal is to implement more assess- ments of student learning in our library sessions, and to incorporate these into a cohesive assessment plan. (“Frame by frame,” continues from page 189)