ACRL News Issue (B) of College & Research Libraries C&RL News ■ July/August 2 0 0 0 / 597 College & Research Libraries news Recipe for disaster or formula for success? Creating and assessing a large scale collaborative library introduction exercise for honors students by Anna Marie Johnson and Melissa Laning W orking with any large group of students in the library is always a challenge. It becomes even more so when the stu number 160 plus and the exercise is sup­ posed to introduce them to all parts of the university library in less than two hours! The logistics alone can cause huge headaches. At the University of Louisville (UL), we have at­ tempted this three years in a row and have learned much about what works and what does not. This article describes the history of the project, our goals, how we have assessed the project’s effectiveness, what we have learned, and what we would recommend. Our previous attempts Working with the Honors sections of Gen­ eral Education 101, the university’s introduc­ tion to campus life, has provided us with a chance to try out instructional methods and content on a large scale. The Honors section classes take place over a three-day period during the weekend before the fall semester begins. In 1997, three instructors from the univer­ sity libraries presented simultaneous sessions to groups of ten to fifty. The sessions dealt only with researching on the Web and using university e-mail. Though the sessions were de productive, it was impossible to know ex­ actly how effective they were, because no ntsformal assessment was conducted. The an­ ecdotal feedback was that the students en­ joyed it, but they wished it had included more about the libraries. Based on this limited anecdotal feedback, we revised the course content for the follow­ ing year by creating two different sessions. The first was held on a Friday morning and consisted of three consecutive sessions called “Critical Evaluation of Web Information.” Each session was taught by a different li­ brary instructor, but the content was the same: a PowerPoint presentation of the key factors relating to the evaluation of information and an exercise using printed copies of two Web pages to be compared for their accuracy, authority, currency, coverage, and objectiv­ ity. The auditorium-style room and the lack of hands-on capability made the session frus­ trating. The second part of the students’ orienta­ tion to the libraries consisted of a Saturday morning “scavenger hunt” exercise. This ex­ ercise required them to answer questions about various resources in different depart­ ments of the main library and then get an “information passport” stamped by someone About the authors Anna Marie Johnson is coordinator fo r library instruction and Melissa Laning is team leader o f assessment and resource planning a t the University o f Louisville, e-mail: annamarie@louisville.edu; malani01@gwise.Iouisville.edu mailto:annamarie@louisville.edu gwise.Iouisville.edu 598 / C&RL News ■ July/August 2000 in that area. Although this introduced the stu­ dents to various parts of the library, it was problematic for several reasons. First, the li­ brary was scheduled to be closed on the day o f the scavenger hunt, so extra personnel were required to open the building and staff it. Second, the size of the group made “traffic jams” a problem. Too many students would crowd an area, all looking for the same re­ source or all asking questions at the same time. There was also no assessment and no group discussion of the experience. The only feedback was again anecdotal, and the stu­ dents reportedly said that they found the e x e r c is e “b o rin g ” and “not c h allen g in g enough.” Learning from our mistakes While the scavenger hunt did not necessarily qualify as a disaster, obviously some new ideas and techniques were needed. It was obvious more planning was needed, espe­ cially in terms o f logistics. It would also be important to make it more engaging for the students. Our audience was Honors students who generally tend to be self-motivated and to expect more challenging material. We worked with the critical Web informa­ tion piece to make it more participatory and “active-learning” oriented. We moved to an­ other room on campus and broke the stu­ dents in discussion groups. The exercises centered around “information dilem m as,” such as “term paper mills” on the Web, and the students were asked to discuss and then share the contents o f their discussion with the rest o f the group. We also changed the format of the scavenger hunt to make it more in-depth and interesting. The students were divided into eight dif­ ferent themes: Science & Technology, Life in the ’80s, Violence, Other Cultures, Wealth, the Arts, Documentary Photos, and Tarzan. The themes were developed by the librar­ ians who created the exercises for them. The thinking was that a theme for the ex­ ercises would provide an opportunity to en­ gage the students in topical discussions. This divided the l6 0 students into groups o f 20. Thinking twenty was still too large, each theme was divided into smaller segments of five students each. For example, the theme o f Violence was broken into Murder, Terror­ ism, Gangs, and Violence in the Media. This L i b r a r y e x e r c i s e t ip # 7 Organizing the exercises around them es helped the librarians generate questions and made the exercises more interesting for the students. way, no more than five students would be likely to be looking for a specific reference book or other resource. At a planning meeting in May, it was de­ cided not to have a specific number o f ques­ tions for the assignment, but to design the questions in such a way that each small group visited at least three or four different areas of the library. The instmctors for this project came from all areas o f the library: Media and Current Periodicals, Information Literacy, Reference, Tech nical Services, and Rare B o ok s and Photoarchives. This allowed for a diversity o f viewpoints as well as expertise. Each vol­ unteer instructor was assigned a theme and was allowed to create his or her own sub­ themes and questions. Questions were shared among the instructors, but only a few o f the questions were standardized across the whole group. This allowed for flexibility and made the most o f each person’s expertise. Some instructors simply created questions relating to their theme, while others created scenarios to help engage the students’ attention and to give context to the questions. L i b r a r y e x e r c i s e t ip # 2 Involving staff from all areas of the library helped generate variety and interest. The questions ranged from the very sp e­ cific, such as “Using the S ta tistic a l A b stra ct o f t h e U n ited States, find w hich state had the highest num ber o f murders in 199 5 ” to the m ore gen eral, “Brow se in the LC call number section o f the reference area that you think is most related to your topic. Write C&RL News ■ July/August 2000 / 599 down the citation for one book that you find there that you think might help with research­ ing this topic. Tell why you think so.” Although the flexibility and variety were nice, there was little standardization and little guidance as to how to write the questions. This may have created som e discrepancies in the students’ experiences. Instructional goals The scavenger hunt con cept is frequently problematic in library instruction. It can de­ generate very quickly into a hide-and-seek game, where the only goal is to find the an­ swer regardless o f the means or the purpose o f the exercise. In writing our questions, the goal was not simply to get the students to use a specific resource, although that would b e beneficial. It was also designed to encour­ age them to think critically about the theme and subtopic they w ere given and also about research in general. W e wanted the students to use higher-level thinking skills and not just write down the first book or answer they cam e to. W e wanted them to think about why they might actually use a particular book, what about it made it look useful? When w e had them search article databases, we Ingredients for success Think through these issues as you de­ velop your library introduction program. P o p u la tio n . Obviously Honors stu­ dents are a unique population o f students and may have more characteristics in com ­ mon, making it easier to write an exercise. Consider the population you want to work with. If possible, find out information about them beforehand. Have they had library experience before? If so, what type? Do they own a PC? How will they use the li­ brary in their course o f study? In s tru c tio n a l G oals. Have some! Write out a list o f goals and objectives before you begin planning and then half it! Take into consideration the other factors you are working with like time and population. What can be reasonably accom plished and what do your students need to know at this point in their education? E x e r c is e s . W hen you write your exer­ cise, keep in mind that the students do not need to know everything by the end o f the day. Ask yourself if you want the stu­ dents to know a few sources well or to have an overview o f a broad range o f sources. This may depend on your popu­ lation and the goals you have made. Also if you are writing the exercises as a group, create som e guidelines and a standard for­ mat. Som eone working in the main library may have a very different perspective on instaiction than som eone working in a pro­ fessional library. Balan ce consistency with flexibility. Although the terms “critical think­ ing” and “active learning” are overused, they do apply here. Rather than have stu­ dents learn sp ecific sou rces, help them learn how to find a source on any topic. Keep them engaged by getting them mov­ ing and thinking. Tim e. The students’ time is valuable. Make certain that what you are asking them to do is not simply a time-filler, but a use­ ful exercise and then make sure th ey k n o w it. L ogistics. This was a key issue for us. If your group is larger than ten people, you may easily have a problem if they are all looking for the same resources. There are also logistical concerns in getting them to the “starting line," so to speak. We used a c olo r- cod ing system with ea ch group, which helped alleviate som e o f the confu­ sion. A ssessm en t. Creating a meaningful and learner-centered evaluation tool is much easier w h en you have created sp ecific learning goals for the session. Although it is valuable to collect som e information on how well the instructor is doing, the most important thing to find out for the purposes o f improving instaiction is how well the student is doing. F e e d b a c k . Use the inform ation gath­ ered through assessm en t to revise your instruction. Also, avoid letting anecd otal ev id en ce from o n e or two p eo p le ov er­ shad ow inform ation co lle cte d from the entire group. 600 / C&RL News ■ July/August 2000 L i b r a r y e x e r c i s e t i p # 3 We wanted the students to think about library research as a concept, not just the day's activity. asked questions about the publication date and how that would affect their topic and whether they would use the article. We also asked about the name o f the journal, if they knew its reputation, and why that might be important. T o move beyond the scavenger hunt concept as we understood it and to chan­ nel the students’ thinking about library re­ search, were key aspects o f our questions. Another primary goal was to help the stu­ dents discover parts o f the library with which they might not otherwise com e into contact. For example, Government Publications, Photo Archives, and Rare Books are rich resources that many students never use. However due to time constraints (the whole exercise was 50 minutes) and the sheer number o f stu­ dents, it was impossible for all students to experience all parts o f the library. To remedy this, w e created a framework around the 50-minute exercise. The groups o f 20 would com e together at the start for a short orientation/explanation/pep talk for roughly 20 minutes. At the end o f the 50- minute exercise, there would b e a ten-minute preparation period and about thirty minutes for the groups o f twenty students to share with one another what they had learned about the different parts o f the library through short oral presentations. To help them prepare for this, the students were given a variety o f guid­ ing questions, such as “Describe your topic briefly”; “What did you have problems with and how did you solve them?”; “What factors affect what resources you use?”; and “Where would you start your next research paper?” The sharing period was meant to allow the students to learn from others’ experiences as well as their own. Feedback and assessm ent At the end o f this session, the students com ­ p le te d a te n -m in u te e v a lu a tio n o f the morning’s activities. Based on our experience, w e knew that feedback from students was extremely valuable in the development o f meaningful and effective instruction sessions; however, we had relied in the past on infor­ mal or indirect information. This time w e cre­ ated a brief survey to elicit direct and focused feedback from students regarding their ex­ perience. The first consideration in our survey de­ sign was deciding what we wanted to learn. Many instruction evaluation surveys ask a series o f questions to determine whether the respondent was satisfied with the instruction. B ecau se feedback from the previous year indicated that the session was boring and too easy, we wanted to know whether the re­ vised approach improved the quality o f the students’ experiences. We were also looking for clues to possible future enhancements. Assessment in academic libraries is also in­ creasingly interested with finding out what students actually learn during information lit­ eracy classes.1 Because student learning is critical to our instruction mission at the UL, this was an important area to investigate fur­ ther. On a m ore practical level, w e also w a n te d to fin d o u t w h ic h o f o u r nonstandardized exercises was most success­ ful in conveying basic information evalua­ tion concepts and library-use skills. The second consideration in the survey design was related to length and format. Given the limited amount o f time scheduled for the evaluation and the fact that it was right before lunch, we knew the survey had to b e short and to the point. For that reason, we limited the evaluation form to one, two- sided page and used mostly multiple choice questions. UL Honors Program Expedition perception question and results 1. T he lib ra ry ex p e d itio n in tr o d u c e d m e to lib ra ry r e s e a r c h tools a n d co llectio n s th a t I d i d n o t a lr e a d y k n o w a b o u t: # o f % respondents Strongly disagree 0 0% Disagree 4 3 % Agree 86 56% Strongly agree 64 42% C&RL News ■ July/August 2000 / 601 The evaluation had three sections. In the first section, we asked the students to iden­ tify their team and topic, and to indicate pre­ vious library usage. In the second section, we asked them to rate from “Strongly Agree” to “Strongly Disagree” a number o f statements about their experience, such as “There was enough time to complete the library exercise for my topic.” The students were also asked specifically about the level of difficulty for their session. In the third section, we asked a series o f multiple-choice questions aimed at discovering if the students knew when to use the libraries’ catalog, when to use an index, and how materials are organized in the li­ brary.2 The UL Libraries have the Bubble Pub­ lishing software, created by Scanning Dynam­ ics, Inc., which allowed us to design and print our own evaluation forms 3 The software also allows us to scan the results and create a simple report from the information collected (see sidebar). Overall, the evaluation results showed that the 1999 Honors Library Expedition was a successfu l program. We w ere esp ecially pleased to see that 88% o f the respondents thought that the level o f difficulty was just right. More importantly, the information will help to make future adjustments to the pro­ gram format and content. For example, on the “experience satisfac­ tion” type questions, we found that almost one-third of the respondents did not feel they had enough time to complete the library ex­ ercises, even though 97% o f them thought they had learned something useful for their classes. We hope to have a longer time period for future Library Expeditions, but if not, we may to need to make the exercises shorter. On the “library knowledge” portion of the evalua­ tion, we found that the most confusion cen­ tered on a question about what cannot be lo­ cated using the library catalog. Using the re­ ports module o f the Bubble Publishing soft­ ware, we can identify which exercises led to more or less accurate responses in this area. This information can be used to shape the development of new exercises. Use o f more formalized evaluation this year allows us to establish a baseline for fu­ ture reference and a way to measure progress. One of the main enhancements planned for next year is the development of specific learn­ ing objectives that will be used for creating/ revising the exercises. The question will be how to retain the creativity and flexibility of the exercise, while giving more structure to the question creation aspect. Also, this event took an extraordinary amount o f planning time on the part o f the volunteers. Will each exercise need to be recreated each year? Can we build on what we already have? These are issues that we need to address. Another possibility would be to combine the General Education 101 experience with a composition class so that the research would have a concrete purpose and would be an integral part o f the curricu­ lum o f the class. While this year certainly was far from a disaster, we will work on our “formula” for next year, adding some of the above elements in an attempt to continually improve the li­ brary experience for this large and vital group o f students. Notes 1. ACRL Task Force on Information Lit­ eracy Competency Standards. “Information Literacy Competency Standards for Higher Education (Draft), Ju n e 1999.” The final, approved version was published in the March 2000 issue o f C&RL News. 2. Two good sources for the format and content of instruction evaluations are: Diana D. Shonrock, ed., Evaluating Library Instruction (Chicago: ALA, 1996). Wanda K. Jo h n sto n , ed., L ibrary a n d L earn in g R esou rce Program s: E valuation a n d S elf Study (Chicago: ACRL, 1998). 3. For more information about Bubble P u b lis h in g S o ftw a r e , s e e : http:// bubblepublishing.com/WwwOffice.htm. ■ UL Honors Program Expedition knowledge question and results 2. To f i n d a b ook o r p eriod ic al ow n ed by the Ekstrom Library, you should use: # o f % respondents Alta Vista 0 0% Reader's Guide 0 0% Minerva 2000 149 97% ProQuest 5 3% 602 / C&RL News ■ July/August 2000