College and Research Libraries ROLAND PERSON Long-Term Evaluation of Bibliographic Instruction: Lasting Encouragement There is a recognized need for evaluation of bibliographic instruction, par- ticularly the long-term effects. This study of a semester-long credit course over a six-year period shows that student appreciation of such bibliographic instruction not only is high at the time of instruction but also frequently increases during the years after the course has been taken. Regardless of their initial reasons for taking the course, students find the instruction valu- able and also recommend such a course to their friends. LIBRARY INSTRUCTION PROGRAMS have a long history in this country, going back even to the nineteenth century, 1 and the in- creasingly lengthy series of annual bibliog- raphies produced by Hannelore Rader indi- cates that the field continues to expand in the attention devoted to it. 2 The concern for means of evaluating these programs, howev- er, is much more recent and, when it does appear, is noteworthy more for the lament at its lack than for the details of its success. As late as 1976, Brewer and Hills could observe that "there are few references to evaluation in the literature of reader in- struction and until very recently they have been virtually non-existent. "3 In a similar vein, J. Martyn calls evaluation that area "rich in speculation but uncommonly poor in demonstrable fact. "4 In recent years, what attempts there have been to develop effective evaluation methods have generally focused on quanti- tative measurement of brief periods of in- struction, i.e., one or more course-related lectures, or pre- and posttesting surround- ing a few hours of instruction. There also Roland Person is assistant undergraduate librar- ian, Southern Illinois University , Carbondale. have been attempts to compare the quality of various formats for presenting library in- formation, e. g., lectures, hands-on experi- ence, team-taught classes, etc. 5 In all of these cases evaluation of the instruction is done at the time, and the question of the long-term effects on student awareness and performance has not been considered. A number of librarians have noted the need for drawing upon the literature and experiences of others in education for assis- tance and guidance in instructional prob- lems, including evaluation and the theory of learning. 6 This would seem especially help- ful in the type of library-use instruction that most resembles other instruction, namely, the full-term course in bibliographic instruc- tion, particularly when offered as an elec- tive and for credit. Yet here it seems the literature on evaluation is virtually nonexis- tent. Perhaps this is because many agree with the opinions of one instructor of such a library course who says "the assessment and evaluation of this is exactly the same as in any other academic course, "7 without pro- viding further details. On the other hand, a number of librar- ians argue that user education is not like other courses, that it is "a skill to be de- veloped, not a subject to be taught"; that I 19 20 I College & Research Libraries • January 1981 the student "must learn how to learn in the future rather than aim at acquiring a body of fact-information" ; and that "learning to use the library is a continuing process. "8 Such a philosophy holds that what is taught (and learned) is a philosophy, an attitude, a strategy or method of seeking information, and thus a quantitative time-of-use evalua- tion would be inadequate or even in- appropriate. What is needed is an evalua- tion of the long-term effects of courses in bibliographic instruction and their effects on students ' later academic attitudes and achievements. If a freshman takes such a course, how does he or she, as a senior, or even a graduate student, look back at its effects on subsequent work? Although sub- · jective , such an evaluation would be more interested in identifying attitudes than in measuring fact retention . 9 With these con- siderations in mind, the author began a study to ascertain the long-term effects and the attitudes of students who had taken a full-term credit course in bibliographic in- struction at one university. In the fall semester of 1974, Morris Li- brary at Southern Illinois University, Car- bondale, began offering GSD 199A, "The Library as an Information Source ." This one-hour, one-credit course was listed in the catalog under the general studies core and satisfied part of those requirements. Three sections, taught by librarians, were offered that first semester, each limited to an enrollment of twenty students. Since that beginning, the course has grown to the point that each semester the library now offers twelve or thirteen sec- tions of the course ; through spring of 1978, some 1,374 students have received a grade in the course. Although one department (the Center for Basic Skills) did for a few semesters require its students to take the course, most students enroll voluntarily for a variety of reasons. Most are freshmen at . the time of enrollment, but there have been some students from all grade levels, includ- ing Ph. D . candidates. Instructors have been drawn from all areas of the library: public services, technical services, and administra- tion . There is a brief, basic syllabus, but each instructor is free to develop the course as desired. The undergraduate librarian coordinates all scheduling and meets' several times each semester with the teachers as a group. In the spring 1979 semester, with the support of the library administration , the author set out to test some hypotheses · about the course that had gradually de- ' veloped, and to try to ascertain how the course might be improved , in the opinion of its graduates . The first hypothesis was that the higher the class level of the respon- dents , the more likely they would be to appreciate the course. This seemed prob- able because a senior, for example, who had taken the course as a freshman would have had more opportunities to apply the library knowledge in a variety of other classes and information problems than a sophomore would. Put another way, we believed that : over time, students' appreciation of the course would continue to increase . A second hypothesis was that regardless of their reason for taking the class, most graduates would come to a similar apprecia- tion of its value . Even if some students took the class merely for the credit, we believed they would ultimately value it as much as someone with seemingly better motives at the beginning. Finally , we wanted to see if distance from the course would provide any different, or more objective, suggestions for ways of im- proving the course compared to those we received during the course evaluation at the end of each semester. The author prepared a four-part qu estion- naire containing twe nty-six items . Part I dealt with biographic and e nrollme nt data; part II concerned specific re actions to the course, using a four-part scale ranging from "strongly agree" to "strongly disagree"; part III consisted of three open-ended questions about strengths and w e aknesses of the course ; part IV asked for suggestions. A summary of part II is displayed in figure 1, and of part III in figures 2 and 3. Using computer-generate d enrollment lists, we were able to determine the num- ber of students currently enrolled who had ever taken GSD 199A. A total of 730 ques- tionnaires was sent out, to both on-campus and off-campus addresses, of which 71 were not deliverable. Thus 659 one-time mailings were sent. No coding of response sheets was done, also to preserve anonymity. A Long-Term Evaluation of Bibliographic Instruction I 21 Cumulative Percentage* Agree Disagree 1. GSD 199A has not helped me in other classes. 21.9 78.1 2. As a result of this course I am a more confident library user. 88.8 11.2 3. I would recommend this course to a friend. 85.0 15.0 4. When I need assistance in the library now, I specifically ask for a librarian to help me. 66.7 33.3 5. I believe my friends who have not had this course are less skilled at using the library than I am. 63.0 37.0 6~ The course helped me feel more comfortable in asking a librarian for help . 78.9 21.1 7. As a result of what I learned in the course, I have helped other students use the library. 75.7 24.3 8. I use the library more now than I would have if I had not taken the course. 45.7 54.3 9. I would not like further training in using the library. 40.4 59.6 10. I have recommended the course to someone else. 69.5 30.5 11 . I would be interested in an advanced library course if it were offered . 48.8 51.2 12. The course should meet more often than once a week. 33.1 66.9 13. The course should be worth more than one credit. 61.3 38.7 *For easier display , respondents' choices of "strongly agree" and "agree " have been combined , as have "dis- agree" and "strongly disagree." Fig. 1 Part II total of 169 questionnaires was returned , for a 25.64 percent response rate.* Students ' ages at the time of response ranged from 18 to 35, with a mean of 21.4; most were between 19 and 23. Seventy-four respondents were male , ninety-five· female; 92 percent were enrolled full time ; seventy- six lived off campus. Most of the respondents had taken the course as freshmen and were now upper- classmen (table 1). Majors and departmental affiliations were widely spread across the colleges, with no more than 20 percent of the total coming from any one area; thus a good cross section of the university popula- tion had been represented by the enroll- ment in this course. Although it is obvious that students in a class taught by a librarian will then recog- nize at least one librarian , one of the empha- ses of the course is that librarians them- selves are a major source of information for students. Thus, it is gratifying to note that *Although at first glance 26 percent may seem a low response rate, one should remember that this was a one-time mailing without follow-up ; the subject was not itself of great importance to students; students in general present difficulties for mail delivery because of frequent address changes and the possible vagaries of campus mail service. some two-thirds of the respondents now specifically ask for a librarian when they need help. In addition, they acknowledge a need to distinguish between librarians and nonprofessional library staff when seeking assistance . Closely related to this distinction is the often-recognized reluctance to admit ignorance by asking for help . Again , more than three-fourths of the students from the course acknowledge that they now feel more comfortable in asking for assistance from a librarian (table 2). In addition to being more willing to ask for help , students who have taken the course clearly have used their knowledge of library skills to help other students. Three- fourths of the respondents spoke of helping other students use the library and directly tied this confidence to their having taken the library course. Moreover, some 85 per- cent said they would recommend the course to a friend (table 3). Putting thought to ac- tion, nearly 70 percent actually had recom- mended the course to at least one other person (table 4). Perhaps because of such word-of-mouth advertising, there have been more students wanting to enroll than could be accommodated in the course each semes- ter it has been offered . Although the most common complaint about the course is the amount of work re- 22 I College & Research Libraries • January 1981 Please list up to three things you feel were major advantages, strengths, and desirable features of the course. [Note: this is an open-response question. For easier display the author has summarized the responses by grouping them into categories.] Percentage of Respondents* 24.9 27.8 30.8 15.4 8.9 7.7 8.9 3.6 5.3 29.6 Taught how to find material Learned how much was available in library Physical layout and location of material Hands-on experience Index to periodicals Classification schemes Card catalogs Handouts Audio-visual materials Other (didn 't fit categories devised above) *Total is more than 100 percent because responden ts could indicate more than one item . Fig. 2 Part III A Please list below up to three things which you felt were disadvantages , weaknesses, or undesirable features of th e course . [Note : this is an open-ended question. For easier display, the author has summarized the responses by grouping them into categories . ] Percentage of Respondents* 17.8 13 .0 8.3 18.9 4. 1 16.0 7.1 4. 1 2.4 3.0 16.0 Time ; class should meet longer or more often Too much work for one hour Not enough credit hours Too general ; not specialized enough Teacher Not stimulating Classroom crowded Classmates hindered learning Tours Readings Other (didn 't fit categories d evised above) *Total is more than 100 pe rce nt because respondents could indicate more than one item. Fig. 3 Part III B quired for just one credit, many recognize the quantity and value of the knowledge and skills involved in information searching. Nearly half would be interested in a more advanced course, and some 40 percent would like further bibliographic training. These results, combined with some of the comments appended to responses, indicate a considerable appreciation of the need for increased emphasis on major- or course- related instruction in which more advanced bibliographic techniques could be related closely to the individual's particular needs. The various divisions of the graduate library at SIU already offe.r much course-related in- struction through lectures to classes . And, TABLE 1 Freshman Sophomore Junior Senior Grad . · Unclassified Class in school when enrolled in GSD 199A 74% 17% 7% 2% Class in school now 2% 34% 26% 34% 3% 1% Long-Term Evaluation of Bibliographic Instruction I 23 TABLE 2 RESPONSE TO : "THE COURSE HELPED ME FEEL MORE COMFORTABLE IN ASKING A LIBRARIAN FOR HELP" (N = 156)* Sophomores Juniors Seniors Percentage of respondents Numbe r Agree 38 38 45 77 .6% Numbe r Di sagree 18 5 12 22.4% *Not eve ry res ponde nt answe red each qu es tion . Although the re we re 169 qu es tionnaires r e turn e d th e N (numbe r) on each tabl e is the total response for that ite m onl y. TABLE 3 RESPONSE TO : "I WOULD RECOMMEND THIS COURSE TO A FRIEND . " (N = 157) Sophomores Juniors Seniors Percentage of respondents TABLE 4 Numbe r Agree 40 42 51 84.7% Number Disagree 16 2 6 15.3% RESPONSE TO : "I HAVE RECOMMENDED THE COURSE TO SOMEONE ELSE. " (N = 154) Sophomores Juniors Seniors Percentage of respondents Numbe r Agree 27 32 46 68 .2% Numbe r Disagree 27 11 11 31.8% of course, some university departments offer bibliographic courses in their own literature. Nevertheless , the survey indi- cates a definite interest in more biblio- graphic instruction and should provide im- petus for both librarians and other faculty to investigate other ways to help meet these needs. In listing the three greatest strengths of the course, with no cues provided on the questionnaire (figure 2), students more often mentioned not only the knowledge of physical location of material (55 percent) but also the realization of just how much a li- brary has to offer (28 percent). They also frequently referred with pride to their new- ly learned ability to find material on their own, knowing the assistance of librarians was readily available when they needed it. Too little time and too much work were the expected responses mentioned as unde- sirable features of the course (figure 3). · However, an unexpected reaction was that of the 19 percent who thought the course was too simple, or too general and not thor- ough enough. Perhaps we have misjudged the students' abilities here. At any rate, we may need to reexamine the syllabus to con- sider broadening rather than simplifying the content. The last class period of each section of the course includes an evaluation on an optical-scanner form designed for use in all university courses. Twenty-six percent of the questionnaire respondents indicated in some way that they have become more appreciative of the value of the course than they were at the time of the in-class eval- uation. This response is in addition to the 20 percent whose opinion of the course hadn ' t changed over this time (and may have been quite favorable in the first place). Overall, some 89 percent of the respon- dents agreed that they had become more confident users of the library as a result of this course. Breaking the response down by class level at time of response, we see that 84.2 percent of the sophomores , 93.2 per- cent of the juniors, and 87.9 percent of the seniors agreed with this judgment. These students clearly felt that they had gained a skill that was lacking in their friends who had not had this instruction (table 5). In re- sponding to the question of whether the course had helped them in other classes, 68.4 percent of the sophomores agreed, 84.1 percent of the juniors agreed, and 82.8 percent of the seniors agreed. When asked whether, as a result of this course, respon- dents had helped other students use the li- brary, 64.9 percent of the sophomores, 75 percent of the juniors, and 86.2 percent of the seniors agreed (table 6). Lastly, 50 per- cent of the sophomores, 74.4 percent of the juniors, and 80.7 percent of the seniors in- dicated that they had recommended the course to someone else. In general, statistical manipulation of the data shows significant support for the first hypothesis, namely, that appreciation of the course would increase over time. In cases in which there was no significant difference between class responses (as in the confi- dence in library skill as a result of the course), the reason is that satisfaction was high at the time of the course and has remained 24 I College & Research Libraries • january 1981 TABLE 5 RESPONSE TO: "I BELIEVE MY FRIENDS WHO HAVE NOT HAD THIS COURSE ARE LESS SKILLED AT USING THE LIBRARY THAN I AM ." (N = 155) Sophomores Juniors Seniors Percentage of respondents TABLE 6 Number Agree 27 31 39 62.6% Number Disagree 29 11 18 37.4% RESPONSE TO: "As A RESULT OF WHAT I LEARNED IN THE COURSE , I HAVE HELPED OTHER STUDENTS USE THE LIBRARY. " (N = 159) Sophomores Juniors Seniors Percentage of respondents umber Number Agree Disagree 37 33 50 75.5% 20 11 8 24.5% high even as much as three years later when seniors look back at their freshman experience. This indeed is welcome and en- couraging news for library-use instructors. The second hypothesis turned out to be nearly impossible to test statistically. By allowing students to indicate more than one reason for taking the course, the question- naire made it impossible to use statistical tests of correlation. However , the results noted above tend to support the conclusion that, regardless of the reason for taking the course, most students came to a similarly strong appreciation of its value. This is borne out by the consistently high figures cited in the preceding paragraphs. Perhaps ironically, this strong satisfaction with the course made it difficult to ascertain any clear weaknesses in the content or method of the course. Less than one-fifth of the respondents agreed on any one iden- tified weakness (remember that the ques- tion was open-ended; no suggested re- sponses were presented). As noted earlier, the most frequent complaints had to do with the time allowed (one hour for one credit). Although the results of this survey do not specifically support all the initial hypoth- eses, they do in fact support the goals and objectives of the course. They indicate that the great majority of students appreciate the value of bibliographic instruction as a formal course, that this appreciation remains high for years after the course was taken, and that this appreciation is evidenced by stu- dents recommending the course to other students. Librarians who are involved in formal bibliographic instruction programs might take heart in these results and consider si~ilar long-term testing of their own pro- grams as one means of further justifying their value. 10 Though this study is hardly definitive , it is a beginning in an area that needs more and better research. If we be- lieve students know a valuable course when they see one, we may conclude that courses in bibliographic instruction are appreciated by those for whom they are designed. We need to document that appreciation, and the reasons for it, in order to improve such instruction.* *The author will gladly furnish details of the questionnaire and statistical tests used to anyone wishing further information. REFERENCES 1. Ravindra N. Sharma, " Bibliographic Educa- tion: An Overview ," Libri 29:329-41 (Dec. 1979); John Mark Tucker, "The Origins of Bibliographic Instruction in Academic Librar- ies , 1876-1914," in New Horizons for Academic Libraries , papers presented at the First National Conference of the Association of College and Research Libraries, Boston, Massachusetts, November 8-11, 1978 (New York : K. G. Saur , 1979), p.268-76 ; L . C. Pugh , " Library Instruction Programmes for Undergraduates: Historical Development and Current Practice, " Library World 71:269 (March 1970). 2. Hannelore B. Rader , " Library Orientation and Instruction-1978 : An Annotated Review of the Literature," Reference Services Review 7:45--56 (Jan./March 1979). The bibliography for 1977 appeared in RSR 6:45-51 (Jan./ March 1978) ; 1976 in RSR 5 :41-44 (Jan./ March 1977); 1975 in RSR 4:91-93 (Oct./Dec. 1976); 1931-75, ed. Maureen Krier, in RSR 4:9--31 (Jan./March 1976). 3. J. G. Brewer and P. J . Hills, "Evaluation of Long-Term Evaluation of Bibliographic Instruction I 25 Reader Instruction ," Libri 26:56 (March 1976). 4 . M. B. Stevenson, User Education Pro- grammes : A Study of Their Development, Organization, Methods, and Assessment, British Library Research and Development Report No. 5320 (London: British Library, 1977), p.27. 5. Larry Hardesty, Nicholas P. Lovrich , Jr. , and James Mannon , " Evaluating Library-Use In- struction, " College & Research Libraries 40:309-17 (July 1979); Frank F . Kuo, " A Comparison of Six Versions of Science Li- brary Instruction ," College & Research Li- braries 34:287-90 (July 1973); Carla J. Stoffie, " Library Instruction : The University of Wis- consin- Parkside Experience ," in Hannelore B. Rader , ed., Academic Library Instruction: Objectives , Programs , and Faculty Involve- ment , Library Orientation Series No. 5 (Ann Arbor, Mich .: Pierian Pr., 1975), p.27-44. 6. Mary Jo Lynch, "Trials, Tactics , and Timing: Some Thoughts on Library Instruction Pro- grams ," in Sui H. Lee, ed . , A Challenge for Academic Libraries: How to Motivate Stu - dents to Use the Library, Library Orientation Series No . 2 (Ann Arbor, Mich .: Pierian Pr., 1973), p . ll ; Thomas Kirk , " Evaluation of Li- brary Orientation and Instruction Programs : A Taxonomy, " in Mary Bolner, ed., Planning and Developing a Library Orientation Pro- gram, Library Orientation Series No. 3 (Ann Arbor, Mich .: Pierian Pr ., 1975), p.41-5l. 7. Mignon Adams, "Effects of Evaluation on Teaching Methods," in Carolyn A. Kirken- dall, ed., Improving Library Instruction: How to Teach and Evaluate, Library Orientation Series No. 9 (Ann Arbor, Mich.: Pierian Pr. , 1979), p.97. 8 . The Education of Users of Scientific and Technical Information, report from a work- shop held at the University of Bath, Septem- ber 14-16, 1973 (Bath University Library, 1973), p.18; Harold E. Potts , "Instruction in Bibliographic Technique for University Stu- dents ," in Association of Special Libraries and Information Bureaux, A Report of Pro- ceedings of the Conference (1926), p.89; Ver- na V. Melum, "1971 Survey of Library Orientation and Instruction Programs," Drexel Library Quarterly 7 :227 (July and Oct. 1971) . 9. For discussions of what is called "illuminative evaluation," see Malcolm Parlett and David Hamilton, "Evaluation as Illumination," in David Tawney, ed . , Curriculum Evaluation Today: Trends and Implications (London: Macmillan, 1976), p.84-101; Colin Harris, " Illuminative Evaluation of User Education Programmes," Aslib Proceedings 29:348-62 (Oct. 1977). 10. For optimistic encouragement in spite of the elusive nature of evaluation , see Peter Taylor, " User Education and the Role of Eval- uation, Unesco Bulletin for Libraries 32:252- 59 (July-Aug. 1978).