Undergraduates’ Use of Google vs. Library Resources: A Four-Year Cohort Study 614 Undergraduates’ Use of Google vs. Library Resources: A Four-Year Cohort Study Carol Perruso Carol Perruso is Associate Librarian for Social Work and Journalism, and Collection Development Officer in the University Library at California State University, Long Beach; e-mail: carol.perruso@csulb.edu. © 2016 Carol Perruso, Attribution-NonCommercial (http://creativecommons.org/licenses/by-nc/3.0/) CC BY-NC. This longitudinal study at a large public university surveyed students of the 2008 freshmen cohort over four years about their use of websites and library resources for their research papers. The three goals of the study were to track changes in reported research behavior over time, to see if students’ reported source choices were associated with librarian instruc- tion and/or if they were associated with instructors’ source requirements. The study found that, as students matured, they used library resources more frequently. Librarian instruction and faculty source requirements both were associated with increased use of library resources. cademic librarians have been assessing the impact of instruction for years, but few studies have looked at undergraduate research behaviors longitu- dinally—and none for a cohort over four years. By surveying a convenience sample of a 2008 class of first-time freshmen each semester for four years, we hoped to develop a better understanding of how undergraduates’ choices of sources changed over time, and how those changes might be associated with librarian instruc- tion and requirements by instructors on the types of sources they used. Specifically, this study by five librarians1 explored our frequent observations that many undergraduates quickly retreat to Google-dominant searching after instruction in the use of scholarly databases, sometimes even during the instruction session. The initial research question was, does Google reliance change over time? Using survey data from 386 first-semester freshmen and more than 75 students in each of the next seven semesters, we found that research behavior does change. Literature Review In her 2009 literature review, MacMillan found that few longitudinal studies have examined undergraduate use of web and library resources for academic research, or the development of student information literacy skills over time, and a recent review of the literature by the author found the same to be true since then.2 Most of the longitu- dinal studies looked at change over relatively short timespans—one to two years.3 The author could find none that tracked a cohort of students over four years. In addition to reviewing longitudinal studies, this literature review focuses on undergraduates’ doi:10.5860/crl.77.5.614 crl15-826 Undergraduates’ Use of Google vs. Library Resources: A Four-Year Cohort Study 615 research paths and, more specifically, on how those paths affect source choice. It will also touch on research about the effects of information literacy instruction and faculty source requirements. Research Path In the early 2000s, about five years after the web became rooted in the public experi- ence4—making it likely that many students had used the Internet before college—a few large cross-sectional studies found that web search engines overshadowed library resources for college students’ academic research.5 A 2002 OCLC study surveyed a nationwide sample of 1,050 mostly undergraduate students and found that 79 percent said they used web search engines for “most” or “every” assignment, while 70 percent said they used library websites for “at least some of their assignments.”6 Kaminski, Seel, and Cullen surveyed 2,102 freshmen at Colorado State University, 58 percent of whom said they went to a web search engine first when researching a paper. Only 23 percent said they consulted library resources first.7 In the Pew study on the academic uses of the Internet by 1,032 degree-seeking students at 27 colleges, Jones and Madden found that 73 percent of students reported using the Internet more than the library, while only 9 percent said they use the library more than the Internet.8 A four-year study, from 1999 to 2003, by the British Joint Information Systems Com- mittee (JISC) surveyed hundreds of students about Internet and online resource use with different samples each year. In the early surveys, undergraduates consistently ranked web search engines No. 1 in frequency of use, with library resources ranked much lower. By the fourth year, web search engines were still the preference; however, bibliographic databases had moved up to second place among undergraduates, with 23.2 percent reporting they had used them.9 Studies during the next few years continued to find web search engines to be the preferred academic research path. In a nationwide survey of 7,421 students conducted in 2005, Jones, Johnson-Yale, Millermaier, and Perez found that search engines topped the list of places to go for information, with 95 percent of respondents saying they used them; library websites were second. However, it is not clear whether the survey was limited to academic research.10 A 2005 OCLC survey of 396 students again found search engines to be “the favorite place to begin,” although 48 percent of students who started their research with a search engine, “ended up at a library Web sites.”11 Like the earlier JISC surveys, Judd and Kennedy’s study of mostly undergraduates in medical and biomedical programs at an Australian university looked at the evolu- tion of Internet use over several years (2005–2009). Information seeking increased over the time span, with Google use rising from 24 percent of the sessions in a campus computer lab to 31 percent. Wikipedia use also increased dramatically, from 2 percent of sessions to 16 percent. Meanwhile, use of the library website and Google Scholar remained steady at 8 percent to 10 percent of sessions, and use of the National Institutes of Health databases declined.12 Running counter to these findings are the results of two surveys in which humani- ties and social science students were asked to rank lists of resources. Course readings ranked first for research, followed by the library website and web search engines in Head’s survey of juniors and seniors at a small liberal arts college.13 A much larger study by Head and Eisenberg of 2,318 undergraduates who had completed at least one semester also found course readings topped the list, followed by web search engines.14 One of the reasons for this might be explained by the content analysis of research paper handouts by Head and Eisenberg, which found that course readings were listed as potential sources in 26 percent to 36 percent of handouts from several colleges.15 616 College & Research Libraries September 2016 In a more recent study, Purdy surveyed 523 undergraduates, mostly freshmen, who ranked 10 information sources for importance to their research. Sixty percent ranked Google first, three times the number who preferred Google Scholar. Only 13.6 percent listed library databases and the library website first.16 The consistency of results in virtually all of these studies seems to confirm that Google and its predecessors have been, and remain, the go-to source for much of un- dergraduates’ academic research. One student comment in a study by Griffiths and Brophy seemed to sum up students’ preference then and now: “Google is very straight forward. You put in your word and it searches…. Bright, eye-catching—simple. Not confusing.”17 While documenting reliance on web search engines and websites, few studies have tried to determine if or how a student’s preferences change over the course of their undergraduate years and what effect librarian instruction have. Instead, as Warwick, Rimmer, Blandford, Gow, and Buchanan found in their in-depth literature review, studies of undergraduate information seeking “have tended to concentrate on how specific search tasks are carried out.”18 One study that did track cohorts of undergradu- ate students over an entire program was that by MacMillan.19 She analyzed research “resumes” completed by journalism students from 2003 to 2008. Each year the students were asked to describe and assess their information literacy skills as they progressed through a three-year program. Analyzing resumes from four cohorts, she found that the use of “article databases and Internet searching saw the most change over time.”20 MacMillan also found that “the range of tools students used, and the sophistication with which they used them increased over time,” and “students broadened the areas they felt comfortable researching.”21 Instruction Effects The most pinpointed analyses of change in undergraduates’ information searching come from the substantial body of citation analyses, surveys, and pretest/posttest comparisons attempting to assess the effectiveness of information literacy instruction. In a systematic review and meta-analysis of such studies, Koufogiannakis found that 75 percent of librarian-instruction studies showed positive results when compared to no instruction.22 However, this body of research presents many challenges: the large number of instruction delivery formats (one-shot sessions, for-credit courses, online tutorials) and methodologies (standardized tools, citation analyses, pretest/posttest, self-assessment and user-satisfaction surveys), as well as the validity and reliability of the instruments.23 As Sobel and Sugimoto put it, Many studies … are focused on change; that is, an increase in scores from one instance of testing to the next …. The time between testing can vary, from the start and finish of a one-hour instruction session, to the start and finish of an un- dergraduate career. These tests are meant to assess student learning. In order to assess the instruction itself, however, it is necessary to provide some baseline for comparison. Some do this by providing a true control group; that is, measuring the learning of those who did not receive instruction training. However, many studies in the literature focus instead on an examination between modes of delivery.24 While a complete review of the literature on the efficacy of librarian instruction is beyond the scope of this study, it is worth highlighting a couple recent studies that address the evolution of information literacy among undergraduates. Cooke and Rosenthal compared bibliographies from research papers written by 75 freshmen English students with those written by 93 students in 300-level and Undergraduates’ Use of Google vs. Library Resources: A Four-Year Cohort Study 617 400-level English and history courses and found that 83 percent of the citations in the more advanced students’ bibliographies were for scholarly sources compared to 52 percent of those in the freshman bibliographies.25 Both sets of students had had instruction, and the professors in the freshman course required at least one scholarly source and no more than one website. Cooke and Rosenthal then com- pared citation analyses done at five other colleges and universities. On average, 75 percent of the citations in bibliographies from upper-level courses at four schools were to scholarly sources.26 Gilbert compared students who had a single instruction session with those who had multiple sessions in the same course in their first semester.27 She found that, before instruction, 75 percent of students start with a Google search. After a single instruction session, 74 percent of students said they started on the library homepage, compared to 59 percent who had multiple instruction sessions (a result that would seem counterintuitive). The citation analysis showed that students who had multiple instruction sessions used fewer sources but a higher percentage of their sources were from the library. They were just as likely to use journal articles as those with one in- struction session but were much more likely to use books. The study also found that those who had multiple instruction sessions were more likely to meet with a librarian at the reference desk. Pickard and Logan compared the results of an ethnographic study of first-generation freshmen with a group of first-generation seniors.28 While both freshmen and seniors mostly searched online, seniors had a more sophisticated and nuanced approach to the research process. Warwick, Rimmer, Blandford, Gow, and Buchanan studied changes in information behavior of a group of information management students over two years. They found that students “avoided the library, and used their expertise to develop strategies to find sufficient information for little effort.” 29 Also, if they were under stress and “a strategy failed or if they were unsure what to do, they tended to return to the pre-university habits of using a textbook or relying on recommended material or to simply repeat an unsuccessful strategy using a different kind of information system.”30 Faculty Source Requirements Studies looking at the effect of faculty requirements on the types of resources students are allowed to use in their papers are rare; even studies that take the concept into ac- count, such as Gilbert, are somewhat unusual.31 Davis tracked bibliographies of eco- nomics students over several years and found that neither librarian instruction nor the combination of instruction and an instructor’s encouragement to use scholarly sources had a significant effect on source use.32 However, in the fourth year of the analyses, there was a sharp increase in citations to scholarly journal articles when the instructor required at least five such sources. Robinson and Schlegl built on this research, compar- ing control groups of political science students who had neither librarian instruction nor source requirements to a group that had librarian instruction and one that had both librarian instruction and source requirements.33 Scholarly sources represented 72 percent of citations for the control groups, and 74 percent for the group that had librarian instruction only. However, 86 percent of the citations of students who had both instruction and were required to use at least four scholarly sources were to scholarly sources. Similarly, citations to nonscholarly sources fell. Theoretical Framework Much has been written about the differences between novice and expert searchers. For this study, we are using Alexander’s Model of Domain Learning as adapted by Thomp- son and Lathey to apply to information literacy.34 Alexander describes three phases 618 College & Research Libraries September 2016 of the evolution, from acclimation to competence and then proficiency, as a journey with “subtle and significant transformations.”35 Thompson and Lathey’s interpreta- tion of those phases describe acclimation as a time of “transition from elementary and secondary school habits to more advanced academic forms of research.”36 Competence should emerge once a student has chosen a major and, using domain knowledge, starts to develop more “advanced strategies for information gathering.”37 The profi- cient student understands the “discourse of the discipline and the systematic means of gathering and reporting information/data in that field.”38 In the current study the researchers were looking to document a small part of the acclimation and competence phases in the places undergraduates start their research and the sources they rely on for the majority of information in their research papers. In the researchers’ anecdotal experience at a large public university, very few undergraduates reach competence, and a student survey about use of Google, and web and library source use, is insuf- ficient to determine proficiency. Research Questions and Hypotheses The study aimed to answer four research questions: 1. Were undergraduates from the 2008 freshman cohort more likely to start their research with library resources or web search engines? Did this change over time? 2. Were undergraduates from the 2008 freshman cohort more likely to use library resources or websites for the majority of the information in their research papers? Did this change over time? 3. Was there an association between librarian instruction and where students started their research and what sources they used for the majority of the infor- mation in their papers? 4. Was there an association between faculty source requirements and where stu- dents started their research and what sources they used for the majority of the information in their papers? From these four questions, we developed six hypotheses: H1. Undergraduates will start their research with library resources more often as they progress in college. H2. Undergraduates will use library resources more often for the majority of infor- mation in their papers as they progress in college. H3. Undergraduates who have had librarian instruction will be more likely to start their research with library resources. H4. Undergraduates who have had librarian instruction will be more likely to use library resources for the majority of the information in their papers. H5. Undergraduates whose instructors specify certain types of sources for their papers will be more likely to start their research with library resources. H6. Undergraduates whose instructors specify certain types of sources will be more likely to use library resources for the majority of the information in their papers. Methodology We chose a longitudinal-panel methodology. According to Taris, a longitudinal panel is a “particular set of participants … repeatedly interviewed using the same question- naires … [which] enables researchers to observe relationships across time rather than relationships at one point in time.”39 A related longitudinal design is the cohort study. A cohort, as defined by Ryder, is an “aggregate of individuals (within some popula- tion) definition who experienced the same event within the same time interval.”40 Undergraduates’ Use of Google vs. Library Resources: A Four-Year Cohort Study 619 Cohort studies are highly ranked in Eldredge’s hierarchy of evidence for evaluating the quality of research, higher than the case studies that populate much of the library science literature.41 The cohort in this study was the 2008 freshman class at a large public Masters I level comprehensive university with 31 percent of students eligible for Pell grants. There were nine waves of data collection, spaced one semester apart. This article reports on the first eight waves. Sample Recruitment For nine waves of data collection, there had to be a large sample to be able to withstand attrition from students dropping out of school or losing interest in the study. As Taris advises, “the sample size needed for the first wave of a longitudinal study increases with the number of waves of that study because people have more opportunities to drop out … in a multi-wave study than in a two-wave study.”42 Our strategy to achieve a large sample was to recruit in classes on the assumption that face-to-face contact would be more successful than e-mail or some other more anonymous recruiting. The original recruitment plan called for contacting a sample of the 4,606 incoming fresh- men in fall 2008 by visiting randomly selected sections of the mandatory university- introduction course to ensure that all members of the cohort had an equal probability of being selected. We were unable to gain permission to access these classes, so we changed the recruiting plan. Our second approach, e-mailing a random sample of the whole freshman class, raised privacy issues. Attempts to visit a random sample of freshman English classes encountered time-consuming Institutional Review Board (IRB) requirements that we contact and get written permission individually from the dozens of mostly temporary instructors before submitting our application. We settled on a convenience sample. We recruited students in four ways and locations at the end of the first semester and through part of their second semester. First, we e-mailed 1,430 students over 18 years old who gave permission on the Freshman Survey for further contact.43 Second, we visited several classes with significant freshman enrollment and instructor permission. Third, we recruited in dormitories, which at the time were about 90 percent freshman. Fourth, we recruited on multiple days in the lobby of the university library and in front of the university bookstore. Students who signed up for the study, after giving informed consent and completing a survey, were given a $5 gift card. We obtained the student’s ID number, which gave us a current e-mail address for future communication. Students were advised that this was a longitudinal study and that they would be surveyed once a semester for up to six years. After duplicates were eliminated, we had signed up a total 451 students, of whom 386 completed the survey and could be confirmed as students who entered the university in fall 2008—8.4 percent of the 2008 freshman class. Representativeness of Sample We compared our sample with the overall 2008 freshman class to determine the sample’s representativeness on bases of gender, ethnicity, and enrollment in remedial classes, using Chi Square tests. Our sample was significantly more female, and had more Asians/Pacific Islanders and African Americans, and fewer Whites than the 2008 freshman class, and it had lower enrollments in remedial courses. However, the number of Latino/Latina students in the sample was representative of the freshman class. Table 1 shows the demographic breakdown of the sample compared with the demographics of the entire 2008 freshman class. Regarding enrollment in remedial courses, more than 40 percent of the 2008 freshman class had to take at least one remedial course, while 32.9 percent of the sample did. 620 College & Research Libraries September 2016 Participation and Attrition Significant attrition was expected, corresponding to the university’s statistics on persistence rates and because of anticipated loss of interest in the study. Students in the study sample were more persistent than 2008 freshmen were as a whole. University figures show that almost 14 percent of the 2008 freshman did not return the second year. However, only 5 percent of our sample was no longer enrolled in the second year of the study. This, along with lower enrollment in remedial courses, were indications that our sample might be academically better prepared than the freshman class overall. Continued participation in the study was more challenging. Of the original sample, 42.7 percent stopped responding after the first semester, despite continued contact and incentives. Proportionately more women continued in the study, with women represent- ing 73.3 percent of those who participated two or more semesters. However, there was no discernible difference in ethnicity of those who continued, with the exception that there were no African American students who participated more than five semesters, and Asian/Pacific Islanders represented a disproportionate percentage (46%) of the students who participated all eight semesters. TABLE 1 Sample Gender and Ethnicity Study Participants Percent of Participants Freshman Class Percent of Freshman Class Latino/Latina 129 33.4% 1,568 34.0% Asian/Pacific Islanders 114 29.5% 1,080 23.4% White 97 25.1% 1,339 29.1% African American/Black 25 6.5% 245 5.3% American Indian/Alaskan Native 3 0.8% 26 0.6% Other/Unknown 18 4.7% 348 7.6% Female 274 71% 1,710 62.9% Male 112 29 % 2,826 37.1% TOTAL 386 4,606 TABLE 2 Participants by Semester Semester Number of Participants Students Who Participated Every Semester Semester 1 386 386 Semester 2 85 85 Semester 3 112 70 Semester 4 83 38 Semester 5 91 23 Semester 6 90 18 Semester 7 78 15 Semester 8 83 13 Undergraduates’ Use of Google vs. Library Resources: A Four-Year Cohort Study 621 To improve participation, we held a drawing each semester for students who filled out a survey. Students could win $150 to $300 in gift cards (depending on the semester and funds). One semester, we also mailed letters to the students’ address of record advising them of the upcoming survey. We also sent several e-mail reminders each semester. We had 386 participants in the first semester, and 78 to 112 in subsequent se- mesters. However, only 13 students completed surveys all eight semesters (see table 2). Survey Instrument The principal goals of the study were to determine what resources students consulted when they started their research for an assigned paper and what resources provided the majority of the information they actually used in their papers. Using survey methodol- ogy, the researchers developed a Research Habits Survey consisting of 10 questions.44 It asked students what courses they were enrolled in, which classes included librarian instruction, which required research papers, and whether professors in those courses required certain types of resources. In addition to this information on courses and research papers, there were two questions on how they conducted their research: one on where they started their research and one on where they obtained the majority of the material they used (see figure 1). Finally, we asked how often the students visited the library website, asked a librarian for help, and how many credits they had completed.45 For the questions on where they started their research and where they found the majority of the material they used, we decided to be as specific as possible. Rather than asking if students used “scholarly” resources, we asked about the use of journal articles or books. Additionally, most librarian instruction at our institution focuses on how to find scholarly articles or books, so the survey terminology would be consistent with what students were hearing in librarian instruction sessions. Participants were able to identify “other” resources, but few did. Results For each semester, we analyzed responses only for the students who reported com- pleting at least one research paper or assignment that required citations (see table 3). Consistent with other studies, a large majority of freshmen in this study reported that Google (or another web search engine) was the starting place for academic research.46 In their first semester, 69 percent of students said they started with Google. At the same time, only 30 percent started with library resources, with a few students mentioning other resources, such as course notes. However, as we hypothesized (H1), students started their research with library resources more often as they progressed in college. During the eight semesters, the percentage of participants starting their research with library resources climbed from FIGURE 1 Survey Questions on Research Behavior Q5. Thinking about all of your research papers in [this] semester, where did you find the MAJORITY of the information you actually used in writing your papers? CHOOSE ONLY ONE ANSWER. a) Websites b) Journal articles c) Books d) Other (please specify) Q6. Again, thinking of your research papers overall during [this] semester, where did you typically START your research? CHOOSE ONLY ONE ANSWER. a) Google (or another Web search engine) b) Library resources (journal articles or books) c) Other (please specify) 622 College & Research Libraries September 2016 30 percent to 45 percent. This increase was not steady from semester to semester, but the percentage grew in all but the fifth and eighth semesters. At the same time, the use of Google as a starting place declined (see figure 2). Perhaps a better measure than where they started their research was what sources students reported using for the majority of the information in their papers. Here again, 70 percent of beginning freshmen reported relying on the web. But the change over four years in this measure was more striking. While only 30 percent of first-semester freshmen reported using library resources for the majority of the information in their papers, 71 percent of eighth-semester study participants did, supporting hypothesis H2 (see figure 3). While several factors could influence these research behavior changes, we consid- ered two variables. First, was librarian instruction in a semester associated with where students started their research and where they found the majority of the information for their papers that semester? Second, was student use of library resources associated with an instructor’s source requirements in a semester? FIGURE 2 Where Students Started Research 68.6% 37.3% 36.5% 37.1%29.9% 59.5% 58.6% 44.9% 0% 10% 20% 30% 40% 50% 60% 70% 80% Pe rc en t of S tu de nt s Google Library Resources Other TABLE 3 Number of Papers Written Total Participants Students Writing Papers Percent Writing Papers Semester 1 386 355 92% Semester 2 85 70 82% Semester 3 112 83 74% Semester 4 83 59 72% Semester 5 91 75 82% Semester 6 90 75 83% Semester 7 78 70 90% Semester 8 83 68 82% Undergraduates’ Use of Google vs. Library Resources: A Four-Year Cohort Study 623 Variables Associated with Where Students Started Research Librarian Instruction Students who had librarian instruction in a semester were more likely to start their research with library resources that semester than those who did not have instruction. This was true in all but two semesters, and in the first semester the difference between those who had librarian instruction and those who did not was statistically significant (Chi-square 8.84, p ≤ 0.01), supporting Hypothesis H3. Overall, in six of the eight se- mesters, students were more likely to start their research with library resources if they had librarian instruction that semester, with the percentage who started with library resources increasing steadily through the sixth semester, but declining in semesters 7 and 8. However, by their eighth semester, students were still 41 percent more likely to start their research with library resources than they were in their first semester if they had librarian instruction (55% vs. 39%). Strikingly, the trend in starting research with library resources was evident even among students who did not have librarian instruction in a given semester, although the percentage was nearly always below that of students who had instruction. Starting with library resources went from 38 percent FIGURE 3 Where Students Found Most Information Used 70% 46% 40% 25% 41% 32% 26% 29% 30% 53% 57% 73% 59% 68% 74% 71% 0% 10% 20% 30% 40% 50% 60% 70% 80% Pe rc en ta ge o f S tu de nt s On the Web Library Resources Other FIGURE 4 Started Research with Library Resources 38% 45% 56% 53% 64% 88% 53% 55% 24% 35% 35% 58% 35% 52% 61% 39% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Pe rc en ta ge o f S tu de nt s Had Instruc�on That Semester No Instruc�on That Semester 624 College & Research Libraries September 2016 in the first semester to 55 percent in the eighth semester for those who had instruction in the semester, compared to a growth from 24 percent to 37 percent for those who did not have librarian instruction (see figure 4). Librarian instruction seemed to be a less distinguishing variable in determining where students found the majority of the information they used in their papers. Of first-semester freshmen who had librarian instruction, 39 percent used library re- sources for most of the information in their papers compared to 26 percent who did not have librarian instruction. In the eighth semester, 75 percent who had instruction used library resources for most of the information in their papers, while 69 percent of those who didn’t have instruction did (see figure 5). The difference was statistically significant only in the first semester (Chi-square 5.255, p ≤ 0.05), raising substantial questions about hypothesis H4. Instructors’ Source Requirements Frequently, professors will require students to use certain types of sources for their re- search papers. For example, students may have to cite at least three journal articles, one book, and no more than one website. Our hypothesis (H5) was that these requirements would be associated with more students starting their research with library sources. In the first semester, 31 percent of the students whose instructors had source require- ments started their research with library resources, compared with only 20 percent of those whose professors did not have source requirements. This was a statistically significant difference (Chi-square 13.843, p ≤ .05). The relationship between instructor requirements and where students started their research varied during the remaining seven semesters, casting doubt on hypothesis H5. Students also were more likely to use library resources for the majority of informa- tion in their papers when professors had source requirements. In the first semester, 33 percent of students whose instructors had source requirements used library resources compared to 12 percent of those whose professors did not, a statistically significant difference (Chi-square 13.066, p ≤ .05). Over the remaining semesters, participants consistently were more likely to use library resources for the majority of the information in their papers when their profes- sors required certain types of resources, but there were insufficient students without source requirements to make meaningful comparisons (see figure 6). FIGURE 5 Used Library Resources for Majority of Information in Paper 39% 56% 68% 64% 100% 75% 26% 57% 79% 57% 67% 69% 0% 20% 40% 60% 80% 100% 120% Pe rc en ta ge o f S tu de nt s Had Instruc�on No Instruc�on Undergraduates’ Use of Google vs. Library Resources: A Four-Year Cohort Study 625 Librarian Instruction vs. Faculty Source Requirements While a true analysis of variance was not possible because of the categorical nature of the data and insufficient numbers of students who had neither librarian instruction nor source requirements past semester 1, analyzing the descriptive statistics for the first semester gives a sense of the impact of librarian instruction and faculty source requirements. Of those who had both librarian instruction and source requirements, 36 percent started with library resources, whereas, of those students whose instructors required certain sources but did not have librarian instruction, just 26 percent started with library resources. Regarding source choices for the majority of the information in their papers, 42 percent of those who had both librarian instruction and source require- ments used library resources, whereas, of those who had source requirements only but not librarian instruction, only 28 percent used library resources. Of those students who had neither librarian instruction nor instructor-required sources, only 5 percent started with library resources and 19 percent used library resources for the majority of the information in their papers (see table 4). Seeking Help In the first two years, about 30 percent of students each semester said they sought a librarian’s help. In the second semester, those who had instruction were more than twice as likely to seek help (48% vs. 21%).47 In the third semester, those who had instruction TABLE 4 Librarian Instruction vs. Faculty Source Requirements Started with Google Started with Library Resources Majority of Information from web sites Majority of information from Library Resources Instruction & Source Requirements 63.2% 36.8% 58.5% 41.5% Instruction Only 44.4% 55.6% 100.0% 0.0% Source Requirements Only 74.4% 25.6% 72.0% 28.0% Neither 95.0% 5.0% 81.0% 19.0% TOTAL 70.2% 29.8% 67.7% 32.3% FIGURE 6 Started with Library Resources 31% 44% 49% 54% 56% 60% 62% 49% 20% 18% 44% 62.5% 17% 58% 53% 28% 0% 10% 20% 30% 40% 50% 60% 70% Pe rc en ta ge o f S tu de nt s Instructor- Required Sources No Instructor- Required Sources 626 College & Research Libraries September 2016 were four times more likely to seek a librarian’s help (51.5% vs. 12.5%). In the fourth semester, students who had instruction were almost three times more likely to seek a librarian’s help than those who did not have instruction (41.2% vs. 13.6%). Discussion Undergraduates’ well-documented reliance on Google for academic research is strongly supported by the results of this longitudinal cohort study. In each semester, at least one third of the participants said they started their research on Google or another search engine, with more than two thirds starting there as first-semester freshmen. Similarly, at least 25 percent of students in any given semester said they used web sources for the majority of the information in their papers, with 70 percent doing so in their first semester. The principal aim of this study was to see if these behaviors changed over time. The study’s results did find substantial change in source use over time. In the first semester, just 30 percent of participants started their research with library resources, but in the eighth semester 45 percent did. More dramatically, while 30 percent of first-semester freshmen used library resources for the majority of the information in their papers, by the eighth semester 71 percent did. The study tested for two possible variables to try to explain this evolution (Hy- potheses H3–H6): ¾ Librarian instruction. In six of eight semesters, students who had librarian instruction were more likely to start their research with library resources (H3) and also were more likely to use library resources for the majority of information in their papers (H4). Results for the first semester were statistically significant in both cases, but the support H3 was greater than that for H4 (see figures 4 and 5). ¾ Instructor requirements. In all semesters but one, a higher percentage of students started their research with library resources if there were instructor-required sources (H5); and, in every semester, a greater percentage of students said they used library resources for the majority of the information in their papers if their instructors required certain types of resources (H6). However, the differences were only statistically significant in the first semester (see figures 6 and 7). Attrition, along with the prevalence of both librarian instruction and faculty source requirements, made it difficult to determine statistically significant dif- ferences after the first semester. Interestingly, however, during those first four semesters, library resources were increasingly used by students who had had FIGURE 7 Majority of Information from Library Resources 33% 55% 60% 74% 63% 69% 79% 78% 12% 36% 57% 37.5% 50% 63% 61% 50% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Pe rc en ta ge o f S tu de nt s Instructor- Required Sources No Instructor- Required Sources Undergraduates’ Use of Google vs. Library Resources: A Four-Year Cohort Study 627 neither librarian instruction nor faculty source requirements, raising the question of a possible maturation effect. Interestingly, regardless of whether students had librarian instruction in a given semester or whether an instructor required specific types of resources, students increasingly started their research with library resources and used library resources for the majority of the information in their papers during their first two years. In semesters 5 through 8, there was more of a gap between those who had librarian instruction or who had instructor-required sources and those who did not, but across the board by the eighth semester more students started their research with library resources and used library resources for the majority of the information in their papers than when they entered the university. Could this be a cumulative effect of librarian instruction or faculty source expectations, or the number of papers written? Another possible explanation is “maturation effect,” which refers to biological or psychological factors such as persistence, motivation, or brain development.48 The author could not find a discussion of this phenomenon specifically in the information literacy literature, but it can be a confounding influence in longitudinal cohort stud- ies. More research is needed on the effect of instructor source requirements and on maturation effect. This study had numerous limitations that future researchers should take into ac- count: ¾ Lack of a random sample. ¾ The sample apparently was more academically prepared than the cohort as a whole. ¾ In addition to the students who had librarian instruction, many completed a short online introduction to the library, which could amplify the effect of librar- ian instruction in the results. ¾ Attrition. Only 41 students completed surveys in each of the first four semes- ters, and only 13 did so for all eight semesters, making it difficult to analyze subgroups. For example, were students who took remedial courses more or less likely to use library resources? ¾ We only asked about two types of resources, those from the web and those from the library. How might the results have differed if we had given participants other choices, such as course readings as Head and Eisenberg did?49 ¾ We did not control for rate of academic progress; each semester was considered equal, while in reality many students had 75 units by the end of their fourth semester while others had only 45. ¾ We did not distinguish between “Google” and “Google Scholar.” Google Scholar is most likely to direct students to library resources, so if a student “started” with Google Scholar, would he or she mark Google or library re- sources? Another question raised by this study is the underlying implication that library resources are “better” than web resources. Librarians, and faculty, are probably bi- ased toward scholarly, peer-reviewed journal articles and academic books, and they might be pleased to see that students increasingly use them over the course of their undergraduate years. But should we not be emphasizing the best resources for a topic or course? For example, for a policy analysis paper, the best resource might be an online government document. Further, Google might be the best place to start all research to get a general understanding of a topic, similar to consulting a reference source. Perhaps the study’s participants figured that out, accounting for the increase in Google use in semester 7. 628 College & Research Libraries September 2016 Conclusion Longitudinal cohort studies with undergraduates are major undertakings plagued by attrition, as evidenced by the fact that only 13 of the original 386 participants in this study completed surveys all eight semesters. However, longitudinal studies can produce a valuable overview of the student learning experience that cross-sectional studies cannot. This is especially important for research skills that often are taught in one-hour, one-shot sessions with no grades or follow-up by which to assess learn- ing, but which are expected to serve students as they navigate the information-dense world. This study looked at one aspect of research skills: Students’ choice of library resources vs. open web sources for academic papers, and the differences in how student source choices change over four years. It also looked at two variables influencing those choices: librarian instruction and faculty source requirements. Like many academic librarians in this field, the goal of the five researchers conducting this study was to document whether librarian instruction improves students’ research and to better understand how we can collaborate with classroom faculty to improve student research skills further. The study’s results provided insight on both goals. The undergraduate participants were more likely to start their research with journal articles and books when they had librarian instruction in a given semester and when their course instructors required certain types of sources. The student participants were also more likely to use these library resources for the majority of the information when they had librarian instruction and faculty source requirements. The results also found that these behaviors increased over the four years of the study. One recommendation to grow out of this study is for faculty who require certain types of resources to make sure their students receive librarian instruction. In the first semester of our study, only 47 percent of students who had assignments with specific source requirements reported that they received librarian instruction. The study results also add further evidence of the need for collaboration on assign- ments between librarians and course faculty. Specifically, collaboration on the types of sources required for success in a given research assignment, as opposed to a blanket prohibition of, or severe limitation on, web sources could result in clearer assignments, better papers, and teachable moments for students when coupled with librarian instruc- tion. For example, a requirement to include trade publications, newspaper articles, or government documents where appropriate could expand students’ source repertoire. More research, however, is needed to better understand the relative contribution of librarian instruction and instructor’s source requirements individually, as well as the synergy between the two. Future researchers should be mindful of the need to have sufficiently large numbers of students who do not receive librarian instruction, and/ or whose professors do not require certain types of resources, to enable meaningful comparisons and analysis. Also needed is research into the cumulative effect of these variables. If students receive librarian instruction and source requirements in their freshman year, for example, is that sufficient? Or is repetition important, and if so, how much repetition? In interviews at the end of the study with 10 of the 13 participants who completed surveys every semester, students emphasized that they want librarian instruction earlier in their college careers and they want it in person. Adding emphasis to these comments were the study results that found that students who had librarian instruction were far more likely to seek out a librarian for additional help—better to have that impact early on. Further evidence from the study pointing to the importance of early librarian instruction was the fact that first-semester students who had librarian instruction were 58 percent more likely to start their research with scholarly sources (38% vs. 24%) and 50 percent more likely to use scholarly sources for the majority of Undergraduates’ Use of Google vs. Library Resources: A Four-Year Cohort Study 629 the information in their papers. Attrition made it difficult to determine whether this early instruction had a lasting effect, pointing in another direction for future research. The question of maturation effect needs further exploration. What factors besides librarian instruction and faculty source requirements influence students’ research be- haviors? The navigability of library websites? Course-specific online research guides? Faculty comments and grades on papers? Peer suggestions? Librarian instruction and source requirements may open the door to learning better research skills, but what are the factors that build on those foundations? Notes 1. Joseph Aubele, Karin Griffin, Susan Jackson, and Catherine Outten were the other research- ers on the study. 2. Margy MacMillan, “Watching Learning Happen: Results of a Longitudinal Study of Jour- nalism Students,” Journal of Academic Librarianship 35, no. 2 (2009): 132–42. 3. Ibid. 4. William F. Slater III, “Internet History and Growth,” Chicago Chapter of the Internet Society (last modified Sept. 2002), available online at www.internetsociety.org/sites/default/ files/2002_0918_Internet_History_and_Growth.ppt [accessed 1 May 2015]. 5. “OCLC White Paper on the Information Habits of College Students: How Academic Librar- ians Can Influence Students’ Web-based Information Choices,” OCLC Online Computer Library Center (last modified June 2002), available online at www5.oclc.org/downloads/community/ informationhabits.pdf [accessed 10 November 2014]; Karen Kaminski, Pete Seel, and Kevin Cul- len, “Technology Literate Students? Results from a Survey,” Educause Quarterly 26, no. 3 (Jan. 1, 2003): 34–40; Steve Jones and Mary Madden, “The Internet Goes to College: How Students Are Living in the Future with Today’s Technology,” Pew Internet & American Life Project (last modi- fied Sept. 15, 2002), available online at www.pewinternet.org/files/old-media/Files/Reports/2002/ PIP_College_Report.pdf [accessed 10 November 2014]. 6. “OCLC White Paper.” 7. Kaminski, Seel, and Cullen, “Technology Literate Students,” 34–40. 8. Jones and Madden, “The Internet Goes to College.” 9. Christine Urquhart, Rhian Thomas, Ray Lonsdale, Siân Spink, Alison Yeoman, Roger Fenton, and Chris Armstrong, “Uptake and Use of Electronic Information Services: Trends in UK Higher Education from the JUSTEIS Project,” Program: Electronic Library & Information Systems 37, no. 3 (July 2003): 168–80; Ray Lonsdale, Rhian Thomas, Jane Durbin, Janet Turner, Chris Arm- strong, Siân Spink, Christine Urquhart, and Roger Fenton, “JUSTEIS (JISC Usage Surveys: Trends in Electronic Information Services) Strand A: Survey of End Users of all Electronic Information Services (HE and FE), with Action Research Report. Final Report 2002/2003 Cycle Four” (2003): 7–110, available online at http://cadair.aber.ac.uk/dspace/handle/2160/304 [accessed 24 November 2014]. 10. Steve Jones, Camille Johnson-Yale, Sarah Millermaier, and Francisco Seoane Pérez, “Aca- demic Work, the Internet and U.S. College Students,” Internet and Higher Education 11 (2008): 165–77. 11. Cathy De Rosa, Perceptions of Libraries and Information Resources: A Report to the OCLC Membership (Dublin, Ohio: OCLC Online Computer Library Center, 2005). 12. Terry Judd and Gregor Kennedy, “A Five-Year Study of On-Campus Internet Use by Un- dergraduate Biomedical Students,” Computers & Education 55, no. 4 (Dec. 2010): 1564–71. 13. Alison J. Head, “Beyond Google: How Do Students Conduct Academic Research?” First Monday 12, no. 8 (Aug. 2007): 1. 14. Alison J. Head and Michael B. Eisenberg, “Lessons Learned: How College Students Seek Information in the Digital Age. Project Information Literacy Progress Report,” Project Information Literacy (Dec. 2009): ERIC. 15. Alison J. Head and Michael B. Eisenberg, “How Handouts for Research Assignments Guide Today’s College Students: Project Information Literacy Progress Report,” Project Information Literacy (July 2010), available online at http://projectinfolit.org/images/pdfs/pil_handout_study_fi- nalvjuly_2010.pdf [accessed 15 October 2015]. 16. James P. Purdy, “Why First-Year College Students Select Online Research Resources as their Favorite,” First Monday 17, no. 9–3 (Sept. 2012). 17. Jillian R. Griffiths and Peter Brophy, “Student Searching Behavior and the Web: Use of Academic Resources and Google,” Library Trends 53, no. 4 (Spring 2005): 539–54. 630 College & Research Libraries September 2016 18. Claire Warwick, Jon Rimmer, Ann Blandford, Jeremy Gow, and George Buchanan, “Cogni- tive Economy and Satisficing in Information Seeking: A Longitudinal Study of Undergraduate Information Behavior,” Journal of the American Society for Information Science & Technology 60, no. 12 (2009): 2402–15. 19. MacMillan, “Watching Learning Happen,” 132–142. 20. Margy MacMillan, “Many Paths, One Journey: Mapping the Routes to Information Lit- eracy,” (unpublished manuscript, June 10, 2010). 21. Ibid. 22. Denise Koufogiannakis and Natasha Wiebe, “Effective Methods for Teaching Information Literacy Skills to Undergraduate Students: A Systematic Review and Meta-Analysis,” E-Lis: E- Prints in Library and Information Science 1, no. 3 (2006): 3–43. 23. Megan Oakleaf, Michelle S. Millet, and Michelle S. Krau, “All Together Now: Getting Faculty, Administrators, and Staff Engaged in Information Literacy Assessment,” portal: Libraries and the Academy 11, no. 3 (2011): 831–52, doi:10.1353/pla.2011.0035; Andrew Walsh, “Information Literacy Assessment: Where Do We Start?” Journal of Librarianship and Information Science 41, no. 1 (Mar. 2009): 19–28. 24. Karen Sobel and Cassidy R. Sugimoto, “Assessment of Learning During Library Instruction: Practices, Prevalence, and Preparation,” Journal of Academic Librarianship 38, no. 4 (2012): 191–204, doi:http://dx.doi.org/10.1016/j.acalib.2012.04.004. 25. Rachel Cooke and Danielle Rosenthal, “Students Use More Books after Library Instruction: An Analysis of Undergraduate Paper Citations,” College & Research Libraries 72, no. 4 (2011). 26. Ibid. 27. Julie K. Gilbert, “Using Assessment Data to Investigate Library Instruction for First Year Students,” Communications in Information Literacy 3, no. 2 (2009): 181–92. 28. Elizabeth Pickard and Firouzeh Logan, “The Research Process and the Library: First- Generation College Seniors vs. Freshmen,” College & Research Libraries 74, no. 4 (2013): 399–415. 29. Warwick, Rimmer, Blandford, Gow, and Buchanan, “Cognitive Economy and Satisficing,” 2402–15. 30. Ibid. 31. Gilbert, “Using Assessment Data,” 181–92. 32. Philip M. Davis, “The Effect of the Web on Undergraduate Citation Behavior: A 2000 Update,” College & Research Libraries 63, no. 1 (2002): 53–60; Philip M. Davis, “Effect of the Web on Undergraduate Citation Behavior: Guiding Student Scholarship in a Networked Age,” portal: Libraries & the Academy 3, no. 1 (2003): 41–51. 33. Andrew M. Robinson and Karen Schlegl, “Student Bibliographies Improve When Profes- sors Provide Enforceable Guidelines for Citations,” portal: Libraries and the Academy 4, no. 2 (2004): 275–90. 34. Patricia A. Alexander, “The Development of Expertise: The Journey from Acclimation to Proficiency,” Educational Researcher 32, no. 8 (2003): 10–14; Gary B. Thompson and Jonathan W. Lathey, “An Integrated Model of Information Literacy, Based Upon Domain Learning,” Copenha- gen, Denmark, Aug. 19–22, 2013, available online at http://informationr.net/ir/18-3/colis/paperC02. html#.Ujt9Qrz5blM [accessed 29 November 2014]. 35. Alexander, “The Development of Expertise,” 10–14. 36. Thompson and Lathey, “An Integrated Model of Information Literacy.” 37. Ibid. 38. Ibid. 39. Toon Taris, Primer in Longitudinal Data Analysis (London, U.K.: SAGE Publications Inc., 2000). 40. Norman B. Ryder, “The Cohort as a Concept in the Study of Social Change,” American Sociological Review 30, no. 6 (1965): 843–61. 41. Jonathan Eldredge, “Evidence-Based Librarianship: The EBL Process,” Library Hi Tech 24, no. 3 (2006): 341–54, doi:10.1108/07378830610692118. 42. Taris, Primer in Longitudinal Data Analysis. 43. “CIRP Freshman Survey,” Higher Education Research Institute (HERI), available online at www.heri.ucla.edu/cirpoverview.php [accessed 20 June 2014]. 44. Two questions were added in the third year asking for participants’ major and anticipated year of graduation. 45. Kaminski, Seel, and Cullen, “Technology Literate Students,” 34–40. 46. Question about seeking a librarian’s help was not asked in the first semester. 47. Davis, “Effect of the Web on Undergraduate Citation Behavior,” 41–51. 48. Roger E. Kirk, “Maturation Effect,” in The Sage Encyclopedia of Social Science Research Meth- ods, eds. Michael S. Lewis-Beck, Alan Bryman, and Tim Futing Liao (Thousand Oaks, Calif.: Sage Publications, 2004). 49. Head and Eisenberg, “Lessons Learned,” ERIC.