272 Information Literacy’s Influence on Undergraduates’ Learning and Development: Results from a Large Multi-institutional Study Kevin Fosnacht* This paper investigated the reliability and validity of the National Survey of Student Engagement’s Experiences with Information Literacy module, an assessment in- strument developed in collaboration with a group of instructional librarians. After identifying three information literacy–related factors in the module, it assessed the relationship between the factors and students’ engagement in Higher-Order Learning and Reflective and Integrative Learning activities and students’ perceived gains. The results from these analyses indicated that information literacy activities were positively and significantly correlated with student engagement and students’ perceived gains. Today undergraduates face a conundrum. They can access more information on their cell- phones at a moment’s notice than previous generations could access in a multistory university library. However, the increasing inclusion of terms like “alternative facts” and “fake news” in the popular discourse highlights how much of the information digested by students is of questionable quality.1 This reality makes knowing how to search for, use properly, and evaluate information a critical skill for the 21st century. Information literacy has been widely recognized as a critical college learning outcome for decades.2 However, assessing information literacy skills has been difficult for librarians and other assessment professionals, as the conventional methods frequently focus on small samples at or within a single institution, testing knowledge of specific skills, are time intensive (rubrics), or use instruments that have not been psychometrically validated.3 Due to these concerns, instructional librarians approached the National Survey of Student Engagement (NSSE) to collaborate with NSSE researchers on expanding the information literacy content in NSSE.4 The collaboration resulted in the creation of the Experiences with Information Literacy module, which institutions can elect to append to the core NSSE instrument. In contrast to other information literacy assessments, the module mainly focuses on student engagement in activities associated with information literacy skill development and the extent to which instructors emphasize the proper use of information, which makes the module’s results ac- tionable. Additionally, most baccalaureate-granting colleges and universities periodically ad- * Kevin Fosnacht is Associate Research Scientist at the Indiana University Center for Postsecondary Research; email: kfosnach@indiana.edu. Author’s note: A previous version of this manuscript was presented at the 2017 meeting of the Association of College & Research Libraries in Baltimore, MD. ©2020 Kevin Fosnacht, Attribution- NonCommercial (http://creativecommons.org/licenses/by-nc/4.0/) CC BY-NC. mailto:kfosnach@indiana.edu http://creativecommons.org/licenses/by-nc/4.0/ Information Literacy’s Influence on Undergraduates’ Learning and Development 273 minister NSSE to their students,5 and 208 institutions have chosen to administer the module to their students in the first four years NSSE has offered the module. Therefore, the module is relatively inexpensive and efficient to administer, and its data are already available to a wide variety of institutions. This study has two main purposes. First, it seeks to establish the reliability and validity of the NSSE Experiences with Information Literacy module. Second, through its investigation of the predictive validity of the module’s factors, it examines how engagement in information literacy activities contributes to students’ learning and development. To accomplish these objectives, the study uses data from nearly 145,000 first-year students and seniors attending 208 colleges and universities. The results have the potential to highlight the usability of the module and provide further evidence of the importance of information literacy in promoting student learning outcomes. Student Engagement Theory Both the study and the creation of the NSSE Experiences with Information Literacy module were informed and guided by student engagement theory. The theory combines empiri- cal research and several well-known concepts in the higher education literature.6 Student engagement uses Pace’s quality of effort concept, which postulates that student learning is a product of students’ quality and quantity of effort, as well as Astin’s student involve- ment theory, which theorizes that student retention is related to involvement in academic and cocurricular activities.7 In addition to focusing on students’ activities, it emphasizes the role of the institution to promote student learning and development in and outside of the classroom. Chickering and Gamson demonstrated a variety of effective educational practices that support student learning.8 Additionally, Kuh and colleagues exhibited how institutions can structure their institutions to improve student learning outside of the class- room.9 When applying the theory to information literacy, the theory views institutions as having the responsibility to emphasize and instruct students on the proper and effective uses of information and providing ample or adequate information resources for students. Furthermore, students are expected to exhort time and effort to search, evaluate, and use information sources properly. Research Questions As the validity of a survey instrument requires multifaceted evidence,10 this study investi- gated the following research questions to establish the validity of the NSSE Experiences with Information Literacy module and examine how engagement in information literacy activities promotes student learning and development. 1. What information literacy factors exist within the NSSE’s Experiences with Informa- tion Literacy module? 2. Do the factors adequately fit the observed data? 3. Are the factors reliably measured? 4. After controlling for other factors, how do the factors correlate to students’ engage- ment in higher-order learning and reflective and integrative learning skills? 5. After controlling for other factors, how do the factors relate to students’ perceived gains? 274 College & Research Libraries March 2020 Methods Data To answer the aforementioned research questions, I used data from the 2014 to 2018 admin- istrations of the National Survey of Student Engagement (NSSE). NSSE examines students’ engagement in educationally beneficial activities, their perceptions of the campus environment, self-assessments of how their college experience has improved their learning and develop- ment, and students’ background characteristics. The 2014 to 2018 timeframe included all NSSE administrations that included the Experiences with Information Literacy module. The module is a set of additional questions that institutions can elect to append to the core NSSE instru- ment that focuses on how often students use and evaluate information, the extent to which instructors emphasized the proper use of information, and an item asking how much students’ college experience has influenced their ability to use information effectively. NSSE staff and a group of instructional librarians collaboratively developed the module.11 The designers of the module used their academic training as well as their experience teaching information literacy skills, and they consulted existing information literacy standards and rubrics when creating the module.12 In keeping with student engagement theory, the module was not designed to test a specific student’s information literacy capabilities. Rather, it seeks to investigate how often students use information literacy skills and how institutions promote the proper use of information through their curriculum, as students’ time and effort in educationally beneficial activities are presumed to promote student learning and development. The module was pilot- tested in 2013 and subsequently refined and officially added to NSSE in 2014. Since 2014, 60,631 first-year and 83,762 senior students responded to the Experiences with Information Literacy module. These students attended 208 U.S. bachelor’s-granting institutions. The response rate was 22 percent for first-year students and 24 percent for seniors. Previous research indicates that NSSE data produces accurate and reliable group means at this response rate.13 Table 1 contains the characteristics of the sample. About two-thirds of the respondents were white. Six out of 10 students were female. Three in four students were 25 years old or younger. More than half of the sample had a parent who earned a bachelor’s degree or higher. The most common major was business; however, the sample was dispersed well among a variety of academic disciplines. A plurality of the respondents attended a research university. Two-thirds of the sample attended a public institution. I obtained the data from the Indiana University Center for Postsecondary Research, which operates NSSE. Researchers interested in replicating the analyses or using the data can request a data use license from the Indiana University Center for Postsecondary Research. The primary data used were 13 of the 14 items on the Experiences with Information Lit- eracy module. The lone item not included in the analyses was the final item asking about how students’ college experiences influenced their ability to use information literacy effectively, which did not comport with the engagement focus of the other items. A list of the items used from the module and their distributions are displayed in table 2. I also used data on two of the NSSE Engagement Indicators: Higher-Order Learning and Reflective & Integrative Learning. The Engagement Indicators are summary measures of various dimensions of student engage- ment. Information on the reliability and validity of the Engagement Indicators is available from NSSE’s Psychometric Portfolio.14 In addition to the Engagement Indicators, I used NSSE’s perceived gains scale, which is a composite of 10 items inquiring about how students’ college Information Literacy’s Influence on Undergraduates’ Learning and Development 275 TABLE 1 Descriptive Statistics of the Sample By Class Level First-year (%) Senior (%) Race/ethnicity Asian/Pacific Islander 6 6 Black/African American 7 7 Hispanic/Latin@ 12 11 White 63 67 International 5 4 Other race/ethnicity 6 5 Sex Female 67 63 Male 33 37 Parental Education Less than high school 5 5 High school 16 19 Some college 11 11 Associate's 9 10 Bachelor's 29 28 Master's 22 19 Doctoral or professional 8 8 Major field Arts & humanities 9 10 Biological sciences 12 9 Physical sci., math, & computer sci. 6 5 Social sciences 11 13 Business 15 17 Communications 4 4 Education 7 7 Engineering 7 7 Health professions 16 15 Social service professions 5 5 All other 4 7 Undecided, undeclared 3 <1 Basic 2010 Carnegie Classification (aggregated) Research universities 42 45 Master's colleges and universities 43 42 Baccalaureate colleges—Arts & Sciences 6 5 Baccalaureate colleges—Diverse Fields 6 5 Other/not classified 3 3 Control Public 68 68 Private 32 32 Note: Values are percentages and may not sum to 100 due to rounding. Sample is unweighted. 276 College & Research Libraries March 2020 TABLE 2 Distribution of the Experiences with Information Literacy module items by class level Variable name Item Response option First- year (%) Senior (%) During the current school year, about how often have you done the following? INL01A Completed an assignment that used an information source (book, article, Web site, etc.) other than required course readings Never 2 2 Sometimes 21 15 Often 37 30 Very often 40 53 INL01B Worked on a paper or project that had multiple smaller assignments such as an outline, annotated bibliography, rough draft, etc. Never 5 8 Sometimes 26 26 Often 39 32 Very often 31 34 INL01C Received feedback from an instructor that improved your use of information resources (source selection, proper citation, etc.) Never 5 9 Sometimes 28 29 Often 40 34 Very often 27 29 INL01D Completed an assignment that used the library’s electronic collection of articles, books, and journals (JSTOR, EBSCO, LexisNexis, ProQuest, etc.) Never 18 13 Sometimes 34 26 Often 28 26 Very often 20 35 INL01E Decided not to use an information source in a course assignment due to its questionable quality Never 25 27 Sometimes 40 39 Often 24 22 Very often 11 13 INL01F Changed the focus of a paper or project based on information you found while researching the topic Never 16 17 Sometimes 43 44 Often 29 26 Very often 12 14 INL01G Looked for a reference that was cited in something you read Never 16 13 Sometimes 37 35 Often 31 31 Very often 15 21 INL01H Identified how a book, article, or creative work has contributed to a field of study Never 20 17 Sometimes 38 36 Often 28 28 Very often 14 18 During the current school year, how much have your instructors emphasized the following? INL02A Not plagiarizing another author’s work Very little 2 3 Some 7 9 Quite a bit 21 21 Very much 70 66 Information Literacy’s Influence on Undergraduates’ Learning and Development 277 experience influenced their learning and development. The reliability of the perceived gains scales was .91. A list of the items contained in the two Engagement Indicators and perceived gains scale is displayed in the appendix. To estimate effect sizes efficiently, I standardized the Engagement Indicators and perceived gains scale to have a mean of 0 and standard deviation of 1. I also used data on a variety of student characteristics to control for differences in students’ background and academic characteristics. The control variables were sex, race/ethnicity, time spent working, major field, college grades, educational aspirations, parental education, age, and enrollment level. The Engagement Indicator and control variables were captured on the core NSSE instrument or provided by institutions. Analyses Following NSSE’s standard practices, I conducted all of the following analyses for first-year and senior students, separately. The initial analyses conducted an exploratory factor analysis (EFA) to identify information-literacy factors in the Experiences with Information Literacy module. I randomly sampled half of the students for use in the EFA (to preserve data for use in subsequent analyses). The Kaiser-Meyer-Olkin measure of sampling adequacy was .90, which Kaiser15 described as “marvelous.” Due to the ordinal nature of the module variables, the EFA used a diagonally weighted least squares estimator, as it does not assume that the variables are continuous.16 I identified the appropriate number of factors through an examina- tion of the scree plot of the extracted factors eigenvalues17 and practical experiences based on the development of the module. The factor loadings were rotated using an oblique, oblimin rotation with a Kaiser normalization. TABLE 2 Distribution of the Experiences with Information Literacy module items by class level Variable name Item Response option First- year (%) Senior (%) INL02B Appropriately citing the sources used in a paper or project Very little 2 4 Some 9 11 Quite a bit 27 25 Very much 61 61 INL02C Using scholarly or peer-reviewed sources in your course assignments Very little 4 6 Some 15 13 Quite a bit 30 25 Very much 51 56 INL02D Questioning the quality of information sources Very little 6 9 Some 20 22 Quite a bit 32 28 Very much 43 41 INL02E Using practices (terminology, methods, writing style, etc.) of a specific major or field of study Very little 8 6 Some 24 18 Quite a bit 30 29 Very much 37 47 Notes: Percentages may not sum to 100% due to rounding. 278 College & Research Libraries March 2020 After identifying an initial set of latent factors in the module, I conducted a confirmatory factor analysis (CFA) to verify the factor structure of the underlying data. CFAs differ from EFAs in that the factor structure is hypothesized by the researcher rather than derived from the data. The CFA was performed using the half of the sample not used in the EFA. Like the EFA, the analysis used a diagonally weighted least squares estimator, as the module variables were ordinal.18 The initial structure of the CFA model was based upon the EFA results. I al- lowed the latent factors to correlate as the EFA results suggested the factors were significantly correlated. The model was subsequently refined based upon the model fit information and modification indices. I used the following model fit information to assess whether the CFA met generally accepted standards. These criteria were a root mean square error of approxima- tion (RMSEA) not significantly different from .05 or lower, comparative fit index (CFI) of .95 or higher, Tucker-Lewis Index (TLI) of .95 or higher.19 I assessed the reliability of the factors identified by calculating Cronbach’s α coefficient for each of the factors. Finally, I investigated the predictive validity of the factors by examining their correlation with NSSE’s Higher-Order Learning and Reflective & Integrative Learning Engagement Indicators and the perceived gains scale, holding constant other characteristics. In these analyses, I standardized the factors to have a mean of 0 and a standard deviation of 1. To estimate the predictive validity of the factors, I estimated two regression models for each of the three outcome measures. The first model contained the control variables previously de- scribed and school-level fixed effects to account for the institution attended. The second model added the information literacy factors. This two-step approach allowed me to investigate how the inclusion of the information literacy factors improved the prediction of these outcomes. Limitations Readers should keep in mind the study’s limitations when interpreting the results. While the sample is large and diverse, it was a convenience sample and may not be generalizable to the larger population of undergraduates in the United States. Institutions choose to administer NSSE and the Experiences with Information Literacy module to their students. Thus, the aver- age institution is most likely more committed to assessment and evaluation and information literacy than the typical four-year institution. For example, research universities were over- represented and baccalaureate colleges were underrepresented in the sample. Additionally, the data were self-reported by students and maybe subject to social desirability bias. Finally, the module focused on engagement in information literacy activities, but it does not investigate the quality of information experiences. For example, an instructor could frequently instruct their students to use peer-reviewed sources, but this emphasis without instructing students how to access and identify peer-reviewed articles is of low quality. Results Exploratory Factor Analysis I began the analyses by estimating an EFA using a random half of the sample for both first-year and senior students discretely. An analysis of the scree plots for both class levels indicated that three factors should be extracted from the data. The rotated loadings for the three factors are available by class in table 3. After analyzing the results, I titled the factors: information use, information evaluation, and instructors’ emphasis. The first four items in the module loaded Information Literacy’s Influence on Undergraduates’ Learning and Development 279 TA B L E 3 Ex p lo ra to ry fa ct o r an al ys is fa ct o rs a n d r o ta te d lo ad in g s b y cl as s le ve l Fi rs t- ye ar Se n io r V ar ia b le It em In fo rm at io n U se In fo rm at io n Ev al u at io n In st ru ct o rs ’ Em p h as is In fo rm at io n U se In fo rm at io n Ev al u at io n In st ru ct o rs ’ Em p h as is IN L0 1A C o m p le te d a n a ss ig n m en t th at u se d a n in fo rm at io n s o u rc e o th er t h an re q u ir ed c o u rs e re ad in g s .6 01 .6 15 IN L0 1B W o rk ed o n a p ap er o r p ro je ct t h at h ad m u lt ip le sm al le r a ss ig n m en ts .8 05 .8 06 IN L0 1C R ec ei ve d fe ed b ac k fr o m a n in st ru ct o r t h at im p ro ve d y o u r u se o f i n fo rm at io n re so u rc es .5 87 .6 18 IN L0 1D C o m p le te d a n a ss ig n m en t th at u se d t h e lib ra ry ’s el ec tr o n ic c o lle ct io n o f a rt ic le s, b o o ks , a n d jo u rn al s .3 59 .3 41 .4 24 IN L0 1E D ec id ed n o t to u se a n in fo rm at io n s o u rc e in a co u rs e as si g n m en t d u e to it s q u es ti o n ab le q u al it y .6 17 .6 60 IN L0 1F C h an g ed t h e fo cu s o f a p ap er o r p ro je ct b as ed o n in fo rm at io n y o u fo u n d w h ile re se ar ch in g t h e to p ic .6 84 .7 17 IN L0 1G Lo o ke d fo r a re fe re n ce t h at w as c it ed in so m et h in g y o u re ad .7 20 .6 63 IN L0 1H Id en ti fie d h o w a b o o k, a rt ic le , o r c re at iv e w o rk h as c o n tr ib u te d to a fi el d o f s tu d y .6 92 .6 01 IN L0 2A N o t p la g ia ri zi n g a n o th er a u th o r’s w o rk .7 85 .8 43 IN L0 2B A p p ro p ri at el y ci ti n g t h e so u rc es u se d in a p ap er o r p ro je ct .8 55 .9 01 IN L0 2C U si n g s ch o la rl y o r p ee r- re vi ew ed s o u rc es in y o u r co u rs e as si g n m en ts .8 30 .8 13 IN L0 2D Q u es ti o n in g t h e q u al it y o f i n fo rm at io n s o u rc es .8 48 .8 17 IN L0 2E U si n g p ra ct ic es o f a s p ec ifi c m aj o r o r fi el d o f st u d y .6 55 .6 59 N o te s: It em la b el s sh o rt en ed . F ac to r l o ad in g s w er e ro ta te d u si n g a n o b lim in ro ta ti o n w it h a K ai se r n o rm al iz at io n . L o ad in g s w it h a n a b so lu te v al u e le ss th an .3 0 ar e n o t d is p la ye d . 280 College & Research Libraries March 2020 onto the information use factor. These items inquired about how often students used various information sources in their coursework or received feedback from an instructor on their information use. The second fac- tor, information evaluation, contained the four remain- ing items in the first question set. These items focused on how students evaluated and used information sources to conduct a richer literature review or revise their topic of study. The five items in the second question all loaded onto the third factor: instructors’ emphasis. These items all inquired about specific activities that instructors can emphasize to improve students’ use of information. One substantial cross-loading greater than .30 was observed for first-year students, but not seniors—the item asking if the student “Completed an assignment that used the library’s electronic collection of articles, books, and journals” loaded onto both the information use and evaluation factors. However, the loading with the larger magnitude was for information use, which comported with seniors’ results. The correlations between the fac- tors ranged from .32 to .49 for first-year students and .37 to .54 for seniors, indicating that the three factors capture independent, but correlated, phenomena. Confirmatory Factor Analysis With the remaining portion of the sample, I conducted a CFA. Figure 1 contains the initial conceptual model between the three factors. As shown in the conceptual model, the three factors are presumed to be latent variables measured by the specific survey items from the module. Additionally, the three latent factors were allowed to correlate, as the EFA results indicated a significant correlation between the factors. Despite the cross-loading observed for first-year students on the use of electronic collections, I attributed this variable to the information use factor due to the higher loading observed for the item on the information use factor and the results from the senior EFA model. When analyzed, the initial conceptual model did not adequately fit the data according to the model fit information. The modification indices indicated that the model needed to account for residual correlations between some of the items. After allowing some of the items to correlate, an adequate model fit was achieved. The RMSEA was .051, 90% CI [.050, .052], the CFI was .988, and the TLI FIGURE 1 Confirmatory Factor Analysis Conceptual Model Note: Error terms not displayed Information Literacy’s Influence on Undergraduates’ Learning and Development 281 FIGURE 2 Confirmatory factor analysis final model for first-year students Notes: Standardized loadings. Standard errors in parentheses. Error terms not displayed. 282 College & Research Libraries March 2020 FIGURE 3 Confirmatory factor analysis final model for senior students Notes: Standardized loadings. Standard errors in parentheses. Error terms not displayed. Information Literacy’s Influence on Undergraduates’ Learning and Development 283 was .983 for the first-year sample. For seniors, the RMSEA was .051, 90% CI [.050, .052], the CFI was .989, and the TLI was .985. All three criteria met the generally accepted standards for CFAs for both classes.20 Figures 2 and 3 contain the final model and standardized factor load- ings between the latent and observed variables for first-year and senior students, respectively. Construct Reliability After confirming the construct validity of the three factors in the model, I estimated the reliability of the three factors by calculating the Cronbach’s α for each composite variable. For first-year students, the αs for information use, information evaluation, and instructors’ emphasis were .77, .80, and .85, respectively. For seniors, the coefficients were .78, .79, and .86 for information use, information evaluation, and instructors’ emphasis, respectively. All α coefficients exceed the generally accepted threshold of .70 used for social science research and thus considered reliable for use in research contexts. Predictive Validity Finally, I assessed the predictive validity of the three factors by estimating regression models predicting the following NSSE composite variables: Higher-Order Learning, Reflective & Inte- grative Learning, and Perceived Gains. For each outcome, I first estimated a regression model TABLE 4 Fixed effect estimates of the relationship between information literacy factors and selected outcomes by class level Higher-Order Learning Reflective & Integrative Learning Perceived Gains   Est. Sig. Est. Sig. Est. Sig. First-year students Information Use 0.20 *** 0.15 *** 0.21 *** Information Evaluation 0.14 *** 0.24 *** 0.19 *** Instructors’ Emphasis 0.19 *** 0.13 *** 0.24 *** R2 change .18 .17 .26 Final R2 .20   .21   .28   Seniors Information Use 0.23 *** 0.19 *** 0.25 *** Information Evaluation 0.12 *** 0.21 *** 0.09 *** Instructors’ Emphasis 0.19 *** 0.13 *** 0.24 *** R2 change .19 .21 .21 Final R2 .22   .26   .26   * p < .05, ** p < .01, *** p < .001 Notes: Models hold constant race/ethnicity, time spent working, major field, grades, transfer status, educational aspirations, parental education, age, and enrollment status. Models include institution- specific fixed effects. The information literacy factors and outcome measures were standardized with a mean of 0 and standard deviation of 1. R2 change is the amount of additional explained variance by the information literacy factors after holding accounting for student and institutional factors. Final R2 is the total amount of variation in the dependent variable explained by the model. 284 College & Research Libraries March 2020 that included the aforementioned control variables and institution-specific fixed effects. The second model added the three information literacy factors. I estimated these models separately for first-year and senior students. As the dependent variables and information literacy fac- tors were standardized, the results represent the expected standard deviation change in the dependent variable for a standard deviation change in an independent variable. Table 4 contains the regression coefficient estimates of the information literacy factors when predicting the outcome measures, holding constant student and institutional characteristics. For each of the three outcome measures, each of the information literacy factors uniquely predicted a significant proportion of the variation in the Higher-Order Learning, Reflective & Integrative Learning, and Perceived Gains scales. Additionally, these relationships were all positive and not trivial, as all of the coefficient estimates were greater than 0.09. Therefore, a standard deviation change in one of the information literacy factors would be expected to result in roughly a tenth to quarter standard deviation change in Higher-Order Learning, Re- flective & Integrative Learning, and Perceived Gains. Despite the significant intercorrelation of the information literacy factors, each of the factors exerted a unique and significant influence on the outcomes. Therefore, the effects of more engagement in different types of information literacy activities appear to be additive. Finally, as demonstrated by the R2 change and final R2 statistics, the information literacy factors accounted for 17 to 26 percent of the variance in the outcome measures. This finding indicates that information literacy activities play an es- sential role in students’ learning and their perceived gains. Discussion In this study, I sought to investigate the validity and reliability of the Experiences with Informa- tion Literacy module using a large multi-institutional sample of first-year and senior students. To answer the first research question, I identified factors within the module through an EFA. The EFA results indicated that there were three latent factors within the module. I titled the factors Information Use, Information Evaluation, and Instructors’ Emphasis. Information Use focuses on students’ basic use of information in their course assignments. Information Evaluation focuses on a deeper and richer use of information by students through activities like assessing information quality, changing the focus of a paper based upon a literature review, and identify- ing the uniqueness of an information source. The final factor, Instructors’ Emphasis, examined the extent to which instructors encouraged the proper use of information in their courses. I subsequently estimated a CFA to confirm that the proposed factors adequately fit the data us- ing a CFA to answer the second research question. After accounting for residual correlations between some of the items, the CFA results adequately fit the data and supported the findings of three information literacy factors identified by the EFA. The third research question asked if the three factors could be reliably estimated. The factors were demonstrated to be reliable, as their Cronbach’s α coefficients ranged from .77 to .86. The fourth question focused on the predictive validity of the three factors in relation to engagement in higher-order learning and reflective and integrative learning activities. After controlling for student characteristics and the institution attended, each of the three factors was significantly and positively correlated to student engagement in these domains. The three information literacy factors also increased the R2 by about .20, indicating that the factors accounted for a substantial portion of the variation in student engagement related to higher-order learning and reflective and integrative learning. The final research question similarly inquired about the predictive validity of the factors in Information Literacy’s Influence on Undergraduates’ Learning and Development 285 relationship to students’ perceived gains. The results were similar to the engagement findings in that the information literacy factors were significantly and positively related to perceived gains and that the factors accounted for a relatively high proportion of the variance explained. Implications As validity is a multifaceted concept,21 the results from the study combine to indicate that the Experiences with Information Literacy module is a psychometrically valid instrument that can be used to assess undergraduates’ engagement with information literacy skills. The results supported the continued and expanded use of the module by institutions and instructional librarians in their assessment and evaluation activities. The module is relatively novel in that it focuses on the process of how students develop information literacy skills rather than directly testing their knowledge, thus adding a new method of assessment for instructional librarians. This feature of the module allows for users to identify specific areas of improvement. For example, an institution’s module results may reveal that instructors placed less emphasis on questioning the quality of information sources. In response, instructional librarians may work with faculty members to emphasize information quality in their lessons, grading, and syllabi. NSSE also provides institutions with the raw data to compare students by their characteristics. Such a subgroup analysis may indicate that instructors in a particular discipline placed less emphasis on the use of scholarly sources, and instructional librarians may work with faculty in that discipline to increase the emphasis on this topic. The validation of the module also leads to multiple lines of future inquiries. First, what student and institutional characteristics are correlated with information use and evaluation and instructors’ emphasis? For example, how does information literacy engagement vary by academic field or parental education? Similarly, are factors like the student-to-librarian ratio or selectivity related to information engagement? Second, how does participation in library instructional sessions influence students’ information engagement? The module data could demonstrate the long-term effectiveness of information literacy programming. Third, does information literacy engagement have compensatory effects on outcomes like retention and GPA? Prior work by NSSE researchers has shown that more general forms of engagement are especially effective in promoting the retention and GPA of underrepresented minority students;22 does this finding hold for information-related engagement? Finally, given the increasing dif- ficulty of determining high-quality news sources, does information-literacy engagement dur- ing college assist students to identify and evaluate news sources in their postcollegiate lives? Conclusion The study’s results demonstrate that the NSSE Experiences with Information Literacy mod- ule is an assessment tool meeting generally accepted standards for validity in social science research. They go on to highlight the significant and positive relationship between first-year and senior students’ engagement in information literacy activities and both Higher-Order Learning and Reflective and Integrative Learning. Additionally, engagement in information literacy activities was positively correlated with students’ self-assessments of their college learning gains, providing additional evidence of the importance of information literacy as a learning outcome. Finally, the study supports the use of the module by both institutions and librarians to assess and improve information literacy outcomes for students. 286 College & Research Libraries March 2020 APPENDIX. Items Comprising the Higher-Order Learning and Reflective & Integrative Learning Engagement Indicators and Perceived Gains Scale Higher-Order Learning During the current school year, how much has your coursework emphasized the following: [Response options: Very little, Some, Quite a bit, Very much] • Applying facts, theories, or methods to practical problems or new situations • Analyzing an idea, experience, or line of reasoning in depth by examining its parts • Evaluating a point of view, decision, or information source • Forming a new idea or understanding from various pieces of information Reflective & Integrative Learning During the current school year, how often have you: [Response options: Never, Sometimes, Often, Very Often] • Combined ideas from different courses when completing assignments • Connected your learning to societal problems or issues • Included diverse perspectives (political, religious, racial/ethnic, gender, etc.) in course discussions or assignments • Examined the strengths and weaknesses of your own views on a topic or issue • Tried to better understand someone else’s views by imagining how an issue looks from his or her perspective • Learned something that changed the way you understand an issue or concept • Connected ideas from your courses to your prior experiences and knowledge Perceived Gains How much has your experience at this institution contributed to your knowledge, skills, and personal development in: [Response options: Very little, Some, Quite a bit, Very much] • Writing clearly and effectively • Speaking clearly and effectively • Thinking critically and analytically • Analyzing numerical and statistical information • Acquiring job- or work-related knowledge and skills • Working effectively with others • Developing or clarifying a personal code of values and ethics • Understanding people of other backgrounds (economic, racial/ethnic, political, religious, nationality, etc.) • Solving complex real-world problems • Being an informed and active citizen Notes 1. For example: Jessica Taylor and Danielle Kurtzleben, “This Week in Trump’s ‘Alternative Facts,’” National Public Radio (last modified Jan. 29, 2017), available online at www.npr.org/2017/01/29/512068148/this-week-in- trumps-alternative-facts [accessed 5 February 2017]; John Wagner, “Trump Rants about ‘Fake News’ as He Marks http://www.npr.org/2017/01/29/512068148/this-week-in-trumps-alternative-facts http://www.npr.org/2017/01/29/512068148/this-week-in-trumps-alternative-facts Information Literacy’s Influence on Undergraduates’ Learning and Development 287 Black History Month,” Washington Post (last modified Feb. 1, 2017), available online at https://www.washington- post.com/news/post-politics/wp/2017/02/01/trump-rants-about-fake-news-as-he-marks-african-american-history- month/ [accessed 5 February 2017]. 2. American Library Association, “Presidential Committee on Information Literacy: Final Report,” American Library Association, available online at www.ala.org/acrl/publications/whitepapers/presidential [accessed 1 Feb- ruary 2017]; Kathleen L. Spitzer, Michael B. Eisenberg, and Carrie A. Lowe, Information Literacy: Essential Skills for the Information Age (Syracuse, NY: ERIC Clearinghouse on Information & Technology, Syracuse University, 1998); Association of College and Research Libraries, “Information Literacy Competency Standards for Higher Education” (Chicago, IL: Association of College and Research Libraries, 2000); Todd J. Wiebe, “The Information Literacy Imperative in Higher Education,” Liberal Education 102, no. 1 (2016). 3. Joseph R. Matthews, The Evaluation and Measurement of Library Services (Westport, CT: Libraries Unlimited Incorporated, 2007); Association of College & Research Libraries, The Value of Academic Libraries: A Comprehensive Research Review and Report. Researched by Megan Oakleaf (Chicago, IL: Association of College & Research Librar- ies, 2010); María Pinto, “Viewing and Exploring the Subject Area of Information Literacy Assessment in Higher Education (2000–2011),” Scientometrics 102, no. 1 (2015). 4. Kevin Fosnacht, “Information Literacy and NSSE Introducing the Experiences with Information Literacy Module,” College & Research Libraries News 75, no. 9 (2014). 5. Indiana University Center for Postsecondary Research, “The Carnegie Classification of Institutions of Higher Education, 2015 Edition” (Bloomington, IN: Indiana University Center for Postsecondary Research, n.d.); National Survey of Student Engagement, “About NSSE,” National Survey of Student Engagement, available online at http://nsse.indiana.edu/html/about.cfm [accessed 1 February 2017]. 6. Alexander C. McCormick, Jillian Kinzie, and Robert M. Gonyea, “Student Engagement: Bridging Research and Practice to Improve the Quality of Undergraduate Education,” in Higher Education: Handbook of Theory and Research, ed. Michael B. Paulsen (Dordrecht, The Netherlands: Springer, 2013). 7. Alexander W. Astin, “Student Involvement: A Developmental Theory for Higher Education,” Journal of College Student Personnel 25, no. 4 (1984); C.R. Pace, Measuring the Quality of College Student Experiences: An Account of the Development and Use of the College Student Experiences Questionnaire (Los Angeles, CA: Higher Education Research Institute, 1984). 8. Arthur W. Chickering and Zelda F Gamson, “Seven Principles for Good Practice in Undergraduate Edu- cation,” AAHE Bulletin 39, no. 7 (1987). 9. G.D. Kuh et al., Involving Colleges: Successful Approaches to Fostering Student Learning and Development outside the Classroom (San Francisco, CA: Jossey-Bass, 1991). 10. Samuel Messick, “Validity,” in Educational Measurement, 3rd edition, ed. Robert L. Linn (New York, NY: American Council on Education and Macmillan, 1989), 13–103. 11. Fosnacht, “Information Literacy and NSSE Introducing the Experiences with Information Literacy Mod- ule.” 12. Association of American Colleges & Universities, “Information Literacy Value Rubric,” Association of American Colleges & Universities, available online at https://www.aacu.org/value/rubrics/information-literacy [accessed 1 February 2017]; Association of College and Research Libraries, “Information Literacy Competency Standards for Higher Education.” 13. Kevin Fosnacht et al., “How Important Are High Response Rates for College Surveys?” Review of Higher Education 40, no. 2 (2017). 14. National Survey of Student Engagement, “Psychomentric Portfolio,” available online at http://nsse.indiana. edu/html/psychometric_portfolio.cfm [accessed 10 February 2017]. 15. Henry F. Kaiser, “An Index of Factorial Simplicity,” Psychometrika 39, no. 1 (1974). 16. Timothy A. Brown, Confirmatory Factor Analysis for Applied Research (New York, NY: Guilford, 2006). 17. Raymond B Cattell, “The Scree Test for the Number of Factors,” Multivariate Behavioral Research 1, no. 2 (1966). 18. Brown, Confirmatory Factor Analysis for Applied Research. 19. Li-tze Hu and Peter M. Bentler, “Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conven- tional Criteria Versus New Alternatives,” Structural Equation Modeling: A Multidisciplinary Journal 6, no. 1 (1999). 20. Hu and Bentler, “Cutoff Criteria for Fit Indexes in Covariance Structure Analysis.” 21. Messick, “Validity.” 22. George D. Kuh et al., “Connecting the Dots: Multi-Faceted Analyses of the Relationships between Student Engagement Results from the NSSE, and the Institutional Practices and Conditions That Foster Student Success,” (Bloomington, IN: Center for Postsecondary Research, 2007). https://www.washingtonpost.com/news/post-politics/wp/2017/02/01/trump-rants-about-fake-news-as-he-marks-african-american-history-month/ https://www.washingtonpost.com/news/post-politics/wp/2017/02/01/trump-rants-about-fake-news-as-he-marks-african-american-history-month/ https://www.washingtonpost.com/news/post-politics/wp/2017/02/01/trump-rants-about-fake-news-as-he-marks-african-american-history-month/ http://www.ala.org/acrl/publications/whitepapers/presidential http://nsse.indiana.edu/html/about.cfm https://www.aacu.org/value/rubrics/information-literacy http://nsse.indiana.edu/html/psychometric_portfolio.cfm http://nsse.indiana.edu/html/psychometric_portfolio.cfm