jan07b.indd Irvin R. Katz ETS research finds college students fall short in demonstrating ICT literacy National Policy Council to create national standards Educational Testing Service (ETS) released preliminary research that is consistent with a suspicion long held by those in academia: while students may be tech savvy when it comes to entertainment, they may not have the critical thinking skills to per­ form the kinds of information management and research tasks necessary for academic success. The research, gathered from 6,300 students who took ETS’s ICT Literacy As­ sessment in 2006, suggests that many of the students who participated lack some of the information and communication technology (ICT) literacy skills expected for college­level work: on average, students earned only about half the points that they could have earned on the test. College faculty and administrators help develop the assessment ETS developed the ICT Literacy Assessment under the guidance of college librarians, faculty, and administrators, who recognized a gap between students’ ICT literacy and the level of ICT skills that colleges and compa­ nies expected them to have, but were not able to create a reliable, valid, cost­effective assessment to measure the breadth or depth of the problem. Working from the ACRL “Information literacy competency standards for higher education” and other key documents, our higher education partners identifi ed seven ICT literacy skill areas that the assessment should measure: defining, accessing, man­ aging, evaluating, integrating, creating, and communicating information. Based on these skill areas and related performance indicators, ETS created an online environment in which students interact with simulated software to solve information prob­ lems, such as searching a library database, integrating information in a spreadsheet, and judging the reliability of information found in an Internet search. The assessment mea­ sures students’ critical thinking skills—not their knowledge of technology—but students must be able to use both the tools (such as a mouse) and generic applications (such as a word processing program) in order to complete the tasks. Two assessments serve colleges and high schools ETS currently offers two levels of the ICT Literacy Assessment: Core and Advanced. Each measures the same ICT literacy con­ struct covering the seven skill areas, and each consists of 14 four­minute tasks that target one skill area each and one 15­minute task that targets two skill areas. The ICT Literacy Assessment–Advanced Level targets students completing a two­year degree or technical program or rising juniors at a four­year insti­ tution. The Core Level assessment (designed to be easier) targets high school seniors and students in their first year of post­secondary education. The preliminary findings were compiled from college students and high school seniors Irvin R. Katz is senior research scientist for the Educational Testing Service, e-mail: ikatz@ets.org © 2007 Irvin R. Katz January 2007 35 C&RL News mailto:ikatz@ets.org who took either the Core or the Advanced level of the test. Findings show a need for improvement By spring 2006, more than 6,300 students at 63 institutions had taken the ICT Literacy As­ sessment, with institutions selecting students to test based on the institution’s assessment goals. Some chose to test students enrolled in a particular course, some recruited a random sample, and some issued an open invitation and offered gift certificates or other incentives. Because the data is not representative of all U.S. institutions or all higher education students, ETS urges caution in using these results to generalize to the greater population of college­age students. Even so, the prelimi­ nary results are rather revealing about the ICT literacy skills of participating students. During a task in which students evaluated a set of Web sites: • Only 52 percent judged the objectivity of the sites correctly • 65 percent judged the authority cor­ rectly • 72 percent judged the timeliness cor­ rectly • Overall, only 49 percent of test­takers iden­ tified the one Web site that met all criteria When selecting a research statement for a class assignment: • Only 44 percent identified a statement that captured the demands of the assign­ ment • 48 percent picked a reasonable, but too broad, statement • 8 percent picked statements that did not address the assignment C&RL News January 2007 ICT Literacy Assessment resources on ETS.org Homepage: www.ets.org/ictliteracy Preliminary fi ndings: www.ets.org /ictliteracy/prelimfi ndings.html When asked to narrow an overly broad search: • Only 35 percent selected the correct revision • 35 percent selected a revision that only marginally narrowed the search results Other results suggest that these students’ ICT literacy need further development: • In a Web search task, only 40 percent entered multiple search terms to narrow the results • When constructing a presentation slide designed to persuade, only 12 percent used only those points directly related to the argument • Only a few test takers accurately adapted existing material for a new audience • When searching a large database, only 50 percent of test takers used a strategy that minimized irrelevant results Of course, there were some positive findings, as well. For example, test takers ap­ peared to recognize that .edu and .gov sites are less likely to contain biased material than .com sites. Eighty percent of test takers correctly com­ pleted an organizational chart based on e­mailed personnel information. Most test takers correctly categorized e­mails and files into folders. And when presented with an unclear assignment, 70 percent of test takers selected the best question to help clarify the assignment. Colleges use findings to guide decisions Participating institutions are using the pre­ liminary results to help stakeholders prioritize their educational goals for the ICT literacy of their student population—in addition to help­ ing guide the instruction of the individuals who took the test. Alexius Macklin, associate professor of library science at Purdue University, said that the preliminary research fi ndings illustrate that most students do not have the ICT lit­ eracy skills needed to complete college­level assignments efficiently. “The reality is that when you give students a research assign­ ment, they go straight to Google without any 36 thought to their actual research question or the information need,” Macklin says. “They draw information from questionable resourc­ es because they don’t know the difference between information they find from an ad or a biased source, and that which they fi nd on an authoritative, timely, objective site. The preliminary research from ETS shows us that a majority of our students are not ICT literate enough to succeed academically . . . they do not currently have the skills to analyze and synthesize information into something man­ ageable and useful for their needs.” While the initial snapshot of students’ ICT literacy skills is bleak, Macklin says, the good news is that it is possible to teach these skills. “The preliminary findings from ETS show us that institutions need to consider how to bet­ ter integrate ICT literacy skills into and across the curricula. It may require initiating an ICT literacy initiative or allocating resources dif­ ferently. It’s important to help our students better evaluate, manage and communicate in­ formation so that they can succeed in school, at work and in life. And now we know that the results are measurable.” As more institutions administer the assess­ ment, further research will clarify ICT literacy performance at individual institutions and across the United States. National Policy Council to set benchmarks In October, the National Forum on Informa­ tion Literacy announced the formation of the National ICT Literacy Policy Council, which is charged with providing leadership in creating national standards for ICT literacy. Lizabeth (Betsy) Wilson, dean of university libraries, University of Washington, and former ACRL president, is one of the leaders who has committed to serving on this council. The council’s chief work will be to review current ICT literacy and information literacy assess­ ments and standards documents, determine the number of assessment levels, name those levels (such as advanced profi ciency, accept­ able proficiency, and minimal profi ciency), and provide descriptions for each level. In short, the council will determine what students should know and be able to achieve at each ICT literacy level. The council will also recruit panels of experts to review the ETS ICT Literacy Assess­ ment to recommend cut points for each of the performance levels. These standards should help educators determine which students meet the ICT literacy standards and which may need additional instruction or remediation. The council will review these recommended cut points to modify and/or accept them as appropriately reflecting national ICT literacy standards. As a first step toward assessing ICT literacy skills nationally, the preliminary fi ndings provide information to colleges and high schools about the ICT literacy of the students who participated, as well as to individual students about their own performance. With more data and further research, ETS hopes to continue to support the educational commu­ nity dedicated to improving the ICT literacy of students. !�SUITE�OF�SERVICES�OFFERED�BY�THE�� !SSOCIATION�OF�2ESEARCH�,IBRARIES 6ISIT�US�IN�3EATTLE "OOTH����� !TTEND�THE�WORKSHOPS� -ONDAY��*ANUARY���� ,IB15!,���������!N�)NTRODUCTION� HTTPS���DB�ARL�ORG�LQINTRO� 9OUR�,IB15!,���#OMMUNITY�� !�2ESULTS�-EETING� HTTPS���DB�ARL�ORG�LQRESULTS� WWW�LIBQUAL�ORG ,IB15!,� &KDUWLQJ�/LEUDU\�6HUYLFH�4XDOLW\��� � January 2007 37 C&RL News