945 Developing a Framework to Improve Information and Digital Literacy in a Bachelor of Paramedic Science Entry-to-Practice Program Nigel Christopher Barr, Bill Lord, Belinda Flanagan, and Roger Carter* The aims of this research were threefold: (1) explore the perceptions of paramedic educators regarding information literacy education (ILE); (2) reveal the information literacy (IL) abilities of paramedic science students; and (3) develop an integrated IL framework for paramedicine. Two key findings arose from this research. First, para- medic educators need to include higher functions of synthesis and creative thinking in ILE. Second, ILE needs to be mapped and scaffolded to incrementally develop these skills across each year of the program. The results informed the development of a programwide ILE framework implemented in 2019 and to be evaluated in 2021. Introduction In Australia, New Zealand, and the United Kingdom, paramedic education has transitioned from the vocational education sector to the university sector. Personal communication with nationally accredited educators in the United States has also indicated a strong interest in transitioning their paramedic training to a baccalaureate. Paramedics in Australia are engaged to provide community–based emergency healthcare,1 and the Australian Health Practitioner Regulation Agency (AHPRA) registers paramedics once they graduate from an accredited, usually three-year, university program. To facilitate and sustain the change in education of paramedics, upskilling of clinicians to serve as faculty staff has occurred. In Australia, Domain 3 of the Higher Education Standards Framework (Threshold Standards) 20152 requires faculty staff to have: a) knowledge of the discipline informed by continuing scholarship, research, or advances in practice; b) skills in contemporary teaching, learning, and assessment principles relevant to the discipline; and c) hold a qualification in a relevant discipline at least one level higher than is awarded for the course of study. Accordingly, many paramedics working as faculty staff within universities have obtained postgraduate qualifications in education or re- search and some have completed a PhD.  Nigel Christopher Barr is Discipline Leader—Paramedicine at the School of Nursing, Midwifery and Paramedicine, email: nbarr@usc.edu.au; Bill Lord is in the Department of Paramedicine at Monash University, email: bill.lord@ monash.edu; Belinda Flanagan is Lecturer in Paramedicine at the School of Nursing, Midwifery and Paramedicine, email: Belinda Flanagan bflanaga@usc.edu.au; Roger Carter is Liaison Librarian, Science, Health & Sport Sciences at the University of the Sunshine Coast Library, email: rcarter@usc.edu.au. ©2020 Nigel Christopher Barr, Bill Lord, Belinda Flanagan, and Roger Carter, Attribution-NonCommercial (https://creativecommons.org/licenses/ by-nc/4.0/) CC BY-NC. 946 College & Research Libraries September 2020 During this transition, Australian government interventions during the last decade have increased participation rates in higher education.3 A new demand-driven system with fund- ing incentives has substantially increased the proportion and diversity of students who are successful in applying for university admission, and this has broadened student preparedness for tertiary education.4 In this era where social inclusion policies have changed patterns of participation, universities still assume the responsibility of maintaining academic standards across all participants.5 However, Australian research has shown significant variance in infor- mation literacy (IL) skills among commencing university students, and often poor acquisition of IL during their program of study.6 Paramedicine is an evolving profession responding to changing community needs,7 situ- ated in a healthcare setting where evidence rapidly changes and diagnostic uncertainty is part of clinical decision making.8 Both AHPRA and the Australian Industry and Skills Committee (Skills IQ) expect paramedics to analyze and critically evaluate the information they have collected to make clinical judgments.9 Hence, IL is a foundational skill in paramedicine and is a critical component of evidence-based practice (EBP), as it allows a practitioner to remain current in their field.10 The skills developed during the formative undergraduate years must equip graduates with the essential problem-solving and critical-thinking skills required for their continued professional development and concomitant delivery of safe and effective healthcare. As such, staff responsible for curriculum design must consider the needs of stu- dents from varying backgrounds and with differing capabilities when embedding information literacy education (ILE) into the curriculum. The necessity of improving the EBP skills of graduating paramedics in Australia has en- abled a productive librarian-faculty collaboration to move forward in this present research project with the aim of genuine curriculum renewal that will graduate clinically effective, information-literate practitioners. This research blends three distinct discipline strands: paramedic education; educational theory; and ILE with a view to effecting real change in the education of Australian paramedics. Developing a sustainable program of ILE was at the centre of this project. Literature Review The urgency of creating an information literate society has long been formally recognized in the United States,11 the UK,12 and Australia.13 The American Library Association14 has de- fined IL as “the set of integrated abilities encompassing the reflective discovery of information, the understanding of how information is produced and valued, and the use of information in creating new knowledge and participating ethically in communities of learning.” The Library and Information Association in the UK15 has broadened this definition and implied it is necessary to empower a full life, “Information literacy is the ability to think critically and make balanced judgements about any information we find and use. It empowers us as citizens to develop informed views and to engage fully with society.” Biggs’ principle of “constructive alignment” is based on the Constructivist theories of Bloom, Vygotsky, Dewey, and others and is now an accepted, widely implemented guide underpinning curriculum development.16 The concept of “Informed learning”17 proposes that, when students are aware of information use in their learning, they become more aware of how they may use information in their future academic, personal, and professional lives. Informed learning echoes constructive alignment18 as embedding ILE across a program, requiring three Developing a Framework to Improve Information and Digital Literacy 947 considerations: (1) identifying the critical aspects of intended learning, which include both us- ing and communicating information and subject content; (2) designing assessment methods to evaluate a student’s ability to use information and subject content; and (3) designing transfor- mative learning activities that enable students to pass through threshold concepts regarding important aspects of subject content by using information in a critical way. A recent systematic review19 has demonstrated wide support for these concepts, but it is not universally applied across programs; and ILE within programs may range from a single 50-minute session assessed with a simple paper survey to fully course-integrated IL programs with robust assessment. There have been repeated attempts to reinvigorate the interest in IL as an essential com- petency in all disciplines,20 including paramedicine,21 and there is a body of research explor- ing how students access information or how to improve specific IL skills in single courses or units of study.22 In the health sciences, the discipline of nursing has had the most attention in relation to ILE. This could simply be because there are far more nurses being trained, in more places, and at any given time than in any of the other health disciplines. There have been active nursing-related ILE projects for decades, including some curriculum-aligned and evaluated examples.23 However, not all such efforts have been satisfactory,24 indicating that curriculum design to guide “what the learner does”25 is critical to success. In the health science disciplines, there are few quality studies exploring the integration of IL across whole programs of study. Much previous librarian-led innovation has failed because of a lack of faculty knowledge or commitment26 as successful ILE demands a committed on- going partnership between library and faculty. The centrality and necessity of incorporating ILE across many disciplines has long been noted,27 and serious attempts have been made to embed ILE into whole programs.28 Notable Australian studies are by Wang,29 which described integration of ILE in an engineering degree; Salisbury and Sheridan,30 which describes ILE renewal across multiple programs; and the decade-long development and implementation of the RSD framework.31 Elsewhere, however, partial efforts, especially involving the first year of programs, have been much more common,32 possibly because they are easier to achieve in the short term and require less high-level institutional commitment. To facilitate the required curriculum change, this project had three aims: (1) to explore the perceptions of paramedic faculty staff working in Australian university programs regarding IL; (2) to explore the IL abilities of a defined cohort of the University of the Sunshine Coast (USC) paramedic students; and (3) to develop a framework for embedding ILE through the entire Bachelor of Paramedic Science curriculum at USC, informed by aims 1 and 2. Methods This project has defined curriculum as a plan for the acquisition of knowledge and skills lead- ing to a credentialed qualification and professional registration. It includes both the official course syllabus, the hidden curricula and extracurricular activities a student is exposed to during a program of study. Information literacy has multiple definitions, as it is context bound. To suit the Australian context, Wang’s Information Literacy Curricular Integration Model33 was chosen to inform the curriculum changes required to develop students’ ability to understand basic scientific data and to evaluate and synthesize information. To contextualize the attributes of IL to Australian tertiary programs and students, our project used the six broad standards of the Australian and New Zealand Information Lit- 948 College & Research Libraries September 2020 eracy (ANZIL) framework34 to outline the breadth of IL expected within Australian curricula (see table 1). Following a period of debate, the Association of College and Research Librar- ies rescinded their Information Literacy Competency Standards for Higher Education (2000) and replaced them with their Framework for Information Literacy in Higher Education (2016).35 It can be argued, however, that a Standards-based approach usefully articulates with the necessary requirements of EBP.36 For our purposes, it was judged best to proceed using the standards based approach of ANZIL rather than the newer ACRL Framework approach. Data Collection This project used sequential mixed-methods37 to investigate two research questions. This was the first research question: How do members of the Network of Australasian Paramedic Academics (NAPA) TABLE 1 Council of Australian Librarians (CAUL) Information Literacy Standards and Elements 1. Recognizes the need for information and determines the nature and extent of the information needed 1.1 Defines and articulates the information needed 1.2 Understands the purpose, scope, and appropriateness of a variety of information sources 1.3 Re-evaluates the nature and extent of the information needed 1.4 Uses diverse sources of information to inform decisions 2. Finds needed information effectively and efficiently 2.1 Selects the most appropriate methods or tools for finding information 2.2 Constructs and implements effective search strategies 2.3 Obtains information using appropriate methods 2.4 Keeps up to date with information sources, information technologies, information access tools, and investigative methods 3. Critically evaluates information and the information-seeking process 3.1 Assesses the usefulness and relevance of the information obtained 3.2 Defines and applies criteria for evaluating information 3.3 Reflects on the information-seeking process and revises search strategies as necessary 4. Manages information collected or generated 4.1 Records information and its sources 4.2 Organizes (orders/classifies/stores) information 5. Applies prior and new information to construct new concepts or create new understandings 5.1 Compares and integrates new understandings with prior knowledge to determine the value added, contradictions, or other unique characteristics of the information 5.2 Communicates knowledge and new understandings effectively 6. Uses information with understanding and acknowledges cultural, ethical, economic, legal, and social issues surrounding the use of information 6.1 Acknowledges cultural, ethical, and socioeconomic issues related to access to, and use of, information 6.2 Recognizes that information is underpinned by values and beliefs 6.3 Conforms with conventions and etiquette related to access to, and use of, information 6.4 Legally obtains, stores, and disseminates text, data, images, or sounds Developing a Framework to Improve Information and Digital Literacy 949 rate the importance of information literacy education in paramedic programs? NAPA is a special inter- est group of Paramedics Australasia (PA), a peak professional body representing Australian and New Zealand paramedics. PA provided in-kind support for this study by sending emails to all members of the NAPA to promote the study. All participants of the study were employed as fac- ulty staff by Australian or New Zealand universities. No other demographic data were collected. A modified Delphi process, as described by Lindstone and Turoff,38 was used with three iterative rounds. This method was chosen because it could inform future educational design by describing the aspects of ILE that paramedic educators value. The Delphi process was conducted online using the survey program Opinio® to achieve consensus on learning outcomes based on the ANZIL standards. NAPA was chosen as the reliability of data from Delphi studies are improved when subjects are drawn from the specialized area of knowledge related to the target issue.39 The modified Delphi process began with an initial survey (see appendix A) sent to the respondent group. This survey had two sections; the first proposed six short-answer questions concerning the participants’ perceptions of the information literacy standards, as defined by the ANZIL framework. The second section asked the participants to rate the value they placed on learning outcomes related to the different skills that comprise information literacy and to indicate at which chronological point in an undergraduate program that the student should be able to demonstrate attainment of that outcome. The learning outcomes were developed from generic IL learning outcomes publicly available on Australian University web pages and presented in the survey (see appendix A) attached with the appropriate ANZIL standard. We did this for two reasons: (1) because of the relative inexperience of paramedic faculty, it was deemed beneficial to present several learning outcomes based on the standards for the par- ticipants to consider; and (2) it may give rise to themes within each standard to help design learning activities attached to individual standards. The first survey section was presented once only to the participants. The results of the second section were analyzed, and the questionnaire was modified for each iterative round. The items that achieved greater than 80 percent consensus were removed so that each itera- tive round of the questionnaire contained only the learning outcomes that had not reached consensus on importance or place in a program. With each iteration, the respondent group was given the opportunity to re-evaluate their original answers to questions that did not achieve consensus based upon examination of the group response. The second research question—How do paramedic students at USC perceive their information literacy abilities?—was explored through an online questionnaire sent to all USC paramedic stu- dents (see appendix B). The findings will inform teaching strategies to enable foundation skills and the incremental development of higher order skills across the three years of the program. The student survey was based on two validated surveys40 developed by CAUL that were linked to the ANZIL framework. The survey was modified for this project based on the recommendations of the project reference group consisting of a librarian, an academic skills advisor, a curriculum designer, and a senior paramedic member of faculty. The survey was piloted with a small convenience sample of four faculty academic staff and 14 students from Bachelor of Nursing and Bachelor of Sport and Exercise Science programs at USC. This enabled the assessment of face validity and refinement of the question stems and response choices to ensure that the questionnaire could be completed within 15 minutes. Construct validity was assessed through discussions with reference group members, email feedback received from pilot participants and their responses to items. 950 College & Research Libraries September 2020 All USC paramedic students (n = 470) were invited to participate by email. Data collection from student participants was conducted online using Opinio®. Participation was anonymous and voluntary. The survey was open for four weeks, with weekly reminders sent by email to potential participants to increase response rate. No incentives were offered. Data Analysis For the Delphi study, the categorical data were summarized using descriptive statistics to determine whether consensus was reached. For the student survey, the nontextual data (cat- egorical, ordinal, and Likert scale responses) were summarized using descriptive statistics for variables. Inferential statistics were then computed using the Fisher’s exact test (FET) to test for associations among variables, gender, and year level in the program. The textual data from the Delphi study and student survey were analyzed through an interactive process of describing, classifying, and connecting information using NVivo Ver- sion 10. This process used a combination of collating data around a small number of a priori codes followed by content analyses, where inductive codes were used to capture emergent themes.41 The content analysis was conducted by one researcher and then checked by a second researcher to improve the accuracy of coding. Ethical clearance was obtained from the University of the Sunshine Coast Human Research and Ethics Committee (A/15/662 and A/17/922). Results and Discussion The results and discussion for the two studies will be presented in turn. We will then discuss how the results have informed curriculum renewal in our program. A wide range of definitions of IL were provided by the Delphi participants. For example, one participant defined IL as “Recognising when information is needed and having the necessary skills to get hold of that information,” and another provided a deeper insight: “identifying an information gap, knowing how and where to find the information required, being able to judge the quality of the information and to use the information appropriately.” Table 2 shows the frequency of the themes that arose from the participant comments. All the Delphi participants indicated that students undertaking a paramedic entry-to-practice qualifica- tion need to develop information literacy skills. The comments provided by participants indicated that educational design needs to ensure that students learn how to challenge, critique, and evaluate information. For example, one participant commented: “students need to be able to locate, evaluate and critique information on a regular basis. These skills are required throughout their professional life.” Delphi participants generally agreed with the CAUL statements regarding benefits associated with the development of IL skills (see table 3). Participant comments also described three additional benefits of ILE: empowerment, safe clinical practice, and improved work readiness. TABLE 2 Themes Arising from NAPA Participants (n = 29) Responses to Defining the Meaning of Information Literacy as a Skill Theme Frequency, Percentage Find/Collect n = 22, 84.6% Critique/Evaluate n = 14, 53.8% Integrate n = 10, 38.5% Understand n = 10, 38.5 Identify Knowledge Required n = 9, 34.6% Communicate n = 5, 19.2% Learn n = 1, 3.8% Manage Data n = 1, 3.8% Developing a Framework to Improve Information and Digital Literacy 951 Consensus on ILE Learning Outcomes and Position within a Program Delphi participants indicated the level of importance (high, moderate, or low) of learning out- comes and place in the program where these outcomes need to be achieved (early, middle, or late). Consensus on the importance of place was achieved if 80 percent of participants voted for the same category of importance low, moderate, or high. After three rounds of voting, half (n = 33) of the presented learning outcomes were con- sidered to be of high importance in a paramedic program. Another 3 (5%) of the learning outcomes were judged to be of medium importance and consensus was not reached on the rest (n = 28, 42%). The lowest level of consensus occurred with learning outcomes grouped under standard 3 “Critically evaluates information and the information-seeking process,” with only one quarter of these learning outcomes reaching consensus. No consensus was gained for the three learning outcomes on assessing the usefulness and relevance of the information obtained, one out of four achieved consensus on learning outcomes relating to reflecting on the information-seeking process and revising search strategies, and only two from the five learning outcomes on defining and applying criteria for evaluating information reached con- sensus on importance. To assist designers of paramedic curricula, we have collated the themes under each ANZIL standard (teaching activities or context) arising from learning outcomes that achieved consensus for a moderate to high level of importance in a paramedic program (see appendix C). In addition, the Delphi participants indicated that many aspects of IL should be taught at the commencement of a program of study (see table 4), such as finding, describing, storing, and managing information. The front loading of foundation IL is a logical position, as these skills support the development of higher order skills such as analysis, evaluation, critique, and synthesis. University graduates must be able to develop IL skills to prepare for a changing practice environment.42 Scope of practice for paramedics has changed substantially over the last de- cade,43 which is associated with a requirement for higher levels of critical thinking and clinical reasoning in paramedic-led healthcare due to increased complexity in decision making. Yet TABLE 3 Level of Agreement (%) of Delphi Participants (n = 19) with Statements Regarding the Benefits to Students Who Have Developed Information Literacy Skills Statements Disagree Neither Agree Students who develop information literacy skills can engage in independent learning through constructing new meaning, understanding, and knowledge. 5.3 0.0 94.7 Students who develop information literacy skills derive satisfaction and personal fulfilment from using information wisely. 10.5 21.1 68.4 Students who develop information literacy skills can search for and use information for decision making, and problem-solve to address personal, professional, and societal issues. 5.3 0.0 94.7 Students who develop information literacy skills can demonstrate social responsibility through a commitment to lifelong learning and community participation. 0.0 26.3 73.7 Note: Statements adapted from the ANZIL information literacy framework. 952 College & Research Libraries September 2020 TABLE 4 Learning Outcomes That Reached Consensus (>80%) for Importance (High, Moderate, or Low) and the Position in the Curriculum (Early, Middle, Late, or No Consensus [NC]) When the Skill Should Be Achieved Learning Outcome Importance Point in Program 1. Recognizes the need for information and determines the nature and extent of the information needed 1.1.1 Explores general information sources to increase familiarity with the topic High Early 1.1.4 Identifies key concepts and terms to formulate the focus questions High NC 1.2.1 Understands how information is organized and disseminated High Early 1.2.2 Recognizes that the context of the topic changes among the disciplines High NC 1.2.3 Differentiates between and values the variety of potential sources of information High NC 1.2.5 Differentiates between primary and secondary information sources High Early 1.3.1 Reviews the initial information to clarify, revise, or refine the question High Middle 1.4.2 Uses a range of sources to understand the issues High NC 1.4.3 Uses information for decision making and problem solving High NC 2. Finds needed information effectively and efficiently 2.1.1 Investigates benefits and applicability of various investigative methods or research questions High NC 2.1.3 Consults with librarians and other information professionals to help identify information access tools High Early 2.2.1 Develops a search plan appropriate to the research question High NC 2.2.2 Identifies keywords, synonyms, and related terms for the information needed High Early 2.2.4 Constructs and implements a search strategy using appropriate commands High Middle 2.4.2 Uses research update services Moderate NC 3. Critically evaluates information and the information-seeking process 3.2.2 Analyzes the structure and logic of supporting arguments or methods High Middle 3.2.3 Recognizes and questions prejudice, bias, deception, or manipulation High NC 3.3.3 Reviews information access tools used and expands to include others as needed Moderate Middle 4. Manages information collected or generated 4.1.2 Differentiates between the types of sources cited and understands the correct citation style for a wide range of resources High Early 4.1.3 Is able to access the correct citation style information High Early 4.1.4 Records all pertinent citation information for future reference and retrieval High Early 4.2.1 Compiles references in the required bibliographic format High Early 5. Applies prior and new information to construct new concepts or create new understandings Developing a Framework to Improve Information and Digital Literacy 953 TABLE 4 Learning Outcomes That Reached Consensus (>80%) for Importance (High, Moderate, or Low) and the Position in the Curriculum (Early, Middle, Late, or No Consensus [NC]) When the Skill Should Be Achieved Learning Outcome Importance Point in Program 5.1.1 Determines whether information satisfies the research or other information need and whether the information contradicts or verifies information used from other sources High Middle 5.1.2 Recognizes interrelationships between concepts and draws conclusions based upon information gathered High Middle 5.1.3 Selects information that provides evidence for the topic and summarizes the main ideas extracted from the information gathered High NC 5.1.5 Extends initial synthesis at a higher level of abstraction to construct new hypotheses Moderate Late 5.2.2 Uses a range of appropriate information technology applications in creating the product (such as report, essay, journal) Moderate Middle 5.2.3 Incorporates principles of design and communication appropriate to the environment Moderate Middle 5.2.4 Communicates clearly and in a style to support the purposes of the intended audience High Early 6. Uses information with understanding and acknowledges cultural, ethical, economic, legal, and social issues surrounding the use of information 6.1.2 Understands and respects indigenous and multicultural perspectives of using information High Early 6.2.2 Applies reasoning to determine whether to incorporate or reject viewpoints encountered High Middle 6.2.3 Maintains an internally coherent set of values informed by knowledge and experience High NC 6.3.1 Demonstrates an understanding of what constitutes plagiarism and correctly acknowledges the work and ideas of others High Early 6.3.2 Participates in online discussions following accepted practices (such as Netiquette) High Early 6.4.1 Understands fair dealing in respect of the acquisition and dissemination of educational and research materials High Early 6.4.2 Respects the access rights of all users and does not damage information resources High Early 6.4.3 Obtains, stores, and disseminates text, data, images, or sounds in a legal manner High Early 6.4.4 Demonstrates an understanding of intellectual property, copyright, and fair use of copyrighted material High Early Note: CAUL, Council of Australian University Librarians; IL, information literacy; NC, indicates that no consensus was reached regarding when the skill should be demonstrated. 954 College & Research Libraries September 2020 the Delphi results indicate that lower-order IL skills are more valued than higher-order skills. This was demonstrated by a lack of consensus regarding learning objectives pertaining to Standard Three: critically evaluates information and the information seeking process. Furthermore, the Delphi participant definitions of information literacy emphasized lower-order skills of finding or collecting information rather than evaluation and synthesis. This finding is unusual because IL is considered the basis for evidence-based-practice,44 but it is consistent with pre- vious Australian studies showing that IL is often underdeveloped in university graduates.45 Student Survey The survey was administered in semester one 2017 and open for four weeks (27 March to 28 April 2017). The survey was sent to 470 students enrolled in the Bachelor of Paramedic Science program at USC from which 111 students started the survey and 77 students fully completed it. Not all participants answered all questions, and the 30 partially completed surveys did not include demographic attributes (see table 5). Respondents reported their sex as males (n = 30), female (n = 47), and missing (n = 30), and no statistically significant associations between sex and response variables were found using = 0.05. The partially completed surveys were included when reporting descriptive statistics but treated as missing when exploring associa- tions between demographic attributes (sex, position in program, age, and level of education) and other variables. Organizing Information Participants were asked: Briefly explain how you organize the information and sources gained through information searches for an assignment topic (Q10). The 68 comments aligned to three themes: decisions, indexing, and storage. The decisions theme included keeping the informa- tion based on: the relevance (n = 9); the quality of information (n = 4); and the date of publica- tion (n = 2). The indexing theme included alphabetical (n = 1), chronological (n = 2), order of relevance (n = 3), reference list (n = 5), making a research book (n = 1), making a themes table (n = 6), and by topic or section of assignment (n = 17 and n = 3 respectively). The storage theme included hard copy (n = 8) or soft copy (n = 7). Of note was that only a minority of students TABLE 5 Demographic Attributes of Student Participants (n = 77, missing = 34), Who Provided Their Demographic Attributes Attribute Detail Sex Male Female 39% (n = 30) 61% (n = 47) Position in Program Entry Middle Year(s) Final Year 48.1% (n = 37) 26.0% (n = 20) 26.0% (n = 20) Age Range Mean 43 years; 17–60 years 23 years Level of Education Year 12 Diploma or Certificate Bachelor’s Degree Other 53.9% (n = 41) 26.3% (n = 20) 13.2% (n = 10) 6.6% (n = 5) Developing a Framework to Improve Information and Digital Literacy 955 used bibliographic software to assist with the organization and citation of information. The benefits of bibliographic software are beyond simple referencing, as many applications allow linking of articles through memo writing and annotating data that is entered. This process may facilitate metacognition relating to the quality of the information being collected.46 Understanding Scholarly Requirements: Cultural, Sources [Diversity and Type] and Citations Participant responses describing what cultural sensitivity meant in the context of writing an assignment fell into three groups: (1) awareness, (2) being respectful, and (3) minimizing bias and prejudice. Twenty-five comments were made regarding awareness and included being aware of differences in philosophies, that cultural perspectives influence needs, and that ethical considerations need to be considered. Twenty-three comments were made about being respectful, which included not causing offense, and 10 comments were made about minimizing bias and prejudice. Students generally reported good scholarly habits when completing written assignments. Strong evidence was found that final-year students were more likely to report increased use of multiple sources of information to reach a conclusion (FET = 23.467, P = 0.002; first-year students 10.8% compared to final-year students 60%). Half of the participants (54.1%, n = 46) indicated that they were required to cite scholarly articles or textbooks to support their argument in a reflective essay, and this perception also increased with each additional year of study (entry year 24.3% compared to final year 95.0%). Students reported increased use of a diverse range of sources with each year in the program (FET = 9.455, P = 0.027; entry year 24.3% compared to final year 65.0%). Participants were also asked to indicate from a list which sources of information they used to complete an assignment (Q4). The students indicated that they sought information mostly from peer-reviewed journals (97.3%) and textbooks (82.0%) followed by government reports (64.9%) and online reference sites such as Medscape (60.4%). Collaborative encyclopaedias such as Wikipedia and news media were the least used (12.6% and 2.7% respectively). No statistically significant changes in response were found between program years. Only the first-year participants reported us- ing news media for assignments where this was not appropriate for the task. However, the use of collaborative encyclopedias such as Wikipedia remained the same across the program, and the use of textbooks for citing high-level evidence in assignments was still high at the end of the program. This may indicate a lack of understanding of assignment requirements, poor design of assessment tasks, or a lack of skills to access published evidence. Approximately three-quarters of students indicated that they often or always worked out how to explain a new idea effectively (73.7%), set out the main ideas from each source, and then combined them to generate new knowledge for an assignment (78.5%). No statisti- cally different changes in response pattern was found between participant year level and the items describing checking whether a concept is still current while developing an answer for an assignment, using a systematic process to help judge the relevance of the information, or using bibliographic software to manage collected information. Most students reported good behaviors of being aware of plagiarism. Three-quarters (74.1%, n = 63) of the survey participants indicated that they have read USC policies regarding plagiarism and there was no statistical evidence of change in response by year level (FET = 7.059, P = 0.112). However, there was a statistical difference (FET = 13.330, P = 0.022) in the reported 956 College & Research Libraries September 2020 use of the USC Style Guide to ensure correct citation of information. From first to third year, there was a slight increase in those reporting they often used the style guide (24.3% to 30.0%) and a large reduction in those reporting they always used the style guide (62.2% to 40.0%). Communicate Scientific Research Approximately half (52.9%, n = 45) of the participants reported that they could effectively communicate statistical research outcomes to others in writing (Q19). Nearly a third (32.9%, n = 28) of participants were not sure, and the remainder (14.1%, n = 12) indicated that they could not. No statistical association was found between response type and year of study. The participants were asked to identify the correct definitions of scientific tests and types of data among a range of choices. Table 6 shows the proportion (%) of correct responses for the statistical concepts explored. Original data was grouped into “I’m not sure,” two incor- rect choices, and one correct choice. The data was dichotomized for analysis into correct and incorrect. Students were asked to use the “I’m not sure” choice instead of guessing. While most students perceived that the available statistical evidence within an article was important to consider when selecting information for an assignment (see table 6), the intention did not necessarily transfer into behavior. Additionally, the findings indicate that students have a poor understanding of foundational statistical tests that are used to describe the strength of scientific evidence. This is of concern when students value making judgments about statisti- cal constructs such as P values, confidence intervals, odds ratios, limitations of studies, and levels of evidence when selecting information. These unexpected results may explain why a substantial proportion of students use inappropriate data such as online discussions, blogs, or podcasts for assignments that require interpretation and application of high-quality data. TABLE 6 Student Perceptions and Behaviors toward Using Statistical Information When Selecting Resources, and Their Understanding of Basic Statistical Concepts Concept Important InformationΩ (%) n = 74 Actively ConsiderΨ (%) n = 101 Able to Define Concept† (%) n = 76 P Value 66.7 28.7 19.5 Confidence Intervals 41.4 23.8 17.1 Odds Ratio 62.7* 18.6* 14.3 Limitations of Study Design 84.1* 50.6* — Level of Evidence 84.5 72.2 — Mean — — 81.6 Median — — 76.3 Number Needed to Treat — — 14.5 ΩPerception: proportion who agreed or strongly agreed with the statement “When you select information for a university assignment, it is important to assess the quality of the information based on the following, if provided” (Q20). ΨBehavior: proportion who reported they often or always considered statistical information when considering to use an article or information within assignments, if supplied (Q5–9). †The proportion of participants who correctly identified the definition of statistical outcomes form a range of choices (Q23–28). *Indicates that statistically significant changes were found between the years of study. However, these changes were too small to be practical differences. — Not explored. Developing a Framework to Improve Information and Digital Literacy 957 There was evidence that students throughout each year of the program did not understand the concept of levels of evidence. Three questions were developed to explore understanding of the National Health and Medical Research Council hierarchy of evidence47 used in the USC paramedic program. These questions asked participants to indicate the study with the higher level of evidence (LOE) from a choice of two in each question. Two-thirds (68.4%, n = 54) cor- rectly identified that systematic reviews of randomised trials had a higher LOE than expert consensus. One-third (37.7%, n = 29) correctly identified that a rigorous qualitative study is considered a higher level of evidence than a case series. Almost half (45.5%, n = 35) correctly identified that a randomized control trial is considered a higher LOE than a systematic review of descriptive or qualitative studies. No statistical evidence for change in responses being associated with year of study in the program is also a concern when students study science research methods in the second year of the program. Informing Curriculum Design Curriculum integration of IL is advocated in Australia48 and internationally,49 with an integra- tive approach to the development of IL skills described as the most effective way of provid- ing ILE.50 This approach involves critical alignment of institutional level policies, program level planning, and course outcomes and activities (see table 7). The pedagogical approach also needs to account for the learners’ previous experience of ILE and must: be inclusive, transformative, encourage reflection and active learning; be grounded and contextualized to disciplines; and allow student collaboration and creativity. Hence, to effectively design curricula and prepare students, faculty staff and librarians must recognize that students in different disciplines use different information-seeking be- haviors. For example, paramedicine students (among others) may prefer independent web browsing rather than using library databases to find information.51 By constructively align- ing the ILE curricula throughout a program, faculty staff can support learning and teaching activities that engage a diverse range of students and facilitate deeper learning. It allows students to build on previous learning, reflect on their weaknesses and strengths in content knowledge and information use. Curriculum and teaching methods must develop informa- tion literacy skills for health science students that include an ability to evaluate the evidence informing practice. The integration of ILE occurs horizontally and vertically across different components and years of study. TABLE 7 Integration of Information Literacy Level of Institution Details Institution Level IL can be integrated into faculty curriculum through faculty teaching and learning policies and guidelines. Program Mapping of staged development of IL competencies. Students need to be provided opportunities to interact with information on an ongoing basis: for example, across curricula from week 1 to week 12 horizontally and from entry into a program of study (year 1) to finishing their study (final year). Course (Unit of Study) Level IL can be integrated into a course curriculum by contextualizing it into the course learning outcomes, course assignments and assessments, and class, laboratory, and online activities. 958 College & Research Libraries September 2020 The results of the Delphi study lend support in acknowledging important themes to embed within the ILE strategy in the USC paramedic program. The results of the student survey demonstrate that IL must be better scaffolded to ensure learning outcomes are met, particularly in light of low self-efficacy regarding communication of scientific information and poor knowledge acquisition. In response to the project findings, the academic team at USC held a series of meetings to develop a framework to better integrate ILE within the program. Several mapping tools were developed to determine and document the changes required. Tables 8 and 9 show the template used to map the ILE learning outcomes across the program and provide an exemplar for standards 1 to 3. TABLE 8 Tool to Map Embedded ILE Information Literacy definitions Insert defined by CAUL <> Insert defined by university <> Insert defined by program <> Year 1 Standards Scaffolds Target courses* Year 2 Standards Scaffolds Target courses* Year 3 Standards Scaffolds Target courses* TABLE 9 Definitions of the ANZIL Standards (1 to 3) for Each Year Level and the Required Scaffolds ANZIL Standard Year Standard Scaffolds 1. Recognizes the need for information and determines the nature and extent of the information needed Year 1: With direction, the student interprets the nature and extent of information the task requires. Teacher-led deconstruction of task requirements. Year 2: To work with minimal direction to interpret the nature and extent of information the task requires. Student-led deconstruction of task requirements. Year 3: To work independently to interpret the nature and extent of information the task requires. Independent deconstruction of task requirements. 2. Finds needed information effectively and efficiently Year 1: With direction, find information required to answer a task using an appropriate search strategy. Teacher-led session to develop the skills necessary to use multiple strategies to find information. Year 2: To work with minimal direction to find the information the task requires. Student-led session to develop effective strategies to find information required. Year 3: Independently finds needed information using a structured approach. Independently uses appropriate search tools to efficiently find required information. Developing a Framework to Improve Information and Digital Literacy 959 Each of the ANZIL standards has several elements. To ensure that all elements were being taught and assessed, these were mapped to courses, assessments, and outcomes. Table 10 shows an exemplar of the form used to map the first-year curriculum against ANZIL Standard 1. TABLE 9 Definitions of the ANZIL Standards (1 to 3) for Each Year Level and the Required Scaffolds ANZIL Standard Year Standard Scaffolds 3. Critically evaluates information and the information- seeking process Year 1: With direction, evaluates the relevance of information. Teacher-led session on assessing the usefulness and relevance of the information. Year 2: Identifies themes within the information and constructs a logical argument. Student-led session on defining and applying criteria for evaluating the information. Year 3: Critically explores themes and critiques the evidence discovered. Student-led session on critical evaluation of information.   TABLE 10 Exemplar of the Tool Used to Map Each Element of the ANZIL Standard 1 to the First-Year Curriculum ILE Standards ILE Themes 101 102 103 Assessment Types Standard 1 Courses Essay Report Reflect Case Study Poster Type (Other) 1.1 Defines and articulates the information needed Explore information sources, identify key concepts and terms o o o PAR101 PAR102 PAR103 o o o o o o o o o o o o o o o 1.2 Understands the purpose, scope and appropriateness of a variety of information sources Organize and disseminate information; topic context is discipline bound; differentiates between a variety of information sources; identifies primary and secondary information sources o o o PAR101 PAR102 PAR103 o o o o o o o o o o o o o o o 1.3 Re-evaluates the nature and extent of the information needed Themes: Reviews the initial information to clarify, revise, or refine the question o o o PAR101 PAR102 PAR103 o o o o o o o o o o o o o o o 1.4 Uses diverse sources of information to inform decisions Uses a range of sources; uses information for decision making and problem solving o o o PAR101 PAR102 PAR103 o o o o o o o o o o o o o o o 960 College & Research Libraries September 2020 The teaching role of librarians has evolved from teaching students how to locate materi- als in the library to include complex and challenging tasks of how to analyze and evaluate information and think critically.52 As such, the teaching of critical thinking and information evaluation is a shared responsibility facilitated through critical alignment of faculty staff and librarians. The importance of curriculum alignment within any framework was made clear by Schaub et al.53 in finding that some of the IL terms least understood by students were those most frequently found in assignment instructions and covered by librarians during informa- tion literacy education sessions. As such, the key to the sustainability of improved ILE is to FIGURE 1 Suggested Framework to Quality Assure the Embedding Information Literacy Education into Bachelor of Paramedic Science Program at USC Note: The Working Party can be one or more staff. Developing a Framework to Improve Information and Digital Literacy 961 ensure that the responsibility for its oversight falls within an academic role and is supported by educational design and IL specialists such as librarians.54 Figure 1 shows the model ac- cepted by the USC paramedic faculty designed to be sustainable within USC and did not require additional resources. The model delineates responsibilities and provides structure and assistance to the teaching team. Conclusion Due to this research being founded in established IL and educational theory, it has wide ap- plicability to other allied health professional education and nonhealth disciplines, especially those regulated by accrediting bodies such as engineering, education, and others. The project aimed to develop a sustainable program of ILE within our paramedic degree program. The new mapping tools and quality structure within the program will help to facili- tate librarians’, academics’, and support staffs’ understanding of pedagogical interventions to improve IL skills of students. Key to integrating ILE across the program of study is that program coordinators, in conjunction with academic staff, librarians, and curriculum design- ers, identify appropriate learning outcomes and constructively align these to develop learning activities and assessment tasks that encourage the mastery of a suite of IL skills appropriate to the year level of the unit of study. The impact of an integrated and constructively aligned ILE is broader than individual student attainment. The immediate impact is that the student body will be the beneficiaries of improved ILE, as this will enhance their professional practice. To find, interpret, analyze, synthesize, and communicate information is an expected graduate attribute by the registering body. It underlies safe and effective practice and builds professional trust within the healthcare team and increases the broader community’s confidence in the delivery of community-based emergency healthcare. For universities, the broader impact will be an opportunity to imple- ment an evidence-based framework for the vertical and horizontal integration of ILE within similar programs. Evidence for the efficacy of the curriculum renewal will be observable via improved student responses to assignment tasks and captured by further research. We aim to retest the students in the coming years and are confident that the evidence-based approach to curricu- lum design will achieve the desired outcomes. 962 College & Research Libraries September 2020 APPENDIX A. NAPA Survey The Council of Australian University Librarians have developed information literacy standards (ANZIL) for use within higher education programs. The purpose of this study is to achieve a consensus view on the important aspects of information literacy education to embed within paramedic education. The ANZIL standards form the basis of the questions and learning outcomes presented in this survey. The next iteration of this survey will be shorter. The open-ended questions will be removed from further iterations of this survey along with any items in which consensus is reached. 1. So that we send you the next iteration of the survey, please enter your email address. Section One—Your thoughts about information literacy 2. Please describe how you define information literacy? 3. Please indicate your level of agreement with each of the following statements: Statement Generally disagree Neither agree or disagree Generally agree Students who develop information literacy skills can engage in independent learning through constructing new meaning, understanding, and knowledge. Students who develop information literacy skills derive satisfaction and personal fulfilment from using information wisely. Students who develop information literacy skills individually and collectively are able to search for and use information for decision making and problem solving to address personal, professional and societal issues; and Students who develop information literacy skills can demonstrate social responsibility through a commitment to lifelong learning and community participation. Please list any additional benefits. 4. Do students undertaking an entry to practice paramedic program need to develop infor- mation literacy skills? Yes / No a. If you responded yes, briefly describe how students acquire those skills. b. If you responded no, briefly describe why you believe this to be the case. 5. Does the program in which you teach have specific learning objectives for the develop- ment of information literacy skills? YES/NO. If possible, please add an example of the learning objectives and the year in which they are taught. Developing a Framework to Improve Information and Digital Literacy 963 In this section of the survey there are 66 examples of learning outcomes relating to the infor- mation literacy standards defined by the Australian and New Zealand information literacy (ANZIL) framework. We are interested in your opinion in two broad areas. The first concerns the value you place on particular learning outcomes related to the different skills that comprise information literacy. The next area concerns the point in an undergraduate entry to practice program of study at which the student should be able to demonstrate attainment of that outcome. Place a tick in the nongreyed-out boxes below to indicate your responses. You may also provide comments. Standards and learning outcomes Importance At which time point in an undergraduate entry to practice program of study should the student be able to demonstrate attainment of that outcome Comments N/A Low Moderate High Early Middle Late 1. Recognizes the need for information and determines the nature and extent of the information needed 1.1 Defines and articulates the information needed Explores general information sources to increase familiarity with the topic Defines or modifies the information needed to achieve a manageable focus Confers with others to identify a research topic or other information need Identifies key concepts and terms in order to formulate the focus questions 964 College & Research Libraries September 2020 Standards and learning outcomes Importance At which time point in an undergraduate entry to practice program of study should the student be able to demonstrate attainment of that outcome Comments N/A Low Moderate High Early Middle Late 1.2 Understands the purpose, scope, and appropriateness of a variety of information sources Understands how information is organized and disseminated Recognizes that the context of the topic changes among the disciplines Differentiates between, and values the variety of, potential sources of information Identifies the intended purpose and audience of potential resources (such as popular vs scholarly, current vs historical) Differentiates between primary and secondary information sources 1.3 Re-evaluates the nature and extent of the information needed Reviews the initial information to clarify, revise, or refine the question Articulates and uses criteria to make information decisions and choices 1.4 Uses diverse sources of information to inform decisions Understands that different sources will present different perspectives Uses a range of sources to understand the issues Uses information for decision making and problem solving Developing a Framework to Improve Information and Digital Literacy 965 Standards and learning outcomes Importance At which time point in an undergraduate entry to practice program of study should the student be able to demonstrate attainment of that outcome Comments N/A Low Moderate High Early Middle Late 2. Finds needed information effectively and efficiently 2.1 Selects the most appropriate methods or tools for finding information Investigates benefits and applicability of various investigative method or research question Investigates the scope, content, and organisation of information access tools Consults with librarians and other information professionals to help identify information access tools 2.2 Constructs and implements effective search strategies Develops a search plan appropriate to the or research question Identifies keywords, synonyms and related terms for the information needed Selects appropriate vocabulary or classification specific to the discipline or information access tools Constructs and implements a search strategy using appropriate commands Implements the search using investigative methodology appropriate to the discipline 2.3 Obtains information using appropriate methods 966 College & Research Libraries September 2020 Standards and learning outcomes Importance At which time point in an undergraduate entry to practice program of study should the student be able to demonstrate attainment of that outcome Comments N/A Low Moderate High Early Middle Late Uses various information access tools to retrieve information in a variety of formats Uses appropriate services to retrieve information needed e.g. document delivery, professional associations, institutional research offices, community resources, experts and practitioners Uses surveys, letters, interviews, and other forms of inquiry to retrieve primary information 2.4 Keeps up to date with information sources, information technologies, information access tools and investigative methods Maintains awareness of changes in information and communications technology Uses research update services Purposefully browses print and electronic sources 3. Critically evaluates information and the information seeking process 3.1 Assesses the usefulness and relevance of the information obtained Assesses the quantity, quality, and relevance of the search results to determine whether alternative information access tools or investigative methods should be utilised Developing a Framework to Improve Information and Digital Literacy 967 Standards and learning outcomes Importance At which time point in an undergraduate entry to practice program of study should the student be able to demonstrate attainment of that outcome Comments N/A Low Moderate High Early Middle Late Identifies gaps in the information retrieved and determines if the search strategy should be revised Repeats the search using the revised strategy as necessary 3.2 Defines and applies criteria for evaluating information Examines and compares information from various sources to evaluate reliability, validity, accuracy, authority, timeliness, and point of view or bias Analyses the structure and logic of supporting arguments or methods Recognises and questions prejudice, bias, deception, or manipulation Recognises the cultural, physical, or other context within which the information was created and understands the impact of context on interpreting the information Recognises and understands own biases and cultural context 3.3 Reflects on the information seeking process and revises search strategies as necessary Determines if the original information need has been satisfied or if additional information is needed 968 College & Research Libraries September 2020 Standards and learning outcomes Importance At which time point in an undergraduate entry to practice program of study should the student be able to demonstrate attainment of that outcome Comments N/A Low Moderate High Early Middle Late Reviews the search strategy Reviews information access tools used and expands to include others as needed Recognises that the information search process is evolutionary and nonlinear 4. Manages information collected or generated 4.1 Records information and its sources Organises the content in a manner that supports the purposes and format of the product e.g. outlines, drafts, poster Differentiates between the types of sources cited and understands the correct citation style for a wide range of resources Is able to access the correct citation style information Records all pertinent citation information for future reference and retrieval 4.2 Organises (orders/ classifies/stores) information Compiles references in the required bibliographic format Creates a system for organising and managing the information obtained e.g. EndNote, card files 5. Applies prior and new information to construct new concepts or create new understandings Developing a Framework to Improve Information and Digital Literacy 969 Standards and learning outcomes Importance At which time point in an undergraduate entry to practice program of study should the student be able to demonstrate attainment of that outcome Comments N/A Low Moderate High Early Middle Late 5.1 Compares and integrates new understandings with prior knowledge to determine the value added, contradictions, or other unique characteristics of the information Determines whether information satisfies the research or other information need and whether the information contradicts or verifies information used from other sources Recognises interrelationships between concepts and draws conclusions based upon information gathered Selects information that provides evidence for the topic and summarises the main ideas extracted from the information gathered Understands that information and knowledge in any discipline is in part a social construction and is subject to change as a result of ongoing dialogue and research Extends initial synthesis at a higher level of abstraction to construct new hypotheses 5.2 Communicates knowledge and new understandings effectively Chooses a communication medium and format that best supports the purposes of the product and the intended audience e.g. poster, report, essay 970 College & Research Libraries September 2020 Standards and learning outcomes Importance At which time point in an undergraduate entry to practice program of study should the student be able to demonstrate attainment of that outcome Comments N/A Low Moderate High Early Middle Late Uses a range of appropriate information technology applications in creating the product (e.g. report, essay, journal) Incorporates principles of design and communication appropriate to the environment Communicates clearly and in a style to support the purposes of the intended audience 6. Uses information with understanding and acknowledges cultural, ethical, economic, legal, and social issues surrounding the use of information 6.1 acknowledges cultural, ethical, and socioeconomic issues related to access to, and use of, information Identifies and can articulate issues related to privacy and security in both the print and electronic environments Understands and respects Indigenous and multicultural perspectives of using information 6.2 Recognises that information is underpinned by values and beliefs Identifies whether there are differing values that underpin new information or whether information has implications for personal values and beliefs Developing a Framework to Improve Information and Digital Literacy 971 Standards and learning outcomes Importance At which time point in an undergraduate entry to practice program of study should the student be able to demonstrate attainment of that outcome Comments N/A Low Moderate High Early Middle Late Applies reasoning to determine whether to incorporate or reject viewpoints encountered Maintains an internally coherent set of values informed by knowledge and experience 6.3 Conforms with conventions and etiquette related to access to, and use of, information Demonstrates an understanding of what constitutes plagiarism and correctly acknowledges the work and ideas of others Participates in online discussions following accepted practices e.g. Netiquette 6.4 Legally obtains, stores, and disseminates text, data, images, or sounds Understands fair dealing in respect of the acquisition and dissemination of educational and research materials Respects the access rights of all users and does not damage information resources Obtains, stores, and disseminates text, data, images, or sounds in a legal manner Demonstrates an understanding of intellectual property, copyright and fair use of copyrighted material 972 College & Research Libraries September 2020 Final Comments Please make any other comments that you feel are important and not captured earlier. Thank you for participating. Our research assistant will email you the next iteration of the survey. The items that have reached consensus and the initial open-ended questions will be removed. As such, the survey should be shorter. Developing a Framework to Improve Information and Digital Literacy 973 APPENDIX B. Student Survey Item Number, Stem, and Response Choices Question Item Stem Response Choice 1 I use diverse sources of information to develop my assignment. (Choose one response only.) None of the time, Some of the time, Often, Always 2 I check whether a concept is still current while developing an answer for an assignment. (Choose one response only.) None of the time, Some of the time, Often, Always 3 I use a systematic process to help me judge the relevance of the information. (Choose one response only.) None of the time, Some of the time, Often, Always 4 Which of the following sources do you use to answer assignments? (Choose all that apply.) government reports; news media; online collaborative encyclopedias, such as Wikipedia; online reference articles, such as those from WebMD or Mayo Clinic; peer-reviewed journal articles; textbooks; other (please list) Preamble to questions 5 to 9: Please indicate how often you evaluate the quality of an article using the following information. 5 P values (if present or if provided) None of the time, Some of the time, Often, Always 6 Confidence intervals (if present or if provided) None of the time, Some of the time, Often, Always 7 Odds ratio (if present or if provided) None of the time, Some of the time, Often, Always 8 Limitations None of the time, Some of the time, Often, Always 9 Levels of evidence None of the time, Some of the time, Often, Always 10 Briefly explain how you organize the information and sources gained through information searches for an assignment topic. Comment 11 Do you use bibliographic software such as Endnote to manage collected information? (Choose one response only.) None of the time, Some of the time, Often, Always 12 When I get a new idea, I work out how to explain it effectively. (Choose one response only.) None of the time, Some of the time, Often, Always 13 When I write an assignment, I set out the main ideas from each source and then combine them to generate new knowledge for an assignment. (Choose one response only.) None of the time, Some of the time, Often, Always 974 College & Research Libraries September 2020 Question Item Stem Response Choice 14 When I write an assignment, I rarely need to use multiple sources of information to reach a conclusion. (Choose one response only.) Strongly Disagree, Disagree, Neither, Agree, Strongly Agree 15 Please tell us what cultural sensitivity means to you in the context of writing an assignment. Comment 16 I use the USC style guide to ensure that I cite my sources accurately. (Choose one response only.) None of the time, Some of the time, Often, Always 17 I have read USC’s policies regarding plagiarism. (Choose one response only.) Yes, No, I’m not sure 18 As a reflective essay is about your personal beliefs, it does not include references to scholarly articles or textbooks. (Choose one response only.) True, False, I’m not sure 19 I can effectively communicate statistical research outcomes to others in writing. (Choose one response only.) Yes, No, I’m not sure Preamble to multiple items in question 20: When you select information for a university assignment, it is important to assess the quality of the information based on the following, if provided. 20 P values (if present or if provided) Strongly Disagree, Disagree, Neither, Agree, Strongly Agree Confidence intervals (if present or if provided) Strongly Disagree, Disagree, Neither, Agree, Strongly Agree Odds ratio (if present or if provided) Strongly Disagree, Disagree, Neither, Agree, Strongly Agree Levels of evidence Strongly Disagree, Disagree, Neither, Agree, Strongly Agree Limitations Strongly Disagree, Disagree, Neither, Agree, Strongly Agree Preamble to multiple items in question 21: In academic research, evidence-based guidelines are presented with a statement of the level of evidence that informed the development. Levels of evidence are ranked. In the following questions, please indicate which form of evidence ranks higher. If you are not sure, please check the “I’m not sure” option. 21 Levels Evidence from (A) a systematic reviews of randomised control trials; or (B) from expert consensus opinion. A is higher, B is higher, I’m not sure. Levels Evidence from (A) a qualitative study; or (B) from a case series. A is higher, B is higher, I’m not sure. Levels Evidence from (A) a randomized control trial; or (B) from systematic reviews of descriptive and qualitative studies. A is higher, B is higher, I’m not sure. Developing a Framework to Improve Information and Digital Literacy 975 Question Item Stem Response Choice 22 At the end of my program of study, I expect that I will have to keep maintaining and updating my knowledge. (Choose one response only.) Strongly Disagree, Disagree, Neither, Agree, Strongly Agree 23 Which of the following statements about an odds ratio (OR) is correct? (Choose one response only.) OR=1 Exposure does not affect odds of outcome, OR>1 Exposure is associated with lower odds of outcome, OR<1 Exposure is associated with higher odds of outcome, I am not confident which answer is correct. 24 Which of the following statements about a confidence interval is correct? (Choose one response only.) A confidence interval of 95% (x, y) is the probability that the true value falling between x and y is 0.95; The size of the confidence interval is not dependent on the sample size or the standard deviation of the study groups; A confidence interval of 95% (x-y) is the probability that the true value falling outside x and y is 0.95; I am not confident which answer is correct. 25 Which of the following statements about P values is correct? (Choose one response only.) A P-value is the probability that an observed difference is due to random chance when the null hypothesis is true; A P-value is the probability that an observed difference is due to random chance when the null hypothesis is false; I am not confident which answer is correct. 26 Which of the following statements is the correct definition for the “mean”? (Choose one response only.) A mean is the value that occurs most often in a data set; The mean is the “middle” value in the list of order ranked numbers in a data set; The average value of numbers in a data set; I am not confident which answer is correct. 27 Which of the following statements is the correct definition for “median”? (Choose one response only.) A median is the value that occurs most often in a data set; The median is the “middle” value in the list of order ranked numbers in a data set; The median is the average value of numbers in a data set; I am not confident which answer is correct. 28 Which of the following statements is the correct definition for “number needed to treat” (NNT)? (Choose one response only.) NNT is the number of patients that must be treated to prevent one additional adverse outcome; NNT is the number of patients that must be treated to eradicate a disease; NNT is the number of patients that must be treated to power a study; I am not confident which answer is correct. 976 College & Research Libraries September 2020 Question Item Stem Response Choice 29 What is your current year of study? If you are studying part-time, enter the full-time equivalent year. (Choose one response only.) Entry year, Middle, Final year 30 What is your gender? (Choose one response only.) Female, Male 31 What is your age in years? 17–21, 22–26, 27–31, 32–36, 37–41, 42–46, 47–51, 52–56, 57–61 32 What is your highest level of education? (Choose one response only.) Bachelor level qualification; Diploma level or certificate; Year 12 secondary education; Other (please state). Developing a Framework to Improve Information and Digital Literacy 977 APPENDIX C. Themes Generated from Learning Outcomes That Reached Consensus Grouped by Each Element of the Council of Australian University Librarians (CAUL) Information Literacy Standards 1. Recognizes the need for information and determines the nature and extent of the information needed 1.1 Defines and articulates the information needed Themes: Explores information sources, identifies key concepts and terms 1.2 Understands the purpose, scope, and appropriateness of a variety of information sources Themes: Organizes and disseminates information; topic context is discipline bound; differentiates between a variety of information sources; identifies primary and secondary information sources Re-evaluates the nature and extent of the information needed Themes: Reviews the initial information to clarify, revise, or refine the question Uses diverse sources of information to inform decisions Themes: Uses a range of sources; uses information for decision making and problem solving 2. Finds needed information effectively and efficiently 2.1 Selects the most appropriate methods or tools for finding information Themes: Evaluates the applicability of various investigative methods or research question; consults with librarians 2.2 Constructs and implements effective search strategies Themes: Develops constructs and implements a search strategy; identifies keywords, synonyms, and related terms 2.3 Obtains information using appropriate methods Themes: nil consensus 2.4 Keeps up to date with information sources, information technologies, information access tools, and investigative methods Theme: Uses research update services 3. Critically evaluates information and the information-seeking process 3.1 Assesses the usefulness and relevance of the information obtained Themes: nil 3.2 Defines and applies criteria for evaluating information Themes: Analyzes the logic of supporting arguments; recognizes and questions prejudice, bias, deception, or manipulation 3.3 Reflects on the information-seeking process and revises search strategies as necessary Theme: Utilizes a range of information access tools 4. Manages information collected or generated 4.3 Records information and its sources Themes: Differentiates between the types of sources cited; uses the correct citation style; keeps accurate citation records 4.4 Organizes (orders/classifies/stores) information Themes: Compiles references in the required bibliographic format 5. Applies prior and new information to construct new concepts or create new understandings 5.1 Compares and integrates new understandings with prior knowledge to determine the value added, contradictions, or other unique characteristics of the information Themes: Determines whether new information discovered contradicts or verifies information from other sources; recognizes relationships and draws conclusions based upon information gathered; selects information that summarizes the main ideas discovered; constructs new hypotheses 978 College & Research Libraries September 2020 Communicates knowledge and new understandings effectively Uses the appropriate software to create an artifact; incorporates principles of design and communication in a style to support the purposes of the intended audience 6. Uses information with understanding and acknowledges cultural, ethical, economic, legal, and social issues surrounding the use of information 6.1 Acknowledges cultural, ethical, and socioeconomic issues related to access to, and use of, information Themes: Respects indigenous and multicultural perspectives of using information Recognizes that information is underpinned by values and beliefs Themes: Applies reasoning; maintains internal consistency 6.3 Conforms with conventions and etiquette related to access to, and use of, information Themes: Understands plagiarism; correctly acknowledges ideas of others; participates appropriately online (netiquette) 6.4 Legally obtains, stores, and disseminates text, data, images, or sounds Themes: Understands fair dealing and intellectual property and fair use; respects the access rights of all users; obtains, stores, and disseminates text, data, images, or sounds in a legal manner Notes 1. Paramedicine Board of Australia, “Professional Standards,” Australian Health Practitioner Regula- tion Agency (AHPRA), available online at https://www.paramedicineboard.gov.au/Professional-standards/ Registration-standards.aspx [accessed 30 March 2019]; Paramedics Australasia, “Paramedic Registration,” www. paramedics.org.au [accessed 19 March 2019]; Gail Whiteford, Mahsood Shah, and Chenicheri Sid Nair, “Equity and Excellence Are Not Mutually Exclusive: A Discussion of Academic Standards in an Era of Widening Par- ticipation,” Quality Assurance in Education 21, no. 3 (2013): 299–310. 2. Department of Education and Training, “Higher Education Standards Framework (Threshold Standards) 2015” (Canberra: Australian Government, 2015), available online at https://www.teqsa.gov.au/hesf-domain-3 [accessed 6 December 2019]. 3. James Griffiths and Marilyn Harrington, “Increasing Participation in Tertiary Education,” in Parliamen- tary Library Briefing Book: Key Issues for the 45th Parliament (Canberra, Australia: Department of Parliamentary Services, 2016), 58. 4. Daniel Edwards and Eva van der Brugge, “Tracking Demand: An Early Audit of Australia’s New Student Demand Driven System,” Joining the Dots Research Briefings 2, no. 1 (2012): 13; Whiteford, Shah, and Nair, “Equity and Excellence Are Not Mutually Exclusive.” 5. Whiteford, Shah, and Nair, “Equity and Excellence Are Not Mutually Exclusive.” 6. Fiona Salisbury and Sharon Karasmanis, “Are They Ready? Exploring Student Information Literacy Skills in the Transition from Secondary to Tertiary Education,” Australian Academic and Research Libraries 42, no. 1 (2011): 43–58; Fiona Salisbury and Linda Sheridan, “Mapping the Journey: Developing an Information Literacy Strategy as Part of Curriculum Reform,” Journal of Librarianship and Information Science 43 (2011): 185–93. 7. Talal AlShammari, Paul A. Jennings, and Brett Williams, “Emergency Medical Services Core Compe- tencies: A Scoping Review,” Health Professions Education 4, no. 4 (2018): 245–58; Nigel Barr et al., “A Qualitative Exploration of Infection Prevention and Control Guidance for Australian Paramedics,” Australasian Journal of Paramedicine 6, no. 14 (2017): Article 3. 8. Emily Adams et al., “Clinical Reasoning of Junior Doctors in Emergency Medicine: A Grounded Theory Study,” Emergency Medicine Journal 34, no. 2 (2017): 70–75; Annemaree Lloyd, “Informing Practice: Information Experiences of Ambulance Officers in Training and On-Road Practice,” Journal of Documentation 65 no. 3 (2009): 396–419. 9. Paramedicine Board of Australia, “Professional Standards”; SkillsIQ, “Ambulance and Paramedic: Industry Reference Committee Industry Skills Forecast,” SkillsIQ Limited, available online at https://www.skillsiq.com. au/site/DefaultSite/filesystem/documents/Industry-Skills-Forecasts-June2017/2018%20Final%20ISFs/SIQ_Skills- Forecast_Ambulance_Paramedics_online.pdf [accessed 30 March 2019]. 10. Susan Kaplan Jacobs, Peri Rosenfeld, and Judith Haber, “Information Literacy as the Foundation for Evidence-Based Practice in Graduate Nursing Education: A Curriculum-Integrated Approach,” Journal of Profes- Developing a Framework to Improve Information and Digital Literacy 979 sional Nursing 19, no. 5 (2003): 320–28. 11. American Library Association, Framework for Information Literacy for Higher Education, American Library Association, available online at www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/infolit/Framework_ILHE. pdf [accessed 24 September 2019]. 12. Chris Armstrong et al., “CILIP Defines Information Literacy for the UK,” available online at http://eprints. rclis.org/7459/1/Article_Update_25102004.pdf [accessed 24 September 2019]; SCONUL Advisory Committee on Information Literacy, “Information Skills in Higher Education: A Sconul Position Paper,” available online at https://www.sconul.ac.uk/sites/default/files/documents/Seven_pillars2.pdf [accessed 24 September 2019]. 13. Australian and New Zealand Information Literacy Framework Principles, Standards and Practice, 2nd ed., Alan Bundy, ed. (Adelaide, South Australia: Library Publications University of South Australia, 2004); John W. Wil- lison, “When Academics Integrate Research Skill Development in the Curriculum,” Higher Education Research & Development 31, no. 6 (2012): 905–19. 14. American Library Association, Framework for Information Literacy for Higher Education. 15. Library and Information Association, “What Is Information Literacy?” available online at https://www. cilip.org.uk/page/informationliteracy [accessed 4 July 2018]. 16. John Biggs and Catherine Tang, Teaching for Quality Learning at University: What the Student Does, 4th ed. (Maidenhead, UK: McGraw-Hill, 2011), 95ff. 17. Christine Susan Bruce et al., “Information Literacy and Informed Learning: Conceptual Innovations for Il Research and Practice Futures,” Journal of Information Literacy IL, no. 1 (2017): 4. 18. John Biggs, “Aligning Teaching and Assessing to Course Objectives,” Teaching and Learning in Higher Education: New Trends and Innovations 2, no. April (2003): 13–17. 19. Allison Erlinger, “Outcomes Assessment in Undergraduate Information Literacy Instruction: A Systematic Review,” College & Research Libraries 79, no. 4 (2018): 442–79. 20. Sharon Weiner, “Information Literacy: A Neglected Core Competency,” available online at https:// er.educause.edu/articles/2010/3/information-literacy-a-neglected-core-competency [accessed 20 September 2019]. 21. Leanne Hamilton, “Embedding Information Literacy into the Prehospital Care Curriculum,” Journal of Emergency Primary Health Care 6, no. 1 (2008): Art 990291; Lloyd, “Informing Practice.” 22. Isaac P. Gilman, “Evidence-Based Information-Seeking Behaviors of Occupational Therapists: A Survey of Recent Graduates,” Journal of the Medical Library Association 99, no. 4 (2011): 307–10; Willison, “When Academics Integrate Research Skill Development in the Curriculum.” 23. Carlos Argüelles, “Curriculum-Integrated Information Literacy (CIIL) in a Community College Nursing Program: A Practical Model,” Community College Journal of Research and Practice 40, no. 11 (2016): 942–53; Lisa Sue Flood, Nanci Gasiewicz, and Terry Delpier, “Integrating Information Literacy across a BSN Curriculum,” Journal of Nursing Education 49, no. 2 (February 2010): 101–04; Susan Franzen and Colleen Bannon, “Merging Information Literacy and Evidence-Based Practice in an Undergraduate Health Sciences Curriculum Map,” Communications in Information Literacy 10, no. 2 (2016), https://doi.org/10.15760/comminfolit.2016.10.2.26; Marilyn Verhey, “Information Literacy in an Undergraduate Nursing Curriculum: Development, Implementation, and Evaluation,” Journal of Nursing Education 38, no. 6 (1999): 252–59. 24. Verhey, “Information Literacy in an Undergraduate Nursing Curriculum.” 25. Biggs and Tang, Teaching for Quality Learning at University. 26. Maryam Derakhshan and Diljit Singh, “Integration of Information Literacy into the Curriculum: A Meta- Synthesis,” Library Review 60, no. 3 (2011): 218–29, https://doi.org/10.1108/00242531111117272. 27. Clive Cochrane, “Embedding Information Literacy in an Undergraduate Management Degree: Lecturers’ and Students’ Perspectives,” Education for Information 24, no. 2/3 (2006): 97–123; Richard Dearden et al., “Aligning Information Literacy with the Faculty Teaching and Learning Agenda,” Australian Academic & Research Libraries 36, no. 4 (2005): 138–52. 28. Dearden, “Aligning Information Literacy with the Faculty Teaching and Learning Agenda”; Helena Ward and Julie Hockey, “Engaging the Learner: Embedding Information Literacy Skills into a Biotechnology Degree,” Biochemistry and Molecular Biology Education 35, no. 5 (2007): 374–80. 29. Xiaoli Wang, “Integrating Information Literacy into Higher Education Curricula and Information Literacy Curricular Integration Model” (doctoral thesis, Queensland University of Technology, 2010), available online at http://eprints.qut.edu.au/41747/. 30. Salisbury and Sheridan, “Mapping the Journey.” 31. University of Adelaide, “Research Skill Development for Curriculum Design and Assessment,” available online at https://www.adelaide.edu.au/rsd/ [accessed 16 March 2019]. 32. Pauline J. Ford, N. Foxlee, and Wendy Green, “Developing Information Literacy with First Year Oral Health Students,” European Journal of Dental Education 13, no. 1 (2009): 46–51; Hamilton, “Embedding Information 980 College & Research Libraries September 2020 Literacy into the Prehospital Care Curriculum.” 33. Wang, “Integrating Information Literacy.” 34. Bundy, Australian and New Zealand Information Literacy Framework Principles, Standards and Practice. 35. Chartered Institute of Library and Information Professionals (CILIP), “Defining Information Literacy for the UK,” available online at https://www.researchgate.net/publication/28804606_Defining_Information_Litera- cy_for_the_UK [accessed 10 July 2020]; American Library Association, “Presidential Committee on Information Literacy: Final Report” (1989), available online at www.ala.org/acrl/publications/whitepapers/progressreport [accessed 24 September 2019]; SCONUL Advisory Committee on Information Literacy, “Information Skills in Higher Education: A SCONUL Position Paper.” 36. Franzen and Bannon, “Merging Information Literacy and Evidence-Based Practice.” 37. John W. Creswell, Research Design: Qualitative, Quantitative and Mixed Methods Approaches, 3rd ed. (London, UK: Sage, 2009), 203–25. 38. Harrold Lindstone and Murray Turoff, The Delphi Method: Techniques and Applications (Boston, MA: Addison- Wesley Publishing Company, 1975), 3. 39. Chia-Chien Hsu and Brian A. Sandford, “The Delphi Technique: Making Sense of Consensus,” Practical Assessment, Research and Evaluation 12, no. 10 (2007): 1–8. 40. Ralph Catts, Information Skills Survey Technical Manual (Adelaide, South Australia: Council of Australian University Librarians, 2005). 41. Steve Stemler, “An Overview of Content Analysis,” Practical Assessment, Research and Evaluation 7, no. 17 (2001): 137–46. 42. Paramedicine Board of Australia, “Professional Capabilities for Registered Paramedics,” available online at https://www.paramedicineboard.gov.au/Professional-standards/Professional-capabilities-for-registered- paramedics.aspx [accessed 18 April 2019]. 43. Barr et al., “A Qualitative Exploration of Infection Prevention and Control Guidance for Australian Para- medics”; Leo McCann et al., “Where Next for the Paramedic Profession? An Ethnography of Work Culture and Occupational Identity,” Emergency Medicine Journal 32, no. 5 (2015): e1–11. 44. Jacobs, Rosenfeld, and Haber, “Information Literacy as the Foundation for Evidence-Based Practice in Graduate Nursing Education.” 45. Salisbury and Sheridan, “Mapping the Journey.” 46. Merinda Kaye Hensley, “Citation Management Software: Features and Futures,” Reference & User Services Quarterly 50, no. 3 (2011): 204–08. 47. National Health and Medical Research Council, “NHMRC Levels of Evidence and Grades for Recom- mendations for Developers of Guidelines,” Australian Government, available online at www.nhmrc.gov. au/_files_nhmrc/file/guidelines/evidence_statement_form.pdf [accessed 22 February 2016]. 48. Alan Bundy, “Enabling the Knowledge Nation: What Australia Needs in the 21st Century,” Australian Library Journal 51, no. 2 (2002): 103–16; Salisbury and Sheridan, “Mapping the Journey”; Wang, “Integrating Information Literacy.” 49. American Library Association, Framework for Information Literacy for Higher Education. 50. Cochrane, “Embedding Information Literacy in an Undergraduate Management Degree”; Derakhshan and Singh, “Integration of Information Literacy into the Curriculum.” 51. Heather Brodie Perry, “Information Literacy in the Sciences: Faculty Perception of Undergraduate Student Skill,” College & Research Libraries 78, no. 7 (2017): 964–77. 52. Heidi Julien, Melissa Gross, and Don Latham, “Survey of Information Literacy Instructional Practices in US Academic Libraries,” College & Research Libraries.79, no. 2 (2018): 179. 53. Gayle Schaub et al., “The Language of Information Literacy: Do Students Understand?,” College & Re- search Libraries.78, no. 3 (2017): 283-96. 54. Xiaoli Wang, “Developing Peoples Information Capabilities: Fostering Information Literacy in Educational Workplace and Community Contexts,” Library and Information Science 8 (2013): 31-49.