822 Writing Abstracts for MLIS Research Proposals Using Worked Examples: An Innovative Approach to Teaching the Elements of Research Design Anita L. Ondrusek, Harold E. Thiele, and Changwoo Yang Anita L. Ondrusek is Associate Professor, and Harold E. Thiele and Changwoo Yang are Assistant Profes- sors, in the MLIS Program at Valdosta State University; e-mail: alondrus@valdosta.edu, hethiele@valdosta. edu, cyang@valdosta.edu. Ashley Rhys and Christie Baker, graduate students in the Marriage and Family Counseling program at Valdosta State University, served as raters for the abstracts in this study. ©2014 Anita L. Ondrusek, Harold E. Thiele, and Changwoo Yang, Attribution-NonCommercial (http://creative- commons.org/licenses/by-nc/3.0/) CC BY-NC The authors examined abstracts written by graduate students for their research proposals as a requirement for a course in research methods in a distance learning MLIS program. The students learned under three instructional conditions that involved varying levels of access to worked examples created from abstracts representing research in the LIS field. A one-way analysis of variance (ANOVA) detected significantly higher scores in areas related to fluency in describing the research design and the required elements of a research proposal in the groups with more exposure to worked examples, while the rhetorical skills necessary to compose a succinct abstract and to relate a proposal to implications in the field were not affected. central tenet of graduate education is the expectation that students will be able to generate new knowledge from the research related to the discipline in which any given graduate student has elected to pursue study. The development of the skills needed by the students to meet this expectation requires a curriculum including coursework that exposes students to the research in the field and, ultimately, prepares students to demonstrate that they can transform that research into an original thesis. To this end, the culminating activities in many graduate programs include the submission of a thesis or a capstone paper. To help students to succeed in meeting exit requirements such as these, most graduate schools provide formal instruction in research methods. The hope is that, beyond graduation, these newly minted professionals will enter their fields armed with the tools they need to conduct research when the opportunities arise. In the library and information science (LIS) discipline, the need for students and practicing librarians to acquire experience in research methods has resulted in recom- doi:10.5860/crl.75.6.822 crl13-526 Writing Abstracts for MLIS Research Proposals Using Worked Examples 823 mendations, and sometimes mandates, from organizations that set standards. For example, the Core Competences of Librarianship adopted by the American Library As- sociation Council provide educators in master’s programs of library and information science with guidelines on the basic knowledge expected from graduates receiving that degree. Among the sections of those guidelines is a declaration that students should understand the fundamentals of quantitative and qualitative research methods and the principles and methods used to assess the value of research.1 Likewise, in its docu- ment, The Research Imperative, the Medical Library Association makes a commitment to “work with academic programs to [ensure] that opportunities to develop quantitative and qualitative research knowledge and skills appear throughout the curriculum.”2 The Association of College and Research Libraries (ACRL) encourages the pursuit of research through a number of initiatives. In 2004, the ACRL Scholarly Communications Committee published a research agenda to encourage research on topics that would enrich knowledge about academic libraries.3 In 2010, ACRL commissioned a report entitled The Value of Academic Libraries, which concludes with a “Research Agenda.” That section poses a set of research questions linked to investigative methods librarians can use to gather evidence of their contribu- tions in helping the institution meet its goals.4 The ACRL Instruction Section’s Research and Scholarship Committee released its updated “Research Agenda for Library Instruc- tion and Information Literacy” in 2011.5 The committee identified four areas of research relevant to academic library instruction programs, and the agenda includes research questions under each of these areas to better define research centering upon instruc- tion. Implicit in all these invitations to do research is the assumption that students and librarians have the skills needed to embark upon research activities. Why the emphasis on an understanding of research design in MLIS education? The responses to this question may include that academic librarians are often expected to publish and that librarians, both in academic and public libraries, need a grounding in research methods to make strategic decisions based on empirical evidence rather than on intuition or past practices. An alternative explanation, equally as valid, is that librarians are “the ones on the front lines… help[ing] students formulate practical research ques- tions” [and operating under the assumption that] “an experienced reference librarian likely knows the field well enough to help students” sort through the many aspects of researching a topic.6 Luo conducted a survey to examine the effects of research methods courses from MLIS education on the work of LIS practitioners and found that 85 percent of the 555 respondents used research skills in their work.7 The top six on-the-job applica- tions, from most-used to least-used, included being able to critically read LIS research articles, to guide patrons in reference consultations in their research needs, to evaluate library programs and services, to assess patrons’ needs, to publish, and to write grants.8 An exercise commonly employed to evaluate how well MLIS students can apply the knowledge and skills necessary to understand research and conduct research is the development of a research proposal. The traditional approach to teaching the methods employed in preparing a research proposal generally entails learning from a textbook supplemented by materials created by the professor. Research shows that this method often produces anxiety in students who are attempting this exercise for the first time.9 For MLIS students who are not required to write a thesis, this approach often seems like a meaningless exercise. There are two possible aspects to teaching research methods that may mitigate the resistance to learning research methods and the angst that accompanies that learning. The first aspect is to relate research to the literature on topics familiar to MLIS students. The second aspect is to focus learning upon a discrete part of writing a research pro- posal. One element of a proposal that addresses the latter aspect is the writing of an 824 College & Research Libraries November 2014 abstract for a proposal. To add relevance to the writing exercise, the use of abstracts from research articles in the LIS literature as exemplars for modeling the writing process introduces meaning into the overall learning process. Abstracts perform several major functions in the dissemination of scholarly, profes- sional, and technical literature—all of which provide a rationale for students to learn to both interpret and write abstracts. Abstracts assist scholars and professionals in keeping abreast of intellectual developments and progress in their respective disciplines and in retrieving specific information.10 They enable scholars and professionals to reduce the amount of time and effort they need to search the ever-increasing output of global intellectual developments and progress.11 With the advent of digital environments, searchers use abstracts to assess relevance quickly and effectively, as they navigate through search engines and gateways.12 In fact, with a growing use of pay-per-view services, the content of the abstract may influence the decision on whether to purchase the article.13 In an effort to provide MLIS students with the skills they need to interpret articles in their own discipline, to advise clients in the evaluation of scholarly literature, and to write abstracts for their own research if the opportunity arises, the authors present a scenario that proposes the use of an instructional method using examples of abstracts to achieve those learning goals. The authors further suggest that this method of in- struction is applicable to developing continuing education on research methods for LIS professionals in the field. In fact, Johnson and Lindsay indicate in their survey of public services librarians that libraries need to increase in-service training on publishing articles, which includes how to structure a research study.14 Teaching by examples is also a method that could be implemented in library instruction by teaching librarians. This fits with Standard Two of the ACRL Information Literacy Standards under the outcome that states that an information-literate student “identifies appropriate inves- tigative methods.”15 By extension, the librarian responsible for instruction that leads to that outcome must be conversant with a variety of research methods. Literature Review In today’s environment, students must internalize a series of skills and abilities that will enable them to handle a wide variety of information-based problems. Abstract- ing is not only a “professional writing activity based on analysis and synthesis,”16 it has a place in academic literacy, defined by Elmborg as “the ability to read, interpret, and produce information valued in academia.”17 The abstracting process encourages students to consciously examine and intentionally integrate skills and knowledge into developing their research proposals by requiring the selection of relevant information, recognition of the structure of the proposal, and reorganization of the selected infor- mation. Thus, an ability to abstract and synthesize is a high-level skill that educators consider a “transcendent learning tool.”18 The instructional concern that arises is how to direct students explicitly in learning both the required elements and the rhetorical structure of abstracts. One approach to guiding students through a problem like this is to model that process for them using examples from their fields of study. In this process, students call upon analogical reasoning to capture a solution and store it mentally with all its parts integrated into a solution—assisting in schema acquisition.19 An instructor can enhance the examples further by denoting the position of each required element using labels or segmentation. This example-based method, referred to as a worked example, has proven effective consistently as an instructional intervention in the study of problem solving.20 A worked example, in its simplest form, is defined as a procedure for solving a problem with one correct answer, and it has long been associated with the teaching of Writing Abstracts for MLIS Research Proposals Using Worked Examples 825 solutions to problems in domains such as algebra, physics, and computer program- ming. Zhu and Simon substituted worked examples for lectures and other classroom activities in physics problem-solving among college students with promising results in knowledge acquisition.21 Owen and Sweller found that students learned solutions to algebra problems, made fewer errors, and required less time to solve test problems similar to those presented to them in worked examples.22 In follow-up studies, stu- dent performances demonstrated that the use of worked examples could substitute for extended instruction and practice in solving algebra problems.23 A review of the worked example literature concluded that worked examples “promote the type of flexible transfer that educators are seeking in their classroom[s].”24 The worked examples used in mathematics and science have been described as classical worked examples. They represent one content level and provide “a lean de- scription of some situation” solved by applying an algorithm.25 However, researchers have developed more complex worked examples to teach skills in other disciplines. Schworm and Renkl describe a double-content example that encompasses a learn- ing domain that obviates the structure of a problem and an exemplifying domain that supplies the subject matter for the example.26 They used double-content examples to teach argumentative skills (the learning domain) via compositions on two different top- ics (the exemplifying domains). In addition, the subjects in their experimental groups received prompts to insert self-explanations of the examples, a technique that Chi et al. discovered enhanced learning.27 Atkinson and Renkl combined self-explanation prompts with fading-out steps from their worked examples, resulting in increased transfer of knowledge among their participants.28 In a study that instructed college students on searching an online library catalog, Ondrusek compared a worke-example treatment to the use of a conceptual model. Both groups received face-to-face instruction, with the worked example presenting the explicit steps of a catalog search. The worked example group performed better on both a written knowledge test and in conducting actual searches than the students trained with the conceptual model.29 Based on the success of that study, the author continued to use worked examples for teaching college students to search online databases and noted distinctions between an example that uses a single diagram to illustrate a pro- cedure and the worked example.30 FIGURE 1 A Single-Diagram Example Illustrating Term Entry for an Online Search 826 College & Research Libraries November 2014 An example using a single diagram states a rule or rules, then shows how to apply those rules (see figure 1). A worked example states a problem, then segments the search into discrete steps that demonstrate a solution to that problem. For instance, a worked example for teaching new searchers to locate citations in a database when they have data such as author, subject, and publication is shown in figure 2. One caveat: research on the transfer of learning indicates that examples should replicate the conditions of learning in any given situation. In other words, the worked examples must look like the search forms used by students in the online collections to which their libraries subscribe. To continue the worked-example sequence, the search problems in figure 3 are par- allel to the original worked example in figure 2 in both structure and solutions. These two search problems also illustrate two instructional techniques that fit well when presenting students with a sequence of worked examples: fading and self-prompting. Renkl, Atkinson, and Merrill showed how to fade completed worked examples by successively eliminating steps leading to the solution.31 The Albert Bandura search (see figure 3, left) uses fading by removing the specific instructions for selecting the correct field qualifiers for a search, and the Ravitch search (see figure 3, right) expects the searcher to identify terms from the search problem independently and to enter those search terms with the correct corresponding field qualifiers—another example of fading. Self-explaining occurs when “the learner… infer[s] information that is miss- ing from… an example’s solution.”32 Step 7 in the Ravitch search, “Tell what you will do before submitting your search,” and the prompt that ends that search, “If you did FIGURE 2 A Worked Example Segmented into the Steps for Term Entry for an Online Search Writing Abstracts for MLIS Research Proposals Using Worked Examples 827 not get these two citations, explain how you will modify your search,” are instances that prompt self-explanations on the part of the learner. In the final stage of learning through worked examples, assessment of learning, students should be able to solve a problem given similar conditions without assistance. In the case of online search- ing, the student would be directed to find the needed references without a worked example for support. In essence, the worked example provides a vehicle for promoting knowledge and skill acquisition in many domains. Its viability as an instructional method for modeling abstract writing—a domain that has not been previously investigated—shows promise. Evolution of the Study: Problem, Purpose, and Questions In the master of library and information science (MLIS) program at the Valdosta State Uni- versity (VSU), a course in research methods (MLIS 7700) is one of the required core courses. The students learn in an online, distance learning setting, using commercial courseware (WebCT was the system in use at the time the study was conducted). The research methods course is rigorous, and students are encouraged to take it toward the end of their program of study. The content of the course covers both qualitative and quantitative research designs, and the culminating assessment is the preparation of a research proposal. The proposal opens with an abstract in which the student must describe the intent and type of research and identify the criteria essential to the proposed research design. In a 2011 pilot assessment of student performance on the research proposal, writing the abstract was one of the outcomes with low scores, indicating a need for improved instruction. Starting in the fall semester of 2011, the MLIS faculty standardized the instructional materials for preparing the research proposal to include worksheets on methods of data collection and analysis and a series of six worked examples that diagram the elements of abstracts from actual LIS research articles. The worked examples were paired into FIGURE 3 Two Worked Examples Demonstrating Fading and Self-Explanatory Prompts 828 College & Research Libraries November 2014 three sets, each set including abstracts from one qualitative study and one quantita- tive study. The worked examples were adjunct learning materials: that is to say, the instructor(s) used them discriminately to teach abstract writing. During the 2011–2012 academic year, three different faculty members taught the research methods course, using the worked examples of abstracts in these varying degrees: Section 1, Fall 2011: The six worked examples of abstracts made available on the teaching website, not specifically recommended by the instructor. Section 2, Spring 2012: The six worked examples of abstracts made available on the teaching website and referred to at the discretion of the students. Section 3, Spring 2012: The six worked examples of abstracts specifically used in an online lecture. Students first submitted drafts of the abstracts for their proposals to a discussion board. The instructor in Section 3 advised students to refer to the worked examples as they composed their abstracts. It was up to the students to make final revisions and place the abstracts into their research proposals. Along with three instructional conditions, the section sizes and demographics sup- ported the feasibility of conducting a quantitative study on the effects of exposure to the worked examples. The ratio of female-to-male students was 3.3:1, and the ratio of white-to-minority students was 8.6:1 (see table 1). These statistics match well to the demographics of other LIS programs on a national level. This situation created an opportunity to investigate the following research questions and to test the related research hypotheses (stated below). Research Questions Does the use of worked examples with graduate students learning to write an abstract for a research proposal affect the quality of the abstract? More specifically, this research is to see how different instructional conditions (in other words, distinctive levels of exposure to worked examples) affect the quality of the abstract in these two dimensions: completeness of description of the proposed research and effective writing. Two hypotheses were proposed for this study. The first hypothesis related to a student’s ability to write an abstract with a complete description: H0 1. There is no significant difference in the mean scores on a measure of the ability of graduate students to write abstracts for research proposals with com- plete descriptions after exposure to worked examples of abstracts from expert research reports. TABLE 1 Demographics of Subjects of the Study Course Section Use of WEs Females Males White Minority Fall 2011 (n=40) Low 32 8 37 3 Spring 2012A (n=22) Medium 15 7 22 3 Spring 2012B (n=24) High 19 5 21 3 Totals (n=86) — 66 20 77 9 Writing Abstracts for MLIS Research Proposals Using Worked Examples 829 HA 1: Studying worked examples on abstracts from expert research reports leads to a significant increase in the ability of graduate students to write abstracts with complete descriptions of their proposed research. The second hypothesis related to a student’s ability to formulate an abstract with effective writing: H0 2. There is no significant difference in the mean scores on a measure of the ability of graduate students to write abstracts more effectively after exposure to worked examples of abstracts from expert research reports. HA 2: Studying worked examples on abstracts from expert research reports leads to a significant increase in the ability of graduate students to write abstracts for their proposed research more effectively. Research Design The design of this study addresses a major concern among investigators of worked examples—the fact that studies have been confined almost exclusively to laboratory experiments.33 In this study, investigators tested the effects of worked examples in an authentic distance learning setting with LIS graduate students (a domain new to worked-example research). Given the natural breakdown into three distinctive levels of exposure to the worked examples in a course otherwise conducted using the same instructional approach, the study proceeded using a three- group quasiexperimental design. The independent and dependent variables are described below. Independent Variable The independent variable was the exposure to the worked examples (WE) as adjunct instruction for writing abstracts. This variable comprises three treatments represent- ing the following: low exposure to worked examples; medium exposure to worked examples; and high exposure to worked examples. Dependent Variable The program assessment embedded in the research methods course (MLIS 7700) is the construction of a research proposal based on either a qualitative or quantitative design. Within that research proposal, the abstract’s quality provides a dependent variable for experimental study. Assumptions The key assumption made was that the students had roughly the same level of knowl- edge concerning writing a research proposal. Their backgrounds were mostly from the humanities and social sciences. The investigators conducted no pretest to determine their general level of prior knowledge. Developing the Worked Examples Since its recognition as an effective teaching format, guidelines governing when and how to use a worked example have evolved. Most proponents agree upon these basic guidelines for the design of worked examples: • Use worked examples for introducing new material to novice learners. • Choose well-defined problems as examples (situations that have a “correct” answer). • Start with typical problems (those that follow all the rules). • Start with a simple example (least amount of steps or parts). 830 College & Research Libraries November 2014 • Begin each worked example with a statement of the problem. • Show the complete solution. • If needed, elaborate upon what happens in each step. • Introduce the worked example(s) first, before any other instruction. • Include practice problems that: — follow the exact same procedure as the demonstration problem. — gradually fade assistance (leave more steps for the learner to complete). The practice of introducing the worked example(s) before students proceed to solv- ing problems on their own is imperative. Without examples to follow, students may resort to a trial-and-error approach to problem solving. With worked examples, the learners start with a worked example and are introduced successively to the elements of a particular problem.34 Having MLIS students write an abstract for a research proposal meets most, if not all, of the conditions of learning in which worked examples are effective. The material is new to these students, making them novices in this domain. Writing an abstract to a research report involves a systematic application of rules for classifying the criteria of a research design (for example, identifying data collection methods). There are enough well-defined research studies in the LIS literature to provide candidate material for abstract examples and companion practice examples. LIS research fits into the social sciences paradigm of research, and the abstracts for this type of research follow the format of an informative abstract (that is, one that summarizes documents pertaining to “experimental investigations, inquiries, or surveys… and state the purpose, methodology, results, and conclusions presented in the original document”35). MLIS faculty emphasize those criteria essential for judging the quality of an abstract as iterated in the APA Publication Manual, Sixth Edition.36 For the purposes of this study, these criteria were divided into two categories and subcategories: Complete description • Includes research problem or research question(s) • Names participants and characteristics • Identifies type of research approach • Expresses the research design clearly • Describes methods(s) for data collection • Describes methods(s) for data analysis • Identifies a strategic application or implications to the profession Effective writing • Nonevaluative language • Informative lead sentence • Succinct summary There is evidence that using worked examples generated by an expert in a domain produces better transfer of learning.37 Every attempt was made to select abstracts from the LIS literature that included all or most of the elements from the categories identified in the APA criteria, thus providing candidate material for the worked examples that was written in an expert manner. The worked examples developed to help students compose abstracts for their research proposals are illustrated in the following figures. They were paired in the following sequence: The first two worked examples provided the students with a fully diagrammed pair of abstracts—one representing a quantitative study (see figure 4) and a companion abstract representing a qualitative study (see figure 5). Following the rules for designing worked examples, the diagram showed the “complete solution” for identifying elements of research design. Writing Abstracts for MLIS Research Proposals Using Worked Examples 831 FIGURE 4 A Worked Example of an Exercise in Identifying Elements of Research Design from a Quantitative Study47 Instructions: There are nine elements of research design labeled in this abstract. Read the ab- stract,* study the labels. Is this the abstract of a study that is qualitative or quantitative? Why? *Abstract used with permission from the American Library Association. FIGURE 5 A Worked Example of an Exercise in Identifying Elements of Research Design from a Qualitative Study48 Instructions: There are nine elements of research design labeled in this abstract. Read the ab- stract,* study the labels. Is this the abstract of a study that is qualitative or quantitative? Why? *Abstract used with permission from Elsevier. 832 College & Research Libraries November 2014 FIGURE 6 A Practice Exercise in Identifying Elements of Research Design from a Quantitative Study49 Instructions: Select labels from the elements provided. Is this the abstract* of a study that is qualitative or quantitative? Why? *Abstract used with permission from Haworth Press. FIGURE 7 A Practice Exercise in Identifying Elements of Research Design from a Qualitative Study50 Instructions: Select labels from the elements provided. Is this the abstract* of a study that is qualitative or quantitative? Why? *Abstract used with permission from John Wiley & Son, Inc. Writing Abstracts for MLIS Research Proposals Using Worked Examples 833 FIGURE 8 An Assessment in Identifying Elements of Research Design from a Quantitative Study51 *Abstract used with permission from the American Library Association. FIGURE 9 An Assessment in Identifying Elements of Research Design from a Qualitative Study52 *Reprinted with permission of the Johns Hopkins University Press. 834 College & Research Libraries November 2014 The second pair of worked examples provided the students with a partially dia- grammed pair of abstracts—one representing a quantitative study (see figure 6) and a companion abstract representing a qualitative study (see figure 7). In this stage of learning from worked examples, the students practice what they learned from the initial worked examples. The worked examples in figures 6 and 7 are characterized as mechanical practice opportunities in that the students can solve the problem by apply- ing rules in exactly the same manner as demonstrated in the initial worked examples.38 To provide more challenge, one abstract included a few elements of research design not presented in the initial worked examples. The final pair of abstracts presented the highest level of challenge in identifying ele- ments of research design. Scaffolding in the form of labels and prompts were removed. These final examples are characterized as independent practice opportunities.39 Again, abstracts represented quantitative research (see figure 8) and qualitative research (see figure 9). Students were instructed to underline and label the elements of research design in each abstract and to identify the design represented in each abstract as ei- ther qualitative or quantitative. Abstracts such as these could be assigned as a graded exercise or used as items on an examination. Methodology Data Collection Abstracts from the research proposals submitted by a total of 86 students in the three sections of the research methods course (MLIS 7700) in 2011–2012 were copied into electronic files. All names were removed and replaced by numerical codes. The ab- stracts representing all three treatments (high, medium, and low exposure to worked examples) were prepared for evaluation in a sequence to prevent a pattern effect. A checklist containing the criteria on which the abstracts were to be evaluated was in- serted under each abstract. Two graduate students (not in the MLIS program) trained to evaluate research proposal abstracts analyzed the abstracts during the summer of 2012. Both graduate students had performed well in a course in research methods for the social sciences in their major (Marriage and Family Counseling). The use of third parties to rate the abstracts was necessary because both investigators in this study had taught the sections of research methods from which the sample abstracts were collected, and this prior exposure would have introduced bias into the scoring process. The use of graduate students to evaluate documents is an accepted practice in research studies in which unbiased assessments of student work is needed.40 Two of the MLIS research methods course instructors trained the graduate students using abstracts selected from research proposals submitted prior to the 2011–2012 academic year. Both students attended a two-hour session on the elements of a well- constructed abstract for social sciences research articles, using the APA Publication Manual, sixth edition, as a guide. Two rounds of practice exercises followed. In the first round, both instructors and both graduate students used the checklist to evaluate eight sample abstracts independently. The inconsistencies between instructor and graduate student scores were ad- dressed in a follow-up training session. In a second round of practice, the two graduate students received six more sample abstracts. Agreement of their scores moved toward consistency, and the MLIS instructors clarified items on which the student evaluators disagreed. Raters could not reach consistency in their scores of the criterion dealing with clear expression of the research design. This criterion was dropped because it was inherent in five other criteria directly related to research design (includes research problem or research questions, names participants and Writing Abstracts for MLIS Research Proposals Using Worked Examples 835 characteristics, identifies type of research approach, and describes data collection methods and data analysis methods). Over a period of two weeks, the student evaluators rated each abstract on a 9-point scale, assigning one point for each criterion that they found in the abstract and zero points for those criteria that were missing. The investigators transferred points to score sheets (see Appendix A) and tabulated three scores for each abstract composed of an individual score from Rater A, an individual score from Rater B, and a final score. They calculated the final score by crediting a point only for those criteria to which both raters had assigned a point. The data analyst for the study transferred these scores into a statistical program. Data Analysis and Results An inter-rater reliability analysis using Cohen’s Kappa was performed to determine consistency between the two raters. There was a total of 87.7 percent agreement, which achieved a kappa = 0.84. The agreement on each criterion was also analyzed, and the percentages of individual scores for each criterion were calculated (see table 2). One-way analysis of variance (ANOVA) followed by post hoc comparisons using the Tukey’s HSD test was conducted to test the differences between group means. Complete Description As stated earlier, the criteria for judging abstract quality was composed of two cat- egories and subcategories: complete description and effective writing. With regard to complete description, it was hypothesized that there is no significant difference in the mean scores on a measure of the ability of graduate students to write abstracts for research proposals with complete descriptions after exposure to worked examples of abstracts from expert research reports. As is shown in table 3, the null hypothesis was rejected. The one-way ANOVA test results indicated that there was a significant effect of exposure to the worked example on the complete description at the p = .05 level for the three conditions [F (2, 83) = 24.61, p = .000, η2 = .59]. Further post hoc comparisons using the Tukey’s HSD were also conducted as a follow-up test to analyze intergroup differences. This test compared the low- and medium-exposure conditions and low- and high-exposure conditions. It also compared medium- and high-exposure conditions. The post hoc analysis revealed significant differences between low-exposure condi- TABLE 2 One-way ANOVA of Complete Description by Level of Exposure to WE Criteria 1 Criteria 2 Criteria 3 Criteria 4 Criteria 5 Criteria 6 Criteria 7 Criteria 8 Criteria 9 Percent Agreement 91.86% 89.53% 100% 93.03% 97.67% 94.18% 68.60% 79.06% 76.74% TABLE 3 One-way ANOVA of Complete Description by Level of Exposure to WE Source df SS MS F p Between groups 2 85.74 42.87 24.61 .000 Within groups 83 144.59 1.74 Total 85 230.337 836 College & Research Libraries November 2014 TABLE 4 One-way ANOVA of Effective Writing by Level of Exposure to WE Source df SS MS F p Between groups 2 1.00 .50 1.54 .218 Within groups 83 26.81 1.74 Total 85 27.81 TABLE 5 One-way ANOVA of Quality of Abstract by Level of Exposure to WE Source df SS MS F p Between groups 2 67.12 33.56 17.64 .000 Within groups 83 157.86 1.90 Total 85 224.98 tion (M = 3.33, 95% CI [2.91, 3.74] and medium-exposure condition (M = 5.09, 95% CI [4.50, 5.69]), and between low-exposure condition (M = 3.33, 95% CI [2.91, 3.74]) and high-exposure condition (M = 5.50, 95% CI [4.94, 6.06]) at the p = .05 level. However, the mean difference between the group of medium exposure and high exposure was not statistically significant. Effective Writing The second null hypothesis stated that there is no significant difference in the mean scores on a measure of the ability of graduate students to write abstracts more effectively after exposure to worked examples of abstracts from expert research reports. The one- way ANOVA results (F (2, 83) = 1.549, p = .218) presented in table 4 indicated that there are no significant differences in the mean scores between the three conditions in regard to the effective writing at the p = .05 level. Therefore, the second null hypothesis was not rejected. In other words, the results suggest that levels of exposure to the worked example did not significantly influence effective writing of the abstracts. Finally, the overarching research question of whether there was the effect of the ex- posure to the worked example (WE) on the abstract quality in low-, medium-, and high- exposure conditions was addressed by testing the one-way ANOVA. The results showed that there was a significant effect of the exposure to the WE on the quality of abstract at the p = .05 level for the three conditions [F (2, 83) = 17.65, p = .000, η2 = .30] (see table 5). The results of the ANOVA post hoc analysis also indicated that the mean score for the low exposure (M = 5.08, 95% CI [4.65, 5.50]) condition was significantly different from the medium (M = 6.64, 95% CI [5.96, 7.31]) and high (M = 7.0, 95% CI [6.44, 7.56]) exposure condition at the p = .05 level. However, there was no significant difference between medium exposure (M = 6.64, 95% CI [5.96, 7.31]) and high exposure (M = 7.0, 95% CI [6.44, 7.56]) conditions at the p = .05 level. In summary, the results suggested that, when students were moderately and highly exposed to the worked examples, they produced a higher-quality abstract than the students receiving little or no exposure to the worked examples. However, there was no significant difference between medium- exposure and high-exposure conditions, Writing Abstracts for MLIS Research Proposals Using Worked Examples 837 TABLE 6 Mean Score of Abstract (Standard Deviations in Parentheses) Level of exposure to WE Low Medium High Complete Description 3.33 (1.3) 5.09 (1.34) 5.50 (1.31) Effective Writing 1.95 (.45) 2.05 (.72) 2.21(.12) Overall Quality 5.08 (1.3) 6.64 (1.52) 7.00 (1.32) Note: Maximum possible score is 6 for completed description, 3 for effective writing, and 9 for overall quality. although the mean score for students receiving the high-exposure condition was slightly higher than the one for medium-exposure condition (see table 6). Discussion Overall, the results of the study suggested that exposure to worked examples of ab- stracts created from research articles on LIS research will improve learning even in a distance learning situation with the associated “noise” factors (potential distractions) and without face-to-face contact.41 The specific research questions posed in this study that focused on whether the use of worked examples with graduate students learning to write an abstract for a research proposal affected the quality of the abstract in two dimensions, completeness of description of the proposal and effective writing, were also answered. The hypothesis HA 1, that studying worked examples on abstracts from expert research reports leads to a significant increase in the ability of graduate students to write abstracts with complete descriptions of their proposed research,was confirmed. The results suggest that even the limited levels of exposure to the worked examples were sufficient to make a difference in meeting the criteria established for a complete abstract. This finding suggests that improvement in complete descriptions in the ab- stract for a research proposal may be a skill that improves with short-term exposure to worked examples. The hypothesis HA 2, that studying worked examples on abstracts from expert research reports leads to a significant increase in the ability of graduate students to write abstracts for their proposed research using writing that is more effective, was not confirmed. The results suggest writing effectiveness is a separate learning skill—one that limited levels of exposure to the worked examples did not significantly influence. In fact, learning to write abstracts is an example of a double-content exercise. The process of structuring the abstract is the learning domain, and the inclusion of the surface features (the different descriptive segments of the abstract) is the exemplifying domain. Using typical worked examples, the students learn the descriptive elements more readily than the compositional elements, and composition, or rhetorical skills, improve through the confluence of worked examples along with other interventions.42 Improvement of those rhetorical skills appeared not to be amenable to significant change during a short exposure to worked examples. This is an important consideration when dealing with novice learners.43 There were a number of limitations inherent in this study. One limitation was the reliance upon a single learning event that restricts generalization of the findings to other learning events. Combining the results with the results from similar studies may 838 College & Research Libraries November 2014 increase generalizability.44 A second limitation is the distance learning setting of the study. Attendance for the online lectures in which the professor introduced worked examples (for the high-exposure group) or recommended worked examples for study (for the medium-exposure group) was high, but not mandated. The archives for both lectures were available, but the researchers did not document the number of students in either group who studied the worked examples at the time of instruction. However, the finding that students in the high- and medium-exposure groups performed better than those students in the low-exposure group suggests that students in the two former groups used the worked examples. Conclusion For students new to research methods, learning the elements of a research design and how to synthesize these elements into an abstract for a research proposal is a complex task. The results from this investigation suggest that exposure of MLIS students to worked examples of informative, expert abstracts from the LIS research literature boosted their fluency with the descriptive components, particularly the vernacular of research design and the elements required in an abstract of a research proposal. The rhetorical skills necessary to compose a succinct abstract and to relate a proposal to implications in the field need additional attention. As for the added benefits of using worked examples of abstracts in a course on research methods in an MLIS curriculum or in a continuing education setting for LIS practitioners, the authors posit that this instructional technique could be an effective means for teaching either audience to interpret research articles. In other words, studying abstracts that include the elements essential to a research design may also strengthen the student’s ability to evaluate abstracts—a skill that librarians call upon when advising clients on the selection of articles for scholarly endeavors. Using worked examples of abstracts from research articles to teach an audience to write abstracts is yet another application for this strategy. The research by Luo and the study by Johnson and Lindsay revealed that librarians, particularly academic librarians in tenure-track positions, are concerned about writing research articles for publication.45 A workshop activity such as presenting actual articles paired with worked examples of abstracts for those articles would provide an introduction to the structure and ele- ments of research articles. As a follow-up exercise, the audience could write their own abstracts for assigned research articles. For the instructional librarian responsible for teaching undergraduates to locate research articles, the worked example method, again, offers a viable solution. This method has proven effective to teach the principles and procedures involved in rule- based learning, and learning to search for articles in online databases fits this defini- tion. The worked-example approach is also a time saver because it can substitute for practice. In a one-shot information literacy session, worked examples can be distributed as handouts or posted to web pages for students to reference after class ends. The findings from this study also invite further research on using the worked example as a teaching tool in the field of library and information science for online learning. Many MLIS programs have moved into distance learning for delivering courses or entire curricula. In an online learning environment, worked examples fit well in that they provide a visual, self-explanatory type of scaffolding for novice learners. Worked example exercises also have the potential to be converted into interactive exercises using software that provides instant feedback.46 Finally, students who learn through worked examples are also gaining exposure to the worked-example approach as an instructional vehicle. Follow-up research on where and how to employ worked examples, the effects of worked examples presented Writing Abstracts for MLIS Research Proposals Using Worked Examples 839 Agree Scores Raters Criteria XXXX A B # Descriptor 1 Nonevaluative language 2 Informative lead sentence 3 Includes research problem or research question(s) 4 Identifies participants and characteristics 5 Identifies type of research approach (qualitative or quantitative) 6 Describes methods(s) for data collection 7 Describes methods(s) for data analysis 8 Identifies strategic application or implications to the profession 9 Succinct summary XXXX