Editorial Considering Registered Reports at C&RL Amy Riegelman* In January 2020, I presented at the Librarians Building Momentum for Reproducibility virtual conference.1 The theme of the presentation was preregistration and registered reports and their role in reproducibility of research results. The presentation was twofold in that it provided background information on these themes and then advocated for the adoption of a registered reports submission track in Library and Information Science journals. I asked attendees to notify me if they wanted to learn more and to join me in contacting LIS journals to advocate for this model. The first journal that we targeted was College & Research Libraries. We drafted a letter that was sent to editor Wendi Arant Kaspar who discussed the topic with the editorial board and ultimately asked me to write a guest editorial for C&RL.2 In this editorial, I will attempt to convey the value of registered reports, how it is consistent with C&RL priorities (e.g., promoting transparency in the research process), and how it could be implemented at C&RL logistically. Registered reports enable research methods to be peer-reviewed before data collection. In practice, this means that a study’s methods and analysis plan are submitted to a journal prior to beginning data collection. The registered report is then peer reviewed, and if accepted, the authors are given a provisional guarantee that the study will be published regardless of posi- tive, null, or negative findings. As explained by the Center for Open Science: This format is designed to reward best practices in adhering to the hypothetico-deductive model of the scientific method. It eliminates a variety of questionable research practices [QRPs], including low statistical power, selective reporting of results, and publication bias, while allowing complete flexibility to report serendipitous findings.3 When a registered report meets journal expectations, authors are given a conditional ac- ceptance (also known as in-principle acceptance) and assured that the journal will publish the completed study if they indeed adhere to the registered methods. Registered reports are considered one method for preventing questionable research prac- tices (QRPs), which are practices that “can spuriously increase the likelihood of finding evi- dence in support of a hypothesis.”4 Fidler and Wilcox describe QRPs as practices that “inflate the rate of false positives in the literature.”5 The high prevalence of QRPs is considered a cause of the replication crisis, and as explained by Dorothy Bishop, “more journals are adopting the * Amy Riegelman is Social Sciences Librarian at University Libraries, University of Minnesota, email: aspringe@ umn.edu. ©2021 Amy Riegelman, Attribution-NonCommercial (https://creativecommons.org/licenses/by-nc/4.0/) CC BY-NC. 2 mailto:aspringe@umn.edu mailto:aspringe@umn.edu https://creativecommons.org/licenses/by-nc/4.0/ Considering Registered Reports at C&RL 3 ‘registered report’ format, in which editors evaluate the experimental question and study design before results are collected—a strategy that thwarts publication bias, P-hacking and HARKing” which are explained in more detail below.6 Registered reports improve efforts for distinguishing between confirmatory and exploratory research. Four examples of QRPs are listed below with more context. Bias Against the Null/Publication Bias happens when studies produce null or negative findings, and instead of publishing, authors self-select to not submit this research to journals. This is also known as the ‘file-drawer problem’ or file-drawer effect in which researchers put studies with nonsignificant results in their file drawers.7 Likewise, publication bias could occur when/if journals choose not to publish null or negative findings. With bias against the null, there is a risk of only the potentially false “by chance” research being published. Hypothesizing After Results are Known (HARK) is when authors present a post hoc hypothesis as though it were an a priori hypothesis. p-hacking is performing multiple analyses in order to find patterns in the data that pres- ent as statistically significant (typically a p-value ≤ 0.05) even though the studies may not have been originally designed to look at those variable relationships. Additionally, p-hacking could include selective reporting of statistically significant observations. Underpowered Studies are studies conducted with low statistical power which may lead to erroneous conclusions, false positives, or false negatives. When conducting experiments in the social sciences and other disciplines, there are researcher degrees of freedom and flexibility in both conducting the studies and analyzing the results. For example, HARKing and/or p-hacking could be used to convey that a study produced statistically significant results consistent with study predictions when in reality, methods and analyses were unplanned, unpredicted and/or cherrypicked. Bias against the null could be allowing false positive studies to flourish in the scholarly literature, while un- derpowered studies could create a “crisis of confidence” in influencing false-positive findings as well as inflated effect sizes.8 Further, meta-analyses, an evidence synthesis method trend- ing in the social sciences, exposes that p-hacking and publication bias distort the cumulative evidence, and Friese and Frankenbach advise that journals, funders, and institutions prioritize the prevention of QRPs to ensure trustworthy evidence.9 Another concern with meta-analyses, when including studies hindered by QRPs, they have a “garbage in, garbage out” problem in which meta-analyses are vulnerable to p-hacking and publication bias and therefore the outcomes could be a distorted view of the evidence base. When a journal commits to helping to realign incentives and moves to adopt a registered reports submission track option, this does not eliminate the opportunity for exploratory research submissions. The exploratory research can co-exist with registered confirmatory research. Transparently differentiating between registered studies and unregistered studies helps make this distinction and could be appropriately interpreted by librarians and other practitioners working to make evidence-based decisions. Registered reports are not limited to quantitative research methods or original data collection. Guidance exists for ways in which research meth- ods and analysis plans could be registered for qualitative research as well as methods analyzing secondary data.10 Librarians who already support systematic reviews and meta-analyses likely already have experience with preregistered methods for secondary data analysis.11 C&RL has a reputation for being a premier venue for LIS scholarship, and with a commit- ment to a gold open access publishing model, scholars and practitioners have equitable access 4 College & Research Libraries January 2021 to the fulltext versions of manuscripts. Additionally, C&RL has recently made a commitment to authorship transparency in adopting portions of the CRediT taxonomy with a scheduled rollout for 2021.12 Adding a registered reports submission track would help ensure that C&RL maintains these high standards but with a heightened awareness toward research methods. As previously stated, registered reports are considered one avenue for thwarting publication bias, p-hacking, and hypothesizing after results are known. The existing C&RL guidance for reviewers already includes two points that speak to the ethos of a registered reports submis- sion track: 1) Is the method used appropriate to the subject? 2) Does the evidence presented support the hypothesis?13 In addition to helping prevent QRPs, C&RL’s TOP (Transparency and Openness Promo- tion) Factor would be positively affected. Under the TOP Factor heading of Publication Bias, level 3 represents the journal’s acceptance of registered reports as a submission option.14 An- nounced on May 5, 2020, Web of Science will be adding TOP Factors to their Master Journal List making the TOP Factor more discoverable for scholars seeking journals to submit their work.15 In terms of communicating a registered reports option to ACRL members and other potential C&RL authors, a new submission option could be clearly articulated in the author instructions as has been done with many of the 266 other journals already incorporating reg- istered reports.16 Helping matters, librarians who have long supported systematic reviews as a consultant or co-author are likely already familiar with the registered reports model since it is used by both Cochrane Reviews and Campbell Systematic Reviews. Additionally, in at least one C&RL article, authors noted that their study was preregistered—“Same question, differ- ent world: Replicating an Open Access Research Impact Study,” from Arendt et al. where the research question and study methods were preregistered in the Open Science Framework Registry.17 Had C&RL already adopted the registered reports submission track, this is one example of a study that could have been peer reviewed prior to data collection. Broadly, journal editors have expressed concerns over implementation and specifically, the technical procedures needed amidst antiquated journal submission websites. Luckily, there are resources available via the Center for Open Science to help C&RL proactively ensure the process is as non-disruptive as possible for editors, reviewers, and authors.18 In terms of changes to the peer review workflow, the Center for Open Science has outlined a process and uses the language Stage 1 and Stage 2. Stage 1 represents the intended hypothesis and methods and Stage 2 represents the post-study write-up. In anticipation for what the techni- cal implementation in Open Journal Systems (OJS)/Public Knowledge Project would entail, there are already existing models in which OJS journals have enabled registered reports. One example is Biolinguistics. The cited resource provides detailed logistical information regarding how submitted manuscripts are triaged for both Stage 1 and Stage 2.19 A summary is below. Stage 1: Study introduction, methods, anticipated timeline, data sharing intentions, analysis plan, and registration plan for an approved repository (e.g., Open Science Framework Registry). Reviewer role at this stage is to review: • research question importance • hypothesis/es rationale • rigor and feasibility of methods https://acrl.libguides.com/c.php?g=525633&p=5370418 Considering Registered Reports at C&RL 5 Stage 2: Study is complete, and the manuscript is ready for review. Reviewer role at this stage is to review: • whether the Stage 1 submitted rationale and hypothesis/es matches the Stage 2 • whether the data collection tested the proposed hypothesis/es • whether the Stage 1 submitted methods were precisely followed • if anything new is presented in either the methods or analysis plan that was not in- troduced in Stage 1. If yes, the authors need to justify this choice and explain that it was methodologically sound • whether the conclusions appropriately reflect the findings/results In conclusion, I, along with the original letter signers, hope that C&RL will seriously con- sider this call for a registered reports submission track. This change would help strengthen the literature published in C&RL and advance the practice of replicable and reproducible research in librarianship. Previous C&RL articles have focused on academic librarian sup- port of reproducible research, and this shift to permitting registered reports in an LIS journal would be aligned with this work as well as taking a step forward to look inwards at our own research practices.20 Chris Chambers, Professor of Cognitive Neuroscience, author of the Seven Deadly Sins of Psychology: A Manifesto for Reforming the Culture of Scientific Practice, and outspoken advocate for registered reports, highlighted ten reasons why journals should offer registered reports in an article in the journal Addiction.21 In this article, he did not mince words when advocat- ing for registered reports; he wrote, “The duty now falls on journals, and journal editors, to help provide solutions. Registered Reports is one such solution and there is no reason for any credible journal in a field suffering from QRPs not to offer the format at the earliest op- portunity.”22 Now we are asking C&RL to do the same. Notes 1. “Librarians Building Momentum for Reproducibility” accessed November 1, 2020, https://vickysteeves. gitlab.io/librarians-reproducibility/; Amy Riegelman. “A Primer on Preregistration (& why I think it should be a submission track in LIS journals)” Presentation at Librarians Building Momentum for Reproducibility, virtual, January 28, 2020. https://osf.io/w4dfh/. 2. Amy Riegelman. Letter Proposing a C&RL Registered Reports Option, June 11, 2020, http://hdl.handle. net/11299/217041. 3. “Registered Reports,” Center for Open Science, accessed November 1, 2020, https://www.cos.io/our-services/ registered-reports. 4. Leslie K. John, George Loewenstein, and Drazen Prelec. “Measuring the prevalence of question- able research practices with incentives for truth telling.” Psychological science 23, no. 5 (2012): 524, dx.doi. org/10.1177/0956797611430953. 5. Fiona Fidler & John Wilcox. Reproducibility of Scientific Results in Stanford Encyclopedia of Philosophy. Accessed November 1, 2020, https://plato.stanford.edu/entries/scientific-reproducibility/. 6. Dorothy Bishop. “Rein in the four horsemen of irreproducibility.” Nature 568, no. 7753 (2019). dx.doi. org/10.1038/d41586-019-01307-2. 7. Robert Rosenthal. “The file drawer problem and tolerance for null results.” Psychological bulletin 86, no. 3 (1979): 638. 8. Geoffrey Schweizer and Philip Furley. “Reproducible research in sport and exercise psychology: The role of sample sizes.” Psychology of Sport and Exercise 23 (2016): 114-122, doi.org/10.1016/j.psychsport.2015.11.005. 9. Malte Friese and Julius Frankenbach. “p-hacking and publication bias interact to distort meta-analytic effect size estimates.” Psychological Methods 24, no. 4 (2019), https://doi.org/10.1037/met0000246. https://vickysteeves.gitlab.io/librarians-reproducibility/ https://vickysteeves.gitlab.io/librarians-reproducibility/ https://osf.io/w4dfh/ http://hdl.handle.net/11299/217041 http://hdl.handle.net/11299/217041 https://www.cos.io/our-services/registered-reports https://www.cos.io/our-services/registered-reports http://dx.doi.org/10.1177/0956797611430953 http://dx.doi.org/10.1177/0956797611430953 https://plato.stanford.edu/entries/scientific-reproducibility/ http://dx.doi.org/10.1038/d41586-019-01307-2 http://dx.doi.org/10.1038/d41586-019-01307-2 https://doi.org/10.1016/j.psychsport.2015.11.005 https://psycnet.apa.org/doi/10.1037/met0000246 6 College & Research Libraries January 2021 10. Alan Jacobs. “Pre-registration and Results-Free Review in Observational and Qualitative Research.” In The Production of Knowledge: Enhancing Progress in Social Science edited by Colin Elman, John Gerring, & James Mahoney, Cambridge: Cambridge University Press, 2020, 221-264; Tamarinde L. Haven and Dr. Leonie Van Grootel. “Preregistering qualitative research.” Accountability in Research 26, no. 3 (2019): 229-244, dx.doi.org/10. 1080/08989621.2019.1580147; Moin Syed and M. Brent Donnellan. “Registered Reports with Developmental and Secondary Data: Some Brief Observations and Introduction to the Special Issue.” Emerging Adulthood 8, no. 4 (2020): 255-258, dx.doi.org/10.1177/2167696820938529. 11. “Managing expectations: What does Cochrane expect of authors, and what can authors expect of Co- chrane?” Cochrane Editorial and Publishing Policy Resource accessed November 1, 2020, https://documentation. cochrane.org/pages/viewpage.action?pageId=117381161. 12. Sarah Fitzgerald, John Budd, Penny Beile, and Wendi Kaspar. “Modeling Transparency in Roles: Mov- ing from Authorship to Contributorship.” College & Research Libraries 81, no. 7 (2020): 1056, dx.doi.org/10.5860/ crl.81.7.1056. 13. “College & Research Libraries Guide for Authors & Reviewers,” ACRL, accessed November 4, 2020, https:// acrl.libguides.com/c.php?g=525633&p=5370418. 14. “TOP Factor Rubric,” Open Science Framework, accessed November 1, 2020, https://osf.io/t2yu5/ 15. “TOP Factor to appear in Master Journal List,” Center for Open Science, accessed November 1, 2020, https:// www.cos.io/about/news/cos-and-the-web-of-science-collaborate-to-bring-top-factor-to-master-journal-list. 16. “Registered Reports,” Center for Open Science, accessed November 1, 2020, https://www.cos.io/our-services/ registered-reports. 17. Julie Arendt, Bettina Peacemaker, and Hillary Miller. “Same question, different world: replicating an open access research impact study.” College & Research Libraries 80, no. 3 (2019): 303. dx.doi.org/10.5860/crl.80.3.303; Hillary Miller, Julie Arendt, and Bettina Peacemaker. “Open Access Citation Analysis” February 19, 2019, https:// osf.io/2ae99/. 18. “Registered Reports,” Center for Open Science, accessed November 1, 2020, https://www.cos.io/our-ser- vices/registered-reports; “Registered Reports Implementation Checklist,” Open Science Framework, accessed November 1, 2020, https://osf.io/8mpji/; “Registered Reports Webinar,” Center for Open Science via YouTube. com, accessed November 1, 2020, https://youtu.be/7MQ5XB1WUp8; “Template Reviewer and Author Guidelines,” Open Science Framework, accessed November 1, 2020, https://osf.io/8mpji/; “Fact sheet for editors,” Open Science Framework, accessed November 1, 2020, https://osf.io/3wct2/. 19. “Info on Registered Reports, Biolinguistics, accessed November 1, 2020, https://www.biolinguistics.eu/ index.php/biolinguistics/registered-reports. 20. Franklin Sayre and Amy Riegelman. “The reproducibility crisis and academic libraries.” College & Research Libraries 79, no. 1 (2018): 2. https://doi.org/10.5860/crl.79.1.2; Franklin Sayre and Amy Riegelman. “Replicable Ser- vices for Reproducible Research: A Model for Academic Libraries.” College & Research Libraries 80, no. 2 (2019): 260. dx.doi.org/10.5860/crl.80.2.260. 21. Christopher D. Chambers. “Ten reasons why journals must review manuscripts before results are known.” Addiction 110, no. 1 (2015), dx.doi.org/10.1111/add.12728; Christopher D. Chambers. The seven deadly sins of psychol- ogy: A manifesto for reforming the culture of scientific practice. Princeton University Press, 2019. 22. Christopher D. Chambers. “Ten reasons why journals must review manuscripts before results are known.” Addiction 110, no. 1 (2015), dx.doi.org/10.1111/add.12728. http://dx.doi.org/10.1080/08989621.2019.1580147 http://dx.doi.org/10.1080/08989621.2019.1580147 http://dx.doi.org/10.1177/2167696820938529 https://documentation.cochrane.org/pages/viewpage.action?pageId=117381161 https://documentation.cochrane.org/pages/viewpage.action?pageId=117381161 http://dx.doi.org/10.5860/crl.81.7.1056 http://dx.doi.org/10.5860/crl.81.7.1056 https://acrl.libguides.com/c.php?g=525633&p=5370418 https://acrl.libguides.com/c.php?g=525633&p=5370418 https://osf.io/t2yu5/ https://www.cos.io/about/news/cos-and-the-web-of-science-collaborate-to-bring-top-factor-to-master-journal-list https://www.cos.io/about/news/cos-and-the-web-of-science-collaborate-to-bring-top-factor-to-master-journal-list https://www.cos.io/our-services/registered-reports https://www.cos.io/our-services/registered-reports http://dx.doi.org/10.5860/crl.80.3.303 https://osf.io/2ae99/ https://osf.io/2ae99/ https://www.cos.io/our-services/registered-reports https://www.cos.io/our-services/registered-reports https://osf.io/8mpji/ http://YouTube.com http://YouTube.com https://youtu.be/7MQ5XB1WUp8 https://osf.io/8mpji/ https://osf.io/3wct2/ https://www.biolinguistics.eu/index.php/biolinguistics/registered-reports https://www.biolinguistics.eu/index.php/biolinguistics/registered-reports https://doi.org/10.5860/crl.79.1.2 http://dx.doi.org/10.5860/crl.80.2.260 http://dx.doi.org/10.1111/add.12728 http://dx.doi.org/10.1111/add.12728