ACRL News Issue (B) of College & Research Libraries 808 / C&RL News ■ Septem ber 2001 On scholarly evaluation and scholarly communication Increasing the availability of quality w o rk by David E. Shulenburger Is the scholarly communication crisis largely a creature o f the faculty evalua­ tion system? D o academ ic departm ent heads, deans, and members o f promotion and tenure committees simply count the faculty members’ publications and award salary increases, promotion, and tenure by the numbers? I f w e reformed the faculty evaluation system, w ou ld the scholarly communication crisis disappear? One commonly encounters anecdotes that appear to support affirmative answers to these questions. Faculty sometimes boast o f publishing the “least publishable unit,” a reference to dividing significant work into several smaller pieces to derive the maxi­ mum number o f articles from it. Others describe mechanical systems they have established that, upon rejection o f a manu­ script by one journal, w ill automatically submit that manuscript to the journal next in the status pecking order, continuing through as many journals as needed until one finally agrees to publish the manu­ script. At least two significant efforts aimed at gaining control o f the scholarly communi­ cation crisis have identified the faculty evaluation system as part o f the problem. In 1997, th e P e w H ig h e r E d u ca tion Roundtable published a treatise entitled “To Publish and Perish,” which urged univer­ sities to “place greater emphasis on qual­ ity rather than quantity in the promotion and tenure process.” In March 2000, a gathering o f academ­ ics, administrators, and librarians drew up the “Tem pe Principles for Emerging Sys­ tems o f Scholarly Publishing,” which have since been endorsed by both the AAU and NASULGC membership. One o f the prin­ ciples states: “To assure quality and reduce proliferation o f publications, the evalua­ tion o f faculty should place a greater em­ phasis on quality o f publications and a reduced emphasis on quantity.” Thus both anecdote and study point to the faculty evaluation system’s role in gen­ erating published scholarship that adds little to the fund o f knowledge. H ow im­ portant is this problem? I have served on and chaired faculty evaluation committees at the school and university level for more than 20 years. Dur­ ing those years, I have review ed many resumes that list publications that are at best marginal when evaluated against the criterion o f generation o f new knowledge. A b o u t th e a u th o r David E. Shulenburger is provost at the University o f Kansas in Lawrence, e-mail: dshulenburger@ku.edu A n n C. SchafSCOMMUNIATIOCCHOLNARLY fner, e d ito r mailto:dshulenburger@ku.edu C&RL N e w s ■ Septem ber 2001 / 809 Editor's note A common theme in the debate about scholarly communication has b e en the need for faculty to publish in a large num­ b e r o f publications to receive tenure. “Publish or perish” is an accepted co n ­ cept in higher education everywhere. Are tenure com m ittees truly blind to the issues o f quality? How do they deter­ mine quality? Do w e have too many low- quality publications, or do they serve a purpose in the scholarly communication process? The answers to these questions are com plex and have b een debated by Why did the faculty m em ber write them? Why were they published? Mark Twain said that one should not criticize others on the grounds that one cannot stand perpendicular to himself. It is very difficult for an author to determine the ultimate worth o f his o r her research. No o n e sets ou t to do inconseq uential work, and having invested weeks, months, or years in a project, it is exp ecting too much o f human beings to judge their work to b e inconsequential. Thus the norm is to write up the work and submit it for peer review so that others make the judgment. But p eer reviewers have similar diffi­ culties. Referees are themselves research­ ers. As researchers they are entangled in the w eb o f knowledge and becom e easily fa sc in a te d by a new d e tail o r by the resubstantiation o f an old one. They look to see w hether the data used should be relied upon, w hether the work followed the methods required to produce valid s ci­ ence, w hether it appropriately built upon the literature, e tc., and then make a judg­ ment from the middle o f the same thicket as to w hether it should be published. R e f e r e e in g w e e d s o u t t h e b a d I have great respect for the refereeing p ro­ cess. While I am aware o f the growing criti­ cism o f this process, I have faith that it librarians, publishers, faculty, and admin­ istrators for many years. For this month’s colum n, w e have in­ vited David Schulenburger, provost o f the University o f Kansas, to share his views on this su bject. Schulenburger is well known for his efforts to help us under­ stand the econom ics o f scholarly commu­ nication and to reform the scholarly com ­ munication process. We hope that this column will help to spur discussion of these issues on your cam pus.— A n n C. Schaffner, an n sch @ rcn .com alm ost alw ays w eeds out bad s cien ce . However, I do not believe that the pro­ cess admits only research that makes a sig­ nificant addition to knowledge. Peer re­ viewers are simply too close to the process to b e exp e cte d to know w hat w ill be judged by future generations to represent significant additions to the discipline. Thus the refereeing process tends to weed out the bad but does not eliminate the insig­ nificant. But b ack to those ré sumé s. Based upon my many discussions with provosts across the nation about the evaluation process, I believe that evaluation com m ittees at the University o f Kansas are similar to those at most research-intensive institutions. In our process, volume o f publication alone carries no weight. Evaluation committees exam ine the perceived significance o f the faculty m em ber’s w ork and if, and only if, it is perceived to b e o f significance do they begin to measure the quantity o f the work. Quantity takes on importance once qual­ ity is e s ta b lis h e d . D o in g v e ry sm all amounts o f quality w ork simply is not suf­ ficient justification for the standard exp ec­ tation that 40 percent o f a faculty member’s time should b e devoted to research. The com m ittee’s judgment o f the ulti­ mate significance o f a faculty m em ber’s work is suspect for the same reason that About th e editor A n n C . Schaffner is a n M B A Candidate a t Sim m o n s Graduate Sch o o l o f M a n a g e m e n t e-mail: annsch@ rcn.com mailto:annsch@rcn.com mailto:annsch@rcn.com 810 / C&RL News ■ September 2001 The real d a m a g e d o n e b y the fa c u lty e v a lu a tio n process . . . is n ot b y re w a rd in g fa c u lty fo r q u a n tity o f pub lication ; it is . . . by b a sin g q u a lity ju d g m e n ts o n the r ig o r o f the peer rev iew process in jo u rn a ls w h e re th eir w o r k appears, a process w h ic h is perceived t o be stro n g e st in th e top -ra n ked journals. p eer reviewers’ evaluations are suspect: committee members simply don’t have the right perspective to make an infallible judg­ ment. The evidence used by evaluation committees comes from their own reading o f the work, their judgment o f the rigor of review given the work by the journal of publication, and, especially in promotion and tenure cases, the opinion of outside reviewers who evaluate the entire body of the faculty member’s work. The latter group is particularly impor­ tant as outside reviewers are chosen b e ­ c a u se they are e xp erts in the facu lty m ember’s field. Given the narrowness o f some fields, only by including external reviewers can real expertise be brought to the evaluation process. By reviewing the entire body of work from the viewpoint o f the discipline, outside reviewers are in a position to judge the cumulative impact o f the faculty member’s work. This evaluation process places essen­ tially zero weight on publication in so- called “backw ater” journals. Evaluation committees generally take for granted that work appearing in such outlets got there either becau se the author judged it to be o f little worth and sent it directly to the journal or because it failed to gain accep ­ tance in one o f the top journals in the field and by default landed in a lesser one. Sometimes such automatic dismissal is a mistake. Sometimes manuscripts that dis­ play extraordinarily significant new knowl­ edge are rejected by top journals because its ideas challenge the orthodox views. Thus a revolutionary idea like plate tec­ tonics reaches the field through lesser jour­ nals and ultimately— through the weight of published findings in low-level, peer-reviewed journals— finds its way over time into the top journals in the field. If faculty evaluation com­ mittees or peer reviewers were true judges of u ltim ate significance, such articles would command great respect at first reading rather than suffer automatic dismissal because o f the low esteem for the publications in which they originally appeared. The real damage done by the faculty evaluation process then is not by reward­ ing faculty for quantity o f publication; it is by rewarding faculty for quality o f publi­ cation and by basing quality judgments on the rigor o f the peer review process in jour­ nals where their work appears, a process which is perceived to be strongest in the top-ranked journals. Evidence that this is true is the lack o f uproar when a library cancels a subscription to a journal per­ ceived to be o f low quality. The lack of turmoil over such decisions confirms that the problem is the reinforcement o f de­ mand for top-quality journals, not the pro­ liferation o f journals of low quality. W h a t can b e d o n e ? What we must do is restore the public goods nature o f journals by reducing the ability o f journals to use the market power they possess to raise prices. There are many efforts now underway to accomplish this aim, and SPARC (Scholarly Publishing and Academic Resources Coalition) represents one such strategy. By sponsoring modestly priced new journals edited and refereed by top scholars, SPARC endeavors to ac­ celerate the supply of prestigious journals and thereby reduce the possibility o f fur­ ther price increases by existing top tier jour­ nals. By creating products like BIOONE, SPARC keeps in the public domain a large group o f journals in the biological sciences for which prices will not b e raised. Three years ago I proposed the creation of NEAR, the National Electronic Article Re­ pository. By making scholarly journal ar­ ticles available for free three months after publication, I surmised demand for the journals would becom e more price elastic. That is, the ability to raise prices would be limited severely by the fact that many purchasers would choose to wait a short time C&RL N ew s ■ Septem ber 2001 / 811 until articles were freely available rather than pay the higher subscription prices. While manuscript authors need no direct return in order to generate articles, publish­ ers do. By having journals retain the exclu­ sive right to an article for three months, the journals would maintain the ability to charge a smaller subscription price, but a subscrip­ tion price that would cover necessary costs. Thus the proposal aimed to keep alive the current refereed journal system. However, my proposal suffered from the lack o f a mecha­ nism to make it happen. Two subsequent developments have created such mechanisms. First, the National Institutes o f Health, under the leadership o f Harold Varmus, cre­ ated PubMed Central, a virtual location in which bio-medical journals could be securely archived. Second, a group o f scholars initiated the PublicLibraryofScience.org petition, which constitutes a pledge that its signers will avoid journals that do not agree to make their con­ tents publicly available six months after pub­ lication. By signing the petition, scientists agree not to subscribe, submit papers, edit or referee papers for journals unless those jour­ nals make articles available to the public af­ ter a lapse o f six months. Public Library o f Science is the conscious- ness-raising mechanism to encourage jour­ nals to move from a profit motive to a public goods orientation. Thus far, about 25,000 sci­ entists have signed the pledge. I am optimis­ tic that many more scientists will join them and this effort will b e effective. These initiatives may soon have an im­ pact on the ability o f journals to raise prices. In fact, I am optimistic that these initiatives will lower prices and reverse the decades of untrammeled inflation. Exploitation o f the economics o f electronic publication, while returning journals to their deserved public goods status, will permit an increased vol­ ume o f quality work to be published and acquired within the reach o f existing library budgets. Universities should not encourage quan­ tity o f publication over quality in faculty evalu­ ations. But the imperative is that quality schol­ arly work has the opportunity to be published in rigorously refereed journals and that it be readily and affordably available to all schol­ ars. ■ 812 / C&RL N e w s ■ S ep tem b e r 2001 L P u b l i c a t i o n s L i b r a r y I & I T n f o r m a A t io n T e c h n o lo g y A s s o c i a t i o n UsabEWNility As s e s s m e n t of Library- R elated Web Sites: M ethods and Case Studies Nicole Campbell, editor 2001 $25 ($23 LITA members) P ra c tic a l a p p ro a c h e s to a na ly z in g th e u s a b ility o f y o u r W e b s ite s . Getting the Most out of Web-based Surveys David Ward 2000 $20 ($18.00 LITA members) Create Web-based surveys and use data with common database and spread­ sheet programs. Getting Mileage Out of Metadata: Applications for the Library Jean Hudgins, Grace Agnew, and Elizabeth Brown 1999 $22 ($19.20 LITA members) State-of-the-art metadata cataloging and standards overview, includes practical steps from project planning to training to merging metadata cataloging into nor­ mal workflow. Digital Imaging of Photographs: A Practical Approach to Workflow Design and Project Management Lisa Macklin and Sarah Lockmiller 1999 $20 ($18 LITA members) A comprehensive approach to management of digital imaging in libraries. Other LITA publications and a printable order form can be found at www.lita.org/litapubs Fax orders to (312) 280-3257 or call 1-800-545-2433, press 1 and ext. 4269. http://www.lita.org/litapubs