College and Research Libraries ( Ranking of Journals in Library and Information Science: A Comparison of Perceptual and Citation-based Measures MaryT. Kim A citation analysis of core library and information science journals was conducted to identify factors associated with subjective rankings of a journal's value in promotion and tenure deci- sions. Prestige rankings from a 1982 survey of ARL directors and library school deans were correlated with nine citation measures: total citation count, impact factor, immediacy index, references per paper, Price's Index, self-citation rate, popularity factor, citation factor, and con- sumption factor, with and without controlling for journal orientation, age, circulation, and index coverage. Results indicate that deans and directors may differ in their weighting of schol- arliness and timeliness when rating journal value, especially when the practitioner-research orientation of the journal is considered. D n a 1982 survey of ARL direc- tors and deans of library schools, David F. Kohl and Charles H. Davis obtained subjective ratings of thirty-one core li- brary science journals in terms of their importance for promotion and tenure decisions. 1 Using a five-point scale, di- rectors and deans rated those journals with which they were familiar. Kohl and Davis then used these ratings to rank or- der the journal set into a perceived hier- archy of journal prestige. Critics of sub- jective journal rankings have argued that such rankings are "artificially pre- cise indicators"2 which may, in fact, not be significantly different from each other. 3 Whether these rankings reflect an actual hierarchy of journal impor- tance or whether they merely group journals into clusters of high and low prestige, members of the-academic com- munity do use them to identify top li- brary and information science (LIS) jour- nals. For example, in a statistical profile of College & Research Libraries (C&RL) on its fiftieth anniversary, Paul Metz de- scribed C&RL as "one of the most widely respected journals in librarian- ship,'' noting that ARL directors and li- brary school deans ranked it first and third, respectively, in terms of "its value for tenure and promotion decisions at their institutions. '' 4 A ''self-fulfilling prophecy" occurs because, once publi- cized, these subjective rankings influ- ence assessment of the best outlet for a journal article and the subsequent weight assigned to it in personnel deci- • 5 s1ons. This author recalls a comment made at a discussion group on negotiating the tenure process held at an Associa- tion for Library and Information Sci- ence Education conference: "If it's not published in one of the top ten or twelve journals listed in the Kohl and Davis study, it doesn't count as much." If these journal rankings do, in fact, establish norms for the field, then Mary T. Kim is an Assistant Professor in the Columbus Program of School of Library Science, Kent State University, Columbus, Ohio 43210. 24 Kohl and Davis' recommendation for additional research appears valid: namely, to take a "close look at the ranking of the journals to determine whether there are objective factors that correlate with journal prestige,'' such factors to be derived by citation analy- sis.6 This paper presents the results of the recommended citation analysis. Its purposes are to determine whether journal characteristics do differentiate between varying levels of perceived LIS journal prestige and to determine if citation-based measures yield similar rankings within the LIS journal net- work. The goal is not to challenge the ranking assigned to specific journals, but to understand more fully the factors contributing to these rankings. CITATION MEASURES Actual journal use, citation-based measures, and subjective judgments are the three indicators typically used to rank journals. Each perhaps taps a dif- ferent aspect of journal "worth," and each is potentially distorted by common and unique sources of bias. R. Todorov and W. Glanzel recently reviewed the more familiar journal citation measures used for journal ranking. A brief sum- mary seems appropriate here. 7 The raw data for computing citation measures are the bibliographic references appear- ing in substantive papers (i.e., source items) within a set of journals for a speci- fied period of time. These references are interpreted as links between journals, journals giving references to and receiv- ing citations from each other. Once a cita- tion analyst decides which subset of journals and source items is appropriate for a given purpose, citation measures may be computed. Total citations are tallies of the citations received by a given journal. Because these are biased in favor of larger jour- nals with more source items, citation an- alysts have developed size-independent measures. The impact factor is computed as the total citations given to a journal for a specified time period divided by the number of source items published in that journal during that same time pe- Ranking of Journals 25 riod. Because the previous two years is the typical time period, the impact factor also measures how quickly authors cite work appearing in a journal. Although corrected for journal size, the impact fac- tor may still be biased in favor of older journals or journals with review articles. The immediacy index is computed as the total citations received from journals published in the same year divided by the number of source items available for citation in a journal that year. Factors such as journal circulation or publication delay may influence both the impact fac- tor and immediacy index. These three indicators measure journal usefulness as an aid in knowledge production. Whether cited positively or negatively, journal contents have stimulated or sup- ported some written endeavor. 8 Derek J. de Solla Price developed two alternative measures for scholarliness and the scientific "hardness" of a jour- nal. He believed high references per paper, the total number of references in a jour- nal divided by the number of source items in a journal, reflected the cumula- tive effect of knowledge building in a scholarly area. Price defined the normal range for scholarship as ten to twenty- two references per paper, with a typical value being fifteen. Price's Index, the pro- portion of the total references in a jour- nal to work published within the last five years, indicates the rapidity of develop- ment of a field and the degree to which journal references were made to the re- search front of the field. Price stated that higher index values indicated harder, scientific journals, while lower index values indicated journals dependent on the archival literature, not the research front. 9 More recently, Sydney J. Pierce has suggested that Price's Index reflects not only the degree to which older re- search has been integrated into the knowledge base of a field, but also the level of consensus as to what constitutes the knowledge base for a field-the lower the index, the lower the consen- sus.10 Obviously, citation norms of a field and the editorial policies of a jour- nal may influence Price's Index and ref- erences per paper. 26 College & Research Libraries The final set of citation measures to be considered deals with popularity and the position of the journal in the infor- mation flow of a field. V. I. Yanovsky proposed ratios of citations to citations and journals to journals. He believed these to be better indices of interaction among sets of journals than other size- independent ratios such as the impact factor. 11 The citation factor is the ratio of the citations received by a journal in a year to the references given by a journal in the same year. This is the inverse of the input/output ratio described by Louis V. Xhignesse and Charles E. Os- good and may be interpreted as the de- gree to which a journal feeds or stores in- formation in the journal network. 12 Journals with higher citation factors would feed information; journals with lower citation factors, store information. The popularity factor, the ratio of the number of journals citing a journal to the number of journals referenced by a jour- nal, has been described as a measure more appropriate for identifying less research-oriented journals. 13 The con- sumption factor, the product of the cita- tion factor and the popularity factor of a given journal, appears to tap dimen- sions other than journal quality. Jour- nals with higher consumption factors tend to be older, with a higher circula- tion rate, and a lower number of refer- ences per paper .14 The self-citation rate, the proportion of citations received by a journal which stem from the journal it- self, also indicates the degree of interac- tion with the journal network. Highly specialized journals in discipline sub- fields or more practitioner-oriented jour- nals with low referencing patterns would probably have higher self-citation rates. Because each of the citation indicators reviewed appears to measure a different dimension of journal significance, they have all been used as ranking measures in this study. Discipline versions, simi- lar to Graeme Hirst's discipline impact factor, have been computed for each of the indices involving total citation counts. 15 January 1991 CORRELATES OF SUBJECTIVE JOURNAL RANKINGS Investigators ranking journals by one method often correlate these rankings with those obtained by another method, the goal being a better understanding of what their rankings actually represent. In a detailed review of journal ranking methods, Alan Singleton discussed problems with each approach and exam- ined the relationship between meth- ods.16 Overall, he noted low correlations between subjective judgments of pres- tige and various citation rankings in the area of physics. In contrast, Bruce C. Bennion and Sunee Karshamroon found a moderately high multiple correlation (R = .74) between perceived usefulness of physics journals and a set of four cita- tion indicators (total source items, im- pact factor, immediacy index, and cita- tion factor). This correlation increased when journal circulation rate was con- sidered (R = .85). 17 Other researchers have reported rank-order correlations between perceived quality and impact factor for journals in the social sciences (rho = .45), and subfields of psychology (rho = .39-.56), sociology (rho = .63), and economics (rho = .87). 18 John C. Smart found a low correlation between impact factor and perceived quality for 678 educational journals (rho = .21). Re- analysis of this relationship within jour- nal types (i.e., core and allied) resulted in higher correlations (rho = .33 and rho = .52).19 Collectively, these studies demon- strate the need to study these ranking re- lationships within disciplines and to consider carefully the type of journals included in the discipline network. Ana- lysts of professional knowledge struc- tures support this latter recommenda- tion, arguing that the journal literature of a profession consists of two different components-research-oriented and practitioner-oriented journals which vary in their referencing patterns and behaviors. 20 Investigators have also reported that some citation indicators have higher cor- relations with perceived quality or use- fulness than others. Michael D. Gordon noted that total citations correlated more highly with perceived prestige (rho = .61) than did the size-independent im- pact factor (rho = .45) or immediacy in- dex (rho = .30). His study suggests that prestige may be more closely related to the size of a journal and dependency on it over time than the speed with which a journal is cited by later works. 21 Smart and C. F. Elton reported a low rank- order correlation between consumption factor and subjective judgments of qual- ity for psychology journals (rho = .13). They concluded that the consumption factor ranked journal quality by some di- mension other than the '' communica- tion of original research'' normally mea- sured by perceived quality rankings. 22 These studies suggest that citation mea- sures do measure different facets of jour- nal significance. Exploring the relation- ship between LIS prestige rankings and the set of citation measures would ex- pand the current understanding of sub- jective rankings of LIS journals. This review identifies the following sources of bias for citation-based and subjective journal rankings: discipline, journal orientation, age, size, and circu- lation. S. M. Dhawan, S. K. Phull, and S. P. Jain would add to this list coverage by indexing services. 23 Where possible, these extraneous factors have been con- sidered in the selection of journals or in the relationships analyzed. HYPOTHESES The author posed the following direc- tional research hypotheses to test com- mon conceptions about prestige rank- ings: Hypothesis Set 1: LIS journals with higher prestige rankings will typically (a) be older journals, (b) have higher cir- culation rates, (c) be covered by more in- dexing services, and (d) be more re- search oriented. Because of the inconsistencies in re- search relating different types of ranking methods, nondirectional research hy- potheses guided tests for the following relationships: Hypothesis Set 2: LIS journal prestige Ranking of Journals 27 rankings will be significantly correlated with journal rankings by (a) total disci- pline citations, (b) discipline impact fac- tor, (c) discipline immediacy index, (d) references per paper, (e) Price's Index, (f) discipline citation factor, (g) disci- pline popularity factor, (h) discipline consumption factor, and (i) discipline self-citation rate. To determine whether the hypothe- sized relationships between prestige and citation measures might be due to common factors such as journal age or circulation, the following set of non- directional hypotheses was tested: Hypothesis Set 3: LIS journal prestige rankings will be significantly correlated with journal rankings by (a) total disci- pline citations, (b) discipline impact fac- tor, (c) discipline immediacy index, (d) references per paper, (e) Price's Index, (f) discipline citation factor, (g) disci- pline popularity factor, (h) discipline consumption factor, and (i) discipline self-citation rate, after controlling for journal age, journal circulation, index coverage, and journal orientation. PROCEDURE Because only citations from journals in the LIS discipline were to be considered when computing discipline citation . measures, the first step in the study was to determine the appropriate journal set. As Patrick Doreian points out, the omis- sion of key journals may distort citation measures more than the inclusion of less significant journals which contribute lit- tle to or receive little from the journal network. 24 Consequently, this study ex- panded the original thirty-one journal set used by Kohl and Davis. English lan- guage journals were added if they were listed as both citing and cited LIS source journals in the Journal Citation Report OCR) of the Social Science Citation Index, were major ALA journals (i.e., not newsletters), and were consistently ref- erenced by journals in the original thirty-one journal set. This iterative pro- cess of addition and deletion resulted in the fifty-two journal network listed in appendix A. 25 Because Kohl and Davis conducted 28 College & Research Libraries their survey in fall1982, this study used citation data from the 1983 and 1984 SSCI Journal Citation Reports to corres- pond to the time frame of development, submission, and final publication for pa- pers written in fall 1982. Using citation data from two years also reduced journal idiosyncracies of subject focus for a given year. Sixteen network journals not covered by JCR required hand tallies of their 1983-84 references. Ten of these sixteen were in the original thirty-one journal network. Three of the original network titles were excluded because of low citations/references (Harvard Library Bulletin and International Journal of Law Libraries) or cessation of publication in 1983 (Library of Congress Quarterly Jour- nal). Discipline citation measures were computed for each of the twenty-eight remaining Kohl and Davis journals. Ap- pendix B contains definitions for these measures. The 1983 Ulrich's International January 1991 Periodicals Directory provided informa- tion on journal age, circulation, and cov- erage by indexing services. Using Price's minimum of ten references per paper as an indication of scholarly orientation, the author divided the journals into pop- ular, practical journals such as Library Journal, Wilson Library Bulletin, and School Library Journal, and more research-oriented journals such as Li- brary Quarterly, College & Research Li- braries, Journal of the American Society for Information Science, and Library Trends. If a journal employed referees in its review process, it was classified as a more schol- arly journal. 26 RESULTS AND DISCUSSION The twenty-eight LIS journals were rank ordered by each of the discipline ci- tation measures as well as the three ''de- mographic" journal characteristics (age, circulation, and index coverage). Table 1 compares the top dozen journals identi- TABLE 1 TOP TWELVE JOURNALS IDENTIFIED BY EACH RANKING METHOD* Ranking Methods /r{i.stigi.s Citation-based Methods g~cogrr~~~~ Journal DTC DIF Dll RP PI DCF DPF DSCR DCSF AGE (RIP) CRL X X X X X X X X X X X X R LQ X X X X X X X X X X X R ]AL X X X X X X X X X X R LRTS X X X X X X X X X X R LibTr X X X X X X X R IT&L X X X X X R ]ASIS X X X X X X X X X X X X R L] X X X X X X X X X X X p Am Lib X X X X X X X X X X X X p RQ· X X X X X X X X X X X R SpLib X X X X X X X X X X R WLB X X X X X X X X X X p LISR X X X X R ]LH X X R ]EL X X X X X R DLQ X X X X X X R Online X X X X X X X p SLMQ X X X X R SL] X X X X X X p ]liM X X X X p LawLJ X X p IP&M~t X X R Micro X X p RSR X p CollMt R PubLi X p ILibRev R LibAc X p *With the exception of DCSF, ranking is in descending order . See appendix A for journal abbreviations . fied by these rankings to the top dozen identified by each of the perceived pres- tige rankings. Table 1 also lists the jour- nal orientation for each title (R or P). The clustering of x' s at the top of table 1 sug- gests that the ranking methods collec- tively do identify a set of top journals. For example, at least eight of the eleven ranking methods ranked College & Re- search Libraries, Library Quarterly, Library Resources & Technical Services, Journal of the American Society for Information Sci- ence, Library Journal, American Libraries, RQ, Special Libraries, and Wilson Library Bulletin among the top dozen journals. Hypothesis Set 1 To test the first set of hypotheses, Spearman rank order correlations were computed between the first" three jour- nal demographics and the two prestige rankings, first for the total journal set (N = 28) and then separately for the re- search (N = 17) and practitioner (N = 11) subsets. 27 An alpha level of .05 was used to test the first two sets of hypothe- ses. Because journal orientation is a di- chotomous variable, prestige rankings were first reduced to a dichotomous level using a median split. Phi coeffi- cients were then computed to analyze the prestige-orientation relationship. As table 2 indicates, the deans' prestige rankings were significantly correlated with journal orientation (phi= .56), con- firming the research hypothesis that the more research-oriented the journal, the higher its prestige ranking. The broader mix of research-practitioner journals in Ranking of Journals 29 the directors' top journals probably ac- counts for the lack of a significant rela- tionship between journal orientation and director prestige rankings. As table 2 further indicates, the data failed to support the hypothesized rela- tionship between circulation and direc- tors' prestige ranking for both the total journal set as well as the research and practitioner subsets. Similarly, no sig- nificant relationships occurred between age and directors' prestige rankings . This finding suggests that the directors' collective assessment of a journal's pres- tige was not based merely on its age, availability, or a wider readership. Table 2 reveals a different pattern for the deans' prestige rankings. Age was sig- nificantly correlated with prestige for the total journal set. This relationship became more apparent when analyzed for research and practitioner journal subsets (rho = .62 and rho = .84, respec- tively). A probable explanation is that older journals have established reputa- tions, whereas the newer journals may still be shifting for position. In contrast, the deans' ranking was significantly re- lated with circulation only within the practitioner set . One possible interpreta- tion of these findings might be that li- brary educators valued publication in more widely circulated practitioner jour- nals, such as Library Journal, American Li- braries, and School Library Journal, be- cause of the service dimension of such writings while publication in more research-oriented journals was valued regardless of the journal's circulation. TABLE2 RANK ORDER CORRELATIONS BETWEEN PRESTIGE RANKINGS AND JOURNAL DEMOGRAPHICS FOR THE TOTAL JOURNAL SET AND FOR THE RESEARCH/PRACTITIONER SUBSETS Journal Demographics Age Circulation Index coverage Journal orientation! • p :S .05, one-tail test t p :S .01, one-tail test Total ARL ~::!~~:ankin~~actitioner (N = 28) (N = 17) (N = 11) .28 .37 .28 .20 .39 .01 .60+ .55* .47 .21 t Phi coefficients are reported for orientation-prestige relationships Total (N = 28) .41* .18 .56+ .56+ LS Prestige Rankings Research Practitioner (N = 17) (N = 11) .62+ .84+ .25 .56* .50* .71+ 30 College & Research Libraries Finally, results reported in table 2 con- firm the hypothesized relationship be- tween index coverage and journal pres- tige, with one exception. Directors' collective prestige ranking was not sig- nificantly related with index coverage for practitioner journals. Also worth noting is the relatively higher correlation between the deans' ranking and the in- dex coverage of practitioner journals (rho = . 71) compared to that for research journals (rho = .50). These findings are consistent with an overall pattern dis- played by table 2: namely, that directors' rankings of research journals and deans' rankings of practitioner journals may more readily be accounted for by journal age, circulation or index coverage than the opposite pairing. To interpret this correctly, the author needed to understand what age, circula- tion, and index coverage represent: scholarliness, popularity, or both. Spearman correlations among these three factors confirmed the expected: older journals circulated more fre- quently and were covered by more in- dexing services. After defining scholarli- ness by the references per paper measure and popularity by the popular- ity and consumption measures, rank or- der correlations based on the total jour- nal set revealed no significant relationship between scholarliness and any of the three journal "demograph- ics." In contrast, significant relation- ships existed between age and the two popularity measures (rho = .41 and rho = .46) as well as index coverage and the two popularity measures (rho = .58 and rho = .59). Particularly interesting were the high correlations between index cov- erage and the popularity measures within the practitioner set (rho = . 93 and rho = . 92). It would appear that the jour- nal demographics represent popularity and consumption somewhat more than scholarliness. A plausible interpretation of table 2, therefore, might be that deans valued publication in practitioner jour- nals because of their popularity and con- sumption while something beyond this shaped their rankings of research jour- nals. In the case of the directors, it ap- January 1991 pears that rankings were generally formed independent of a journal's pop- ularity or consumption. If such a bias ex- isted for directors, it seems to have been directed towards research journals. Hypothesis Set 2 To test the second set of hypotheses, Spearman rank-order correlations were computed between the prestige- and citation-based rankings for the total journal set as well as two subsets. These are italicized for each measure in table 3. As hypothesized, both deans and di- rectors assigned higher rankings to those journals receiving more direct cita- tions. Total discipline citation counts were significantly correlated with pres- tige rankings, regardless of journal ori- entation. As table 3 indicates, the strength of this relationship seems more consistent across journal type for direc- tors than for deans. Values of rho ranged from .65 to .71 for directors and from .54 to .71 for deans. The highest value for the deans occurred for the practitioner subset. It is interesting to see what oc- curs when examining the relationship between the size-adjusted impact factor and prestige rankings. While the rela- tionship between prestige and discipline impact factor was similarly significant across journal types for directors, it ap- pears to be strongest among research journals. For the deans, the impact fac- tor was significantly correlated with prestige for the research journals (rho = .60). Once corrected for size and, by ex- tension, popularity, the relationship be- tween citation frequency (e.g., impact factor) and deans' prestige ranking dis- appeared for the practitioner journals. This appears consistent with earlier findings that total citation counts tend to be biased towards large journals. As demonstrated by table 3, findings did not support the hypothesized rela- tionship between a journal's immediacy index ranking and the deans' prestige ranking for any of the journal group- ings. The directors, however, did value more highly those journals with a higher immediacy index (rho = .41). This was especially true among practitioner jour- nals (rho = .63). Similarly, only for the directors did the relationship between prestige and a journal's Price's Index even approach significance (rho = .55, p = .081, practitioner set; rho = .45, p = .068, research set) . Together these find- ings indicate that there was a tendency for directors to value more highly those journals reporting recent developments and being quickly cited by current writ- ers in the field. The overall absence of a significant re- lationship between Price's Index and the prestige rankings is not surprising given Ranking of Journals 31 the bimodal distribution of citation age reported for the LIS field. Susan Bonzi' s study of LIS citing behavior found mate- There was a tendency for directors to value more highly those journals report- ing recent developments and being quickly cited by current writers in the field. TABLE 3 RANK ORDER CORRELATIONS BETWEEN PRESTIGE AND DISCIPLINE CITATION MEASURES WITH AND WITHOUT CONTROLLING FOR JOURNAL AGE, CIRCULATION, INDEX COVERAGE, AND ORIENTATION ARL ~::!~~~:ankin~~actitioner LS Prestige Rankings Discipline Measures / Total Total Research Practitioner Controls (N = 28) (N = 17) (N = 11) (N =28) (N = 17) (N = 11) Total Citations .71+ .68+ . 65* .57+ .54* .71* Age .65+ .63+ Circulation .70+ .74+ .60+ Index .64+ .82+ Im.,eact Factor .70+ .78+ .67* .50+ .60* .46 ge .71+ .74+ .77+ .53+ .63+ Circulation .72+ .79+ .51+ Index .59+ .71+ Immediacy Index .41* .31 .63* .34 .38 .31 Age .82+ Circulation .72+ Index Price's Index .21 .45 .55 -.17 .14 .12 Age Circulation Index References per Paper .22 .02 .04 .48* .10 .02 Age .56+ Circulation .57+ Index Self-citation Rate -.53* -.46 -.28 -.52+ -.58* -.30 Age -.SOt -.52+ -.65+ Circulation -.48+ - .53+ -.62+ Index Citation Factor .19 .33 .23 .20 .56* .64* Age Circulation Index PoAularity Factor .35 .26 .41 .33 .51* .56 ge Circulation Index Consumption Factor .30 .33 .52 .30 .61+ . 74+ Age Circulation Index * p !S; .05, two-tail test t p !S; .01 , two-tail test 32 College & Research Libraries rials less than five years old and materi- als more than fifteen years old were cited most frequently. 28 The research-oriented journals, in the present study, had a lower average Price's Index than the practitioner-oriented journals (.47 vs. .59). Given the mix of practitioner- research journals in the top twelve pres- tige journals and the lower, rather than higher, index values for the research journals, this author believes that Price's Index may be an inappropriate measure of journal quality within a professional field. Price's Index may be an inappropriate measure of journal quality within a pro- fessional field. As expected, the deans' prestige rank- ing was significantly related to a jour- nal's average number of references per paper (rho = .48), while this was not the case for the directors' ranking. Because journal orientation was based on the number of references per paper, the low correlations within journal subgroups were also anticipated. Earlier in this paper, it was noted that journals with higher self-citation rates tend to be more highly specialized within sub-fields of a discipline. This seems to be the case for LIS journals as well. Journals such as Law Library Jour- nal, School Library Media Quarterly, Jour- nal of Library History, Philosophy & Com- parative Librarianship, and Reference Services Review had self-citation rates of .50 or higher. As reported in table 3, the prestige rankings of both directors and deans were inversely correlated with self-citation rates for all journals com- bined (rho = -.53 and rho = -.52, re- spectively). This finding also applied to the deans' assessment of research jour- nals. Journals with higher self-citation rates tended to receive fewer citations from the LIS journal network and ranked lower on the discipline con- sumption factor. Consequently, this au- January 1991 thor concludes that, generally speaking, both directors and deans valued publica- tion in journals which hold more central positions in the information flow of the journal network. The final set of relationships concerns journal popularity as measured by disci- pline popularity factor, citation factor, and consumption factor. For these three measures, the data failed to support the hypothesized relationships with the di- rectors' prestige rankings, consistent with the pattern revealed earlier in table 2. When the relationships were ana- lyzed separately for the research and practitioner subsets, the deans assigned significantly higher prestige rankings to those research and practitioner journals having higher citation factors (rho = .56 and rho = .64, respectively). Practi- tioner journals with higher citation fac- tors, such as School Library Journal, Amer- ican Libraries, Library Journal, and Wilson Library Bulletin, tend to feed information to the LIS journal network. The same ap- plies to research journals such as Journal of Education for Library and Information Sci- ence, Library Quarterly, College & Research Libraries, and the Journal of the American Society for Information Science. The rank order correlation between the deans' prestige ranking and the discipline pop- ularity factor was significant only for the research journals. In contrast, the data supported the hypothesized relation- ship between deans' prestige ranking and the consumption factor for both the research and practitioner sets (rho = .61 and rho = .74, respectively). Practi- tioner journals with higher discipline consumption factors, such as Wilson Li- brary Bulletin, American Libraries, and Li- brary Journal, did receive somewhat higher prestige rankings from the deans. This confirmed the already re- ported research which suggested that the consumption factor better identifies older journals with higher circulation rates and fewer references per paper. The consumption factor may function differently, however, when used to rank research journals. The research journals with higher consumption factors and higher deans' prestige rankings, such as Library Quarterly, Journal of the American Society for Information Science, and College & Research Libraries, were older journals but tended to have more references per article and represented a wider range of circulation rates. Hypothesis Set 3 In order to test the last set of hypothe- ses, regression analysis was used to re- move the effect of each journal's demo- graphic factor from the nine citation measures and the Kohl and Davis pres- tige ratings. The three variables-age, circulation, and index coverage-were normalized through a logarithmic trans- formation prior to the regression analy- sis. Spearman rank order correlations were then computed on the remaining residual scores. Because of the number of tests performed, alpha level was set at the .Ollevel. To emphasize more clearly any patterns which may exist, table 3 re- ports only significant relationships un- der each citation measure. The following discussion emphasizes these patterns; the reader interested in actual correla- tion values is referred to table 3. Given the previously determined finding that journal age, circulation, and index coverage correlate significantly with journal popularity and consump- tion, the results reported in table 3 are not surprising. When controlling indi- vidually for age, circulation, and index coverage, the relationship between total citations and deans' prestige rankings all but disappeared. This confirms the biased nature of total citations often at- tributed to age, size, and frequency of circulation. Controlling for circulation did not eliminate the previously re- ported relationship for the total journal set because of the low correlation be- tween circulation and deans' prestige ranking reported earlier in table 2. In contrast, the directors' prestige rankings continued to correspond to citation . count rankings, even after removing the effect of age, circulation, and index cov- erage. As table 3 demonstrates, this rela- tionship held for the total journal set but less clearly for the journal subsets. Removing the effect of age, circula- Ranking of Journals 33 tion, and index coverage did not alter the overall relationship between either prestige ranking or the impact factor for the total journal set. Because impact fac- tor already adjusts the size bias present in total citation counts, this was as ex- pected. When the relationships within subgroups were considered, four of the six correlations between prestige and impact factor continued to be significant for the research journals while only one was significant for the practitioner jour- nals. A different pattern occurred for the immediacy-prestige relationships when controlling for the three demographics. The relationship previously reported be- tween the directors' ranking and the im- mediacy index disappeared for the total journal set but appeared to be even stronger for the practitioner journals. Similarly, even after controlling for age, circulation, and index coverage, the deans' prestige ranking continued to be correlated with the scholarliness mea- sure of references per paper. J Both directors and deans valued publica- tion in journals which hold more central positions in the information flow of the journal network. Controlling for the three demograph- ics did not alter the nonsignificant rela- tionships previously reported for Price's Index. Also, the significant relationships between the deans' prestige rankings and the three popularity type measures-citation factor, popularity, and consumption-disappeared when controlling for the three factors. These findings are consistent with the fact that age, circulation, and index coverage were also shown to be related to popu- larity and consumption. Finally, table 3 indicates that control- ling for the demographics did not signif- icantly alter the inverse relationships be- tween prestige rankings and self-citation rate. The directors' ranking 34 College & Research Libraries continued to be significantly related to self-citation rate for the total journal set and the deans' ranking significantly re- lated for the total set as well as the re- search subset. CONCLUSIONS As stated earlier, this study attempted to gain a better understanding of subjec- tive journal rankings within the LIS field. The purpose was not to challenge the ranking of specific journals, nor to provide current rankings for LIS jour- nals. Rather, an attempt was made to an- swer the following questions about a specific set of journal rankings . Were these subjective rankings consistent with those derived by more objective, citation-based measures; were these rankings biased by perhaps less schol- arly factors such as journal age, circula- tion, or popularity; and if such relation- ships existed, were they consistent across journal types and journal raters? Discipline citation measures identified a core of top journals which overlapped well with the core listings of directors and deans. This study tested three sets of hypoth- eses. Some general patterns emerged. 1. Collectively, the discipline citation measures identified a core of top jour- nals which overlapped well with the core listings of the directors and deans for a similar time period. This consis- tency between the citing behavior of contributors and the LIS journal litera- ture suggests that the prestige rankings did represent norms for the LIS field at the time of the study. 2. In 1982, library school deans and ARL directors valued publication in journals which fed information to the network and had an impact on current writing in the field. Library school deans specifically valued publication in jour- nals with a research orientation as re- January 1991 fleeted by a higher number of references per paper and in older practitioner jour- nals with higher consumption values. ARL directors valued a mix of research- practitioner journals, but specifically valued research journals which tended to be cited, on the average, more heavily than other journals and practitioner journals which tended to be cited, on the average, more quickly than other jour- nals. 3. Library school deans and ARL di- rectors appeared to use different criteria in judging the value of a publication for tenure and promotion. Scholarliness, as defined by references per paper, and journal consumption were correlates for deans but not directors. Timeliness of in- formation emerged as a factor for direc- tors but not deans. Beyond these general patterns, study findings support the need to consider re- search and practitioner journals sepa- rately when analyzing knowledge struc- tures in a professional field. For example, the journal consumption fac- tor appeared to be somewhat more ap- propriate for identifying quality practitioner-oriented journals; the disci- pline impact factor for identifying qual- ity research journals; and the discipline immediacy index for identifying quality practitioner journals. In addition, this study has offered an approach for devel- oping discipline versions of citation measures for journals not currently cov- ered by available citation indexes and has presented data for an initial assess- ment of the construct validity of such measures. Discipline versions of total ci- tation count and popularity-consump- tion-citation factors, for the most part, functioned as anticipated. Each measure was related to uncorrected ratings of prestige, but failed to be related once the biases of age, circulation, or index cover- age had been removed. The size- adjusted discipline impact factor also functioned as expected for research jour- nals, being correlated with prestige even after the effects of journal demographics were removed. Given the small sample size in this study, especially within orientation Ranking of Journals 35 groups, additional research on an ex- panded journal network is needed to confirm these patterns, to determine the stability of the prestige rankings and de- tected relationships over time, and to provide further testing of the discipline citation measure approach. REFERENCES AND NOTES 1. David F. Kohl and Charles H. Davis, "Ratings of Journals by ARL Library Directors and Deans of Library and Information Science Schools," College & Research Libraries 46:40-47 Oan. 1985). 2. Ralph A. Weisheit and Robert M. Regoli, "Ranking Journals," Scholarly Publishing 15:323 Ouly 1984). 3. William E. McGrath, ''Ratings and Rankings: Multiple Comparisons of Mean Ratings,'' College & Research Libraries 48:169-73 (Mar. 1987). 4. Paul Metz, "A Statistical Profile of College & Research Libraries," College & Research Libraries 50:42 Oan. 1989). 5. Weisheit and Regoli, "Ranking Journals," p.317. 6. Kohl and Davis, "Ratings of Journals," p.47. 7. R. Todorov and W. Glanzel, "Journal Citation Measures: A Concise Review," Journal of Infor- mation Science 14:47-56 (1988). 8. Robert B. Archibald and David H. Finifter, "Biases in Citations-based Ranking of Journals," Scholarly Publishing 18:32 Oan. 1987). 9. Derek J. De Solla Price, Little Science, Big Science . .. and Beyond 18:132 (New York: Columbia University Pr., 1989), p.161-79. 10. Sydney J. Pierce, "Characteristics of Professional Knowledge Structures: Some Theoretical Implications of Citation Studies," Library and Information Science Research 9:157 Ouly/Sept. 1987). 11. V.I. Yanovsky, "Citation Analysis Significance of Scientific Journals," Scientometrics 3:223-33 (May 1981). 12. Louis V. Xhignesse and Charles E. Osgood, "Bibliographical Citation Characteristics of the Psychological Journal Network in 1950 and in 1960," American Psychologist 22:778-91 (1967). 13. Todorov and Glanzel, "Journal Citation Measures," p.52. 14. J. C. Smart and C. F. Elton, "Consumption Factor Scores of Psychology Journals: Scientome- tric Properties and Qualitative Implications," Scientometrics 4:349-60 (Sept. 1982). 15 . Graeme Hirst, "Discipline Impact Factors: A Method for Determining Core Journal Lists," Journal of the American Society for Information Science 29:171-72 Ouly 1978). 16. Alan Singleton, "Journal Ranking and Selection: A Review in Physics," Journal of Documenta- tion 32:258-89 (Dec. 1976). 17. Bruce C. Bennion and Sunee Karschamroon, "Multivariate Regression Models for Estimating Journal Usefulness in Physics," Journal of Documentation 40:217-27 (Sept. 1984). 18. Michael D. Gordon, "Citation Ranking versus Subjective Evaluation in the Determination of Journal Hierarchies in the Social Sciences,'' Journal of the American Society for Information Science 33:55-57 Oan. 1982); M. J. White andK. G. White, "Citation Analysis of Psychology Journals," American Psychologist 32:301-5 (1977); T. Roche and D. L. Smith, "Frequency of Citations as a Criterion for the Rankings of Departments, Journals, and Individuals," Sociological Inquiry 48:49-57 (1978); C. C. McDonough, "The Relative Quality of Economics Journals Revisited," Quarterly Review of Economics and Business 15:91-97 (1975). 19. John C. Smart, "Perceived Quality and Citation Rates of Education Journals," Research in Higher Education 19:175-82 (1983). 20. Pierce, "Characteristics," p.163. 21. Gordon, "Citation Ranking," p.57. 22. Smart and Elton, "Consumption Factor Scores," p .358. 23 . S. M. Dhawan, S. K. Phull, and S. P. Jain, "Selection of Scientific Journals: A Model," Journal of Documentation 36:24-41 (Mar. 1980). 36 College & Research Libraries J~nuary 1991 24. Patrick Doreian, "Measuring the Relative Standing of Disciplinary Journals," Information Proc- essing & Management 24:45-56 (1988). 25. Journals are referred to by the titles in use during the study period, 1983-84. 26 . Refereeing policy was defined by consensus using the following sources: Daniel O'Connor and Phyllis Van Orden, "Getting into Print," College & Research Libraries 39:389-96 (Sept. 1978); Norman Stevens and Nora Stevens, eds., Author's Guide to Journals in Library and Infor- mation Science (New York: Haworth Pr ., 1982); Mary Ann Bow an, comp ., Library and Information Science Journals and Serials (Westport, Conn. : Greenwood Pr., 1985); and John Budd, "Publica- tion in Library & Information Science: The State of the Literature," Library Journal113:125-31 (Sept. 1, 1988). 27. For reasons of consistency the author used the term prestige ranking throughout most of this study. The reader is reminded that Kohl and Davis collected ratings of journals which were then used to rank order the journals . 28. Susan Bonzi, "Characteristics of a Literature as Predictors of Relatedness Between Cited and Citing Works," Journal of the American Society for Information Science 33:213 (July 1982). APPENDIX A. DISCIPLINE JOURNALS ANALYZED Journals from Kohl and Davis Study American Libraries [AmLib] (P) Collection Management [CollMgt] (R) College & Research Libraries [CRL] (R) Drexel Library Quarterly [DLQ] (R) Information Processing & Management [IP- &MG11 (R) Information Technology & Libraries [IT&L] (R) International Library RevieuJ [ILibRev] (R) Journal of Academic Librarianship [JAL] (R) Journal of Education for Librarianship [JEL] (R) Journal of Information and Image Management [filM] (P) [formerly Journal of Micrographics; Micrographics Today] Journal of Library History, Philosophy & Compar- ative Librarianship [JLH] (R) Journal of the American Society for Information Science [JASIS] (R) Law Library Journal [LawLn (P) Library Acquisitions: Practice & Theory [LibAcq] (P) Library Journal [Ln (P) Library Quarterly [LQ] (R) Library & Information Science Research [LISR] (R) . Library Resources & Technical Services [LRTS] (R) Library Trends [LibTr] (R) Microform Review [MicroR] (P) Online [Online] (P) Public Libraries [PubLib] (P) Reference Services RevieuJ [RSR] (P) RQ(R) School Library Journal [SLn (P) School Library Media Quarterly [SLMQ] (R) Special Libraries [SpLib] (R) Wilson Library Bulletin [WLB] (P) Additional LIS Journals The American Archivist Aslib Proceedings Bulletin of the Medical Library Association Behavioral & Social Sciences Librarian Canadian Library Journal Collection Building Database Government Publications RevieuJ IFLA Journal International Classification International Forum on Information and Docu- mentation Journal of Documentation Journal of Information Science Journal of Librarianship Journal of Library Administration Libri Online RevieuJ Program Resource Sharing & Information Networks Scholarly Publishing Scientometrics The Serials Librarian Technical Services Quarterly Top of the NeuJs (R) indicates a research orientation; (P) indicates a popular, practitioner orientation. APPENDIX B. DEFINITIONS OF CITATION MEASURES Unless stated otherwise, citation counts mentioned in each of the following definitions are tal- lies of initial journal references appearing in the major source items in 1983-84 issues of the fifty- two LIS journal set: Ranking of Journals 37 1. Total discipline citations [TDC]: total citations received by a journal, 2. Discipline impact factor [DIF]: total citations received by source items appearing in 1981-1984 issues of a journal divided by the total number of source items appearing in 1981- 83 issues of that journal, 3. Discipline immediacy index [DII]: total citations received by 1983 and 1984 source items of a journal from discipline journals published in the same year, divided by the total number of source items appearing in 1983-84 issues of that journal, 4. References per paper [RP]: number of references appearing in 1983-84 source items of a jour- nal divided by the number of 1983-84 source items, 5. Price's Index [PI]: the proportion of total references in 1983-84 source items of a journal given to works published in the preceding five years (ie., 1979-1983 and 1980-1984), 6. Discipline citation factor [DCF]: total citations received by a journal, divided by the number of references given by that journal in 1983-84, 7. Discipline popularity factor [DPF]: total number of LIS journals citing a journal in 1983-84 divided by the total number of LIS journals referenced by a journal in 1983-84, 8. Discipline consumption factor [DCSF]: the product of the discipline citation factor and the discipline popularity factor, and 9. Discipline self-citation rate [DSCR] : the proportion of citations received which are self- citations . IN FORTHCOMING ISSUES OF COLLEGE & RESEARCH LIBRARIES Automation in College Libraries by Richard Hume Werking Designing Library Instruction for Undergraduates by Delia Neuman The Changing Domain of Subject Access by Prudence W. Dalrymple and Jennifer A. Younger Role Identity of Women Academic Librarians by Pamela J. Cravey Scientific Journal Prices by Kenneth E. Marks, Steven P. Nielsen, Craig H. Peterson, and Peter E. Wagner