lawson.p65


464  College & Research Libraries September 1999

Assessing Technology-based Projects 
for Promotion and/or Tenure in ARL 
Academic Libraries 

Karen G. Lawson and Nancy L. Pelzer 

Little is known about how technology-based projects (computer soft­
ware, articles in electronic journals, Internet-based materials, videotapes 
and audiotapes) are reviewed for promotion and/or tenure purposes in 
academic libraries. Reviewers might evaluate projects with traditional 
criteria or attempt to revise criteria to accommodate computer-related 
work. To address this issue in more detail, the authors conducted a study 
to assess how technology-based projects are evaluated in the promo­
tion and/or tenure process for academic librarians in Association of Re­
search Libraries. Survey results show that, while projects, particularly 
World Wide Web–based materials, are being evaluated in some ARL 
academic libraries, little has been developed as a core set of measures 
or assessments for promotion and/or tenure decisions. 

hanges in methods of publica­
tion and in the nature of re­
search and its resulting prod­
ucts are raising difficult 

questions in the arena of the scholarly 
reward system. Several years ago, as ad­
ministrators began to talk positively 
about technology and its benefits to the 
University community, efforts such as de­
veloping computer programs, writing 
reviews of software, and publishing in 
electronic journals were perceived by 
some in the academic world as coming 
into their own as legitimate forms of 
scholarship.1 By 1997, the winds on cam­
puses appeared to have shifted and the 
earlier enthusiasm of scholars was being 
tempered by skeptical departments and 
committees reviewing candidates for pro­
motion and/or tenure.2 The Modern Lan­
guage Association’s Guidelines for Evalu­

ating Computer-related Work in the Modern 
Languages specifies “recognition of con­
tributions by faculty members” as one of 
the guidelines for support of computer 
technology and recommends that “col­
leges and universities should develop a 
written policy concerning the evaluation 
of electronic publications in the tenure 
and promotion process so that faculty 
members can make their decision about 
appropriate ways to distribute their re­
search.”3 These guidelines, however, lack 
suggestions as to how specific kinds of 
computer-related activity should be 
evaluated within the traditional catego­
ries of professional practice, research, and 
service or how the traditional categories 
might best be revised to accommodate 
work with new technologies. 

While technology and promotion 
and/or tenure are widely discussed in 

Karen G. Lawson is an associate professor at Iowa State University; e-mail: klawson@iastate.edu. Nancy 
L.Pelzer is an associate professor at Iowa State University; e-mail: npelzer@iastate.edu. 

464 

mailto:npelzer@iastate.edu
mailto:klawson@iastate.edu


 

 

Assessing Technology-based Projects 465 

library literature as discrete topics, the 
two issues are rarely covered jointly. The 
use of technology on campuses and in 
libraries is hardly new, but there has been 
an obvious shift within the past decade. 
Technology-based projects are no longer 
controlled solely by an institutional “sys­
tems analyst” and are now conceived, 
produced and disseminated directly by 
faculty and staff. Concurrently, these 
products are included by faculty and 
staff in their promotional evaluation and 
tenure review files. In 1992, Edward 
Shreeves wrote that “Faculty often report 
that, while their interest in creating and 
working with electronic information 
sources is keen, their mentors urge them 
to produce traditional scholarship for 
publication in mainstream journals if 
they want to be eligible for tenure.”4 A 
librarian, who may or may not be a fac­
ulty member, can submit a technology-
based project for review for promotion 
or tenure evaluation. How that project is 
evaluated by the library administration, 
promotional reviewer, or tenure review 
committee may vary widely. The review­
ing body might evaluate the project with 
traditional criteria or may attempt to re­
vise evaluation criteria to accommodate 
computer-related work. 

While technology and promotion 
and/or tenure are widely discussed 
in library literature as discrete topics, 
the two issues are rarely covered 
jointly. 

In order to address this issue in more 
detail, the authors conducted a study to 
assess how technology-based projects are 
evaluated in the promotion and/or tenure 
process for librarians in Association of Re­
search Libraries (ARL) academic libraries. 
This group of institutions was chosen be­
cause their emphasis on research and the 
possible faculty status of their librarians 
indicated that institutions within this 
group would be facing the same set of 
problems regarding the evaluation of tech­
nology-based projects as their counterparts 
on college and university campuses. 

Background 
How are technology-based projects evalu­
ated for promotion and/or tenure in aca­
deme in general? Ernest Boyer suggests 
that the scholarly reward system can be­
come more flexible and vital by acknowl­
edging that scholarship can find expres­
sion in nontraditional ways, stating that 
“preparing quality computer software, 
for example, is increasingly a function of 
serious scholars. And even videocassette 
and television offer opportunity for com­
municating ideas to nonspecialists in cre­
ative new ways.”5 Martha Gilliland writes 
that “the tenure system is regularly 
blamed for many of the perceived ills of 
the university” and that “faculty mem­
bers’ notions of development … tend to 
emphasize their own fields of scholarship 
rather than teaching methods, the use of 
technology, or greater understanding of 
the needs and perceptions of constituen­
cies.”6 There are an increasing number of 
theoretical articles that discuss the pros 
and cons of “new scholarship” and offer 
differing perspectives on how computer-
related work could be evaluated in the 
tenure and promotion process—revision 
of guidelines, creation of new guidelines 
and criteria, and morphing of existing 
guidelines and criteria. A small number 
of studies have attempted to gain insights 
about what institutions are doing to de­
velop technology-based projects, how 
they are perceived in the promotion and/ 
or tenure process, or if and how tradi­
tional guidelines have been changed.7 In 
a survey sent to all deans of schools of 
education in the American Association of 
Colleges for Teacher Education, Nancy E. 
Seminoff and Shelley B. Wepner found 
that while institutions do not value tech­
nology-based projects as much as tradi­
tional scholarly works, faculty and ad­
ministrators believe that technology-
based projects merit equivalent value to 
textbooks and journal articles for tenure 
and promotion. Most respondents to the 
Seminoff and Wepner survey indicated 
that their institutions had not yet estab­
lished criteria for evaluating technology-
based projects.8 



 

466 College & Research Libraries 

How are technology-based projects 
being evaluated in academic libraries 
and what rethinking, if any, is taking 
place regarding the place of technology-
based projects in promotion and/or ten­
ure of librarians? In 1997 Pamela S. 
Bradigan and Carol A. Mularski ana­
lyzed the specific criteria used by aca­
demic library directors in the tenure and 
promotion process, but included only 
journal articles and monographs as the 
two most common forms of publications 
produced by academic librarians.9 Joan 
M. Leysen and William K. Black, in a 
survey on peer review in Carnegie Re­
search Libraries, asked library adminis­
trators to comment on the relative impor­
tance of contributions in electronic for­
mat in the peer review process.10 There 
have been no studies that specifically 
discuss if and how technology-based 
projects are being assessed in promotion 
and/or tenure decisions in ARL aca­
demic libraries, the materials used to 
evaluate their merit, and the criteria as­
signed for the assessment of technology-
based projects. 

Methodology 
On January 4, 1999, a survey was mailed 
to library administrators at 109 ARL aca­
demic libraries. An e-mail reminder was 
sent on January 20, 1999, with a final mail 
reminder to non-respondents sent on 
February 4, 1999. The survey instrument 
was composed of forty-six categorical 
and six open-ended questions which 
sought to assess how technology-based 
projects are evaluated in the promotion 
and/or tenure process for librarians at 
these institutions. For purposes of this 
study, technology-based projects were 
defined as computer software (including 
CD-ROM), or the publication of articles 
in electronic journals, or the develop­
ment of Internet-based materials (includ­
ing Web pages, tutorials, or digitization), 
or videotapes and audiotapes. The cover 
letter sent with the survey included an 
instruction that allowed an administra­
tor to refer the questionnaire to another 
person if they felt that person was more 

September 1999 

familiar with the evaluation of technol­
ogy-based projects at that library. In ad­
dition to institutional information, spe­
cific inquiries were made addressing the 
perceived need for technology-based 
projects in these libraries, the actual oc­
currence of technology-based projects, 
factors that are used to determine the 
value of these projects, criteria that are 
used to determine the value of these 
projects in promotion and/or tenure de­
cisions, and perceived credibility of tech­
nology-based projects in the promotion 
and/or tenure process. Respondents 
whose libraries had not experienced 
technology-based projects as compo­
nents of librarians’ promotion and/or 
tenure reviews during the past three 
years were asked to skip a portion of the 
survey which focused on active assess­
ment of these projects. Results are pre­
sented as proportional analyses using the 
calculation of frequencies and percent­
ages. 

Findings 
Seventy-seven responses from the one 
hundred and nine libraries contacted 
yielded a total of sixty-nine useable sur­
veys, for a final return rate of 63%. Not 
every respondent answered every ques­
tion; therefore, frequency data are given, 
as necessary, in the text. Of the sixty-nine 
respondents, fifty-eight (91%) were 
Carnegie I or II institutions. Thirty-six 
(52%) of the responding institutions in­
dicated that all librarians were faculty, 
while at four (6%) institutions some li­
brarians were faculty and at twenty-nine 
institutions (42%) librarians were not fac­
ulty. In trying to assess the overall cli­
mate for scholarship requirements at 
ARL academic libraries, a question was 
posed about whether scholarly/creative 
activity is REQUIRED for promotion 
and/or tenure of librarians at their insti­
tution. Forty-two (61%) of the libraries 
reported “Yes.” However, in a related 
question which asked whether schol­
arly/creative activity must appear in a 
refereed publication, fifty-three (77%) of 
the libraries said “No.” Forty-four (64%) 

http:process.10


 

  
  

 
 

  
  

 

 
 

 
 

 
  

 
   

 
 

 

�  
-�

E-
�.

�

�u

"'=
=' 

�=
  U

 u"
..=

5 E
u-<

==
:=5

 �.
"'u

� �
 =.

u-.
' .=

 -�
.


i;
9 

0; 
V �

-E.
-§  

 -
-

§ 
U 

§ 
-

� 
; E

E i
E E;

V 
Z0

§,�
 ;

 EE 
i .

0 
V.-

i .
0 

V.-
Z0

E
 V 

.
 V 

. 
iE 

E;V
 0;

-
:

 0;
-

: 
--9

 
- V

 l§
VE-E

;E-
0§V

 

 
V0§

V 
§

� 
§

� 
§

� 
§

� 
§

�
§

� 
l 

0 
--9

E l
§VE

 ; 
E-0

§ 
OO 

O00
 

M0 
O00

 
NO 

O00
 

Oo 
O00

 
N0 

O00
 

O0 
O00

 
--9

 
E �

§-
V-EE

 Z 
: 

OO 
O0 

M0 
Oo 

NO 
O0 

Oo
O00

 
N0 

O00
 

O0 
OO 

i 
V �

 E 
§ 

U0
§VE

-E;
 §E

V 
OM 

00 
Mo

00 
NO

00 
O0

0O 
No 

0O 
O0

00 
J 0

' V
V-0

§ 
0 §

-E-0
§ 

O0 
0M 

MO
0O 

NN
OO 

OO
0o 

N0 
0O 

OO
oO 

. 
0 E

;§-
EE '

0 
i .

0 
V.-

O0 
o0 

MO
o0 

NN
00 

OO
00 

N0 
0O 

OO
oO 

Z 
: -§

 --
9 

E J
 0'

 VV
-0§

 
OO 

0O 
MO

0O 
NM

o0 
OO

Oo 
N0 

0N 
OO

00 

Assessing Technology-based Projects 467 

of the sixty-nine respondents said 
that technology-based projects 
had been a component of promo­
tion and/or tenure reviews dur­
ing the last three years, while 
twenty-five (36%) libraries indi­
cated that this was not the case. 

Almost all of the group of 
forty-four libraries agreed that 
the reasons for creating technol­
ogy-based projects were to im­
prove library instruction or to 
meet a library or university need 
(see table 1). Eighty-six percent of 
these respondents believed that 
these projects enhanced aware­
ness of and responsiveness to 
needs of constituents outside 
their university. Eighty-three per­
cent of ARL academic libraries 
( n = 4 0 )  f e l t ,  a l s o ,  t h a t  t h e s e  
projects were important for pro­
viding professional recognition 
to their librarians, as well as pro­
viding opportunity for research/ 
scholarship (78%, n=40). A higher 
percentage of ARL academic li­
braries with faculty status than 
those without faculty status saw 
technology-based projects as an 
opportunity for scholarship and 
professional recognition. A some­
what lesser importance was as­
cribed to “need in the library pro­
fession,” with only 61% of forty-
one libraries indicating “yes” to 
this question. Answers to the 
question “Are there any other 
reasons why librarians at your li­
brary create technology-based 
projects?” included personal in­
terest of the developer, the fact 
that some projects become com­
mercial successes for their au­
thors, an opportunity for collabo­
ration with teaching faculty on 
the development of technology-
based products for curricular or 
research use, developing distance 
learning programs, and partici­
pation in an important mode of 
scholarly communication. 



  
 

  
   

   
   

   
   

   
   

   
  

   
  

   
   

 
   

   
   

  
   

   
  

   
 

   
 

   
  

 
   

   
 

   
   

 
   

   
   

  
   

 
   

 
 

   
 

   
 

   
  

    
   

  
 

   
   

   
    

   
   

   
  

   
   

   
 

   
   

   
   

   
   

   
   

   
   

   
   

    
  

   
  

   
   

 
   

   
   

   
  

   
   

   
 

   
    

  
   

  
   

   

 

468 College & Research Libraries 

 
 

   
 

 
   

 
  

  
   

 
 

    
  

  
 

  
   

   
   

 
  

  
 

 
 

 
 

 
 

 
 

 
 

 
 

 
  

 
 

 
 

 
 

 
 

 
 

 
 

 
 

 
 

 
  

   
  

   
  

   
  

   
  

   
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  

   
 

 
 

 
 

 
 

 
 

 
 

 
 

 
 

  
   

  
  

  
  

 
  

  
 

 
 

 
 

  
  

 
  

 
  

 
 

  
  

  
   

 
  

  
 

 
  

  

September 1999 

Specific Technology-
based Projects That are 
Being Assessed 
Table 2 shows the actual oc­
currence of various technol­
ogy-based projects reported 
from the responding librar­
ies. 

The projects most com­
monly reported as being 
components of promotion 
and/or tenure decisions 
were World Wide Web 
(WWW) subject or re­
source pages, tutorials or 
other major WWW 
projects. Electronic journal 
publications were most im­
portant to those libraries 
which had faculty status or 
that required scholarship. 
Other projects evaluated, 
according to comments re­
ceived in response to the 
question “Are there any 
other types of technology-
based projects that have 
been components of pro­
motion and/or tenure re­
views in your Library?” 
were curricular, internal, or 
Internet-based. Curricular 
projects included a com­
puter-based instruction 
lesson, CD-ROMs, collabo­
rative projects with teach­
ing faculty to design exten­
sive course web sites, a 
video-streamed instruction 
module, and a grant pro­
posal to integrate technol­
ogy into the curriculum. 
Some internal projects 
evaluated were local en­
hancements to an online 
system and an electronic 
reserves project. Internet-
based activities included 
the creation of Web pages 
for professional or 
consortial groups, devel­
opment of a Web search en­



 

  
 

  
   

  
   

   
    

   
   

   
 

   
  

   
   

   
   

   
   

   
  

   
 

   
 

   
  

 
   

   
 

   
   

 
   

   
   

  
   

 
   

 
 

   
 

   
 

   
  

    
   

  
 

   
 

   
    

   
   

   
  

  
 

   
   

   
   

   
  

   
   

  
   

   
   

 
   

   
   

   
   

   
  

   
 

   
   

   
   

  
   

   
   

    
  

   
  

' "
 

 
 

 
 

 
 

 
 

 
 

 
 

 
 

  
  

 
 

  
  

 
 

  
  

 
 

  
  

 
 

  
  

 
 

  
  

 
 

  
  

 
 

  
  

 
 

. '
= 

 
' 

   
 

  
 


 
  

  
 

 
 

 
 

 
 

 
 

 
 

 
 

 
  

 
 

 
 

 
 

 
 

 
 

 
 

 
 

 
 

 
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

 
  

  
  

 

Assessing Technology-based Projects 469 

gine, some Web/gateway de­
signs, and being an electronic 
journal editor-in-chief. 

Components Used to 
Evaluate Technology-
based Projects 
Respondents indicated that 
the component most used to 
evaluate the merit of a tech­
nology-based project was the 
project itself. 

Other factors used fr e­
quently (about 75% of the 
time) were a project summary 
or project description (see 
table 3). End-user comments 
and peer review were consid­
ered to be important by 58% 
(n=40) and 57% (n=42) of the 
respondents, respectively. 
Print components, whether 
appearing in refereed or non-
refereed journals, and project 
procedures were the least im­
portant to the respondents. 
Another concrete component 
considered by one evaluator 
in response to the question 
“Are there any other materi­
als that you might evaluate 
when making a decision 
about the merit of technology-
based projects?” was an 
award given to a technology-
based project. Other respon­
dents considered word of 
mouth fr om the field and 
value to the national and in­
ternational community. 

Criteria Used to Determine 
the Merit of Technology-
based Projects 
The survey asked respon­
dents to indicate which in a 
group of seven criteria for de­
termining the merits of tech­
nology-based projects for pro­
motion and/or tenure were 
CURRENTLY being used in 
their library.



   
   

  
   

 
   

 
   

  
 

   
   

 
   

   
 

   
   

   
  

   
 

   
 

 
   

 
   

 
   

  
    

   
  

 
   

 
   

    
   

   
   

   
  

   
   

   
   

   
   

    
   

   
   

   
    

   
   

   
  

   
    

   
  

   
   

   
  

   
   

  
   

  
   

  
   

   
   

  
�.

� .



 '
'"'

': 
� �

".:
 ' o

.  
"'=

 o
:o�

...
''"

� �
.o.

"' 
' : 

   
.


 
 

 
 

 
 

 
 

 
 

 
 

 
 

 
 

 
 

  
  

 
 

  
  

 
 

  
  

 
 

  
  

 
 

 
 

  
  

 
 

  
  

 
 

  
  

 
  

  
 

 
 

 
 

 
 

 
 

 
 

 
 

 
  

 
 

 
 

 
 

 
 

 
 

 
 

 
 

 
 

 
 

 
 

 
 

 
 

 
 

 
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  

470 College & Research Libraries September 1999 

Over 80% of the report­
ing libraries (n=42) consid­
ered the creativity or 
uniqueness of a technol­
ogy-based project and its 
scholarly contribution to 
library science to be the 
primary factors in assess­
ing the merit of a project 
(see table 4). Local recog­
nition and national/inter­
national recognition of the 
project were factors used 
in assessing merit in 78% 
(n=41) and 73% (n=41) of 
the respondents, respec­
tively, while 62% also in­
cluded the “logical devel­
opment of ideas” (n=39). 
Institutions with faculty 
status or where scholar­
ship was required as­
signed a somewhat higher 
value to national/interna­
tional recognition of 
projects, while those insti­
tutions with non-faculty 
status assigned a higher 
status to local recognition. 
Responses to the question 
“Are there any other crite­
ria that are being used to 
determine the merit of 
technology-based projects 
for promotion and/or ten­
ure in your library?” indi­
cated that the other main 
criterion being used is 
whether a project is a prac­
tical utility for meeting the 
needs of a targeted audi­
ence. Only one institution 
specifically noted a will­
ingness to consider a con­
tribution by a librarian in 
any area of scholarship, 
not just in library science. 

In a related question, 
the same group of seven 
criteria were listed and re­
spondents were given an 
opportunity to indicate



 Assessing Technology-based Projects 471 

whether, in their opinion, the criteria that 
SHOULD be used to assess these projects 
were different from those currently be­
ing used. No differences were seen in re­
spondents’ answers to the two questions. 
Extensive comments were offered by re­
spondents who had no technology-based 
projects to evaluate in the past three 
years, as well as those who had. Respon­
dents with experience evaluating tech­
nology-based projects would look for 
usefulness and user-friendliness (to lo­
cal users and users at other research li­
braries), promotion and enhancement of 
library services, and evidence of effective 
functioning (as an information medium 
and of broad-based access and use). The 
difficulty of the problem solved and will­
ingness to experiment/take risks with 
technology-based projects, thereby con­
tributing to the research mission of the 
University and the profession were other 
criteria offered by two Associate Deans. 
When asked if their library’s promotion 
and/or tenure document included spe­
cifically written criteria for technology-
based projects 88% of ARL academic li­
braries answered “No” (n=59). However, 
in libraries that have faculty status or 
where scholarship is required, an affir­

mative answer was somewhat higher 
(17%, n=36; 19%, n=42). 

Regarding Credibility of Technology-
based Projects 
Respondents were almost split evenly 
over whether they felt that technology-
based projects deserved more credibility 
in the promotion and/or tenure process 
than they may currently have at their in­
stitutions: 49% answered “Yes,” while 
51% said “No” (n=63). 

Results were similar when broken 
down into Carnegie only institutions, and 
institutions with and without scholarship 
requirements. However, a sharp diver­
gence was noted in those libraries with 
or without faculty status. Respondents 
from institutions with faculty status felt 
decidedly (61%, n=33) that these projects 
did deserve more credibility, while those 
institutions without faculty status felt that 
they did not (69%, n=26)(see figure 1). 
Only one respondent from each of the fac­
ulty/non-faculty categories felt that tech­
nology-based projects are already treated 
equally with print products in promotion 
and/or tenure reviews. 

In a final question, tenure track insti­
tutions only were asked if technology-

FIGURE 1
Do Technology-based Projects at ARL Deserve More Credibility? 



 

472 College & Research Libraries September 1999 

based projects alone (without accompa­
nying traditional scholarship) would 
ever be sufficient evidence of scholarship 
for the granting of tenure. Sixty percent 
of the respondents felt positively that this 
could be the case in the future. Several 
added the disclaimer that “ever” was a 
long time and that they did not expect a 
change in the near future. 

Discussion 
Many unknowns have surrounded the 
issue concerning the evaluation of tech­
nology-based projects in the promotion 
and/or tenure process in various aca­
demic disciplines. “As candidates for 
jobs and promotions stock their portfo­
lios with Internet-related accomplish­
ments, many evaluation committees are 
skeptical. Behind the closed doors, com­
mittee members are asking questions that 
betray equal parts confusion and suspi­
cion. Should a candidate’s Internet 
project count? Is it teaching, scholarship, 
or service? Does editing an electronic 
journal require the same kind of rigor as 
editing a print journal? Who is referee­
ing all this stuff, anyway?”11 A respon­
dent to our survey who answered “NO” 
to the question “In the last three years, 
have any technology-based projects that 
have been developed at your library been 
components of promotion and/or tenure 
reviews?” commented that “at a Univer­
sity library where technology plays a 
major role, candidates should not expe­
rience any difficulty with technology-
based projects.” However, evaluators 
and those being evaluated at institutions 
where these projects have been submit­
ted for review and are increasingly in­
cluded in promotion and/or tenure files 
are experiencing a variety of difficulties. 
In an effort to explore the current climate 
in ARL academic libraries, we asked 
these institutions to identify the compo­
nents and criteria used to evaluate these 
projects. From a group of more than forty 
libraries that had dealt with technology-
based projects in promotion and/or ten­
ure decisions within the last three years, 
we have gathered a snapshot of the pro­

cess at the time of the survey. Not sur­
prisingly, we found that the majority of 
technology-based projects in ARL aca­
demic libraries were initiated in response 
to library or university need and that the 
majority of these projects were Internet 
related. We also found that, while some 
of the factors for dealing with these 
projects in promotion and/or tenure de­
cisions were similar to those occurring 
in other areas of academia, others were 
quite different. 

ARL academic libraries with faculty 
status took the initiative to combine “pro­
fessional recognition,” “opportunity for 
scholarship,” and “need in the profes­
sion” with library/university needs as 
reasons for creating technology-based 
projects to a higher degree than did non-
faculty institutions, reflecting a higher 
probable need for scholarly activity. The 
level of creative thinking that may be in­
volved in the conception and realization 
of technology-based projects may easily 
equal that of traditional scholarship. On 
a related issue, we found that refereed 
print components are used to make a de­
cision about the merit of a technology-
based project far more by librarians with 
faculty status than by those without fac­
ulty status, most probably because of 
more pressure to produce any type of ref­
ereed publication. 

The survey showed that Internet-re­
lated technology dominates the types of 
technology-based projects appearing in 
promotion and/or tenure decisions 
within the past three years. No doubt li­
braries are responding to a market need 
for information that is in a convenient, 
easily accessible form. Many of these 
products are needed during a limited 
time frame and it may not be important 
that they remain part of a permanent 
record. However, this remains a problem 
to be resolved, as illustrated in the case 
of the archiving and retrieval of elec­
tronic journal articles.12 The fact that elec­
tronic journal articles were most impor­
tant to libraries that have faculty status 
or that require scholarship bears out a 
finding from a 1997 Canadian Policy 

http:articles.12


 Assessing Technology-based Projects 473 

Conference on Scholarly Communication 
in the Next Millennium, that the “percep­
tion among scholars that work published 
electronically is less legitimate from a 
scholarly standpoint than work pub­
lished on paper in a prestigious journal 
is diminishing. As scholars become con­
nected to the Internet and involved in 
electronic communication they see the 
enormous benefits to be had.” 13 

Only twenty-four of the responding 
forty-two ARL academic libraries said 
that they have used peer review to evalu­
ate the merit of technology-based project 
in the past three years, perhaps reflect­
ing the lack of traditional sources for peer 
review of these projects. Respondents 
comments indicated a variety of prob­
lems with peer review: consistency of 
peer review procedures, not enough ven­
ues for national/international recogni­
tion, and unsureness about the peer re­
view process in electronic journals. Many 
respondents from tenure-track institu­
tions felt that peer acceptance and peer 
review are critical for technology-based 
projects if they are to be taken seriously 
in the promotion and/or tenure process. 
The American Association of the Col­
leges of Teacher Education has an “In­
novative Use of Technology Award” pro­
gram and one survey respondent would 
like to see “ALA/ARL sponsor a prize 
or two for this type of enterprise.” 

The survey results indicate that the 
evaluation criteria used currently to de­
termine the value of technology-based 
projects in ARL academic libraries is for 
the most part in alignment with Seminoff 
and Wepner ’s study of deans of schools 
of education14. The deans and ARL aca­
demic libraries disagree about the rela­
tive value of two criteria. The criterion 
used most frequently by ARL academic 
libraries to assess the merit of technol­
ogy- based projects was “exhibits 
uniqueness or creativity,” while the 
deans of the schools of education listed 
“uniqueness or novelty” near the bottom 
(number 8 of 10) of a similar list of crite­
ria. The education deans’ third most im­
portant criteria for evaluating technol­

ogy-based projects was “well-researched 
data collected prior to development.” 
This is in concurrence with the MLA 
“Guidelines for Evaluating Computer-re­
lated Work in the Modern Languages”15 

which state that “faculty members are re­
sponsible for making a case for the value 
of their projects, articulating the intellec­
tual assumptions underlying their work, 
and documenting their time and effort 
… Faculty members should be prepared 
to explain what theory informs their 
work, why their work is useful to the 
discipline, and the evidence of rigor and 
intellectual content in their work.” Con­
versely, ARL academic libraries ranked 
“includes a thorough review of prior 
developments in the project’s area of re­
search” at the bottom of their list of cri­
teria. All other criteria in the survey of 
ARL academic libraries are ranked simi­
larly to the survey of education deans. 
Many respondents indicated a concern 
that their technology-based projects 
would have more credibility only when 
the projects of teaching faculty had more. 
Comments from respondents indicated 
that this is even a greater issue for librar­
ians whose promotion and/or tenure 
files are reviewed by university commit­
tees or administrators. 

Conclusions 
It is clear that there is an additional bur­
den involved for librarians who produce 
technology-based projects and include 
them in promotion and/or tenure files. 
Because review committees and admin­
istrators currently lack a depth of expe­
rience in evaluating these projects, the 
author must take the initiative in provid­
ing as much documentation of the project 
as possible. The concept that the “docu­
mentation of any faculty work should 
stress two dimensions (1) the quality of 
the work and (2) the significance of the 
work. In many instances faculty provide 
promotion and/or tenure committees 
with detailed information as to the qual­
ity of the effort; however, they do not 
present a case for the value of their work, 
describing its impact or explaining in 



474 College & Research Libraries September 1999 

what ways and for whom this work has 
significance”16 is doubly important if a 
technology-based project is up for re­
view. Submission of documentation that 
shows that projects have received inter­
nal or external funding, have earned an 
award or other professional recognition, 
and reviews and citations of work either 
in print or in electronic journals are all 
useful. 

Inclusion of evaluation processes for 
technology-based projects should be in­
corporated into existing library guide­
lines for promotion and/or tenure re­
views. If an institution believes that tech­
nology-based work, like other forms of 
scholarship, teaching, and service, 
should be evaluated as an integral part 
of a faculty member ’s accomplishments, 
language related to teaching, scholarly/ 
creative activities, and professional ser­
vice throughout the guidelines should 
also refer to computer-related work. 
“When an archeologist uses a Computer-
Aided design program to reconstruct a 
site, or a rhetorician moderates a four-
month discussion online, or an historian 
collaborates with 120 colleagues to pro­
duce a polylog, departments and tenure/ 
promotion committees will need to know 
how to evaluate those efforts.”17 Certified 
peer review is still an important factor 
for a successful review in academe, and 
academic librarians who must face Uni­
versity-wide review for promotion and/ 
or tenure must pay special attention to 
this criterion. “Whether the current form 
of peer review remains in place for long 
is of importance only inasmuch as it re­
mains a viable measure of the life of the 
scholarly community. It is widely agreed 
that the current scholarly communication 
system of editorial boards, reviewers, 
and publishers will continue in place for 
at least another decade and probably 
much longer. Although the number and 
variety of scholarly publications that are 
exclusively electronic has grown tremen­
dously since the 1980s, there has not yet 
been the kind of fundamental change 
that would spell the end of the current 
regime and the start of the new regime.

Rather, there is a gradual transformation. 
As more scholars establish their own 
communication networks and the cred­
ibility of the work being disseminated 
grows, the efficacy of this new means of 
communication will become evident and 
attractive.”18 It is also incumbent on ad­
ministrators to make review committees 
understand whether or not these projects 
are valued at their institution. 

Academic librarians can play an 
important role in addressing the 
serious questions about technology-
based projects that are before us. 

Library evaluators seem prepared to 
take more risks and step outside tradi­
tional guidelines when evaluating tech­
nology-based projects when compared 
with their counterparts in other areas of 
academe. They are willing to assess the 
content as well as the container. Along 
with Janice Walker, they are finding that 
“ we are left with three choices: first, 
make our electronic work somehow “fit” 
into existing guidelines and be able to 
justify it along traditional lines; second, 
do what we’re doing now and not have 
it count for purposes of tenure and pro­
motion; or, third, change the definitions 
of what is ‘valued’ to fit what we’re do­
ing.”19 The respondents to the survey felt 
that there will be a “general change to 
recognize electronic scholarly activity,” 
and that an increase in the credibility of 
technology-based projects “will be inevi­
table as [they] will become increasingly 
the norm as the WWW is used for ser­
vices and instruction as well as access 
tools.” “The effort to broaden the mean­
ing of scholarship simply cannot succeed 
until the academy has clear standards for 
evaluating this wider range of scholarly 
work. After all, administrators and pro­
fessors accord full academic value only 
to work they can confidently judge.” 20 

The results of our survey show that 
ARL academic librarians recognize that 
technology-based projects are valuable 
and should be seriously considered in 
promotion and/or tenure reviews, but 



 

 

Assessing Technology-based Projects 475 

that there has been little initiative to regu­
larize its consideration and evaluation. 
A question in the survey asked tenure-
track institutions “In your opinion, will 
technology-based projects alone (without 
traditional scholarship) ever be sufficient 
evidence of scholarship for the granting 
of tenure?” The majority of respondents 
replied “yes,” but always with a caveat. 
Comments offered were generally of this 
type: “Yes, but depends greatly on sub­
stance, quality and peer acceptance,” 
“Yes, if there is strong and effective docu­
mentation of use and value in the aca­
demic community of the specific 
projects,” and “Yes, I would not confuse 
the medium with the content. It is the 

content of the project that should be 
evaluated.” It is apparent that, although 
the environment is dynamic and con­
stantly evolving, there is a need for a core 
set of measures and assessment tech­
niques that evaluate technology-based 
projects. Academic librarians can look at 
the evaluation criteria and guidelines 
used by other academic institutions and 
other disciplines in the consideration of 
technology-based projects and compare 
them with their own organization’s 
evaluation processes. Academic librar­
ians can play an important role in ad­
dressing the serious questions about 
technology-based projects that are before 
us. 

Notes 

1. Thomas J. DeLoughry, “Professors Report Progress in Gaining Recognition for Their 
Use of Technology,” Chronicle of Higher Education 39, no. 26 (March 3, 1993): A19, A21. 

2. Lisa Guernsey, “Scholars Who Work With Technology Fear They Suffer in Tenure Re­
views,” Chronicle of Higher Education 43, no. 39 (June 6, 1997): A21-22. 

3. Modern Language Association.  MLA Committee on Computers and Emerging Tech­
nologies in Teaching and Research, “Guidelines for Evaluating Computer-related Work in the 
Modern Languages,” ADE Bulletin 114 (1996): 44-46. 

4. Edward Shreeves, “Between the Visionaries and the Luddites: Collection Development 
and Electronic Resources in the Humanities,” Library Trends 40, no. 4 (spring 1992): 579-95. 

5. Ernest L. Boyer, Scholarship Reconsidered: Priorities of the Professoriate (Princeton, 
N.J.:Carnegie Foundation for the Advancement of Teaching, 1990), 36. 

6. Martha W. Gilliland, “Organizational Change and Tenure: We Can Learn From the Cor­
porate Experience,” Change 29, no. 3 (May/June 1997): 30-33. 

7. For example, see: Devorah A. Lieberman and John Rueter, “The Electronically Aug­
mented Teaching Portfolio,” in The Teaching Portfolio : a practical guide to improved performance 
and promotion/tenure decisions 2nd ed., Peter Seldin (Bolton, MA.:Anker Publishing Company, 
1997), 46-57. Blaise Cronin and Kara Overfelt, “E-Journals and Tenure,” Journal of the American 
Society for Information Science 46, no. 9 (October 1995): 700-2. Nancy E. Seminoff and Shelley B. 
Wepner, “What Should We Know About Technology-Based Projects for Tenure and Promo­
tion?,” Journal of Research on Computing in Education 30, no. 1 (fall 1997): 67-82. “Tenure and 
Technology: New Values, New Guidelines” in Kairos: A Journal for Teachers of Writing in Webbed 
Environments 2, no. 1 (spring 1997) ( http://english.ttu.edu/kairos) includes: Cindy Nahrwold 
“‘Just’ Professing; A Call for the Valuation of Electronic Scholarship, Seth R. Katz “One 
Department’s Guidelines for Evaluating Computer-related Work, Janice R. Walker “Fanning 
the Flames: Tenure and Promotion and Other Role-Playing Games,” Janet Cross and Kristian 
Fuglevik “Jesters Get Serious.” 

8. Nancy E. Seminoff and Shelley B. Wepner, “Are Technology-Based Projects Valued as 
Scholarship for Tenure and Promotion?,” Journal of Computing in Teacher Education 11, no. 3 
(1995): 5-10. 

9. Pamela S. Bradigan and Carol A. Mularski, “Evaluation of Academic Librarians’ Publi­
cations for Tenure and Initial Promotion,” Journal of Academic Librarianship 22, no. 5 (September 
1996): 360-65. 

10. Joan M. Leysen and William K. Black, “Peer Review in Carnegie Research Libraries,” 
College & Research Libraries 59, no. 6 (November 1998): 512-522. 

11. Guernsey, “Scholars Who Work With Technology,” A21. 
12. Charlotte E. Ford and Stephen P. Harter, “The Downside of Scholarly Electronic Pub­

lishing: Problems in Accessing Electronic Journals through Online Directories and Catalogs,” 
College & Research Libraries 59, no. 4 (July 1998): 335-346. 

http://english.ttu.edu/kairos


476 College & Research Libraries September 1999 

13. Kenneth Field, “Faculty Perspective on Scholarly Communication,” Canadian Journal of 
Communication 22, no. 3/4 (1997): 161-178. 

14. Seminoff and Wepner, “What Should We Know,” 70. 
15. Modern Language Association, “Guidelines for Evaluating,” 45. 
16. Robert M. Diamond, Serving on Promotion and Tenure Committees: A Faculty Guide. (Bolton, 

MA.:Anker Publishing Co. Inc., 1994), 18. 
17. Paul LeBlanc “Pulling Out the Rug: Technology, Scholarship, and the Humanities” in 

The Politics and Processes of Scholarship edited by Joseph M. Moxley and Lagretta T. Lenker 
(Westport, Connecticut; London: Greenwood Press, 1995), 122-123. 

18. Field, “Faculty Perspective,” 165. 
19. Walker, “Fanning the Flames,” (http://english.ttu.edu/kairos/2.1/index_f.html) 
20. Charles E. Glassick, Mary Taylor Huber, and Gene I. Maeroff, Scholarship Assessed: Evalu­

ation of the Professoriate (San Francisco: Jossey-Bass Publishers, 1997), 5. 

http://english.ttu.edu/kairos/2.1/index_f.html