College and Research Libraries


EVA L. KIEWITT 

A User Study of a Computer 

Retrieval System 

A study was conducted of the users of the Indiana University 
PROBE computer retrieval program using the Resources in Education 
ERIC data base. The critical incident survey identified the PROBE 
user characteristics, needs, and satisfaction. A typical user profile was 
determined and overall satisfaction was found to- be 79.5 percent. As 
a result of the study, changes were effected in the retrieval program 
concerning time lag, price, use of computer fields, and query 
specificity. 

A BETTER UNDERSTANDING OF EDUCATORS 
and their information needs can be ac-
quired by studying them as users of an 
information system. Since the purpose 
of gathering and disseminating infor-
mation is to satisfy these users, their 
needs and requirements must be studied 
to assure the utility of an information 
system. Both in planning and in evaluat-
ing a system, a user's interface is of 
critical importance. 

If inexpensive and rapid retrieval 
systems are developed and are easily ac-
cessible, school teachers and administra-
tors may be encouraged to obtain and 
use relevant research. Thus, research 
would be available to the practitioners 
as well as to the university students and 
faculty. 

When describing the installation of 
a coordinated information network in 
the New York State Education Depart-
ment, Hull and Benson stated: 

The improvement of educational pro-

Eva L. Kiewitt is assistant professor, 
Graduate Library School, Indiana U niversi-
ty, Bloomington. 

458 I 

cesses and systems depends on the 
availability of appropriate information 
for rational decisions. Information 
must be available in a form which can 
be understood by a decision-maker, 
and at a time appropriate to a deci-
sion. :Many a school superintendent or 
classroom teacher has acted out of ex-
pediency because he was unaware of 
alternative plans of action. The quality 
and amount of information available 
to the decision-maker can be expected 
to influence the outcome.l 

Users of a computer retrieval pro-
gram at Indiana University were studied 
in the spring of 1973 to determine their 
characteristics and satisfaction. The 
data base used in the retrieval program 
was the Resources in Education (RIE), 
formerly Research in Education, mag-
netic tapes from the Educational Re-
sources Information Center (ERIC) 
system. 

ERIC was established by the U.S. Of-
fice of Education in 1966 in response to 
a growing concern for control of pro-
liferating research reports in the field 
of education. It is now under the spon-
sorship of the National Institute of Ed-



ucation ( NIE). The national network 
is composed of sixteen specialized sub-
ject centers, or clearinghouses, which 
select, screen, and abstract educational 
research reports and disseminate this re-
search by means of microfiche collec-
tions and magnetic tapes. The two ma-
jor ERIC index tools, Resources in Ed-
ucation (RIE) and Current Index to 
Journals in Education (CI]E), are 
available both in hard copy and on 
machine-readable magnetic tapes. RIE 
was begun in 1968 as a monthly abstract 
journal with subject, author, and insti-
tution indexes to the ERIC research re-
ports. 

In order to make information readily 
available to both practitioners and re-
searchers, many libraries and informa-
tion centers began to develop local re-
trieval computer programs. PROBE, a 
batch-mode retrieval program of the 
ERIC data base, was developed in 1970 
by Ronald Tschudi at Indiana Univer-
sity. The program has a variable-field 
format which gives it a broad range of 
capabilities. Details of the program are 
described in a technical report pub-
lished in 1972 by Tschudi and Mere-
dith, 2 and the history of its develop-
ment was reported in 1973.3 

The retrieval program was supported 
by aid from the University, a one-year 
LSCA grant from HEW awarded 
through the Indiana State Library, and 
a $4.00 fee per search from the users. 
With the aid from the grant it was pos-
sible to expand service to educators 
throughout the state of Indiana and ac-
cept some out-of-state requests for 
searches. 

USER NEEDS AND STUDIES 

After the retrieval system had been 
in use more than a year, it was decided 
to evaluate the system by studying the 
needs and satisfaction of the users. The 
user "feedback" ·could help determine 
changes needed to improve the system 
and satisfy the users. 

User Study I 459 

The first major evaluation study of 
information retrieval systems was the 
Cranfield investigation in England. Four 
indexing systems were tested and the 
studies were summarized by Cleverdon 
in 1962. The major results of the tests 
were not only the comparison of the in-
dexing systems but methods of evalua-
tion. 4 Swanson reviewed the Cranfield 
experiments in 1965, citing some of the 
major findings. He stated the value of 
the project but criticized the inaccurate 
interpretations and generalizations of 
the Cranfield data. 5 

In discussing evaluation of systems, 
Cleverdon stated: 

In future work in evaluation testing, 
the users must be included as an inte-
gral part of the system and their re-
quirements must be evaluated along 
with the other system components. 6 

Few systems are viable without "feed-
back" from the user, since information 
is stored in anticipation of some user's 
needing it. After reviewing studies of 
user needs from 1953 to 1966, Coover 
said: 

A determination of the needs of users 
is absolutely essential to the manage-
ment of an information center .... In 
fact, the value of the Information 
Center can be known only in terms of 
its satisfying the users' needs. 7 

There are various techniques of an-
alyzing user needs. One method used 
frequently in user studies is the critical 
incident survey, which relates to a par-
ticular incident of searching for infor-
mation. This method was used in Lan-
caster's evaluation of the MEDLARS 
Demand Search Service of medical liter-
ature from 1966 to 1968. He studied 
twenty-one user groups and 607 requests 
and analyzed 302 actual searches.8 

In each of the volumes of Annual 
Review of Information Science and 
Technology from 1966 to the present, 
a chapter has been devoted to informa-
tion needs and uses. A review of these 



460 I College & Research Libraries • November 1975 

studies of user needs and the methods 
used to determine their requirements 
showed that the most viable method of 
studying the PROBE users was the criti-
cal incident technique. 

CRITICAL INCIDENT SURVEY 

A critical incident survey of PROBE 
users was conducted in the spring of 
1973. As each user requested a computer 
search by means of mail, telephone, or 
"walk-in," a search request form was 
completed, either by the librarian or by 
the user. The request form included 
space for basic information about the 
patron, a description of the problem, 
and .a place for descriptors or other ele-
ments to be used in formulating the 
search query. 

The description of the patron's sub-
ject of interest was discussed thoroughly 
with the individual if the request was 
made in person or by telephone. If the 
search arrived by mail, the librarian ex-
amined the user's statement of the prob-
lem or request and chose the terms to be 
searched by the computer. 

Boolean logic was used to combine 
the selected descriptors (subject head-
ings) to form the computer query, to-
gether with the logical connectors "and" 
(conjunction), "or" (disjunction), and 
"not" (negation ) . 

Since the PROBE computer program 
is very flexible, it is possible to search 
the tapes in many ways. The majority 
of users have been able to find descrip-
tors listed in the Thesaurus of ERIC 
Descriptors to fit their interests. 9 How-
ever, if no descriptor is judged to be 
relevant, a search m.ay be made through 
other bibliographic fields, such as the 
title, the author, the identifier (proper 
nouns, such as Indiana University), or 
through the abstract field. The abstract 
m.ay contain words or phrases that are 
not used as descriptors, and a search 
through this summary paragraph with 
the use of natural language terms may 

locate a relevant article not found by 
descriptor search. 

The user received a complete printout 
of the first 100 abstracts and up to 700 
additional citations, if the inquiry gen-
erated that many hits. 

The search request forms were sent 
daily, M<?nday through Friday, to a key-
punch operator, who then submitted 
them at the Wrubel Computer Center 
in batch-mode, three times each week. 

Following completion of each com-
puter search, either the patron was noti-
fied of the receipt of the search or the 
printout was mailed. A cover letter was 
attached to each printout asking that 
the enclosed questionnaire be completed 
and returned after the user's appraisal 
of the results. 

From the questionnaires the charac-
teristics of the user were obtained: the 
purpose of the search, satisfaction, and 
willingness to pay. Information about 
age, sex, position, level of education, 
and subject interest area was requested 
in order to obtain a profile of the users. 
Satisfaction was inferred from answers 
to questions concerning relevancy of 
the output, the worth of nonrelevant 
matches, the number of documents new 
to the user, usefulness of the informa-
tion, and good features and problems 
of the computer search. 

USER CHARACTERISTICS 

The questionnaire, with a cover letter 
of explanation, was included with the 
computer output of 200 consecutive 
searches. Of these 200 questionnaires, 
156 ( 78 percent) were returned. Some 
of the users requested more than one 
search during the evaluation period, 
and, consequently, the number of indi-
vidual users who returned the question-
naire was 141. 

The majority of users were from col-
lege and university campuses. Students 
1and faculty from Indiana U nivers1ty 
comprised 63.1 percent; from other In-
diana universities, 14.9 percent; users 



from Indiana elementary and secondary 
schools, 12.8 percent; and users from 
out-of-state locations, 9.2 percent. 

The age group of the users ranged 
from eighteen to twenty-one to forty-
six to fifty-five years of age. The largest 
number of users (sixty-four, or 47.8 
percent) were in the twenty-six to thirty-
five age-group category. 

The educational levels of the respon-
dents ranged from high school gradu-
ates (ten, or 7.1 percent) to those with 
doctorates (twenty-six, or 18.4 percent). 
The largest number of users reported 
the master's degree as their highest aca-
demic degree (fifty-six, or 39.7 percent). 
The majority of the users were male 
(eighty, or 56.7 percent), with sixty-one, 
or 43.3 percent, female. 

The occupations of the users were 
quite varied. The primary user was the 
graduate student (sixty-seven, or 47.5 
percent), with college faculty as the 
second largest group (twenty, or 14.2 
percent). The greatest number of users 
were associated with a college or univer-
sity campus. This was anticipated since 
the ERIC data base is primarily orient-
ed toward research; also, the PROBE 
service and most ERIC collections are 
located on university campuses. How-
ever, twenty-nine ( 20.6 percent) of the 
users listed elementary, middle, or high 
school as their primary institutional as-
sociation. 

Sixty ( 42.5 percent) of the users in-
dicated that they had conducted re-
search within the past five years; and 
thirty-eight ( 26.9 percent) of the users 
stated that they had published articles 
or books or presented papers during 
that period. 

The individual r~spondents were 
queried concerning their use of ERIC 
material. Of the users, 107 (75.9 per-
cent) indicated they had easy access to 
ERIC collections, usually at one of the 
four state universities or private col-
leges in Indiana. Seventy-two percent of 
the users stated that they found the use 

User Study' I 461 

of microfiche acceptable but not pref-
erable. They noted that it is easy to read 
and store, the necessary data are avail-
able on microfiche, and it is less expen-
sive than hard copy. 

The PROBE office was interested in 
learning where users had obtained in-
formation concerning the availability 
of the PROBE computer service. The 
greatest number ( 27.2 percent) stated 
that they had heard of PROBE from 
a colleague. The three other methods of 
communication that were most effective 
were: librarian, 22.8 percent; teacher, 
21.7 percent; and brochure or flyer, 10.9 
percent. 

SATISFACTION OF PROBE USERS 

The satisfaction of the users was as-
certained by requesting information on 
the purposes of their PROBE searches 
and asking for the overall usefulness 
or value of their results. 

A number of users indicated that 
they requested the computer search for 
more than one reason, as is shown in 
Table 1. 

The users indicated research as the' 
major purpose for a PROBE search ( 49 
percent); of this number, forty-two 
specifically mentioned that it was for a 
dissertation. A few others said that they 
were searching for a dissertation topic. 

TABLE 1 

PuRPOSES OF THE CoMPUTER SEARCHES 

Number of 
Purpose of Search User Purposes Percent 

Research project 101 49.0 
( Dissertation, 
specifically, 42) 

Assignments or term paper 41 19.9 
Preparation of speech, 21 10.2 

report, or article 
Curriculum development 17 8.2 
Program improvement 9 4.4 
To keep a breast in a field 8 3.9 
Preparation or updating of 

course bibliographies 
4 1.9 

Browsing 2 1.0 
Other 3 1.5 
Total purposes 206 '100.0 



462 I College & Research Libraries • November 1975 

The crucial question concerning satis-
faction was how the users rated each 
search as to its usefulness to their study 
or work. Of the 156 searches conducted 
for 141 users, their value was indicated 
in 151 of the responses, .as shown in 
Table 2. 

TABLE 2 
USEFULNESS OF THE DESCRIPTOR SEARCHES 

Number of 
Value of the Search Searches Percent 

Very high value 42 27.8 
High value 52 34.5 
Medium value 26 17.2 
Little value 16 10.6 
No value 15 9.9 
Total 151 100.0 

Ninety-four ( 62.3 percent) of the 
searches were considered of high or very 
high value to the users; twenty-six ( 17.2 
percent) of medium value; and thirty-
one ( 20.5 percent) of little or no value. 
Thus, overall, 120 ( 79.5 percent) con-
sidered the searches of value compared 
with 31 ( 20.5 percent) who felt that 
their searches were of little or no value. 

The users indicated that they found 
more relevant reports than nonrelevant 
in their searches. However, since some 
nonrelevant material was found, the 
users were asked to indicate whether 
they felt it was worthwhile examining 
this material. Fifty-six percent of them 
felt that it was worthwhile to examine 
this additional material. 

The time lag from request of the 
search to the receipt is often very im-
portant to users. Thus, it was necessary 
to determine if the users felt the turn-
around time of the PROBE search was 
adequate for their needs. During the 
evaluation period the PROBE schedule 
had been to process the searches three 
times a week in batch-mode. Unless 
mailing time was involved, the user usu-
ally could obtain a search in two or 
three days. Eighty-eight percent of the 
users felt that the length of time from 

request to receipt of their printout was 
either fast or reasonable. 

A large number of users ( 81.5 per-
cent) indicated that much of the infor-
mation obtained in the computer 
searches was new. The novelty ratio was 
60.9 percent, which shows that over half 
of the relevant information obtained 
was new to the user. 

Users were asked to identify both 
good features .and problems with their 
searches. A larger number of users indi-
cated satisfaction with the computer 
search than dissatisfaction. The three 
good features of the computer search 
most frequently identified were time 
saving, speed, .and comprehensiveness. 
The two problems listed most often 
were lack of adequate descriptors and 
time lag. 

Users were also requested to indicate 
their willingness to pay. Sixty-eight per-
cent of the users were willing to pay 
from $5 to $10 per search, and 32 per-
cent stated their willingness to pay more 
than $10 per search. Some were willing 
to pay as much as $60. Thus, the ma-
jority were quite willing to pay within 
the present price range, and approxi-
mately one-third were willing to pay 
from $10 to $60. 

SuMMARY OF USER STUDY 

A profile of the most typical PROBE 
user, determined from the question-
naires, showed that he was a male grad-
uate student, between the ages of twen-
ty-six and thirty-five, working on re-
search at Indiana University (Blooming-
ton Campus). Other larger user cate-
gories were college faculty members, 
twenty, or 14.2 percent; undergraduate 
students, sixteen, or 11.3 percent; and 
administrators, fifteen, or 10.6 percent. 
Overall satisfaction of the users proved 
to be high since 79.5 percent of the 
searches were considered of value. 

Because of the user study, the follow-
ing changes were effected in the retrieval 
program: the time lag was shortened, 



the price per search was raised slightly 
(from $4 to ·$6), more computer fields 
were used to allow flexibility, and the 
queries were structured with more speci-
ficity. 

The number of searches requested 
has remained stable, the price increase 
not having altered the demand for such 
a service. The improvements effected in 

User Study I 463 

the program following the evaluation 
have kept it a popular retrieval system 
in Indiana and the number of out-of-
state requests has increased steadily. 
More detailed information regarding 
the study may be obtained from the au-
thor's complete description of the evalu-
ation.10 

REFERENCES 

1. W. L. Hull and Gregory Benson, Jr., In-
stalling a Coordinated Information Net-
work in a State Education Agency: A Case 
Study of the Decision Process in New York. 
Final Report (Albany, N.Y.: New York 
State Education Dept., Feb. 1972), p .15. 
(ERIC, ED 061 428). 

2. R. R. Tschudi and J. C. Meredith, The 
"PROBE" Retrieval Program-A Descrip-
tion (Bloomington: Indiana Univ., Feb. 
1972). (ERIC, ED 059 596). 

3. Eva L. Kiewitt, PROBE Computer Search 
of the ERICtapes (Bloomington, Ind.: In-
diana Univ., May 1973). (ERIC, ED 070 
050). 

4. C. W. Cleverdon, Factors Determining the 
Performance of Indexing Systems ( Cran-
field, England: College of Aeronautics, 
1966 ). 

5. D. R. Swanson, "The Evidence Underlying 

the Cranfield Results," Library Quarterly 
35:1-20 (Jan. 1965). 

6. C. W. Cleverdon, "Design and Evaluation 
of Information Systems," Annual Review 
of Information Science and Technology 
6:68 ( 1971). 

7. R. W. Coover, "User Needs and Their Ef-
fect on Information Center Administration: 
A Review, 1953/66-," Special Libraries 60: 
446-4 7 ( Sept. 1969 ) . 

8. F. W. Lancaster, Evaluation of the MEV-
LARS Demand Search Service (Washing-
ton, D .C.: National Library of Medicine, 
Jan. 1968). (ERIC, ED 022 494). 

9. Thesaurus of ERIC Descriptors (New 
York: CCM Information Corp., 1972). 

10. Eva L. Kiewitt, "PROBE: Computer 
Searches of the ERICT APES-An Evalua-
tion of a Pilot Study" (Ph.D. dissertation, 
Indiana Univ., 1973).