june05b.indd


Harvey Varnet and Martha Rice Sanders 

Asking better questions 
Small-scale assessment measures that inform ongoing work 

The
late
Dick
Palmer,
professor
of
library
and
 information
 science
 at
 Simmons

College,
loved
words.
He
had
served
in
the

navy
as
an
intelligence
offi
cer,
code­breaking

Japanese
during
World
War
II.
Palmer
loved

to
challenge
his
classes
to
define
the
word

knowledge.
His
favorite
definition
was
“that

information
 which
 allows
 us
 to
 ask
 better

questions.”
Palmer
was
a
pioneer
in
informa­
tion
science
and,
though
he
didn’t
know
it
at

the
time,
he
was
in
front
of
the
assessment

movement
 in
 higher
 education.
 He
 always

asked
“What
for?”
If
you
didn’t
have
an
an­
swer,
he
quickly
followed
that
remark
with

“Why?”
 The
 approach
 was
 gentle
 but
 fi
rm

and
purposeful.


In
small­
and
medium­sized
colleges
and

universities,
undertaking
library
assessment
is

definitely
a
challenge.
First,
there’s
fi
guring

out
who
has
the
time
or
on­staff
expertise
to

do
this
in
a
meaningful
way;
getting
staff
in­
terested
in
this
topic
is
not
always
easy.
Then

there’s
 the
 complexity
 of
 linking
 anything

you
do
in
the
library
to
the
overall
campus

assessment
 program
 (if
 indeed
 one
 is
 fully

formulated).


In
the
northeast,
NEASC
(the
New
England

Association
of
Schools
and
Colleges)
has
been

the
prime
driver
in
the
assessment
movement.

On
 the
 NEASC
 Web
 site,1
 the
 Commission

on
Institutions
of
Higher
Education
outlines

standards
and
other
policies,
including
an
im­
portant
“Policy
on
Institutional
Effectiveness.”

NEASC
has
attempted
to
assist
accreditation

site
visitors
by
formulating
what
it
calls
“pilot

site
 visit
 questions.”2
 While
 the
 questions

relating
to
Standard
7,
Library
and
Informa­
tion
Resources,
are
helpful,
they
are
limited


in
 scope.
 One
 area
 these
 questions
 do
 ask

site
visitors
to
consider
is
the
development

of
 information
 literacy
 skills—meaning,
 for

most
of
us,
bibliographic
instruction
or
some

library
and/or
institutional
attempt
to
foster

such
skills.
This
is
an
area
where
many
of
our

colleagues
do
an
excellent
job
of
measuring

their
effectiveness.
Otherwise,
what
to
do?


The culture of evidence 
One
less
intimidating
way
to
look
at
assess­
ment
is
to
see
our
efforts
as
part
of
the
great­
er
culture of evidence.
Peggy
Maki,
a
noted

authority
on
assessment
and
the
director
of

assessment
at
the
American
Association
for

Higher
Education,
talks
about
support
units

and
how
outcomes
assessment,
while
help­
ful,
may
not
always
fit
our
operations.
How­
ever,
libraries
perform
lots
of
services,
with

accompanying
 data/statistics,
 that
 can
 in­
form
our
decision
making.
In
Maki’s
words,

aggregating
the
data,
analyzing
it,
and
then

doing
something
with
it—asking
better
ques­
tions—is
an
important
aspect
in
helping
to

create
the
culture
of
evidence
that
what
we

are
doing
is
meaningful
and
worthwhile.


Small-scale measures 
At
Providence
College,
we’ve
tried
to
adopt

techniques
that
tell
us
something
about
our­
selves.
On
an
annual
basis,
we
look
at
inter­
library
loan
(ILL)
fill­rate.
These
data
are
easy


Harvey Varnet is library director, e-mail: hvarnet@ 
providence.edu, and Martha Rice Sanders is coordinator 
of collections services and catalog librarian, e-mail: 
sandersm@providence.edu, at Providence College in 
Rhode Island 
© 2005 Harvey Varnet and Martha Rice Sanders 

June 2005  461 C&RL News 

mailto:sandersm@providence.edu
http:providence.edu


to
gather,
as
they
are
primarily
OCLC­based.

The
 data
 tell
 us
 that
 we
 are
 very
 effi
cient

at
 getting
 requests
 from
 students
 and
 fac­
ulty
out
to
other
colleges—in
less
than
one

day,
 typically.
 The
 chart
 below
 also
 shows

us
 how,
 over
 time,
 the
 fill
 rates
 for
 copies

(mostly
 journal
 articles)
 and
 monographs

are
 remarkably
 consistent.
 This
 allows
 our

service
staff
to
tell
students
and
faculty,
with

some
 confidence,
 what
 they
 can
 expect
 if

and
 when
 we
 don’t
 have
 an
 item
 and
 it
 is

not
 available
 nearby
 in
 a
 sister
 consortium

library.


Interlibrary Loan Fill Rates—Borrowing

When
 considering
 budgeting
 challenges,

we
occasionally
look
at
book
use
rates.
Our

process
 is
 to
 review
 whether
 the
 books
 we

acquired
 two
 fiscal
 years
 ago
 circulated
 or

not.
 This
 tells
 us
 in
 some
 measure
 that
 we

are
buying
the
books
that
are
needed
to
sup­
port
 the
 curriculum.
 With
 a
 campus
 that
 is

not
 research
 driven
 but
 research
 interested,

we
surely
acquire
books
for
their
long­term

value.
 However,
 our
 mission
 is
 to
 provide

an
 undergraduate,
 liberal
 arts
 education,
 so

our
purchasing
decisions
are
made
with
this

in
 mind.
 One
 might
 ask,
 what
 is
 “success”

when
 looking
 at
 use
 rates.
 One
 interesting

way
to
look
at
this
is
via
Richard
Trueswell’s3


seminal
work
years
ago
that
outlined
the
80/20

rule—over
 time,
 80
 percent
 of
 circulation

tends
to
come
from
20
percent
of
a
collection.

Our
fi
scal
year
2002
acquisitions
circulated
as

indicated
 below—not
 bad,
 considering
 the

compressed
time
period
of
analysis
and
the


Book Circulation—FY 2002 Purchases 

0
 =
3,261


1­5
=
2,875


5+
=
56



Total
acquisitions
=
6,192



oft­reported
general
downward
trend
in
book

circulation.


Another
small­scale
approach
we’ve
taken

is
 to
 ask
 faculty
 for
 the
 bibliographies
 of

papers
 or
 presentations
 that
 their
 seminar

students
 take
 or
 from
 major
 departmental

courses.
We’ve
had
limited
success
at
getting

these
 bibliographies,
 but
 each
 set
 we
 have

looked
at
has
proven
useful.
Our
goal
in
the

review
of
the
bibliographies
is
to
determine

if,
within
the
library
or
our
consortium,
we

can
provide
80
percent
of
the
materials
the

students
 needed.
 This
 approach
 also
 deals

in
reality—we
look
at
actual
resources
used,

versus
 the
 students’
 speculation
 that
 “they

don’t
have
anything
I
need
to
do
my
course

work.”


Our
 analysis
 of
 bibliographies
 in
 social

work
benefited
both
areas.
The
perception

that
we
had
very
few
journal
subscriptions

was
incorrect
(we
were
able
to
prove
this)

and
this
helped
change
the
tone
of
our
dis­
cussions
with
the
department.
We
reviewed

the
senior
seminar
papers
for
two
years
and

discovered
that
we
were
able
to
supply
62.5

percent
of
the
citations
in
2001
and
87
percent

in
2002.
When
the
department
came
up
for
its

professional
accreditation,
we
were
an
active

and
welcomed
partner
in
the
process,
and
the

library
was
cited
as
a
program
strength.


In
music,
we
were
able
to
review
the
bib­
liographies
for
two
courses.
In
each
course,

we
held
more
than
90
percent
of
the
books

used
and
100
percent
of
the
journals
cited.

This
data
helped
dispel
the
perception
among

the
music
department
faculty
that
the
library

held
none
of
the
titles
needed
by
students
for

doing
their
research.
In
addition,
the
library

purchased
one
book
for
the
collection
that

was
accessed
by
a
student
through
ILL.


In
chemistry,
analysis
of
student
seminar

citations
revealed
that
we
held
or
provided


462C&RL News June 2005



access
 to
 77
 percent
 of
 the
 journals
 cited.

For
books,
46
percent
were
available
in
the

consortium,
though
we
held
none
used
by

students.
This
lead
to
an
interesting
dialogue

with
 the
 chair
 of
 the
 department.
 After
 re­
viewing
 what
 we
 did
 not
 hold,
 the
 chair

indicated
that
we
ought
not
to
buy
the
works

cited
 because,
 though
 useful
 for
 students’

papers,
 the
 books
 were
 not
 a
 particularly

good
fit
for
our
collection.


Larger-scale measures 
Another
 aspect
 of
 operational
 analysis
 is

benefiting
from
the
overall
assessment
work

that
the
college
undertakes.
Similar
to
many

institutions,
Providence
College
administers

the
 Student
 Satisfaction
 Inventory
 (SSI)—in

our
case,
to
freshmen
and
seniors.


The
 Noel­Levitz4
 SSI
 has
 two
 library­re­
lated
questions:
Library
staff
are
helpful
and

approachable?
 and
 Library
 resources
 and

services
are
adequate?
While
these
questions

may
not
be
ideally
stated
(for
many
of
us,
this

is
actually
four
and
not
two
questions),
the

results
give
us
a
look
at
how
our
primary
cli­
entele
generally
perceive
us.
The
SSI
uses
the

gap
analysis
technique
for
assessment,
with

the
“gap”
being
the
difference
between
stu­
dents’
ratings
of
importance
and
their
ratings

of
satisfaction,
each
on
a
1
to
7
point
scale
(7

is
best).
We
have
data
from
three
such
SSI
sur­
veys
(1997,
2000,
2003).
In
2003,
for
the
staff 
question,
students
rated
us
well,
with
a
gap
of

0.26.
For
the
resources and services
question,

students
weren’t
quite
as
satisfi
ed—our
gap

is
1.02.
In
both
cases
our
scores
are
slightly

ahead
 of
 our
 national
 peers.
 And,
 in
 both

cases,
the
gap
has
decreased
from
the
previ­
ous
assessment,
a
trend
in
the
right
direction.

Our
scores
for
the
staff
question
also
rank
us

as
one
of
the
highest­rated
service/support

areas
on
campus.


Because
 the
 college
 has
 an
 ongoing
 as­
sessment
program,
we
received
institutional

support
in
2003
to
participate
in
the
LibQual+

survey.5
This
nationally
normed
survey
was

developed
at
Texas
A
&
M
University
in
co­
operation
 with
 the
 Association
 of
 Research

Libraries.
 This
 survey
 measures
 four
 areas:


affect
of
service,
access
to
information,
per­
sonal
control,
and
library
as
place.


Our
 library
 was
 built
 in
 1969,
 and
 we

knew
it
was
badly
in
need
of
updating
and

renovating.
What
we
learned
from
LibQual+

was
 that
 students
 felt
 strongly
 about
 the

library
as
place,
in
our
case
in
the
negative.

The
radar
chart
produced
in
our
results
sum­
mary
is
very
red—not
a
good
color
to
see,

but
a
clear
message
to
the
library
and
college

administration.


We
 began
 a
 long­term
 renovation
 pro­
cess
in
the
summer
of
2003,
completing
the

library’s
 main
 floor
 in
 the
 spring
 of
 2004.

Our
 plan
 is
 to
 participate
 in
 Libqual+
 next

year,
 again
 surveying
 sophomores
 and
 ju­
niors
to
determine
if
their
perceptions
have

changed.


As
 the
 library
 improved
 physically,
 it

seemed
 to
 us
 that
 students
 were
 coming

in
to
use
the
library
in
larger
numbers.
We

keep
evening
attendance
headcounts,
so
we

decided
to
compare
the
current
(renovation)

year
to
the
previous
year’s
counts.
The
chart

below
 shows
 how
 attendance
 was
 up
 for

each
week
analyzed.
The
counts
for
each
year

were
taken
during
the
week
before
Thanks­
giving,
in
the
first
two
weeks
of
December,

and
for
all
of
January.


We
reported
this
information
to
the
college

administration,
and
they
in
turn
reported
it

to
 the
 Board
 of
 Trustees
 at
 its
 late
 January

meeting.
There
was
general
recognition
that

the
renovations
were
well
worth
the
expense

involved,
and
the
Board
indicated
its
support

for
the
next
phase
of
renovations.


The future 
We
 plan
 to
 continue
 with
 a
 mix
 of
 assess­

June 2005  463 C&RL News 



ment
activities.
We
are
now
planning
to
look

much
 more
 closely
 at
 our
 instruction
 pro­
gram.
We
teach
between
50
and
70
classes

per
year,
and
we
need
to
know
more
from

the
 students
 as
 to
 whether
 the
 information

they
receive
has
utility
and
value.


We
 are
 also
 interested
 in
 some
 general

assessment
of
the
high­profile
activity
of
ref­
erence
desk
service.
We’re
working
with
our

campus
assessment
coordinator
on
a
point­
of­use
survey
for
reference,
hoping
that
we’ll

get
 useful
 feedback
 from
 users.
 While
 we

get
 anecdotal
 information
 about
 reference

assistance,
this
information
tends
to
be
at
the

extremes—either
someone’s
very
pleased
or

quite
the
opposite.
We’re
hopeful
that
more

user­response
data
will
give
us
a
balanced
pic­
ture
of
this
primary
and
important
service.


We
will
continue
to
analyze
and
review

interlibrary
loan
data
and
book
use
rates.
We

also
 plan
 to
 review
 course­related
 bibliog­
raphies
from
those
faculty
willing
to
share.

This
work
has
proven
to
be
very
benefi
cial

and
has
qualitatively
changed
the
discussions

we
have
regarding
library
services
with
our

teaching
 colleagues.
 In
 our
 next
 review
 of

bibliographies,
we’ll
attempt
to
see
if
there

is
any
correlation
between
the
grade
students

earn
on
their
papers
and
their
use
of
refereed,

professional
sources.


The
 college
 will
 continue
 to
 do
 the
 SSI

survey
on
a
regular
basis,
and
we’ll
extract

that
data
to
see
how
students
are
perceiving

us
 in
 general.
 And,
 finally,
 we
 are
 hopeful

for
institutional
support
this
coming
year
to

do
another
LibQual+
survey.
The
cost
for
this

activity
is
quite
different
from
all
our
other

assessment
efforts,
but
the
return
on
invest­
ment
is
well
worth
the
effort.


In
 its
 “Standards
 Revision
 Discussion

Paper,”
 NEASC
 notes:
 “Evidence
 is
 funda­
mental
to
institutional
decision­making
.
.
.

Increasingly,
 as
 will
 be
 reflected
 in
 the
 re­
vised
Standards,
the
Commission
expects
that

institutions
 gather
 and
 use
 evidence
 as
 the

basis
for
this
evaluation
and
improvement.”6


At
Providence
College,
we
in
the
library
are

attempting
 to
 take
 this
 evidence­based
 ap­
proach
seriously.
We’re
confident
our
“mixed”


approach
to
analysis
is
on
track.
The
blend

of
analytical
activities
we’ve
undertaken
has

helped
to
make
us
a
better
campus
service.

This
mixed
approach
is
also
on
a
scale
such

that
the
evaluation
work
doesn’t
overwhelm

any
other
ongoing
activity.


To
conclude
with
more
wisdom
from
Dick

Palmer,
 “Just
 count—it’s
 amazing
 what
 the

numbers
tell
you
about
your
prejudices.”


Providence College profi le 
Providence
 College
 (www.providence.edu)

is
 a
 primarily
 undergraduate,
 liberal
 arts,

Catholic
institution
that
is
committed
to
fos­
tering
academic
excellence
through
the
sci­
ences
 and
 humanities.
 The
 college
 enrolls

approximately
4,200
FTE
undergraduate
stu­
dents
and
330
FTE
graduate
students.
There

are
250
faculty
offering
degree
programs
in

more
than
40
areas.


Notes 
1.
 New
 England
 Association
 of
 Schools


and
Colleges
Web
site:
www.neasc.org/cihe

/assessment.htm.


2.
NEASC
Pilot
Questions
Web
site:
www.

neasc.org/cihe/site_visit_questions.PDF.


3.
Those
unfamiliar
with
Dr.
Trueswell’s

work
should
review
one
or
more
of
the
fol­
lowing
 citations:
 Richard
 Trueswell,
 “Some

behavioral
 patterns
 of
 library
 users:
 The

80/20
 rule,”
 Wilson Library Bulletin,
 1969;

43:
458–461.
Richard
Trueswell,
“User
circula­
tion
satisfaction
vs.
size
of
holdings
at
three

academic
 libraries,”
 College and Research 
Libraries (May
1969):
30(3):
204–13.
Richard

Trueswell,
“Determining
the
optimal
number

of
 volumes
 for
 a
 library’s
 core
 collection,”

Libri
 (International
 Library
 Review)
 1976;

19(1):
49–60.


4.
The
corporate
Web
site
for
Noel­Levitz,

Inc.
 can
 be
 found
 at
 www.noellevitz.com

/solutions/retention/satisfaction/index.

asp.
 To
 review
 gap
 analysis
 use,
 see
 “The

Measurement
 of
 Gaps
 in
 Service
 Quality”

by
Robert
Lewis
and
David
Klein
(pages
33

–
38)
in The Service Challenge: Integrating for 
Competitive Advantage (Chicago,
 American

Marketing
Association,
1987).


464C&RL News June 2005

http:www.noellevitz.com
www.neasc.org/cihe
http:www.providence.edu


5.
 The
 underpinnings
 of
 LibQual+
 are

based
on
the
pioneering
“servqual”
work
of

A.
Parasuman
and
his
colleagues.
See
Valarie

A.
Zeithaml,
A.
Parasuraman,
and
Leonard
L.

Berry,
Delivering quality service: Balancing 
customer perceptions and expectations
(New

York,
Free
Press,
1990).
For
Parasuman
et
al.,

the
only
opinion
that
matters
is
the
customers


—all
others
are
irrelevant.
The
Libqual+
Web

site
can
be
found
at
www.libqual.org


6 . 
 N E A S C ’ s 
 “ S t a n d a r d s 
 R e v i ­
sion
 Discussion
 Paper”
 (issued
 in
 De­
cember
 2003)
 can
 be
 found
 at
 www.

neasc.org/cihe/revisions/summary_of

_significant_changes.pdf.
The
quote
from
the

paper
is
on
page
three,
second
bullet.


(“Internet Resources” continued from page 457) 

and
recovery,
and
grief. Access:
http://www.

helphorizons.com/.


Organizations 
•
 American Counseling Association. 

The
 flagship
 professional
 organization
 has

an
impressively
supportive
Web
site,
which

includes
the
online
periodical
Counseling To­
day,
the
ACA
Code
of
Ethics,
crisis
fact
sheets,


and
 assistance
 in

choosing
a
gradu­
ate
program.
Sepa­
rate
 headings
 for

students,
consum­

ers,
and
counselors
guide
users
to
the
proper

area.
Access:
http://www.counseling.org.


•
American Psychological Association. 
This
useful
site
includes
the
APA
Help
Center

with
articles
and
brochures
on
psychological

issues;
quick
links
for
students,
including
the

ever­popular
 APA
 style
 helper;
 and
 online

publications
 consisting
 of
 the
 Monitor on 
Psychology
and
the
 Mental Health Patients’ 
Bill of Rights.
The
Online
Testing
and
Assess­
ment
section
is
essential
for
counselors
and

students.
Access:
http://www.apa.org.


•
National Board of Certifi ed Counsel­
ors. This
 organization
 provides
 standards,

certification,
and
a
Code
of
Ethics
for
coun­
selors.
The
section
on
Internet
counseling
is

valuable
as
is
the
state
credentialing
boards

list.
Access: http://www.nbcc.org/.


E-journals 
•
 Internet Journal of Mental Health. 

This
peer­reviewed
journal
contains
articles

on
topics
ranging
from
HIV
patient
counsel­

ing
to
the
psychological
implications
of
neck

cancer.
Access:
http://www.ispub.com/ostia/

index.php?xmlFilePath=journals/ijmh/front.

xml.


•
Journal of Technology in Counseling. 
This
online
journal
specializes
in
combining

technology
 with
 counseling
 to
 produce
 ar­
ticles
on
such
topics
as
digital
portfolios
and

distance
 learning.
 Access:
 http://jtc.colstate.

edu/.


Note 
1.
 Merriam­Webster’s Online Diction­

ary,
 retrieved
 April
 28,
 2005,
 from
 www.

britannica.com/dictionary.


June 2005  465 C&RL News 

http://jtc.colstate
http://www.ispub.com/ostia
http:http://www.nbcc.org
http:http://www.apa.org
http:http://www.counseling.org
http:helphorizons.com
http://www
http:www.libqual.org