Richardson Understanding the Reference Transaction 211 Understanding the Reference Transaction: A Systems Analysis Perspective John V. Richardson Jr. The reference transaction can be understood as a system and mod­ eled as such. By adopting a problem-solving paradigm other than the one traditionally used in this area (i.e., flowcharting), a systems analy­ sis of the question-answering process can yield new insights. In par­ ticular, this article identifies fifteen functional requirements of the pro­ cess as well as fit criteria to measure success in question negotiation. In addition, the process is graphically illustrated using four diagrams of the reference transaction and provides a top-level perspective of the process. n this article, the author pro- poses that the process of ques- tion answering, technically called the reference transaction in the field of library and information sci- ence, be defined as a face-to-(inter)face process involving an inquirer and a librar- ian, or surrogate, within an information- seeking environment (i.e., a library-like setting). Although the process can readily be defined as such, it has not been so clearly conceptualized or successfully modeled by reference practitioners or re- searchers. Witness the thirty years of ref- erence evaluation studies that suggest a low level of reference accuracy (about 50%), which could be attributed to a poor conceptualization of the question-answer- ing process. Furthermore, several differ- ent graphical representations have been proposed for understanding this area, but without any explicit justifications or ex- planations of their inherent biases. Hence, there is a strong need for a theoretical resolution using new knowledge so that reference practitioners and researchers can possess an accepted visualization of the reference process. Without a doubt, the topic, with a long history since its introduction in 1876 by Samuel S. Green, is an important one to understand.1 For example, the several thousand new practitioners who enter the profession each year need to be able to visualize this process accurately, if they are to engage in it successfully. Having such a model would be a useful teaching aid for instructors. Furthermore, the in- quirers who ask more than 295 million reference questions in U.S. libraries2 each John Richardson, the 1996/97 OCLC Visiting Distinguished Scholar, is a professor in the Graduate School of Education and Information Studies at UCLA; e-mail: jrichard@ucla.edu. 211 mailto:jrichard@ucla.edu 212 College & Research Libraries May 1999 year would benefit directly by having their questions answered more accurately and more efficiently by reference librarians who clearly understand the process. A model could make the question-answer- ing process more systematic. This inves- tigation offers insight into other question- and-answer settings such as help desk situ- ations, counseling, or interviewing more generally. In summary, one could argue that the profession faces a serious situa- tion because the system requirements have never been codified and diffused as such. Background Starting in the early to mid-1960s, the flow- charting model (a.k.a. block diagram or logic diagram) of the reference transac- tion became the predominant approach to understanding the process. This shift occurred because of the pioneering work of at least six people: Allan M. Rees and Tefko Saracevic, Robert Hayes and Gary Carlson, Jesse Shera, F. S. Stych, and Charles Bunge.3 To see graphically the culmination of the flowcharting ap- proach, look at Knowledge-Based Systems for General Reference Work.4 The advantage of the systems analysis perspective is that it provides a top-level view of the process. Today, this model still influences the way we teach the process to novice prac- titioners. Hence, it is important to exam- ine its implications. Coming out of busi- ness data processing, flowcharting merges data and process and graphically represents functions over time.5 Depend- ing on the level of granularity, the flow- charting approach accentuates the linear- ity of the system, its discrete elements or components, and the procedural detail of the process. Furthermore, the flowchart approach examines the transaction itself as a closed system that exists indepen- dently of all other transactions. In other words, it is a good technique for seeking the individual trees, but not the forest. By the 1960s, some people recognized the necessity of distinguishing between data and process.6 Looking for these other approaches, the author adopted an alternative prob- lem-solving paradigm—systems analy- sis. It is different in its assumptions and constitutes a new and unique way (there appears to be no published lit- erature on the reference transaction that uses this technique) to think about t h e p r o c e s s . F o r e x a m p l e , s y s t e m s analysis encourages one to identify ex- plicitly the system’s users, goals, and requirements in contrast to process. Al- though the reference literature men- tions such things in passing, no one has brought together in one place a com- plete list of the requirements as stated below. In addition, it encourages the graphic representation of data flows (see the four diagrams) and begins to suggest how the input is transformed into an output. Thus, it is a unique way of representing the problem and leads to new insights as mentioned below. The advantage of the systems analysis perspective is that it provides a top-level view of the process. In that sense, it re- veals the forest rather than the trees, so that one can see how the subsystems in- teract. Although the diagrams may seem rather intuitively obvious, they are origi- nal because the previous work has always approached the process in a piecemeal fashion. For instance, the system’s log re- quirement (see R 2.10 below) encourages continuous quality improvement that could not have happened so easily be- fore. Rather than merely making a tick mark that a question was attempted, the librarian can check to see what percent- age of questions are unanswered due to inadequate resources. Administrators can examine the economic implications and decide to allocate reference funds based on data. Theoretical Framework The problem as identified above can be described further as the interaction be- tween what exists (i.e., the system of ques- Understanding the Reference Transaction 213 tion answering or the reference transac- tion as a process) and what is unknown (i.e., the system requirements and vari- ous models including context diagrams, data flow diagrams, and flowcharts). In this article, the author treats the reference process (a.k.a. the reference interview and question negotiation) as a system fol- lowing “the definition of a system as (1) something consisting of a set (finite or infinite) of entities (2) among which a set of relations is specified, so that (3) deduc- tions are possible from some relations to others or from the relations among the entities to the behavior or the history of the system.”7 In systems analysis, the overall strategy in dealing with a large system is to partition it into a number of smaller, manageable pieces. In this case, the individual pieces are diagrammati- cally represented as (1) data flow dia- grams and events as well as (2) a flow- chart (see results section below). Furthermore, the author assumes that the reference process is: (1) goal directed in that inquirer and librarian are both en- gaged in solving or resolving8 an infor- mation problem, and (2) well bounded in that the problem-solution boundaries (i.e., the number of reference questions asked and reference questions answered) are finite.9 In the case of this article, en- larging our circle of knowledge about the system requirements (i.e., into the dimen- sion of the unknown mentioned above) would be aided by the existence of a use- ful model of the reference transaction. Next, this article suggests that a useful theoretical bridge between the two states is that of systems analysis including its at- tendant concepts such as goals, events, re- quirements, and criteria of fit (defined be- low). A general problem-solving tech- nique, systems analysis is the “craft of un- derstanding and specifying systems by building models of them”10 and contains techniques such as structured systems analysis and design. Some succinct defini- tions follow. An event is a response to an outside (i.e., of the system) stimulus, such as the inquirer asking a question. Fur- thermore, there are two types of events: external to the system and temporal. For example, the inquirer asking a question is external to the question-answering pro- cess. Temporal events are those that are time dependent, such as following up on user satisfaction. When used within the context of systems analysis, the word re- quirement means the real (i.e., essential or logical) activity of the system without re- gard to its actual physical implementation. By contrast, a nonfunctional requirement is any property of the system such as its performance, usability, or legality. Goals and Objectives The ultimate goals of this article are to present a model of the reference process derived from a formal systems analysis and thereby to improve the field’s un- derstanding of this important transac- tional system. More specifically, this ar- ticle (1) identifies the reference process’s goal, functional as well as nonfunctional requirements, related system events, and fit criteria; (2) posits several competing graphical models of the reference process including context diagrams and flow- charts; and (3) proposes tests for their goodness of fit. Research Questions and Hypothesis The key question is why the system does not work better than it does, especially in light of 50 percent accuracy findings. Is there a deficiency in our understanding of (1) who its users are; (2) what its goals, requirements, and fit criteria are; and (3) what models might be usefully employed? Answering these questions will improve as well as advance our understanding of the reference transaction because they are fundamental in terms of the system’s na- ture and its boundaries. The explicit hy- pothesis is that the user requirements fall under three system responses: query ne- gotiation, query resolution, and satisfac- tion assessment. Methodological Considerations Herein, the explicit assumption is that systems analysis is an appropriate tech- nique for understanding the reference 214 College & Research Libraries May 1999 transaction. To ensure the reliability and validity of the results reported below, this analytic-deductive article is based on data gathered from: (1) knowledge derived from an academic library apprenticeship (i.e., the author worked as a reference librarian at the University of Kentucky for several years); (2) intensive obtru- sive observations of reference service in a public setting (i.e., the Santa Monica (California) Public Library in 1990) and extensive unobtrusive observations of reference service in a research library set- ting (i.e., UCLA’s Young Research Li- brary); and (3) review of the research literature as well as introductory refer- ence textbooks11 and of professional state- ments regarding normative standards of practice. In this sense, then, the research reported below is based on what is done in practice12 as well as what is said (the reported research on the reference pro- cess).13 Finally, the findings draw heavily upon the analytical techniques of systems design and analysis.14 Results Following the common process of analy- sis, this section discusses (1) the users in order to create (2) a context diagram, fol- lowed by (3) an explicit statement of the goal and (4) system requirements as well as a set of (5) event diagrams. System Users The two primary users of the question- answering system are inquirers and librar- ians. Inquirers can be characterized using a variety of attributes such as age, appear- ance, college preference, living density (e.g., county size and rank), gender, edu- cation, educational plans, employment sta- tus, ethnicity, family size, geographical mo- bility, income, language ability, marital status, occupation, reading skill, and place of residency (including urban versus ru- ral). A common approach is to set a default stereotype model of the inquirer consist- ing of a set of “clusters of characteristics.”15 New librarians, novices, and advanced be- ginners constitute the other primary user FIGURE 1 Question Answering Context Diagram Note: IQ = Initial Question AS = Answer with Satisfaction Query QN = Question Negotiation RQ = Real Question FS = Follow-up long-term satisfaction Inquirer Answer Question Refer Request Data ILL/ Colleague/ Collection Reference Statistics Log Resources: Reference Collection, WorldCat, First Search/EPIC Tick IQ RQ QN FS http:analysis.14 http:cess).13 Understanding the Reference Transaction 215 FIGURE 2 Event 1. Inquirer Initiates Question group. The identified attributes of librarians are personality traits, education, specific course work, and subject expertise. A second- ary set of customers is reference department heads. System Goal As mentioned earlier, the system’s goal is to resolve or solve the inquirer ’s information problem INQUIRER UNDERSTAND QUESTION QUESTION LOG RECORD QUESTION INITIAL QUESTION CLARIFYING QUESTION by providing an answer to his or her question, the sine qua non of all reference service in libraries. Stated negatively, it is not supposed to make the user go away uninformed or unsatis- fied. Figure 1 presents a graphic abstraction of the question-answering system context. As a model, its purpose is to aid in under- standing the process and its boundaries. Data flows into the process from two sources: the inquirer and the reference col- lection. The process terminates with a re- turn of information to the user or a refer- ral to another agent and a record in the reference log. Requirements, Events, and Fit Criteria Requirements are a set of user specifica- tions indicating what the system is sup- posed to do.16 They tell someone (usually the system analyst or designer) what needs to be done rather than how to do it. Often requirements are divided into two categories: those that are functional (i.e., a must-do list) and those that are non- functional (i.e., system constraints or prop- erties). Unfortunately, many, if not most, reference departments do not have writ- ten mission statements;17 if they did, it would be easier to identify the system re- quirements. Hence, it is unclear whether all reference librarians would subscribe to the following system requirements. As mentioned above, there are two types of events. The first type starts outside the sys- tem; for example, the inquirer initiates a question (see figure 2) or asks the real ques- tion (see figure 3). An example of the sec- ond type of event, temporal, is the need to follow up on the long-term satisfaction of the inquirer (see figure 4). Fit criteria measure the ability to know when the system designer is done; in other words, the newly designed system per- forms as required. In the following text, the requirements are discussed as they re- late to the requirements of event 1 (R1), event 2 (R2), and event 3 (R3). A complete list of events is presented in table 1. Non- functional requirements make up the re- mainder of the results section. Event 1. Inquirer Asks Initiating Question (Query Negotiation) R1.1 Answer All Questions Theoretically, the system must have an “end-product: the information sought by the user.”18 To answer any and every ques- tion includes dangerous topics (e.g., how to build a hydrogen bomb (referred to as the Howard Morland question because Morland was arrested by the FBI for at- tempting to gather data on the topic), ille- gal consequences (e.g., how to grow mari- juana plants or how to pick a lock), politically sensitive topics (e.g., Neo-Nazi literature in Germany or the United States), sexually explicit topics (e.g., how does one download kiddie porn from alt.binaries.pictures.erotica), legal topics (e.g., which tax form does one need or what does the law mean to my case where se- lection or interpretation can be considered practicing law), medical questions (e.g., what is the proper drug dosage or is the disease one has fatal), financial questions (e.g., stock quotes in order to determine what stock to purchase), and existential 216 College & Research Libraries May 1999 INQUIRER ANSWER QUESTION QUESTION NEGOTIATION ANSWER attempting to answer itFIGURE 3 and serves as a narrow-Event 2. Inquirer Asks Real Question ing strategy. Because highly successful librar- ians use open-ended ques- tions, the system could use QUESTION LOG RESOURCES: REAL QUESTION REFERENCE questions phrased as fol-RECORD COLLECTION lows: (1) “What are you DATA trying to do with this in- formation?” or “How are ILL/ you going to use this in- COLLEAGUE/ COLLECTION formation?” or (2) “How REFER REQUEST KNOWLEDGE HUMAN MEMORY/ INTELLECT questions (e.g., what is the meaning of life). Practical requirements may exempt these preceding questions as well as newspaper competitions or crossword puzzle games. Some libraries simply limit the number of questions an individual user can ask per interaction. The reason for stating this re- quirement is that the goal of the system is to fulfill (i.e., solve the information need) or satisfy (i.e., resolve the need) the inquirer ’s information problem, perhaps within some limits. One fit criterion would be to ask the completed system any ques- tion to see if the system can answer it. R1.2 Question Negotiation Must Be Open Ended At the beginning, the system’s (e.g., the librarian’s) initial ques- tion negotiation (i.e., “a process of iterative refor- did this question arise?” or, perhaps best of all, (3) “Give me some context . . . .” In other situations, questions such as these might be used: (4) “Are you looking for material of a particular type (length, language, level of difficulty, etc.)?” or (5) “How much time do you have?” Because public libraries have fewer cues about the inquirer, perhaps the selection-type question (example 4 above) would be more appropriate. R1.3 Record Data Related to Question The system must record items related to the question in a question log, for only in this way can reference work become data driven. A log would enable generated ba- sic statistical information22 to be used in librarian deployment, reference collection development work, determining hours of operation, and any subsequent failure analysis. To determine whether the sys- FIGURE 4 mulation and refinement Event 3. Librarian To Follow Up Satisfaction of the initial question”)19 must be asked as an open- ended, neutral, or selec- tion-type question. Prior research by Geraldine B. King20 and Brenda Dervin and Patricia Dewdney21 reveal that framing ques- tions in that form will re- sult in the maximum of relevant information about the query before SATISFACTION QUERY INQUIRER QUESTION LOG SATISFACTION QUERY RECORD SATISFACTION RESPONSE Understanding the Reference Transaction 217 tem is successful, the log could be exam- ined for date and time, question, tick marks, or other relevant records. Figure 3 graphically represents the ques- tion-answering process after the inquirer asks the initial question. By far, this event is the most complicated because it involves resources, intellectual as well as physical ones. At this stage of the process, there are additional requirements. Event 2. Inquirer Asks the Real Question (Query Resolution) R2.1 Persistence of System The system must be persistent during ques- tion negotiation. This persistence will re- sult in a more complete clarification of the inquirer ’s information need. Moreover, in some cases, this attribute is necessary to answer the real question. All the reference textbook authors have consistently identi- fied this requirement as desirable.23 To test the fit, one might ask: Does the system keep asking questions until the inquirer says “enough”? R2.2 Closed-Ended Confirmation Near the end of question negotiation, a confirming statement about what is meant must be asked as a closed-ended question. Feedback is necessary to identify the negotiated question and also to bring about closure and to move to the next stage of the system. Does the system or librarian say, “If I under- stand you, you are looking for . . .” or ask, “Is this what you meant?” This re- quirement is a type of monitoring.24 R2.3 Determination of Acceptable Answer At the end of question negotiation, the sys- tem must know what is an acceptable an- swer by asking a closed-ended question. As just mentioned, feedback is necessary to identify the inquirer ’s requirements. Does the system ask, “Is this what you need?” or “Is this what you expect?” The former may be more user-friendly be- cause it is stated positively. Rees and Saracevic found certain restrictions on acceptable answers,25 or what Marilyn D. White refers to as “internal constraints.”26 R2.4 Obligatory Data Source The system must have a source of data because its goal is to always cite a source TABLE 1 Event List for Reference Transactions E1. Inquirer asks initiating question (query negotiation) IN: Initial question (IQ) OUT: Clarifying question or statement (CQ) OUT: Record question in log E2. Inquirer asks the real question (query resolution) IN: Real question (RQ) OUT: Clarifying statement (CS) IN: Information resources to answer question (INFO) IN: Human intellect to process data (KNOW) OUT: Answer with satisfaction statement (AS) IN: Satisfaction statement (SS) OUT: If not able to answer question, librarian initiates ILL request to answer inquirer’s question (REQUEST) OUT: Librarian records type of question (RECORD) E3. Librarian to follow up (short- or long-term) satisfaction (assessment) OUT: Follow-up question (FS) IN: Satisfaction response (SR) http:monitoring.24 http:desirable.23 218 College & Research Libraries May 1999 (i.e., make appropriate recommendations based on sources). The fit criterion might be stated as: Does the answer statement come with a print or electronic reference? For example, in a public library setting, How do you spell “cat”? could be an- swered as part of one’s commonsense knowledge—c-a-t. Yet, the question could be about Computer-Aided Tomography or Clear-Air Turbulence, in which case it is capi- talized. R2.5 Memory Requirement The system must have a memory because it may encounter recurring questions. Several of the reference textbook authors have commented on the need for the hu- man librarian to have a good memory27 and S.D. Neill has written about the three different types of memory devices.28 Here, the fit criterion might be whether a log of previously asked questions exists. R2.6 Types of Sources Because the system must respond with an appropriate source, the source of data should: be accurate, be appropriate to age level, be authoritative, have a bibliogra- phy attached, be comparable, be complete, be current,29 be easy to use, be illustrated, be indexed, be reliable, be revised fre- quently, be unique, be well organized, and be in their native-language.30 There is an obvious dependency here because it relates to the accuracy, speed, and sat- isfaction requirements. Following is a multipart test of the fit. When the sys- tem recommends an answer: • Are the sources prepared by knowl- edgeable individuals or institutions and published by known publishers? • Are the materials current? • Are the sources consistent over time? • Do the sources match one’s personal experience? • Are the copyright dates within five years or less? • Does readability match intellectual/ mental age? • Are there a table of contents, headings, bibliography, and index with cross-refer- ences? R2.7 Complete, Accurate Answers The system must provide “complete, ac- curate answers”31 to the inquirer ’s query because this is the fundamental to its goal. What would constitute a possible test of fit? Can the system answer a test set (N = 10 or 20) of typical questions much as we have done over thirty years of human reference accuracy studies?32 Another possibility would be a panel of expert judges reviewing the system’s recom- mendations. R2.8 Timely Answers Ideally, the system must provide timely answers because the inquirer can become impatient. Again, there is a dependency here on satisfaction. In part, user satisfac- tion depends on receiving an answer within an expected time frame. Since the 1920s, speed has been one of the desir- able characteristics of a successful refer- ence librarian.33 Based on our knowledge of reference practice, real questions are answered in two to five minutes—direc- tional questions in less than one minute; instructional (including how to use equip- ment) and research questions in more than five minutes.34 Technically, time per- formance is a nonfunctional requirement. R2.9 System Referrals The system must initiate referral to another individual or institution or interlibrary loan (ILL) if it has exhausted all of its resources (e.g., no source is available in the library to answer question).35 Again, the system’s goal is to answer questions, and there is a de- pendency relationship on satisfaction. A test of fit would be whether a colleague (knowl- edgeable and willing), another collection, or a completed ILL request can match each unanswered question. R2.10 Question Logging The system must log the answer (e.g., the reference source) to the question. One or two reasons for this requirement is that we want to establish the accuracy of an- swer as well as prepare reports for man- agement in order to justify resource allo- cation36 and service staffing. The fit criterion http:question).35 http:minutes.34 http:librarian.33 http:native-language.30 http:devices.28 Understanding the Reference Transaction 219 would be the existence of an accurate and up-to-date log. Event 3. Librarian to Follow Up Satisfaction (Assessment) R3.1 Short-term Customer Satisfaction The system’s answer must fully satisfy the inquirer (based on measures such as relevance and pertinence) in the short term. One of the system’s goals is the ability to review the results and bring about closure; otherwise, the inquirer goes away and may not come back. Hence, there is a strong dependency on satisfaction here. The fit criterion might be whether the system asks, Does this answer your question completely? or Do you have what you want? although there is a risk of a pat question and response. R3.2 Long-term Customer Satisfaction The system must “create high satisfac- tion” in the long term because the user may not know if the answer is really use- ful until it is applied to the information problem (e.g., back home, in the labora- tory, or in the study). Two appropriate tests would be whether the inquirer says yes to “Was the answer useful?” one or two weeks later, and perhaps most im- portant, whether he or she returns with another question.37 With the existence of the data flow diagrams, we no longer have to rely solely on flowcharting as the primary technique of understanding. Nonfunctional Requirements As mentioned earlier, nonfunctional re- quirements are those that operate as constraints on the system. In this ar- ticle, these nonfunctional properties are listed seriatim without any detailed elaboration at this point: • The system must be able to get to an answer. • Time/speed; • Operation. Ideally, the system should be available anytime, day or night. Of course, practically speaking, the historic physical constraint has been twofold: (1) when the library building is open; and (2) when the reference staff is available to answer ques- tions; • The system must accept new sources as well as new editions. • The system must protect inquirer con- fidentiality. • Each and every question issue. The physical log can be forgotten or misrepre- sented (e.g., recording tick marks irre- sponsibly). • The system must perform to standard or normative practice (e.g., it must avoid malpractice and consequential damages). • Each and every question issue; • Of the shelf solutions (see the proto- type review). Implications and Conclusions With the existence of the data flow dia- grams, we no longer have to rely solely on flowcharting as the primary tech- nique of understanding. Of course, we still could use graphic representations of those other dimensions of the refer- ence transaction. The point is that we now a graphic technique of how librar- ians collect enough information to in- crease the chances of a good outcome (i.e., a correct answer and high satis- faction). As is obvious, this new per- spective also shifts the focus onto the dimension of data and time. Further- more, though it may not be completely exhaustive, we now have the most com- prehensive set of requirements to date. Codified and diffused, these can be found in a single convenient place by students in classrooms and by beginning librarians. Inasmuch as we already have second-gen- eration knowledge-based systems for gen- eral reference work,38 we are in a position to build a truly expert system for answer- ing queries. For the Future As mentioned earlier, the fact that this systems analysis perspective is an al- ternative model to the flowcharting ap- proach suggests that there are yet other http:question.37 220 College & Research Libraries May 1999 paradigms to solve the reference trans- a c t i o n p r o b l e m ( e . g . , e m p h a s i z i n g other dimensions in a matrix format). There are at least three other scenarios as well: object-oriented, linear pro- gramming, and parallel processing. Imagine, for instance, an object-ori- ented approach where the domain would be examined for nouns or things. The nouns would be examined for their prop- erties and what they can do. An APL ap- proach uses linear programming tech- niques, such as matrix transformations. It might mean we would first create a matrix of data. In other words, the set of reference questions could be an array or matrix. The matrix of known data (i.e., reference query) could be compared to the solution matrix (i.e., the reference re- sources). Then, an examination of the shared or common dimensions between the two matrices could yield some use- ful insights. Similarly, a massively paral- lel processing approach might mean a nonsequential solution to the reference transaction problem. Here, the universe of all known answers would be pre- sented to a filter until an acceptable an- swer is reached—effective perhaps, but not very efficient. Nonetheless, some- one might profitably examine this ap- proach in more detail. Based on the author ’s review of the published litera- ture, we do not seem to know which is the best paradigm. Thus, should we change paradigms? Does the fact that the reference tools are changing from print to electronic mean anything in terms of the model we employ to understand the process? So, one might really ask whether there is a better model. What are some of these proposed models good for, and what are they not good for? Acknowledgments First and foremost, the author thanks UCLA for his yearlong sabbatical during the 1996/97 academic year as well as OCLC Inc. for offer- ing him the position of Visiting Distinguished Scholar in their Office of Research and Spe- cial Projects. During the development of this systems design and analysis project, the au- thor enjoyed the support and encouragement of many individuals at OCLC including: Dr. Terry R. Noreault, Director, Research and Special Projects; Dr. Keith E. Shafer, Senior Research Scientist, his esteemed consultation monitor as well as Ralph LeVan, Research Sci- entist II; Greg Feldman-Hill, Electronic Pub- lications Division; Karen Nowak, Operations Division; and Larry Olszewski, the OCLC In- formation Center as well as James Robertson, principal of the Atlantic Systems Guild, who helped me to clarify my thinking about the context of question answering and how to model it. As always, Bradley Watson, OCLC Office of Research, helps further clarify the written expression of my ideas. At UCLA, the author wishes to acknowledge Marcia Bates, Robert M. Hayes, Matthew Saxton, and Pei-Ling Wang for their thoughtful read- ing of this manuscript. An earlier version of this paper entitled “Understanding the Ques- tion Answering Process: A Systems Ap- proach,” was presented on 18 March 1997 at the University of North Carolina, Chapel Hill as the 7th Annual Lucile Kelling Henderson Lecture. Notes 1. In 1876, Samuel S. Green articulated “The Desirableness of Establishing Personal Inter- course and Relations between Librarians and Readers,” American Library Journal 1 (Nov. 1876): 74–81. For further information, interested readers might consult John V. Richardson, “Samuel Swett Green, 1837–1918” in the American National Biography vol.9 (New York: Oxford Univ. Pr., 1999), 507. 2. American Library Association, Library Advocacy Now! Quotable Facts about America’s Li- braries (Chicago: ALA Public Information Office, circa 1996), 2. 3. Allan M. Rees and Tefko Saracevic, “Conceptual Analysis of Questions in Information Retrieval Systems,” in Automation and Scientific Communication, Topic 8: Information Storage and Retrieval: Annual Meeting of the American Documentation Institute, Part II, ed. Hans P. Luhn (Washington, D.C.: American Documentation Institute, 1963), 175–77; Robert Hayes and Gary Carlson, Search Strategy by Reference Librarians: Part 3 of the Final Report on the Organization of Large Understanding the Reference Transaction 221 Files, NSF Contract C-280 (Sherman Oaks, Calif.: Hughes Dynamics, Advanced Information Systems, 1964); Jesse Shera, “Automation and the Reference Librarian,” RQ 3 (July 1964): 3–7; F. S. Stych, “Decision Factors in Search Strategy: Teaching Reference Work,” RQ 12 (winter 1972): 143–47; Charles Bunge, “Charting the Reference Query,” RQ 8 (summer 1969): 245–50. 4. See John Richardson, Knowledge-based Systems for General Reference Work: Applications, Problems, and Progress (San Diego, Calif.: Academic Pr., 1995), figure 4.1. 5. Robert M. Hayes has argued that “the process of systems analysis can best be pictured in terms of four dimensions: Data, Functions, Components, and Time” in his UCLA GSLIS 240 “Information Systems Analysis and Design” course (see his winter quarter 1990 syllabus “Sec- tion 4. The Dimensions of Systems Analysis,” p. 30). A complete systems analysis might plot the interaction of all four of his dimensions. For example, following this logic, plotting the components over time would result in a Gantt chart. See Wallace Clark, The Gantt Chart: A Working Tool of Management (New York: Ronald Pr. Co., 1922). 6. Flowcharting, American Standard, 1963. As early as 1910, Louis Brandeis brought to- gether Fred W. Taylor, H. L. Gantt, and C. G. Barth, among others. See “Economies through Scientific Management,” in Evidence Taken by the Interstate Commerce Commission in the Matter of Proposed Advances in Freight Rates by Carriers, August to December 1910, vol. 8, Briefs of Counsel, Docket No. 3400 (61st Congress, 3rd Session, Senate Doc. No. 725; U. S. Serial Set 5911) (Washing- ton, D.C.: GPO, 1911), 4758–4803. 7. Anatol Rapport, “Systems Analysis: General Systems Theory,” in International Encyclope- dia of Social Sciences , ed. David L. Sills (New York: Macmillan and Free Pr., 1968), 453. 8. Problems can be resolved (i.e., a good enough answer), solved (i.e., the best possible answer), or dissolved (i.e., changed), according to Russell L. Ackoff, “The Art and Science of Mess Management,“ Interfaces 11 (Feb. 1981): 20–21. 9. Noam Chomsky argues that the number of sentences is infinite, and so although it is true that a finite number of reference questions are asked, one might wish to pursue the unasked questions of inquirers as well. In his Tractatus Logico-Philosophicus (London, Routledge, and Kegan Paul, 1921), Wittgenstein argued within the context of his system that an askable question was an answerable question (see paragraph 6.5). So, even existential questions such as Why do we exist? might be answerable. 10. Robertson, James, and Suzanne Robertson, Complete Systems Analysis (New York: Dorset House, 1994), 314. 11. A perusal of the introductory chapters in the basic textbooks by James I. Wyer (1930), Margaret Hutchins (1944), Louis Shores (1954), Francis Cheney and Wiley J. Williams (1980), and the latest edition of William Katz (1992) generated the first pass on stating requirements. See also, John V. Richardson Jr., “Teaching General Reference Work: The Complete Paradigm, 1890–1990,” Library Quarterly 62 (Jan. 1992): 55–89. 12. For example, see American Library Association, Information Services for Information Con- sumers: Guidelines for Providers (Chicago: ALA, Reference and Adult Services Division, Stan- dards and Guidelines Committee, 1990); reprinted RQ 30 (winter 1990): 262–65. 13. Richardson, Knowledge-based Systems for General Reference Work, chapter 4. 14. Jerry Fitzgerald, Ardra Fitzgerald, and Warren D. Stallings, Fundamentals of Systems Analysis (New York: Wiley, 1987), 7–8. 15. Elaine A. Rich, “Building and Exploiting User Models” (Ph.D. diss., Carnegie Mellon University, 1979). More recently, this idea has been expanded by Robert B. Allen, “User Models: Theory, Method, and Practice,” International Journal of Man-Machine Studies 32 (May 1990): 511– 43. 16. Alan M. Davis, Software Requirements: Objects, Functions, and States, 2nd ed. (Englewood Cliffs, N.J.: PTR Prentice-Hall, 1993). 17. William A. Katz, Reference and Online Services Handbook: Guidelines, Policies, and Procedures for Libraries (New York: Neal-Schuman, 1982). 18. ALA, Information Services for Information Consumers, section 1.1. 19. Norman J. Crum, “The Librarian—Customer Relationship Dynamics of Filling Re- quests for Information,” Special Libraries 60 (May/June 1969): 269–77. 20. Geraldine B. King, “The Reference Interview: Open and Closed Questions,” RQ 12 (winter 1972): 157–60. 21. Brenda Dervin and Patricia Dewdney, “Neutral Questioning: A New Approach to the Reference Interview,” RQ 25 (summer 1986): 506–13. 22. ALA, Information Services for Information Consumers, section 5.6. 23. Richardson, Knowledge-based Systems for General Reference Work, table 1.4. 24. Marcia Bates, “Information Search Tactics,” Journal of the American Society for Information Science 30 (July 1979): 205–14. 25. Rees and Saracevic, “Conceptual Analysis of Questions in Information Retrieval Sys- 222 College & Research Libraries May 1999 tems,” 175–77. 26. Marilyn D. White, “Evaluation of the Reference Interview,” RQ 24 (fall 1985): 76–84. 27. Richardson, Knowledge-based Systems for General Reference Work, table 1.4. 28. S. D. Neill, “The Reference Process and Certain Types of Memory: Semantic, Episodic, and Schematic,” RQ 23 (summer 1984): 417–23. 29. ALA, Information Services for Information Consumers, sections 2.1 and 2.4. 30. Many of these criteria appear in the incomplete and now outdated Reference Books Bulletin Manual (Chicago: ALA, 1990). Some of the online criteria are well addressed in James Rettig’s October 1996 Online article “Beyond ‘Cool’: Analog Models for Reviewing Digital Resources,” which can be found at http://www.onlineinc.com/onlinemag/SeptOL/rettig9.html. 31. ALA, Information Services for Information Consumers, section 1.3. 32. Matthew Saxton, “Evaluation of Reference Service: A Meta-Analytic Study,” Library Quarterly 67 (July 1997): 267–89. 33. Richardson, Knowledge-based Systems for General Reference Work, table 1.4. 34. Peter Hernon and Charles R. McClure, “Unobtrusive Reference Testing: The 55 Percent Rule,” Library Journal 111 (Apr. 1986): 37–41. 35. ALA, Information Services for Information Consumers, sections 1.11 and 1.13. The factors influencing referral are covered in George S. Hawley, Referral Process in Libraries: Characteriza- tion and an Exploration of Related Factors (Metuchen, N.J.: Scarecrow, 1987). 36. This requirement was suggested as early as the 1940s. See Elizabeth O. Stone, “Methods of Evaluating Reference Service,” Library Journal 67 (Apr. 1942): 296–98. 37. On use and nonuse, see John Lubans, “Nonuse of the Academic Library,” College & Research Libraries 32 (Sept. 1971): 362–67. 38. See, for example, John Richardson, “Question Master: An Evaluation of a Web-based Decision-Support System for Use in Reference Environments,” College & Research Libraries 59 (Jan. 1998): 29–37. http://www.onlineinc.com/onlinemag/SeptOL/rettig9.html