may11b.indd C&RL News May 2011 274 Research shows that interpersonal com-munication between library staff and patrons is just as important to the patron as whether a question was answered cor- rectly.1 But how can you accurately capture and evaluate such interactions? One possible method is mystery shopping. In a mystery shopping exercise, a shop- per poses as a patron and asks a library staff member a question. Immediately following the encounter, the shopper reports on the service provided. The corporate sector has used mystery shopping for years to mea- sure whether employees provide quality customer service and adhere to guidelines and policies.2 Beginning in 2007, we began mystery shopping exercises at our libraries’ reference and front desks at Radford Uni- versity and Longwood University. Secret #1: Lay the groundwork When we first discussed mystery shopping, we realized that we needed to have a clear idea of how we would use our findings and how it would most benefit the libraries and their staff. Did we want to use it as a snapshot to measure current service only to start a cyclical training program, or perhaps a combination of the two? Establishing goals would allow us to measure the success of our exercises. Ultimately, we found it best to work with the immediate desk supervi- sors to identify the goals; interestingly, each one settled on a different desired outcome. Knowing that the mystery shopping exercises could reveal potentially sensitive information, we each designated people other than the desk supervisors to receive and compile the shopping data. We also came to agreement as to who could have ac- cess to the information. At both Radford and Longwood, the desk supervisors wanted re- ports that measured service overall, without identifying individual workers. The resulting reports would be shared library-wide. At Longwood University there was an addi- tional component; individual staff members were shopped, and the supervisor shared the confidential results with each staff member. Before we even mentioned the program with the desk supervisors, the authors dis- cussed with each other how the gathered data could be used to improve service. Mys- tery shopping takes considerable effort to plan and carry out, not just by the research- ers but also by the staff involved. We didn’t want to demoralize everyone by launching a large evaluative process that goes nowhere. Everyone involved in the exercises agreed that each area shopped should have con- crete plans to use the data. At our libraries, each service desk decided to create training programs to address areas identified by the mystery shopping that could be improved. Candice Benjes-Small and Elizabeth Kocevar-Weidinger Secrets to successful mystery shopping A case study Candice Benjes-Small is coordinator of information literacy and outreach at Radford University’s McConnell Library, e-mail: cbsmall@radford.edu, and Elizabeth Kocevar-Weidinger is instruction/reference services librarian/professor Greenwood Library at Longwood University, e-mail: kocevarweidingerea@longwood.edu © 2011 Candice Benjes-Small and Elizabeth Kocevar-Weidinger May 2011 275 C&RL News Secret # 2: Involve all stakeholders Library administration. When we floated the idea of mystery shopping by our library ad- ministrators, the response at both institutions was enthusiastic. Our library administrators were eager for us to provide both quantita- tive and qualitative data about how well we do what we do at our service points. Desk supervisors. Once we got the okay from the higher-ups, it was on to the desk supervisors. At Radford University, the refer- ence and circulation desk supervisors were intrigued. Neither had the time to personally observe all of his or her workers on the desk and loved the idea of getting a better sense of what goes on at the desk. At Longwood University, the desk supervisors were inter- ested in generating staff accountability at service desks and proving what we believed to be true: that our staff did provide quality customer service. Employees being shopped. This was our biggest challenge. All of the groundwork we had done earlier prepared us to answer their questions and assuage their concerns. At the same time, we had to be flexible with our planning. We knew a successful exercise would involve the employees as much as possible. Therefore, staff were involved from the ground up. They developed the custom- er service guidelines by which they would be evaluated. This generated buy-in from the start. Although we put together a frame- work ahead of time, we solicited feedback about each step of the exercise. We made many modifications to the exercise based on feedback from the employees, and, because transparency is imperative, we made sure these modifications were communicated. In our first round of mystery shopping, we also told the employees the beginning and end dates of the exercise. They wouldn’t know when exactly they were being shopped, but they knew the window. This increased their comfort level with our “beta” mystery shop. Over time, once staff were comfortable with the process, we were able to stop giving advance notice, and let them know mystery shopping would happen regularly. Institutional review board and human resources. At both Radford and Longwood, we ran the exercises as research studies and went through formal IRB review. If you are mystery shopping unionized employees, be sure to discuss your plan with the union. Secret #3: Establish model behaviors Quick quiz: how many of you have written guidelines or standards for appropriate cus- tomer service at the reference and circulation desks? We didn’t either. But we all know what proper desk behavior is, right? Indi- vidually, sure. But coming to consensus was surprisingly challenging. Consistent service is imperative. We were going to evaluate everyone using the same shopping instru- ment, so we needed everyone to understand what behaviors are expected. The conversations among the desk employees as to what constitutes model behaviors were fascinating. We set ground rules for open but courteous discussion. We used existing model behavior lists, such as RUSA’s “Guidelines for behavioral perfor- mance of reference and information service providers”3 to generate discussion about generally accepted standards, to guide us in the process, and to help us from re-creating the wheel. Secret #4: Design an effective mystery shopping instrument In both libraries, we had lists of model be- haviors that were far longer than could be measured by one shopper. Our first mystery shopping exercises revealed that if the shop- ping evaluation tool were too long, some of the shoppers would take the form with them to the desk and fill it out right in front of the employee. In the end, we worked with desk supervisors to prioritize the behaviors to be measured and shorten the length of the shopper questionnaire. For each behavior to be measured, we created a scale identifying what is unac- ceptable, acceptable, and optimal for each measured behavior. Example: Behavior to be measured: Were you greeted promptly? C&RL News May 2011 276 Possible answers: Unacceptable: I was not greeted. Acceptable: I was greeted after I said hi or hello. Optimal: Employee imme- diately greeted me. We then formulated the questions the mystery shoppers would ask the staff. We wanted the question to be involved enough to elicit the behaviors being evaluated, without being so complicated the shopper would not be able to sustain the transaction. Examples that have worked for us: Do you have any books by [author name]? Can you help me find a scholarly article on im- ages of women in advertising? Examples of the types of questions to avoid: Directional questions such as, Where is the bathroom? These answers do not require much inter- action. Complicated questions that lead to a lengthy reference interview should be avoided as well. Often the shopper will not have the time for a personal consult, or will not have the background on the topic to fake it for long. At Radford University, we worked with business majors. We had the most suc- cess feeding them questions like “Can you help me find an article about outsourcing factory work to Asia?” At Longwood Univer- sity, we had a variety of multi-disciplinary questions designed to elicit the behavior being evaluated and determine if the staff person could answer the question correctly. Secret #5: Set your shoppers up for success We decided that we needed to use shop- pers from outside the library to maintain objectivity. At both universities, we worked with business faculty to use their students as shoppers. The faculty members graciously gave extra-credit or participation points to students who shopped us. Although profes- sional mystery shoppers could be hired, we were satisfied with our arrangement. The advantage with using on-campus shoppers is that they are familiar with the academic environment and their expectations may be more in line with our average user than an outside shoppers might be. We presented the project to the student shoppers in their classes and described mystery shopping in general and the spe- cifics of our plans. Even after sharing these details, we had to emphasize to the students that the point was to evaluate the service, not to test the students’ library knowledge. Some students expressed discomfort at ap- proaching the reference desk with a simple “Do you have this book” question, explain- ing that they knew how to use the catalog. Role playing a mystery shopping occurrence helped them better understand their func- tion, as well as teaching them appropriate shopper behavior. As we continued with our mystery shop- ping exercises, we were able to also share some changes in the libraries that had come about because of past results. These ex- amples often made the project ‘real’ to the students and increased how seriously they took their roles. We reviewed the mystery shopping in- strument the students would use, reminded them not to bring it to the desk when con- ducting their shop, and clarified the proce- dure for returning the completed instrument. At Radford University, we eventually moved to an online form, which simplified the pro- cess greatly. To make sure the same librarian wasn’t shopped repeatedly, we scheduled our shoppers. At Longwood, we provided staff pictures to our shoppers that were to help them identify whom they were to shop. This greatly helped ensure accuracy of who was shopped. Secret #6: Share the findings At both libraries, librarians who did not staff the reference or front desks during the mys- tery shopping study were given the raw data. Since Radford University was using an online survey tool that aggregated the shoppers’ data, the Radford University author focused on eliminating identifying details from the free comments section. Final reports, as- sessing overall the service provided, were given to the desk supervisors. The supervi- (continues on page 287) May 2011 287 C&RL News embargo after print publication; for many presses, this will be their first experience with simultaneous release. Other university press collections are underway also, such as those respectively of Cambridge and Oxford University Presses. The report was posted in a MediaCom- mons edition (an AAUP first)2 and has pro- voked some lively response from librarians, university administrators, and university press staff. Some comments add new ideas or examples or expand on the complexity of the issues discussed. Others reveal significant skepticism about the capacity of university presses to embrace fully a more open and format- unbounded future. In my personal view, this open exchange feels like an exciting step forward into the healthy dialogue that we have long desired. University presses share so much com- mon ground with academic libraries. This includes supporting faculty research, writ- ing, and teaching, and valuing the quality, accessibility, and usefulness of published work. It is only natural that today’s engage- ment of both presses and libraries with new technologies and new methods of dissemination is informed by our re- spective legacy business models (which are strikingly different). These different perspectives are exciting because of how much we can learn from each other and accomplish together. I hope that you will find the AAUP taskforce report helpful as one small step in advancing the conversation. Notes 1. “Sustaining Scholarly Publishing: New Business Models for University Press- es,” www.aaupnet.org/resources/reports /business_models/index.html. 2. mediacommons.futureofthebook.org /mcpress/sustaining. sors then met with the staff to review the results. Successes were celebrated and areas for improvement were identified. At Longwood University, the desk su- pervisor decided to use the mystery shop- ping reports as part of the staff evaluation process. Such a move entailed adjusting the staff member’s work description to include customer service as a responsibility and mystery shopping as a measure of progress. Conclusion Based on our studies, we conclude that mystery shopping is an accurate and efficient method to measure existing service at library service points. Used iteratively, it provides cyclical feedback as one pinpoints areas for improvement, provides training, repeats the mystery shopping process, and determines if shoppers report staff behaviors that meet established expectations. Mystery shopping is not productive as a mechanism to “spy on” or “catch” employees misbehaving. Instead, a sound mystery shopping program is a team effort that includes all stakeholders in a process that leads to improved interpersonal communication between our public service staff and patrons. Notes 1. J. D. Durrance, “Factors That Influence Reference Success: What Makes Questioners Willing to Return,” Reference Librarian 23 (1995): 243–65. 2. T. Van der Wiele, M. Hesselink, J. Van Iwaarden, “Mystery shopping: A tool to develop insight into customer service provision,” Total Quality Management 16 (200):529–41. 3. “Guidelines for Behavioral Performance of Reference and Information Service Provid- ers,” accessed January 6, 2011, www.ala.org /ala/mgrps/divs/rusa/resources/guidelines /guidelinesbehavioral.cfm. (“Secrets to successful mystery shopping,” cont. from page 276)