United States General Accounting Office Report to the Ranking Minority Member, Committee on Science, House of Representatives July 1995 LCM UTA NCO) MONITORING ere OTs nae ra) Model» make Sd GAO/PEMD-95-19 '''' GAO United States General Accounting Office Washington, D.C. 20548 Program Evaluation and Methodology Division RA B-261325 7S Y July 19, 1995 ( Mees The Honorable George E. Brown, Jr. | 199 Ranking Minority Member PU BL Committee on Science House of Representatives Dear Mr. Brown: This report responds to your request that we identify features of a model nutrition monitoring system and examine approaches to incorporating those features in the National Nutrition Monitoring and Related Research Program (NNMRkP). It builds on two earlier reports. The first, Nutrition Monitoring: Progress in Developing a Coordinated Program (GAO/PEMD-94-23), assessed the NNMRRP’s planning activities. The second, Nutrition Monitoring: Data Serve Many Purposes; Users Recommend Improvements (GAO/PEMD-95-15), surveyed users of nutrition monitoring data about how the data are used and what changes are needed to the data collection activities. This report describes features of a model nutrition monitoring system and presents the strengths and limitations of various strategies for achieving four of the model features. As arranged with your office, we will be sending copies of this report to the Director of the Office of Management and Budget, the Interagency Board for Nutrition Monitoring and Related Research, the agencies responsible for data collection, and interested congressional committees. We will also make copies available to others upon request. The major contributors to this report are listed in appendix V. If you have any questions or would like additional information, please call me at (202) 512-3092. Sincerely yours, So) Kwai-Cheung Chan Director of Program Evaluation in Physical Systems Areas '' Executive Summary Purpose Background Nutrition-related data are used to monitor food safety, study the relationship between diet and health, inform agricultural policies, evaluate food assistance programs, and help food industries develop new products. However, past evaluations have criticized federal nutrition monitoring activities as poorly coordinated and insufficiently responsive to the needs of data users. At the request of the former House Committee on Science, Space, and Technology, Gao defined the features of a model monitoring program and compared current and potential approaches to responding to the needs of data users. The current National Nutrition Monitoring and Related Research Program (NNMRRP) consists of a combination of national surveys, federally- supported surveillance systems operated by the states, and other research and data collection activities. The 1990 National Nutrition Monitoring and Related Research Act (P.L. 101-445) established an Interagency Board that gave the Departments of Agriculture (UsDA) and Health and Human Services (HHS) primary responsibility for developing and implementing a 10-year comprehensive plan for federal nutrition monitoring.! In reviewing the Interagency Board’s progress in coordinating the program, an earlier GAO report found that coordination between UsDA and HHS had improved, but the 10-year plan was still inadequate. It neither established priorities nor provided a framework for evaluating existing or potential monitoring activities.2 Drawing on a survey of users of nutrition monitoring data, a second GAO report found that the data collected by the NNMRRP support an extensive range of purposes, but changes could increase the utility and credibility of the data.? For this report, Gao first defined the features of a model program by reviewing past evaluations of federal nutrition monitoring and the 10-year plan, consulting experts, and surveying users of nutrition data. Then, with the assistance of expert advisers and studies of illustrative programs, GAO identified and compared current and potential approaches to incorporating model features in the NNMRRP. (See chapter 1.) 1Ten-Year Comprehensive Plan for the National Nutrition Monitoring and Related Research Program, 58 Fed. Reg. 111 (June 11, 1993), pp. 32752-806. 2See Nutrition Monitoring: Progress in Developing a Coordinated Program (GAO/PEMD-94-23; May 27, 1994). 3See Nutrition Monitoring: Data Serve Many Purposes; Users Recommend Improvements (GAO/PEMD-95-15; June 20, 1995). Page 2 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Results in Brief GAO’s Analysis Executive Summary GAO identified four features of a model program: It would have a coordinated set of activities, provide data continuously, generate reliable "inferences about important t subpopulations and small geographic areas, and support state and local monitoring activities. While the NNMRRP already has some elements of a model program, other strategies may lead to improved nutrition monitoring capabilities. Alternate approaches to coordination, such as an independent central office with authority over the NNMRRP agencies or a single lead agency, may not provide any clear advantages to the current structure, which relies on the Interagency Board. However, such approaches do offer useful tools that could be adopted by the Board, such as the use of formal budget reviews and the establishment of a central contact for data users. The current source-of— continuous data—the state-based surveillance systems—cannot meet the needs for national-level information or for in-depth dietary intake_ information. Alternatives are to attach modules of nutrition-related questions to other ongoing surveys or to field a core set of questions continuously, supplemented periodically by questions of intermittent or emerging interest. Current approaches to providing more information on subpopulation groups and small geographic areas are to oversample selected groups as part of the national surveys and to collect data on specific high-risk groups through the surveillance systems. These approaches could be complemented by special studies and indirect estimation. Finally, to support state and local monitoring activities, HHS provides a combination of technical and financial assistance for the state-based surveillance systems. One option to provide more relevant data to localities is community-based data collection. Features of a Model Nutrition Monitoring Program Coordination of the data collection activities is needed to ensure that the diverse needs for the data are met efficiently. For example, NNMRRP’S major national surveys collect complementary data, with one focusing on health and nutritional status and the others emphasizing food consumption. Combining these data would increase their utility. (See chapter 2.) Continuous data collection, “repeated regularly and frequently” (as defined in the 10-year plan), is needed to track trends in diet-related health risks, such as those targeted in HHS’ year 2000 health objectives. In addition, continuous data are important in evaluating the effects of policy changes, such as replacing current food assistance programs with block grants. Page 3 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Executive Summary The dietary habits and nutritional problems of a subpopulation or small geographic area may deviate from the national profile. Data on these. groups are needed to appropriately target programs that address their nutrition-related problems, such as diabetes in some minority groups. Finally, states and localities need relevant nutrition monitoring data to plan programs that respond to the specific health and nutrition concerns of their populations. For example, currently available data have been used to initiate “5-a-Day” programs that encourage consumption of fruits and vegetables and to establish anemia as a priority issue for nutrition education. Current and Potential Approaches to Achieving a Model Program While the Interagency Board appears to have improved communication and cooperation among the agencies, its 10-year plan did not establish clear priorities for nutrition monitoring across agencies. The Board has conducted a priority-setting exercise, but it has made little progress in developing a framework to guide NNMRRP decision-making. The two alternate approaches to coordinating nutrition monitoring that Gao examined suggest mechanisms for strengthening the work of the Interagency Board. Appointing an NNMRRP administrator, for example, could provide a central contact point for users of nutrition monitoring data. In addition, regular budget reviews could be conducted to ensure that priority objectives of the programs are funded. (See chapter 3.) None of the three major national surveys are implemented continuously. Moreover, the surveillance systems, which are the current source of continuous data, cannot meet the needs of data users, who depend on national-level data because of limitations in the amount and kind of data they gather and in the populations they target. A continuous national survey could be more efficient in terms of costs per respondent than the current, intermittent surveys, especially if a smaller, core set of data elements are collected continuously and supplemented periodically by topical modules. However, activities that are now implemented sequentially would have to be funded and staffed to operate concurrently. Another approach to gathering information more frequently is the inclusion of nutrition-related questions on other ongoing surveys. (See chapter 4.) At present, the state-based surveillance systems and oversampling in the national surveys provide information on subpopulations and small geographic areas. Two surveillance systems focus on low-income pregnant women and children, but not all states participate and data are collected only from participants in public health and nutrition programs. Page 4 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Executive Summary Recommendations Agency Comments Oversampling of specific subpopulations as part of a national survey is most appropriate for groups that are geographically clustered and that do not require different data collection procedures than those used for the general population. Its advantage is that the data on the group of interest can be compared to national estimates. Two alternate ways of gathering information on subpopulations and small geographic areas are special studies and indirect estimation programs. Special studies are appropriate for populations that are geographically dispersed or require different data collection procedures, but they can be costly relative to other approaches. Indirect estimates, which use data from other sources to predict the value for the group or area of interest, are likely to be less costly than collecting new data for direct estimation. However, gauging the bias introduced by the models that produce the indirect estimates is difficult; thus, they may not be as useful as direct estimates. (See chapter 5.) Through the state surveillance systems, HHS provides assistance in the collection, analysis, and interpretation of state and local data. Although the data collection instruments can be modified from state to state, the systems are organized to collect standardized information across the participating states. As a result, they may have limited sensitivity to state or local differences. An alternate strategy is to design monitoring programs specifically for local needs. However, unlike the data collected by the surveillance systems, community-based data could probably n not be aggregated to describe statewide needs. (See chapter 6.) GAO is not making recommendations in this report. In their written comments on a draft of this report, officials from USDA and HHS (1) found the report to be factual, (2) agreed with the model program features identified by Gao, (3) agreed with the importance of the model features discussed in the report, and (4) provided additional detail on the progress that has been made by the Interagency Board and its working groups toward improving NNMRRP. Both usDa’s and HHS’ comments as well as comments received from three members of the National Nutrition Monitoring Advisory Council noted that limited resources affect the NNMRRP’s ability to incorporate the model features. (See appendixes III and IV for comments from USDA and HHS and our response.) Page 5 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Contents Executive Summary 2 Chapter 1 we 10 The C + Nutrition Core Data Collection Activities of the NNMRRP 10 € Current NUUIMON structure of the NNMRRP 13 Monitoring Pr ogram Objectives, Scope, and Methodology 14 Organization of the Report 18 Chapter 2 Features of a Model Prog = iis eatures of a Model Program A Model Nutrition The Selected Features 22 Monitoring Program Approaches to Achieving the Selected Model Features 29 Chapter 3 31 A h t Current Approach: the Interagency Board 31 pproac es O Potential Options 36 Coordinating the Conclusions 41 Nutrition Monitoring Program Chapter 4 42 A h to th Current Approach: the State-Based Surveillance Systems 42 pproac es to tne , Potential Options 45 Continuous Collection — Conclusions 49 of Data Chapter 5 50 A h t Current Approaches 51 pproac €sS to Potential Options 53 Supporting Reliable Conclusions 57 Inferences About Subpopulations Page 6 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Contents Chapter 6 59 A h t Current Approach: the State-Based Surveillance Systems 59 pproac €s to Potential Option: Community-Based Nutrition Monitoring 61 Assisting States and Conclusions 63 Localities Appendixes Appendix I: List of Experts 64 Appendix II: Sources for the Model Features 66 Appendix III: Comments From the Department of Agriculture 73 Appendix IV: Comments From the Public Health Service 80 Appendix V: Major Contributors to This Report 85 Related GAO Products 88 Tables Table 1.1: Selected Uses of NNMRRP Data by Content Area 10 Table 1.2: Major NNMRRP Activities by USDA and HHS ll Table 2.1: Features of a Model Nutrition Monitoring Program and 20 Their Sources Table 2.2: Examples of State and Local Uses of Nutrition Data 28 Table 2.3: Features of a Model Program and Current and Potential 30 Approaches Table 3.1: Approaches to Coordinating Nutrition Monitoring 31 Activities Table 3.2: Responsibilities and Coordination Tools of Three 37 White House Offices Table 4.1: Approaches to Providing Continuous Data 42 Table 5.1: Approaches to Supporting Inferences About 51 Subpopulations Table 6.1: Approaches to Assisting States and Localities 59 Table II.1: Features of a Model Program Suggested by Past 66 Evaluations Table II.2: Examples of Planned NNMRRP Activities Associated 68 With Model Features Table II.3: Advisory Panel Suggestions Associated With Model 69 Features Table II.4: Major Themes in the Suggestions Made by Data Users 71 Page 7 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Contents Abbreviations ARS Agricultural Research Service BRFSS Behavioral Risk Factor Surveillance Survey CPS Current Population Survey CSFII Continuing Survey of Food Intake by Individuals DHKS Diet and Health Knowledge Survey HHANES Hispanic Health and Nutrition Examination Survey HHS Department of Health and Human Services HPCCP High Performance Computing and Communications Program NCST National Council on Science and Technology NFCS Nationwide Food Consumption Survey NHANES National Health and Nutrition Examination Survey NNMRRP National Nutrition Monitoring and Related Research Program OMB Office of Management and Budget ONDCP Office of National Drug Control Policy OSTP Office of Science and Technology Policy PATCH Planned Approach to Community Health PedNSS Pediatric Nutrition Surveillance System PHS Public Health Service PNSS Pregnancy Nutrition Surveillance System USDA U.S. Department of Agriculture WIC Special Supplemental Food Program for Women, Infants, Page 8 and Children GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Page 9 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program ''Chapter 1 The Current Nutrition Monitoring Program This chapter first provides background information on the National Nutrition Monitoring and Related Research Program. Then, the objectives, scope, and methodology of our review of current and potential approaches to achieving a model program are described. The chapter concludes with an overview of the organization of the rest of the report. Core Data Collection Activities of the NNMRRP The NNMRRP is a complex system of data collection and research activities, including national surveys, state surveillance activities, and a variety of research programs. Over time, the NNMRRP has developed activities focused on five content areas: (1) food and nutrient consumption; (2) nutritional and health status; (3) dietary knowledge, attitudes, and behavior; (4) food composition; and (5) food supply. As shown in table 1.1, the information produced by these activities is used for a variety of purposes, from supporting basic research on human nutritional needs to informing policy decisions about health, agriculture, and food programs. EN aE TR EU SS ES AN on TE] Table 1.1: Selected Uses of NNMRRP Data by Content Area Content area Food and nutrient consumption Selected uses? Assess potential exposure of children to pesticide residues Develop standards for food stamp eligibility Evaluate impact of farm commodity programs on demand and pricing Monitor trends in diet-related risks of chronic diseases Nutritional and health status Evaluate food assistance programs Research the relationship between fat consumption and cancer Target food fortification to foods eaten by people with nutrient deficiencies Update infant growth charts Inform policies on cholesterol screening and treatment Dietary knowledge, attitudes, and behavior Develop food labeling policies Inform nutrition education programs Food composition Support estimation of the consumption of nutrients and food components related to chronic disease Plan meals for military personnel and other groups Monitor trends in nutrient availability and food safety Food supply Manage federal marketing and agricultural policies @These uses were selected to illustrate the variety of purposes supported by NNMRRP data. They are not an exhaustive list. Table 1.2 lists specific activities managed by the Departments of Agriculture and Health and Human Services, which have major Page 10 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 1 The Current Nutrition Monitoring Program responsibilities for the five areas. Other agencies, including Commerce, Defense, and the Environmental Protection Agency, also participate in the NNMRRP. Table 1.2: Major NNMRRP Activities by USDA and HHS Department Content area USDA HHS Food and nutrient Nationwide Food National Health and consumption Consumption Survey Nutrition Examination Survey Continuing Survey of Food Intake by Individuals Total Diet Study Nutritional and health status No major role National Health and Nutrition Examination Survey Pregnancy Nutrition Surveillance System Pediatric Nutrition Surveillance System Dietary knowledge, attitudes, and behavior Diet and Health Knowledge Survey Health and Diet Survey Behavioral Risk Factor Surveillance System Food composition National Nutrient Data Bank Food Label and Package Survey Food supply Food and Nutrition Supply Series No major role USDA Activities uspDA has major responsibilities for collecting information in all of the content areas except nutritional and health status. UsDA gathers data on food and nutrient consumption through two national surveys—the Nationwide Food Consumption Survey (NFcs) and the Continuing Survey of Food Intake by Individuals (csFri). In the past, NFcs has gathered nationally-representative information on the food consumption behavior of households and individuals. It provides detailed data on household costs for food. One of the major uses for these data is the development of the Thrifty Food Plan, which is the basis for calculating food stamp benefits. Implemented decennially, Nrcs suffered from severe response rate problems (less than 40 percent) in 1987-88. As a result, the individual food consumption portion of the NFCs is expected to be dropped in the future. Page 11 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 1 The Current Nutrition Monitoring Program Since the mid-1980s, csFil has supplemented NFcs by providing regular information on individual dietary intake. The data collected in its most recent implementation (1994-96) will be used to describe both general and low-income populations. The Diet and Health Knowledge Survey (DHKs), which collects data on dietary knowledge, attitudes, and behavior, is a follow-up to csFi. Together, CSFII and DHKS are intended to inform policies relating to food production and marketing, food safety, food assistance, and nutrition education. The USDA activities focused on food composition and food supply are not surveys. For food composition, USDA gathers data from food industries and other sources on the nutrient content of foods. These data support the dietary surveys by translating the foods consumed into their nutrient components. Food supply is estimated by deducting data on exports, year-end inventories, and nonfood use from data on production, imports, and beginning inventories. HHS Activities HHS is responsible for the National Health and Nutrition Examination Survey (NHANES) and the state-based surveillance systems. These data collection activities provide information on the content areas (1) food and nutrient consumption, (2) nutritional and health status, and (3) dietary knowledge, attitudes, and behavior. Like NFCS and CSFII, NHANES collects data from a nationally-representative sample. However, NHANES’ unique contribution is its use of physical examinations and clinical and laboratory tests as well as traditional survey methods to gather information. NHANES’ data support research on the relationship between diet and health and inform health policy decisions, such as the promotion of cholesterol screening. After two earlier implementations in the 1970s, NHANES has just completed its third administration (1988-94). NHANES has been supplemented by the Hispanic Health and Nutrition Examination Survey (HHANES) in the early 1980s, follow-up surveys of respondents to NHANES I, and follow-up matches of the records from NHANES II to the National Death Index and other vital statistics records. The state-based surveillance systems were set up to provide quick information to states to use in planning and managing nutrition and health programs. They include the Pediatric Nutrition Surveillance System (PedNSS), Pregnancy Nutrition Surveillance System (PNss), and Behavioral Risk Factor Surveillance System (BRFSS). Participating states collect the data for these systems with technical and other kinds of assistance from HHS. Both PedNSS and PNSS rely on data from clinic records from publicly Page 12 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 1 The Current Nutrition Monitoring Program Structure of the NNMRRP funded health, nutrition, and food assistance programs, primarily the Special Supplemental Food Program for Women, Infants, and Children (WIC). PedNSS monitors nutritional status among low-income, high-risk children, while pNss focuses on low-income, high-risk pregnant women, measuring nutrition-related problems and behavioral risk factors associated with low birthweight. In contrast to PedNss and PNSS, data for BRFSS are gathered through telephone interviews, with respondents (adults 18 years and over) sampled through random digit dialing. In addition to a core set of questions on various health risk factors, BRFSS includes optional modules for the assessment of dietary fat and fruit and vegetable consumption. Other major HHS monitoring activities include the Total Diet Study, which . analyzes nutrient and contaminant levels in the food supply; the Food Label and Package Survey, which monitors nutrition labeling practices; and the Health and Diet Survey, which assesses dietary knowledge and practices as they relate to health problems. Although the United States has one of the most comprehensive monitoring programs in the world, several problems with nutrition monitoring activities have been identified over the past two decades. Of key concern has been the lack of coordination and compatibility of different data collection activities. This encompasses differences across surveys in methods for assessing dietary intake and nutritional status, sampling designs, population descriptors and other measures, and the timing and reporting of results. To improve the coordination of federal nutrition monitoring activities and the quality of the data collected, the Congress passed the National Nutrition Monitoring and Related Research Act of 1990 (P.L. 101-445). The act established an Interagency Board, jointly chaired by UsDA and HHS, to coordinate activities across the various agencies involved in nutrition monitoring. The Interagency Board was charged with developing a strategic plan that would establish a comprehensive nutrition monitoring and related research program. This plan—known as the 10-year comprehensive plan—was published in the Federal Register on June 11, 1993. It outlines a set of planning activities, including a general time frame and lead agencies for each activity. The activities are organized around six objectives, which are to Page 13 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 1 The Current Nutrition Monitoring Program Objectives, Scope, and Methodology provide for a comprehensive NNMRRP through continuous and coordinated data collection; improve the comparability and quality of data across the NNMRRP; improve the research base for nutrition monitoring; develop and strengthen state and local capacity for continuous and coordinated nutrition monitoring data collection that complements national nutrition surveys; improve methodologies to enhance comparability of NNMRRP data across federal, state, and local levels; and improve the quality of state and local nutrition monitoring data. In addition, an Advisory Council of experts from outside the federal government was created to guide the Interagency Board on scientific and technical matters. (Chapter 3 contains more information on the Interagency Board and its activities.) Objectives This is the third and final report in a series responding to a request from the former House Committee on Science, Space, and Technology. The first report reviewed past evaluations of federal nutrition monitoring and examined the progress of the NNMRRP since the passage of the 1990 act.? It concluded that (1) a coherent program for nutrition monitoring was not yet in place and (2) although there has been progress in coordinating the program, the 10-year plan is incomplete because it does not include a framework for evaluating current and potential activities or detailed plans for achieving the objectives. Based on a survey of users of nutrition monitoring data, the second report described the purposes for which nutrition data are used and summarized respondents’ suggestions for improving NNMRRP activities.? These suggestions, which addressed such issues as the timing of the surveys, their coverage of subpopulations, and the ease with which the data could be used, were consistent with the concerns raised by past evaluations of ‘Nutrition Monitoring: Progress in Developing a Coordinated Program (GAO/PEMD-94-23). Nutrition Monitoring: Data Serve Many Purposes; Users Recommend Improvements (GAO/PEMD-95-15). Page 14 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 1 The Current Nutrition Monitoring Program federal nutrition monitoring and indicated a continued need to address the long-standing problems of federal nutrition monitoring. Completing our response to the Committee’s request, this report builds on our earlier work to meet two objectives: (1) define a model nutrition monitoring program and (2) compare the current system with potential options for implementing the components of a model program in the NNMRRP. Scope and Methodology Defining the Features of a Model Program Before defining the features of a model program, we first limited our review to three of the five content areas covered by the NNMRRP: (1) food consumption and dietary intake; (2) health and nutritional status; and (3) knowledge, attitudes, and behavior. These three variables were selected for both substantive and methodological reasons. Substantively, the selected elements provide the data that support the planning and evaluation of interventions that directly affect health, such as nutrition education and food assistance programs. A substantive argument could also be made for the inclusion of food composition data, which are used to translate the information on what foods are eaten into estimates of nutrient intake. However, because an earlier GAO project focused on the NNMRRP’s food composition activities, we did not include this in our review.® Methodologically, the three selected variables are linked because they rely on data obtained from individuals through surveys and physical examinations. In contrast, food composition information is based on chemical analyses of foods, and food supply is determined from macroeconomic data. Our focus on the components of the NNMRRP that rely on surveys facilitates the comparison of current and potential approaches to achieving a model program. To identify features of a model nutrition monitoring program, we used four sources: reviews of previous evaluations of federal nutrition monitoring activities, review of the objectives and related activities outlined in the 10-year comprehensive plan developed by the Interagency Board, consultation with expert advisers, and our survey of data users. 3See Food Nutrition: Better Guidance Needed to Improve Reliability of USDA’s Food Composition Data (GAO/RCED-94-30; Oct. 25, 1993). Page 15 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 1 The Current Nutrition Monitoring Program These sources are detailed below, and supporting material is provided in appendix II. (See chapter 2 for a description of the features.) Past Evaluations of Federal Nutrition Monitoring. Past evaluations by such groups as the National Academy of Public Administrators, the Joint Nutrition Monitoring Evaluation Committee, and the National Research Council identified several concerns about the federal nutrition monitoring program. Because these evaluations also informed the Interagency Board’s development of its 10-year plan, we used them as the starting point for our identification of features of a model nutrition monitoring program. Our report, Nutrition Monitoring (GAO/PEMD-94-23), discusses these past evaluations and NNMRRP progress in addressing their recommendations. (Table II.1 in appendix II lists the criticisms identified in these evaluations and categorizes them by the features of a model program that they suggest. ) The 10-Year Comprehensive Plan. As described above, the NNMRR Act required the Interagency Board to develop a plan for the program. The plan outlined six objectives, three with a federal focus and three emphasizing state and local monitoring, and listed 68 planned activities. These were reviewed for responsiveness to the model features suggested by the past evaluations. (See table II.2 in appendix II for examples of the 68 activities listed in the plan.) Consultation With Expert Advisers. To assist us at critical decision points in this project, we organized three panels. (The members of each panel are listed in appendix I.) The Core Policy Panel consisted of nationally-known experts in nutrition and nutrition monitoring policy. These panelists were consulted throughout the project. In addition, they helped us develop a framework of purposes for nutrition monitoring data that guided our survey of data users. The Methodology Panel included renowned experts in such fields as sampling, survey design, dietary assessment, and nutritional epidemiology. This panel met to help us identify promising approaches to critical elements of a nutrition monitoring system. In addition, the panelists assisted us on issues related to their areas of specialization. The Data Users Panel consisted of users of the nutrition monitoring data, chosen to reflect the broad range of purposes that the data must serve, including the support of state and local nutrition programs, academic research, food industry research, and the development and evaluation of Page 16 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 1 The Current Nutrition Monitoring Program Comparing Approaches to Attaining the Model Features federal food assistance programs. As with the Methodology Panel, this panel was convened once to help us identify promising approaches to nutrition monitoring. Individual panelists were consulted later about specific issues related to their expertise. Through the process of reviewing materials and participating in panel meetings, the expert advisers generated several suggestions for possible changes to the NNMRRP, examples of which are given in table II.3 in appendix II. In addition, they reviewed a draft of this report. Suggestions From the Survey of Data Users. Our survey of users of nutrition monitoring data focused on primary users of 14 of the NNMRRP surveys and surveillance systems. Primary users were defined as those who directly access the data rather than use information that has already been processed and interpreted by others in reports and other documents. Since there is no single list of primary users of NNMRRP data, we obtained lists of known and potential data users such as people who had requested the data from NNMRRP agencies, attendees at nutrition-related workshops, state and local government officials working in nutrition, and members of associations for nutrition professionals.* A major portion of the survey was dedicated to determining how the respondents used the data. We also asked whether changes are needed to better meet the respondent’s information and data quality needs. If the respondent indicated a need for change, we asked for suggestions in the following categories: (1) data elements collected, (2) data collection methods, (3) units of analysis, (4) time of data collection, (5) population group coverage, (6) geographic area coverage, and (7) ease of use. These comments were analyzed to identify major themes for each of three groups of data collection activities—USDA surveys, HHS surveys, and HHS state-based surveillance systems. In appendix II, table II.4 identifies the themes associated with the features of a model program. From the features that were identified, we selected four as the focus of the second objective—the comparison of current and potential approaches to achieving the model program. We focused on features that reflect long-standing concerns about federal nutrition monitoring, that encompass other desired characteristics, and that generate debate about how they should be addressed. 4See Nutrition Monitoring (GAO/PEMD-95-15) for details of the survey. Page 17 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 1 The Current Nutrition Monitoring Program For each of the selected features, we identified current activities of the NNMRRP through interviews with staff in NNMRRP agencies, attendance at meetings of the Interagency Board and the Advisory Council, and reviews of program documents. Potential approaches were identified through literature review, analysis of our survey results, and consultation with the expert advisers. We did not identify the universe of potential approaches. Instead, our search was focused on those approaches deemed promising and feasible by our expert advisers. To assess each of the potential approaches, we first identified programs from the same set of sources that helped us define the features of a model program—that is, the literature, our survey of data users, and expert advisers. Where possible, we limited our consideration to programs that had some linkage to nutrition monitoring. For example, for separate studies of subpopulation groups, we looked at the experience with the Hispanic Health and Nutrition Examination Survey in the early 1980s. To describe the strengths and weaknesses of the programs illustrating the potential options, we reviewed program documents and related literature and interviewed managers and staff. This review was conducted between October 1993 and December 1994 in accordance with generally accepted government auditing standards. Strengths and Limitations of Our Methodology Organization of the Report This report describes the results of a systematic examination of current and potential approaches to selected features of a model nutrition monitoring program. The strength of the review is its reliance on multiple sources of information. In addition to surveying users of nutrition monitoring data, consulting with experts, and reviewing both technical literature and program documents, we also interviewed officials and program staff in nutrition monitoring programs and in programs illustrating the alternatives. The major limitation of our work is its prospective nature. Because we were examining potential changes to the NNMRRP, hard evidence of the costs or effectiveness of the options was not available. Instead, the strengths and weaknesses of the options relative to the current nutrition monitoring system are supported primarily by logic and stated in tentative terms. Given this limitation, the report makes no recommendations for specific changes to the NNMRRP. In response to the first objective of the project, chapter 2 describes the model program and provides information on the selection of four model Page 18 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 1 The Current Nutrition Monitoring Program features as the focus of the report. (The sources used to develop the model are described above.) The second objective—comparing current and potential approaches to each model feature—is addressed in chapters 3-6, organized by the four model features. Specifically, chapter 3 examines coordination options; chapter 4 compares alternate approaches to providing continuous data; chapter 5 discusses different methods of supporting inferences about subpopulation groups and small geographic areas; and chapter 6 reviews approaches to assisting state and local monitoring activities. Appendix I provides additional detail on the expert advisers to the project, and appendix II describes the sources for the model features. Agency comments on a draft of the report are in appendixes III and IV. Page 19 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program ''Chapter 2 A Model Nutrition Monitoring Program This chapter responds to the first objective of our review—the definition of a model nutrition monitoring program. First, the major features identified from the sources described in chapter 1 are outlined. From this set of features, we selected four as the focus for our response to the second objective of the review—the comparison of current and other approaches to achieving the model characteristics. The chapter describes our selection process and details the importance of the four features that are the subject of the rest of the report. Features of a Model Program Depending on the purposes that the data serve, the specific elements of a model nutrition monitoring program change. For example, researchers and program managers interested in food safety need detailed information on dietary intake, including specific brand names of the foods consumed. In contrast, a nutrition educator may place a higher priority on information about dietary knowledge, attitudes, and behaviors. However, at a more general level, some common ideal characteristics can be identified. Focusing on this general level, we used the sources described in chapter 1 to identify a number of features of a model program. Table 2.1 lists these features and the sources that support them. Table 2.1: Features of a Model Nutrition Monitoring Program and Their Sources Feature Source? Coordinated system that responds efficiently Past evaluations to the diverse needs of the data users 10-year plan Expert advisers Continuous or more frequent collection of data Past evaluations 10-year plan Expert advisers Survey of data users Support for reliable inferences about subpopulation Past evaluations groups and small geographic areas 10-year plan Expert advisers Survey of data users (continued) Page 20 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 2 A Model Nutrition Monitoring Program Feature Assistance to states and localities Source® Past evaluations 10-year plan Expert advisers Survey of data users Strong research base, including improved methods for assessing dietary intake and nutritional status High response rates and low respondent burden Past evaluations 10-year plan Expert advisers Survey of data users Past evaluations Survey of data users Timely processing and dissemination of survey information Past evaluations 10-year plan Survey of data users On-going evaluation of the system’s content and methods, including a review of the information needs Expert advisers Collection of both household and individual data Expert advisers Survey of data users Maintenance of data comparability over time Expert advisers Longitudinal data (collection of information on the same individuals or households over time) Expert advisers Survey of data users @For more information on these sources, see the discussion in chapter 1 and the tables in appendix II. All of the features identified are clearly important elements of a comprehensive nutrition monitoring program. However, we selected four characteristics as the focus of our review: (1) a coordinated set of activities that (2) provides data on a continuous basis, (3) supports inferences about important population groups, and (4) assists state and local monitoring activities. These features encompass other desired characteristics, respond to long-standing concerns about federal nutrition monitoring, and generate debate about how they can be achieved. The first criterion for selection was to focus on the most general concerns. With these features, other desirable characteristics can be considered even Page 21 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 2 A Model Nutrition Monitoring Program The Selected Features though they are not emphasized. Specifically, a mechanism for evaluating the NNMRRP’s content and methods, including a review of the information needs of the data users, is described in chapter 3 as an element of a coordinated program. The theme of evaluating options in relation to the needs for data also underlies the discussion of the alternatives for the other features. Three other features—the comparability of data over time, the collection of longitudinal data, and the timeliness of data release—are related to the continuous collection of data and, as such, are considered briefly in chapter 4. The four features were also selected because they respond to long-standing criticisms of federal nutrition monitoring activities. Concerns about coordination, the continuity and timeliness of the data, the availability of information on subpopulations, and the role of states and localities were raised as early as 1977 by witnesses before a House subcommittee. In contrast, the concerns with response rates and the level of the data (individual or household) can be traced to the problems with the last NFcs (described in chapter 1), which uspa has taken steps to avoid in the future. The selected features are also not easily addressed; that is, there is debate about the best approach to achieving each feature. For example, some expert advisers stated that assistance to states and localities should focus on data analysis and interpretation, while others argued for a larger state and local role in data collection. Similarly, to provide information on subpopulation groups, national surveys could be supplemented by such means as the surveillance systems and oversampling or the national surveys could be abandoned and their resources dedicated to special studies of specific groups. In contrast to the debates about how best to achieve these and the other selected features, there has been consensus about the kind of dietary intake methodology that will be used by the NNMRRP Surveys and ongoing research to improve these methods through automation and other means. An ideal federal nutrition monitoring program would have a coordinated set of activities that provides data on a continuous basis, covers important population groups, and supports state and local monitoring activities. Coordination is the key both to the efficiency of the system and its responsiveness to the needs of data users. A continuous flow of data ‘Hearings before the House Subcommittee on Domestic and International Scientific Planning, Analysis and Cooperation, Committee on Science and Technology, July 26, 27, and 28, 1977. Page 22 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 2 A Model Nutrition Monitoring Program would ensure that the information on the nation’s nutritional status was up-to-date and would also enable the tracking of dietary behavior and nutritional status over time. Information on population groups that are vulnerable or growing rapidly is needed to plan, manage, and evaluate programs intended to prevent or ameliorate nutritional problems. Because state and local governments are often the location of such programs, they need assistance in either interpreting available data or collecting their own information. The importance of these features is further explained in the following sections. A Coordinated Nutrition Monitoring Program The coordination of nutrition monitoring activities has implications for both the utility of the information produced and the costs of the program as a whole. The utility of the information is constrained when data from different data collection activities cannot be easily combined. For example, research on the relationship between diet and health could be strengthened if csFi’s data on dietary intake could be combined with NHANES’ data on health and nutritional status. However, because of differences in the sampling designs and nutrition measures, combining the data from the two surveys is difficult and controversial. Similarly, poor integration of the data collected by the state-based surveillance systems with the national surveys presents a barrier to meaningful comparisons of state and national populations. To the extent that the lack of coordination results in unnecessary duplication, a fragmented system can also increase the costs of nutrition monitoring. For example, the current NNMRRP includes two surveys focused on dietary knowledge, attitudes, and behaviors—one operated by USDA and the other by HHS. The Interagency Board plans to review these surveys for duplication. While some overlap can be useful as a quality check on data from different sources, unnecessary redundancy in the system uses resources that could be used for currently unmet data needs—such as the need for information on specific subpopulations at risk for nutrition-related problems. The need for a coordinated system was supported by three of the four sources used to identify features of a model program. In addition to a general concern about the lack of coordination, past evaluations of federal nutrition monitoring have criticized the incompatibility of the data gathered by different enterprises. Specifically, these evaluations have called for compatible methods of assessing dietary intake, a core set of standardized measures for the major surveys, compatible sampling Page 23 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 2 A Model Nutrition Monitoring Program techniques in the national surveys, and integrated reporting. Another criticism related to coordination focused on the absence of a systematic process for determining the needs for nutrition monitoring data across the different data collection activities. Although needs are assessed for individual activities, no comprehensive assessment of needs in relation to the total system of activities has taken place. Coordination is also a major theme of the 10-year plan. Four of the six objectives discussed in the plan focus on coordinating data collection and improving the comparability of the data. Of the specific activities listed for each objective, some of those focused on coordination are: coordinating the planning for coverage, tracking, and reporting of findings from surveys and surveillance systems; identifying ways to increase comparability within a dietary method to improve the quality and usefulness of data; and establishing a mechanism for improved coordination among federal agencies that collect and use survey information about knowledge, attitudes, and behavior to assess gaps and duplications in existing surveys. The third source of support for the need for coordination comes from meetings of the advisory panels. The panelists noted that the different agencies involved in nutrition monitoring have different missions and priorities and, hence, coordination is difficult. Their major suggestions for improving coordination were to give coordination responsibility to a single lead agency, to coordinate from an interagency body with permanent staff and enforcement authority, to locate nutrition monitoring in statistical agencies within the user departments or within a central statistical agency, to centralize the congressional appropriations process for nutrition monitoring activities, and to ensure informed review of data collection plans by qualified staff at the Office of Management and Budget. Our survey queried respondents about changes to specific data collection activities, rather than the NNMRRP as a whole, so coordination was not a major theme of the comments provided by the data users. The Continuous Collection of Data The Interagency Board defines continuous data as data collection that is “repeated regularly and frequently.” Two consequences follow when data are not regularly available. First, because the kinds of foods that are available and the eating patterns of the American people change rapidly, the data become outdated quickly. Compounded by delays between data collection and data release, long intervals between administrations of the 2Ten-Year Comprehensive Plan, 58 Fed. Reg. 32806 (1993). Page 24 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 2 A Model Nutrition Monitoring Program surveys diminish the relevance of the data to the current situation and, hence, their utility in program planning and management. For example, if any of the policy changes currently being considered are implemented— such as the consolidation of food assistance programs—up-to-date information at regular intervals before and after the change will be needed to monitor any positive or negative effects of such policy changes on the population. A second consequence of pauses in data collection is that potential efficiencies of a continuous survey operation are lost. Each implementation of a national survey requires extensive planning, including reviews of the needs of data users, development and testing of data collection procedures, and all the steps involved in approving a contract. An ongoing data collection operation could streamline some of these processes. In addition, when surveys are not in the field continuously or even at dependable intervals, they may attempt to meet as many of the needs of data users as possible when they are administered. For example, the low response rates of the 1987-88 NFcs have been partially attributed to the burden on survey respondents resulting from its attempt to obtain both household and individual data with one interview. In contrast, an ongoing survey could consist of a core set of questions and rotating modules of questions that address the needs of specific users. The continuous collection of data was a theme in all four of our sources for the features of a model program. Past evaluations of nutrition monitoring have called for the continuous collection and timely release of nutrition-related data. Although the 10-year plan does not list specific activities focused on the continuous collection of data, two of the plan’s objectives indicate the Interagency Board’s concern with the timeliness of the data collected at the federal and state and local levels. The expert advisers also emphasized the need for regularly available data and suggested the following mechanisms for the ongoing collection of the information: continuous national nutrition surveys, addition of nutrition-related modules to existing surveys, reliance on program data that are already collected, and collection of longitudinal data (that is, data collected from the same sample over time). Finally, responses to our survey of the users of nutrition data not only stated a desire for continuous data, but also indicated that data that are Page 25 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 2 A Model Nutrition Monitoring Program collected at regular and frequent intervals serve some important purposes. For example, data that are currently available on a regular basis have been used to measure progress toward the Healthy People 2000 objectives and to evaluate policies such as the fortification of infant formula.° Support for Reliable Inferences About Subpopulation Groups and Small Geographic Areas The need for continued and improved information on subpopulation groups and small geographic areas is supported by several arguments. First, information is needed on specific populations known to be at risk for nutrition-related problems, such as Native Americans or homeless persons, in order to identify their needs and develop and target assistance programs. Second, some subpopulations, including Hispanics and the elderly, are growing rapidly. Their dietary patterns or nutritional needs may be different from the population as a whole; thus, information about these groups is needed to monitor their needs and to understand their effect on estimates of the prevalence of various nutritional problems in the overall population. Finally, the samples for the three major NNMRRP surveys are designed to yield national estimates. However, much of the planning for health and nutrition programs is conducted at the state and local levels. Hence, states and localities also need information on nutrition-related indicators for their populations. For the reasons given above, past evaluations of federal nutrition monitoring have criticized the program for not covering specific population groups and geographic areas. Although none of the overall objectives of the 10-year comprehensive plan focus on the need for information on subpopulation groups, several of the activities do. For example, one planned action is to develop and implement a plan for improved coverage of groups at nutritional risk. Our advisory panels also discussed the importance of information on subpopulations, noting differences in the kinds of foods consumed in different regions of the country and by different ethnic and racial groups as well as differences in nutritional needs at different ages. Their suggestions for improving the availability of data on subpopulation groups and small geographic areas included different sampling strategies for different populations, contracts with states and localities to gather information on geographically-based populations, and indirect estimation to support inferences about subpopulation groups and small geographic areas. 3The Healthy People 2000 objectives, published in 1990, are targets for improving the health of the population by the year 2000. The targets are phrased in specific terms and with reference to baseline information so that progress toward the objectives is measurable. Nutrition-related objectives address such issues as overweight, breast-feeding, consumption of fruits and vegetables, iron deficiency, and others. Page 26 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 2 A Model Nutrition Monitoring Program The availability of data on important subpopulations and states and localities was one of the changes to the current data collection activities requested by respondents to our survey of data users. The users also emphasized the importance of information on subpopulations by describing the uses supported by currently available data on subpopulation groups such as determining dietary needs of the elderly, assessing differences between blacks and whites in the effect of obesity on diabetes, informing policies on the fortification of infant and toddler foods, and targeting a blood pressure screening program to the Mexican-American population. Assistance to States and Localities for Nutrition Monitoring State and local governments are interested not only in the applicability of federal nutrition monitoring data to their jurisdictions, but also in having federal assistance in collecting and interpreting their own data. The major justification for the emphasis on state and local monitoring is the wide range of uses that states and localities have for nutrition data. The examples in table 2.2, drawn from responses to our survey of data users, illustrate the utility of existing NNMRRP data for state and local governments. While NNMRRP data collection systems meet some of the state and local needs for nutrition monitoring data, state and local officials have called for additional technical assistance in analyzing and interpreting existing sources of data and for federal support in collecting their own data. Page 27 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 2 A Model Nutrition Monitoring Program Table 2.2: Examples of State and Local Uses of Nutrition Data® a ee ee RE a ee | Purpose Specific use Problem identification Determine nutritional habits of school-age children Identify which nutrition problems are most prevalent among WIC participants Assess fruit and vegetable consumption and need for intervention Program planning or Plan cardiovascular interventions for state residents with policy-making diabetes Establish anemia as a priority issue for nutrition education Develop a program targeted toward the prevention of obesity in children Seek iron fortification of flour used in ethnic and large county bakeries Launch a “5-a-Day” program to encourage consumption of fruits and vegetables Select ethnic target groups for Preventive Health Block Grants Map anemia rates to identify highest risk areas for targeting resources Program evaluation or Monitor progress toward Healthy People 2000 objectives, program management such as improvement in rates of breast-feeding Assess access to food assistance Justify procurement of hematological equipment Expand nutrition technical assistance to county health departments aProvided by state and local officials who responded to our survey. In addition to the support provided by the survey of data users, our other sources indicated the importance of building state and local capacity for - nutrition monitoring. For example, past evaluations recommended assisting state and local nutrition monitoring activities. These recommendations are mirrored in the Interagency Board’s 10-year comprehensive plan, which clearly signals the importance of states and localities in the NNMRRP by devoting three of its six objectives to strengthening state and local monitoring activities. The expert advisers also noted the role of states and localities in nutrition monitoring, but disagreed about the responsibility states should have for data collection. Some argued for state-based data collection that feeds into a federal system, and others argued for less state responsibility for data collection, but for increased consideration of state needs in federal data Page 28 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 2 A Model Nutrition Monitoring Program collection activities. Their suggestions for assisting states and localities include providing financial assistance to states to determine their own data needs, creating a federal-state partnership in which states can provide funds for some extra sampling or extra questions on federal surveys, « developing standardized modules of interest for state data collection activities, ¢ assisting state collection of data on subpopulations, and providing technical assistance in data interpretation. The rest of the report describes the strengths and limitations of current Approaches to and potential approaches to achieving the selected model features. These Achieving the approaches are listed in table 2.3. The alternate strategies were selected Selected Model from the suggestions generated by the expert advisers using the criteria of Features responsiveness to criticisms of the current approach and feasibility. For example, the current approach to coordination—the Interagency Board—is criticized for its lack of authority over the member agencies. In contrast, an independent central authority could have influence over the NNMRRP agencies. The options were considered feasible if they were already used in other programs with similar issues (such as the lead agency approach for other cross-agency programs), past activities of the NNMRRP (such as the special study approach for information on subpopulations), or related current activities by NNMRRP agencies (such as indirect estimation). Page 29 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 2 A Model Nutrition Monitoring Program Table 2.3: Features of a Model Program and Current and Potential Approaches Model feature Current approach Potential option Coordinated system of Interagency Board and Coordination through a activities Working Groups central authority Coordination from a single lead agency Continuous collection of data State-based surveillance systems Continuous nationally-representative survey Addition of nutrition questions to existing ongoing surveys Support for reliable inferences about subpopulation groups and small geographic areas Oversampling of selected groups in the national surveys State-based surveillance systems Regular, separate studies of subpopulations Development of indirect estimation programs Assistance to states and localities Page 30 State-based surveillance systems Community-based nutrition monitoring GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program ''Chapter 3 Approaches to Coordinating the Nutrition Monitoring Program As described in chapter 2, a model program would have a mechanism for coordinating the various nutrition monitoring activities to maximize the utility of the data and minimize the costs of its collection. This chapter first reviews the status of current NNMRRP activities to improve coordination. Then, two other possible coordination mechanisms— coordination by a central authority and coordination by a single lead agency—are examined. (Table 3.1 provides an overview of the strengths and limitations of the various approaches to coordination. ) LT a a TE UO US OPE Er ET ap PI Table 3.1: Approaches to Coordinating Nutrition Monitoring Activities Approach Strength Limitation Current: Interagency Board and Working Groups Improved communication and cooperation among the agencies Compilation of a coordinated budget report Difficulties setting priorities across agencies Inadequate assessment of the needs for data by different kinds of users Difficulties in disaggregating the costs of the nutrition component of multipurpose programs Limited authority, staff, and financial resources for coordination Potential Coordination through a central authority Increased ability to establish priorities Alignment of agency priorities with overarching goals through centralized budget reviews A central contact point for state and local governments, interest groups, Congress, and other federal agencies Conflict between research concerns and policy concerns Difficulties in disaggregating the costs of multipurpose programs Limited staff and financial resources for coordination Coordination from a single lead agency Improved communication and cooperation among the agencies Compilation of a coordinated budget report A central contact point for state and local governments, interest groups, Congress, and other federal agencies Difficulties setting priorities across agencies Difficulties in disaggregating the costs of multipurpose programs Limited authority, staff, and financial resources for coordination Decreased attention to the data needs of other agencies Current Approach: the Interagency Board Page 31 As described in chapter 1, the NNMRRP meets multiple needs for nutrition-related data. Yet, historically, these needs have not been met by GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 3 Approaches to Coordinating the Nutrition Monitoring Program an integrated program with the capacity for evaluating data needs and making adjustments as those needs change. Instead, a fragmented system of activities developed over the decades as new needs for nutrition data were identified. For example, in the early 1930s, uspA developed its first national survey of household food consumption because data on the food supply provided no information about the distribution of food at the household and individual levels. Similarly, the nutrition component was added to the National Health Examination Survey in the early 1970s in response to a need for more information about hunger, and state-based surveillance systems were established in recognition of the primary role of states in providing services to populations at risk of nutritional problems. To address concerns about the lack of coordination across the agencies involved in nutrition monitoring, the National Nutrition Monitoring and Related Research Act of 1990 required the Secretaries of Agriculture and Health and Human Services to implement a coordinated program of nutrition monitoring. The act specified several tools: an Interagency Board, the development of a comprehensive plan for the program, a council of outside advisers, and an integrated budget. The Interagency Board created by the act has the difficult task of coordinating numerous data collection and analysis activities across several agencies that have traditionally had separate and distinct missions and operations. The Board has two chairpersons, one selected by the Secretary of Agriculture and one by the Secretary of Health and Human Services. For uspA, the chair is the Under Secretary for Research, Education, and Economics. The HHS chair is the Assistant Secretary for Health. Membership on the Board includes representatives of various agencies in USDA and HHS, as well as the Bureau of the Census, Agency for International Development, Bureau of Labor Statistics, Department of Defense, Department of Veterans Affairs, and Environmental Protection Agency, among others. The Executive Secretary for the Board rotates between USDA and HHS every 2 years. To facilitate coordination, the Interagency Board established working groups focused on survey comparability, food composition, and federal-state linkages and information dissemination. The Secretaries and the Interagency Board were charged with developing a 10-year strategic plan, which was published in the Federal Register on June 11, 1993. The plan outlines a set of planning activities, including a general time frame and the lead agencies for each activity. The activities are organized around six objectives, listed in chapter 1 (see p. 14). The Page 32 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 3 Approaches to Coordinating the Nutrition Monitoring Program Interagency Board clearly recognizes the need for improved coordination since four of these objectives focus on either the coordination of the data collection activities or the comparability of the data. To advise the Board on the development and implementation of the NNMRRP, the act established the National Nutrition Monitoring Advisory Council. The members of the Council represent academic institutions and other interested parties drawn from outside the federal government. The act also required the Interagency Board to submit annually a coordinated budget for nutrition monitoring. Strengths Both the concern that preceded passage of the act and the structure it created appear to have improved communication and cooperation among the agencies. The Board and its working groups provide mechanisms for communication and joint decision-making. Specific actions that demonstrate the increased coordination include the development of common population descriptors for use in conducting and reporting the 1994-96 csFi and the next NHANES, a marketing and distribution plan for NNMRRP reports, an automated dietary intake interview that would facilitate timely data release and linkage across CSFII and NHANES, and a common set of questions on food security (a concept that addresses the certainty about having enough to eat) to be used in the Current Population Survey. In addition, a jointly funded research project explored the possible linkage of CSFII and NHANES sampling plans. Alternate sampling designs were evaluated using the criteria of (1) ability to satisfy the separate objectives of the two surveys, (2) benefits in overall costs or analytic power, and (3) feasibility, especially in terms of burden on the survey respondents. The draft report from the contractor on the project emphasized the compromises one or both surveys would have to make to link their sample designs.! For example, while the combination of the NHANES and CSFII into a single survey could yield a rich database of information on diet and health, the likely increase in respondent burden could reduce response rates and response quality. Another alternative—using linked samples for NHANES and csFii—could decrease CSFIl’s precision because NHANES’ sampling design is determined by the survey’s reliance on mobile examination 1An Evaluation of Linked Survey Designs for NHANES and CSFIl, prepared for NCHS by Westat, July 29, 1994. Page 33 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 3 Approaches to Coordinating the Nutrition Monitoring Program centers to conduct physical exams of the respondents.” The Interagency Board concluded that the two surveys should remain independent, although work on improving the comparability of the data should continue. As required by the act, the Interagency Board submits an annual budget for the NNMRRP to the Congress. The budget report includes costs allocated for data collection, related research, information dissemination and exchange, and technical assistance; however, these different types of costs are not distinguished. Instead, the funds dedicated to nutrition monitoring and related activities are reported only by agency, not by type of activity. The budget report for fiscal years 1994-96 indicated that a total of $157.7 million was dedicated to nutrition monitoring or related research in 1994. Of that, $30 million was reported by the Centers for Disease Control and Prevention (which has responsibility for the HHS surveys and state-based surveillance systems) and $9.3 million was reported by the Human Nutrition Information Service (which had responsibility for NFcs and csFil).? The remainder was accounted for by agencies whose primary involvement in nutrition monitoring is related research. For example, HHS’ National Institutes of Health reported $25.9 million dedicated to NNMRRP activities and UsDa’s Agricultural Research Service accounted for $50.7 million. The budget report is useful in communicating to the Congress a general sense of the cost of the NNMRRP across agencies. However, funds for nutrition monitoring cannot always be disaggregated from other purposes of the data collection and research programs. As a result, the budget report contains only approximate amounts dedicated to nutrition monitoring and related research. Moreover, with the recent incorporation of the office responsible for NFcs and CSFII into ARs, determining which funds are dedicated to the monitoring activities and which are used for related research will be more difficult. Limitations The literature on the development of objectives to increase accountability for program results indicates that (1) objectives should be written in terms that can be used to judge progress toward achieving them and (2) implementation plans and specific measures of progress should be ?HHS maintains only three of these expensive mobile units and, as a result, has a sampling design that minimizes the geographic dispersion of respondents. However, HHS is reviewing alternatives to the mobile examination centers. ’The Human Nutrition Information Service and its functions have been merged with the Agricultural Research Service (ARS). Page 34 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 3 Approaches to Coordinating the Nutrition Monitoring Program developed for the goals and objectives.‘ In the 10-year plan, the objectives are stated in general, global terms so that it is not clear when an objective can be considered achieved. For example, the first objective is to “provide for a comprehensive NNMRRP through continuous and coordinated data collection.” Neither the objective itself, nor the text following it clearly defines what. the terms “comprehensive” or “coordinated” mean. Similarly, the activities are too vague to be considered implementation plans for the objectives; for example, no activity directly relates to the development of continuous data collection. As another illustration, the activity, “identify ways to increase comparability within a dietary method to improve the quality and usefulness of data,” specifies neither the degree of quality required nor the uses that will be facilitated. Without more concrete, measurable objectives, there is little accountability for the program because progress toward the objectives cannot be assessed. (Other examples of activities listed in the 10-year plan can be found in table II.2 in appendix II.) In addition, the plan did not describe how activities would be ranked by importance or addressed within current fiscal constraints. However, since the plan was published, the Interagency Board drafted an approach for ranking the 68 activities listed in the plan into three categories of priorities: (1) essential (mandatory, legislatively required), (2) necessary (critical but not mandatory), and (3) beneficial. An interagency implementation group, involving around 60 agency representatives, applied the approach. Twenty-five of the activities were ranked in the high-priority category, 33 were ranked as next most important, and only 10 were ranked as beneficial but not critical. While this is an important first step in setting priorities, the Interagency Board has not yet linked the top-ranked tasks to the costs and benefits of completing them. Such a framework could be used to understand the trade-offs in selecting one or another approach to each task. Since an assessment of the effectiveness of the different approaches to the objectives requires information on the uses of the data, another obstacle to the Interagency Board’s implementation of a coordinated system is the absence of a comprehensive assessment of how nutrition monitoring data are used across the federal government and by data users in other settings. 4Performance Measurement: An Important Tool in Managing for Results (GAO/T-GGD-92-35; May 5, 1992). Page 35 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 3 Approaches to Coordinating the Nutrition Monitoring Program Potential Options For example, although UsDA and HHs held a joint workshop to assess the needs of users of dietary intake data in August 1994, the workshop included only representatives of federal agencies. Finally, the ability of the Interagency Board to coordinate the NNMRRP is limited by the lack of resources dedicated to coordination. The Interagency Board has no staff, although two people—one from USDA and one from HHS—have been given primary responsibility for organizing NNMRRP activities. The NNMRR Act gives the Secretaries the option of appointing an administrator for the program, but so far they have chosen not to exercise that option. While recognizing the progress made by the Interagency Board, we also considered other coordination mechanisms. Two options for improving coordination—through a central authority or by a lead agency—are reviewed in detail in this section. In addition, other options discussed by our expert advisers are briefly presented. Coordination Through a Central Authority The suggestion of coordination through a central authority came in response to the lack of enforcement power held by the Interagency Board over the member agencies. The kind of central, coordinating agency envisioned by the expert advisers is most clearly exemplified by the executive offices in the White House. Therefore, to examine the advantages and disadvantages of having a central authority provide coordination, we reviewed the literature (including prior GAO reports and congressional hearings) and interviewed agency officials in three White House offices that have coordination responsibilities: Office of Management and Budget (OMB), Office of Science and Technology Policy (osTP), and Office of National Drug Control Policy (oNDcP). As indicated by the brief descriptions of each office provided in table 3.2, the coordination tools used by the White House offices are similar to those used by NNMRRP’s Interagency Board, including interagency committees and working groups, development of plans, and review of budgets. Page 36 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 3 Approaches to Coordinating the Nutrition Monitoring Program Table 3.2: Responsibilities and Coordination Tools of Three White House Offices Potential Strengths aE I Office Responsibility Tool OMB Ensure coordination and Review and approve agency cooperation where federal agency budgets lines of authority overlap Review and approve data Assist in the development of collection instruments efficient coordinating mechanisms Expand interagency cooperation OSTP Coordinate research and National Council on Science and development activities in other Technology (NCST), which is agencies chaired by the President and includes Cabinet Secretaries and Review and participate in the selected agency heads annual formulation of the budgets related to science, research, and § NCST subcommittees and working development groups Review of agency budgets for interagency programs Commissioning studies of controversial issues ONDCP Through the National Drug Control Interagency working groups Strategy, establish policies, objectives, and priorities for federal drug control programs Coordinate federal budget for drug activities Review of agency plans for implementing the National Drug Control Strategy Review and certification of drug control budgets One potential advantage of elevating the coordination of nutrition monitoring to a high-level central authority is the increased participation of high-level officials in coordination activities. This participation could, in turn, increase the ability of the coordinating body to establish priorities. Currently, some of the agencies participating in the Interagency Board are represented by administrators or directors, while other agencies are represented by staff members who have no authority to change agency activities or establish priorities. In contrast, the political visibility of a program under White House management could encourage agencies to send representatives in positions of authority, capable of establishing priorities and committing resources to support them. The political visibility that comes with the participation of a high-level central authority may also contribute to the effectiveness of the various coordination tools. For example, the Interagency Board compiles budgets Page 37 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 3 Approaches to Coordinating the Nutrition Monitoring Program Potential Limitations obtained from each of the agencies involved in nutrition monitoring without reviewing them for consistency with the activities identified in the 10-year comprehensive plan. In contrast, the White House offices use the budget review process to bring activities of the agencies into line with overarching policy goals. For example, onpcP can threaten to decertify an agency’s drug budget if it is not consistent with the National Drug Control Strategy. Decertification has no practical ramifications, but it sends a politically important message about the priority given to drug control activities by the White House. An additional potential advantage of having a central authority provide coordination is that it can be a central location for assistance to all data users. For example, ONDCP has a Bureau of State and Local Affairs that works with state and local government agencies involved in drug control activities. The Bureau serves as a clearinghouse for information about state and local activities and uses conferences to increase communication with and among state and local officials. In addition, the Bureau can communicate the concerns of state and local governments and community groups to the federal agencies involved in drug control programs. For nutrition monitoring, such an office could provide a central contact point for users in a variety of settings, including federal, state, and local governments, food industry, and health care organizations. Although the increased political visibility of a high-level central coordination office may facilitate the development of priorities, it may also increase the potential for political pressure on the data collection and research. A conflict between ONDCP and HHS over the data collection and reporting of drug data illustrates this issue. HHS was concerned about the degree of ONDCP’s involvement in how the data were collected and reported, while ONDCP expected HHS to meet its data needs. Political influence on the scientific agenda is also a concern for OSTP, where priorities may change with changing administrations even if scientific issues remain the same. In addition, although White House offices have more influence over the budgets reported by the different agencies, the budgets developed for other programs that cut across agency jurisdictions share the limitation of the coordinated NNMRRP budget. Specifically, agency activities may serve multiple purposes, thus making it difficult to determine how much of the overall costs are dedicated to the interagency program. For example, Coast Guard patrol boats serve drug interdiction purposes, but are also used in search and rescue missions that are not related to drug control. Page 38 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 3 Approaches to Coordinating the Nutrition Monitoring Program Thus, the Coast Guard can only estimate the portion of its resources dedicated to supporting national drug control efforts. Ambiguity about the portion of a multipurpose program that serves the interagency purpose makes it difficult to monitor the costs of the program. Although central authorities like the White House offices are financed separately from the agencies they oversee, resource limitations persist. For example, HHS staff attributed part of their conflict with ONDcP to a lack of technical and substantive expertise among ONDCP staff. Similarly, a past director of ostTP identified limited staff resources as a reason why long-term planning received less attention than short-term problems. Coordination Through a Lead Agency Potential Strengths Potential Limitations Another possible approach to coordinating the NNMRRP is locating the responsibility within a single agency. To investigate this approach, we discussed the option with our expert panels and used program documents and evaluations conducted by GAO and the Congressional Budget Office to examine the experience of the High Performance Computing and Communications Program (HpccpP).° Like the NNMRRP, the HPCCP involves multiple agencies with different strengths and missions. Unlike the NNMRRP, the oversight of the HPCcP is located in osTP, which delegated the responsibility for coordinating the activities across the agencies to a single agency, the National Library of Medicine. While not a major player in high-performance computing, the National Library of Medicine was seen as an independent, unbiased participant with interest in and knowledge about the technology. The Library of Medicine’s role has been to pull together materials for program reports, convene meetings, and provide a clearinghouse. HPCCP’s National Coordination Office shares two of the strengths of the Interagency Board: It appears to have facilitated communication among the agencies and coordination of individual activities. In addition, it has provided the Congress with a budget that looks at the costs of high- performance computing activities across agencies. Moreover, the National Coordination Office has the added advantage of providing the Congress and the public with a central contact point for information on high-performance computing. Just as the HPCCP shares the NNMRRP’s strengths, it also has some of its limitations. Specifically, no mechanism is in place to set priorities across *The purpose of HPCCP is to further the development and dissemination of U.S. supercomputer and high-speed computer network technologies. Page 39 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 3 Approaches to Coordinating the Nutrition Monitoring Program different agency activities. In a recent review, GAO suggested that the HPCCP needs “an explicit technical agenda, identifying and prioritizing specific technology challenges and establishing a framework of expected costs and results . . . [to] clarify the program’s goals and objectives, focus efforts on critical areas, and serve as a baseline for measuring program progress and results.”® In addition, HPccP did not have uniform guidelines for which research activities should be included in the budgets submitted by the different agencies, mirroring the difficulty the other coordination mechanisms have in tracking funds used for multipurpose programs. The National Coordination Office also shares the NNMRRP’s lack of budget and staff resources for coordination. An additional concern about the lead agency model was raised by our expert advisers: If responsibilities for nutrition monitoring were located in one agency, nutrition monitoring might become a monopoly, serving the needs of only one agency. Its current dispersion across agencies allows the different components to serve different purposes. However, safeguards—such as that used for the HPpccP when it was located in an agency that did not have a large investment in high-performance computing, relative to some of the other agencies—could address this concern. Other Coordination Suggestions In addition to coordination by an interagency body with permanent staff and enforcement authority or a single lead agency, other suggestions for improving coordination of the NNMRRP were to locate nutrition monitoring in statistical agencies within the user departments, centralize the congressional appropriations process for nutrition monitoring activities, and ensure informed review of data collection plans by qualified staff at the Office of Management and Budget. The first idea—locating nutrition monitoring in statistical agencies within the departments that use the data—was intended to focus the program on the quality of the data collected. However, a concern about this suggestion was that it could result in decreased responsiveness to the needs of the data users. We did not pursue it because the major surveys are already located in statistical or research branches of HHS and USDA. (NHANES is ®High Performance Computing and Communications: New Program Direction Would Benefit From a More Focused Effort (GAO/AIMD-95-6; Nov. 4, 1994), p. 22. Page 40 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 3 Approaches to Coordinating the Nutrition Monitoring Program Conclusions operated by the National Center for Health Statistics, and NFcs and CSFII were recently relocated to ARS.) Similarly, oMB already has responsibility for reviewing the data collection activities of the agencies, although OMB staff reported that they rely on the agencies to describe coordination efforts. Finally, we did not review the advantages and disadvantages of centralizing the congressional appropriations process for nutrition monitoring activities because its relevance to the utility and efficiency of the data collection activities was not clear. Lack of coordination across program activities has implications for both the effectiveness and the efficiency of the NNMRRP as a whole. The Interagency Board has made progress toward coordinating activities across the different agencies. However, the other approaches to coordination suggest mechanisms that could further strengthen the work of the Interagency Board. First, because its responsibilities are shared by USDA and HHS, the Interagency Board does not provide a central contact point for users of nutrition monitoring data. A central contact could be established if the Secretaries of USDA and HHS used their option to appoint an administrator of the program or if responsibility for responding to requests for information about the program was assigned to a single agency, as it was for high-performance computing. Second, the Interagency Board does not review the agency budgets it compiles for consistency with the overarching priorities of the program. Before such a review could occur, specific objectives and priority activities would need to be identified. Then, the Interagency Board could work with OMB to secure funding of NNMRRP priorities. Page 41 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program ''Chapter 4 Approaches to the Continuous Collection of Data Evaluating programs intended to reduce diet-related chronic disease, tracking progress toward health objectives, and monitoring changes in our diets are the kinds of activities that require continuous data. The advantages and disadvantages of the current and alternate approaches to collecting continuous data are summarized in table 4.1 and detailed below. Table 4.1: Approaches to Providing Continuous Data Approach Strength Limitation Current: state-based surveillance systems Quick availability of data on specific issues Connection between data elements and program decisions Efficient relative to national surveys because of the reliance on program records (PedNSS and PNSS) or telephone interviews (BRFSS) No in-depth food intake or health data Uneven quality and completeness of data across states Limited to populations participating in publicly funded programs (PedNSS and PNSS) or living in households with telephones (BRFSS) Incomplete participation of states in PedNSS and PNSS Potential A continuous, nationally-representative survey Up-to-date information on the population Increased flexibility to respond to new issues by adding modules to a core set of data elements Increased efficiency as start-up costs diminish Costs of conducting all survey processes concurrently Difficulties in identifying a core set of data elements Inflexibility of core data elements Addition of nutrition questions to existing ongoing surveys Efficient means of gathering data at frequent intervals Dependent on flexibility of surveys Current Approach: the State-Based Surveillance Systems Although the provision of continuous data is one of the objectives identified in the Interagency Board’s 10-year plan, not one of the three major NNMRRP surveyS—NFCS, CSFII, or NHANES—is implemented continuously. In fact, planned future administrations of two of the surveys have been postponed, potentially compromising the ability to monitor trends in diet-related health risks over time and evaluate the effect of any changes in food assistance policy. NFCs has been administered at approximately 10-year intervals since the 1930s. Its next implementation was planned to begin in 1996, but is now tentatively scheduled for 1998, depending on funding. CsFI, originally Page 42 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 4 Approaches to the Continuous Collection of Data intended to provide continuous data on dietary intake, has had three separate administrations: 1985-86, 1989-91, and 1994-96. After a l-year pause in 1997, it is expected to resume for another 3-year period in 1998. Since the National Health Examination Survey gained a nutrition component in 1971, NHANES has been fielded three times: 1971-75, 1976-80, and 1988-94. Like NFCS, its future implementation is uncertain because of budget constraints. Its planned implementation in 1997 is now expected to be postponed. Because of the lack of certainty about the implementation of the national nutrition surveys, the state-based surveillance systems are currently the primary source of continuous data in the NNMRRP. As described in chapter 1, PedNSS and PNSS rely on data from clinic records from publicly funded health, nutrition, and food assistance programs and BRFSS collects information through telephone interviews, with respondents (adults 18 years and over) identified through random digit dialing. While a valuable source of quick information for state and local program managers, the surveillance systems do not meet the needs of researchers or program decisionmakers who require either national data or in-depth food intake data. Strengths Compared to the national surveys, one of the strengths of the state-based surveillance systems is that they not only provide data continuously, but they are also able to process and report the data relatively quickly. For PedNSS and PNSS, information is collected as part of the process of receiving services from wic and other publicly funded health, nutrition, and food assistance programs. Because they depend on program records, PedNSs and PNss do not burden respondents the way surveys dependent on interviews do. The information is transmitted from the records of local health and nutrition programs to the state, which then forwards the records to HHs for analysis. Similarly, the data collected by the states for BRFSS are sent to HHS for processing. According to HHs officials, all three systems report data back to the states within a year and generally in less than 9 months. HHS has also helped states conduct their own analyses by distributing a standardized software package. Another strength of the state-based surveillance systems is that the data they collect are directly linked to program decisions. Although PedNss and PNSS include only a few indicators of nutritional deficiencies and behaviors, they are selected to support state data needs for program planning and management. For example, the data are used to target Page 43 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 4 Approaches to the Continuous Collection of Data resources for the WIC program. Similarly, BRFSS data have been used to inform decisions about nutrition education programs, such as campaigns to encourage the consumption of five servings of fruits and vegetables a day. Limitations In contrast to the national surveys, the surveillance systems do not permit examination of diverse diet-health associations across the entire population. Instead of collecting extensive biochemical, anthropometric, and interview data, they focus on a narrow range of variables relevant to specific programs or nutritional risks. For example, PedNss collects clinical data on weight and height, monitors infant feeding practices, and assesses anemia. PNSS collects information on anemia and behaviors associated with low-birthweight babies. BRFSS asks respondents to report on their consumption of fat, fruits, and vegetables. This focus limits the breadth of uses that can be supported by the data; however, as noted above, it also limits the burden placed on respondents. While the systems are currently limited in the amount of dietary data they collect, HHS is exploring other methods of gathering these data. For example, with HHS support, the University of Texas examined the use of bar code data to look at dietary patterns. They concluded that the technology is not yet ready for use, but through a partnership with food manufacturers could be a promising method for the future. In addition, with USDA, HHS is evaluating the feasibility of collecting additional dietary data in the clinics that provide the PedNss and PNSS records. Another concern about surveillance systems is the quality and completeness of the data across the different states. For example, for PedNSS, error can be introduced by variations in practice in weighing infants, such as with or without the baby’s winter clothes, or by clerical errors in entering the data in states that do not have automated data systems. PNss, which attempts to collect a wider range of information than PedNSS, suffers from missing data on several variables, such as the pregnancy risk factors of smoking and alcohol consumption. However, HHS provides technical assistance to help states standardize their data collection procedures. In fact, by flagging biologically implausible values for the physical measures, HHS analyses of the surveillance data help identify clinics that may have poor procedures. The surveillance systems are also limited because, within the participating states, only certain groups of the population are covered. As described Page 44 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 4 Approaches to the Continuous Collection of Data Potential Options earlier, PedNSS and PNSS primarily provide information on mothers and children participating in the wic program. As a result, data may not be available on other populations that are potentially at risk, such as homeless people or older children not eligible for wic. In contrast, BRFSS has a wider target population, collecting data from randomly selected adults 18 years and over. However, neither adults in households without telephones nor children are covered by BRFSS. Since there is evidence that some health risk factors, such as smoking, are associated with living in a household without a telephone, this could affect estimates of the extent of diet-related risk factors as well. Finally, not all states participate in the PedNss and PNSS surveillance systems and, therefore, they are not good sources of national-level data on their populations of interest. In 1993, 38 states participated in PedNSs and only 20 states participated in PNss. (In contrast, BRFSS has good state coverage; in 1993, only Wyoming did not participate.) In addition to the state-based surveillance systems, we reviewed two other approaches to providing continuous data. The first option is a national nutrition-related survey that is operated continuously. To describe the strengths and weaknesses of this approach, we consulted with our expert advisers and interviewed managers of the current national nutrition monitoring surveys about the continuous operation of the surveys. The second option is the inclusion of nutrition-related questions on continuous nOn-NNMERP surveys. The strengths and weaknesses of this alternative were explored by examining the NNMRRP’s recent experience developing food insecurity questions for the Current Population Survey (cps). In addition, our expert advisers and the data users who responded to the survey made other suggestions, which are also briefly presented. A Continuous NNMRRP Survey An ongoing national survey is a possible approach to providing continuously updated information that addresses some of the limitations of the state-based surveillance systems. One of the current national surveys, the CSFII, was developed in response to calls for the continuous collection of individual data on food intake. It has not been implemented regularly in the past; however, current plans are for the ongoing implementation of CSFII as a 3-year survey followed by a 1-year pause for planning and development before the next 3-year period. Page 45 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 4 Approaches to the Continuous Collection of Data Potential Strengths Potential Limitations An ongoing survey has the potential to yield continuously updated, timely data for monitoring the nutritional status of the nation’s population. The increased timeliness of the data could decrease the current pressure from collaborating federal agencies to include components on each implementation of the periodic surveys. For example, in contrast to the recent implementation of NHANES, which attempted to meet as many data needs as possible within a single survey, the ongoing implementation of NHANES could contain a core set of data items that would be collected continuously, supplemented at intervals by rotating modules. Such a streamlining would also reduce the burden on respondents. Moreover, the surveys could become more flexible by distinguishing between variables that change rapidly, variables that need regular but not continuous monitoring, and variables of emerging policy importance. The core set of items in continuous implementation could gather information on rapidly changing variables. Topical modules on issues that do not change as rapidly could be included on a regular schedule. Finally, as new issues arise, additional questions could be added. In addition to increased timeliness and flexibility, a continuous survey operation could be more efficient than periodic surveys because current costly start-up activities, such as planning, designing sampling strategies, and training interviewers, would be diminished. Thus, the data could be collected for less cost per respondent. However, as described below, without concomitant streamlining of the survey, overall costs could increase. While an ongoing survey could save money on start-up costs, it could increase costs overall because activities that are now funded and staffed sequentially would require a continuous flow of resources to be conducted concurrently. For example, as NHANES is currently implemented, staff change their activities as the survey moves through the phases of planning, implementation, analysis, and dissemination. If NHANES was implemented continuously, all of these activities would be going on at the same time. According to both usDA and HHs officials, the primary constraints on the continuous operation of their surveys are the need for dependable funding and sufficient staff resources. As described above, the absence of dependable funding has affected the frequency with which the national surveys are now implemented. The development of a survey that continuously collects data on a core set of items and intermittently collects data on other issues raises two difficult Page 46 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 4 Approaches to the Continuous Collection of Data issues. First, the definition of the core items is complex. An expert panel convened by the Federation of American Societies for Experimental Biology was charged with identifying a set of core indicators to assess the nutritional status of difficult-to-sample populations. The report summarizing the work of the panel noted that (1) the suitability of an indicator changes with the purposes for the data and (2) information on the determinants and the consequences of each indicator is also needed.! The panel ended by identifying three sets of indicators—minimal, intermediate, and comprehensive—without recommending specific measures for the indicators. The second issue with a continuous survey is the potential inflexibility of the core items once they are selected and implemented. While opportunities to test new methods are enhanced because the survey is in the field continually, making changes to the data collection procedures can be difficult because of the pressure to ensure that the measures are consistent over time. Without such assurance, changes in an indicator such as obesity may be the result of changes in how obesity is measured rather than changes in the prevalence of the condition itself. Our expert advisers suggested a survey with built-in periods of transition to allow the survey to incorporate new methods and new data elements as they emerge. Adding Nutrition Questions to Existing Continuous Surveys A third approach to obtaining frequently up-dated information is the addition of nutrition-related questions to existing continuous surveys. The potential of existing surveys to provide data regularly on some variables is demonstrated by plans that included NNMRRP questions on food security in the April 1995 Current Population Survey conducted by the Census Bureau.” Food security is a concept intended to go beyond the idea of hunger to measure the availability of food for a family or individual. The food security questions were developed by an interagency working group cochaired by HHs’ National Center for Health Statistics and usDa’s Food and Consumer Service. While the working group developed questions, USDA reserved space on the cps for 1995. The question development process included determining how the data would be used and soliciting input from both federal and other data users. The food security module contains both core questions and supplemental ones. The 1.4 Anderson, ed., “Core Indicators of Nutritional State for Difficult-to-Sample Populations,” Journal of Nutrition, 120:11S (Nov. 1990), 1559-99. The CPS is a monthly survey of about 60,000 households across the country. Page 47 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 4 Approaches to the Continuous Collection of Data Potential Strengths Potential Limitations Census Bureau will include the complete module. NNMRRP surveys could include the smaller core set of questions. If the initial implementation of the questions yields useful data, USDA plans to continue to support the inclusion of the food security questions annually in the cps. The same strategy of piggy-backing nutrition-related questions on continuous surveys could be used with other surveys, such as the Survey of Income and Program Participation or the National Health Interview Survey. The latter has the added advantage of collecting health data that could be linked to nutritional indicators. The major advantage of this approach is the efficiency with which data on specific issues can be gathered. USDA will pay for the cost that the food security questions add to the cps without having the responsibility or the cost of fielding and managing the survey itself. A potential disadvantage of this approach is that, just as core questions on nutritional status could be inflexible, existing continuous surveys can be hard to change because of their momentum. Moreover, existing surveys have their own set of constraints and limitations. For example, while cps could accommodate the addition of questions about food security, it probably could not accommodate a module obtaining data on an individual’s dietary intake over the last 24 hours. A 24-hour recall instrument requires considerable training to administer and adds substantially to the burden on the respondent. For these reasons, opportunities to piggy-back nutrition-related questions on other surveys may be limited. Other Options for Continuous Data In addition to the approaches reviewed, other actions were suggested by our expert advisers and data users who responded to our survey. Specifically, longitudinal surveys were suggested as an efficient way to collect data over time because a new sample does not have to be selected every time a survey is fielded. Since original respondents are followed up in subsequent administrations of the survey, longitudinal surveys can also be useful for tracking individual-level changes in food consumption behavior. However, longitudinal surveys have their own costs, including the need to collect additional data so that respondents can be found for later surveys and the likelihood of attrition as respondents either drop out or cannot be found. Page 48 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 4 Approaches to the Continuous Collection of Data Conclusions Another suggestion was the collection of survey data using automated survey technology. Because direct entry of the data into an automated system can speed the processing and aggregation of the survey, it can also accelerate the release of the data for analysis. However, it does not affect the regularity with which the data are collected in the first place. HHS already uses automated data collection for NHANES. USDA is planning to automate the next administration of the CSFII. The funding constraints that have caused the postponement of two of the national surveys jeopardize the availability of periodic data on the population as a whole. Although the state-based surveillance systems provide continuous information, they are inadequate to meet the need for data on the population at large or on in-depth nutrition and health status. Moreover, if approved, proposals to collapse funds for the wic program with other food assistance programs into a block grant for the states could affect the major source of data for two of the state-based surveillance systems, PedNSS and PNSS. Although the NNMRRP has pursued such creative solutions as including food security questions on the cps, the availability of up-to-date information could worsen. To decide how best to meet the needs for continuous data in the future, the NNMRRP would first need to analyze the purposes that require frequently collected data and the current mechanisms for supporting those purposes. Within this framework, the strengths and limitations of the different approaches to increasing the frequency with which important indicators are measured could be weighed according to which purposes are supported and which are diminished. Page 49 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program ''Chapter 5 Approaches to Supporting Reliable Inferences About Subpopulations Subpopulations can be defined by geographic location as well as by age and sex (such as infants or elderly women), physiological characteristics (such as pregnancy), ethnicity or race (such as Hispanic or Native American), income, and the intersection of any of these groups (such as low-income children). As described in chapter 2, information on subpopulations is needed to appropriately target and evaluate programs that address nutrition-related programs. As summarized in table 5.1, this chapter describes current approaches and some potential options to responding to the calls for better information on subpopulation groups and small geographic areas in the NNMRRP. Other issues of federal assistance to states and localities for nutrition monitoring are discussed in chapter 6. Page 50 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 5 Approaches to Supporting Reliable Inferences About Subpopulations Table 5.1: Approaches to Supporting Inferences About Subpopulations Approach Current Oversampling of selected groups in the national surveys State-based surveillance systems Strength Less costly than a separate survey Ability to compare the subpopulation to the general population Collection of data on high-risk populations: PedNSS (low-income children) and PNSS (low-income pregnant women) Collection of data representative of the state population of adults, 18 years or older (BRFSS) Potential expansion to include programs that serve other at-risk populations Limitation Appropriate only for special populations that are covered by the national sampling frame (such as children in low-income households) Primarily appropriate for populations that are geographically clustered or large Costs of screening for members of the group No in-depth food intake or health data Uneven quality and completeness of data across states Incomplete participation of states in PedNSS and PNSS Potential Special studies Ability to tailor data collection instruments and content to the subpopulation Collection of data on subpopulations that are not adequately covered by the sampling frame used for national surveys Inability to compare to the rest of the population unless the special study is conducted in tandem with a national survey using comparable instruments Costs of tailoring data collection instruments, developing the sampling frame, and screening Difficulties in developing a sampling frame that allows for generalization Development of indirect estimation programs Efficient means of generating estimates for subpopulations Historical use in other areas (estimates of population, crop yields) Response to limitations of other sources of information, such as cost and relevance Difficult to assess the quality of the estimate Diminished confidence in the estimates because they are based on models rather than direct observation Current Approaches Subpopulation groups are covered in two ways by current NNMRRP activities. First, the three national surveys use oversampling of certain groups to ensure the selection of enough respondents to support subpopulation estimates. NHANES focuses on racial and ethnic Page 51 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 5 Approaches to Supporting Reliable Inferences About Subpopulations subpopulations, while csri and NFCs have included subpopulations defined by income, reflecting uspa’s focus on food assistance to low-income populations. The second way in which data on subpopulations are gathered is through the state-based surveillance systems—PedNSs, PNSS, and BRFSS. Oversampling Strengths Limitations Oversampling includes members of subpopulation groups in asample at a rate greater than their proportion in the population. The purpose of oversampling is to ensure that data will be collected on enough group members to support inferences about the group as a whole. Oversampling is already used in the major NNMRRP surveys. NHANES Ill (1988-94) oversampled non-Hispanic blacks and Mexican-Americans, as well as persons 60 years or older and children 1-5 years old. NHANES staff indicated that two groups (Hispanics and persons 75 years and older) are likely to be important groups in the next implementation of the survey because both are growing and have significant health-related issues to study. In addition to estimates of the general population, the current administration of CSFII (1994-96) is expected to produce estimates for low-income populations through oversampling. Also, staff of both UsDA and HHS surveys stated that the national surveys could oversample a state’s population, but that the state would have to finance the added costs. Oversampling has two major strengths. First, because it can be used in conjunction with a national survey, it has efficiencies of scale. Specifically, the planning and implementation costs are diminished because they are part of a larger survey. Second, data on both the group and the rest of the population are collected at the same time and with the same survey procedures, facilitating the comparison of the two population groups. Oversampling implies that the subpopulation group of interest is included in the sampling frames used to identify participants in the national surveys.! However, not all special populations are well covered by a national sampling frame. For example, homeless individuals, persons who live in institutions, and American Indians and Alaska Natives living on reservations would not be included in the national household sampling frame. Even for those individuals who are included in the sampling frame, oversampling may not be appropriate because of the costs incurred in The sampling frame is the group of units from which a sample is actually drawn. So, for example, both NHANES and CSFII sample from a sampling frame of households in the 50 states and the District of Columbia. Page 52 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 5 Approaches to Supporting Reliable Inferences About Subpopulations screening for members of the group. Screening is the process of asking questions at sampled households to identify whether they represent (or include representatives of) the subpopulation of interest. Screening adds to the costs of the survey because enough households have to be sampled and screened to identify the smaller number of households or individuals that meet the definition of the subpopulation. To reduce screening costs, oversampling is most effective for subpopulations that are geographically clustered or fairly well represented in the general population, such as persons with low income who are often clustered by neighborhood. In contrast, oversampling is not appropriate for groups that are few in number or geographically dispersed, such as pregnant women. A possible response to some of the limitations of oversampling is the use of multiple sampling frames. For some subpopulations, alternative frames or lists may be available that can be used in conjunction with the national sampling frame. Samples can be selected from both the subpopulation frame and the general population frame, and weighted estimates can then compensate for the fact that some group members could be selected from two different sources. For example, to oversample for the frail elderly, the elderly individuals identified in the sample drawn from the national sampling frame could be supplemented by samples drawn from lists of elderly who participate in congregate meals programs. This approach is useful because, relative to screening in the general population, it is an inexpensive way to identify members of the subpopulation. State-Based Surveillance Systems Potential Options Although identified as a limitation in the previous chapter, the focus of the state-based surveillance systems on particular subpopulations can also be seen as a strength. As described in chapters 1 and 4, states use BRFSS to collect data on the health behaviors of their adult population. PedNss and PNSS are sources of information on the nutritional status of low-income mothers and children. In addition, HHS is exploring opportunities to expand the program to other populations, such as schoolchildren. However, the other disadvantages of the systems, such as the limited amount of nutritional data they collect and the incomplete participation of the states, diminish their utility as a source of information on subpopulations. The two major alternate approaches to providing subpopulation data are special studies and indirect estimation. Special studies are those that use a separate sampling frame from a national survey and are not necessarily Page 53 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 5 Approaches to Supporting Reliable Inferences About Subpopulations conducted at the same time as a national survey. To describe the strengths and weaknesses of special studies as an approach to collecting information on subpopulations, we examined HHs’ past experience with the Hispanic Health and Nutrition Examination Survey (HHANES) and discussed the option with our expert advisers. Indirect estimation uses data that are not direct observations of the group of interest to develop inferences about the subpopulation. Our review of this approach relied on reviews of technical literature, interviews with usDA and Bureau of Census staff responsible for indirect estimation programs, and consultation with methodological experts on our panel of advisers and in HHS. These and the current approaches address the suggestions made by the expert advisers and data users who responded to our survey. Special Studies: the HHANES Model Potential Strengths An alternate approach to covering subpopulations is conducting a special study. A model of this approach is the Hispanic Health and Nutrition Examination Survey conducted in 1982 by HHS. HHS developed HHANES in response to recommendations made by the National Academy for Public Administration, which identified the Hispanic population as growing, likely to have low income, and potentially at risk for health and nutrition problems. HHANES was conducted as a separate study rather than integrated into the national survey because NHANES II was already completed and funding was not available to conduct HHANES as part of a national survey. One of the advantages of conducting HHANES as a separate study was the opportunity to change the content of the survey instrument to address issues of special relevance to the Hispanic population. For example, unlike NHANES II, HHANES gathered information on health services use and gallstone disease. In addition, HHS took steps to address the cross-cultural issues of applying HHANES to different Hispanic populations. The survey instrument was translated into the idiomatic Spanish of each of the three Hispanic groups surveyed (Cuban-Americans, Mexican-Americans, and Puerto Ricans) and an appropriate plan for reaching out to the respondents was developed. For example, unlike the regular NHANES, which relies primarily on press releases and the formal leaders (such as mayors) of the places they have sampled to communicate the importance of participation in the survey, HHANES used informal leaders (such as church leaders) and Spanish-language media. The opportunity to tailor a special survey to the population of interest is an advantage of the Page 54 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 5 Approaches to Supporting Reliable Inferences About Subpopulations Potential Limitations approach, but it can be complicated for a population whose members speak multiple distinct languages and have varying degrees of assimilation into the U.S. population. Although not demonstrated by HHANES, another advantage of a special study is the ability to study populations that are not appropriately studied through oversampling. As described above, this includes groups that are not included in the national sampling frame, such as homeless persons, and groups that are geographically dispersed or not well represented in the general population, such as pregnant and lactating women. Unlike oversampling, in which the subpopulation is surveyed at the same time and usually with the same procedures as the population as a whole, the data collected by special studies may not be comparable to the population as a whole. For example, differences found between HHANES data on Hispanic groups and NHANES data on the nation could be the result of national changes in health and nutrition status during the gap between the two surveys rather than actual differences between the groups. Of course, conducting special studies in tandem with a national survey is possible. In fact, HHs is considering conducting special subpopulation surveys at the same time as the next NHANES. Special studies that are conducted in addition to the national surveys clearly add to the overall cost of data collection. Specific costs for surveys of ethnic populations include bilingual interviewers, the translation of the survey instrument, and outreach to the group. Other costs are the development of a separate sampling frame and screening potential respondents to identify those that belong to the group of interest. Finally, if a special study is conducted at the same time as the national survey, the burden on the survey support facilities, such as the laboratories that analyze blood and urine specimens, is increased, which may slow down the processing and release of the data. If a special study is conducted because the group of interest is not well covered by the national sampling frame, it needs a sampling frame that allows for generalization to the subpopulation group, which may be difficult to construct. For example, to use survey results to draw conclusions about the population of people who are without homes, one needs to sample from a complete list of the members of that group. For the homeless population, such a list would be very costly to construct. As a result, other means—such as sampling shelters—would have to be used. The survey results, though, would probably not be generalizable to the Page 55 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 5 Approaches to Supporting Reliable Inferences About Subpopulations overall population of homeless people because these other means are likely to be incomplete. Indirect Estimation Potential Strengths Indirect estimation (also known as small area or synthetic estimation) refers to procedures that use values of the variable of interest from an area or time other than the area and time of interest. For example, to develop an indirect estimate of the prevalence of iron-deficiency anemia in a county, the national estimate of iron-deficiency anemia can be adjusted based on the county’s demographic profile. Both UsDA and HHs have experience with indirect estimation. Since the early part of the century, uspa has had a program to develop indirect estimates of crop yields. Although some information is available from state surveys of nonprobability samples of farmers, the USDA program adjusts these less dependable estimates so that they aggregate to the more reliable regional and national estimates that are based on a survey of a national probability sample of farmers. HHs has no regular program to produce indirect state estimates, but since 1968, it has supported the occasional development and evaluation of indirect estimates from National Health Interview Survey data. In addition, HHs produced state estimates from the National Natality and National Fetal Mortality Surveys conducted in 1980 using demographic data from the states to adjust the national estimates. Indirect estimation models range from the relatively straightforward adjustments of national or regional estimates to match local demographic profiles to more complex models. The major strength of indirect estimation is that, compared to increasing survey sample sizes to obtain data for direct estimates at state and local levels, it is far less costly. In addition, it is a means of extending the usefulness of costly national survey data to inform decisions made at state or local levels of government. Moreover, it is an established approach. Indirect estimates of such variables as state and local populations, employment and unemployment, and crop yields are already used by the federal government in formulas for determining eligibility and benefit amounts for federal programs. Some state governments have also used indirect estimation to conduct analyses for economic and other types of programs. Indirect estimation also responds to limitations of the data on which direct estimates might be based. Program records, such as those used by PedNSS and PNSS, have the advantage of timeliness since they are usually collected continuously, but their relevance may be limited because the data are Page 56 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 5 Approaches to Supporting Reliable Inferences About Subpopulations Potential Limitations Conclusions collected for specific administrative purposes, not just for nutrition monitoring. Sample survey data, however, have the advantage of relevance, but the data can be costly to obtain at a level of detail that will support estimates for states, localities, or other subpopulation groups. In contrast, indirect estimation is an approach to producing timely, relevant, and detailed information without a major increase in cost. The major limitation of indirect estimation is the difficulty of determining the quality of the estimate. The best way is to compare it to the true value for the population. For example, comparing an indirect estimate of a county’s population to census data on the county’s population would enable an assessment of the bias of the estimate. However, since indirect estimation is used when data are not available for direct estimation, such a comparison is usually not feasible. Moreover, the quality of the estimates yielded by a model changes for different populations and for different times. In other words, even if a model yields estimates that prove to be unbiased in comparison to direct observations for the same year, estimates from the same model for subsequent years may be biased if the relationships between the variables change over time. This limitation has an implication for the use of indirect estimates. Specifically, indirect estimates may be difficult to defend in the political arena because they are based on models rather than direct observations. However, in the absence of direct data from other, more expensive approaches, indirect estimation is preferable to no information at all. Although indirect estimation has been used successfully with other federal surveys, there are constraints on the development of indirect estimation programs for the major nutrition monitoring surveys. According to USDA and HHS staff, major obstacles include the lack of staff resources to support the program and the complexity of the surveys. Despite these concerns, both agencies have long experience with indirect estimation in other arenas, which could be applied to nutrition monitoring. Information on subpopulation groups and small geographic areas is used to identify nutrition-related problems that are associated with specific populations and to target programs to those most at risk. Different strengths and weaknesses are associated with the four approaches to meeting this need that we reviewed. Surveillance systems and indirect estimation programs are likely to be less expensive than oversampling or special surveys. However, oversampling and special surveys can yield Page 57 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 5 Approaches to Supporting Reliable Inferences About Subpopulations more detailed information than the surveillance systems that rely on program records and more dependable estimates than indirect estimation models. Of the four approaches, only indirect estimation has not been a part of NNMRRP activities at one time or another. From programs in other areas, this approach appears to be a potentially efficient means of expanding the information available on subpopulations and small geographic areas. However, before the Interagency Board can determine what priority to give to this promising approach, a complete picture of what needs would be met by each option is required. Page 58 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program ''Chapter 6 Approaches to Assisting States and Localities State and local governments are interested not only in the applicability of federal nutrition monitoring data to their jurisdictions, but also in having federal assistance in collecting and interpreting their own data. This chapter follows on the previous discussion about NNMRRP support for reliable inferences about state and local populations to describe current and potential options for assisting states and localities in their own monitoring activities. The NNMRRP currently assists states through the state-based surveillance systems, which provide technical and other assistance to states. The alternative that we examined, community-based nutrition monitoring, is a response to states and localities that are interested in building their capacity to collect their own data. The strengths and weaknesses of these two approaches to assisting states and localities are summarized in table 6.1 and detailed below. Table 6.1: Approaches to Assisting States and Localities Approach Current: state-based surveillance systems Strength Limitation Balance between federal need for Limited flexibility for state and local standardization and state need for tailoring governments Development of states’ capacity for data Limited support for local use of the data collection and interpretation Potential for improvements on current foundation Potential: community-based nutrition Development of local capacity for Technical and financial resources required monitoring collecting and using data tailored to to assess needs or implement responses to community concerns needs Limited federal investment Tension between federal interest in standardization and local interest in flexibility Other suggestions by our expert panelists—such as providing financial and technical assistance, developing standardized modules of interest for state data collection activities, and assisting state collection of data on subpopulations—can be implemented as part of either of the two approaches. Current Approach: the State-Based Surveillance Systems The strengths and limitations of the surveillance systems as sources of continuous data and of information on state and other population groups have been discussed in chapters 4 and 5. However, they have additional Page 59 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 6 Approaches to Assisting States and Localities strengths and limitations related to their ability to meet state and local needs for assistance. Strengths Surveillance systems currently balance the federal interest in information that is collected in a standard format across states and the states’ needs for flexibility. For example, with BRFSS, HHS supplies states with standardized modules of questions, training in collecting the data, and support in analysis and reporting. In addition, changes to the survey content are made in consultation with state participants in the surveillance system, and states have the opportunity to add their own questions to the survey. The surveillance systems have also played a role in building state capacity for data collection and analysis. Specifically, some states have used their experience with BRFSS to implement their own telephone surveys using the random digit dialing procedure. In addition, standardized software developed by HHS for PNSS has enabled states to generate their own reports. Yet another strength of the surveillance systems is the foundation of federal-state partnership it provides for future improvements of federal technical assistance to state and local nutrition monitoring activities. For example, as mentioned in previous chapters, HHS is exploring ways to use the surveillance systems to gather additional nutritional data, such as more in-depth information on dietary intake, and to cover new populations, such as schoolchildren. Additional technical assistance in data analysis and interpretation could also be provided through the surveillance system structure. Limitations While the surveillance systems could be expanded and improved to further respond to the interest of states and localities in receiving more technical assistance, they are limited in their flexibility because of the federal interest in standardization across states. In addition, users of surveillance system data who responded to our survey identified some concerns about the surveillance systems. Specific issues included the availability of the data to localities, timeliness of HHS’ processing of the data, and the formats of the reports that HHs provided. Recommendations for improving the surveillance systems included increasing local access to data, reducing the time it takes HHS to process the data, simplifying reports for local users, and providing additional technical and financial assistance in data collection and interpretation. Page 60 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Potential Option: Community-Based Nutrition Monitoring Chapter 6 Approaches to Assisting States and Localities The federal-state linkage forged by the surveillance systems could be further extended to support community-based nutrition monitoring. To explore the strengths and limitations of this approach, we reviewed the literature on two models of community-based programs. HHS has funded Planned Approach to Community Health (PATCH) projects, which used the BRFSS survey instrument as the basis for a community needs assessment. The survey data, in combination with interviews with knowledgeable informants in the community, were used to plan health promotion programs. HHS provided technical assistance in the needs assessment portion of the projects and small awards of a few thousand dollars for project activities. Researchers at Cornell University developed a similar model specifically for nutrition monitoring that was pilot-tested in three New York counties with funding from the State Department of Health and technical assistance from the university. In their approach, a coalition of interested community members first articulates potential information needs. Then, the group selects specific needs on the basis of the likelihood that the data will be used by the community. To meet the selected information needs, feasible sources of routinely available data are identified. According to the model, data collection should depend on procedures that are already in place, from such sources as program participation records, patient charts, and school health screenings. In its reliance on program records, this approach is similar to PedNSS and PNSS. However, there is no expectation that the same issues will be targeted or the same sources of data used in each community. To facilitate the final component of the system—the communication of monitoring results—a network of users must be cultivated and the appropriate vehicle for communicating the results must be used. Drawing on the experience of these projects and HHS’ PATCH program, the following strengths and limitations of community-based nutrition monitoring were identified. Potential Strengths Evaluations of both types of community-based programs found evidence that local capacity for collecting and using data was developed. By involving community members in assessing needs, PATCH built skills in identifying health risks and cultivated community and organizational supports for health promotion programs. In the New York counties that pilot-tested the Cornell model of community-based nutrition monitoring, data were collected and compiled to describe access to food and nutrition services and nutritional health of county residents. To disseminate the information and guide decisions based on the data, interagency coalitions Page 61 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 6 Approaches to Assisting States and Localities were formed. These coalitions brought together local nutrition-related professionals, so that nutrition interventions were better coordinated and information was shared. The limited federal investment in the community-based programs is another strength. HHS supported PATCH with small grants and technical assistance, while the community-based nutrition monitoring projects were sponsored by the state and received technical assistance from the state’s land grant institution. Moreover, there is some evidence that the activities put in place by these projects will continue. For example, Cornell reports that “At the end of the [3-year] pilot stage, one county obtained local funding to continue their monitoring activities, including the issue-based coalitions; in the other counties, local nutrition councils are coordinating continued monitoring efforts.”! The program has also spread to other counties, some of which have initiated monitoring activities without outside funding. Potential Limitations The experience with community-based programs indicates that communities require a set of resources—specifically, technical skills and dedicated personnel—to fully benefit from the projects. One evaluation of PATCH found that the projects were most effective in communities that already had human services and health professionals who were involved in community health promotion efforts. In addition, the projects that had directors seemed to have the greatest implementation successes. Both kinds of projects required considerable time and effort to collect and interpret data. Moreover, while HHs provided technical assistance in using the BRFSS survey in the local needs assessments, additional assistance was needed to help communities set priorities based on the data. The reliance on BRFSS was a limitation for PATCH because BRFSS did not necessarily include the issues of primary concern for the community. For example, one PATCH project was specifically interested in water quality, which is not addressed in BRFSS. In contrast, the Cornell model emphasized local sources of data. Based on the initial experience with PATCH, HHS no longer expects communities to use BRFSS for its needs assessment. “Community-Based Nutrition Monitoring,” Division of Nutritional Sciences, Cornell University, and Division of Nutrition, N.Y. State Department of Health, Mar. 1993. Page 62 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Chapter 6 Approaches to Assisting States and Localities Conclusions Data from the HHs surveillance systems are already extensively used in state and local program planning. In addition, the systems are flexible and are exploring ways to increase the information they collect on dietary intake. However, they seem less responsive to the needs of localities than to the needs of states. Recent efforts to build local capacity for data collection and interpretation indicate that community-based programs are a promising approach to responding to local needs for nutrition information. Before the Interagency Board can decide what priority to place on community-based nutrition monitoring, however, it must first identify the objectives of the NNMRRP that would be furthered and the importance of these objectives relative to others that are competing for program resources. Page 63 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program ''Appendix I List of Experts This appendix lists the expert advisers who assisted with this project. As described in chapter 1, the advisers were organized into three panels—core policy panel, methodology panel, and data users panel. Core Policy Panel Johanna Dwyer, D.Sc., R.D., Francis Stern Nutrition Center, New England Medical Center, and Tufts University Schools of Medicine and Nutrition Jean-Pierre Habicht, M.D., Ph.D., Division of Nutritional Sciences, Cornell University Catherine Woteki, Ph.D.! Methodology Panel Norman Bradburn, Ph.D., Director, National Opinion Research Center Marilyn Buzzard, Ph.D., Director, Nutrition Coordinating Center, University of Minnesota Ricardo O. Castillo, M.D., M.P.H., Co-Director, Pediatric Gastroenterology, Stanford University Medical Center Alan R. Kristal, Dr.P.H., Fred Hutchinson Cancer Research Center and Department of Epidemiology, University of Washington James Lepkowski, Ph.D., Institute for Social Research, University of Michigan Cheryl Ritenbaugh, Ph.D., Department of Family and Community Medicine, University of Arizona Laura Sims, Ph.D., Department of Nutrition and Food Science, University of Maryland Data Users Panel Elizabeth Barnett, Ph.D., North Carolina Department of Environment, Health, and Natural Resources Doris Disbrow, Dr.P.H., R.D., Center for Health Education 1Dr. Woteki withdrew from the panel when she was appointed to the Office of Science and Technology Policy in the White House. During her participation in our work, she was the Director of the Food and Nutrition Board of the National Academy of Sciences. Page 64 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix I List of Experts Pamela Haines, Dr.P.H., R.D., Department of Nutrition, University of North Carolina Jay Hirschman, M.P.H., Senior Analyst, Food and Consumer Service, U.S. Department of Agriculture Karen J. Morgan, Ph.D., Senior Director, Nutrition and Consumer Affairs, Nabisco Brands Barbara Petersen, Ph.D., Technical Assessment Systems Page 65 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program ''Appendix II Sources for the Model Features Past Evaluations of Federal Nutrition Monitoring The model features that are the focus of this report were derived from four sources. First, we reviewed previous evaluations, from which 13 historical criticisms of federal nutrition monitoring were identified. Second, we considered the objectives and related activities outlined in the 10-year comprehensive plan developed by the Interagency Board. Third, through an iterative process of panel meetings and review of written materials, a number of suggestions for improvements to the NNMRRP were generated by the expert advisers. Fourth, the data users who responded to our survey provided written comments describing what changes to specific data collection activities would increase their use of the data. Table II.1 lists the criticisms identified in past evaluations and categorizes them by the features of a model program that they suggest. (See Nutrition Monitoring (GAO/PEMD-94-23), pp. 5-6.) Table II.1: Features of a Model Program Suggested by Past Evaluations Feature Past criticism Coordinated system that responds efficiently to the diverse needs of the data users Lack of coordination among nutrition monitoring activities Information needs of users not systematically determined Lack of compatibility in methods for assessing dietary intake Core set of standardized measures not yet developed for major surveys Compatible sampling techniques not used for national surveys Reporting by national surveys not integrated Continuous or more frequent collection of data Data not collected continuously Support for reliable inferences about subpopulation groups and small geographic areas Specific population groups not covered by major surveys Specific geographic areas not represented by major surveys Assistance to states and localities State and local data needs not fully addressed Improved methodology for assessing dietary intake and nutritional status Improvement needed in methodology for assessing dietary intake and nutritional status Low respondent burden and high response rates Sampling problems with NFCS Timely processing and dissemination of data Page 66 Need for more timely dissemination of survey information GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix II Sources for the Model Features ee rr The 10-Year Table II.2 provides examples from the 68 activities listed in the 10-year . comprehensive plan that correspond to the desirable features identified Comprehensive Plan from the review of previous evaluations of federal nutrition monitoring. (Other desirable features suggested by the 10-year plan activities but not addressed in this report focus on strengthening the food composition and food supply data systems. ) Page 67 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix II Sources for the Model Features Re aie 22 A NESE VS Table 1.2: Examples of Planned NNMRRP Activities Associated With Model Features Feature Planned activity® Coordinated system that responds efficiently to the diverse needs + Coordinate the planning for coverage, tracking, and reporting of of the data users findings from surveys and surveillance systems Identify ways to increase comparability within a dietary method to improve the quality and usefulness of data Establish a mechanism for improved coordination among federal agencies that collect and use survey information about knowledge, attitudes, and behavior to assess gaps and duplications in existing surveys Continuous or more frequent collection of data No specific activities listed® Support for reliable inferences about subpopulation groups and Develop and implement a plan for improved coverage of small geographic areas subgroups of the population at nutritional risk or at risk for underconsumption or overconsumption of nutrients and food components Evaluate the feasibility of alternate statistical methodologies for creating state and local estimates Assistance to states and localities Provide assistance for the development and maintenance of state structure, staff, and programs to support their participation in NNMRRP Expand the coverage of current state and local nutrition monitoring activities in selected population groups through technical assistance and grant awards Improved methodology for assessing dietary intake and nutritional Conduct research to develop, improve, and validate laboratory status measures of nutritional status Develop and evaluate procedures for determining usual intakes of foods and nutrients from surveys employing 24-hour recall measures of dietary intake Low respondent burden and high response rates No specific activities listed° Timely processing and dissemination of data No specific activities listed? @See the Ten-Year Comprehensive Plan, 58 Fed. Reg. 111 (June 11, 1993), pp. 32752-806. bAlthough no specific activities are planned to ensure the continuous collection of data, two of the six objectives include continuous data and one of the activities mentions coordinating the timing of surveys. CUSDA has separated the household and individual portions of the NFCS survey in order to reduce respondent burden and improve response rates. In addition, USDA signed interagency agreements with the Bureau of Census for assistance in designing and conducting the individual and household surveys, as well as for research on improving methods for collecting household food use data. 4Although none of the activities identified under the objectives specifically address the timeliness of data processing and dissemination, part of the overarching goal stated for the plan is “efficiently disseminating and exchanging information with data users.” In addition, the plan discusses possible mechanisms for increasing awareness and dissemination of the data and states that the Interagency Board plans to establish a central clearinghouse. Page 68 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix II Sources for the Model Features Consultation With Expert Advisers The expert advisers to the project (listed in appendix I) provided guidance by reviewing materials and participating in panel meetings. Through this process, several suggestions were generated, and their advantages and disadvantages were discussed. The suggestions outlined below in table II.3 do not indicate consensus among the panelists. Table I1.3: Advisory Panel Suggestions Associated With Model Features Feature Coordinated system that responds efficiently to the diverse needs of the data users Suggested approach A single lead agency An interagency body with permanent staff and enforcement authority Location in statistical agencies within the user departments or within a central statistical agency Centralization of the congressional appropriations process for nutrition monitoring activities Informed review of data collection plans by qualified people at OMB Continuous or more frequent collection of data Continuous national nutrition surveys Addition of modules to existing surveys Reliance on program data that are already collected Collection of longitudinal data Support for reliable inferences about subpopulation groups and small geographic areas Surveys of population subgroups instead of national-level surveys Different sampling strategies for different populations (for example, list-based sampling for populations that are rare in a broad-based sample) Contracts with states and localities to gather information on geographically-based populations Indirect estimation to support inferences about population subgroups and small geographic areas Assistance to states and localities? Financial assistance to states to determine their data needs A federal-state partnership in which states can provide funds for some extra sampling or extra questions on federal surveys Federal development of standardized modules of interest for state data collection activities Federal assistance to state collection of data on subpopulations Technical assistance in data interpretation Page 69 (continued) GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix II Sources for the Model Features Feature Suggested approach ebb methodology for assessing dietary intake and nutritional © Continued or increased research on nutrition monitoring methods status Postmarket surveillance data (such as that gathered through bar code scanners) Automated collection of dietary data Reliance on program records Low respondent burden and high response rates Provide more feedback to respondents about survey results Timely processing and dissemination of data Assistance to data users in interpreting and analyzing the data Development of core analysis data sets that are focused on specific issues or groups A data clearinghouse Automated data collection aThe panelists disagreed about the responsibility states should have for data collection, with some arguing for state-based data collection that feeds into a federal system and others arguing for less state responsibility for data collection, but increased consideration of state needs in federal data collection activities. In addition to comments that could be categorized by the model features suggested by the historical criticisms, the expert advisers also had other suggestions for improving the system, including develop an ongoing evaluation of the system’s content and methods, including a review of the information needs; continue to collect both household and individual level data; have a system to maintain data comparability over time; and use longitudinal designs to measure change. Suggestions From the Survey of Data Users The comments that users of nutrition monitoring data provided in response to our survey were analyzed to identify major themes for each of three groups of data collection activities—USDA surveys, HHS surveys, and HHS state-based surveillance systems. Table II.4 identifies the themes associated with the features of a model program. More detailed summaries can be found in Nutrition Monitoring (GAO/PEMD-95-15). Page 70 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix II Sources for the Model Features - Table II.4: Major Themes in the Suggestions Made by Data Users Model feature Major Theme USDA surveys HHS surveys HHS surveillance systems Coordinated system that responds efficiently to the diverse needs of the data users a a a Continuous or more frequent collection of data Continuous or more frequent data collection Continuous or more frequent data collection b Support for reliable inferences about subpopulations geographic areas Increased sample sizes and broadened coverage More detail on racial, ethnic, age, and income groups Refined geographic coverage, specifically state and substate data More detail on racial, ethnic, and age groups Data that can support estimates for small geographic areas More detail on subpopulation groups in the reporting of the data Increased ability to look at substate geographic divisions Assistance to states and localities c Simplified reports that are more readily used at the local level Additional technical and financial assistance in data collection and interpretation Better controls on the quality of the data collected Improved methodology for assessing dietary intake and nutritional status More specificity and detail about foods and better data on food composition Improved questions about dietary behavior More information about health and demographic variables Higher quality dietary recall data More information on health habits and outcomes More detailed data on food consumption Improved dietary intake methods More data on dietary intake Reduced respondent burden and improved response rates Reduced respondent burden and improved response rates Timely processing and dissemination of data Improved timeliness and documentation of the data Dissemination of the data in formats that facilitate access and analysis Page 71 Improved timeliness and documentation of the data Dissemination of the data in formats that facilitate access and analysis Improved timeliness of HHS’ return of the data to states (Table notes on next page) GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix II Sources for the Model Features @Coordination was not a major theme of the comments provided in response to our survey of data users probably because the survey queried respondents about changes to specific data collection activities rather than NNMRRP as a whole. >Users of surveillance system data were unlikely to identify continuous data as a major theme since these systems collect data continuously. Because our survey asked for comments about the two data collection systems most frequently used by the respondent, state and local users were more likely to comment on the surveillance systems. As a result, their interest in changing the national surveys to better meet their needs was not gauged. ¢The HHS systems have not suffered the same response rate problems that USDA's 1987-88 NFCS experienced. The themes reflect the strengths and weaknesses of the different data collection activities. For example, response rates were probably a major issue to users of USDA surveys because of the response rate problems of NFcs. Similarly, assistance to states and localities was a theme from the users of the surveillance systems, many of whom manage state and local programs. In addition to the themes that correspond to the features of a model program, other major issues in the suggestions from the data users were the collection of longitudinal data (from users of USDA survey data) and the collection of both individual and household data (from users of data from USDA and HHS surveys). Page 72 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program ''Appendix III Comments From the Department of Agriculture Note: GAO comments EL gaa re ar | supplementing those in the report text appear at the j ; fax, United States Agricultural Office of the Washington, D.C. end of this appendix. }) Department of Focearch Administrator 20250 We? Agriculture Service JUN 13> 1995 Mr. Kwai-Cheung Chan Director Program Evaluation in Physical Systems Areas Program Evaluation and Methodology Division General Accounting Office Washington, D.C. 20548 Dear Mr. Chan: This is in response to your May 17, 1995, memo to Mr. Dan Glickman to review and comment on the General Accounting Office Draft Report, entitled "Nutrition Monitoring: Current Program Has Strong Elements But Still Needs Better Coordination of Activities." Enclosed is the Department of Agriculture’s response. PAS FOX: ( \ Dia Le 4 R. D. PLOWMAN Administrator Enclosure CONCURRENCE : hel! A, hbe, DATE: << B— KARL N. STAUBER Under Secretary Research, Education, and Economics Agricultural Ores see Service Page 73 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix II I Comments From the Department of Agriculture United States Agricultural National Beltsville, Maryland Department of Research Program 20705 Agriculture Service Staff June 6, 1995 SUBJECT: TO: FROM: As noted i Administra have been Enclosed a need any a contact me Enclosures ces E. Kniplin W. Martine F. Schwenk D. Reed Ces iis" Service U.S. General Accounting Office Draft Report "Nutrition Monitoring: Current Program Has Strong Elements But Still Needs Better Coordination of Activities" Gene P. Spory Associate Deputy Director Financial Management Jacqueline L. Dupont {¢ — National Program Leader Human Nutrition n the enclosed memo dated May 17, 1995, to the ARS tor, you are compiling comments from agencies and designated to furnish the Department's reply. re ARS comments in reply to OIG's request. If you dditional information, please do not hesitate to on 301/504-6275. ig z Page 74 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix III Comments From the Department of Agriculture See comment 1. See comment 2. See comment 3. See comment 4. The Agricultural Research Service has reviewed the GAO draft report "Nutrition Monitoring: Current Program Has Strong Elements But Still Needs Better Coordination of Activities. We are committed to a strong nutrition monitoring program as agree with the features GAO identified as a model program. These features are emphasized in the Ten-Year Comprehensive Plan and are reflected in a variety of ways throughout the Program. One of the strengths of the Program is its diversity of users as well as providers which requires constant attention to coordination and communication, but brings together extensive expertise. Our work with the Department of Health and Human Services and other agencies in coordinating activities continues to be an overriding theme and focus in ARS. Within the resources available, ARS will build on the strengths of the current nutrition monitoring program to address priority needs. Specific comments regarding the report follow: Page ES-2: Statements made throughout the report do not provide a consistent tone. For example, in the first paragraph on this page, it is stated that priorities were not established for the Ten-Year Plan. However, on page 3-9 and 3-10, there is discussion about the priorities assigned to Ten-Year Plan activities since the Plan was published. Another example is in Appendix II-13: USDA has taken the necessary steps to avoid response rate problems in the future, as acknowledged on page 2- 6. However, the many activities implemented by ARS in its current survey, the CSFII 1994-96, to improve response rates and reduce respondent burden, as well as to improve the quality of data collection, should be acknowledged here also for consistency within the report. Page ES-3: The Interagency Board Working Group Federal STRIDE investigated the possibility of establishing some type of clearing house for nutrition monitoring products. However, resource limitations were a primary barrier especially in view of the fact that each agency involved in nutrition monitoring had distribution mechanisms for their products and reports. Page ES-7: First paragraph. We agree with this statement. The second paragraph could be amended to state: 1. The Board has been quite effective in coordinating the technical aspects of monitoring. It could be improved by across department consistent membership with budget decision-making authority from all agencies. Also, there should be a professionally qualified career civil service leader named as assistant to the Board Chairs who is appointed by the Secretaries. The charge to the Board should be strengthened to support its development of a prospective budget to be proposed for inclusion in departmental and OMB budgets. Page 1-5: Because the NFCS is the only nutrition monitoring survey that gathers information about cost of food used at the household level, we believe that point should be included in the first paragraph. Page 75 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix III Comments From the Department of Agriculture See comment 5. There is no mention about the role that the CSFII and DHKS play regarding health policy. Because use of the CSFII and DHKS data has been extensive in establishing and tracking progress toward the Year 2000 Healthy People Nutrition Objectives, we believe the important use of these data in the nutrition-related health policy area should not be overlooked. In fact, we have worked quite extensively with the Department of Health and Human Service in this endeavor. The following reference documents USDA data use: U.S. Department of Health and Human Services, Public Health Service. Healthy People 2000 Midcourse Review and 1995 Revisions (in press). Page 1-10: The National Nutrition Monitoring Advisory Council wasn't established to specifically guide the Interagency Board. P.L. 101-445 states that "The Council shall assist in carrying out the purposes of this Act, provide scientific and technical advice on the development and implementation of the coordinated program and comprehensive plan, and serve in an advisory capacity to the Secretaries" (of Agriculture and Health and Human Services). Page 2-8: Under the auspices of the Interagency Board, work is underway to review surveys focused on dietary knowledge, attitudes, and behaviors conducted by USDA and HHS for enhancing comparability. By design some questions are the same across surveys. USDA's Diet and Health Knowledge Survey is the only national survey to provide information on knowledge and attitudes that is linked to information on dietary intakes for the same individuals. This unique feature makes it possible to validate the nutritional effect of knowledge and attitude issues covered by the HHS surveys; use resources efficiently to maintain trend data on key issues; and expand available information on various groups, such as food label users, nutritionally knowledgeable individuals, and diet-health aware individuals. USDA and HHS have worked diligently toward coordinating the content of each survey so that together they provide more comprehensive information than either could provide alone, and so that unnecessary duplication is avoided. These efforts support an activity in the 10-year comprehensive plan for national nutrition monitoring that was established specifically to ensure such coordination. See comment 6. Page 3-4: The USDA chair for the Interagency Board for Nutrition Monitoring and Related Research is the Under Secretary for Research, Education, and Economics. See comment 7. Page 3-8: The ARS accounting system identifies exactly how funds are allocated by the Current Research Information System (CRIS). The surveys identified in the 10-year plan are under the Management Unit, Survey Systems/Food Consumption Laboratory of the Beltsville Human Nutrition Research Center. Adherence to the mission stated in the Monitoring Act is overseen by the National Program Staff of ARS. Page 76 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix III Comments From the Department of Agriculture Ss Page 3-9: the Ten-Year Plan activities were intentionally ee comment 8. written to be general so that involved agencies could design implementation plans based on the most critical needs and resources available. To use one of your examples regarding increasing comparability within a dietary method, the computer- assisted dietary interview project was deemed to be a critical need by USDA and HHS. In planning for the project, both the degree of quality required and the uses were focuses of a scientific consultation and Federal users workshop, respectively. Page 3-14: Membership on the IBNMRR has traditionally been See comment 9. agency staff who have strong technical expertise in nutrition monitoring which has served the Board well for communication and coordination functions. Board members are encouraged to work within their agencies to assure that agency activities are supportive of nutrition monitoring goals. Page 4-3: Following the sentence related to NFCS, the current information is as follows: After the 1987-88 NFCS, the household portion of the survey has not been conducted. Plans for a continuing survey of households will be made in relation to need, expected rate of change in food behavior, and availability of resources. With consistent base funding, it is expected that a long-range plan (20 years) will be developed to integrate the surveys so as to match anticipated rates of change in different factors. Pages 4-10, 4-11: Sufficient staff resources and dependable funding are equally essential for continuous data collection. ARS has committed its resources to quality survey management and timely release of data. Certainly, timely release of data, if the CSFII were fielded very year, would be difficult with current resources. As discussed on page 4-12, opportunities to develop, test, and incorporate new methods and data elements will be needed to continue to improve the quality and ensure the usefulness of the data collected. See comment 10. . Page 5-5: The CSFII 1994-96 sampling will provide estimates for the U.S. general and low-income populations. Ethnic subgroups will be represented proportionately as they are in the general population. Page 77 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix III Comments From the Department of Agriculture The following are GAO’s comments in response to the June 13, 1995, letter from the Department of Agriculture. 1. What uspaA finds inconsistent with the report reflects the inconsistency of NNMRRP. The 10-year plan, a natural location for a statement of priorities within expected funding levels, specified neither priorities nor expected funding levels. As noted in our report, the later priority-setting exercise was an improvement but was also limited in its utility because it did not specify whether data needs and feasibility were considerations in ranking the objectives. 2. The central contact for data users that is suggested by the alternate approaches to coordination entails much more than a clearinghouse for data products. For example, the Office of National Drug Control Policy acts not only as a source of information for state and local drug enforcement interests, but also as a conduit of information about state and local concerns to the federal agencies. 3. USDA makes several recommendations on how the Board can be improved. These recommendations are consistent with the strengths that we identified in the other approaches to coordination. 4, The Consumer Expenditure Survey also provides information about the cost of food used at the household level. However, because of the detail about the expenditures on specific foods gathered by the NFcs, we have amended the description of the NFCS. 5. Following the classification provided by the NNMRRP document, The Directory of Federal and State Nutrition Monitoring Activities, CSFII and DHKS focus on food and nutrient consumption. Table 1.1 indicates that data on food and nutrient consumption are used for health- as well as food-related purposes. 6. The sentence has been changed to reflect this information. 7, Because the coordinated NNMRRP budget reports amounts dedicated to nutrition monitoring by agency rather than by purpose, the accounting system used by ARs is not relevant. All NNMRRP expenditures by ARS will be reported together, whether they are dedicated to surveys or related research. Page 78 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix III Comments From the Department of Agriculture 8. The fact that the plan was intentionally vague does not diminish the degree to which the lack of detail limits its utility as a statement about the specific objectives of the program, the activities planned to meet the objectives, and the resources needed to support the plans. 9. While technical expertise can be supplied by staff, only members with organizational authority can make decisions about agency priorities and resources. USDA’s comment regarding p. ES-7 (responded to in comment 3 above) suggests the importance of having members with budget decision-making authority. 10. The statement has been changed to reflect this information. Page 79 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program ''Appendix IV Comments From the Public Health Service Note: GAO comments supplementing those in the report text appear at the end of this appendix. anne Me DEPARTMENT OF HEALTH & HUMAN SERVICES Public Health Service oe MALIN ny rare Rockville MD 20857 JUN [| 6 toas Mr. Kwai-Cheung Chan Director of Program Evaluation in Physical Systems Areas Program Evaluation and Methodology Division U.S. General Accounting Office Washington, D.C. 20548 Dear Mr. Chan: The Public Health Service has So the General Accounting eee 8 er report entitled fe) 3 urrent trong Elements But s ds Better ST PPR of Activities. Our comments on the draft report are attached. We appreciate the opportunity to review the draft report before it is finalized. Sincerely gue gD, i is L. LN Deputy Assistant aiorukeey for Health (Management and Budget) Attachment Page 80 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix IV Comments From the Public Health Service PUBLIC HEALTH SERV S_ON E_ GENERAL Ai IN FFICE D “NUTRITION IN 3 TRONG ELEMENT BUT STILL NEEDS BETTER IRDINAS OF TIVITIES" ENE: Overall, the General Accounting Office (GAO) draft report is factual with regard to the National Nutrition Monitoring and Related Research Program (NNMRRP). The draft report expresses current limitations, potential alternatives, and identifies the consequences of various options. The GAO has identified four major features of a model nutrition monitoring program, each of which is specifically addressed as a major objective and activity in the NNMRRP’s 10-year plan. The Interagency Board’s priorities for 1994 include: coordinated reporting of data for the Third Report on Nutrition Monitoring and Year 2000 health objectives, survey coordination (sample design research and automated dietary data collection), development of a food security assessment tool, and food safety and labeling research. These activities received high-priority for staff resources. Progress updates were regularly made at the Interagency Board meetings and are summarized in the annual 10-year plan progress documents. Contrary to the statement on page ES-2, the 10-year plan does include two planned activities to evaluate progress on the 10-year plan and the NNMRRP activities: (1) annual progress See comment 1 reviews by the Interagency Board and (2) formal evaluations of . the 10-year plan in 1997 and 2002. Since 1992, the Interagency Board has summarized plans and progress on each 10-year plan activity and held annual progress reviews on high-priority activities. This information has been distributed to the Interagency Board and the 10-year plan working groups, and the executive summaries of 1992 and 1993 progress have been widely disseminated to Federal and non- Federal nutrition monitoring data users, as well as to GAO. Continuous data is defined by the NNMRRP as a “survey or surveillance system in which data collection is repeated regularly and frequently." This does not imply that data collection must actually be continual or annual, but rather that data can be expected to be available on a regular, planned basis. Plans for a continuously-fielded National Health and Nutrition Examination Survey (NHANES), beginning in 1997, have been proposed, but are contingent upon securing adequate funding. The State-based surveillance systems are continual in nature, but are not nationally representative. Of utmost concern is the need to use methods in the States that are comparable to methods used in national surveys to allow for linkage and comparability of data. Page 81 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix IV Comments From the Public Health Service With the current fiscal restraint in the Government, it would have been helpful for GAO to provide cost estimates for the proposed scenarios. While it is reasonable to suggest that States and localities collect data on subgroups and special populations, one needs to consider how extensive or detailed the data would be. It would be far more abbreviated than the national data. For example, States do not currently have the resources to collect the type of extensive dietary data that the national surveys do. As a result, State data would not be comparable to national data. Additional research to develop brief, accurate nutrition tools is needed. The comparability of surveillance systems at the State and local community level might be improved with further research to establish simple and less costly measures for nutrition monitoring. The exact timing, content, and size of the next NHANES will be determined with the 1997 budget process and beyond, and there is no definitive conclusion at this time. The Centers for Disease Control and Prevention and HHS share commitment to the continual flow of nutrition and health data from national surveys such as the NHANES to meet NNMRRP data needs. TECHNICAL COMMENTS a ottom of e 3-8 and continuin See comment 2. on page 3-9: GAO can not assume that a working group is not planning courses of action or being accountable. The 10-year plan does not detail specifically how an activity will be met, since flexibility and creativity are necessary to achieve progress in light of resources available to implement activities. However, the annual progress reports indicate that the working group assigned to each activity develops an approach to the problem and reports on.progress. A review of these documents would indicate which working groups are making progress and how they are making progress in meeting 10-year plan activity timelines. eqinning on -17: In addition to GAO's See comment 3. description of problems in staffing a central authority, staff that currently work on the Interagency Board and supporting staff have other functions within their own agencies-- nutrition monitoring tasks are often only one of many tasks they perform. In order to staff a central authority, it would be necessary to pull staff from their current jobs, where they are very familiar with the nutrition monitoring activities that are occurring, or, hire new staff who are unfamiliar with what has progressed in the monitoring program and with the work of the member agencies. Page 82 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix IV Comments From the Public Health Service S 4 Sentence beginning at the bottom of page 4-7 and ending on ee comment 4. page 4-8: This sentence implies that inclusion of all States in a surveillance system would provide national data. A coordinated, integrated sample design would be needed so that the data collected would still be representative of the State and could also be aggregated to produce nationally representative data. scu on _beginni at the bottom o - on page 5-6, and the footnote on page 5-5: The report notes that homeless persons or persons who live in institutions See comment 5 would not be included in the national household sampling : frame. We note that American Indians and Alaska Natives residing on reservations are likewise not currently included in the national sampling frame. The discussion of potential strengths and limitations of direct and indirect approaches in special studies includes no mention of issues relevant to American Indians and Alaska Natives. Page 83 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Appendix IV Comments From the Public Health Service The following are GAO’s comments on the June 16, 1995, letter from the Public Health Service. 1. As PHs states, the 10-year plan does plan to conduct evaluations in 1997 and 2002. However, neither these evaluations nor the Board’s annual summaries of progress are a strategy for weighing the costs against the benefits of current or new activities. A framework for comparing existing and potential monitoring activities in such terms as their feasibility, ability to meet data needs, and other considerations is still needed. 2. The paragraph referred to (now on pp. 34-35) is not about the progress of the working groups, but about the guidance provided by the 10-year plan for the program as a whole. While the progress reports indicate specific examples of improvements within working groups, the 10-year plan does not constitute a comprehensive proposal. Such a proposal would include specific objectives, detailed plans for how to meet the objectives, and cost estimates and funding requests to support the plans. The availability of resources affects the ability to meet an objective, but is not a reason for not developing specific plans in the first place. 3. In contrast to such coordinating bodies as the Interagency Board and the National Library of Medicine, the central authorities that we reviewed have personnel who are not detailed from the agencies. While the potential disadvantage that PHS suggests (that is, unfamiliarity with NNMRRP) can be easily overcome, the advantage of independence from the interests of any single agency is difficult to achieve in the other approaches. 4, The sentence has been changed to clarify that, with current levels of state participation, the PedNSs and PNSSs can not be used to provide nationally representive data of the low-income mothers and children receiving assistance through the wic program. 5. This section has been changed to reflect the lack of coverage of American Indians and Alaska Natives living on reservations by the national household sampling frame. Page 84 _GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program ''Appendix V Major Contributors to This Report * John Oppenheim, Assistant Director Program Evaluation Leslie Riggin, Project Manager and Methodology Lé Xuan Hy, Adviser Division Venkareddy Chennareddy, Referencer Elizabeth W. Scullin, Communications Analyst Page 85 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Page 86 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Page 87 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Related GAO Products Nutrition Monitoring: Data Serve Many Purposes; Users Recommend Improvements (GAO/PEMD-95-15; June 20, 1995). High Performance Computing and Communications: New Program Direction Would Benefit From a More Focused Effort (GAO/AIMD-95-6; Nov. 4, 1994). Nutrition Monitoring: Progress in Developing a Coordinated Program (GAO/PEMD-94-23; May 27, 1994). Food Nutrition: Better Guidance Needed to Improve Reliability of UsDA’s Food Consumption Data (GAO/RCED-94-30; Oct. 25, 1993). (973397) Page 88 GAO/PEMD-95-19 Features of a Model Nutrition Monitoring Program '' Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (301) 258-4097 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. i PRINTED ON (|) RECYCLED PAPER '' United States General Accounting Office Washington, D.C. 20548-0001 OeCOr NM sre Penalty for Private Use $300 Address Correction Requested Bulk Mail Postage & Fees Paid GAO Permit No. G100 ''BERKELEY LIBRARIES CO08b69b15b ''Aan Ge SS3N90Nd NI ''