N u) 5 f- e 5 T i f NOAATM NWS FCST-18 A UNITED STATES DEPARTMENT OF COMMERCE PUBLICATION NOAA Technical Memorandum NWS FCST-18 y Office of Meteorological Operations Weather Analysis and Prediction Division SILVER SPRING, MD. May 1972 U.S. DEPARTMENT OF COMMERCE National Oceanic and Atmospheric Administration National Weather Service Long-Term Verification Trends of Forecasts by the National Weather Service DUANES.COOLEY AND ROBERT G. DEROUIN NOAA TECHNICAL MEMORANDA National Weather Service, Weather Analysis and Prediction Division Series The Weather Analysis and Prediction Division of the Office of Meteorological Operations is specifically responsible for the management of forecasting services for the public, aviation, marine, agriculture, and fire weather interests. NOAA Technical Memoranda in the Weather Analysis and Prediction Division series com- municate scientific and technical information relating to field forecasting operations. The series includes information on present techniques, procedures, and performance data. Background information and detail on selected service operations are also given. The series provides a means for the personnel in the National Weather Service Headquarters and Regional Offices to report on fore- casting methods of general interest and wide application. NOAA Technical Memoranda in the Weather Analysis and Prediction Division series facilitate rapid distribution of material which may be preliminary in nature and which may be published formally elsewhere at a later date. Publications 1 to 4 by this Division are in the former series, Weather Bureau Technical Notes (TN) , Notes to Forecasters (FCST); publications 5 to 15 are in the former series, ESSA Technical Memoranda, Weather Bureau Technical Memoranda (WBTM) . Beginning with FCST 16, publications are now part of the series, NOAA Technical Memoranda, National Weather Service (NWS). Publications listed below are available from the National Technical Information Service, U.S. Department of Commerce, Sills Bldg., 5285 Port Royal Road, Springfield, Va. 22151. Price: $3.00 hard copy; $0.95 microfiche. Order by accession number shown in parentheses at end of each entry. Weather Bureau Technical Notes TN 8 FCST 1 On the Use of Probability Statements in Weather Forecasts. Charles F. Roberts, September 1965. (PB-174 647) TN 13 FCST 2 Local Cloud and Precipitation Forecast Method (SLYH) . Matthew H. Kulawiec, September 1965. (PB-168 610) TN 16 FCST 3 Present and Future Operational Numerical Prediction Models. Charles F. Roberts. October 1965. (PB-169 126) TN 23 FCST 4 Forecasting the Freezing Level Objectively as an Aid in Fore- casting the Level of Icing. Jack B. Cox, December 1965. (PB-169 247) ESSA Technical Memoranda WBTM FCST 5 Performance of the 6- Layer Baroclinic (Primitive Equation) Model. Julius Badner, September 1966. (PB-173 426) WBTM FCST 6 Forecasting Mountain Waves. Philip A. Calabrese, September 1966. (PB-174 648) WBTM FCST 7 A Method for Deriving Prediction of Soil Temperature From Medium- Range Weather Forecasts. Charles F. Roberts, June 1967. (PB-175 773) WBTM FCST 8 Recent Trends in the Accuracy and Quality of Weather Bureau Fore- casting Service, Charles F. Roberts and John M. Porter, November 1967. (PB-176 953) WBTM FCST 9 Report on the Forecast Performance of Selected Weather Bureau Offices for 1966-1967. C. F. Roberts, J. M. Porter, and G. F. Cobb, Decem- ber 1967. (PB-177 043) WBTM FCST 10 Size of Tornado Warning Area When Issued on Basis of Radar Hook Echo. Alexander Sadowski, May 1969. (PB-184 613) (Continued on inside back cover) U.S. DEPARTMENT OF COMMERCE National Oceanic and Atmospheric Administration National Weather Service NOAA Technical Memorandum NWS FCST-1! LONG-TERM VERIFICATION TRENDS OF FORECASTS BY THE NATIONAL WEATHER SERVICE Duane S. Cooley and Robert G. Derouin Office of Meteorological Operations Weather Analysis and Prediction Division SILVER SPRING, MD . May 1972 UDC 551. 509. 5: 354. 82 (73) "1915/1971" 354.82 Government departments 551.5 Meteorology .509 Synoptic forecasts .5 Verification of forecasts (73) United States of America "1915/1971" Period 1915-1971 CONTENTS Page Abstract 1 1 . Introduction 1 2 . Guidance forecasts 1 3. City and national forecasts 4 4. Tornado and hurricane statistics 4 5 . Extended forecasts 4 6 . Summary and conclusions 9 7 . Acknowledgments 11 References 11 LONG-TERM VERIFICATION TRENDS OF FORECASTS BY THE NATIONAL WEATHER SERVICE Duane S. Cooley and Robert G. Derouin ABSTRACT. Averages of daily forecast verification records of the National Weather Service (NWS) (pre- viously the United States Weather Bureau) over a period of many years are summarized and illustrated below. Results of studying precipitation, tempera- ture, wind, and pressure predictions made during the past two or three decades show that progress, though sometimes slow, is definitely being made in improving weather forecasts. 1. INTRODUCTION This paper is an update and extension of earlier reports on the accuracy of selected National Weather Service forecasts (Roberts et al. 1967a, and Cressman 1970). The graphs in this report were produced through the joint efforts of the National Weather Service Headquarters, the National Meteoro- logical Center (NMC) , the National Hurricane Center (NHC) , the National Severe Storms Forecast Center (NSSFC), and selected forecast offices. Sec- tion 2 illustrates progress in central guidance forecasts received by the field office forecasters. Section 3 summarizes records from a few major cities and the nation as a whole. Section 4 shows the records relating to tornadoes and hurricanes, and section 5 illustrates the progress in extended forecasting. Since there is no completely acceptable way of defining fore- cast accuracy, a number of different verification indexes are used. 2. GUIDANCE FORECASTS Guidance forecasts produced by NMC are scored in a number of different ways, depending on the forecast parameter and the manner in which it is presented. Nearly all these forecasts have shown improvement during the past decade or so. Maps that present the forecast shape of the sea level pressure pattern or the mid-tropospheric pattern are used by the field forecast offices to guide fore- casters in making local and statewide weather forecasts. NMC scores these maps using the S]^ score (Teweles and Wobus 1954) and progress in forecasting the maps is shown in figures 1 and 2. An arbitrary skill score in percent is also shown in the figures* The progress in 12-hour forecasts by NMC's Analysis and Forecast Division (A&FD) is shown in figures 3 and 4. The verifications are based on the threat score which is the ratio of the correct forecasts to the number of forecasts and occurrences, less the correct forecasts. Figure 3 shows the results from 60 points for measurable precipitation while figure 4 shows the results from areas with 4 or more inches of snow. 85% 65% j; 30 r 45% A. PRE-NWP. B. APRIL 1958 TO FEBRUARY 1964. BASED ON NWP 500-MB BAROTROPIC AND 3-LEVEL BAROCLINIC GUIDANCE. C. MARCH 1964 TO MAY 1966. BASED ON REED SURFACE GUIDANCE. D. JUNE 1966 TO AUGUST 1967. BASED ON P.E. SURFACE GUIDANCE. E. SEPTEMBER 1967 TO DECEMBER 1971. BASED ON P.E. SURFACE GUIDANCE. El INDICATES HUMAN IMPROVEMENT. Figure 1.--NMC average 30-hour surface S L scores A. PRE NWP. B. APRIL 1958 TO JUNE 1962. BASED ON BAROTROPIC GUIDANCE. C. JULY 1962 TO JUNE 1964. BASED ON 3-LEVEL BAROCLINIC GUIDANCE. D. JULY 1964 TO JANUARY 1971. BASED ON 3-LEVEL BAROCLINIC GUIDANCE TO JUNE 1966 AND P.E. BAROCLINIC GUIDANCE THEREAFTER. E. JANUARY 1971 TO DECEMBER 1971. BASED ON P.E. BAROCLINIC GUIDANCE. □ INDICATES HUMAN IMPROVEMENT Figure 2.--NMC average 36-hour 500-mb S. scores THREAT SCORE % i960 1962 1964 1966 1968 1970 YEAR Figure 3.--A&FD verification of measurable precipitation for a L2-hour period 25 I — 20 THREAT SCORE 15 % • 3-YEAR AVERAGE 1:2:1 ♦ RAW VALUE J I I I 'I'' 62-63 63-64 64-65 65-66 66 67 67 68 68-69 69-70 70-71 YEAR Figure 4.--A&FD verification of heavy snow forecasts for a 12-hour period 3. CITY AND NATIONAL FORECASTS Long-term verification records of various parameters are available for a few stations, and for a sum of stations over the conterminous United States. Nearly all these records show improvement in skill over the years. Figure 5 shows a decline in large temperature errors at Salt Lake City since 1949. Figure 6 shows a gradual improvement in precipitation and temperature forecasts for Chicago since 1942, reaching near 90 percent correct in the late 1960's. Figures 7 and 8 show the improvement in pre- cipitation forecasting for Washington, D.C. since 1945, and nationally (Alaska and Hawaii excluded) for 150 to 250 stations since 1959, respec- tively. (Although the 0-12 hour score is not shown nationally, it is currently about 87 percent, which is 5 percent better than 1959.) The latter three graphs are based on the ratio of correct forecasts to the total forecasts. A probability of 50 percent or more is considered a precipitation forecast. Correct precipitation forecasts involve the occurrence (non-occurrence) of measurable precipitation except for Chicago, where a trace was verified either way. In addition, a temperature forecast for Chicago was considered to be correct if it was within - 10 degrees (or less, depending on the season) of the observed. Since 1966 the data for the 150 to 250 stations are from the official verification program (Roberts et al. 1967b); the yearly value represents the period April through March. 4. TORNADO AND HURRICANE STATISTICS The most destructive storms, in loss of life and damage to property, are tornadoes and hurricanes. Property damage and deaths from these storms, by 5-year periods, are shown in figures 9 and 10, respectively. Both figures show a steady increase in the economic loss, and a decrease in the number of deaths from the early 1900" s until the present. Figure 11 shows the number of tornadoes and tornado deaths normalized to the fre- quency per million population. As the population expanded, more tornadoes were reported, but there were fewer deaths per million. The decrease in the death rate from tornadoes and hurricanes is a reflection of the fore- casts and warnings issued by the NSSFC in Kansas City, the NHC in Miami, and the field offices. In addition, both the reporting network and com- munication system have improved over the years. The accuracy of the official 24-hour hurricane forecasts is shown in figure 12. Since the mid 1950's, the mean errors of the hurricane tracks have decreased by nearly 33 percent. It should be pointed out, however, that the verification system was changed, since the forecasts from 1956- 67 were verified 28 hours after data time while those from 1968-71 were verified 24 hours after data time. 5. EXTENDED FORECASTS The long-term performance accuracy of 5-day precipitation and temperature forecasts is measured statistically by a "skill score." This score, fre- quently used in forecast verification, represents the mean performance • 5-YEAR AVERAGE 1:2:4:2:1 ■ 3-YEAR AVERAGE 1:2:1 D RAW VALUE NUMBER OF ERRORS >10° 1949 1951 1953 1955 1957 1959 1961 1963 1965 1967 1969 1971 YEAR Figure 5. --Annual number of maximum temperature errors equal to or greater than 10°F for Salt Lake City 95 r PERCENT CORRECT 90 85 80 75 • 5-YEAR AVERAGE 1:2:4:2:1 ♦ 3-YEAR AVERAGE 1:2:1 o RAW VALUE J—l I 1 J I— J I I— L JL_1 J— »- 1942 1945 1950 1955 YEAR I960 1965 1970 Figure 6„ --Percent of correct precipitation and temperature forecasts for Chicago, based on 2 forecasts per day over 2 periods PERCENT CORRECT • 5-YEAR AVERAGE 1:2:4:2:1 ♦ 3-YEAR AVERAGE 1:2:1 o RAW VALUE 1945 1949 1953 1957 YEAR 1961 1965 1969 Figure 7, •Percent of correct precipitation forecasts for Washington, D.C., based on 4 forecasts per day over 3 periods 90 85 PERCENT 80 CORRECT 75 12-24 HOUR FORECAST HOUR FORECAST >••-" .>•■ — r' • 3-YEAR AVERAGE 1:2:1 ■ RAW VALUE 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 YEAR Figure 8. --Percent of correct local precipitation and no precipitation forecasts nationally (average of 150 to 250 stations) 1600 1200 1000 2100. 600 - = 400 200 to o in o in o in o in o in cd ■ — cvi csi co CO 1 "* -4- in in co cor— ct? oo o"> oo cn en en go en en oo oo PERIOD * Two-year average PERIOD 2400 2100 c^, 1800 | 1500 o 1200 CO § 900 s 600 Figure 9„ --Tornado damage in millions of dollars (adjusted to 1968) and tornado deaths; 5 -year averages 7000 6000^ 300 uJjJ CTl ^■0'^-CJ5^-CT>^-CT5^-CT> ^— cnj cvj co co ^^ ^^ in in fr * cp in caincoincainco »— N cn <<) n LO CD IT) CD LO CD LO CD LO CD LO CD tO cd cd ^^csicsicoeo^^mmco s persistV^ 20 PERSIST >l 10 C^-On in " \ 1 ^\ 1 1 1 1 1 n , HiV-^"""^ 2 3 4 5 6 2 3 4 5 6 DAY DAY I.0LD PROGRAM FROM FEBRUARY 1968 TO JANUARY 1970 I. NEW PROGRAM FROM FEBRUARY 1970 TO DECEMBER 1971 Figure 14. --Extended forecast surface pressure prog verifications 11 little improvement in short-range forecasts on the mesoscale (aviation terminals and severe storms) , and it is in these areas where more emphasis is needed. 7. ACKNOWLEDGMENTS This report would not have been possible without the help of the Emergency Warnings Branch of the NWS Headquarters, the Western Region Headquarters, NMC, NHC, NSSFC, and the Chicago, Salt Lake City, and Washington, D.C. Forecast Offices. Thanks are also due to Mrs. Nora Keel for typing the report. REFERENCES Cressman, George P., "Public Forecasting—Present and Future," A Century of Weather Progress , American Meteorological Society, Boston, Mass., 1970, pp. 71-77. Roberts, Charles F. , and Porter, John M. , "Recent Trends in the Accuracy and Quality of Weather Bureau Forecasting Service," ESSA Technical Memo - randum WBTM FCST-8, November 1967a, 14 pp. Roberts, Charles F. , Porter, John M. , and Cobb, Geraldine F. , "Report on the Forecast Performance of Selected Weather Bureau Offices for 1966-67," ESSA Technical Memorandum WBTM FCST-9, December 1967b, 52 pp. Teweles, Sidney, and Wobus , Hermann B. , "Verification of Prognostic Charts," Bulletin of the American Meteorological Society , Vol. 35, No. 10, December 1954, pp. 455-463. (Continued from inside front cover) WBTM FCST 11 Report on Weather Bureau Forecast Performance 1967-1968 and Com- parison With Previous Years. Charles F. Roberts, John M. Porter, and Geraldine F. Cobb, March 1969. (PB-184 366) WBTM FCST 12 Severe Local Storm Occurrences 1955-1967. Staff, SELS Unit, NSSFC, Maurice E. Pautz, Editor, September 1969. (PB-187 761) WBTM FCST 13 On the Problem of Developing Weather Forecasting Equations by Sta- tistical Methods. Charles F. Roberts, October 1969. (PB-187 796) WBTM FCST 14 Preliminary Results of an Empirical Study of Some Spectral Character- istics of Skill in Present Weather and Circulation Forecasts. Charles F. Roberts, November 1969. (PB-188 529) WBTM FCST 15 Weather Bureau Forecast Verification Scores 1968-69 and Some Per- formance Trends From 1966. Robert G. Derouin and Geraldine F. Cobb, May 1970. (PB-192 949) NOAA Technical Memoranda NWS FCST 16 Weather Bureau April 1969-March 1970 Verification Report With Special Emphasis on Performance Scores Within Echelons. Robert G. Derouin and Geraldine F. Cobb, April 1971. (COM-71-00555) NWS FCST 17 National Weather Service, May 1970-April 1971 Public Forecast Verifi- cation Summary. Robert G. Derouin and Geraldine F. Cobb, March 1972. (COM-72-10484) "mat A0000750413S5