SCRIPUC SAN DIEGOLƏRARGRAPHY LIBRARY OPTICAL SYSTEMS GROUP Jan 1990 TECHNICAL NOTE NO. 217 UNIVERSITY OF CALIFORNIA, SAN DIEGO 3 1822 03387 3449 AUTOMATED CLOUD COVER & VISIBILITY SYSTEMS FOR REAL TIME APPLICATIONS J. E. Shields T. L, Koehler M. E. Karr R. W. Johnson UNIVERSITY OF CALIFORNIA SAN DIEGO The material contained in this note is to be considered proprietary in nature and is not authorized for distribution without the prior consent of the Marine Physical Laboratory and the Geophysics Laboratory TTY Contract Monitor, Dr. J. W. Snow Atmospheric Sciences Division ERSI AN ORN UCHS *1866 Prepared for The Geophysics Laboratory, Air Force Systems Command United States Air Force, Hanscom AFB, Massachusetts 01731 under contract No. F19628-88-K-0005 SCRIPPS INSTITUTION OF OCEANOGRAPHY MARINE PHYSICAL LAB San Diego, CA 92152-6400 TECHNICAL NOTE NO. 217 AUTOMATED CLOUD COVER & VISIBILITY SYSTEMS FOR REAL TIME APPLICATION This Technical Note contains the extended summary of the January 1990 presentation made at the Cloud Impacts on DOD Operations and Systems 1989/90 Conference at the Naval Post Graduate School, Monterey, California. www. SUMMARY WA Responding to a well recognized need by many in both the modeling and operational communities for an improved capability in the collection and assessment of whole sky cloud characteristics, as well as sector visibility variabilities and statistics, a new generation of video based imaging systems has been developed and fielded by the Marine Physical Laboratory. One of these systems, the Whole Sky Imager, has been de- ployed at several widely separated portions of the United States, and has gathered several million images appropriate for determining cloud cover at very high spatial and temporal resolution. Cloud coverestimates derived from a 7-month sample of these cloud images shows very good agreement with observed sky cover amounts. The capabilities of the Whole Sky Imager and the other imaging systems is discussed, followed by an overview of the current status and quality of the WSI data base. BAK was dass BA we TABLE OF CONTENTS Summary ........ A. : List of Illustrations... E 1.0 2.0 i Introduction ....... Automated Systems for Cloud Assessment and Visibility Determination 2.1 Whole Sky Imager.. 2.2 Real Time Cloud System .... 2.3 Portable WSI... 2.4 Night-time WSI ..................... 2.5 Horizon Scanning Imager .......... 2.6 Composite Cloud/Visibility System ... . 3.0 . WSI Cloud Data Archive 3.1 Background........... 3.2 Current Extent of the Data Base ...... 3.3 Cloud Determination ........ v vicino Aww ww wWN 4.0 WSI Evaluation and Results . 4.1 Comparison with Observer ....... 4.2 Specific Temporal Dynamics Case Study W W W 5.0 Conclusion .... W WW 6.0 Acknowledgements WWWWW v 7.0 References .................. w wwwwwwwwww222 cereweweweweweweewww-WXYZYWwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww de . LE LIST OF ILLUSTRATIONS Page Fig. # Figure Title 1 Whole Sky Imager Camera Assembly.. 2 Sample WSI Image, acquired at 650 nm. CHOMANN MUSSANNININ .... 1 ................ WIWINU SAVIMAS. WAVAMIWKOVOU Sample Cloud No cloud Decision Image.. Sample One-Minute Cloud/No cloud Decision Image ... MRSA WWW 5 Sample image from Real Time Cloud System...... 6 Portable Whole Sky Imager ......... ..... 8 .......... Home Ver.. WSI Data Base, number of acquired data days ....... WSI Basic Image Processing Flow Chart ............. Distribution of Total Cloud Cover determinations, WSI and Weather Observer....... Total Cloud Cover determination difference for WSI minus observer Total Cloud Cover Time series, WSI and Observer ............. Maximum Cloud Free Arc Time series, WSI ................ WSI Cloud Cover Image at 1420, from above time series.. WSI Cloud Cover Image at 1450, from above time series...... 32 10 11 12 13 14 MALARI 1. INTRODUCTION A family of imaging systems for use in automated cloud assessment and sector visibility determination has been developed by the Marine Physical Laboratory of Scripps Institution of Oceanography (Johnson, 1989). This paper will give a brief overview of these systems, including the Whole Sky Imager (WSI) for long term cloud studies, the Real Time Cloud system for on-line operational support, and the Horizon Scanning Imager (HSI) for sector visibility determinations. Each of these systems is an automated unit operating under micro- computer control, gathering digital imagery suitable for automated processing and analysis. The WSI systems have been operating at several locations during the last year, acquiring images once a minute, 12 hours a day. These data are archived and subsequently converted to cloud/no cloud decision images. The resulting cloud database can be utilized for a variety of applications including model evaluation and development (Hering, 1989). We will review the acqui- sition and processing of the WSI cloud database, and discuss some of the emerging statistical relations from the WSI cloud determinations. Fig. 1. Whole Sky Imager Camera Assembly . Fig. 2. Sample WSI Image, acquired at 650 nm image, the center is at the zenith overhead, and the edges are near the horizon. The black square is the sun occultor, which shades the lens in order to minimize stray light. A tower may also be seen in the field of view near the occultor. In the processing, a determination is made at each point in the image, yielding a cloud deci- sion image at full spatial resolution. The cloud decision image is illustrated in Fig. 3. In this illustration the areas identified as sky are blue, the occultoris black indicating a "no data" region, and the pixels identified as thin or opaque cloud are yellow and white respectively. 2.0 AUTOMATED SYSTEMS FOR CLOUD ASSESSMENT AND VISIBILITY DETERMINA- TION 2.1 Whole Sky Imager The Whole Sky Imager (WSI) is a ground-based electronic imaging system, which monitors the upper hemisphere. It is a passive, i.e. non-emissive system, which acquires calibrated multi-spectral images of the sky dome. The system, shown in Fig. 1, views the sky through a series of spectral and neutral density filters, using a fisheye lens to acquire most of the sky dome. A fixed gain CID (charge injection device) solid state camera is utilized, and a full set of radiometric and geometric calibrations are acquired prior to fielding the system. Data are acquired in 512 x 480 format, which yields 1/3 degree spatial resolution. This corresponds to a 17 meter footprint, for a cloud layer at 3 km height. In the field, cloud images are acquired under control by an IBM AT-class microcomputer. This is a stand- alone unit, requiring essentially no user intervention; control of all peripheral functions is fully automatic. Four digital images are acquired every minute, and archived on 8 mm tape, for post-processing. Approxi- mately 1.2 gigabytes are acquired and archived per week at each site. A sample radiance image is shown in Fig. 2. In this field spatial and temporal dynamics. The WSI acquires calibrated multi-spectral sky radiances every minute. From these, automatic cloud determinations are pro- duced with 1/3 degree spatial resolution. The system is currently deployed at 7 sites throughout the country, and as of 1 Jan 90, had acquired an average of 14 months of data per site. The data will be further discussed in Section 3. 2.2 Real Time Cloud System An outgrowth of the WSI is the Real Time Cloud system, which provides automatic cloud assessments in real time in the field, for on-line operational support. This system acquires sky radiances, much like the WSI. The user may input a track of interest, such as the track of a satellite or drone. The cloud determinations are then made from the measured radiances and presented to the user in color-coded image format, i.e. blue for sky, white for opaque cloud, and yellow for thin cloud as shown in Fig. 5. In the user's display, the track is color coded by Fig. 3. Sample Cloud/No cloud Decision Image. This shows the full resolution results taken at 10 minute intervals. WHOLE SKY IHAO Full resolution images such as shown in Fig. 3 are saved every ten minutes. At one minute intervals, the subset shown in Fig. 4 is saved. (Fig 4 is the one minute cloud decision image.) This image consists of a subset of rows and columns, each saved at full resolution (i.e. we save 33 rows, and 33 columns, for 33 x 512+33 x 480 resolution). Thus for cloud free arc length statistics, we have full resolution, but for cloud areal statistics such as cloud cover, we have reduced resolution in the one minute data. Our early studies indicated an uncertainty of approximately a 1% in the cloud cover computed from the one minute data as compared with the ten minute full resolution data. SECOUR COLUMBIA, HO 13. Mag 2130 Fig. 5. Sample image from Real Time Cloud System kehadiran segundo este finale d i s ogg i orno in giornata di venditionin rekisterin Note user-input track. altitude, and superimposed on the image. Percent cloud cover, both total and opaque, are presented to the user. (Currently cloud coveris assessed only for the sky dome; techniques to compute cloud cover specifically along the track are in development.) If desired, the user may view previously stored images, which are video looped to display temporal changes in the cloud cover. The current system is connected by optical fiber to a WSI unit, and has its own micro-computer for independent image acquisition and display. The system is currently de- ployed in New Mexico, for support of the HELSTF laser site, however it has application to a wide variety of operational and test scenarios. Fig. 4. Sampie One-Minute Cloud/No cloud Decision Image. This shows the subset saved at 1 minute intervals. In summary, the Whole Sky Imager is a system for cloud assessment, specifically for the archival of cloud 2.3 Portable WSI The portable WSI unit is shown in Fig. 6. This unit, designed for short term deployments and/or special applications, is similar to the WSI, but has a more transportable support and environmental control hous- ing. The portable system is currently installed at the University of Wisconsin, for a cooperative study with Eloranta and Grund, using their HSRL lidar system. The intent here is to relate the WSI cloud determina- tions to the cloud optical depth determined by the lidar system. targets and the horizon, the visibility is computed, using contrast transmittance theory. This system essentially simulates the human determi- nation of visibility, but it has several advantages. The human must theoretically have black targets (i.e. -1 inherent contrast), of given angular size, seen directly against the horizon. With the HSI, since measured radiances are compared directly, the target need not be of specific size and does not need to be directly adjacent to the horizon. And since inherent contrast is a user specified input in the computation, the requirement for a black target may be relaxed somewhat. Numerous computations of uncertainty in the com- puted visibility as a function of parameters such as the target inherent contrast show that the system is most accurate when the target range is close to the limit of visibility. For this reason, the system is enabled to make an active selection of which targets to use in determining visibility, depending on the conditions that obtain at the time of measurement. This active target selection is one of the advantages this system has in comparison with staring photometers. Finally, it should be noted that unlike a point scatter meter, this system measures visibil- ity for the integrated path of sight, and returns the visibility for all sectors. 2.6 Composite Cloud/Visibility System Finally, in addition to the above systems, we have the composite system, which has the capabilities of both the HSI, for sector visibility determinations, and the Real Time Cloud system, for cloud field determinations. This system, currently on site at Geophysics Lab, is under development for use as an automated observing station. Our intent is to develop night-time capability for both visibility determination and cloud assessment with this system. 3.0 WSI CLOUD DATA ARCHIVE Fig. 6. Portable Whole Sky Imager 2.4 Night-time WSI We have recently designed a night-time WSI, and are beginning the fabrication and test stage. This unit utilizes an image intensifier, in conjunction with a CID camera. Flux level may be controlled through a combi- nation of voltage control on the intensifier, and variable on-chip camera integration time. Analysis of night-time sky radiances acquired by our group during the 1960's (Duntley, 1970) indicates that the daytime cloud deci- sion algorithms should be reasonably applicable at night, down to quarter moon illumination levels. 2.5 Horizon Scanning Imager The Horizon Scanning Imager (HSI) is a system designed for for automated determination of sector visi- bility. Like the WSI, this system utilizes a CID camera, but this time with a narrow field of view and a photopi- cally corrected response. A sequence of images around the horizon is acquired, at predetermined azimuthal angles. When the instrument is first installed, and at any time later, black targets of interest within each scene are identified. From the measured relative radiance of the The Whole Sky Imager has been operating routinely at several sites in the continental US for over a year. This section discusses the background of this data archive, and then reviews the data processing and cloud algo- rithms. 3.1 Background The WSI data base was acquired primarily in support of the SDI ground-based laser program. A variety of models predict joint multi-site probabilities of cloud free lines of sight (CFLOS) for earth to space, and cloud free arcs (CFARC). These joint probabilities depend on a W CUNEONATOVA NUMRINO* WHAT WWS H 5 6 Fig. 7. WSI Data Base, number of acquired data days. Sites 1-3 and in New Mexico, 4 in Calif., 5 in Montana, 6 in Florida, and 7 in Missouri. variety of intermediate level relationships, such as CFLOS and CFARC as a function of sky cover and zenith angle, site-to-site correlation as a function of separation dis- tance, and temporal relations such as recurrence and persistence probabilities. The WSI data base is in many ways an ideal set for model evaluation. The model input, i.e. sky cover, is measured directly. The various intermediate level re- sults mentioned above may be computed directly from the digitized data base, and the final multi-site joint probabilities may be computed directly from the data. As a result, WSI can provide the data base for testing existing models, development of new models, and docu- mentation of a mini-climatology of CFLOS and CFARC at specific sites of interest. There appears to be considerable interest in the user community in using these data in a variety of other applications. For example, there is interest in creating a climatology of optically thin cloud cover, perhaps as a function of optical depth. Additionally, the fact that absolute radiometric calibrations are acquired for the instruments means that a data base of calibrated radiance could be generated from the data, for background clutter and other applications. 3.2 Current Extent of the Data Base We are currently archiving WSI data at seven sites: three in New Mexico, and one each in California, Mon- tana, Missouri, and Florida. Four images are archived every minute, 12 hours a day, for approximately 87,000 images per month per site (5 gigabytes per month per site). As of 1 Jan 90, we had acquired an average of 14 months of data per site. The number of days of acquired data per site is shown in Fig. 7. Although this is a large data base, it is not as unwieldy as might be expected, due to the 2 gigabyte capacity of the 8 mm tapes. The raw data to date fits on approximately 400 cassettes, and the cloud data will fit on approximately 100 cassettes (25 with compaction). Since these data are acquired in digital format, automated quality control and data reduc- tion processes are very fast and reproducible in compari- son with photographic techniques. Initially, we had some trouble with hardware driven system down-times. Installation of better air filters in the computers corrected the dust-caused problems with the tape drives, and installation of chillers directly on the camera housings corrected a problem with heat sensitiv. ity in the cameras in the warmer locations. Since these upgrades were accomplished, most stations have been quite reliable. As a result, we have had at least 4 sites archiving concurrent data essentially every day since Feb 89, with 6 concurrent sites 2/3 of the time, and 7 sites yielding concurrent data 1/3 of the time. 3.3 Cloud Determination The cloud determination sequence is illustrated in Fig. 8. We start with four measured radiances; a blue, a red, and a blue and a red trimmed with neutral density to acquire those regions which are offscale bright in the first two filters. A variety of additional neutral densities and aperture settings are used to bring the radiances to the proper onscale level. A number of calibrations are then applied to these four acquired radiance images. The most significant are the linearity calibration, which corrects for any non-linearity of the basic sensor, and the absolute calibration, which corrects for differences in the pass bands of the spectral filters, non-neutrality of the neutral density filters, and so on. (The resulting cali- brated radiance images are not saved at this time, due to the extra processing time required.) Ratios of red to blue radiance are then computed, including any correction for small differences in image size. The best blue/red pair to use is then selected on a pixel by pixel basis, to generate the composite ratio image. These ratios are then saved to tape in image format, for further processing. The ratio tapes are next processed to yield the cloud tapes. At this point in time, the cloud algorithm is a very simple thresholding scheme; any pixel with a red/blue ratio above a certain value is identified as cloud. Sepa- rate thresholds distinguish thin cloud from opaque cloud. In general this scheme works quite well. The algorithm has no trouble with clouds of varying brightness (e.g. dark grey to white). "Also, since the determination is BASIC MAGERY CORRECTED IMAGERY COMPOSITE RATO DELIVERABLE DATABASE CALIBRATION FUNCTIONS AADIANCE CONVERSIONS CLOUDVNO-CLOUD DECISION ALGORITHMS CALIB. BLUE IMAGE BLUE MAGE 512x512 x 8 RADIOMETRIC LINEARITY BLUERED RATIOS RADIOMETRIC SENSITIVITY UP.SUN PED IMAGE 512 X 512 x 8 CALIB RED MAGE DERIVED PAODUCTS DOWN SUN OPTICAL DISTORTIONS IMAGE RATIO COMPUTATIONS NEAR HORIZON SPATIAL DISTRIBUTIONS COMPOSITE BLUEIRED RATIO IMAGE OPTIMUM CLOUDVNO-CLOUD IMAGE TWILIGHT FIELD OF VIEW DEFINITION TEMPORAL DISTRIBUTIONS BLUE + N.D. MAGE 512 X 512x8 CALIB (BLUE) MAGE DAWN STATISTICAL PARAMETERS SENSOR CHIP UNIFORMITY AS REO'D PIXEL SELECTIONS FOR OPTIMUMIZED COMPOSITE BLUEIRED) RATIOS REGISTRATION ADJUSTMENTS REDEND IMAGE 512x512x8 CALIB. FREDY MAGE FLUX CONTROL THRESHOLDS Fig. 8. WSI Basic Image Processing Flow Chart cloud algorithm identifies some clear cases as 1/10 cover, but in all other cloud categories the match in frequency of observance is excellent. TOVKY E CAP* TO AP# F # B 1989 RELATIVE THQUENCY IN o 1 21 7 131 14 15 16 1 SKY COVER AMUNT (EENTH made on a pixel by pixel basis, small clouds are readily identified. Comparison of the radiance images with the processed cloud images shows that the cloud algorithm generally agrees well with the visual assessments. We are also investigating directionally-dependent corrections to the cloud determination algorithms, based on the Hering FASCAT radiance model (Hering, 1985). Since the clear day ratios tend to be slightly higher at the horizon and near the sun, a correction which depends on scattering angle and relative air mass should enhance the accuracy of the cloud determination. It should be em- phasized that an enhanced algorithm would only make small changes in the final cloud images, since the direc- tional variance in the ratio normally is quite moderate; indeed, one would expect only selective improvement, since the fixed threshold results compare very well with standard observers, as discussed below. 4.0 WSI EVALUATION AND RESULTS 4.1 Comparison with Observer Much of the wsi data from the Columbia site has been processed using the preliminary (fixed threshold) algorithm, in order to give a preliminary assessment of the accuracy of these techniques. Fig. 9 shows the cloud cover distribution from 7 months of WSI data, compared with the values of total sky cover reported on the Na- tional Weather Service Form 10's. The WSI values are from the one minute image at the reported time of the weather observation. This plot is for the six hours surrounding local apparent noon. The comparison be- tween WSI and observer is in general quite good. The Fig. 9. Distribution of Total Cloud Cover determinations, WSI and Weather Observer. Another indication of data quality is a direct case-by- case comparison between WSI and observer. For Fig. 10, the difference between WSI and observer has been computed for each case. That is, a WSI value of 7/10 and observer value of 5/10 would be a difference of 2 categories. The distribution of category differences is shown in Fig. 10. The majority of the cases show a category difference of 0. The average difference is less than half a category, i.e. much less than 1/10 cloud cover. 4.2 Specific Temporal Dynamics Case Study Although the observer to WSI comparison is gener- ally good, the WSI data can show much more variance COLUMBIA, M APY 31 T *** TOTA 1450, both WSI and observer show approximately 20% cloud cover, but during the intervening hour the cloud cover increased to nearly 80%. The cloud images at the middle and end of this hour are shown in Figs. 13 and 14. The fastest rate of change during this hour occurred at between 1400 and 1413, when the cloud cover changed from 20 to 74%. 10 ལོའི་ས་དང་ད་ RELATIVE FREQUENCY (9) COD CIVER CATERY DIFFT MENCE (W SERWUM Fig. 10. Total Cloud Cover determination difference for WSI minus observer. than the observer data, simply due to the limited tempo- ral frequency of the observer values. One particularly dynamic day, 14 April at Columbia, Mo., is illustrated in Figs. 11 and 12. In Fig. 11, which shows the cloud cover determinations for both the WSI and the observer, the observer values are consistent with the WSI, but they certainly do not show the true variability, due to their limited temporal frequency. For example, at 1350 and Fig. 13. WSI Cloud Cover Image at 1420, from above time series. COLUMBIA ME 14 APRIL **** NEWWE TOTAL CLOUD COYER (1) | 13 14 15 16 17 18 19 20 21 2 2 ne (49) Fig. 11. Total Cloud Cover Time series, WSI and Observer COLUMBIA, M) 14 APR 1998 130 Fig. 14. WSI Cloud Cover Image at 1450, from above time series. EN 8.8 HAK CTARE (06) 8 1 SI 8 8 8 One can also see, in Figs. 13 and 14, a horizontal line part way down the image in the Northem sky. This line represents an arc traveling from horizon to horizon, rising to a 45 degree zenith angle. The maximum cloud free arc length (CFARC) along this arc has been plotted in Fig. 12. Comparing Figs. 11 and 12, one can see that in general there is a tendency for short CFARCs to occur with high cloud cover. As expected however, the maxi- mum CFARC is not always well related to the cloud cover at a given point in time. For example, at 1450, the TORTAS w 12 13 14 15 16 20 18 THE (GM) Fig. 12. Maximum Cloud Free Arc Time series, WSI K-0005. Our thanks to Mr. Donald Grantham and I William Snow of GL for their guidance. The HSI effort is sponsored by Geophysics Lab contract #F19628-88- C-0154 under the guidance of Dr. H. Albert Brown. The authors would also like to recognize the outstanding efforts of our colleagues at Marine Physical Laboratory: Wayne Hering, Monette Karr, Gene Zawadski, Jack Varah, Harry Sprink, Melissa Ciandro, Tim Romedy, John Malo, and Peter Pak, for technical support, and Phil Rapp and Carole Robb for publications support. 1 cloud free arc length was quite short, in spite of the low cloud cover. Examination of the cloud image for this time (Fig. 14) shows that this occurred because the cloud band happened to lie over the selected arc. 5.0 CONCLUSION With the development of a family of digital imaging systems, it has been possible to support a number of applications requiring monitoring of the cloud and visi- bility environment. The Whole Sky Imager has been monitoring specific sites for post analysis and evalu- ation of cloud fields, with high temporal and spatial resolution. The Real Time Cloud system is currently supporting the need for immediate feed-back of the cloud field over the sky dome and in specific directions, for operational support. The portable system may be used for short term test and evaluation programs. The Horizon Scanning Imager can provide visibility deter- minations, for both prevailing and sector visibilities. And finally, the Composite system should provide both real time cloud and visibility results, for a variety of applications including automated weather stations. : The WSI has been used specifically to gather a data base of over a million images for assessment of cloud fields. The data consists of cloud decision images, containing an assessment of the cloud condition in each direction with 1/3 degree spatial resolution, and one minute temporal resolution. These data are currently undergoing processing and quality evaluation. The preliminary results appear quite good, in comparison with standard meteorological observations. The cloud images may be used in several applications, including determination of cloud free line of sight and cloud free arc probabilities, and multi-site joint probabilities. 6.0 ACKNOWLEDGEMENTS 7.0 REFERENCES Duntley, S. Q., R. W. Johnson, J. I. Gordon, and A. R. Boileau, (1970). Airbome Measurements of Optical Atmospheric Properties at Night, University of Cali- fornia, San Diego, Scripps Institution of Oceanogra- phy, Visibility Laboratory, SIO Ref. 70-7, AFCRL- 70-0137, NTIS NO. AD 870 734. Hering, W. S. and R. W. Johnson, (1985). The FASCAT Model Performance Under Fractional Cloud Condi- tions and Related Studies, University of California, : San Diego, Scripps Institution of Oceanography, Visi- bility Laboratory, SIO Ref. 85-7, AFGL-TR-84-0168, NTIS NO. AOA 085 451. Hering, W. S., (1989). Evaluation of Stochastic Models for Estimating the Persistence Probability of Cloud- Free Lines-of-Sight, University of California, San Diego, Scripps Institution of Oceanography, Marine Physical Laboratory, GL-TR-89-0275. Johnson, R. W., W. S. Hering, and J. E. Shields, (1989). Automated Visibility and Cloud Cover Measurements with a Solid-State Imaging System, University of California, San Diego, Scripps Institution of Oceanog- raphy, Marine Physical Laboratory, SIO 89-7, GL- TR-89-0061. This work was sponsored by Geophysics Lab, Air Force Systems Command, under contract #F19628-88- WORKS WWWWWWW PRO