8. File format and content
The MODIS Level 2 aerosol product files (MOD04_L2 for Terra and MYD04_L2 for Aqua for the 10 km product, MOD04_3K for Terra and MYD04_3K for Aqua for the 3 km product, known commonly as MxD04) are stored in Hierarchical Data Format (HDF). HDF is a multi-object file format for sharing scientific data in multi-platform distributed environments. HDF files can be accessed through HDF library subroutine and function calls, which can be downloaded from the HDF web site (https://www.hdfgroup.org). Proprietary software, such IDL and MATLAB have built in routines to process IDL files. Also, free tools, including HDFLook (https://www.hdfeos.org/software/HDFLook.php) and Panoply http://www.giss.nasa.gov/tools/panoply/ are available, and can be found on the MODIS atmosphere web site (http://modis-atmos.gsfc.nasa.gov/tools.html).
All C6 MxD04 data files (and corresponding L1B and geolocation data files used to create them) are available on-line (at no charge) through the user-friendly Level 1 and Atmosphere Archive and Distribution System (LAADS; http://ladsweb.nascom.nasa.gov/data/search.html). MxD04 HDF file names have this naming convention:
where x is O or Y for Terra or Aqua, RR describes the resolution of the product (L2 for 10 km, 3K for 3 km), YYYY, DDD and HHMM are four digit year, three digit Julian day, and time of day in UTC, CCC is the collection (006 for C6), and YEARDAYHRMNSC represents when the file was processed.
The MxD04_L2 files include many parameters, each stored as a Scientific Data Set (SDS) within the HDF file. Only a subset of these parameters are available in the 3 km files. In addition to the SDSs listed in previous sections of this document (in Table 8 for land and Table 3 for ocean), the HDF file includes SDSs for location (Latitude, Longitude), geometry (Solar and Sensor zenith and azimuth angles). The full SDS list can be found on the MODIS-atmosphere web site (http://modis-atmos.gsfc.nasa.gov/MOD04_L2/format.html), and may be updated as needed.
The MxD04_L2 product contains data that has a spatial resolution (pixel size) of 10 x 10 kilometers (at nadir), with larger size closer to swath edges. Each MxD04 product file represents a five-minute time interval of L1B observations (a granule), such that the output grid is 135 pixels in width (cross track) and 203 pixels in length (along track). Every tenth granule is 135 by 204 pixels. The geolocation (location and geometry) of each output pixel is computed from aggregation of 10x10 boxes of L1B 1 km input, taking the average of the 4 central (5,5), (5,6), (6,5), (6,6) 1-km L1B input pixels. Again, more details can be found within the MODIS-atmosphere web site (http://modis-atmos.gsfc.nasa.gov/MOD04_L2/grids.html), and may be updated as needed.
8.1 Notes on global sampling
Although MODIS views a swath approximately 2300 km wide, there are daily gaps near the equator and multiple viewings of the poles. Near swath edges, view angles exceed 55˚, such that individual pixels can represent three times the surface area of nadir views. Thus parts of the globe may be over or under sampled by each MODIS instrument each day.
During a given month, each instruments’ MxD04_L2 products represent MODIS observations on 1.1x108 pixels (at 10 x 10 km). Except for the Deep Blue algorithm (not included in this version of the ATBD), the MODIS aerosol algorithms rely on observing dark surface targets, meaning that glint (over ocean), deserts (over land), ice, snow and clouds are avoided, and aerosol properties are not retrieved over them. Thus out of the monthly MxD04 coverage, only ~15% can be used for aerosol retrieval. These ‘valid’ pixels are comprised of approximately 75% ocean and 25% land retrievals, but varies monthly. High confidence data (QAC=3) account for approximately 60% and 25% of the valid data over land and ocean, respectively, while at least marginal confidence data (QAC≥1) accounts for about 85% and 98%, respectively. The algorithm over land has stricter criteria for high confidence than that over ocean.
The fraction of retrieval coverage versus MODIS-observed coverage varies by month. In addition, due to changing surface and cloud conditions, the QAC (quality) distribution varies by month. Fig. 40 shows how the number of retrievals (pixel counts, or Pk, black circles) and their QAC (plotted as QC, square symbols) varies by month in 2003, for Terra over land and ocean separately. Clearly, both the quality and quantity of the over-land retrievals has a strong seasonal dependence, with strong maxima in both during the northern summer. There is much less seasonal dependence over ocean. Interestingly, over the ocean, the maximum in pixel counts is not associated with the maximum in confidence. Note the relative dominance of QAC=3 over land, but QAC=1 over ocean, due the stricter QA criteria over each surface.
Figure 40: Time series (Terra, 2003) of monthly total counts over A) land and B) ocean. For each panel, black circles represent the monthly total pixel counts (Pk), the squares are the total pixel counts that are a given QC value, and the red triangles are the total confidence for that month (i.e., Qk = ΣiQi,k). Note the different seasonal dependencies of the curves. This data is from C5, but is expected to represent C6 coverage as well.
8.1 Recommendations for data usage
8.11 Total τ
According to the provisional validation (Section 5), the C6 MODIS retrievals of total τ can be validated as compared to AERONET. The expected uncertainties for τ over land remains the same from C5, but has been modified over ocean to account for a positive mean bias against AERONET.
In C6, the quantitative DT land and ocean joint t product (i.e., ‘Optical_Depth_Land_And_Ocean’) include data of minimum QAC>1 over ocean and QAC=3 over land. While we recommend using that SDS for some quantitative studies, for some applications it may be more appropriate to only examine the land or ocean separately derived τ SDSs (‘Effective_Optical_Depth_Average_Ocean’ and ‘Corrected_Optical_Depth_Land’). The DT land and ocean joint qualitative SDS ‘Image_Optical_Depth_Land_And_Ocean’ reports all valid (non-fill value) AOD, over land and ocean, regardless of QAC value. This product can be used to trace aerosol plumes and make aerosol maps, but should not be used for quantitative study.
It also should be repeated that over land, the algorithm permits retrieval of small negative τ values in order to avoid arbitrary biases in long-term statistics. MODIS does not have enough sensitivity over land to retrieve aerosol to better than ±0.05, meaning that in very clean conditions the algorithm cannot determine τ = 0.00 from 0.05 or -0.05. It is up to a user to decide what to do with these values, whether to force them to zero or some small positive value (for use within models). However, these retrievals of negative τ are considered to be good retrievals, and are assigned QAC=3, unless other conditions apply. Over ocean, however, retrievals of negative τ are not permitted, so that they are automatically forced to zero, but still can be considered as high quality retrievals.
8.1.2 Size parameters
The comprehensive C5 over land validation found that the size parameters over land lacked quantitative information (Levy et al., 2010). Therefore, over-land Ångstrom Exponent (AE) and fine-mode AOD SDSs have been removed from the C6 aerosol products. Values for η are reported when retrieved τ >0.2 and QAC≥1; however, these are considered to be diagnostics to the algorithm and do not necessarily have scientific value. Over ocean, η and Ångstrom exponent are reported, as long as τ is valid and the observed reflectance (at 0.86 μm) is above some threshold. Provisional validation shows that 68% of over ocean AE retrievals are within 0.41 of AERONET. This number rises to 80% if only retrievals with an AOD greater than 0.2 are considered (Levy et al., 2013).
8.1.3 Other derived and diagnostic parameters
At this point, the MODIS aerosol team has not evaluated the reliability of other parameters, including certain derived parameters (e.g. Mass Concentration) and diagnostic parameters (cloud fraction). The user is cautioned that there is no expected quantitative value to these parameters.
Occasionally, bugs and processing errors can slip into product production. The modis-atmosphere website maintains a list of any such errors, the data which is affected at http://modis-atmos.gsfc.nasa.gov/validation.html.
On these web pages, we have described the fundamentals of the dark target algorithm used for deriving aerosol properties from MODIS observed spectral reflectance, including updates in the newest Collection 6. In fact, this algorithm is comprised of two separate algorithms, one for deriving aerosol properties over ocean, the other over land. The quality assurance (QA) plan has been overhauled and is now more useful to users within the aerosol community. Operational C6 processing began in July 2014.
The core inversion of the C6 over-ocean algorithm (C6-O) remains similar to the pre-launch MODIS algorithm formulated by Tanré et al., (1997) and ATBD-96, as well as that described for C5 over ocean (C5-O) described in ATBD-09 and C4 over ocean (C4-O) by Remer et al., (2005). However, updates have been made to the look up tables, radiative transfer, cloud and other masking, and to the descriptions of the retrieved products. The expected error estimation is changed from ±0.03+0.05t to an asymmetrical formula (+(0.04 + 0.10τ), −(0.02 + 0.10τ)),
The algorithm for deriving C6 aerosol over-land (C6-L) is an update of the C5-L algorithm, which in turn was a complete overhaul from the pre-launch algorithm formulated by Kaufman et al., (1997) and ATBD-96 and the C4-L algorithm described by Remer et al., (2005). In C5-L, we updated a number of assumptions, including the representative global aerosol optical models, the VISvs2.11 surface reflectance parameterization, and the statistical implications of deriving below zero aerosol optical thickness. We also have converted the algorithm over land from an independent two-channel retrieval to a simultaneous three-channel inversion, in order to make use of aerosol information contained in the SWIR (2.11 μm) channel. In C6-L, we have updated the cloud mask to allow heavy smoke retrievals, fine-tuned the assignments for aerosol type as function of season/location, corrected bugs in the Quality Assurance Confidence (QAC) logic, and added diagnostic parameters such topographic altitude.
The C6 algorithm was tested on 8 months of data (Jan/July of 2003, 2008 and 2010 and Apr/Sept of 2008), and compared to the well-understood and documented C5 algorithm, and evaluated at co-located AERONET sites. Over land, AOD retrievals meet the expected accuracy levels (±0.05±0.15τ) defined by Remer et al., (2005), and a high bias at low AOD has been eliminated. MODIS/AERONET t regression had an equation of: y = 0.949x + 0.013, R = 0.86, with 69% within error. Over ocean, the expected accuracy of MODIS retrieved t are more stringent (+(0.04 + 0.10τ), −(0.02 + 0.10τ)), and the algorithm is performing very well (regression: y = 0.97x + 0.014, R = 0.937).
The provisional validation suggests that the operational C6 retrieval will be capable of deriving τ over both land and ocean within the defined expected uncertainty. We also expect that improved radiometric calibration has enhanced the quality of the aerosol retrieval, particularly allowing for trend analyses which were not possible with the C5-L aerosol products. The DT-algorithm is a mature algorithm, with known uncertainties and a continually improving track record. Even with the known limits of the DT products, we expect that Collection 6 of the MODIS retrievals (both Terra and Aqua) will provide an anchor for developing a long-term aerosol climate data record.