Model Evaluation
Grading Model Performance
Physically-based model evaluation is an essential element of all GMI work. We have developed a variety of methods for evaluating and grading model performance in the stratosphere. These methods have been described in several GMI publications (Douglass et al., 1999; Strahan and Douglass, 2004; and Douglass et al., 2004), which can be found under GMI Publications.
The diagnostics presented here are part of a larger effort to evaluate chemistry-climate models (CCMs). The Chemistry-Climate Validation Activity for SPARC (CCMVal) is project that addresses the need for consistent evaluation/validation of CCMs with detailed descriptions of the stratosphere. Participants in CCMVal have developed a number of diagnostics for stratospheric dynamics, transport, chemistry, and radiation. Please visit the CCMVal Evaluation Table for a complete listing of the diagnostics.
In this section, you can download model diagnostics packages. Each package contains the necessary data sets, IDL routines, and instructions for applying the diagnostic to your own model results.
Model Diagnostic Tests
The Antartic Vortex Transport and Mixing Test
This diagnostic examines a model’s ability to produce and maintain the dynamical barrier to transport found at the edge of the lower stratospheric Antarctic vortex in spring. The presence of a realistic vortex edge is important for containment of chemical perturbed air and the simulation of O3 loss. This diagnostic is described in detail in Strahan and Douglass [2004].
Required data set: This test uses 12 years of HALOE CH4 and temperature measurements between 44oS-80oS from September 1 through November 30. Data from 2002 is excluded because of the unusually early vortex breakup. HALOE temperature measurements are used to interpolate the CH4 observations to isentropic surfaces. Probability distribution functions are calculated on the two potential temperature surfaces for 6 space-time bins: for each of the 3 months (Sep, Oct, Nov), there is one bin each for the latitude ranges 44-60oS and 60-80oS. Because of the low interannual variability in southern hemisphere dynamics, it is feasible to combine the measurements from 12 years of observations. HALOE’s sparse sampling (15 profiles around a latitude circle per day) prohibits the calculation of meaningful PDFs from a single year’s data.
You may download this ‘processed’ data set from this web page. The data file containing HALOE pdfs was created in IDL by the ‘save’ command. It can be read into an IDL session using the ‘restore’ command. This data file contains HALOE CH4 on seven isentropic surfaces (450, 600, 800, 1000, 1200, 1400, 1600K), although the ‘official’ test only includes the lowest two levels.
If you are unable to use IDL data sets, please contact Susan Strahan for data in an alternative format.
Analysis software:
There are several IDL procedures that you may download for the calculation of PDFs from your model output. The routine requires model output on a regular grid. Please read the comments inside ‘calc_model_pdf.pro’ for more detailed instructions. This procedure calls a number of other IDL procedures, all of which are available here.
Download this test: You may download a detailed description of this test and an example of how to grade a model with this test. There are 6 required IDL routines and 2 IDL savesets with the necessary HALOE data. They have been tarred together and can be downloaded here.