Identifying and removing structural biases in climate models with history matching

Abstract

We describe the method of history matching, a method currently used to help quantify parametric uncertainty in climate models, and argue for its use in identifying and removing structural biases in climate models at the model development stage. We illustrate the method using an investigation of the potential to improve upon known ocean circulation biases in a coupled non-flux-adjusted climate model (the third Hadley Centre Climate Model; HadCM3). In particular, we use history matching to investigate whether or not the behaviour of the Antarctic Circumpolar Current (ACC), which is known to be too strong in HadCM3, represents a structural bias that could be corrected using the model parameters. We find that it is possible to improve the ACC strength using the parameters and observe that doing this leads to more realistic representations of the sub-polar and sub-tropical gyres, sea surface salinities (both globally and in the North Atlantic), sea surface temperatures in the sinking regions in the North Atlantic and in the Southern Ocean, North Atlantic Deep Water flows, global precipitation, wind fields and sea level pressure. We then use history matching to locate a region of parameter space predicted not to contain structural biases for ACC and SSTs that is around 1 % of the original parameter space. We explore qualitative features of this space and show that certain key ocean and atmosphere parameters must be tuned carefully together in order to locate climates that satisfy our chosen metrics. Our study shows that attempts to tune climate model parameters that vary only a handful of parameters relevant to a given process at a time will not be as successful or as efficient as history matching.

Appendices

Appendix 1: Building emulators for history matching

What follows is a brief description of the methods we used to construct emulators for the constraints described in this paper. An emulator for element $i$ of $f(x)$ might typically be fitted as

$$\begin{aligned} f_{i}(x) = \sum _{j}\beta _{ij}g_{j}(x) + \epsilon _{i}(x) \end{aligned}$$

(2)

where $g(x)$ is a vector of specified functions of $x, \beta$ is a matrix of coefficients, and $\epsilon (x)$ is a stochastic process with a specified covariance function. As discussed in Sect. 2 there are many ways to build emulators and the way that is chosen will depend on the size of the PPE available, the type of constraint we wish to emulate and the relationships between the data and the parameters that we find. In this study we had access to large ensembles, and each of our constraints was a univariate quantity and so required less sophisticated modelling than a spatial field or time series might. Hence we fit the emulator mean functions, $\beta g(x)$ in Eq. (2) using a stepwise regression procedure described below.

The functions we consider adding to $g(x)$ were linear, quadratic and cubic terms in each of the parameters with up to third order interactions between all parameters considered. Switch parameters were treated as factors (variables with a small number of distinct possible “levels”) and interactions between factors and all continuous parameters were permitted. For a list of the parameters varied in the ensemble see Appendix 2.

Our fitting procedure begins with a “forward selection”, where we permit each allowed term to be added to $g(x)$ in its lowest available form. For example, if the linear term for $x_1$ is not yet in $g(x), x_1$ is available for selection but $x_1^{2}$ is not. If $x_1$ is already in $g(x)$ then all first order interactions with the other linear parameters in $g(x)$ are included and then $x_1^{2}$ is available for selection. So, suppose $g(x)$ is $(1, x_2)$, then the selection of $x_1$ implies that $g(x)$ will become $(1,\,x_1,\,x_2,\,x_1*x_2)$. If $x_1$ is selected, at the next iteration we may select any of the other parameters but we may also include quadratic terms $x_2^{2}$ and $x_1^{2}$. We add the interactions in this way, and do similar for third order interactions when quadratic terms have been included, so that the resulting emulator will be robust to changes of scale (see Draper and Smith 1998, for discussion). The term that is added to $g(x)$ at each iteration is the term of those available that reduces the residual sum of squares the most after fitting by ordinary least squares.

When it becomes clear that adding more terms is not improving the predictive power of the emulator (a judgement made by the analyst based on looking at the proportion of variability explained by the emulator and at plots of the residuals from the fit) we begin a backwards elimination algorithm. This removes terms from $g(x)$, strictly one at a time, with the least contribution to the sum of squares explained by the fit without compromising the quality of the fit. Lower order terms are not permitted to be removed from $g(x)$ whilst higher order terms remain. We stop when removing the next term chosen by the algorithm leads to a poorer statistical model. For more details on stepwise methods such as these see Draper and Smith (1998).

We allow $\epsilon (x)$ in Eq. (2) to be mean zero error with variance specified by the residual variability from the fits and no correlation between $\epsilon (x)$ and $\epsilon (x^{\prime })$ for $x\ne x^{\prime }$. Though this lack of correlation might not be appropriate if we had smaller ensembles or, perhaps, if we had completed a number of waves of history matching and were focussing on a densely sampled subset of parameter space, it is computationally efficient and a reasonable enough approximation to the data here to be adopted for pragmatism. Including a more complex correlation would reduce our emulator uncertainty and likely lead to more parameter space being ruled out, though at a computational cost. Note that, with zero correlation between any points, the emulator will not interpolate the ensemble members and will have non-zero variance at each of them. Though the model is deterministic (in the sense that running it twice for the same values of the model inputs returns the same answer), it also displays sensitive dependence to initial conditions, hence the fitted variance at the design points represents the model’s internal variability. This form of emulator effectively assumes that internal variability is constant throughout parameter space, and hence can be estimated from the ensemble as part of the fitting procedure.

Following the fitting of each emulator we validate its quality using $10\,\%$ of the ensemble that was chosen randomly and reserved from the training data prior to the fit. This procedure involves checking that the emulator accurately predicts each of the unseen ensemble members to within the accuracy specified by emulator uncertainty. If the emulators pass this diagnostic check, we then use them in our history matching.

We give details of our emulator for the ACC strength in HadCM3 to illustrate the complexity of the mean function and the performance of the predictions. The terms selected in $g(x)$ are displayed in Table 1. Each header corresponds to the label given to each of the parameters in Table 2. Numbers on the diagonal of the table refer to the order of the parameter included in the emulator. For example, the number 2 implies that both quadratic and linear terms in that parameter were included in $g(x)$. Numbers on the upper triangle refer to the inclusion (1) or not (0) of interactions between the two relevant parameters in $g(x)$. So, reading from the first row of the table, the term (entcoef $*$ ct) is included in $g(x)$, but the term (entcoef $*$ rhcrit) is not. Variables in bold on the lower triangle indicate the inclusion of the given third order interaction. For example, the table indicates that the term (AH1_$\hbox {SI}^{2}*$CWland) is in $g(x)$. In addition to the terms in the table, the factor r_layers and a linear term in parameter charnock are included, as is 1 so that an intercept is fitted.

Table 1 A table indicating which terms are in $g(x)$ for our emulator of ACC in Eq. (2)

Figure 13 shows a validation plot for the ACC emulator. For 65 PPE members, chosen randomly, that were reserved from the emulator at the fitting stage, the data are sorted by ACC strength and plotted in red. We overlay the emulator predictions (black points) and the uncertainty on those predictions (error bars). The uncertainty represents approximately 2 standard deviations for each prediction. We can see that the predictions are generally good with most unseen PPE members laying within the uncertainty on the prediction. In fact, our uncertainty specification may be too conservative, in that we have allowed for more uncertainty in the predictions than is required. If this is the case, that would lead to less space ruled out by history matching, not more, and it is our preference to remain conservative when ruling out regions of parameter space.

Fig. 13

Predicted ACC strength (black points) with error bars showing approximately 2 standard deviations of the emulator uncertainty for each of the withheld PPE members (red points)

Appendix 2: Tables

Tables 2, 3 and 4 give descriptions and ranges for the parameters and the settings of switches used in our ensemble. Some parameters have relationships with other model parameters that were given to us by the Met Office so that a change in one leads to a derivable value for the other. CWland also determines CWsea, the cloud droplet to rain threshold over sea ($\hbox {kg/m}^3$), MinSIA also determines dtice (the ocean ice diffusion coefficient) and k_gwd also determines kay_lee_gwave (the trapped lee wave constant for surface gravity waves $\hbox {m}^{3/2}$).

Table 2 Parameter descriptions and model section

Table 3 Ranges for each of the continuous parameters varied in our ensemble

Table 4 Switch parameters and their settings in our ensemble

Appendix 3: Another NROY ACC model

In the main text we present the behaviour of one of the NROY ACC models, arguing that correcting the ACC strength seems to improve the ocean circulation. Though we do not reproduce all of the figures from the main text, in order to save space, we show the BSF of another of these models in Fig. 14 to indicate that the chosen model was not a “one-off”. This model has a slightly more physical looking sub polar gyre, at the expense of a more diffuse gulf stream. The cold bias in the North Atlantic (not shown) was also greater in this model.

Fig. 14

The barotropic streamfunction (BSF, Sv) for a different ensemble member with realistic ACC strength (another of the blue dots from Fig. 1)

Appendix 4: Anomalies from two additional precipitation climatologies

In the main text we present precipitation anomalies from the ERA40 climatology. However, we caution the reader against interpreting improvements seen in the improved ACC run as robust. We present anomalies from two alternative precipitation climatologies, the CPC Merged Analysis of Precipitation (CMAP) (Xie and Arkin 1997) and NCEP/NCAR reanalysis (Kalnay et al. 1996) in Fig. 15. These plots indicate that the standard model does have a tendency towards higher than observed precipitation along the ITCZ and over the maritime continent, and the improved ACC run we present tends to exhibit lower than observed precipitation over the western equatorial Pacific. The improved ACC run does perform better than the standard run everywhere outside the tropics, where the standard run has a tendency towards higher than observed precipitation.

Fig. 15

The top two panels show differences from the CPC Merged Analysis of Precipitation (CMAP) climatology (standard above improved ACC model). The bottom two panels show differences from the NCEP reanalysis precipitation climatology (standard above improved ACC model)