Comparing Individual CMIP5 Models To Observations

... because it just never gets old.

Background

One of the Denizens at Judy's writes in response to my question of Mike Flynn:

---------------------

PA | June 24, 2016 at 11:05 am | Reply

Again: Where is your CMIP5-compatible model which beats the present state of the art at its own game?

The Russian INMCM4 is arguably the most accurate or one of the most accurate in terms of predictive trend

The INMCM4 model predicts little future warming.

Question asked and answered.

---------------------

To which I reply:

---------------------

brandonrgates | June 25, 2016 at 9:42 pm |

PA,

The Russian INMCM4 is arguably the most accurate or one of the most accurate in terms of predictive trend. The INMCM4 model predicts little future warming.

Comparing to HADCRUT4 annual means over 1880-2005:

Model                 RMSE   GMST 2100  2.0 C Year
-------------  ----------- ----------- -----------
CESM1-CAM5          0.0093        5.10        2043
CNRM-CM5            0.0104        4.52        2047
inmcm4              0.0121        3.60        2057
GISS-E2-H           0.0123        3.95        2043
MIROC-ESM-CHEM      0.0126        6.36        2030


Model            Slope Dev   GMST 2100  2.0 C Year
-------------  ----------- ----------- -----------
MIROC-ESM-CHEM      0.0036        6.36        2030
inmcm4              0.0070        3.60        2057
FGOALS-g2           0.0222        4.15        2046
CNRM-CM5            0.0299        4.52        2047
GFDL-ESM2M          0.0463        3.54        2051

“Slope Dev” is the absolute deviation of the model vs. observed change in C/century. Projections are from RCP8.0 for all models using a “pre-industrial” temperature baseline of 1861-1890. There is practically zero correlation between implied future sensitivity and either hindcast trend or hindcast skill as judged by root mean squared error.

Question asked and answered.

Not really. According to Mike, ECS shouldn’t just be low — it should be zero.

---------------------

The response misses the point and includes the often seen shonky plots from Spencer and Christy:

---------------------

PA | June 26, 2016 at 1:16 am |

brandonrgates | June 25, 2016 at 9:42 pm |
inmcm4 1.68 3.60 2057

Really?

Most plots show INMCM4 at the bottom of the pack. I notice you don’t link to support your clams.

---------------------

Much has been written elsewhere about the issues with these plots.  The main problem is that the combination of 5-year running means converging to zero on 1979 exaggerates the divergence.  Plus, nobody's really sure which balloon datasets were used or how they were averaged -- balloon datasets typically contain temperature trends for multiple pressure levels, and since various levels of the atmosphere are changing temperature at different rates, it's essential to report which levels were used and how they were weighted in putting together the observational mean.

There's a Better Way

It's much simpler and less fraught to simply compare teh modulz to global surface temperature.  I'll use HadCRUT4 for this post, at annual resolution.  GISTemp only really makes a difference prior to 1950, and even then not much.

First, let's look at the rankings like I provided for PA for all the models in RCP8.5:

Model	RMSE	Slope Dev	GMST 2100	2.0 C Year
GFDL-ESM2M	0.0162	0.0463	3.54	2051
inmcm4	0.0121	0.0070	3.60	2057
GFDL-ESM2G	0.0187	0.0995	3.64	2056
GISS-E2-R	0.0133	0.0492	3.69	2048
GISS-E2-H	0.0123	0.1233	3.95	2043
NorESM1-M	0.0127	0.1042	4.13	2049
FGOALS-g2	0.0133	0.0222	4.15	2046
MIROC5	0.0146	0.1013	4.22	2043
NorESM1-ME	0.0135	0.0807	4.39	2044
bcc-csm1-1	0.0275	0.2211	4.44	2046
MRI-CGCM3	0.0212	0.1594	4.46	2048
CNRM-CM5	0.0104	0.0299	4.52	2047
CCSM4	0.0261	0.3219	4.56	2043
IPSL-CM5B-LR	0.0180	0.2216	4.56	2044
CESM1-BGC	0.0252	0.2807	4.57	2039
EC-EARTH	0.0178	0.2365	4.59	2044
FIO-ESM	0.0178	0.1302	4.59	2050
MPI-ESM-MR	0.0212	0.2356	4.62	2044
MPI-ESM-LR	0.0203	0.2472	4.74	2041
ACCESS1-3	0.0191	0.2176	5.10	2040
CESM1-CAM5	0.0093	0.0509	5.10	2043
ACCESS1-0	0.0152	0.0609	5.19	2040
HadGEM2-AO	0.0144	0.0927	5.23	2043
CSIRO-Mk3-6-0	0.0180	0.1736	5.23	2041
CMCC-CM	0.0142	0.1252	5.46	2038
IPSL-CM5A-LR	0.0272	0.3228	5.54	2036
CMCC-CMS	0.0200	0.2725	5.59	2036
BNU-ESM	0.0345	0.4550	5.60	2031
IPSL-CM5A-MR	0.0169	0.1566	5.62	2036
CanESM2	0.0173	0.0699	5.71	2031
HadGEM2-ES	0.0183	0.2228	5.75	2034
GFDL-CM3	0.0203	0.2392	5.84	2029
HadGEM2-CC	0.0201	0.3349	5.85	2031
MIROC-ESM	0.0131	0.0901	6.05	2033
MIROC-ESM-CHEM	0.0126	0.0036	6.36	2030

The spread by 2100 is about 2.8 degrees C, and INMCM4 is indeed near the bottom of the pack.  Here's a plot:

Now for RCP4.5, first the rankings (9999 means the model never hits the 2.0 C threshold):

Model	RMSE	Slope Dev	GMST 2100	2.0 C Year
GFDL-ESM2G	0.0187	0.0995	1.57	9999
GFDL-ESM2M	0.0162	0.0463	1.63	2073
FIO-ESM	0.0178	0.1302	1.69	9999
inmcm4	0.0121	0.0070	1.81	9999
GISS-E2-R-CC	0.0120	0.0166	1.88	9999
GISS-E2-H-CC	0.0216	0.3068	1.94	2084
FGOALS-g2	0.0133	0.0222	1.95	2102
GISS-E2-R	0.0109	0.0175	2.09	2086
bcc-csm1-1-m	0.0230	0.1697	2.10	2066
bcc-csm1-1	0.0275	0.2211	2.14	2066
CESM1-BGC	0.0252	0.2807	2.26	2051
NorESM1-M	0.0127	0.1042	2.26	2066
MPI-ESM-MR	0.0193	0.2239	2.27	2056
GISS-E2-H	0.0119	0.1349	2.29	2060
IPSL-CM5B-LR	0.0180	0.2216	2.32	2060
CCSM4	0.0261	0.3219	2.34	2060
MIROC5	0.0146	0.1013	2.37	2054
MPI-ESM-LR	0.0203	0.2472	2.47	2052
NorESM1-ME	0.0135	0.0807	2.51	2051
EC-EARTH	0.0178	0.2365	2.54	2057
CNRM-CM5	0.0154	0.0938	2.55	2055
MRI-CGCM3	0.0212	0.1594	2.59	2072
BNU-ESM	0.0345	0.4550	2.67	2034
CMCC-CM	0.0142	0.1252	2.80	2049
CMCC-CMS	0.0200	0.2725	2.83	2043
IPSL-CM5A-LR	0.0272	0.3228	2.84	2046
ACCESS1-0	0.0152	0.0609	2.87	2049
IPSL-CM5A-MR	0.0169	0.1566	2.89	2043
ACCESS1-3	0.0191	0.2176	2.90	2047
CSIRO-Mk3-6-0	0.0180	0.1736	2.99	2046
HadGEM2-CC	0.0201	0.3349	3.01	2039
HadGEM2-ES	0.0183	0.2227	3.01	2040
CanESM2	0.0173	0.0699	3.02	2036
HadGEM2-AO	0.0144	0.0927	3.08	2043
CESM1-CAM5	0.0093	0.0509	3.13	2046
MIROC-ESM	0.0131	0.0901	3.14	2034
MIROC-ESM-CHEM	0.0126	0.0036	3.29	2038
GFDL-CM3	0.0203	0.2392	3.51	2027

And the plot:

Unsurprisingly INMCM4 is again amongst the coolest of the bunch.

To be continued ... I want to post this so as to be able to use the images in my response to PA.

Other Interesting Stuff

The following two plots show that there's little correlation between trend over the hindcast portion and trends over the centennial projection:

It is interesting that there are three distinct groups in the forward-looking projections.  I've no idea what this means, but I can suppose it has something to do with the fact that various models use common codes and methods between -- i.e., they're not all entirely built from scratch.