The Coupled Model Intercomparison Project, which began in 1995 under the auspices of the World Climate Research Programme (WCRP), is now in its sixth phase (CMIP6). CMIP6 orchestrates somewhat independent model intercomparison activities and their experiments, which have adopted a common infrastructure for collecting, organising, and distributing output from models performing common sets of experiments.
This document shows how to explore which CMIP6 data are available and how to download the data once you have figured out what you need. The data will be downloaded asynchronously - no need to stare at a screen for hours. The downloaded data will reside in a shared directory and hence will also be accessible to your collaborators.
The instructions are geared towards members of the niwa02916 group - send a message to firstname.lastname@example.org if you are a NIWA employee and want to become part of this group. Other NeSI users may want to read this, which explains how to install the synda tool. Once installed, you can then type similar commands to the ones below to test your configuration.
On mahuika or w-mauivlab01.maui.nesi.org.nz:
This will load the Anaconda3 environment and set the ST_HOME variable. You should also now be able to invoke synda commands, a tool that can be used to synchronise CMIP data with Earth System Grid Federation archives. A full list of options can be obtained with
Below we demonstrate how synda might be used.
Find some datasets
CMIP6 datasets are organised by institution_id, experiment_id, variable etc. A full list can be glanced from https://esgf-node.llnl.gov/search/cmip6/. A possible search might involve
synda search institution_id=NCAR experiment_id=1pctCO2 variable=ta
as well as some other datasets.
Find out how big the datasets are
Once you know what you want to download, it's a good idea to check the size of the dataset:
synda stat CMIP6.CMIP.NCAR.CESM2-WACCM.1pctCO2.r1i1p1f1.day.ta.gn.v20190425
This prints "Total files count: 16, New files count: 16, Total size: 48.7 GB, New files size: 48.7 GB". The "New" indicates that the files have not yet been downloaded. You can see that there are 16 files to download, taking nearly 50GB of disk space.
Download/install the dataset
synda install CMIP6.CMIP.NCAR.CESM2-WACCM.1pctCO2.r1i1p1f1.day.ta.gn.v20190425
The first time you may get a message requesting you to start a daemon, if so do
synda daemon start
This will put your request in a queue. The transfer will take place as a background process so you can close your terminal if you want and come back later to check progress.
The data will end up under $ST_HOME/data/CMIP6/CMIP/NCAR/CESM2-WACCM/1pctCO2/r1i1p1f1/day/ta/gn/v20190425 in this case.
You can type
to see progress.
Note that if another researcher (or you) decide execute the same synda install command then synda will recognise the files to be already installed and will print the message
INFO: Nothing to install (matching files are already installed or waiting in the download queue)