What is synda?
Synda is a command line tool to search and download files from the Earth System Grid Federation (ESGF) archive. Synda is a useful tool if you need to download Climate Model Intercomparison Project Phase 6 (CMIP6) data in particular. Synda supports different download protocols (e.g. HTTPS) and can download files in parallel.
We'll describe the steps to install and use synda on Mahuika or Maui_ancil.
How to install synda
1. Load Anaconda3
module load Anaconda3
2. Create a conda environment and activate the environment
conda create --prefix=<SYNDA_ENV>
where <SYNDA_ENV> is a directory of your choice. Activate your new environment
conda activate <SYNDA_ENV>
3. Install synda (currently version 3.10)
conda install -c IPSL synda
4. Configure synda
Set the ST_HOME environment variable and populate
$ST_HOME, for instance
export ST_HOME=/nesi/nobackup/<YOUR PROJECT>/synda_home
Note: you may want to add the above "export ST_HOME=<...>" somewhere near the bottom of your ~/.bashrc file.
To search and download climate model data you will in addition need to create an account on one or more of the ESGF nodes, e.g. https://esgf-node.llnl.gov/projects/esgf-llnl/. This will require you to provide a user name (USER_NAME) and a password - you will receive an openID in return. Copy your openID string as you will need later.
To configure and store your ESGF credentials, type
Paste in the openID you received when creating your ESGF account. If you created an account on esgf-node.llnl.gov then your openID will be:
(USER_NAME is the user name you chose when creating the account). You can change your default configuration by editing $ST_HOME/conf/sdt.conf. I have
indexes = esgf-node.llnl.gov
default_index = esgf-node.llnl.gov
Find all the datasets for given ocean surface temperature ("tos"), variant label ("r1i1p1f1") produced by institution "NOAA-GFDL" for historical data and table Id "Omon":
synda search project=CMIP6 realm=ocean variable=tos variant_label=r1i1p1f1 institution_id=NOAA-GFDL table_id=Omon experiment_id=historical
This will return
Choose one of the datasets. To find out how big the dataset is, type:
Total files count: 9
New files count: 9
Total size: 262.8 MB
New files size: 262.8 MB
To download the dataset, type
synda get CMIP6.CMIP.NOAA-GFDL.GFDL-CM4.historical.r1i1p1f1.Omon.tos.gn.v2018070
The datasets (NetCDF files) will be downloaded locally in your directory.