Follow

NCO

Description

The NCO toolkit manipulates and analyzes data stored in netCDF-accessible formats, including DAP, HDF4, and HDF5. It exploits the geophysical expressivity of many CF (Climate & Forecast) metadata conventions, the flexible description of physical dimensions translated by UDUnits, the network transparency of OPeNDAP, the storage features (e.g., compression, chunking, groups) of HDF (the Hierarchical Data Format), and many powerful mathematical and statistical algorithms of GSL (the GNU Scientific Library).

The NCO home page is at http://nco.sourceforge.net.

Available modules

Packages with modules

Module NeSI Cluster
nco/4.4.8 fitzroy
NCO/4.4.4-ictce-5.4.0 pan

Licensing requirements

NCO is free and open-source. It is made available under the terms of version 3 of the GNU General Public Licence.

Example scripts

Example script for the Pan cluster

#!/bin/bash -e

#SBATCH --job-name    MyNCOJob
#SBATCH --account     nesi99999
#SBATCH --time        01:00:00
#SBATCH --mem-per-cpu 4G
#SBATCH --output      MyNCOJob.%j.out # Include the job ID in the names of the
#SBATCH --error       MyNCOJob.%j.err # output and error files

# LoadLeveler has an annoying habit of transferring parts of the user's
# environment as it existed at the time of submission to the job. Clear any
# loaded modules.
module purge

module load NCO/4.4.4-ictce-5.4.0

# Compute vector magnitudes "mag" from variables "x", "y", "z" in "input.nc""
# and store result in a new NetCDF file "output.nc"
ncap2 -v -s 'mag=sqrt(x^2 + y^2 + z^2)' input.nc output.nc

Example script for the Fitzroy cluster

#!/bin/bash -e

#@ job_name         = MyNCOJob
#@ account_no       = nesi99999
#@ class            = General
#@ wall_clock_limit = 01:00:00
#@ initialdir       = /hpcf/working/nesi99999/MyNCOJob
#@ output           = $(job_name).$(jobid).out
#@ error            = $(job_name).$(jobid).err
#@ queue

# LoadLeveler has an annoying habit of transferring parts of the user's
# environment as it existed at the time of submission to the job. Clear any
# loaded modules.
module purge

module load nco/4.4.8

# Compute vector magnitudes "mag" from variables "x", "y", "z" in "input.nc""
# and store result in a new NetCDF file "output.nc"
ncap2 -v -s 'mag=sqrt(x^2 + y^2 + z^2)' input.nc output.nc

Further notes

Please be mindful of memory requirements of the NCO tools ("operators") when handling very large data files: "record"-type operators load only one record at a time into memory, whereas "ensemble"-type operators load values for all the records at once, requiring much more memory. Click here for more information.

Comments

Powered by Zendesk