What is Cylc
Cylc is a general purpose workflow engine that can also orchestrate cycling systems very efficiently. It is used in production weather, climate, and environmental forecasting on HPC, but is not specialised to those domains.
Using a workflow engine may enable you to run large parametric or sensitivity studies while ensuring scalability, reproducibility and flexibility. If you have embarrassingly parallel jobs then Cylc might be a good solution for you. The workflow engine will allow for the concurrent execution of parallel jobs, depending on the task graph and available resources on the platform. One advantage of this approach over running a monolithic, parallel executable is that each task will require less resources that the complete problem, it is thus easier for each task to slip into the queue and start running.
See the NeSI Snakemake page for another, possible choice.
In this article, we show how you can create a simple workflow and run it on NeSI's platform. Consult the Cylc documentation for more elaborate examples, including some with a cycling (repeated) graph pattern. One of the strengths of Cylc is that simple workflows can be executed simply while allowing for very complex workflows, with thousands of tasks, which may be repeated ad infinitum.
Cylc uses ssh to automatically start schedulers on configured "run hosts" and to submit jobs to remote platforms, so if you haven't already done so you need to allow ssh to connect to other hosts on the HPC cluster without prompting for a passphrase (all HPC nodes see the same filesystem, so this is easy):
ssh-keygento generate a public/private key pair with no passphrase (when it asks for a passphrase, just hit enter)
- add your own public key to your authorized keys file:
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
- check that your keys, authorized_keys file, ssh directory, and home directory all have sufficiently secure file permissions. If not,
sshwill silently revert to requiring password entry. See for example https://www.frankindev.com/2020/11/26/permissions-for-.ssh-folder-and-key-files/
- make sure your home directory has a maximum of 750 permissions
Now you should be able to run
ssh mahuika02(for example) without being asked for a passphrase.
How to select the Cylc version
Cylc has been installed on Māui and Mahuika, there is no need to load any module,
$ which cylc
(Access if via the NeSI module, which is loaded by default.)
Be aware that the default version
$ cylc version
is not the latest, and that configuration file and Cylc commands have changed significantly at version 8.
New Cylc users should use version 8 or later,
$ cylc list-versions
cylc -> 7.9.1
You can select your Cylc version by setting the environment variable CYLC_VERSION, for instance,
$ export CYLC_VERSION=8.0.1
$ cylc version
At the time of writing, the latest version is 8.0.1.
A simple example of a Cylc workflow
To demonstrate Cylc, let's start with a workflow, which we call "simple",
$ mkdir -p ~/cylc-src/simple
$ cd ~/cylc-src/simple
Create/edit the following flow.cylc file containing
[scheduling] # Define the tasks and when they should run
R1 = """ # R1 means run this graph once
taskA & taskB => taskC # Defines the task graph
[runtime] # Define what each task should run
[[root]] # Default settings inherited by all tasks
platform = mahuika-slurm # Run "cylc conf" to see platforms.
[[[directives]]] # Default SLURM options for the tasks below
--account = nesi99999 # CHANGE
script = echo "running task A"
[[[directives]]] # specific SLURM option for this task
--ntasks = 2
script = echo "running task B"
script = echo "running task C"
In the above example, we have three tasks (taskA, taskB and taskC), which run under SLURM (hence platform = mahuika-slurm). Type
cylc config --platform-names
to see a list of platforms. The SLURM settings for taskA are in the [[[directives]]] section.
How to interact with Cylc
Cylc takes command lines. Type
$ cylc help all
to see the available commands. Type
$ cylc help install # or cylc install --help
to find out how to use a specific command (in this case "install").
Installing a workflow
Prior to running a workflow, it must be installed to a run directory. Due to limited disk space in home directories on NeSI, Cylc has been configured to symlink the standard run directories to project directories, if $PROJECT is defined. Hence, you need to set
$ export PROJECT=nesi99999 # CHANGE
Then install the workflow with
cylc install simple
Validating the workflow
It's a good idea to check that there are no syntax errors in flow.cylc,
$ cylc validate simple
Valid for cylc-8.0.1
Looking at the workflow graph
A useful command is
$ cylc graph simple
which will generate a png file, generally in the /tmp directory with a name like /tmp/tmpzq3bjktw.PNG. Take note of the name of the png file. To visualise the file you can type
$ display /tmp/tmpzq3bjktw.PNG # ADJUST the file name
Here, we see that our workflow "simple" has a "taskC", which waits for "taskA" and "taskB" to complete,
The "1" indicates that this workflow graph is executed only once.
Different ways to interact with Cylc
Every Cylc action can be executed via the command line. Alternatively, you can invoke each action through a terminal user interface (tui),
$ cylc tui simple
Another alternative, is to use the graphical user interface
$ cylc gui
Read below on how access the web interface running on NeSI using your local browser.
Connecting via Jupyter
If you're connecting through https://jupyter.nesi.org.nz you'll need to replace anything before the ":" with https://jupyter.nesi.org.nz/user/USERNAME/proxy/ to get access to the web graphical user interface (where USERNAME is your NeSI user name). Hence the URL becomes https://jupyter.nesi.org.nz/user/USERNAME/proxy/8888/cylc?token=TOKEN
Connecting via SSH
First open ssh tunnelling, so that a given port on your local machine (e.g. your laptop) maps to the Cylc UI Server’s port on the HPC. On your local machine, type
$ ssh -N -L PORT:localhost:PORT HOST
where PORT is a valid port number and HOST can be maui or mahuika. See the NeSI page for the range of allowed ports (currently 1024-49151). Choose any number in this range but make sure your port number is fairly unique to avoid clashing with other users. Option -N is optional: it opens the connection without logging you into the shell.
Then ssh to the host (e.g. mahuika)
$ ssh HOST
and add the following to $HOME/.cylc/uiserver/jupyter_config.py on the HOST.
where PORT and HOST match the values you selected when opening the ssh tunnel.
You're now ready to fire up the web graphical interface
$ cylc gui
Just copy the URL that looks like
into your web browser. (Again substitute HOST and PORT with the values chosen above.)
How to execute a workflow
To execute the workflow type
$ cylc play --no-detach simple
The "--no-detach" option makes scheduler run in the foreground so you can see its output in your terminal. Without this option it will "daemonize" so it can keep running even if you log out.
$ cylc scan
will list all running and installed workflows.
Checking the output
$ cylc cat-log simple//1/taskA # note // between workflow and task ID
of the first cycle of taskA. The "1" refers to the task iteration, or cycle point. Our simple workflow only has one iteration (as dictated by the R1 graph above).
How to clean or remove a workflow
$ cylc clean simple
will remove the file structure associated with workflow "simple".
Where jobs, results and log files are stored
Cylc will create a directory under $HOME/cylc-run. On NeSI, the output of the runs will be stored in the project directory, with a symbolic link pointing from the user home directory to the project directory
$ ls -l $HOME/cylc-run/simple/run1
lrwxrwxrwx 1 pletzera pletzera 54 Aug 5 03:19 /home/pletzera/cylc-run/simple/run1 -> /nesi/nobackup/nesi99999/pletzera/cylc-run/simple/run1
More can be found about Cylc here, including what Cylc is and how you can leverage Cylc to submit parallel jobs.