Follow

Pan Bulletin: 12 December 2017

Contact

If you have any questions about the recent outage or the content of this bulletin, please contact support@nesi.org.nz.

Scheduled outage

The next planned outage of Pan is scheduled for Thursday 15th February 2018.

Per-Job Limits

We now have default constraints on job size (512 CPUs) and duration (3 weeks). Exceptions can be made by request, on a per-project basis, but 3 weeks is already half of the usual time between planned Pan outages, so half of the time a 3 week job can't start because it won't finish before the outage begins.

There is now a page documenting this and other aspects of how jobs are prioritised by Slurm on Pan.

Updated Applications

For once, there have been no new applications added since the previous bulletin. There are however new versions of Python, R, BioConductor, SAMtools, Cufflinks, and MrBayes.

Our latest Python environment modules, with Python 2.7.14 and 3.6.3, bring some changes to the way we organise some Python packages. We now have two new environment modules which extend them:

Python-GPU

Contains pycuda, pygpu, and scikit-cuda. So these are no longer in the base Python environment module.

Python-Geo

Contains GDAL, pyModis, RIOS, Fiona, Shapely, descartes and pygrib, which are Python packages for geospatial data I/O, mostly based on GEOS and the OSGEO libraries GDAL and OGR. This is intended to replace the existing Python-GDAL environment modules.

Platform Migration

In a few months we will be moving your data and our scientific software applications from Pan on to the new NeSI platform. You can help us with this by:

  • Moving to the latest versions of our environment modules so that we don't have to recreate the older ones. We log when modules get loaded, and only those which are still in use will be preserved.
  • Aiming to reduce the amount of data on disk toward the end of the year, and putting any easily replaced data (e.g., that which is publicly available) under a directory named "nobackup" in your project directory.

Installing Software

Our team is currently working on several aspects of the new system which will replace Pan, and as a consequence have less time work on installing software. You are welcome to install software into your home or project directory yourself if you have the skills to be comfortable doing this. Here are some tips:

Python packages from the Python Package Index

$ ssh build-wm
$ module load Python/3.6.3-gimkl-2017a
$ pip --user install packagename

Do this on build-wm if you want to be sure that any resulting extension modules will work on all compute nodes of Pan without causing "Invalid Instruction" errors.

R packages from CRAN

$ ssh build-wm
$ module load R/3.4.2-gimkl-2017a
$ R
> package.install("packagename")

Do this on build-wm if you want to be sure that any resulting extension modules will work on all compute nodes of Pan without causing "Invalid Instruction" errors.

Fetching other software

You can copy software onto the cluster just like other files, via your machine and scp, or directly from a public URL with the wget command.

For Java, linux executables, and interpreted languages that might be all that is required.

Compiling software from source code

For simple tools the system compiler may be sufficient, but for scientific software it is generally best to compile using our most recent "toolchain", so do module load gimkl/2017a before compiling the code, and again when running it. This will load a recent version of the GCC compilers, together with Intel MPI and the Intel MKL libraries which you may or may not need. Of course you are welcome to use additional environment modules to provide libraries, don't attempt to combine different toolchains - stick to versions with a gimkl-2017 toolchain suffix or no such toolchain. Compile on build-wm if you want the resulting executable to work on all compute nodes of Pan.

Introductory Workshops

If you are interested in participating in a Hands-on Introduction to NeSI workshop, please contact us via training@nesi.org.nz.

Consultancy service

Do you think you could be using NeSI more efficiently? Our technical experts can work with you to help optimise your code and workflow. For more information see our website or contact us at support@nesi.org.nz.

Comments

Powered by Zendesk