Description

Python is a programming language that lets you work more quickly and integrate your systems more effectively.

The Python home page is at http://python.org.

On the NeSI systems Python is available in various versions with already installed packages. On Mahuika you will find Python/2... and Python/3... modules, on Maui cray-python and on Maui_ancil Anaconda2 and Anaconda3 is provided.

Licensing requirements

All versions of Python available on NeSI platforms are owned and licensed by the Python Software Foundation. Each version is released under a specific open-source licence. The licences are available on the Python documentation server.

Example scripts

#!/bin/bash -e

#SBATCH --job-name    MyPythonJob
#SBATCH --time        01:00:00
#SBATCH --mem-per-cpu 3G
#SBATCH --output      MyPythonJob.%j.out # Include the job ID in the names of
#SBATCH --error       MyPythonJob.%j.err # the output and error files

module load Python/3.6.3-gimkl-2017a

python MyPythonScript.py

Python Packages

Programmers around the world have written and released many packages for Python, which are not included with the core Python distribution and must be installed separately. Each Python environment module comes with its own particular suite of packages, and the system Python has its own installed packages.

The provided packages can be listed using

$ module load Python/3.6.3-gimkl-2017a   # another Python module (see above)
$ python -c "help('modules')"

Adding packages

In addition to relying on the packages installed with Python by our staff, we suggest to install additional packages yourself into a project directory as follow. Thus you can easily share the installation with collaborators and prevent filling your $HOME quota. In the following we install a python3 package called  prodXY into /nesi/project/<projectID>/PyPackages and create a module for it.

Using Easybuild

Easybuild is a Package provisioning tool, please read also here. Therefore you need a configuration file, e.g. for PyPI python packages (here cutadapt), which could be located in /nesi/project/<projectID>/easybuildinstall/easyconfigs and the following would be named as cutadapt-2.3-gimkl-2018b-Python-3.7.3.eb:

# Easybuild Python package template
# Author: Mandes Schoenherr
# NeSI - the New Zealand eScience Infrastructure

easyblock = "PythonPackage"

name = 'cutadapt' ### specify the name here
version = '2.3' ### specify the version

homepage = 'https://cutadapt.readthedocs.io/' ### add a reference URL and a description below
description = """ cutadapt removes adapter sequences
 from high-throughput sequencing data. """

toolchain = {'name':'gimkl', 'version':'2018b'}
source_urls = [PYPI_SOURCE]
sources = [SOURCELOWER_TAR_GZ]

python = 'Python'
pyver = "3.7.3"
pyshortver = '.'.join(pyver.split('.')[:2])
versionsuffix = "-%s-%s" % (python, pyver)
dependencies = [ (python, pyver) ]
sanity_check_paths = {
 'files': ['bin/cutadapt'],
 'dirs': ['lib/python%s/site-packages/' % pyshortver],
}

moduleclass = 'bio' ### change the type of software

Then we can install this package using:

export NESI_EASYBUILD_PROJECT_ID=<projectID>
module load project
eb cutadpt-2.3-gimkl-2018b-Python-3.7.3.eb

After successful install we should see the module using:

module --ignore-cache avail cutadapt

And the package can be used by all project members (after specifying the projectID) using:

module load project
module load cutadapt 

 

Manual install and provisioning

Depending on the package you can install packages using:

• the Python Package Manager PIP

$ pip install --prefix /nesi/project/<projectID>/PyPackages cogent

• use source code

$ python setup.py install --prefix /nesi/project/<projectID>/PyPackages

 

Python will not find the package by default, therefore you need to add the location of the package to the $PYTHONPATH. Furthermore, some packages provide binaries, which are accessible by adding the directory to $PATH

$ export PYTHONPATH=/nesi/project/<projectID>/PyPackages/lib/python3.6/site-packages:$PYTHONPATH
$ export PATH=/nesi/project/<projectID>/PyPackages/bin:$PATH

NOTE: If you install multiple packages in the same location, e.g. /nesi/project/<projectID>/PyPackages, you just need to set these environment variable once.

Creating a modulefile

These environment variables can be handled in a module file. Thus the packages (and others if desired) can be accessed by you and the project collaborators.

Therefore we create a modulefile let's say at /nesi/project/<projectID>/modulefiles/PyXtra and assume we installed a Python package into /nesi/project/<projectID>/PyPackages:

#%module
module load Python/3.7.3-gimkl-2018b
   # provide a description
whatis "The packageXY for python."
proc ModulesHelp { } {  puts stderr "This module loads the packageXY. It requires python3." }

set PKG_PREFIX /nesi/project/<projectID>/PyPackages
   # add the location of binaries to PATH, such they are immediately accessible
prepend-path PATH $PKG_PREFIX/bin
   # add to PYTHONPATH to access python packages
prepend-path PYTHONPATH $PKG_PREFIX/lib/python3.7/site-packages/

To use this module (and all other modules you create in that directory) you add the following to your $HOME/.bashrc (need to source it, or re-login to activate the changes):

module use /nesi/project/<projectID>/modulefiles

Then you can simple load the module via:

module load PyXtra

 

Further notes

 iPython

iPython (interactive Python) is an enhanced tool for accessing a Python command line. It is available in many NeSI Python modules.

Starting iPython

To open an iPython console, simply run the ipython command:

[jblo123@build-wm ~]$ module load Python/3.6.3-gimkl-2017a
[jblo123@build-wm ~]$ ipython

Listing available functions

You can use iPython to list the functions available that start with a given string. Please note that if the string denotes a module (i.e., it has a full stop somewhere in it), that module (or the function you want from it) must first be imported, using either an "import X" statement or a "from X import Y" statement.

import os
os.<TAB>   # List all functions in the os module
os.O_<TAB> # List functions starting with "O_" from the os module
len<TAB>   # List functions starting with "len"

Here,

o<TAB>

or even

os<TAB>

and expect to see the methods and values provided by the os module - you have to put the full stop after the "os" if you want to do that.

Getting information about an object

In iPython, you can query any object by typing the object name followed by a question mark (?), then hitting Enter. For instance:

In [1]: x = 5
In [2]: x?
Type:        int
String form: 5
Docstring:
int(x=0) -> int or long
int(x, base=10) -> int or long

Convert a number or string to an integer, or return 0 if no arguments
are given.  If x is floating point, the conversion truncates towards zero.
If x is outside the integer range, the function returns a long instead.

If x is not a number or if base is given, then x must be a string or
Unicode object representing an integer literal in the given base.  The
literal can be preceded by '+' or '-' and be surrounded by whitespace.
The base defaults to 10.  Valid bases are 0 and 2-36.  Base 0 means to
interpret the base from the string as an integer literal.
>>> int('0b100', base=0)
4

You can also do this on functions (len?), methods (os.mkdir?) and modules (os.path?). If you try to do it on something that isn't defined yet, Python will tell you that the object in question couldn't be found.

Quitting iPython

Just enter the quit command at the iPython prompt.