About Data Transfer Services

NeSI supports two methods for transferring data onto and off our platforms.

scp

For the transfer of modest amounts of data (convenient, but slow). This service is only available to persons who are authorised to access Mahuika or Māui ssh services.

Globus Online

For transferring large amounts of data transfer between Data Transfer Nodes (DTN). With Globus Online, very high data transfer rates are achievable. This service allows data to be accessible to any person who has a Globus Online account.

Note that Globus Online is not currently available!

scp

scp can be used to transfer small amounts of data to/from the HPCF. It is readily available on a terminal on Linux and MacOS, or via WinSCP and MobaXterm on Windows. Note that two-factor authentication will be required for file transfer sessions.

The lander node filesystem is distinct from the HPCF filesystems, and so copying a file from your desktop to the HPCF is by default a two-step process. If you are using an OpenSSH based ssh, which will be the case for most MacOS and Linux users, then this inconvenience can be avoided by setting up an SSH config file such as:

Host mahuika
    User your_username
    Hostname login.mahuika.nesi.org.nz
    ProxyCommand ssh -W %h:%p lander02.nesi.org.nz

You can add a similar entry for Maui by just replacing "mahuika" with "maui"

With that file present at ~/.ssh/config, the command

ssh mahuika

should take you there directly, jumping across the lander node on the way, and

scp <filename> mahuika:~/

will do the same for your file transfers.

Passwordless scp via ssh multiplexing

The following method allows passwordless scp while you have an ssh session open:

ControlPath ~/.ssh/sockets/%r@%h:%p

Host mahuika User your_username Hostname login.<host>.nesi.co.nz ProxyCommand ssh -W %h:%p lander02.nesi.org.nz ControlMaster auto ControlPersist 1

For full details on this see https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing.

 

Data Transfer Node Service [NOT YET Available]

A Data Transfer Node (DTN) provides a high-performance method to transfer data to and from the HPCF storage. This is achieved using Globus Online, a web-based service that solves many of the challenges encountered moving large volumes of data between systems. While NeSI supports use of other data transfer tools and protocols such as scp, Globus Online provides the most comprehensive, efficient, and easy to use service for most NeSI users.

To use Globus from within NeSI, you need a Globus ID. If you don't have one, sign up at https://www.globusid.org, following the instructions at Register for a Globus ID).

Table 13: NeSI Data Transfer Node Endpoints.

Endpoint Name

Description

Recommended Use

nesi#hpcf-dtn?

Endpoint for NeSI Storage on the HPCF at Greta Point

 

nesi#uoa-dtn?

 

 

nesi#otago-dtn01

Endpoint 01 for the High Capacity Research Storage Cluster, U of Otago

 

nesi#otago-dtn02

Endpoint 02 for the High Capacity Research Storage Cluster, U of Otago

 

nesi#otago-dtn03

Endpoint 03 for the High Capacity Research Storage Cluster, U of Otago

 

nesi#otago-dtn04

Otago AWS Endpoint

 

nesi#otago-dtn-chc01

Endpoint 01 for the High Capacity Research Storage Cluster, Christchurch Campus

 

nesi#otago-dtn-wlg01

Endpoint 04 for the High Capacity Research Storage Cluster, Wellington Campus

 

nesi#mwlr-dtn?

Endpoint for Manaaki Whenua Landcare Research

 

Others?

 

 

Note: To transfer files into/out of your laptop or desktop computer, you can install a Globus connect (personal) server to configure an endpoint on your personal device.

Use of NeSI DTNs

Sign in to https://www.globus.org with your id. You will be taken to the transfer page https://www.globus.org/app/transfer

Select the Endpoints you wish to move files between (start typing "nesi#" to see the list of NeSI DTNs to select from).

Globus defaults to your home directory. Navigate your path to an appropriate directory (see Globus Paths, Permissions, Storage Allocation).

Transfer the files.

Check your email for confirmation the job has succeeded.

Personal computer to NeSI

To transfer files into/out of your laptop or desktop computer, you can install a Globus connect (personal) server to configure an endpoint on your personal device.

 

 

Labels: globus dtn
Was this article helpful?
0 out of 0 found this helpful