Globus Paths, Permissions, Storage Allocation

Note: after the system upgrade of June-July 2018, some of the globus endpoints or data-transfer nodes (DTN) which are named in this documentation will no longer be in service. Please check back for updated information once the new filesystems are in use.

In Linux, your "current working directory" designates the path from "/" to the directory where you are currently located. Use the pwd command ( "print working directory") to see this:


username@login-01:~$ pwd
/home/username

Globus session - default location

When you ssh in to a NeSI cluster, your terminal (shell) session opens in your home directory, designated by the path /home/<your_username> aliased to '~'.

Similarly, if you point Globus to a filesystem endpoint, it will also open a session in your home-directory there.

home_dir.png

There's just one catch: Your cluster home-directory has a 2GB allocation so is not ideal for large data-transfers. So before you start transferring files, navigate to a suitable path for the transfer.

Project Paths

The path to your project folder is /projects/

cluster_project.png

Note:  When you select a new path, make sure to press either "enter" or the "Go" button to activate the "change-directory" command.

FYI: A simple way to verify that you're in the right path is to put a small file in your projects directory and make sure Globus can see it when that path is selected.

How much storage space do you have?

On the cluster, get a disk-space usage summary with the command fs_my_quota_usage:


[username@login01 ~]$   fs_my_quota_usage

Your disk space usage summary
 =============================

/home/username
 Disk space: quota = 2.00 GB, usage = 1.28 MB (0.06%)
 File count: usage = 32 (no quota set)

/projects/project_code
 Disk space: quota = 30.00 GB, usage = 9.59 MB (0.03%)
 File count: quota = 1000000, usage = 7 (0.00%)

 Note: These figures may be up to 15 minutes out of date.

Storage allocation - the scratch filesystem

For transfer of files which exceed this capacity, designate the /scratch directory on the cluster.

The command df -h /scratch will show you how much space is available there at any time:


  $ df -h /scratch
Filesystem            Size  Used Avail Use% Mounted on
/dev/gpfsscratch       13T  6.9T  5.8T  55% /scratch

If you are bulk-transferring data onto the scratch directory of cluster with Globus,

  1. transfer the data to somewhere in the subdirectory /scratch/jobs/<your_username>.
  2. Note: Please move your data out of this scratch space after a transfer from another Globus DTN. The /scratch filesystem is not backed up.
Here's a sample session displaying transfer to the scratch filesystem, to a subdirectory /scratch/jobs/abcd123/data_transfer. It shows an scp session to copy the data off the filesystem, and then deletion of the data.

[abcd123@login-01 ~]$ cd /scratch/jobs/abcd123
[abcd123@login-01 scratch] $ mkdir data_transfer
(perform data transfer into this directory)
[abcd123@login-01 scratch/jobs/abcd123 ] $ tar czvf data_transfer.tar.gz data_transfer/
[abcd123@login-01 scratch/jobs/abcd123] $ scp -r data_transfer.tar.gz destination_filesystem/destination_directory
[abcd123@login-01 scratch/jobs/abcd123] $ rm -rf data_transfer 
[abcd123@login-01 scratch/jobs/abcd123] $ cd
scratch_data_transfer.png
Labels: globus
Was this article helpful?
0 out of 0 found this helpful