Follow

Disk Space Allocations and Usage

Disk Space Allocations (Quotas)

NeSI serves a large number of researchers and computational projects with diverse resource requirements. In order to best meet the needs of the research community, all persistent user-writable storage spaces on the Pan cluster have disk space allocations, also known as quotas, and many of these spaces also have inode quotas. An "inode" is something in a directory, like another directory, a file, or a symbolic link.

What Quotas does NeSI Give?

Home directories have disk space allocations, while project directories have both disk space and inode allocations. If at any time you think you will need more disk space or more inode capacity for your project, please email support@nesi.org.nz.

How Long do Quotas Last?

Some disk allocations are awarded for the lifetime of the project, or, if the allocation involves a user's home directory, for the duration of that user's access to the cluster. Other disk allocations are set to expire at the start of a specified day. When we agree to award a disk space or inode allocation, we endeavour to tell the requester when the allocation is scheduled to expire.

Please be aware that, unless a particular grant of disk space or inode capacity forms part of a contract between you (or your team or institution) and a relevant NeSI collaborator, your disk allocations are not guaranteed entitlements, and we may need to lower an unexpired disk space or inode quota for operational reasons. If at any time we need to do this in connection with your home directory or a project you are involved in, we will endeavour to give you at least several weeks' notice, and will work with you and your team to mitigate any adverse effects such a disk space or inode reduction may have on your research. On the other hand, once a granted disk space or inode allocation has expired, we may revoke that allocation at any time.

What are My Current Quotas?

To find out your current disk space quotas, on the Pan login node you can run:

fs_my_quota_usage

This command will produce output like the following:

      Your disk space usage summary
      =============================

      /home/jblo123
            Disk space: quota = 20.00 GB, usage = 4.84 GB (24.22%)
            File count: usage = 105339 (no quota set)

      /projects/nesi00357
            Disk space: quota = 30.00 GB, usage = 0 KB (0.00%)
            File count: quota = 1000000, usage = 1 (0.00%)


      Note: These figures may be up to 15 minutes out of date.

Disk Space Usage

We provide various commands to help you better understand your disk space usage.

What is My Current Usage?

You can find out your current usage in two ways:

Can I Get a File-by-File Breakdown of Disk Usage?

The du command will provide this information. However, it has some limitations and inaccuracies:

  • du does quick and dirty estimates of file sizes, and rounding errors may also be introduced when summing these estimates to form totals.
  • Even when the estimates are correct, du calculates the space taken up by files, while df calculates the space taken up by wholly or partially used disk blocks, which is generally larger (though there are things called sparse files that are exceptions to this rule). There are flags that can be passed to du to change its behaviour in this regard.
  • Every file within the directory that has been deleted but is still in use by a process somewhere will be ignored by du but counted by df.
  • Unlike du, df includes in its calculations entities other than files, such as metadata entries.

These things being said, du is usually a good way to get a general idea as to which files or directories are responsible for consuming large amounts of available disk space or inode seats.

Comments

Powered by Zendesk