I've run out of storage space

There are two tracked resources in the NeSI filesystem, disk space and inodes (number of files) .

Trying to write to a filesystem over it's inode or disk quota will cause an error (and probably kill your job).

Current file-count and disk space can be found using nn_storage_quota.

Filesystem         Available      Used     Use%     Inodes     IUsed     IUse%
home_user123 20G 1.957G 9.79% 92160 21052 22.84%
project_nesi99999 2T 798G 38.96% 100000 66951 66.95%
nobackup_nesi99999 6.833T 10000000 2691383 26.91%

Note

There is a delay between making changes to a filesystem and seeing the change in nn_storage_quota, immediate file count and disk space can be found using the commands du --inodes and du -h respectively.

There are a few ways to deal with file count problems

  • Use /nesi/nobackup/<projectcode>
    The nobackup directory has a significantly higher inode count and no disk space limits. Files here are not backed up, so best used for intermediary or replaceable data.
  • Delete Unnecessary files
    Some applications will generate a large number of files during runtime, using the command du --inodes -d 1 | sort -hr (for inodes) or du -h -d 1 | sort -hr for disk space.  You can then drill down into the directories with the largest file count deleting files as viable.  
  • Tarball
    Many files can be compressed into a single 'tarball' 
    tar -czf name.tar /path/containing/files/
    Then when files need to be accessed again they can be un-tarred using,
    tar -xzf tarname.tar
  • Contact Support
    If you are following the recommendations here yet are still concerned about indoes or disk space, open a support ticket and we can raise the limit for you.
Labels: disk quota exceeded
Was this article helpful?
1 out of 1 found this helpful