File systems on the HPC system HYDRA


Your home directory is in the GPFS file system /u (see below).
AFS is only available on the login nodes and on the interactive nodes in order to access software that is distributed by AFS. If you don't get automatically an AFS token during login, you can get an AFS token with the command /usr/bin/klog.krb5. There is no AFS on the compute nodes, so you have to avoid any dependencies on AFS in your job script.
If you get the message
LoadL_starter: 2512-906 Cannot set user credentials.
LoadL_starter: AFS token(s) were captured for the job step, however AFS was not found running on this node
in your job log files, you can ignore it, it's not an error message but only for information.
There are two global, parallel file systems of type GPFS (/u and /ptmp), symmetrically accessible from all HYDRA cluster nodes, plus the migrating file system /r interfacing to the HPSS archive system:

/u (a symbolic link to /hydra/u) for permanent user data (source files, config files, etc.).
(0.75 PB, RAID 6, no system backups)
Your home directory is in /u. If you need a private shell profile, you have to provide it by yourself.
The default disk quota in /u is 2.5 TB. You can check your disk quota in /u with the command:

   /usr/lpp/mmfs/bin/mmlsquota u

GPFS offers the possibility to share files with colleagues. If you are interested, please look here.

/ptmp (a symbolic link to /hydra/ptmp) for batch job I/O.
(3.8 PB, RAID 6, no system backups)
Files in /ptmp that have been not accessed for more than 12 weeks will be removed automatically. The period of 12 weeks may be reduced if necessary (with prior notice).

As a current policy, no quotas are applied on /ptmp. This gives users the freedom to manage their data according to their actual needs without administrative overhead. This liberal policy presumes a fair usage of the common file space. So, please do a regular housekeeping of your data and archive/remove files that are not used actually.


Archiving data from the GPFS file systems to tape can be done using the migrating file system /r (see below).

/r (a symbolic link to /ghi/r) for migrated data, available only on the login nodes and on the interactive nodes
Each user has a subdirectory /r/<initial>/<userid> to store his/her data. For efficiency, files should be packed to tar, cpio or zip files (with a size of 1 to 500 GByte) before archiving them in /r, that is, please avoid archiving small files. When this file system /r gets full above a certain value, files will be transferred from disk to tape, beginning with the largest files which have been unused the longest time.

If you access a file which has been migrated to tape, the file will automatically be transferred back from tape to disk. This of course implies a delay. You can manually force the recall of a migrated file by using any command which opens the file. You can recall in advance all files needed by some job with a command like

file myfiles/*

You can see which files are resident on disk and which ones have been migrated to tape with the command ghi_ls (located in /usr/local/bin), optionally with the option -l. Here is a sample output:

hydra01% ghi_ls -l
G -rw-r--r--   1  ifw       rzs             22 Nov 21 15:12 a1
H -rw-------   1  ifw       rzs   138958551040 Sep 18 22:22 abc.tar
H -rw-r--r--   1  ifw       rzs     1073741312 May 06 2009  core
G -rw-r--r--   1  ifw       rzs              0 Jun 20 2008  dsmerror.log
B -rw-r--r--   1  ifw       rzs     1079040000 Aug 03 2010  dummyz3
The first column states where the file resides: a 'G' means the file is resident on disk; a 'H' means the file has been transferred to the underlying HPSS tape archiving system; a 'B' means premigrated to tape (the file has already been copied to HPSS but is still present on disk and can be removed immediately if the system needs to free disk space).


Please note: If you want to "tar" files that are already located in /r, please carefully check in the contents of the resulting TAR file whether all migrated files were correctly retrieved and included into the TAR file.
Don't use "gzip" or "compress" on files that are already located in /r. It's not necessary because all files are automatically compressed by hardware as soon as they are written to tape.

For more information about /r see :
HSM (HPSS) for users of the IBM High-Performance Computer


Please, DON'T use the file system /tmp for your scratch data. Instead, use /ptmp which is accessible from all HYDRA cluster nodes, and set the environment variable TMPDIR to /ptmp/<userid> in your job scripts.

Document Actions