LocalDataStorage

Local (data) storage refers to all data that is stored physically at UMIACS, i.e., on hard drives either in servers in datacenters managed by UMIACS staff, or in UMIACS-supported workstations. The opposite of this is cloud storage which is stored on third-party providers' data hosting platforms.

The below sections outline the different categories of local storage available at UMIACS. Although technically local storage by the above definition, UMIACS also hosts an Object Store that is documented outside the scope of this page.

Network Home Directory Local Storage

We provide network home directory local storage to each of our users through NFShomes home directories.

This home directory can be accessed via either

   /nfshomes/<username> (NFS)

or

   \\isilondata.umiacs.umd.edu\nfshomes\<username> (CIFS)

and is backed up nightly using the Institute's TSM backup system. It also has Snapshots enabled for easy user restores.

Users are given a 30GB, non-expandable Quota. You will need to use either platform-specific local storage, directly-attached local storage, or other network-attached local storage for increased space.

On user account closure, the account's NFShomes home directory goes into our Archives.

UNIX Local Storage

UNIX hosts use redundant, backed-up network file shares for user directories (above section). Research data storage (below section) is also stored on redundant, possibly-backed-up network file shares and is generally available under /fs/

All UNIX hosts also have local storage available for transitory use. These directories may be used to store temporary, local COPIES of data that is permanently stored elsewhere or as a staging point for output.

These directories may not, under any circumstances, be used as permanent storage for unique, important data. They are not backed up or archived by UMIACS. UMIACS staff cannot recover damaged or deleted data from these directories and will not be responsible for data loss if they are misused. Additionally, on our SLURM compute clusters, these volumes may have an automated cleanup routine that will delete unmodified data after some number of days. You can check the page for the specific cluster you are using for more information.

Please note that /tmp in particular is at risk for data loss or corruption as that directory is regularly used by system processes and services for temporary storage.

These directories include:

 - /tmp
 - /scratch0, /scratch1, ... (/scratch#)
 - any directory named in whole or in part "tmp", "temp", or "scratch".

Windows and macOS Local Storage

Windows and macOS hosts at UMIACS store user directories on their primary internal drives (C:\Users for Windows, /Users for macOS). Supported, UMIACS-managed hosts automatically back up user data on these drives nightly using the Institute's TSM backup system. If you have a supported, UMIACS-managed host that has other internal or external hard drives attached to it, or other partitions on its primary internal drive, please be aware that these drives/partitions are not backed up. Laptops and non-standard hosts are not automatically backed up and should be manually backed up by their users.

On host decommission, user directories go into our Archives.

Direct-Attached Local Storage

Direct-attached local storage refers to devices like USB flash drives and USB hard drives, which are very popular for easily expanding storage capacity on a host. However, these devices are significantly more vulnerable to data loss or theft than internal or networked data storage. In general, UMIACS discourages the use of direct-attached local storage when any other option is available. Please note that these devices are prone to high rates of failure and additional steps should be taken to ensure that the data is backed up and that critical or confidential data is not lost or stolen.

Direct-attached local storage is not backed up or archived by UMIACS.

Network-Attached Local Storage

Some labs have network-attached local storage space dedicated for datasets, models, and project storage. These shares are typically named in the form /fs/<lab>-<purpose> (i.e., /fs/cml-models or /fs/vulcan-projects).

Network-attached local storage may or may not be backed up and/or archived by UMIACS. Details of a specific share's retention policy should be stated along with the documentation of the share's access / usage policy. If you find an documentation network-attached local storage space in this wiki that does not state its retention policy, please contact staff.

Network-Attached Local Scratch Storage

One specific sub-category of network-attached local storage is network-attached local scratch storage. These shares are named similarly to local scratch or temporary storage, but with the lab's name included (i.e., /fs/cbcb-scratch or /gammascratch), are intended for scratch/temporary storage, and are subject to the same policies as local scratch/tmp directories, discussed above.

Network-attached local scratch storage is not backed up or archived by UMIACS.

UNIX Local Storage Commands

Below are a few different CLI commands that may prove useful for monitoring your local storage usage and performance. For additional information, run [command] --help or man [command]

df - Shows descriptive file system information

Usage: df [OPTION]... [FILE]...
Show information about the file system on which each FILE resides,
or all file systems by default.

du - Shows disk usage of specific files. Use the -d flag for better depth control.

Usage: du [OPTION]... [FILE]...
  or:  du [OPTION]... --files0-from=F
Summarize disk usage of each FILE, recursively for directories.

free - Shows current memory(RAM) usage. Use the -h flag for a human readable format.

Usage:
 free [options]

quota - Shows quota information, this is useful for viewing per filesystem limits in places such as a home directory.

quota: Usage: quota [-guqvswim] [-l | [-Q | -A]] [-F quotaformat]
	quota [-qvswim] [-l | [-Q | -A]] [-F quotaformat] -u username ...
	quota [-qvswim] [-l | [-Q | -A]] [-F quotaformat] -g groupname ...
	quota [-qvswugQm] [-F quotaformat] -f filesystem ...

iostat - Shows drive utilization, as well as other utilizations. Pair this with the watch command for regular updates.

Usage: iostat [ options ] [ <interval> [ <count> ] ]
Options are:
[ -c ] [ -d ] [ -h ] [ -k | -m ] [ -N ] [ -t ] [ -V ] [ -x ] [ -y ] [ -z ]
[ -j { ID | LABEL | PATH | UUID | ... } ]
[ [ -T ] -g <group_name> ] [ -p [ <device> [,...] | ALL ] ]
[ <device> [...] | ALL ]