LocalDataStorage: Difference between revisions

From UMIACS
Jump to navigation Jump to search
(merging UNIXDataPolicies with LocalDataStorage)
No edit summary
(18 intermediate revisions by 8 users not shown)
Line 2: Line 2:


==Windows Local Storage==
==Windows Local Storage==
Windows hosts at UMIACS store user directories on their local drives. Supported, UMIACS-managed hosts are automatically backed up using the Institute's backup system. Laptops and non-standard hosts are not automatically backed up and be manually backed up by their users.
Windows hosts at UMIACS store user directories on their local C drives. Supported, UMIACS-managed hosts automatically back up user data on the C drive nightly using the Institute's backup system. If you have a supported, UMIACS-managed host that has other internal or external hard drives attached to it, or partitions other than C on its primary hard drive, please be aware that these drives/partitions '''are not''' backed up. Laptops and non-standard hosts are not automatically backed up and should be manually backed up by their users.


==UNIX Remote Storate==
==UNIX Remote Storage==
We provide storage to each of our users in our UNIX offerings through the [[OpenLAB]] [[NFShomes]].
We provide storage to each of our users in our UNIX offerings through the [[NFShomes]].


This home directory,
This home directory,
Line 11: Line 11:
     /nfshomes/username
     /nfshomes/username


is backed up nightly into our [[TSM]] backup system.  If you wish to have files restored pleased follow these [[TSMRestore|instructions]].  We also now provide [[Snapshots]] on this file system.
is backed up nightly into our [[TSM]] backup system.  This volume has [[Snapshots]] enabled for easy user restores.


Users are given a 2 gigabyte [[Quota]].
Users are given a 20 gigabyte [[Quota]]. This quota is non-expandable and you will need to use either local or other network storage for increased space.


==UNIX Local Storage==
==UNIX Local Storage==
Line 20: Line 20:
All UNIX machines also have local storage available for transitory use.  These directories may be used to store temporary, local '''''COPIES''''' of data that is permanently stored elsewhere or as a staging point for output.
All UNIX machines also have local storage available for transitory use.  These directories may be used to store temporary, local '''''COPIES''''' of data that is permanently stored elsewhere or as a staging point for output.


These directories may not, '''''under any circumstances''''', be used as permanent storage for unique, important data.  UMIACS staff cannot recover damaged or deleted data from these directories and will not be responsible for data loss if they are misused.
These directories may not, '''''under any circumstances''''', be used as permanent storage for unique, important data.  UMIACS staff cannot recover damaged or deleted data from these directories and will not be responsible for data loss if they are misused. Additionally, these volumes may have an automated cleanup routine that will delete unmodified data after some number of days. You can check the page for the specific cluster you are using for more information.


Please note that '''/tmp''' in particular is at risk for data loss or corruption as that directory is regularly used by system processes and services for temporary storage.
Please note that '''/tmp''' in particular is at risk for data loss or corruption as that directory is regularly used by system processes and services for temporary storage.
Line 27: Line 27:


   - /tmp
   - /tmp
   - /scratch0, /scratch1, /scratch*
   - /scratch0, /scratch1, ... (/scratch#)
   - any directory named in whole or in part "tmp", "temp", or "scratch".
   - any directory named in whole or in part "tmp", "temp", or "scratch".


==Locally Attached Storage==
==Locally Attached Storage==
Locally attached storage like USB flash drives and USB hard drives are very popular.  However, these devices are significantly more vulnerable to data loss or theft than internal or networked data storage.  In general, UMIACS discourages the use of locally attached network storage when any other option is available.  Please note that these devices are prone to high rates of failure and additional steps should be taken to ensure that the data is backed up and that critical or confidential data is not lost or stolen.
Locally attached storage like USB flash drives and USB hard drives are very popular.  However, these devices are significantly more vulnerable to data loss or theft than internal or networked data storage.  In general, UMIACS discourages the use of locally attached network storage when any other option is available.  Please note that these devices are prone to high rates of failure and additional steps should be taken to ensure that the data is backed up and that critical or confidential data is not lost or stolen.
==Network Scratch Storage==
Some labs have network-attached storage dedicated for scratch/temporary storage. These shares are named in the same manner as local scratch or temporary storage (i.e. /fs/lab-scratch or /lab/scratch0 ) and are subject to the same policies as local scratch/tmp (discussed above.)
==UNIX Storage Commands==
Below are a few different CLI commands that may prove useful for monitoring your storage usage and performance. For additional information, run <code>[command] --help</code> or <code>man [command]</code>
df - Shows descriptive file system information
<pre>
Usage: df [OPTION]... [FILE]...
Show information about the file system on which each FILE resides,
or all file systems by default.
</pre>
du - Shows disk usage of specific files. Use the -d flag for better depth control.
<pre>
Usage: du [OPTION]... [FILE]...
  or:  du [OPTION]... --files0-from=F
Summarize disk usage of each FILE, recursively for directories.
</pre>
free - Shows current memory(RAM) usage. Use the -h flag for a human readable format.
<pre>
Usage:
free [options]
</pre>
quota - Shows quota information, this is useful for viewing per filesystem limits in places such as a home directory.
<pre>
quota: Usage: quota [-guqvswim] [-l | [-Q | -A]] [-F quotaformat]
quota [-qvswim] [-l | [-Q | -A]] [-F quotaformat] -u username ...
quota [-qvswim] [-l | [-Q | -A]] [-F quotaformat] -g groupname ...
quota [-qvswugQm] [-F quotaformat] -f filesystem ...
</pre>
iostat - Shows drive utilization, as well as other utilizations. Pair this with the <code>watch</code> command for regular updates.
<pre>
Usage: iostat [ options ] [ <interval> [ <count> ] ]
Options are:
[ -c ] [ -d ] [ -h ] [ -k | -m ] [ -N ] [ -t ] [ -V ] [ -x ] [ -y ] [ -z ]
[ -j { ID | LABEL | PATH | UUID | ... } ]
[ [ -T ] -g <group_name> ] [ -p [ <device> [,...] | ALL ] ]
[ <device> [...] | ALL ]
</pre>

Revision as of 13:37, 9 May 2022

UMIACS recommends that any and all important data be stored on a redundant, backed-up file server. However, there are a number of cases where this is not feasible.

Windows Local Storage

Windows hosts at UMIACS store user directories on their local C drives. Supported, UMIACS-managed hosts automatically back up user data on the C drive nightly using the Institute's backup system. If you have a supported, UMIACS-managed host that has other internal or external hard drives attached to it, or partitions other than C on its primary hard drive, please be aware that these drives/partitions are not backed up. Laptops and non-standard hosts are not automatically backed up and should be manually backed up by their users.

UNIX Remote Storage

We provide storage to each of our users in our UNIX offerings through the NFShomes.

This home directory,

   /nfshomes/username

is backed up nightly into our TSM backup system. This volume has Snapshots enabled for easy user restores.

Users are given a 20 gigabyte Quota. This quota is non-expandable and you will need to use either local or other network storage for increased space.

UNIX Local Storage

UNIX machines use redundant, backed-up network file shares for user directories. Research data storage is also stored on redundant, backed-up network file shares and is generally available under /fs/

All UNIX machines also have local storage available for transitory use. These directories may be used to store temporary, local COPIES of data that is permanently stored elsewhere or as a staging point for output.

These directories may not, under any circumstances, be used as permanent storage for unique, important data. UMIACS staff cannot recover damaged or deleted data from these directories and will not be responsible for data loss if they are misused. Additionally, these volumes may have an automated cleanup routine that will delete unmodified data after some number of days. You can check the page for the specific cluster you are using for more information.

Please note that /tmp in particular is at risk for data loss or corruption as that directory is regularly used by system processes and services for temporary storage.

These directories include:

 - /tmp
 - /scratch0, /scratch1, ... (/scratch#)
 - any directory named in whole or in part "tmp", "temp", or "scratch".

Locally Attached Storage

Locally attached storage like USB flash drives and USB hard drives are very popular. However, these devices are significantly more vulnerable to data loss or theft than internal or networked data storage. In general, UMIACS discourages the use of locally attached network storage when any other option is available. Please note that these devices are prone to high rates of failure and additional steps should be taken to ensure that the data is backed up and that critical or confidential data is not lost or stolen.

Network Scratch Storage

Some labs have network-attached storage dedicated for scratch/temporary storage. These shares are named in the same manner as local scratch or temporary storage (i.e. /fs/lab-scratch or /lab/scratch0 ) and are subject to the same policies as local scratch/tmp (discussed above.)

UNIX Storage Commands

Below are a few different CLI commands that may prove useful for monitoring your storage usage and performance. For additional information, run [command] --help or man [command]

df - Shows descriptive file system information

Usage: df [OPTION]... [FILE]...
Show information about the file system on which each FILE resides,
or all file systems by default.

du - Shows disk usage of specific files. Use the -d flag for better depth control.

Usage: du [OPTION]... [FILE]...
  or:  du [OPTION]... --files0-from=F
Summarize disk usage of each FILE, recursively for directories.

free - Shows current memory(RAM) usage. Use the -h flag for a human readable format.

Usage:
 free [options]

quota - Shows quota information, this is useful for viewing per filesystem limits in places such as a home directory.

quota: Usage: quota [-guqvswim] [-l | [-Q | -A]] [-F quotaformat]
	quota [-qvswim] [-l | [-Q | -A]] [-F quotaformat] -u username ...
	quota [-qvswim] [-l | [-Q | -A]] [-F quotaformat] -g groupname ...
	quota [-qvswugQm] [-F quotaformat] -f filesystem ...

iostat - Shows drive utilization, as well as other utilizations. Pair this with the watch command for regular updates.

Usage: iostat [ options ] [ <interval> [ <count> ] ]
Options are:
[ -c ] [ -d ] [ -h ] [ -k | -m ] [ -N ] [ -t ] [ -V ] [ -x ] [ -y ] [ -z ]
[ -j { ID | LABEL | PATH | UUID | ... } ]
[ [ -T ] -g <group_name> ] [ -p [ <device> [,...] | ALL ] ]
[ <device> [...] | ALL ]