Datasets

UMIACS hosts a number of datasets in read-only mode on some shared filesystems used by some of our SLURM computing clusters. The motive behind this is to provide commonly-used datasets in a well-defined location in order to de-duplicate their use elsewhere, thus reducing overall storage usage.

Clusters

• CML
  • List of datasets -- faculty approver Tom Goldstein
• Vulcan
  • List of datasets -- faculty approver Abhinav Shrivastava

Requesting a new dataset

You can request a new dataset by contacting staff with a link to the dataset's official download location (no torrents or other peer-to-peer re-hosting). Staff will first contact the faculty approver for the cluster to ensure they approve. Staff will then inspect the dataset and see if there are any terms and conditions that must be agreed to before downloading.

If there are no terms and conditions, staff will download/extract the dataset, copy it to the appropriate location depending on what cluster you are requesting it for, and let you know when it is available for use on that cluster.

If there are terms and conditions, staff will first need to have the dataset's terms and conditions approved by UMD's Office of Research Administration (ORA). Since the dataset will be hosted by us in a location accessible by all users of a cluster, not all of which will have individually agreed to the terms and conditions, having a single person (staff member or not) agree to a set of terms and conditions is not sufficient to publish it. After approval, staff will perform the same steps as mentioned above (download/extract, copy to appropriate location, and let you know when ready).

Dataset use

All datasets are read-only to users. Any intermediate data generated from a dataset will need to be stored in a location other than the shared filesystem hosting the dataset and other datasets.

Exceptions may be granted if there is a set of intermediate data generated from a dataset that you believe will be useful to a subset of a cluster's users. If you suspect this is the case for some of your generated data, please contact staff. We will reach out to the faculty approver for the cluster (as in the above section) and, upon approval, copy the intermediate data into the shared filesystem.