Datasets: Difference between revisions
No edit summary |
No edit summary |
||
| (5 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
UMIACS hosts a number of datasets in read-only mode on some shared filesystems used by | UMIACS hosts a number of datasets in read-only mode on some shared filesystems used by our [[SLURM]] computing clusters. The motivation behind this is to provide publicly accessible datasets in a well-defined location in order to de-duplicate their use elsewhere, thus reducing overall storage usage. | ||
==Dataset Directories== | ==Dataset Directories== | ||
* [[Nexus/CML | CML]] (<code>/fs/cml-datasets</code>) | * [[Nexus/CML | CML]] (<code>/fs/cml-datasets</code>) | ||
** [https://info.umiacs.umd.edu/datasets/list/?q=CML List of datasets] -- faculty approver Tom Goldstein | ** [https://info.umiacs.umd.edu/datasets/list/?q=CML List of datasets] -- faculty approver Tom Goldstein | ||
* [[Nexus/GAMMA | GAMMA]] (<code>/fs/gamma-datasets</code>) | |||
** [https://info.umiacs.umd.edu/datasets/list/?q=GAMMA List of datasets] -- faculty approver Dinesh Manocha | |||
* [[Nexus]] (<code>/fs/nexus-datasets</code>) | * [[Nexus]] (<code>/fs/nexus-datasets</code>) | ||
** [https://info.umiacs.umd.edu/datasets/list/?q=Nexus List of datasets] | ** [https://info.umiacs.umd.edu/datasets/list/?q=Nexus List of datasets] | ||
| Line 10: | Line 12: | ||
==Requesting a new dataset== | ==Requesting a new dataset== | ||
You can request a new dataset by [[HelpDesk | contacting staff]] with a link to the dataset's official download location. | You can request a new dataset by [[HelpDesk | contacting staff]] with a link to the dataset's official download location. Mirrors and torrents/other peer-to-peer re-hosting are not allowed unless sanctioned by the dataset owners. | ||
* '''CML/Vulcan''': If the uncompressed/final dataset size is over 100GB, staff will first contact the faculty approver for the cluster to ensure they approve of using the storage space. If the size is under 100GB, no faculty approval is required. Staff will then inspect the dataset and see if there are any terms and conditions that must be agreed to before downloading. | * '''CML/GAMMA/Vulcan''': If the uncompressed/final dataset size is over 100GB, staff will first contact the faculty approver for the cluster to ensure they approve of using the storage space. If the size is under 100GB, no faculty approval is required. Staff will then inspect the dataset and see if there are any terms and conditions that must be agreed to before downloading. | ||
* '''Nexus''': Please let staff know which faculty member's research you are working on that requires use of the dataset you are requesting. Then, if the uncompressed/final dataset size is over 50GB, the [https://www.umiacs.umd.edu/people/computing-staff Director of Computing Facilities] must first approve of using the storage space. If the size is under 50GB, no approval is required. Staff will then inspect the dataset and see if there are any terms and conditions that must be agreed to before downloading. | * '''Nexus''': Please let staff know which faculty member's research you are working on that requires use of the dataset you are requesting. Then, if the uncompressed/final dataset size is over 50GB, the [https://www.umiacs.umd.edu/people/computing-staff Director of Computing Facilities] must first approve of using the storage space. If the size is under 50GB, no approval is required. Staff will then inspect the dataset and see if there are any terms and conditions that must be agreed to before downloading. | ||
If there are no terms and conditions, staff will download/extract the dataset, copy it to the appropriate location depending on what cluster you are requesting it for, and let you know when it is available for use on that cluster. | If there are no terms and conditions, staff will download/extract the dataset, copy it to the appropriate location depending on what cluster you are requesting it for, and let you know when it is available for use on that cluster. | ||
If there are terms and conditions in excess of well-defined [https://creativecommons.org/about/cclicenses/ Creative Commons licenses], staff will first need to have the dataset's terms and conditions approved by [https://ora.umd.edu/ UMD's Office of Research Administration (ORA)]. Since the dataset will be hosted by UMIACS | If there are terms and conditions in excess of well-defined [https://creativecommons.org/about/cclicenses/ Creative Commons licenses], staff will first need to have the dataset's terms and conditions approved by [https://ora.umd.edu/ UMD's Office of Research Administration (ORA)]. Since the dataset will be hosted by UMIACS, as an institution, in a location accessible by all users of a cluster, not all of whom will have individually agreed to the terms and conditions, having a single person, staff member or not, agree to a set of terms and conditions is not sufficient to host it. After approval by ORA, staff will perform the same steps as mentioned above (download/extract, copy to appropriate location, and let you know when ready). | ||
==Dataset use== | ==Dataset use== | ||
Latest revision as of 16:08, 16 February 2024
UMIACS hosts a number of datasets in read-only mode on some shared filesystems used by our SLURM computing clusters. The motivation behind this is to provide publicly accessible datasets in a well-defined location in order to de-duplicate their use elsewhere, thus reducing overall storage usage.
Dataset Directories
- CML (
/fs/cml-datasets)- List of datasets -- faculty approver Tom Goldstein
- GAMMA (
/fs/gamma-datasets)- List of datasets -- faculty approver Dinesh Manocha
- Nexus (
/fs/nexus-datasets) - Vulcan (
/fs/vulcan-datasets)- List of datasets -- faculty approver Abhinav Shrivastava
Requesting a new dataset
You can request a new dataset by contacting staff with a link to the dataset's official download location. Mirrors and torrents/other peer-to-peer re-hosting are not allowed unless sanctioned by the dataset owners.
- CML/GAMMA/Vulcan: If the uncompressed/final dataset size is over 100GB, staff will first contact the faculty approver for the cluster to ensure they approve of using the storage space. If the size is under 100GB, no faculty approval is required. Staff will then inspect the dataset and see if there are any terms and conditions that must be agreed to before downloading.
- Nexus: Please let staff know which faculty member's research you are working on that requires use of the dataset you are requesting. Then, if the uncompressed/final dataset size is over 50GB, the Director of Computing Facilities must first approve of using the storage space. If the size is under 50GB, no approval is required. Staff will then inspect the dataset and see if there are any terms and conditions that must be agreed to before downloading.
If there are no terms and conditions, staff will download/extract the dataset, copy it to the appropriate location depending on what cluster you are requesting it for, and let you know when it is available for use on that cluster.
If there are terms and conditions in excess of well-defined Creative Commons licenses, staff will first need to have the dataset's terms and conditions approved by UMD's Office of Research Administration (ORA). Since the dataset will be hosted by UMIACS, as an institution, in a location accessible by all users of a cluster, not all of whom will have individually agreed to the terms and conditions, having a single person, staff member or not, agree to a set of terms and conditions is not sufficient to host it. After approval by ORA, staff will perform the same steps as mentioned above (download/extract, copy to appropriate location, and let you know when ready).
Dataset use
All datasets are read-only to users. Any intermediate data generated from a dataset will need to be stored in a location other than the shared filesystem hosting the dataset and other datasets.
Exceptions may be granted if there is a set of intermediate data generated from a dataset that you believe will be useful to a subset of a cluster's users. If you suspect this is the case for some of your generated data, please contact staff. We will follow the above procedure and upon whatever approvals may be necessary, copy the intermediate data into the shared filesystem.