Data Storage - Revision history

Mbaney at 14:34, 31 October 2025

2025-10-31T14:34:19Z

← Older revision		Revision as of 14:34, 31 October 2025
Line 44:		Line 44:

	==What are some data storage best practices?==		==What are some data storage best practices?==
	# Store data that should persist indefinitely, whether filesystem, Object Store, or cloud, in either a faculty member's storage allocation or a shared storage allocation. This reduces the likelihood that the data is lost when any one individual who has access to it has their account decommissioned due to an expiring appointment, inactivity, bad behavior, or otherwise. This is detailed further on [[Publishing Data \| this page]], specifically as it pertains to publication, and by the Division of IT [https://itsupport.umd.edu/itsupport?id=kb_article_view&sysparm_article=KB0018225#mcetoc_1hntftl5o23 here], specifically as it pertains to Google Drive.		# Store data that should persist indefinitely, whether filesystem, Object Store, or cloud, in either a faculty member's storage allocation or a shared storage allocation. This reduces the likelihood that the data is lost when any one individual who has access to it has their account decommissioned due to an expiring appointment, graduation, inactivity, bad behavior, or otherwise. This is detailed further on [[Publishing Data \| this page]], specifically as it pertains to publication, and by the Division of IT [https://itsupport.umd.edu/itsupport?id=kb_article_view&sysparm_article=KB0018225#mcetoc_1hntftl5o23 here], specifically as it pertains to Google Drive.
	#* As an extension of this, be aware of who the actual owner of a file is in cloud data storage services. Files not owned by you in cloud storage can appear to be present in shared storage allocations or in your own personal allocation, but may in fact just be links to files owned by other individuals. If those individuals have their account decommissioned, the links to those files will no longer work, as the actual file will have been removed. ~~The~~		#* As an extension of this, be aware of who the actual owner of a file is in cloud data storage services. Files not owned by you in cloud storage can appear to be present in shared storage allocations or in your own personal allocation, but may in fact just be links to files owned by other individuals. If those individuals have their account decommissioned, the links to those files will no longer work, as the actual file will have been removed.
	# Be aware of what the retention policy for data in a specific cloud data storage service is '''before''' deciding to store your data there. It is much harder to negotiate restoring data with cloud storage providers than it is with [[HelpDesk \| UMIACS Staff]], so ensure you are OK with accepting the risk.		# Be aware of what the retention policy for data in a specific cloud data storage service is '''before''' deciding to store your data there. It is much harder to negotiate restoring data with cloud storage providers than it is with [[HelpDesk \| UMIACS Staff]], so ensure you are OK with accepting the risk.

Mbaney: /* How is my data retained? */

2024-12-12T16:00:50Z

How is my data retained?

← Older revision		Revision as of 16:00, 12 December 2024
Line 37:		Line 37:


	* '''[[OBJ \| UMIACS' Object Store]]''' is not retained in any way, but decommissioned data is sent to [[Archives]]. There are, however, redundancies in the Object Store that protect against data loss.		* '''[[OBJ \| UMIACS' Object Store]]''' is not retained in any way, but decommissioned data is sent to [[Archives]]. There are, however, redundancies in the Object Store that protect against data loss due to hardware failure.

Mbaney: /* What are some data storage best practices? */

2024-12-12T15:59:58Z

What are some data storage best practices?

← Older revision		Revision as of 15:59, 12 December 2024
Line 46:		Line 46:
	# Store data that should persist indefinitely, whether filesystem, Object Store, or cloud, in either a faculty member's storage allocation or a shared storage allocation. This reduces the likelihood that the data is lost when any one individual who has access to it has their account decommissioned due to an expiring appointment, inactivity, bad behavior, or otherwise. This is detailed further on [[Publishing Data \| this page]], specifically as it pertains to publication, and by the Division of IT [https://itsupport.umd.edu/itsupport?id=kb_article_view&sysparm_article=KB0018225#mcetoc_1hntftl5o23 here], specifically as it pertains to Google Drive.		# Store data that should persist indefinitely, whether filesystem, Object Store, or cloud, in either a faculty member's storage allocation or a shared storage allocation. This reduces the likelihood that the data is lost when any one individual who has access to it has their account decommissioned due to an expiring appointment, inactivity, bad behavior, or otherwise. This is detailed further on [[Publishing Data \| this page]], specifically as it pertains to publication, and by the Division of IT [https://itsupport.umd.edu/itsupport?id=kb_article_view&sysparm_article=KB0018225#mcetoc_1hntftl5o23 here], specifically as it pertains to Google Drive.
	#* As an extension of this, be aware of who the actual owner of a file is in cloud data storage services. Files not owned by you in cloud storage can appear to be present in shared storage allocations or in your own personal allocation, but may in fact just be links to files owned by other individuals. If those individuals have their account decommissioned, the links to those files will no longer work, as the actual file will have been removed. The		#* As an extension of this, be aware of who the actual owner of a file is in cloud data storage services. Files not owned by you in cloud storage can appear to be present in shared storage allocations or in your own personal allocation, but may in fact just be links to files owned by other individuals. If those individuals have their account decommissioned, the links to those files will no longer work, as the actual file will have been removed. The
	# Be aware of what the retention policy is for data in cloud data storage ~~services~~ '''before''' deciding to store your data there. It is much harder to negotiate restoring data with cloud storage providers than it is with [[HelpDesk \| UMIACS Staff]], so ensure you are OK with accepting the risk.		# Be aware of what the retention policy for data in a specific cloud data storage service is '''before''' deciding to store your data there. It is much harder to negotiate restoring data with cloud storage providers than it is with [[HelpDesk \| UMIACS Staff]], so ensure you are OK with accepting the risk.

Mbaney at 15:59, 12 December 2024

2024-12-12T15:59:18Z

← Older revision		Revision as of 15:59, 12 December 2024
Line 23:		Line 23:
	==How can I transfer my data?==		==How can I transfer my data?==
	* '''[[FilesystemDataStorage \| Filesystem data storage]]''' is typically best transferred using commands or programs available in the operating system on which it is stored or most commonly accessed from. A number of these commands are covered [[LocalDataTransfer \| here]].		* '''[[FilesystemDataStorage \| Filesystem data storage]]''' is typically best transferred using commands or programs available in the operating system on which it is stored or most commonly accessed from. A number of these commands are covered [[LocalDataTransfer \| here]].


	* '''[[OBJ \| UMIACS' Object Store]]''' has a [https://obj.umiacs.umd.edu/obj built-in web interface] for transferring data in and out of it, which is usually the best option for individual or relatively small amounts of files/folders. For bulk transfers, one of many [[S3Clients \| compatible clients]] can be used.		* '''[[OBJ \| UMIACS' Object Store]]''' has a [https://obj.umiacs.umd.edu/obj built-in web interface] for transferring data in and out of it, which is usually the best option for individual or relatively small amounts of files/folders. For bulk transfers, one of many [[S3Clients \| compatible clients]] can be used.


	* '''[[CloudDataStorage \| Cloud data storage]]''' transfer methods vary based on the service provider. Most major providers will provide simple upload/download functionality, which works well for individual or relatively small amounts of files/folders. For bulk transfers, [[Rclone]] is one program that [[HelpDesk \| UMIACS staff]] often uses, as it has compatibility with many major providers.		* '''[[CloudDataStorage \| Cloud data storage]]''' transfer methods vary based on the service provider. Most major providers will provide simple upload/download functionality, which works well for individual or relatively small amounts of files/folders. For bulk transfers, [[Rclone]] is one program that [[HelpDesk \| UMIACS staff]] often uses, as it has compatibility with many major providers.

Mbaney at 15:59, 12 December 2024

2024-12-12T15:59:01Z

Mbaney at 15:57, 12 December 2024

2024-12-12T15:57:21Z

← Older revision		Revision as of 15:57, 12 December 2024
Line 6:		Line 6:
	Before choosing where to store any of your data, consider how you may need to interact with that data in the short-term and in the long-term, and who you might need to share the data with. You can copy data between the different types of data storage listed below, but it may be unnecessarily cumbersome if you don't choose the right place for the "master copy" of your data.		Before choosing where to store any of your data, consider how you may need to interact with that data in the short-term and in the long-term, and who you might need to share the data with. You can copy data between the different types of data storage listed below, but it may be unnecessarily cumbersome if you don't choose the right place for the "master copy" of your data.

	* '''[[FilesystemDataStorage \| Filesystem data storage]]''' is best suited for data that you are actively working on. Some examples are code that you are developing or using (perhaps to run computational jobs in [[SLURM]] with), results from computational jobs that ~~have already run but~~ have not yet been published in a paper, or files that need to be processed by another desktop-based or server-based application.		* '''[[FilesystemDataStorage \| Filesystem data storage]]''' is best suited for data that you are actively working on. Some examples are code that you are developing or using (perhaps to run computational jobs in [[SLURM]] with), results from computational jobs that have not yet been published in a paper, or files that need to be processed by another desktop-based or server-based application.
	*: Data on UMIACS' filesystem storage can be moved between different UMIACS-supported hosts and shared with other [[Accounts#UMIACS_Account \| UMIACS account]] holders using common methods such as <tt>cp</tt>, File Explorer, Finder, and more.		*: Data on UMIACS' filesystem storage can be moved between different UMIACS-supported hosts and shared with other [[Accounts#UMIACS_Account \| UMIACS account]] holders using common methods such as <tt>cp</tt>, File Explorer, Finder, and more.

	* '''[[OBJ \| UMIACS' Object Store]]''' is best suited for long-term storage of static data, such as data referenced by published papers. It is also suitable for transferring specific versions of data in and out of UMIACS via large singular files, such as archive (tar/zip/etc.) files.		* '''[[OBJ \| UMIACS' Object Store]]''' is best suited for long-term storage of static data, such as data referenced by published papers. It is also suitable for transferring specific versions of data in and out of UMIACS via large singular files, such as archive (tar/zip/etc.) files.
	*: Data in UMIACS' Object Store can be moved in and out of it, to filesystem storage (whether UMIACS' or otherwise), via its [https://obj.umiacs.umd.edu/obj built-in web interface] or one of many [[S3Clients \| compatible clients]]. Data can be shared publicly via simple download links or [[OBJ/WebHosting \| static websites]] that can visualize ~~the data in a more accessible way~~. UMIACS account holders can sponsor [[Accounts#Guest_/_Collaborator_Account \| Collaborator accounts]] for external collaborators that may need to upload and download ~~new versions of~~ data without needing direct access to UMIACS-supported hosts.		*: Data in UMIACS' Object Store can be moved in and out of it, to filesystem storage (whether UMIACS' or otherwise), via its [https://obj.umiacs.umd.edu/obj built-in web interface] or one of many [[S3Clients \| compatible clients]]. Data can be shared publicly via simple download links or via [[OBJ/WebHosting \| static websites]] that can visualize it. UMIACS account holders can sponsor [[Accounts#Guest_/_Collaborator_Account \| Collaborator accounts]] for external collaborators that may need to upload and download data without needing direct access to UMIACS-supported hosts.

	* '''[[CloudDataStorage \| Cloud data storage]]''' is best suited for collaborative data, such as ~~simple or rich~~ text documents, ~~PDF forms~~, spreadsheets, presentations, pictures, videos, and more. Many cloud storage service providers also provide web-based apps attached to their storage service that can be used to edit these types of data without ever having to download it to a specific device.		* '''[[CloudDataStorage \| Cloud data storage]]''' is best suited for collaborative data, such as text documents, PDFs, spreadsheets, presentations, pictures, videos, and more. Many cloud storage service providers also provide web-based apps attached to their storage service that can be used to edit these types of data without ever having to download it to a specific device.
	*: The methods that can be used to move data in and out of cloud ~~storage to filesystem~~ storage vary based on the specific service provider, but often involve web-based or mobile app-based upload and download. Data can be shared with others across the web via accounts associated with that service provider, and often publicly via simple download links.		*: The methods that can be used to move data in and out of cloud storage vary based on the specific service provider, but often involve web-based or mobile app-based upload and download. Data can be shared with others across the web via accounts associated with that service provider, and often publicly via simple download links.

	If you need help deciding where the best place to store your data is, please [[HelpDesk \| contact staff]].		If you need help deciding where the best place to store your data is, please [[HelpDesk \| contact staff]].

Mbaney at 15:38, 12 December 2024

2024-12-12T15:38:47Z

← Older revision		Revision as of 15:38, 12 December 2024
Line 15:		Line 15:
	*: The methods that can be used to move data in and out of cloud storage to filesystem storage vary based on the specific service provider, but often involve web-based or mobile app-based upload and download. Data can be shared with others across the web via accounts associated with that service provider, and often publicly via simple download links.		*: The methods that can be used to move data in and out of cloud storage to filesystem storage vary based on the specific service provider, but often involve web-based or mobile app-based upload and download. Data can be shared with others across the web via accounts associated with that service provider, and often publicly via simple download links.

	If you need help deciding where the best place to store your data ~~may be~~, please [[HelpDesk \| contact staff]].		If you need help deciding where the best place to store your data is, please [[HelpDesk \| contact staff]].

	==How can I transfer my data?==		==How can I transfer my data?==
Line 30:		Line 30:
	*: [[Archives]]: Final copies of specific types of data, stored on an archive server managed by staff. Taken when specific types of data is decommissioned and retained for up to 5 years (see page for more details).		*: [[Archives]]: Final copies of specific types of data, stored on an archive server managed by staff. Taken when specific types of data is decommissioned and retained for up to 5 years (see page for more details).

	* '''[[OBJ \| UMIACS' Object Store]]''' is not retained in any way, but decommissioned data is sent to [[Archives]].		* '''[[OBJ \| UMIACS' Object Store]]''' is not retained in any way, but decommissioned data is sent to [[Archives]]. There are, however, redundancies in the Object Store that protect against data loss.

	* '''[[CloudDataStorage \| Cloud data storage]]''' retention policies vary based on the service provider. You can often buy additional storage or storage protection for a set price per renewal period. There is also sometimes a Trash folder that stores accidentally deleted data for some period of time, often 30 days.		* '''[[CloudDataStorage \| Cloud data storage]]''' retention policies vary based on the service provider. You can often buy additional storage or storage protection for a set price per renewal period. There is also sometimes a Trash folder that stores accidentally deleted data for some period of time, often 30 days.

Mbaney at 15:34, 12 December 2024

2024-12-12T15:34:08Z

← Older revision		Revision as of 15:34, 12 December 2024
Line 9:		Line 9:
	*: Data on UMIACS' filesystem storage can be moved between different UMIACS-supported hosts and shared with other [[Accounts#UMIACS_Account \| UMIACS account]] holders using common methods such as <tt>cp</tt>, File Explorer, Finder, and more.		*: Data on UMIACS' filesystem storage can be moved between different UMIACS-supported hosts and shared with other [[Accounts#UMIACS_Account \| UMIACS account]] holders using common methods such as <tt>cp</tt>, File Explorer, Finder, and more.

	* '''[[OBJ \| UMIACS' Object Store]]''' is best suited for ~~data that is going to remain static, but needs to remain accessible for very~~ long ~~periods~~ of ~~time~~, such as data referenced by published papers. It is also suitable for transferring specific versions of data in and out of UMIACS via large singular files, such as archive (tar/zip/etc.) files.		* '''[[OBJ \| UMIACS' Object Store]]''' is best suited for long-term storage of static data, such as data referenced by published papers. It is also suitable for transferring specific versions of data in and out of UMIACS via large singular files, such as archive (tar/zip/etc.) files.
	*: Data in UMIACS' Object Store can be moved in and out of it, to filesystem storage (whether UMIACS' or otherwise), via its [https://obj.umiacs.umd.edu/obj built-in web interface] or one of many [[S3Clients \| compatible clients]]. Data can be shared publicly via simple download links or [[OBJ/WebHosting \| static websites]] that can visualize the data in a more accessible way. UMIACS account holders can sponsor [[Accounts#Guest_/_Collaborator_Account \| Collaborator accounts]] for external collaborators that may need to upload and download new versions of data without needing direct access to UMIACS-supported hosts.		*: Data in UMIACS' Object Store can be moved in and out of it, to filesystem storage (whether UMIACS' or otherwise), via its [https://obj.umiacs.umd.edu/obj built-in web interface] or one of many [[S3Clients \| compatible clients]]. Data can be shared publicly via simple download links or [[OBJ/WebHosting \| static websites]] that can visualize the data in a more accessible way. UMIACS account holders can sponsor [[Accounts#Guest_/_Collaborator_Account \| Collaborator accounts]] for external collaborators that may need to upload and download new versions of data without needing direct access to UMIACS-supported hosts.

Mbaney at 18:39, 18 November 2024

2024-11-18T18:39:32Z

← Older revision		Revision as of 18:39, 18 November 2024
Line 1:		Line 1:
	This is a landing page for all topics related to data storage that are available at UMIACS~~. It is under active development~~.		This is a landing page for all topics related to data storage that are available at UMIACS.

	==Where can I store my data?==		==Where can I store my data?==
Line 14:		Line 14:
	* '''[[CloudDataStorage \| Cloud data storage]]''' is best suited for collaborative data, such as simple or rich text documents, PDF forms, spreadsheets, presentations, pictures, videos, and more. Many cloud storage service providers also provide web-based apps attached to their storage service that can be used to edit these types of data without ever having to download it to a specific device.		* '''[[CloudDataStorage \| Cloud data storage]]''' is best suited for collaborative data, such as simple or rich text documents, PDF forms, spreadsheets, presentations, pictures, videos, and more. Many cloud storage service providers also provide web-based apps attached to their storage service that can be used to edit these types of data without ever having to download it to a specific device.
	*: The methods that can be used to move data in and out of cloud storage to filesystem storage vary based on the specific service provider, but often involve web-based or mobile app-based upload and download. Data can be shared with others across the web via accounts associated with that service provider, and often publicly via simple download links.		*: The methods that can be used to move data in and out of cloud storage to filesystem storage vary based on the specific service provider, but often involve web-based or mobile app-based upload and download. Data can be shared with others across the web via accounts associated with that service provider, and often publicly via simple download links.

			If you need help deciding where the best place to store your data may be, please [[HelpDesk \| contact staff]].

	==How can I transfer my data?==		==How can I transfer my data?==

Mbaney at 14:12, 18 November 2024

2024-11-18T14:12:04Z

← Older revision		Revision as of 14:12, 18 November 2024
Line 2:		Line 2:

	==Where can I store my data?==		==Where can I store my data?==
	There is no one answer. Different types of data are used differently and have different optimal storage strategies. Trying to pick a one size fits all option will lead to things being sub-optimal overall.		There is no one answer. Different types of data are used differently and have different optimal storage strategies. Trying to pick a one-size-fits-all option ''will'' lead to things being sub-optimal overall.

	Before choosing where to store any of your data, consider how you may need to interact with that data in the short-term and in the long-term, and who you might need to share the data with. You can copy data between the different types of data storage listed below, but it may be unnecessarily cumbersome if you don't choose the right place for the "master copy" of your data.		Before choosing where to store any of your data, consider how you may need to interact with that data in the short-term and in the long-term, and who you might need to share the data with. You can copy data between the different types of data storage listed below, but it may be unnecessarily cumbersome if you don't choose the right place for the "master copy" of your data.

← Older revision		Revision as of 15:57, 12 December 2024
Line 6:		Line 6:
	Before choosing where to store any of your data, consider how you may need to interact with that data in the short-term and in the long-term, and who you might need to share the data with. You can copy data between the different types of data storage listed below, but it may be unnecessarily cumbersome if you don't choose the right place for the "master copy" of your data.		Before choosing where to store any of your data, consider how you may need to interact with that data in the short-term and in the long-term, and who you might need to share the data with. You can copy data between the different types of data storage listed below, but it may be unnecessarily cumbersome if you don't choose the right place for the "master copy" of your data.

	* '''[[FilesystemDataStorage \| Filesystem data storage]]''' is best suited for data that you are actively working on. Some examples are code that you are developing or using (perhaps to run computational jobs in [[SLURM]] with), results from computational jobs that ~~have already run but~~ have not yet been published in a paper, or files that need to be processed by another desktop-based or server-based application.		* '''[[FilesystemDataStorage \| Filesystem data storage]]''' is best suited for data that you are actively working on. Some examples are code that you are developing or using (perhaps to run computational jobs in [[SLURM]] with), results from computational jobs that have not yet been published in a paper, or files that need to be processed by another desktop-based or server-based application.
	*: Data on UMIACS' filesystem storage can be moved between different UMIACS-supported hosts and shared with other [[Accounts#UMIACS_Account \| UMIACS account]] holders using common methods such as <tt>cp</tt>, File Explorer, Finder, and more.		*: Data on UMIACS' filesystem storage can be moved between different UMIACS-supported hosts and shared with other [[Accounts#UMIACS_Account \| UMIACS account]] holders using common methods such as <tt>cp</tt>, File Explorer, Finder, and more.

	* '''[[OBJ \| UMIACS' Object Store]]''' is best suited for long-term storage of static data, such as data referenced by published papers. It is also suitable for transferring specific versions of data in and out of UMIACS via large singular files, such as archive (tar/zip/etc.) files.		* '''[[OBJ \| UMIACS' Object Store]]''' is best suited for long-term storage of static data, such as data referenced by published papers. It is also suitable for transferring specific versions of data in and out of UMIACS via large singular files, such as archive (tar/zip/etc.) files.
	*: Data in UMIACS' Object Store can be moved in and out of it, to filesystem storage (whether UMIACS' or otherwise), via its [https://obj.umiacs.umd.edu/obj built-in web interface] or one of many [[S3Clients \| compatible clients]]. Data can be shared publicly via simple download links or [[OBJ/WebHosting \| static websites]] that can visualize ~~the data in a more accessible way~~. UMIACS account holders can sponsor [[Accounts#Guest_/_Collaborator_Account \| Collaborator accounts]] for external collaborators that may need to upload and download ~~new versions of~~ data without needing direct access to UMIACS-supported hosts.		*: Data in UMIACS' Object Store can be moved in and out of it, to filesystem storage (whether UMIACS' or otherwise), via its [https://obj.umiacs.umd.edu/obj built-in web interface] or one of many [[S3Clients \| compatible clients]]. Data can be shared publicly via simple download links or via [[OBJ/WebHosting \| static websites]] that can visualize it. UMIACS account holders can sponsor [[Accounts#Guest_/_Collaborator_Account \| Collaborator accounts]] for external collaborators that may need to upload and download data without needing direct access to UMIACS-supported hosts.

	* '''[[CloudDataStorage \| Cloud data storage]]''' is best suited for collaborative data, such as ~~simple or rich~~ text documents, ~~PDF forms~~, spreadsheets, presentations, pictures, videos, and more. Many cloud storage service providers also provide web-based apps attached to their storage service that can be used to edit these types of data without ever having to download it to a specific device.		* '''[[CloudDataStorage \| Cloud data storage]]''' is best suited for collaborative data, such as text documents, PDFs, spreadsheets, presentations, pictures, videos, and more. Many cloud storage service providers also provide web-based apps attached to their storage service that can be used to edit these types of data without ever having to download it to a specific device.
	*: The methods that can be used to move data in and out of cloud ~~storage to filesystem~~ storage vary based on the specific service provider, but often involve web-based or mobile app-based upload and download. Data can be shared with others across the web via accounts associated with that service provider, and often publicly via simple download links.		*: The methods that can be used to move data in and out of cloud storage vary based on the specific service provider, but often involve web-based or mobile app-based upload and download. Data can be shared with others across the web via accounts associated with that service provider, and often publicly via simple download links.

	If you need help deciding where the best place to store your data is, please [[HelpDesk \| contact staff]].		If you need help deciding where the best place to store your data is, please [[HelpDesk \| contact staff]].

← Older revision		Revision as of 15:34, 12 December 2024
Line 9:		Line 9:
	*: Data on UMIACS' filesystem storage can be moved between different UMIACS-supported hosts and shared with other [[Accounts#UMIACS_Account \| UMIACS account]] holders using common methods such as <tt>cp</tt>, File Explorer, Finder, and more.		*: Data on UMIACS' filesystem storage can be moved between different UMIACS-supported hosts and shared with other [[Accounts#UMIACS_Account \| UMIACS account]] holders using common methods such as <tt>cp</tt>, File Explorer, Finder, and more.

	* '''[[OBJ \| UMIACS' Object Store]]''' is best suited for ~~data that is going to remain static, but needs to remain accessible for very~~ long ~~periods~~ of ~~time~~, such as data referenced by published papers. It is also suitable for transferring specific versions of data in and out of UMIACS via large singular files, such as archive (tar/zip/etc.) files.		* '''[[OBJ \| UMIACS' Object Store]]''' is best suited for long-term storage of static data, such as data referenced by published papers. It is also suitable for transferring specific versions of data in and out of UMIACS via large singular files, such as archive (tar/zip/etc.) files.
	*: Data in UMIACS' Object Store can be moved in and out of it, to filesystem storage (whether UMIACS' or otherwise), via its [https://obj.umiacs.umd.edu/obj built-in web interface] or one of many [[S3Clients \| compatible clients]]. Data can be shared publicly via simple download links or [[OBJ/WebHosting \| static websites]] that can visualize the data in a more accessible way. UMIACS account holders can sponsor [[Accounts#Guest_/_Collaborator_Account \| Collaborator accounts]] for external collaborators that may need to upload and download new versions of data without needing direct access to UMIACS-supported hosts.		*: Data in UMIACS' Object Store can be moved in and out of it, to filesystem storage (whether UMIACS' or otherwise), via its [https://obj.umiacs.umd.edu/obj built-in web interface] or one of many [[S3Clients \| compatible clients]]. Data can be shared publicly via simple download links or [[OBJ/WebHosting \| static websites]] that can visualize the data in a more accessible way. UMIACS account holders can sponsor [[Accounts#Guest_/_Collaborator_Account \| Collaborator accounts]] for external collaborators that may need to upload and download new versions of data without needing direct access to UMIACS-supported hosts.

← Older revision		Revision as of 15:59, 12 December 2024
Line 5:		Line 5:

	Before choosing where to store any of your data, consider how you may need to interact with that data in the short-term and in the long-term, and who you might need to share the data with. You can copy data between the different types of data storage listed below, but it may be unnecessarily cumbersome if you don't choose the right place for the "master copy" of your data.		Before choosing where to store any of your data, consider how you may need to interact with that data in the short-term and in the long-term, and who you might need to share the data with. You can copy data between the different types of data storage listed below, but it may be unnecessarily cumbersome if you don't choose the right place for the "master copy" of your data.


	* '''[[FilesystemDataStorage \| Filesystem data storage]]''' is best suited for data that you are actively working on. Some examples are code that you are developing or using (perhaps to run computational jobs in [[SLURM]] with), results from computational jobs that have not yet been published in a paper, or files that need to be processed by another desktop-based or server-based application.		* '''[[FilesystemDataStorage \| Filesystem data storage]]''' is best suited for data that you are actively working on. Some examples are code that you are developing or using (perhaps to run computational jobs in [[SLURM]] with), results from computational jobs that have not yet been published in a paper, or files that need to be processed by another desktop-based or server-based application.
	*: Data on UMIACS' filesystem storage can be moved between different UMIACS-supported hosts and shared with other [[Accounts#UMIACS_Account \| UMIACS account]] holders using common methods such as <tt>cp</tt>, File Explorer, Finder, and more.		*: Data on UMIACS' filesystem storage can be moved between different UMIACS-supported hosts and shared with other [[Accounts#UMIACS_Account \| UMIACS account]] holders using common methods such as <tt>cp</tt>, File Explorer, Finder, and more.


	* '''[[OBJ \| UMIACS' Object Store]]''' is best suited for long-term storage of static data, such as data referenced by published papers. It is also suitable for transferring specific versions of data in and out of UMIACS via large singular files, such as archive (tar/zip/etc.) files.		* '''[[OBJ \| UMIACS' Object Store]]''' is best suited for long-term storage of static data, such as data referenced by published papers. It is also suitable for transferring specific versions of data in and out of UMIACS via large singular files, such as archive (tar/zip/etc.) files.
	*: Data in UMIACS' Object Store can be moved in and out of it, to filesystem storage (whether UMIACS' or otherwise), via its [https://obj.umiacs.umd.edu/obj built-in web interface] or one of many [[S3Clients \| compatible clients]]. Data can be shared publicly via simple download links or via [[OBJ/WebHosting \| static websites]] that can visualize it. UMIACS account holders can sponsor [[Accounts#Guest_/_Collaborator_Account \| Collaborator accounts]] for external collaborators that may need to upload and download data without needing direct access to UMIACS-supported hosts.		*: Data in UMIACS' Object Store can be moved in and out of it, to filesystem storage (whether UMIACS' or otherwise), via its [https://obj.umiacs.umd.edu/obj built-in web interface] or one of many [[S3Clients \| compatible clients]]. Data can be shared publicly via simple download links or via [[OBJ/WebHosting \| static websites]] that can visualize it. UMIACS account holders can sponsor [[Accounts#Guest_/_Collaborator_Account \| Collaborator accounts]] for external collaborators that may need to upload and download data without needing direct access to UMIACS-supported hosts.


	* '''[[CloudDataStorage \| Cloud data storage]]''' is best suited for collaborative data, such as text documents, PDFs, spreadsheets, presentations, pictures, videos, and more. Many cloud storage service providers also provide web-based apps attached to their storage service that can be used to edit these types of data without ever having to download it to a specific device.		* '''[[CloudDataStorage \| Cloud data storage]]''' is best suited for collaborative data, such as text documents, PDFs, spreadsheets, presentations, pictures, videos, and more. Many cloud storage service providers also provide web-based apps attached to their storage service that can be used to edit these types of data without ever having to download it to a specific device.
	*: The methods that can be used to move data in and out of cloud storage vary based on the specific service provider, but often involve web-based or mobile app-based upload and download. Data can be shared with others across the web via accounts associated with that service provider, and often publicly via simple download links.		*: The methods that can be used to move data in and out of cloud storage vary based on the specific service provider, but often involve web-based or mobile app-based upload and download. Data can be shared with others across the web via accounts associated with that service provider, and often publicly via simple download links.


	If you need help deciding where the best place to store your data is, please [[HelpDesk \| contact staff]].		If you need help deciding where the best place to store your data is, please [[HelpDesk \| contact staff]].
Line 29:		Line 33:
	*: [[NightlyBackups]]: Daily copies of specific file systems, sent to a backup server managed by staff. Taken daily and retained for up to 90 days.		*: [[NightlyBackups]]: Daily copies of specific file systems, sent to a backup server managed by staff. Taken daily and retained for up to 90 days.
	*: [[Archives]]: Final copies of specific types of data, stored on an archive server managed by staff. Taken when specific types of data is decommissioned and retained for up to 5 years (see page for more details).		*: [[Archives]]: Final copies of specific types of data, stored on an archive server managed by staff. Taken when specific types of data is decommissioned and retained for up to 5 years (see page for more details).


	* '''[[OBJ \| UMIACS' Object Store]]''' is not retained in any way, but decommissioned data is sent to [[Archives]]. There are, however, redundancies in the Object Store that protect against data loss.		* '''[[OBJ \| UMIACS' Object Store]]''' is not retained in any way, but decommissioned data is sent to [[Archives]]. There are, however, redundancies in the Object Store that protect against data loss.


	* '''[[CloudDataStorage \| Cloud data storage]]''' retention policies vary based on the service provider. You can often buy additional storage or storage protection for a set price per renewal period. There is also sometimes a Trash folder that stores accidentally deleted data for some period of time, often 30 days.		* '''[[CloudDataStorage \| Cloud data storage]]''' retention policies vary based on the service provider. You can often buy additional storage or storage protection for a set price per renewal period. There is also sometimes a Trash folder that stores accidentally deleted data for some period of time, often 30 days.