OBJ

UMIACS Object Store

An object store is a web-based storage solution focused on reliability, scalability and security. It is best suited for public content storage/distribution, archiving data or secure data sharing between users. Our Object Storage can be used through the web interface, the command line UMobj utilities, third-party graphical clients, and even programmatically using many popular programming languages. We support a subset of the Amazon Simple Storage Services (S3) API, built around a technology called Ceph.

Terminology

S3-like storage thinks in terms of buckets and keys. Keys are analogous to files. A bucket is simply a container to a set of keys. There is no actual hierarchy inside of a bucket, but the standard UNIX path separator, a forward slash (/) at the end of a Key name, is interpreted by many clients (including this web site and our umobj utilities) as being a directory delimiter. This allows you to copy data from your local filesystems to your buckets through umobj or third-party clients. You may specify who has what types of access to your buckets via Access Control Lists (ACLs) at the bucket level or the individual key level.

Your data is protected from individual machine failure via replication within the cluster. All data is checksummed in accordance with the Amazon S3 protocol to ensure that data in transit is valid before it is accepted by the cluster. However, there are no backups or snapshots of this data in the cluster, so if a user deletes a key or bucket in the object store, there is no way to restore that information.

Getting Started

UMIACS users are allocated 50GB of storage. Faculty are allocated 1TB. To get started, log in and you will be redirected to the initial help page. You can also find the link from our https://intranet.umiacs.umd.edu site as "OBJbox Object Store".

Buckets

You can create and browse your buckets (containers that hold data) by visiting your buckets page. You can also set bucket-level Access Control Lists (ACLs) from this page. Bucket-level ACLs get implicitly inherited to all the keys within the bucket. However, individual keys can have additional specific ACLs applied for more granular control.

Bucket names must be unique. When you create a bucket it will notify you if the name is already taken.

Keys (files)

After selecting a bucket, you will be able to create folders and upload files within that bucket. Listed files can be downloaded, deleted, or assigned a specific ACL by the key owner/creator.

Please note: Local file system ownership and permissions, and special files (such as symlinks) can not be represented in the object store. We highly suggest that if you are securing data into the object store and need these to be faithfully maintained that you use a local archive tool (tar, zip, etc..) to collect the data and then upload the resulting archive file(s).

Hosting a Website in your Bucket

On the settings page for each bucket, there is a section allowing you to specify a configuration for a website hosted in your bucket. You can choose keys located in your bucket to act as the index file (what will be displayed when someone visits your website) and the error file (what will be displayed when an error is encountered loading your website). These settings can be updated at any time.

Because hosting a website through your bucket requires others to have access to any keys associated with your website, setting up a website configuration will automatically set the visibility of your bucket to public and remove the bucket level ACLs. You will not be able to set the bucket back to private without first removing the website configuration, which can be done on the bucket settings page.

The Object Store only has the capability to support static content and client-side scripting in bucket websites.

Deleting Keys (files)

Within the web interface you can delete files one-by-one. If you want to remove a bunch of files, you will need to use a different client as described below.

This is dangerous as there are no backups of files in the object store. Be careful to only delete the data you intend to delete.

Clients

There are several clients that can be used (sometimes with a limited set of features) on your desktop to gain access to the Object Store. All supported UMIACS systems have a copy of our umobj utilities which provide command line access to the object store. We also have an article in our wiki on 3rd party clients that lists and explains the details. These clients need to be configured with your Access and Secret Keys as described below.

Access Key and Secret Key

Each user has one or more pairs of Access Keys and Secret Keys that are used as a credential to not expose your password when using the object store. These can be obtained by clicking on your username in the upper right-hand corner. You'll use these to identify and authenticate yourself to the Object Store.

When using the umobj utilities, you will need to make sure you have added these credentials in your local shell initialization files. There are links to files that have these automatically generated for the 3 most popular UNIX shell families (bash/sh, csh/tcsh, and zsh). Please make sure that whatever file(s) you copy these credentials into can not be read by other users (eg. chmod 600 filename).

Note: Each Access Key and Secret Key are specific to a particular object store, so if you are accessing multiple object stores you may want to write the credentials for each to separate files and then source each file when you want to use the associated object store. Please contact "staff [at] umiacs.umd.edu" if you have any questions.

Lab Groups

Lab Groups allow a group of users to share data while avoiding the need for complex ACLs by maintaining group ownership. Designated Lab Group managers can grant granular access (read, write, full control, manager) to buckets owned by the Lab Group. All objects owned by a Lab Group count against the group quota. Lab Groups can be navigated using the menu with username in the top right corner of the page. Note: this will only appear if you are a member of at least one Lab Group. At this point, you can browse the Object Store as the Lab Group and obtain your unique Access Key and Secret Key pair using the instructions above. To switch to another Lab Group or back to your own buckets, click the menu again and select another user or group.

Managing Lab Groups

Lab Groups have many different levels of membership: Managers, FULL_CONTROL, READ/WRITE, and READ. Managers can add or remove Lab Group Members while every other access level cannot. If you hold the Manager role in a Lab Group, you can add and remove users using the Manage LabGroups page, which is available under the Manage menu at the top of the page. After selecting a Lab Group, you can add users by typing their username into the search field and selecting a membership role.

Requesting a Lab Group

To request a Lab Group for your project, please contact "staff [at] umiacs.umd.edu"