UMobj: Difference between revisions

From UMIACS
Jump to navigation Jump to search
No edit summary
No edit summary
Line 1: Line 1:
This is a set of console utilities that are available on all UMIACS-supported Linux (RHEL5, RHEL6, Ubuntu) machines.  They are provided to support our [http://ceph.com Ceph] Object Stores so that our users may interact with them on the command line.  In addition, each Ceph Object Store in UMIACS has a fully-functional web application that allows access to your storage from a web browser.
This is a set of console utilities that are available on all UMIACS-supported Linux (RHEL5, RHEL6, Ubuntu) machines.  They are provided to support our [http://ceph.com Ceph] Object Stores so that our users may interact with them on the command line.  In addition, each Ceph Object Store in UMIACS has a fully-functional web application that allows access to your storage from a web browser.


For UMIACS-supported linux machines, these utilities are installed at <code>/opt/UMobj/bin</code>.  The UMobj binaries should already be in your [[PATH]].
For UMIACS-supported linux machines, these utilities are already installed and should be available in the standard [[PATH]].


==Prerequisites==
For more information see the [[GitLab]] page for [https://gitlab.umiacs.umd.edu/staff/umobj/tree/master umobj].
 
To use any of these utilities you either have to pass in your access and secret keys on the command line or you need to set some appropriate environmental variables.  You can obtain your access and secret keys (which are different from your username and password) by visiting the appropriate Ceph Object Store.  The default store is https://obj.umiacs.umd.edu/obj/.
 
Your access and secret keys for that Ceph Object Store can be retrieved by visiting https://obj.umiacs.umd.edu/obj/user.  You can then set, for example, in bash the following environment variables:
 
  export OBJ_ACCESS_KEY_ID="31sdfadDFAHFDN+344qOEIS"
  export OBJ_SECRET_ACCESS_KEY="NDSMK3233adfahadflkkPDSH092DSJKDKDJKFDLSFLNK"
 
If you are not using the main Institute-wide Object Store you can tell the utilities to use an alternative server.
 
  export OBJ_SERVER="gembox.cbcb.umd.edu"
 
You can optionally pass the <code>--access_key</code>, <code>--secret_key</code> and <code>--server</code> switches to the utilities.
 
==lsobj==
 
To list buckets and keys in the Object Store you can use the <b>lsobj</b> command.  If given without an argument it will list your buckets. (this will only list buckets that you created)
 
<pre>
$ lsobj
bob
test
zeta
</pre>
 
You can then give it a bucket name to list the contents within that bucket.  If you were granted access to a bucket that you didn't create you can also use this command to display its contents.  This will list all the keys in your bucket, which can take a long time.
 
<pre>
$ lsobj test
-rwx--- 2013-10-04T15:25:09.000Z     1.8 KB UMIACSCA.pem
-rwx--- 2013-10-04T15:25:24.000Z   311.4 KB cover.jpg
-rwx--- 2013-10-04T15:27:39.000Z     0.0 b foo/
-rwx--- 2013-10-04T15:27:40.000Z     0.0 b foo/bar
-rwx--- 2013-10-04T15:25:32.000Z   31.8 KB screenshot.jpg
-rwx--- 2013-10-04T15:26:48.000Z     8.3 KB thunderbird.xpm
================================================================================
TOTAL:    353.3 KB 6 Files
</pre>
 
===Directories===
In an object store there are only buckets and keys (key=value store).  This means that your traditional POSIX directory structure is only emulated using the UNIX "<b>/</b>" character within key names.  Any key in the bucket ending in a / will be interpreted by the <code>lsobj</code> utility and website as a directory.  You can also list only subdirectories with the <code>lsobj</code> utility.
 
<pre>
$ lsobj test:foo/
foo/
-rwx--- 2013-10-04T15:27:39.000Z     0.0 b foo/
-rwx--- 2013-10-04T15:27:40.000Z     0.0 b foo/bar
================================================================================
TOTAL:      0.0 b  2 Files
</pre>
 
==mkobj==
mkobj creates buckets and directories in the Object Store.  <b>Please note that bucket names are unique in the Object Store, so you may very well get an error back that the name has already been used.</b> 
 
<pre>
$ mkobj foo
Created bucket foo.
$ lsobj
bob
foo
test
zeta
</pre>
 
You can also create directories within your buckets to provide a way to group your data.
 
<pre>
% mkobj foo:bar/
% lsobj foo
-rwx--- 2013-10-04T15:38:38.000Z     0.0 b bar/
================================================================================
TOTAL:      0.0 b  1 Files
</pre>
 
==cpobj==
Copying files to the Object Store can be done per-file or recursively both to and from the Object Store.
 
To copy a single file to the Object Store you can use <code>cpobj</code> and specify a bucket with a trailing <code>:</code> (with an optional additional path).
 
<pre>
% cpobj test.png foo:
100% |##############################################################################################|
% lsobj foo
-rwx--- 2013-10-04T15:38:38.000Z     0.0 b bar/
-rwx--- 2013-10-07T20:06:48.000Z   18.3 KB test.png
================================================================================
TOTAL:    18.3 KB 2 Files
</pre>
 
To copy a directory of files you will need to use the <code>-r</code> or <code>--recursive</code> flags.
 
<pre>
% lsobj foo
================================================================================
TOTAL:      0.0 b  0 Files
% cpobj -r /tmp/stuff foo:stuff
100% |##############################################################################################|
100% |##############################################################################################|
100% |##############################################################################################|
% lsobj foo
-rwx--- 2013-10-08T00:13:52.000Z     0.0 b stuff/
-rwx--- 2013-10-08T00:13:55.000Z   26.0 KB stuff/WindowsSecurity.png
-rwx--- 2013-10-08T00:13:54.000Z   226.5 KB stuff/changepass.tiff
-rwx--- 2013-10-08T00:13:55.000Z   18.3 KB stuff/test.png
================================================================================
TOTAL:    270.8 KB 9 Files
</pre>
 
==rmobj==
You can delete your buckets and keys with <code>rmobj</code>.  It can take a bucket and work recursively, asking you to delete all the files and the bucket itself.  It can also just delete specific files in a bucket when given on the command line.
 
<pre>
% rmobj -r foo
Are you sure you want to remove all the contents of the bucket 'foo'? [yes/no] yes
Do you want to remove the bucket 'foo'? [yes/no] no
</pre>
 
<pre>
% lsobj foo
-rwx--- 2013-10-09T18:44:20.000Z     1.0 KB setup.cfg
-rwx--- 2013-10-09T18:44:17.000Z   781.0 b setup.py
-rwx--- 2013-10-09T18:44:09.000Z   289.0 b test-requirements.txt
================================================================================
TOTAL:      2.0 KB 3 Files
% rmobj foo:setup.cfg foo:setup.py
% lsobj foo
-rwx--- 2013-10-09T18:44:09.000Z   289.0 b test-requirements.txt
================================================================================
TOTAL:    289.0 b  1 Files
</pre>
 
You can also remove directories within a bucket.  To do so, you will need to pass the <code>-r</code> flag.  This will prompt you for the removal of every key under that directory unless the <code>-f</code> flag is passed as well.
<pre>
% lsobj foo
-rwx--- 2013-10-10T21:58:17.000Z     0.0 b research/
-rwx--- 2013-10-10T21:58:17.000Z     0.0 b research/papers/
-rwx--- 2013-10-10T21:58:18.000Z     0.0 b research/papers/paper1.tex
-rwx--- 2013-10-10T21:58:18.000Z     0.0 b research/papers/paper2.tex
-rwx--- 2013-10-10T21:58:18.000Z     0.0 b research/posters/
-rwx--- 2013-10-10T21:58:18.000Z     0.0 b research/posters/poster1.tex
-rwx--- 2013-10-10T21:58:18.000Z     0.0 b research/posters/poster2.tex
================================================================================
TOTAL:      0.0 b  7 Files
% rmobj -rf foo:research/posters/
% lsobj foo
-rwx--- 2013-10-10T21:58:17.000Z     0.0 b research/
-rwx--- 2013-10-10T21:58:17.000Z     0.0 b research/papers/
-rwx--- 2013-10-10T21:58:18.000Z     0.0 b research/papers/paper1.tex
-rwx--- 2013-10-10T21:58:18.000Z     0.0 b research/papers/paper2.tex
================================================================================
TOTAL:      0.0 b  4 Files
</pre>
 
==chobj==
You can change the ACL on a specific key(s) and specify multiple canned ACL policies in a single command.
 
<pre>
% chobj -p liam:FULL_CONTROL foo:test-requirements.txt</pre>

Revision as of 18:57, 29 September 2014

This is a set of console utilities that are available on all UMIACS-supported Linux (RHEL5, RHEL6, Ubuntu) machines. They are provided to support our Ceph Object Stores so that our users may interact with them on the command line. In addition, each Ceph Object Store in UMIACS has a fully-functional web application that allows access to your storage from a web browser.

For UMIACS-supported linux machines, these utilities are already installed and should be available in the standard PATH.

For more information see the GitLab page for umobj.

Retrieved from "https://wiki.umiacs.umd.edu/umiacs/index.php?title=UMobj&oldid=6300"