UMobj: Difference between revisions

From UMIACS
Jump to navigation Jump to search
No edit summary
 
(41 intermediate revisions by 5 users not shown)
Line 1: Line 1:
This is a set of console utilities that are available on all UMIACS supported machines.   They are provided to support our Ceph Object Stores so our users may interact with them on the command line.  In addition each Ceph Object Store in UMIACS has fully functional web application that allows access to your storage from any web browser.
The UMobj suite of utilities provide command-line access to the [https://obj.umiacs.umd.edu/obj UMIACS Object Store].  UMobj is preinstalled on all UMIACS-supported RHEL8 machines. For other operating systems or non UMIACS-supported hosts, we encourage use of one of the many [[S3Clients#Command_Line_Clients | third-party command line clients]] that exist.


For UMIACS support machines these utilities are installed at <code>/opt/UMobj/bin</code> however they should already be available in your path.
==When to use UMobj==
Use umobj if:
* you have a large number of files to upload (hundreds or thousands of files)
* you are uploading large files (files greater than 4GB in size)


==Prerequisites==
==Setup==
We highly recommend setting a few environmental variables containing your credentials for
convenience.  When logged into the Object Store web interface (see list above), you can
find these credentials on the user page.  E.g. https://obj.umiacs.umd.edu/obj/user/


To use any of these utilities you either have to pass in on the command line your access and secret keys or you need to set some appropriate environmental variables.  You can obtain your access and secret keys (which are different from your username and password) by visiting the appropriate Ceph Object Store.  The default store is https://obj.umiacs.umd.edu/UMobjstore/user.
For example, if you use the <tt>bash</tt> shell, you can add something like the following to your
 
<tt>.bashrc</tt> or <tt>.bash_profile</tt>.
In that page will be your access and secret keys for that Ceph Object Store.  You can then set for example in bash the following environment variables.
 
  export OBJ_ACCESS_KEY_ID="31sdfadDFAHFDN+344qOEIS"
  export OBJ_SECRET_ACCESS_KEY="NDSMK3233adfahadflkkPDSH092DSJKDKDJKFDLSFLNK"
 
If you are not using the default Ceph Object Store you can tell the utilities to use an alternative server.
 
  export OBJ_SERVER="gembox.cbcb.umd.edu"
 
You can optionally use the <code>--access_key</code>, <code>--secret_key</code> and <code>--server</code> switches to the utilities.
 
==lsobj==
 
To list the object store you can use the <b>lsobj</b> command.  If given without a argument it will list your buckets (this will only list buckets that you created). 
 
<pre>
$ lsobj
bob
test
zeta
</pre>
 
You can then give it a bucket to list the contents within the bucket (if you were granted access to a bucket you didn't create you can also use it).  This will list all the keys in your bucket which can take a long time.


<pre>
<pre>
$ lsobj test
export OBJ_ACCESS_KEY_ID="<ACCESS_KEY>"
-rwx--- 2013-10-04T15:25:09.000Z     1.8 KB UMIACSCA.pem
export OBJ_SECRET_ACCESS_KEY="<SECRET_KEY>"
-rwx--- 2013-10-04T15:25:24.000Z   311.4 KB cover.jpg
export OBJ_SERVER="obj.umiacs.umd.edu"
-rwx--- 2013-10-04T15:27:39.000Z     0.0 b foo/
-rwx--- 2013-10-04T15:27:40.000Z     0.0 b foo/bar
-rwx--- 2013-10-04T15:25:32.000Z   31.8 KB screenshot.jpg
-rwx--- 2013-10-04T15:26:48.000Z     8.3 KB thunderbird.xpm
================================================================================
TOTAL:    353.3 KB 6 Files
</pre>
</pre>


===Directories===
Or, in tcsh, you can do the following or add it into your <tt>.tcshrc</tt>.
In a object store there are only buckets and keys (key=value store).  This means your traditional posix directory structure is somewhat faked out using the UNIX "<b>/</b>" character.  Any key in the bucket ending in a / will be interpreted by the <code>lsobj</code>utility and website as a directory.   You can list only subdirectories with the <code>lsobj</code> utility.


<pre>
<pre>
$ lsobj test:foo/
setenv OBJ_ACCESS_KEY_ID "<ACCESS_KEY>"
foo/
setenv OBJ_SECRET_ACCESS_KEY "<SECRET_KEY>"
-rwx--- 2013-10-04T15:27:39.000Z     0.0 b foo/
setenv OBJ_SERVER "obj.umiacs.umd.edu"
-rwx--- 2013-10-04T15:27:40.000Z     0.0 b foo/bar
================================================================================
TOTAL:      0.0 b  2 Files
</pre>
</pre>


==mkobj==
(substituting in your actual ACCESS_KEY and SECRET_KEY for your personal account or [[OBJ#LabGroups | LabGroup]] in the [https://obj.umiacs.umd.edu/obj/user/ Object Store]).
Creates buckets and directories in the object store.  <b>Please note that bucket names are unique in the object store, so you may very well get an error back that the name is already used.</b> 
 
<pre>
$ mkobj foo
Created bucket foo.
$ lsobj
bob
foo
test
zeta
</pre>
 
You can also create directories within your buckets to provide a way to group your data.
 
<pre>
% mkobj foo:bar/
% lsobj foo
-rwx--- 2013-10-04T15:38:38.000Z     0.0 b bar/
================================================================================
TOTAL:      0.0 b  1 Files
</pre>
 
==cpobj==
Copying files to the Object Storage can be done per file or recursively both to and from the object store.
 
To copy a single file to the Object Store you can use <code>cpobj</code> and specify a bucket with a trailing <code>:</code> (with an optional additional path) .
 
<pre>
% cpobj test.png foo:
100% |##############################################################################################|
% lsobj foo
-rwx--- 2013-10-04T15:38:38.000Z     0.0 b bar/
-rwx--- 2013-10-07T20:06:48.000Z   18.3 KB test.png
================================================================================
TOTAL:    18.3 KB 2 Files
</pre>
 
To copy a directory of files you will need to use the <code>-r</code> or <code>--recursive</code> flags.
 
<pre>
% lsobj foo
================================================================================
TOTAL:     0.0 b  0 Files
% cpobj -r /tmp/stuff foo:stuff
100% |###################################################################################################################|
100% |###################################################################################################################|
100% |###################################################################################################################|
% lsobj foo
-rwx--- 2013-10-08T00:13:52.000Z     0.0 b stuff/
-rwx--- 2013-10-08T00:13:55.000Z   26.0 KB stuff/WindowsSecurity.png
-rwx--- 2013-10-08T00:13:54.000Z   226.5 KB stuff/changepass.tiff
-rwx--- 2013-10-08T00:13:55.000Z   18.3 KB stuff/test.png
================================================================================
TOTAL:    270.8 KB 9 Files
</pre>


==rmobj==
==Detailed Usage==
For an example of how to use UMobj, please see [[UMobj/Example]].


==chobj==
For complete usage information, please see the [[GitLab]] page for [https://gitlab.umiacs.umd.edu/staff/umobj/blob/master/README.md#umobj umobj].

Latest revision as of 15:57, 10 May 2024

The UMobj suite of utilities provide command-line access to the UMIACS Object Store. UMobj is preinstalled on all UMIACS-supported RHEL8 machines. For other operating systems or non UMIACS-supported hosts, we encourage use of one of the many third-party command line clients that exist.

When to use UMobj

Use umobj if:

  • you have a large number of files to upload (hundreds or thousands of files)
  • you are uploading large files (files greater than 4GB in size)

Setup

We highly recommend setting a few environmental variables containing your credentials for convenience. When logged into the Object Store web interface (see list above), you can find these credentials on the user page. E.g. https://obj.umiacs.umd.edu/obj/user/

For example, if you use the bash shell, you can add something like the following to your .bashrc or .bash_profile. 

export OBJ_ACCESS_KEY_ID="<ACCESS_KEY>"
export OBJ_SECRET_ACCESS_KEY="<SECRET_KEY>"
export OBJ_SERVER="obj.umiacs.umd.edu"

Or, in tcsh, you can do the following or add it into your .tcshrc. 

setenv OBJ_ACCESS_KEY_ID "<ACCESS_KEY>"
setenv OBJ_SECRET_ACCESS_KEY "<SECRET_KEY>"
setenv OBJ_SERVER "obj.umiacs.umd.edu"

(substituting in your actual ACCESS_KEY and SECRET_KEY for your personal account or LabGroup in the Object Store).

Detailed Usage

For an example of how to use UMobj, please see UMobj/Example.

For complete usage information, please see the GitLab page for umobj.

Retrieved from "https://wiki.umiacs.umd.edu/umiacs/index.php?title=UMobj&oldid=11795"