S3Clients: Difference between revisions
(→mc) |
No edit summary |
||
| (11 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
Many popular S3 desktop clients can be used to access the UMIACS Object Store. These tools complement the [[UMobj]] command line utilities and the built-in web interface by providing integration with the native file explorer on your desktop machine. | Many popular S3 desktop clients can be used to access the [[OBJ | UMIACS Object Store]]. These tools complement the [[UMobj]] command line utilities and the built-in web interface by providing integration with the native file explorer on your desktop machine. | ||
'''Note''': Many of these clients have features that are not supported by our Object Store in UMIACS. One prominent example of this is permissions. We suggest you instead manage permissions from the [https://obj.umiacs.umd.edu/obj built-in web application] for the Object Store. | '''Note''': Many of these clients have features that are not supported by our Object Store in UMIACS. One prominent example of this is permissions. We suggest you instead manage permissions from the [https://obj.umiacs.umd.edu/obj built-in web application] for the Object Store. | ||
| Line 6: | Line 6: | ||
==Cyberduck== | ==Cyberduck== | ||
https://cyberduck.io/ | |||
This is a free Windows and Mac S3 browser (it is however nagware that asks for a donation). It supports our S3 Object | This is a free Windows and Mac S3 browser (it is however nagware that asks for a donation). It supports our S3 Object Store using the "S3 (Amazon Simple Storage Service)" drop down menu choice in the add bookmark dialog. | ||
[[Image:Cyberduck.png|400px]] | [[Image:Cyberduck.png|400px]] | ||
* '''Server''' - This | The following fields are required: | ||
* '''Server''' - This is your object store (<code>obj.umiacs.umd.edu</code>) | |||
* '''Access Key ID''' - This is your access key as provided to you in the object store | * '''Access Key ID''' - This is your access key as provided to you in the object store | ||
* '''Password''' - This is your secret key as provided to you in the object store | * '''Password''' - This is your secret key as provided to you in the object store | ||
You will be prompted for your secret key when you connect and may choose to save the password. | You will be prompted for your secret key when you connect and may choose to save the password. | ||
==WinSCP== | |||
* https://winscp.net/eng/index.php | |||
This is a free Windows file transfer application. It supports our S3 Object Store using the "Amazon S3" drop down menu choice under File protocol when logging in. | |||
[[Image:WinSCPS3.png|400px]] | |||
The following fields are required: | |||
* '''Host name''' - This is your object store (<code>obj.umiacs.umd.edu</code>) | |||
* '''Access key ID''' - This is your access key as provided to you in the object store | |||
* '''Secret access key''' - This is your secret key as provided to you in the object store | |||
==Transmit== | ==Transmit== | ||
* http://panic.com/transmit/ | * http://panic.com/transmit/ | ||
This is a paid file transfer application for Mac. It supports our S3 Object | This is a paid file transfer application for Mac. It supports our S3 Object Store using the "S3" menu choice after clicking the plus sign to add a favorite. | ||
[[Image:Transmit.png|400px]] | [[Image:Transmit.png|400px]] | ||
* '''Server''' - This | The following fields are required: | ||
* '''Server''' - This is your object store (<code>obj.umiacs.umd.edu</code>) | |||
* '''Access Key ID''' - This is your access key as provided to you in the object store | * '''Access Key ID''' - This is your access key as provided to you in the object store | ||
* '''Secret''' - This is your secret key as provided to you in the object store | * '''Secret''' - This is your secret key as provided to you in the object store | ||
| Line 33: | Line 47: | ||
==s3cmd== | ==s3cmd== | ||
Command line client for accessing S3-like services. | Command line client for accessing S3-like services. | ||
* http://s3tools.org/s3cmd | * http://s3tools.org/s3cmd | ||
You need to configure a file like <code>~/. | You need to configure a file like <code>~/.s3cfg</code> that looks like the following with your ACCESS_KEY and SECRET_KEY substituted. | ||
<pre> | <pre> | ||
[default] | [default] | ||
access_key = ACCESS_KEY | access_key = <ACCESS_KEY> | ||
host_base = obj.umiacs.umd.edu | host_base = obj.umiacs.umd.edu | ||
host_bucket = %(bucket)s.obj.umiacs.umd.edu | host_bucket = %(bucket)s.obj.umiacs.umd.edu | ||
secret_key = SECRET_KEY | secret_key = <SECRET_KEY> | ||
use_https = True | use_https = True | ||
</pre> | </pre> | ||
==mc== | ==mc== | ||
The | The MinIO Client is a comprehensive single binary (Go) command line client for cloud based storage services. | ||
* https://min.io/download | |||
You can run this client on supported UMIACS systems through adding it via our software [[Modules|module]]. | |||
<pre>module add mc</pre> | |||
You will need to setup a cloud provider for Obj by running the following command (substituting in your actual ACCESS_KEY and SECRET_KEY for your personal account or [[OBJ#LabGroups | LabGroup]] in the [https://obj.umiacs.umd.edu/obj/user/ Object Store]). | |||
<pre> | <pre> | ||
mc config host add obj http://obj.umiacs.umd.edu <ACCESS_KEY> <SECRET_KEY> | |||
</pre> | </pre> | ||
You | You can see what host(s) you have configured with the command <code>mc config host ls</code>. | ||
<pre> | <pre> | ||
mc config host | $ mc config host ls | ||
... | |||
obj | |||
URL : http://obj.umiacs.umd.edu | |||
AccessKey : (redacted) | |||
SecretKey : (redacted) | |||
API : s3v4 | |||
Path : auto | |||
... | |||
</pre> | </pre> | ||
| Line 80: | Line 103: | ||
derek/derek_support/umiacs-49168.log | derek/derek_support/umiacs-49168.log | ||
</pre> | </pre> | ||
The full MinIO Client documentation can be found here: https://min.io/docs/minio/linux/reference/minio-mc.html. | |||
==rclone== | |||
Command line client for accessing S3-like services and much more. | |||
* https://rclone.org/downloads | |||
We maintain rclone documentation on a separate page found [[Rclone | here]]. | |||
Latest revision as of 16:48, 25 October 2024
Many popular S3 desktop clients can be used to access the UMIACS Object Store. These tools complement the UMobj command line utilities and the built-in web interface by providing integration with the native file explorer on your desktop machine.
Note: Many of these clients have features that are not supported by our Object Store in UMIACS. One prominent example of this is permissions. We suggest you instead manage permissions from the built-in web application for the Object Store.
Graphical Clients
Cyberduck
This is a free Windows and Mac S3 browser (it is however nagware that asks for a donation). It supports our S3 Object Store using the "S3 (Amazon Simple Storage Service)" drop down menu choice in the add bookmark dialog.
The following fields are required:
- Server - This is your object store (
obj.umiacs.umd.edu) - Access Key ID - This is your access key as provided to you in the object store
- Password - This is your secret key as provided to you in the object store
You will be prompted for your secret key when you connect and may choose to save the password.
WinSCP
This is a free Windows file transfer application. It supports our S3 Object Store using the "Amazon S3" drop down menu choice under File protocol when logging in.
The following fields are required:
- Host name - This is your object store (
obj.umiacs.umd.edu) - Access key ID - This is your access key as provided to you in the object store
- Secret access key - This is your secret key as provided to you in the object store
Transmit
This is a paid file transfer application for Mac. It supports our S3 Object Store using the "S3" menu choice after clicking the plus sign to add a favorite.
The following fields are required:
- Server - This is your object store (
obj.umiacs.umd.edu) - Access Key ID - This is your access key as provided to you in the object store
- Secret - This is your secret key as provided to you in the object store
These settings can be saved as a favorite for easy access. Transmit also allows you to mount your Obj buckets as local disks, which will support easy drag-and-drop of files.
Command Line Clients
s3cmd
Command line client for accessing S3-like services.
You need to configure a file like ~/.s3cfg that looks like the following with your ACCESS_KEY and SECRET_KEY substituted.
[default] access_key = <ACCESS_KEY> host_base = obj.umiacs.umd.edu host_bucket = %(bucket)s.obj.umiacs.umd.edu secret_key = <SECRET_KEY> use_https = True
mc
The MinIO Client is a comprehensive single binary (Go) command line client for cloud based storage services.
You can run this client on supported UMIACS systems through adding it via our software module.
module add mc
You will need to setup a cloud provider for Obj by running the following command (substituting in your actual ACCESS_KEY and SECRET_KEY for your personal account or LabGroup in the Object Store).
mc config host add obj http://obj.umiacs.umd.edu <ACCESS_KEY> <SECRET_KEY>
You can see what host(s) you have configured with the command mc config host ls.
$ mc config host ls ... obj URL : http://obj.umiacs.umd.edu AccessKey : (redacted) SecretKey : (redacted) API : s3v4 Path : auto ...
You can then use the normal mc commands like the following to list the contents of a bucket.
$ mc ls obj/iso [2017-02-10 16:45:04 EST] 3.5GiB rhel-server-7.3-x86_64-dvd.iso [2017-02-13 12:21:33 EST] 4.0GiB rhel-workstation-7.3-x86_64-dvd.iso
There is also the ability to search for file globs of specific files using the find sub-command for mc.
$ mc find derek/derek_support --name "*.log" derek/derek_support/mds_20170918/ceph-mds.objmds01.log derek/derek_support/satellite.log derek/derek_support/umiacs-49168.log
The full MinIO Client documentation can be found here: https://min.io/docs/minio/linux/reference/minio-mc.html.
rclone
Command line client for accessing S3-like services and much more.
We maintain rclone documentation on a separate page found here.