S3Clients

From UMIACS
Jump to navigation Jump to search

Many popular S3 desktop clients can be used to access the UMIACS Object Store. These tools complement the UMobj command line utilities and the built-in web interface by providing integration with the native file explorer on your desktop machine.

Note: Many of these clients have features that are not supported by our Object Store in UMIACS. One prominent example of this is permissions. We suggest you instead manage permissions from the built-in web application for the Object Store.

Graphical Clients

Cyberduck

https://cyberduck.io

This is a free Windows and Mac S3 browser (it is however nagware that asks for a donation). It supports our S3 Object Store using the "S3 (Amazon Simple Storage Service)" drop down menu choice in the add bookmark dialog.

Setting up Object Store access in Cyberduck

The following fields are required:

  • Server - This is your object store (obj.umiacs.umd.edu)
  • Access Key ID - This is your access key as provided to you in the object store
  • Password - This is your secret key as provided to you in the object store

You will be prompted for your secret key when you connect and may choose to save the password.

WinSCP

This is a free Windows file transfer application. It supports our S3 Object Store using the "Amazon S3" drop down menu choice under File protocol when logging in.

Setting up Object Store access in WinSCP

The following fields are required:

  • Host name - This is your object store (obj.umiacs.umd.edu)
  • Access key ID - This is your access key as provided to you in the object store
  • Secret access key - This is your secret key as provided to you in the object store

Transmit

This is a paid file transfer application for Mac. It supports our S3 Object Store using the "S3" menu choice after clicking the plus sign to add a favorite.

Setting up Object Store access in Transmit

The following fields are required:

  • Server - This is your object store (obj.umiacs.umd.edu)
  • Access Key ID - This is your access key as provided to you in the object store
  • Secret - This is your secret key as provided to you in the object store

These settings can be saved as a favorite for easy access. Transmit also allows you to mount your Obj buckets as local disks, which will support easy drag-and-drop of files.

Command Line Clients

s3cmd

Command line client for accessing S3-like services.

You need to configure a file like ~/.s3cfg that looks like the following with your ACCESS_KEY and SECRET_KEY substituted.

[default]
access_key = <ACCESS_KEY>
host_base = obj.umiacs.umd.edu
host_bucket = %(bucket)s.obj.umiacs.umd.edu
secret_key = <SECRET_KEY>
use_https = True

mc

The AIStor Client (formerly known as the MinIO Client) is a comprehensive single binary (Go) command line client for cloud based storage services.

You can run this client on supported UMIACS systems through adding it via our software module.

module add mc

You will need to setup a cloud provider for Obj by running the following command (substituting in your actual ACCESS_KEY and SECRET_KEY for your personal account or LabGroup in the Object Store). The obj example name is configurable and used to differentiate between multiple different cloud based storage systems that you configure.

Alias

mc alias set obj https://obj.umiacs.umd.edu <ACCESS_KEY> <SECRET_KEY>

You can see what host(s) you have configured with the command mc config host ls.

$ mc alias list
...
obj
  URL       : https://obj.umiacs.umd.edu
  AccessKey : (redacted)
  SecretKey : (redacted)
  API       : s3v4
  Path      : auto
...

You can then use the normal mc commands note that the alias name you have used is required to differentiate between the local file system.

Buckets

You can create a bucket mc mb obj/my_bucket_name. Buckets in the Object Store share a single name space and you will get a friendly error Error response code BucketAlreadyExists. which you will need to choose a different bucket name.

Listing

Listing keys inside a bucket.

$ mc ls obj/iso/RedHat-E10
[2026-07-29 10:56:46 EDT]     0B EC-53 /
[2026-07-29 10:56:46 EDT] 9.7GiB EC-53 rhel-10.2-aarch64-dvd.iso
[2026-07-29 11:00:18 EDT]  10GiB EC-53 rhel-10.2-x86_64-dvd.iso

Copy

Copying data into a bucket from the local file (eg. rhel-10.0-x86_64-dvd.iso in this case).

$ mc cp rhel-10.0-x86_64-dvd.iso obj/iso/RedHat-E10
...-10.0-x86_64-dvd.iso:        █████████████████████████████████████████████████████████████████████████ 100.0% 7.9 GiB   90 MiB/s

Mirror

Often users will just want to mirror a local set of files. Note that it will mirror recursively the files in the directory but you will need to be explict if you about the destination path in this case to ensure that you get the directory created in the destination.

$ ls -l test/
total 16
-rw-rw-r--. 1 dyarnell dyarnell 8688 Aug 19 15:17 cpuinfo
-rw-rw-r--. 1 dyarnell dyarnell   23 Aug 19 15:17 uptime
$ mc mirror /tmp/test obj/staff-test/test
                                ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░   0.0% 0 B       0 B/s
$ mc ls obj/staff-test/test/
[2026-08-19 15:18:45 EDT] 8.5KiB STANDARD cpuinfo
[2026-08-19 15:18:45 EDT]    23B STANDARD uptime

Find

There is also the ability to search for file globs of specific files using the find sub-command for mc.

$ mc find dyarnell/dyarnell_support --name "*.log"
dyarnell/dyarnell_support/mds_20170918/ceph-mds.objmds01.log
dyarnell/dyarnell_support/satellite.log
dyarnell/dyarnell_support/umiacs-49168.log

The full AIStor Client documentation can be found here: https://docs.min.io/aistor/reference/cli/.

rclone

Command line client for accessing S3-like services and much more.

We maintain rclone documentation on a separate page found here.