Rclone

Rclone is a command line program useful for syncing files and directories. Its functionality is similar to rsync, but has additional capabilities that support cloud storage services such as Dropbox, Open Drive, Amazon S3, and many more.

Below are directions on how to remote setup rclone on a headless machine. If you would like to set up specific Google Drive or Dropbox remotes, you can go directly to the section labeled "Google Drive" or "Dropbox".

Remote Setup on A Headless System

To remote setup rclone, you will first have to SSH into one of UMIACS' hosts and load the rclone module by typing module load rclone. If this does not work, check that the rclone module is available by running the command module avail rclone. If the module is not available, SSH to a host where it is.

If you are instead installing rclone on a system that you have administrative privileges on, downloads are available at https://rclone.org/downloads/.

Once you have loaded the module or installed the software, there are two ways to remote configure rclone. The first is by copying the rclone config file, and the second is by using rclone authorize.

Rclone configuration file

To copy the rclone config file, first configure rclone on your desktop machine. This can be achieved by running the command rclone config to set up the file and then finding the location of the configuration file using the command rclone config file, which should yield output similar to what is below:

$ rclone config file
Configuration file is stored at:
/home/user/.rclone.conf

This file can then be transferred to the remote box using scp, copy/paste, ftp, sftp, etc. The command rclone config file can also be run on the remote box to find the correct location to which the file should be moved.

Rclone authorize

To configure using rclone authorize instead, run the command Remote config on your headless machine, and then select the option for working on a headless machine by answering the first prompt with an n. The resulting output should look like this:

Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> n

For this to work, you will need rclone available on a machine that has a web browser available.
Execute the following on your machine:
	rclone authorize "amazon cloud drive"
Then paste the result below:
result>

Next, on your main desktop machine, run the command rclone authorize "amazon cloud drive". Follow the link that the output directs you to; you should now have received a secret token on your desktop machine console, in a message similar to the one below:

rclone authorize "amazon cloud drive"
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Got code
Paste the following into your remote machine --->
SECRET_TOKEN
<---End paste

Go back to your headless machine and paste the secret message on the console where it says result>. Finally, approve the token by answering the prompts that follow:

result> SECRET_TOKEN
--------------------
[acd12]
client_id = 
client_secret = 
token = SECRET_TOKEN
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d>

If there are any issues with the remote setup, visit https://rclone.org/remote_setup/ for more information.

Google Drive

In order to sync files to Google Drive on Linux systems, use of the command line program Rclone is necessary. The first step is to SSH into one of UMIACS' hosts with X11 forwarding enabled (use the -XY flag) or (if the host is a workstation) log in over GUI and launch the terminal. If Rclone is already in the module tree, then load the Rclone module using:

module load rclone

If the module is NOT installed on the machine, install it from here: https://rclone.org/downloads/

Once installed, load into it using the previous command.

To make a new remote, run the command rclone config. This should produce output similar to what is below:

n) New remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
n/r/c/s/q> 

Choose to make a new remote, and give it a name. There should then be prompt asking for the type of storage to configure. Input drive for Google Drive.

Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / 1Fichier
   \ "fichier"
 2 / Alias for an existing remote
   \ "alias"
 ...
 ...
37 / premiumize.me
   \ "premiumizeme"
38 / seafile
   \ "seafile"
Storage> drive

When prompted for your Google Application Client Id and Secret, it is okay to leave those blank. Rclone will then prompt for a scope that it should use when requesting access from your drive, which you can provide by selecting one of the numbers on the screen. The suggested value for full operation is 1.

Google Application Client Id
Setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a string value. Press Enter for the default ("").
client_id>
OAuth Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>
Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Full access all files, excluding Application Data Folder.
   \ "drive"
 2 / Read-only access to file metadata and file contents.
   \ "drive.readonly"
   / Access to files created by rclone only.
 3 | These are visible in the drive website.
   | File authorization is revoked when the user deauthorizes the app.
   \ "drive.file"
   / Allows read and write access to the Application Data folder.
 4 | This is not visible in the drive website.
   \ "drive.appfolder"
   / Allows read-only access to file metadata but
 5 | does not allow any access to read or download file content.
   \ "drive.metadata.readonly"
scope> 1

You can leave the root_folder_id and the service_account_file blank. You will then be prompted to edit the advanced config. Enter the default value unless the application requires further customization.

Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n>

You will then be prompted to Use auto config. Enter Y.

Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n>

The window will then prompt for sign in with a Google account. Sign into your UMD account using your $USERNAME@umd.edu email. You will then be brought to the UMD CAS Dual Factor Authentication Screen. Sign in with your normal credentials and dual factor authenticate. Allow rclone to access your Google Account.

You will then be prompted to Configure this as a team drive. Enter the default value unless the application requires further customization.

Configure this as a team drive?
y) Yes
n) No (default)
y/n>

Finally, confirm your selections if they are correct, edit them if they are incorrect.

--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

Common Use

List directories in top level of your drive

rclone lsd $REMOTENAME$:

List all the files in your drive

rclone ls $REMOTENAME$:

To copy a local directory to a drive directory called backup

rclone copy /home/source $REMOTENAME$:$NEWBACKUPNAME$

More commands and usage can be found at: https://rclone.org/drive/

Creating a cronjob

A cronjob will allow you to automate backups. To create a cronjob for an Rclone command, first you must find the path of Rclone. This can be done using the which command.

[user@workstation ~]$ which rclone
/opt/common/rclone-v1.53.4-linux-amd64/rclone

Then use crontab -e to edit your crontab. Once in the crontab, you can input your Rclone command using the command path found previously. An example of an Rclone cronjob that would occur every hour on the hour would be:

0 * * * * /opt/common/rclone-v1.53.4-linux-amd64/rclone copy /$PATHTOBEBACKEDUP$/ $REMOTENAME$:$BACKUPNAME$

Dropbox

These instructions assume that you have SSH'd into one of UMIACS' hosts and have successfully been able to load the rclone module using the command module load rclone, or installed rclone yourself on a system that you have administrative privileges on. If this is not the case, please do so before proceeding with the Dropbox remote setup.

First, make a new remote by running the command rclone config. This should produce output similar to what is below:

n) New remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
n/r/c/s/q> 

Choose to make a new remote, and give it a name. The setup will then prompt you to choose a type of storage to configure. You can either choose the number from the options that corresponds to "Dropbox", or simply type in dropbox. When prompted for your Dropbox App Key and Secret, it is okay to leave those blank.

The console will then redirect you to a browser page that will prompt you to enter a code displayed on the console and authorize rclone for access. Once you have followed the instructions on the webpage, the console will display your credentials and ask for approval to confirm the remote.