Transferring Data to OURRstore Using Rclone

Overview

Rclone is a command-line program that helps manage files on cloud and remote systems. In particular, it is useful for transferring large sizes and amounts of files. This document details how to configure Rclone to transfer files on a local system to OURRstore, a tape archive managed by the University of Oklahoma open to all researchers at EPSCoR institutions (including the University of Kentucky).

Contents

Assumptions 

• You have already created accounts and sent tape drives to OURRstore.

• All of the files are between 1GB - 1TB in size (20 - 200 GB preferably).

• You have a SSH key pair, with the public key authorized for use on the OURRstore DTN (dtn2.oscer.ou.edu).

Setting Up an SSH Key Pair

If you need to generate a public-private key pair, run the following command to generate one. This command generates a RSA formatted key, which should be usable by almost all systems.

You will have two files; a file with no extension (the private key) and a file with a .pub extension (the public key).

On the OURRstore DTN (Data Transfer Node), you may need to create a ~/.ssh/authorized_keys file to let the system know your public key. SSH into the DTN using your OU credentials you created when setting up an account and run the following commands to create this file and set the appropriate permissions. SFTP will not work without these permissions being set correctly.

Once the authorized_keys file is created on the DTN, copy the contents of the public key on your system into it.

Configuring Rclone 

Run the following command to begin setup for the OURRstore connection. Any options not specified in the following steps can usually be skipped; just hit "Enter" to set the default value of an option.

1. Enter "n" to create a new connection.

   No remotes found - make a new one n) New remote s) Set configuration password q) Quit config n/s/q> n

2. When selecting the type of connection, select “sftp”.

3. Enter a name for the connection, dtn2.oscer.ou.edu for the host, and the username for your OSCER account if it is different from your current one.
   
   

   SSH host to connect to Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Connect to example.com \ "example.com" host> dtn2.oscer.ou.edu SSH username, leave blank for current username. Enter a string value. Press Enter for the default (""). user>

4. Enter the path for your private SSH key, usually located in .ssh.
   
   

   SSH password, leave blank to use ssh-agent. y) Yes type in my own password g) Generate random password n) No leave this optional password blank (default) y/g/n> n Raw PEM-encoded private key, If specified, will override key_file parameter. key_pem> Path to PEM-encoded private key file, leave blank or set key-use-agent to use ssh-agent. Leading '~' will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`. Enter a string value. Press Enter for the default ("") key_file> /path/to/key_file

    

5. At this point, all the other config questions can be left blank (i.e. use the default options) to finish configuring the OURRstore connection. If you haven't already, make sure you have copied your public key from the computer where you configured rclone into ~/.ssh/authorized_keys on the OURRstore DTN.

Transferring Files to OURRstore 

Run the following command to copy files to the tape archive on OURRstore. Typically, you copy files to /archive/ourrstore/<groupname>/tape_2copies/on the OURRstore DTN to back them up to tape.

You can also copy files to your OURRstore home directory with the following command.