Data Transfer from Supercomputers

You can use Rclone to transfer data from command-line environments such as supercomputers and Linux servers.

1. Install Rclone

With administrator privileges (e.g., Ubuntu):

$ sudo apt install rclone

Without administrator privileges on shared machines:

$ wget https://downloads.rclone.org/rclone-current-linux-amd64.zip
$ unzip rclone-current-linux-amd64.zip
Copy the binary file rclone from rclone-version-linux-amd64/ to your PATH.

2. Issue Nextcloud Device Password

Access ARIM-mdx Nextcloud to issue a device password.

Note

This process is required for each device (i.e., supercomputer or server) individually. It should not be reused.

1. Top-right icon > “Settings”
nextcloud-setting

2. Go “Security” > “Devices & sessions”
3. Enter App name (any name, such as supercomputer’s name), then click “Create new app password” .
nextcloud-devicepass

4. User ID and Password will be issued. These will be used in the Rclone configuration below.
nextcloud-passgen

Note

Once you close or move away from this screen, you won’t be able to see the password again. Please keep this screen open until you complete the setup, or make note of the generated password. After completing the setup, please discard the password.

2. Getting Nextcloud WebDAV URL

1. Click “File Settings” at the bottom left of any folder page
nextcloud-geturl

2. Get the WebDAV URL: https://arim.mdx.jp/nextcloud/remote.php/dav/files/{username}

3. Configuring Rclone

Configure on the computing server using the generated User ID, Password, and WebDAV URL:

$ rclone config

Answer the following questions for configuration:

  1. No remotes found, make a new one? → Enter “n” (first letter of new) ↵

  2. Enter name for new remote. → Enter any name for the connection. Example: Enter “arim” ↵

  3. Option Storage. → Select connection method. Enter “webdav” ↵

  4. Option url. → Enter the WebDAV URL obtained above ↵

  5. Option vendor. → Enter “nextcloud” ↵

  6. Option user. → Enter the generated User ID ↵

  7. Option pass. → Confirm to enter password. Enter “y” ↵

  8. Enter the password: → Enter the issued Password ↵ (password input is hidden)

  9. Confirm the password: → Re-enter the same password for confirmation ↵

  10. Option bearer_token. → Press ↵ without entering anything

  11. Edit advanced config? → Press ↵ without entering anything

  12. Configuration complete. → Press ↵ without entering anything

  13. Current remotes: → Enter “q” to quit ↵

This creates a configuration file at ~/.config/rclone/rclone.conf.

4. Basic Usage of Rclone

Display directories:

$ rclone lsd arim:

Display deeper directories:

$ rclone lsd arim:large/xxx/yyy/zzz

Copy files (from local to ARIM-mdx Nextcloud):

$ rclone copy yyy/ arim:large/xxx/yyy/

Copy files (from ARIM-mdx Nextcloud to local):

$ rclone copy arim:large/xxx/yyy yyy/

Remote mount (verify with df after mounting):

$ mkdir arim
$ rclone mount --daemon arim:large/xxx/ arim/

Note

For systems with login node resource restrictions like ISSP’s kugui supercomputer, you need to limit GO language program (Rclone) parallelism using environment variables:

$ export GOMAXPROCS=1

5. Detailed Usage

You can refer to rclone and subcommand options with these commands:

$ rclone --help
$ rclone subcommand --help

To see detailed help for each subcommand, replace the subcommand part with actual commands (e.g., copy, lsd, mount, etc.).

6. Reference