Difference between revisions of "Rclone"

From UFRC
Jump to navigation Jump to search
 
(9 intermediate revisions by 4 users not shown)
Line 5: Line 5:
 
<!--CONFIGURATION: OPTIONAL (|1}} means it's ON)-->
 
<!--CONFIGURATION: OPTIONAL (|1}} means it's ON)-->
 
|{{#vardefine:conf|}}          <!--CONFIGURATION-->
 
|{{#vardefine:conf|}}          <!--CONFIGURATION-->
−
|{{#vardefine:exe|}}            <!--ADDITIONAL INFO-->
+
|{{#vardefine:exe|1}}            <!--ADDITIONAL INFO-->
 
|{{#vardefine:job|}}            <!--JOB SCRIPTS-->
 
|{{#vardefine:job|}}            <!--JOB SCRIPTS-->
 
|{{#vardefine:policy|}}        <!--POLICY-->
 
|{{#vardefine:policy|}}        <!--POLICY-->
Line 18: Line 18:
 
{{App_Description|app={{#var:app}}|url={{#var:url}}|name={{#var:app}}}}|}}
 
{{App_Description|app={{#var:app}}|url={{#var:url}}|name={{#var:app}}}}|}}
  
−
Rclone is a command line program to manage files on cloud storage.
+
Rclone is a command line program to manage files on cloud storage, including Amazon Drive, Backblaze B2, Citrix Sharefile, Dropbox, Google Cloud Storage, Google Photos, Google Drive, Microsoft Azure Blob Storage, OpenDrive, Zoho, and more.
  
 
<!--Modules-->
 
<!--Modules-->
Line 32: Line 32:
 
{{#if: {{#var: exe}}|==Additional Information==
 
{{#if: {{#var: exe}}|==Additional Information==
  
−
WRITE_ADDITIONAL_INSTRUCTIONS_ON_RUNNING_THE_SOFTWARE_IF_NECESSARY
+
==Examples==
 +
===Add DropBox Remote===
 +
<div class="mw-collapsible mw-collapsed" style="width:70%; padding: 5px; border: 1px solid gray;">
 +
''Expand to see an example to add DropBox Remote.''
 +
<div class="mw-collapsible-content" style="padding: 5px;">
  
 +
I've verified that the https://rclone.org/dropbox/ and https://rclone.org/drive/ procedures worked on HiPerGator. Here's dropbox remote setup:
 +
 +
# Start an Open OnDemand Desktop session: https://ood.rc.ufl.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/hipergator/session_contexts/new
 +
# Once the session starts connect to it and open a terminal by clicking on its icon.
 +
# Use 'New > Open Tab' menu option or 'Ctrl+Shift+T' key combo in the terminal to make sure you have two tabs to start a web browser
 +
#*Tab1:
 +
#**<pre>module load rclone</pre>
 +
#*Tab2:
 +
#**<pre>module load ubuntu firefox</pre>
 +
#Follow the https://rclone.org/dropbox/ procedure. Note that the ubuntu module has chrome and firefox browsers, so the web authentication part of defining a remote can be done by pasting the generated link into it.
 +
 +
Tab1:
 +
<pre>
 +
$ rclone config
 +
Current remotes:
 +
 +
Name                Type
 +
====                ====
 +
gdrive              drive
 +
 +
e) Edit existing remote
 +
n) New remote
 +
d) Delete remote
 +
r) Rename remote
 +
c) Copy remote
 +
s) Set configuration password
 +
q) Quit config
 +
e/n/d/r/c/s/q> n
 +
name> dbox
 +
Option Storage.
 +
Type of storage to configure.
 +
Choose a number from below, or type in your own value.
 +
1 / 1Fichier
 +
  \ (fichier)
 +
...
 +
12 / Dropbox
 +
  \ (dropbox)
 +
...
 +
46 / seafile
 +
  \ (seafile)
 +
Storage> dropbox
 +
Option client_id.
 +
OAuth Client Id.
 +
Leave blank normally.
 +
Enter a value. Press Enter to leave empty.
 +
client_id>
 +
Option client_secret.
 +
OAuth Client Secret.
 +
Leave blank normally.
 +
Enter a value. Press Enter to leave empty.
 +
client_secret>
 +
Edit advanced config?
 +
y) Yes
 +
n) No (default)
 +
y/n> n
 +
Use auto config?
 +
* Say Y if not sure
 +
* Say N if you are working on a remote or headless machine
 +
 +
y) Yes (default)
 +
n) No
 +
y/n> y
 +
2022/09/28 15:29:19 NOTICE: If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=2B9dREDACTED_gzfrZg
 +
2022/09/28 15:29:19 NOTICE: Log in and authorize rclone for access
 +
 +
</pre>
 +
<At this point copy the link above and paste it into 'firefox' URL bar. Log in and authorize the connection. Return to Tab1>
 +
<pre>
 +
2022/09/28 15:29:19 NOTICE: Waiting for code...
 +
2022/09/28 15:30:47 NOTICE: Got code
 +
--------------------
 +
[dbox]
 +
type = dropbox
 +
token = {"access_token":"sl.BQJDjte4UkfWREDACTEDyRcXqyi-oAcmScjT_-mn3b81BAc-pLgPcX5PtDRRREDACTEDwIaulrxz5ncFYrLQyummJpG5","token_type":"bearer","refresh_token":"93j-33PYjZgAAAAAAAREDACTED_NwenaxE3Y5VBbLD93g6r","expiry":"2022-09-28T19:30:48.69886106-04:00"}
 +
--------------------
 +
y) Yes this is OK (default)
 +
e) Edit this remote
 +
d) Delete this remote
 +
y/e/d> y
 +
Current remotes:
 +
 +
Name                Type
 +
====                ====
 +
dbox                dropbox
 +
gdrive              drive
 +
 +
e) Edit existing remote
 +
n) New remote
 +
d) Delete remote
 +
r) Rename remote
 +
c) Copy remote
 +
s) Set configuration password
 +
q) Quit config
 +
</pre>
 +
Test the Dropbox remote:
 +
<pre>
 +
$ rclone lsd dbox:
 +
          -1 2022-09-28 15:31:03        -1 GCP 2019 Photos
 +
          -1 2022-09-28 15:31:03        -1 SKITE
 +
          -1 2022-09-28 15:31:03        -1 Sent files
 +
          -1 2022-09-28 15:31:03        -1 UFIT-RC-SysSecPlan
 +
</pre>
 +
 +
To copy a local file/directory to a remote dropbox location:
 +
<pre>
 +
$ rclone copy /path/to/local/source remote:/path/to/remote/
 +
</pre>
 +
 +
</div></div>
 
|}}
 
|}}
 
<!--Job Scripts-->
 
<!--Job Scripts-->

Latest revision as of 20:07, 30 October 2023

Description

rclone website  

Rclone is a command line program to manage files on cloud storage, including Amazon Drive, Backblaze B2, Citrix Sharefile, Dropbox, Google Cloud Storage, Google Photos, Google Drive, Microsoft Azure Blob Storage, OpenDrive, Zoho, and more.

Environment Modules

Run module spider rclone to find out what environment modules are available for this application.

System Variables

  • HPC_RCLONE_DIR - installation directory

Additional Information

Examples

Add DropBox Remote

Expand to see an example to add DropBox Remote.

I've verified that the https://rclone.org/dropbox/ and https://rclone.org/drive/ procedures worked on HiPerGator. Here's dropbox remote setup:

  1. Start an Open OnDemand Desktop session: https://ood.rc.ufl.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/hipergator/session_contexts/new
  2. Once the session starts connect to it and open a terminal by clicking on its icon.
  3. Use 'New > Open Tab' menu option or 'Ctrl+Shift+T' key combo in the terminal to make sure you have two tabs to start a web browser
    • Tab1: 
      • module load rclone
    • Tab2: 
      • module load ubuntu firefox
  4. Follow the https://rclone.org/dropbox/ procedure. Note that the ubuntu module has chrome and firefox browsers, so the web authentication part of defining a remote can be done by pasting the generated link into it.

Tab1: 

$ rclone config
Current remotes:

Name                 Type
====                 ====
gdrive               drive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> dbox
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
 1 / 1Fichier
   \ (fichier)
...
12 / Dropbox
   \ (dropbox)
...
46 / seafile
   \ (seafile)
Storage> dropbox
Option client_id.
OAuth Client Id.
Leave blank normally.
Enter a value. Press Enter to leave empty.
client_id> 
Option client_secret.
OAuth Client Secret.
Leave blank normally.
Enter a value. Press Enter to leave empty.
client_secret> 
Edit advanced config?
y) Yes
n) No (default)
y/n> n
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine

y) Yes (default)
n) No
y/n> y
2022/09/28 15:29:19 NOTICE: If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=2B9dREDACTED_gzfrZg
2022/09/28 15:29:19 NOTICE: Log in and authorize rclone for access

<At this point copy the link above and paste it into 'firefox' URL bar. Log in and authorize the connection. Return to Tab1> 

2022/09/28 15:29:19 NOTICE: Waiting for code...
2022/09/28 15:30:47 NOTICE: Got code
--------------------
[dbox]
type = dropbox
token = {"access_token":"sl.BQJDjte4UkfWREDACTEDyRcXqyi-oAcmScjT_-mn3b81BAc-pLgPcX5PtDRRREDACTEDwIaulrxz5ncFYrLQyummJpG5","token_type":"bearer","refresh_token":"93j-33PYjZgAAAAAAAREDACTED_NwenaxE3Y5VBbLD93g6r","expiry":"2022-09-28T19:30:48.69886106-04:00"}
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
dbox                 dropbox
gdrive               drive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config

Test the Dropbox remote: 

$ rclone lsd dbox:
          -1 2022-09-28 15:31:03        -1 GCP 2019 Photos
          -1 2022-09-28 15:31:03        -1 SKITE
          -1 2022-09-28 15:31:03        -1 Sent files
          -1 2022-09-28 15:31:03        -1 UFIT-RC-SysSecPlan

To copy a local file/directory to a remote dropbox location: 

$ rclone copy /path/to/local/source remote:/path/to/remote/




Retrieved from "https://help.rc.ufl.edu/mediawiki/index.php?title=Rclone&oldid=28228"