Difference between revisions of "Rclone"

From UFRC
Jump to navigation Jump to search
 
(8 intermediate revisions by 4 users not shown)
Line 18: Line 18:
 
{{App_Description|app={{#var:app}}|url={{#var:url}}|name={{#var:app}}}}|}}
 
{{App_Description|app={{#var:app}}|url={{#var:url}}|name={{#var:app}}}}|}}
  
Rclone is a command line program to manage files on cloud storage.
+
Rclone is a command line program to manage files on cloud storage, including Amazon Drive, Backblaze B2, Citrix Sharefile, Dropbox, Google Cloud Storage, Google Photos, Google Drive, Microsoft Azure Blob Storage, OpenDrive, Zoho, and more.
  
 
<!--Modules-->
 
<!--Modules-->
Line 33: Line 33:
  
 
==Examples==
 
==Examples==
===Add DropBox Remote==
+
===Add DropBox Remote===
 
+
<div class="mw-collapsible mw-collapsed" style="width:70%; padding: 5px; border: 1px solid gray;">
 +
''Expand to see an example to add DropBox Remote.''
 +
<div class="mw-collapsible-content" style="padding: 5px;">
  
 
I've verified that the https://rclone.org/dropbox/ and https://rclone.org/drive/ procedures worked on HiPerGator. Here's dropbox remote setup:
 
I've verified that the https://rclone.org/dropbox/ and https://rclone.org/drive/ procedures worked on HiPerGator. Here's dropbox remote setup:
  
1. Start an Open OnDemand Desktop session: https://ood.rc.ufl.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/hipergator/session_contexts/new
+
# Start an Open OnDemand Desktop session: https://ood.rc.ufl.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/hipergator/session_contexts/new
 
+
# Once the session starts connect to it and open a terminal by clicking on its icon.
2. Once the session starts connect to it and open a terminal by clicking on its icon.
+
# Use 'New > Open Tab' menu option or 'Ctrl+Shift+T' key combo in the terminal to make sure you have two tabs to start a web browser
 
+
#*Tab1:
3. Use 'New > Open Tab' menu option or 'Ctrl+Shift+T' key combo in the terminal to make sure you have two tabs
+
#**<pre>module load rclone</pre>
 +
#*Tab2:
 +
#**<pre>module load ubuntu firefox</pre>
 +
#Follow the https://rclone.org/dropbox/ procedure. Note that the ubuntu module has chrome and firefox browsers, so the web authentication part of defining a remote can be done by pasting the generated link into it.
  
 
Tab1:
 
Tab1:
module load rclone
+
<pre>
 
 
Tab2:
 
module load ubuntu
 
firefox
 
 
 
to start a web browser.
 
 
 
4. Follow the https://rclone.org/dropbox/ procedure. Note that the ubuntu module has chrome and firefox browsers, so the web authentication part of defining a remote can be done by pasting the generated link into it.
 
 
 
Tab1:
 
 
 
 
$ rclone config
 
$ rclone config
 
Current remotes:
 
Current remotes:
Line 109: Line 103:
 
2022/09/28 15:29:19 NOTICE: Log in and authorize rclone for access
 
2022/09/28 15:29:19 NOTICE: Log in and authorize rclone for access
  
 
+
</pre>
 
<At this point copy the link above and paste it into 'firefox' URL bar. Log in and authorize the connection. Return to Tab1>
 
<At this point copy the link above and paste it into 'firefox' URL bar. Log in and authorize the connection. Return to Tab1>
 
+
<pre>
 
2022/09/28 15:29:19 NOTICE: Waiting for code...
 
2022/09/28 15:29:19 NOTICE: Waiting for code...
 
2022/09/28 15:30:47 NOTICE: Got code
 
2022/09/28 15:30:47 NOTICE: Got code
Line 137: Line 131:
 
s) Set configuration password
 
s) Set configuration password
 
q) Quit config
 
q) Quit config
 
+
</pre>
 
Test the Dropbox remote:
 
Test the Dropbox remote:
 
+
<pre>
 
$ rclone lsd dbox:
 
$ rclone lsd dbox:
 
           -1 2022-09-28 15:31:03        -1 GCP 2019 Photos
 
           -1 2022-09-28 15:31:03        -1 GCP 2019 Photos
Line 145: Line 139:
 
           -1 2022-09-28 15:31:03        -1 Sent files
 
           -1 2022-09-28 15:31:03        -1 Sent files
 
           -1 2022-09-28 15:31:03        -1 UFIT-RC-SysSecPlan
 
           -1 2022-09-28 15:31:03        -1 UFIT-RC-SysSecPlan
 +
</pre>
  
 +
To copy a local file/directory to a remote dropbox location:
 +
<pre>
 +
$ rclone copy /path/to/local/source remote:/path/to/remote/
 +
</pre>
  
 +
</div></div>
 
|}}
 
|}}
 
<!--Job Scripts-->
 
<!--Job Scripts-->

Latest revision as of 20:07, 30 October 2023

Description

rclone website  

Rclone is a command line program to manage files on cloud storage, including Amazon Drive, Backblaze B2, Citrix Sharefile, Dropbox, Google Cloud Storage, Google Photos, Google Drive, Microsoft Azure Blob Storage, OpenDrive, Zoho, and more.

Environment Modules

Run module spider rclone to find out what environment modules are available for this application.

System Variables

  • HPC_RCLONE_DIR - installation directory

Additional Information

Examples

Add DropBox Remote

Expand to see an example to add DropBox Remote.

I've verified that the https://rclone.org/dropbox/ and https://rclone.org/drive/ procedures worked on HiPerGator. Here's dropbox remote setup:

  1. Start an Open OnDemand Desktop session: https://ood.rc.ufl.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/hipergator/session_contexts/new
  2. Once the session starts connect to it and open a terminal by clicking on its icon.
  3. Use 'New > Open Tab' menu option or 'Ctrl+Shift+T' key combo in the terminal to make sure you have two tabs to start a web browser
    • Tab1:
      • module load rclone
    • Tab2:
      • module load ubuntu firefox
  4. Follow the https://rclone.org/dropbox/ procedure. Note that the ubuntu module has chrome and firefox browsers, so the web authentication part of defining a remote can be done by pasting the generated link into it.

Tab1:

$ rclone config
Current remotes:

Name                 Type
====                 ====
gdrive               drive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> dbox
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
 1 / 1Fichier
   \ (fichier)
...
12 / Dropbox
   \ (dropbox)
...
46 / seafile
   \ (seafile)
Storage> dropbox
Option client_id.
OAuth Client Id.
Leave blank normally.
Enter a value. Press Enter to leave empty.
client_id> 
Option client_secret.
OAuth Client Secret.
Leave blank normally.
Enter a value. Press Enter to leave empty.
client_secret> 
Edit advanced config?
y) Yes
n) No (default)
y/n> n
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine

y) Yes (default)
n) No
y/n> y
2022/09/28 15:29:19 NOTICE: If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=2B9dREDACTED_gzfrZg
2022/09/28 15:29:19 NOTICE: Log in and authorize rclone for access

<At this point copy the link above and paste it into 'firefox' URL bar. Log in and authorize the connection. Return to Tab1>

2022/09/28 15:29:19 NOTICE: Waiting for code...
2022/09/28 15:30:47 NOTICE: Got code
--------------------
[dbox]
type = dropbox
token = {"access_token":"sl.BQJDjte4UkfWREDACTEDyRcXqyi-oAcmScjT_-mn3b81BAc-pLgPcX5PtDRRREDACTEDwIaulrxz5ncFYrLQyummJpG5","token_type":"bearer","refresh_token":"93j-33PYjZgAAAAAAAREDACTED_NwenaxE3Y5VBbLD93g6r","expiry":"2022-09-28T19:30:48.69886106-04:00"}
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
dbox                 dropbox
gdrive               drive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config

Test the Dropbox remote:

$ rclone lsd dbox:
          -1 2022-09-28 15:31:03        -1 GCP 2019 Photos
          -1 2022-09-28 15:31:03        -1 SKITE
          -1 2022-09-28 15:31:03        -1 Sent files
          -1 2022-09-28 15:31:03        -1 UFIT-RC-SysSecPlan

To copy a local file/directory to a remote dropbox location:

$ rclone copy /path/to/local/source remote:/path/to/remote/