Difference between revisions of "Transfer Data"

Revision as of 15:31, 25 May 2023

Overview

This document describes transferring data between a local computer (client) and HiPerGator (HPG). For file sharing on the cluster see Sharing Files

Note: ResVault users: find more info in this page

Tools

There are a variety of command-line, GUI, and even web-based tools available for transferring data to or from HiPerGator. Command-line tools include, for example, 'cp, mv, scp, rsync, sftp, wget, curl, or ncftp (from the ncftp environment module]'. GUI tools like Cyberduck, WinSCP, BitVise SFTP and similar tools can be used on your local computer. Globus data transfer tool is available via a web interface in addition to command-line and GUI versions that can be run on your local computer. Please visit the HPG how-to video series on Data Transfer for more details.

Transferring Data within HiPerGator

Login servers on HiPergator (hpg.rc.ufl.edu) can be used for rsync, copying files with cp or mv depending on the permissions or filesystem ACLs (extended permissions).

Between a local computer and HiPerGator

Globus

If your data files are large (hundreds of megabytes or gigabytes) then try Globus first.

SFTP/Rsync

For smaller file sizes, or if Globus is not an option, use SFTP/rsync/scp by connecting to the 'hpg.rc.ufl.edu' or 'sftp.rc.ufl.edu'. Set port to 22 if you have to specify it. Use your gatorlink credentials to connect. Make sure MFA (multi-factor authentication) is taken into account to avoid having to go through MFA for every file transferred. You can also use terminal interfaces like Tabby and MobaXterm, which include an GUI SFTP function embedded.

Note: If you are using Cyberduck on a MacOS computer and notice that it is using the old password resulting in a security lockout uncheck 'Use Keychain' in 'Preferences > General'.

Rclone

Rclone is a command-line data transfer tool that support transferring data from or to dozens of cloud storage providers and remote locations. Please see https://rclone.org/docs/ for information on a specific storage provider.

JupyterHub

The easiest way to upload data using JypyterHub is by using the Upload Files button included in the File Browser (Ctrl+Shift+F) menu.

From HiPerGator to a remote system/site

If you are logged into HiPerGator and need to transfer data to or from a remote system or a site use the login nodes. Transfers can also be made from within developmental sessions. You can use ftp (lftp command), sftp, scp, rsync, or Globus to transfer data out.

Downloading Data from an External Server or Website

Most "Download" buttons on external websites have the option to right-click and select an option similar to "Copy link address". Once the link address is copied in the clipboard, you can paste it in your HPG terminal following your "wget" or "curl" commands, such as: $ wget https://github.com/author/software/main.zip"

Another option is to use the firefox module via Open OnDemand to navigate to a website, then download files into a target directory.

Alternatively, users can download files to their local computers and upload them into HPG using one of the methods described in the "Between a local computer and HiPerGator" section above.

Providing access to your data

If you need to share data with another RC user or publicly for collaboration, view instructions at Providing Access To Data

@@ Line 5: / Line 5: @@
 ==Overview==
 This document describes transferring data between a local computer (client) and HiPerGator (HPG). For file sharing on the cluster see [https://help.rc.ufl.edu/doc/Sharing_Files Sharing Files]
+* Note: ResVault users: [https://help.rc.ufl.edu/doc/How_To_ResVault#How_to_Transfer_Data_from.2Fto_RV find more info in this page]
 ==Tools==
-There are a variety of command-line, GUI, and even web-based tools available for transferring data to or from HiPerGator. Command-line tools include, for example, 'cp, mv, scp, [[Rsync|rsync]], sftp, wget, curl, or ncftp (from the ncftp environment module]'. GUI tools like Cyberduck, WinSCP, BitVise SFTP and similar tools can be used on your local computer. [[Globus|Globus data transfer tool]] is available via a web interface in addition to command-line and GUI versions that can be run on your local computer. [https://help.rc.ufl.edu/doc/How_To_Videos#data%20transfer Please visit the HPG how-to video series on Data Transfer for more details.]
+There are a variety of command-line, GUI, and even web-based tools available for transferring data to or from HiPerGator. Command-line tools include, for example, 'cp, mv, scp, [[Rsync|rsync]], sftp, wget, curl, or ncftp (from the ncftp environment module]'. GUI tools like Cyberduck, WinSCP, BitVise SFTP and similar tools can be used on your local computer. [[Globus|Globus data transfer tool]] is available via a web interface in addition to command-line and GUI versions that can be run on your local computer. [https://help.rc.ufl.edu/doc/Training_Videos#data%20transfer Please visit the HPG how-to video series on Data Transfer for more details.]
 ==Transferring Data within HiPerGator==
@@ Line 20: / Line 22: @@
 ;Note: If you are using [https://cyberduck.io/ Cyberduck] on a MacOS computer and notice that it is using the old password resulting in a security lockout uncheck 'Use Keychain' in 'Preferences > General'.
+====Rclone====
+[[Rclone]] is a command-line data transfer tool that support transferring data from or to dozens of cloud storage providers and remote locations. Please see [https://rclone.org/docs/ https://rclone.org/docs/] for information on a specific storage provider.
 ====JupyterHub====
-The easiest way to upload data using JypyterHub is by using the Upload Files button included in the File Browser (Ctrl+Shift+F) menu.
+The easiest way to upload data using JypyterHub is by using the Upload Files button included in the File Browser (Ctrl+Shift+F) menu.<br>
-[[File:Jupyterhub upload.png|frameless|left]]
+[[File:Jupyterhub upload.png|frameless]]
 ===From HiPerGator to a remote system/site===
 If you are logged into HiPerGator and need to transfer data to or from a remote system or a site use the login nodes. Transfers can also be made from within [[Development_and_Testing|developmental sessions]]. You can use ftp (lftp command), sftp, scp, rsync, or Globus to transfer data out.
-==Providing access to your data==
+==Downloading Data from an External Server or Website==
-===Within HiPerGator===
+Most "Download" buttons on external websites have the option to right-click and select an option similar to "Copy link address". Once the link address is copied in the clipboard, you can paste it in your HPG terminal following your "wget" or "curl" commands, such as:
-If you need to share data with another RC user there are two common options to select from.
+<code>$ wget https://github.com/author/software/main.zip"</code>
-* First, a [[Globus|Globus Shared Collection]] can be created on any directory you own. Either read, write, or read/write access can be granted to any Globus user and revoked at any time.
-* Second, you can submit a [https://support.rc.ufl.edu support request] to have the receiving user added to your group. You may also have to change permissions on the respective directory tree to make the source directory and the files within readable by the receiving user by running 'chmod g+rx <directory>' command on the minimal set of directories that will allow access to the group starting from the source directory and 'chmod g+r *' inside the source directory to make the files readable by the group.
-* In some cases it is possible to set up a filesystem [https://help.rc.ufl.edu/doc/ACLs ACL] (access control list) to provide more complicated permission structure for a directory tree or file(s) without changing file ownership or linux permissions. We'd be happy to review your request to see whether setting an ACL may help in a particular case.
-===Outside of HiPerGator===
+Another option is to use the firefox module via Open OnDemand to navigate to a website, then download files into a target directory.
-If you need to serve data to the outside either publicly or with password protection for collaborations you can:
-* Set up a shared end-point in [[Globus]] as shown above.  Either read, write, or read/write access to the SEP can be granted to any Globus user and revoked at will by you. Note that the data does not necessarily have to be in /blue. Log into [http://globus.org Globus]; go to the 'Groups' menu and select 'Search For Groups'. Search for 'University of Florida Research Computing' and request to be added to the group. Once added, you will be able to install [https://www.globus.org/globus-connect-personal Globus Connect Personal] and create a Personal Shared End-Point (PSEP) on your local computer, which will work the same way a SEP works on HiPerGator giving you the flexibility to transfer data either way with any other Globus user.
-* Submit a [https://support.rc.ufl.edu support request] and we'll help you get set up with serving static data from [https://data.rc.ufl.edu https://data.rc.ufl.edu]
-* If you want to create your own application for data sharing you can purchase a unit service for our PubApps web application infrastructure.
-==From HiPerGator to Third-Party Cloud Storage Service==
+Alternatively, users can download files to their local computers and upload them into HPG using one of the methods described in the "Between a local computer and HiPerGator" section above.
-There are 2 recommended options to transfer data directly from HPG to Dropbox/similar services:
-===Mapping Drives===
-Map your /blue /orange storages as a drive in your personal computer using the path: \\exasmb.rc.ufl.edu.
-For instructions on how to do so, please see [https://help.rc.ufl.edu/doc/Samba_Access#Mapping_Drive_Shares Mapping Drive Shares]
-===Open OnDemand===
+==Providing access to your data==
-{|cellpadding="20"
+If you need to share data with another RC user or publicly for collaboration, view instructions at [[Providing Access To Data]]
-|- style="vertical-align:top;"
-|
-You can use [https://ondemand.rc.ufl.edu Open OnDemand] to open a new firefox browser from your HiPerGator desktop session.
-*Start a new HiPerGator Desktop OOD session and configure it similarly to the first two images, specifically
-**Cluster partition (--partition, -p) SET TO gpu
-**Generic Resource Request (--gres). SET TO gpu:geforce:1
-*Following the third image, start a new terminal from the Desktop session and load firefox by running:
-** $ module load firefox
-** $ firefox
-*The firefox browser will start and you should be able to login to your Dropbox account.
-*Open the File System icon and navigate to your desired path, then you should be able to drag and drop files and folders (HPGtoDB_5).
-||
-[[Image:HPG_OOD_2y.png | frameless | top | 570px]]
-[[Image:HPG_COMB.png | frameless | top | 600px]]
-<br>
-[[Image:LAST HPG.jpg | frameless | bottom | center | 1000px]]
-|}
-DRAFTING