Revision as of 13:15, 20 October 2021

Overview

This document describes transferring data between a local computer (client) and HiPerGator (HPG).

Tools

There are a variety of command-line, GUI, and even web-based tools available for transferring data to or from HiPerGator. Command-line tools include, for example, 'cp, mv, scp, rsync, sftp, wget, curl, or ncftp (from the ncftp environment module]'. GUI tools like Cyberduck, WinSCP, MobaXTerm SFTP and similar tools can be used on your local computer. Globus data transfer tool is available via a web interface in addition to command-line and GUI versions that can be run on your local computer.

Transfers within HiPerGator

Login servers on HiPergator (hpg.rc.ufl.edu) can be used for rsync, copying files with cp or mv depending on the permissions or filesystem ACLs (extended permissions).

Transfers between a local computer and HiPerGator

Globus

If your data files are large (hundreds of megabytes or gigabytes) then try Globus first.

SFTP/Rsync

For smaller file sizes, or if Globus is not an option, use SFTP/rsync/scp by connecting to the 'hpg.rc.ufl.edu' or 'sftp.rc.ufl.edu'. Set port to 22 if you have to specify it. Use your gatorlink credentials to connect.

Note: If you are using Cyberduck on a MacOS computer and notice that it is using the old password resulting in a security lockout uncheck 'Use Keychain' in 'Preferences > General'.

Transferring data from HiPerGator to a remote system/site

If you are logged into HiPerGator and need to transfer data to or from a remote system or a site use the login nodes. Transfers can also be made from within developmental sessions. You can use ftp (lftp command), sftp, scp, rsync, or Globus to transfer data out.

Providing access to your data

Within HiPerGator

If you need to share data with another RC user there are two common options to select from.

First, a Globus Shared Collection can be created on any directory you own. Either read, write, or read/write access can be granted to any Globus user and revoked at any time.
Second, you can submit a support request to have the receiving user added to your group. You may also have to change permissions on the respective directory tree to make the source directory and the files within readable by the receiving user by running 'chmod g+rx <directory>' command on the minimal set of directories that will allow access to the group starting from the source directory and 'chmod g+r *' inside the source directory to make the files readable by the group.
In some cases it is possible to set up a filesystem ACL (access control list) to provide more complicated permission structure for a directory tree or file(s) without changing file ownership or linux permissions. We'd be happy to review your request to see whether setting an ACL may help in a particular case.

Outside of HiPerGator

If you need to serve data to the outside either publicly or with password protection for collaborations you can:

Set up a shared end-point in Globus as shown above. Either read, write, or read/write access to the SEP can be granted to any Globus user and revoked at will by you. Note that the data does not necessarily have to be in /blue. Log into Globus; go to the 'Groups' menu and select 'Search For Groups'. Search for 'University of Florida Research Computing' and request to be added to the group. Once added, you will be able to install Globus Connect Personal and create a Personal Shared End-Point (PSEP) on your local computer, which will work the same way a SEP works on HiPerGator giving you the flexibility to transfer data either way with any other Globus user.
Submit a support request and we'll help you get set up with serving static data from https://data.rc.ufl.edu
If you want to create your own application for data sharing you can purchase a unit service for our PubApps web application infrastructure.

@@ Line 1: / Line 1: @@
-[[Category:How-To]]
+[[Category:How-To]][[Category:Transfer Data]]
 ==Overview==
 This document describes transferring data between a local computer (client) and HiPerGator (HPG).
 =Tools=
-There is a variety of command-line, GUI, and even web-based tools available for transferring data to or from HiPerGator. Some examples include 'cp, mv, scp, [[Rsync|rsync]], or sftp' on the command-line, FileZilla, Cyberduck, WinSCP, or MobaXTerm Gui sotware, or [[Globus|Globus data transfer tool]] available via a web interface in addition to command-line and GUI versions. The instructions below can be applied to virtually all of the tools mentioned above.
+There are a variety of command-line, GUI, and even web-based tools available for transferring data to or from HiPerGator. Command-line tools include, for example, 'cp, mv, scp, [[Rsync|rsync]], sftp, wget, curl, or ncftp (from the ncftp environment module]'. GUI tools like Cyberduck, WinSCP, MobaXTerm SFTP and similar tools can be used on your local computer. [[Globus|Globus data transfer tool]] is available via a web interface in addition to command-line and GUI versions that can be run on your local computer.
-=Servers=
+=Transfers within HiPerGator=
-It is no longer possible to use interactive SSH login servers on HiPergator (hpg2.rc.ufl.edu) i.e. gator3 and gator4 - for rsync or SFTP transfers. Use the dedicated servers described below for data transfers. At the same time, the data transfer servers do not allow interactive logins via ssh, so do not try to log on into them.
+Login servers on HiPergator (hpg.rc.ufl.edu) can be used for [[Rsync|rsync]], copying files with cp or mv depending on the permissions or filesystem ACLs (extended permissions).
-==From a Local computer to HiPerGator==
+==Transfers between a local computer and HiPerGator==
+===Globus===
 If your data files are large (hundreds of megabytes or gigabytes) then try [[Globus]] first.
-For smaller file sizes Use SFTP/rsync/scp by connecting to the '<code>sftp.rc.ufl.edu</code>' data transfer server. SFTP uses port 22 if you have to specify it.
+===SFTP/Rsync===
+For smaller file sizes, or if Globus is not an option, use SFTP/[[Rsync|rsync]]/scp by connecting to the '<code>hpg.rc.ufl.edu</code>' or '<code>sftp.rc.ufl.edu</code>'. Set port to 22 if you have to specify it. Use your gatorlink credentials to connect.
-==Transfers from a remote location to HiPerGator==
+;Note: If you are using [https://cyberduck.io/ Cyberduck] on a MacOS computer and notice that it is using the old password resulting in a security lockout uncheck 'Use Keychain' in 'Preferences > General'.
-If you are already logged into HiPerGator and need to transfer data to or from a remote system log into '<code>dtn1.ufhpc</code>' node from any other node within HiPerGator. This server has a high-throughput network connection to the outside. Once on that node you can use command-line tools like wget, curl, sftp, or ncftp (from the ncft environment module) to transfer data to or from remote systems.
+==Transferring data from HiPerGator to a remote system/site==
+If you are logged into HiPerGator and need to transfer data to or from a remote system or a site use the login nodes. Transfers can also be made from within [[Development_and_Testing|developmental sessions]]. You can use ftp (lftp command), sftp, scp, rsync, or Globus to transfer data out.
+==Providing access to your data==
+===Within HiPerGator===
+If you need to share data with another RC user there are two common options to select from.
+* First, a [[Globus|Globus Shared Collection]] can be created on any directory you own. Either read, write, or read/write access can be granted to any Globus user and revoked at any time.
+* Second, you can submit a [https://support.rc.ufl.edu support request] to have the receiving user added to your group. You may also have to change permissions on the respective directory tree to make the source directory and the files within readable by the receiving user by running 'chmod g+rx <directory>' command on the minimal set of directories that will allow access to the group starting from the source directory and 'chmod g+r *' inside the source directory to make the files readable by the group.
+* In some cases it is possible to set up a filesystem ACL (access control list) to provide more complicated permission structure for a directory tree or file(s) without changing file ownership or linux permissions. We'd be happy to review your request to see whether setting an ACL may help in a particular case.
+===Outside of HiPerGator===
+If you need to serve data to the outside either publicly or with password protection for collaborations you can:
+* Set up a shared end-point in [[Globus]] as shown above.  Either read, write, or read/write access to the SEP can be granted to any Globus user and revoked at will by you. Note that the data does not necessarily have to be in /blue. Log into [http://globus.org Globus]; go to the 'Groups' menu and select 'Search For Groups'. Search for 'University of Florida Research Computing' and request to be added to the group. Once added, you will be able to install [https://www.globus.org/globus-connect-personal Globus Connect Personal] and create a Personal Shared End-Point (PSEP) on your local computer, which will work the same way a SEP works on HiPerGator giving you the flexibility to transfer data either way with any other Globus user.
+* Submit a [https://support.rc.ufl.edu support request] and we'll help you get set up with serving static data from [https://data.rc.ufl.edu https://data.rc.ufl.edu]
+* If you want to create your own application for data sharing you can purchase a unit service for our PubApps web application infrastructure.

Difference between revisions of "Transfer Data"