Difference between revisions of "Transfer Data"

From UFRC
Jump to navigation Jump to search
Line 1: Line 1:
 
[[Category:How-To]]
 
[[Category:How-To]]
 
==Overview==
 
==Overview==
This document describes transferring data between a local computer (client) and HiPerGator (HPG). For the duration of the HiPerGator 1.0 to HiPerGator 2.0 (HPG1->HPG2) transition both HPG1 and HPG2 information will be presented.
+
This document describes transferring data between a local computer (client) and HiPerGator (HPG).
  
 
=Tools=
 
=Tools=
There is a variety of command-line, GUI, and even web-based tools available for transferring data to or from HiPerGator and between HPG1 and HPG2. Some examples include 'cp, mv, scp, [[Rsync|rsync]], or sftp' on the command-line, FileZilla, Cyberduck, WinSCP, or MobaXTerm Gui sotware, or [[Globus|Globus data transfer tool]] available via a web interface in addition to command-line and GUI versions. The instructions below can be applied to virtually all of the tools mentioned above.
+
There is a variety of command-line, GUI, and even web-based tools available for transferring data to or from HiPerGator. Some examples include 'cp, mv, scp, [[Rsync|rsync]], or sftp' on the command-line, FileZilla, Cyberduck, WinSCP, or MobaXTerm Gui sotware, or [[Globus|Globus data transfer tool]] available via a web interface in addition to command-line and GUI versions. The instructions below can be applied to virtually all of the tools mentioned above.
  
 
=Servers=
 
=Servers=
It is no longer possible to use interactive SSH login servers on either HiPergator 1 or 2 - gator.rc.ufl.edu or hpg2.rc.ufl.edu - for rsync or SFTP transfers. Use the dedicated servers described below for data transfers. At the same time, the data transfer servers do not allow interactive logins via ssh, so do not try to log on into them.
+
It is no longer possible to use interactive SSH login servers on HiPergator (hpg2.rc.ufl.edu) i.e. gator3 and gator4 - for rsync or SFTP transfers. Use the dedicated servers described below for data transfers. At the same time, the data transfer servers do not allow interactive logins via ssh, so do not try to log on into them.
  
==HiPerGator 1.0==
+
==From a Local computer to HiPerGator==
To transfer data to or from HiPerGator 1.0, with its main high-performance filesystem available at /scratch/lfs, connect to the '<code>rsync.rc.ufl.edu</code>' data transfer server. The name of this server will be changed to point to HiPerGator 2.0 after the HPG1->HPG2 transition is complete.
+
To transfer data from your local computer to or from HiPerGator, with its main high-performance filesystem available at /ufrc, connect to the '<code>sftp.rc.ufl.edu</code>' data transfer server. SFTP uses port 22 if you have to specify it.
  
==HiPerGator2==
+
==Transfers from HiPerGator==
To transfer data to or from HiPerGator 2.0, with its main high-performance filesystem available at /ufrc, connect to the '<code>sftp.rc.ufl.edu</code>' data transfer server. SFTP uses port 22 if you have to specify it.
+
If you are already logged into HiPerGator and need to transfer data to or from a remote system log into '<code>dtn1.ufhpc</code>' node from any other node within HiPerGator. This server has a high-throughput network connection to the outside.
 
 
==HPG1->HPG2==
 
There are two main mechanisms of data transfers between HiPerGator 1.0 (HPG1) and HiPerGator 2.0 (HPG2). For large files a ''streaming'' approach of [[Globus]] works very well. For many small files a 'cp' or [[Rsync|'rsync']] may work better.  If in doubt, try [[Globus]] first.
 
 
 
To transfer data between /scratch/lfs and /ufrc with cp or [[Rsync|rsync]] log into '<code>dtn1.ufhpc</code>' from any other node within HiPerGator. This server has a read-only connection to /scratch/lfs, so the data must flow from /scratch/lfs to /ufrc, but since both filesystems are available at the same time both the linux copy command '<code>cp</code>' and '<code>rsync</code>' will work.
 

Revision as of 01:15, 22 September 2016

Overview

This document describes transferring data between a local computer (client) and HiPerGator (HPG).

Tools

There is a variety of command-line, GUI, and even web-based tools available for transferring data to or from HiPerGator. Some examples include 'cp, mv, scp, rsync, or sftp' on the command-line, FileZilla, Cyberduck, WinSCP, or MobaXTerm Gui sotware, or Globus data transfer tool available via a web interface in addition to command-line and GUI versions. The instructions below can be applied to virtually all of the tools mentioned above.

Servers

It is no longer possible to use interactive SSH login servers on HiPergator (hpg2.rc.ufl.edu) i.e. gator3 and gator4 - for rsync or SFTP transfers. Use the dedicated servers described below for data transfers. At the same time, the data transfer servers do not allow interactive logins via ssh, so do not try to log on into them.

From a Local computer to HiPerGator

To transfer data from your local computer to or from HiPerGator, with its main high-performance filesystem available at /ufrc, connect to the 'sftp.rc.ufl.edu' data transfer server. SFTP uses port 22 if you have to specify it.

Transfers from HiPerGator

If you are already logged into HiPerGator and need to transfer data to or from a remote system log into 'dtn1.ufhpc' node from any other node within HiPerGator. This server has a high-throughput network connection to the outside.