Revision as of 14:32, 21 June 2016

Overview

This document describes transferring data between a local computer (client) and HiPerGator (HPG). For the duration of the HiPerGator 1.0 to HiPerGator 2.0 (HPG1->HPG2) transition both HPG1 and HPG2 information will be presented.

Tools

There is a variety of command-line, GUI, and even web-based tools available for transferring data to or from HiPerGator and between HPG1 and HPG2. Some examples include 'cp, mv, scp, rsync, or sftp' on the command-line, FileZilla, Cyberduck, WinSCP, or MobaXTerm Gui sotware, or Globus data transfer tool available via a web interface in addition to command-line and GUI versions. The instructions below can be applied to virtually all of the tools mentioned above.

Servers

It is no longer possible to use interactive SSH login servers on either HiPergator 1 or 2 - gator.rc.ufl.edu or hpg2.rc.ufl.edu - for Rsync or SFTP transfers. Use the dedicated transfer servers described below for data transfers. At the same time, the data transfer servers do not allow interactive logins via ssh.

HiPerGator 1.0

To transfer data to or from HiPerGator 1.0, with its main high-performance filesystem available at /scratch/lfs, connect to the 'rsync.rc.ufl.edu' data transfer server. The name of this server will be changed to point to HiPerGator 2.0 after the HPG1->HPG2 transition is complete.

HiPerGator2

To transfer data to or from HiPerGator 2.0, with its main high-performance filesystem available at /ufrc, connect to the 'sftp.rc.ufl.edu' data transfer server.

HPG1->HPG2

There are two main mechanisms of data transfers between HiPerGator 1.0 (HPG1) and HiPerGator 2.0 (HPG2). For large files a streaming approach of Globus works very well. For many small files a 'cp' or 'rsync' may work better. If in doubt, try Globus first.

To transfer data between /scratch/lfs and /ufrc with cp or rsync log into 'dtn1.ufhpc' from any other node within HiPerGator. This server has a read-only connection to /scratch/lfs, so the data must flow from /scratch/lfs to /ufrc, but since both filesystems are available at the same time both the linux copy command 'cp' and 'rsync' will work.

@@ Line 7: / Line 7: @@
 =Servers=
-;Note: Using the SSH login (gator, hpg2) servers via gator.rc.ufl.edu or hpg2.rc.ufl.edu for SFTP transfers is not allowed. Use the dedicated transfer servers described here for rsync/sftp. Note that rsync and sftp servers do not allow interactive logins via ssh - only sftp and rsync.
+It is no longer possible to use interactive SSH login servers on either HiPergator 1 or 2 - gator.rc.ufl.edu or hpg2.rc.ufl.edu - for Rsync or SFTP transfers. Use the dedicated transfer servers described below for data transfers. At the same time, the data transfer servers do not allow interactive logins via ssh.
 ==HiPerGator 1.0==
@@ Line 16: / Line 16: @@
 ==HPG1->HPG2==
-There are two main mechanisms of data transfers between HiPerGator 1.0 (HPG1) and HiPerGator 2.0 (HPG2). For large files a ''streaming'' approach of [[Globus]] works very well. For many small files a 'cp' or [[Rsync|'rsync']] may work better.  If in doubt, try Globus first.
+There are two main mechanisms of data transfers between HiPerGator 1.0 (HPG1) and HiPerGator 2.0 (HPG2). For large files a ''streaming'' approach of [[Globus]] works very well. For many small files a 'cp' or [[Rsync|'rsync']] may work better.  If in doubt, try [[Globus]] first.
 To transfer data between /scratch/lfs and /ufrc with cp or [[Rsync|rsync]] log into '<code>dtn1.ufhpc</code>' from any other node within HiPerGator. This server has a read-only connection to /scratch/lfs, so the data must flow from /scratch/lfs to /ufrc, but since both filesystems are available at the same time both the linux copy command '<code>cp</code>' and '<code>rsync</code>' will work.

Difference between revisions of "Transfer Data"