Difference between revisions of "Transfer Data"
Moskalenko (talk | contribs) |
|||
| Line 16: | Line 16: | ||
==HPG1->HPG2== | ==HPG1->HPG2== | ||
| − | There are two main mechanisms of data transfers between HiPerGator1 (HPG1) and HiPerGator2 (HPG2). For large | + | There are two main mechanisms of data transfers between HiPerGator1 (HPG1) and HiPerGator2 (HPG2). For large files a ''streaming'' approach of [[Globus]] works very well. For many small files a 'cp' or [[Rsync|'rsync']] may work better. If in doubt, try Globus first. |
To transfer data between /scratch/lfs and /ufrc with cp or [[Rsync|rsync]] log into '<code>dtn1.ufhpc</code>' from any other node within HiPerGator. This server has a read-only connection to /scratch/lfs, so the data must flow from /scratch/lfs to /ufrc, but since both filesystems are available at the same time both the linux copy command '<code>cp</code>' and '<code>rsync</code>' will work. | To transfer data between /scratch/lfs and /ufrc with cp or [[Rsync|rsync]] log into '<code>dtn1.ufhpc</code>' from any other node within HiPerGator. This server has a read-only connection to /scratch/lfs, so the data must flow from /scratch/lfs to /ufrc, but since both filesystems are available at the same time both the linux copy command '<code>cp</code>' and '<code>rsync</code>' will work. | ||
Revision as of 17:27, 13 June 2016
Overview
This document describes transferring data between a local computer (client) and the HiPerGator (HPG). For the duration of the HiPerGator1 to HiPerGator2 (HPG1->HPG2) transition both HPG1 and HPG2 information will be presented.
Tools
There is a variety of command-line, Gui, and even web-based tools available for transferring data to or from HiPerGator and between HPG1 and HPG2. Some examples include 'cp, mv, scp, rsync, or sftp' on the command-line, FileZilla, Cyberduck, WinSCP, or MobaXTerm Gui sotware, or Globus data transfer tool available via a web interface in addition to command-line and Gui versions. The instructions below can be applied to virtuall all of the tools mentioned above.
Servers
- Note
- using login (gator) nodes via gator.rc.ufl.edu or hpg2.rc.ufl.edu for sftp transfers is no longer allowed as the connections overwhelm the load-balancing solution that makes the login servers highly-available. Use the dedicated transfer servers for the task.
HiPerGator1
To transfer data to or from HiPerGator1 with its main high-performance filesystem available at /scratch/lfs connect to the 'rsync.rc.ufl.edu' data transfer server. The name of this server will be changed to point to HiPerGator2 after the HPG1->HPG2 transition is complete.
HiPerGator2
To transfer data to or from HiPerGator 2.0 ,with its main high-performance filesystem available at /ufrc, connect to the 'sftp.rc.ufl.edu' data transfer server.
HPG1->HPG2
There are two main mechanisms of data transfers between HiPerGator1 (HPG1) and HiPerGator2 (HPG2). For large files a streaming approach of Globus works very well. For many small files a 'cp' or 'rsync' may work better. If in doubt, try Globus first.
To transfer data between /scratch/lfs and /ufrc with cp or rsync log into 'dtn1.ufhpc' from any other node within HiPerGator. This server has a read-only connection to /scratch/lfs, so the data must flow from /scratch/lfs to /ufrc, but since both filesystems are available at the same time both the linux copy command 'cp' and 'rsync' will work.