Difference between revisions of "Rsync"

From UFRC
Jump to navigation Jump to search
 
(10 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
[[Category:Transfer Data]]
 +
{|align=right
 +
  |__TOC__
 +
  |}
 +
 +
==Latest: Rsync 3.2.7==
 +
Rsync v3.2.7 is available as a module. Usage <code>ml rsync</code>
 +
 
==Overview==
 
==Overview==
 
Rsync, which stands for "remote sync," is a utility for copying and syncing data between two computer systems. UFRC now provides a URL for users to rsync their data to and from the cluster. The basic syntax of the rsync command is:
 
Rsync, which stands for "remote sync," is a utility for copying and syncing data between two computer systems. UFRC now provides a URL for users to rsync their data to and from the cluster. The basic syntax of the rsync command is:
<source lang="bash" enclose="div">
 
[user@laptop ~]$ rsync options source destination
 
</source>
 
  
 +
[user@laptop ~]$ rsync options source destination
  
{| class="wikitable" style="width:50%;"
+
{| class="wikitable" style="width:60%;"
 
|-  
 
|-  
 
| style="text-align:center;" colspan="2"| '''Some common options used with rsync commands'''
 
| style="text-align:center;" colspan="2"| '''Some common options used with rsync commands'''
Line 36: Line 42:
  
 
If Jane's username on her computer is the same as her UFRC username, she can run:
 
If Jane's username on her computer is the same as her UFRC username, she can run:
<source lang="bash" enclose="div">
+
[jane@laptop ~]$ rsync -av build_tree_xyz rsync.rc.ufl.edu:~
[jane@laptop ~]$ rsync -av build_tree_xyz rsync.rc.ufl.edu:~
 
</source>
 
 
 
  
 
If Jane's username on her computer is different than her UFRC username, she needs to run:
 
If Jane's username on her computer is different than her UFRC username, she needs to run:
<source lang="bash" enclose="div">
+
[jane@laptop ~]$ rsync -av build_tree_xyz jsmith@rsync.rc.ufl.edu:~
[jane@laptop ~]$ rsync -av build_tree_xyz jsmith@rsync.rc.ufl.edu:~
 
</source>
 
  
 
==Copying files from the cluster==
 
==Copying files from the cluster==
 
{| style="border:1px solid #0021A5; background-color:#f0f8ff; width:50%; padding:5px;"
 
{| style="border:1px solid #0021A5; background-color:#f0f8ff; width:50%; padding:5px;"
| style="padding:10px;" | '''Example''': Jane's job writes some outputs to a directory on scratch '/scratch/lfs/jsmith/build_tree_xyz_outputs/' and she wants to rsync those outputs to her laptop for further analysis.
+
| style="padding:10px;" | '''Example''': Jane's job writes some outputs to a directory on scratch '/blue/mylab/jsmith/build_tree_xyz_outputs/' and she wants to rsync those outputs to her laptop for further analysis.
 
|}
 
|}
 
  
 
Jane should run:
 
Jane should run:
<source lang="bash" enclose="div">
+
[jane@laptop ~]$ rsync -av jsmith@rsync.rc.ufl.edu:/blue/mylab/jsmith/build_tree_xyz_outputs ~
[jane@laptop ~]$ rsync -av jsmith@rsync.rc.ufl.edu:/scratch/lfs/jsmith/build_tree_xyz_outputs ~
 
</source>
 
  
 
+
==Copying files between directories==
==Copying files from /scratch/lfs to /ufrc==
 
  
 
{| style="border:1px solid #0021A5; background-color:#f0f8ff; width:50%; padding:5px;"
 
{| style="border:1px solid #0021A5; background-color:#f0f8ff; width:50%; padding:5px;"
| style="padding:10px;" | '''Example''': Jane, of the "doe" group wants to move her important data from /scratch/lfs/ to /ufrc to complete her migration to HiPerGator 2.0.
+
| style="padding:10px;" | '''Example''': Jane, of the "doe" group wants to copy a directory tree from one /blue directory to another.
 
|}
 
|}
  
Jane should login to dtn1 after logging into hpg2.rc.ufl.edu and then run rsync as follows:
+
Jane should login to hpg.rc.ufl.edu and then run rsync as follows:
<source lang="bash" enclose="div">
+
 
[janedoe@JanesMac]$ ssh jane@hpg2.rc.ufl.edu
+
[janedoe@JanesMac]$ ssh jane@hpg.rc.ufl.edu
<Enter password>
+
<Enter password>
[jane@gator4 ~]$ ssh dtn1
+
[jane@login1 ~]$ rsync -av /blue/mygroup/jane/important_data /blue/mygroup/share/project/
[jane@dtn1 ~]$ rsync -av /scratch/lfs/jane/important_data/ /ufrc/doe/jane/
+
 
</source>
+
Note the absence of a slash in the source directory. If a slash is present at the end of the source directory name then the ''contents'' of the source directory will be copied and not the directory itself.
  
 
==References==
 
==References==

Latest revision as of 21:25, 28 July 2023

Latest: Rsync 3.2.7

Rsync v3.2.7 is available as a module. Usage ml rsync

Overview

Rsync, which stands for "remote sync," is a utility for copying and syncing data between two computer systems. UFRC now provides a URL for users to rsync their data to and from the cluster. The basic syntax of the rsync command is:

[user@laptop ~]$ rsync options source destination
Some common options used with rsync commands
Option Meaning
-a archive mode; allows copying files recursively and also preserves symbolic links, files permissions, user & group ownerships and timestamps
-v increase verbosity
-r copies data recursively
-z compress file data
-h output numbers in a human-readable format

Copying files to the cluster

Example: Jane has started a new project called 'build_tree_xyz' on her laptop. She wishes to copy this project to the cluster to run some jobs related to the project. Jane runs rsync from her laptop to sync the project into her home directory.


If Jane's username on her computer is the same as her UFRC username, she can run:

[jane@laptop ~]$ rsync -av build_tree_xyz rsync.rc.ufl.edu:~

If Jane's username on her computer is different than her UFRC username, she needs to run:

[jane@laptop ~]$ rsync -av build_tree_xyz jsmith@rsync.rc.ufl.edu:~

Copying files from the cluster

Example: Jane's job writes some outputs to a directory on scratch '/blue/mylab/jsmith/build_tree_xyz_outputs/' and she wants to rsync those outputs to her laptop for further analysis.

Jane should run:

[jane@laptop ~]$ rsync -av jsmith@rsync.rc.ufl.edu:/blue/mylab/jsmith/build_tree_xyz_outputs ~

Copying files between directories

Example: Jane, of the "doe" group wants to copy a directory tree from one /blue directory to another.

Jane should login to hpg.rc.ufl.edu and then run rsync as follows:

[janedoe@JanesMac]$ ssh jane@hpg.rc.ufl.edu
<Enter password>
[jane@login1 ~]$ rsync -av /blue/mygroup/jane/important_data /blue/mygroup/share/project/

Note the absence of a slash in the source directory. If a slash is present at the end of the source directory name then the contents of the source directory will be copied and not the directory itself.

References

There are also plenty of existing tutorials available on the web describing the various rsync options.