Difference between revisions of "Parallel Computing"

Latest revision as of 19:52, 16 January 2023

Parallel computing refers to running multiple computational tasks simultaneously. The idea behind it is based on the assumption that a big computational task can be divided into smaller tasks which can run concurrently.

Types of parallel computing

Parallel computing is used only for the last row of below table;

	Single Instruction	Multiple Instructions	Single Program	Multiple Programs
Single Data	SISD	MISD
Multiple Data	SIMD	MIMD	SPMD	MPMD

In more details;

Data-parallel(SIMD): Same operations/instructions are carried out on different data items, simultaneously.
Task Parallel(MIMD): Different instructions on different data carried out concurrently.
SPMD: Single program, multiple data, not synchronized at individual operation level

SPMD and MIMD are essentially the same because any MIMD can be made SPMD. SIMD is also equivalent, but in a less practical sense. MPI (Message Passing Interface) is primarily used for SPMD/MIMD.

Shared Memory vs. Distributed Memory

For explanation of the different memory models (shared, distributed, or hybrid), visit Memory: Shared vs Distributed

Communication In Parallel Computation

Communications in parallel computing takes advantage of one of the following interfaces;

OpenMp
MPI (MPI, OpenMPI)
Hybrid

OpenMp is used for communication between tasks running concurrently on the same node with access to the shared memory. Assume you have a machine which each one of its nodes contains 16 cores with shared access to 32 GB of memory. If you have an application which is parallelized and can use up to 16 cores, the tasks running on each node will communicate using OpenMP.

Assume use of the same machine; if you want to run the same application on 16 nodes using only one core on each node, communication between different nodes is necessary since memory is not shared between nodes. MPI (Message Passing Interface) utilizes this communication. MPI is used for communication between tasks which use distributed memory.

Again, assume use of the same machine; what if you want to use two nodes (8 cores on each one). The tasks running on each node communicate using OpenMP while the tasks running on different nodes communicate using MPI. This communication style is called hybrid programming since it takes advantage of hybrid memory.

It is common to mistakenly assume OpenMP and OpenMPI are the same! But, OpenMPI is the name of recent MPI versions and should not be mistaken with OpenMP.

Run OpenMP Applications

Compiler	Compiler Options	Default behavior for # of threads (If not set)
GNU (gcc, g++, gfortran)	-fopenmp	as many threads as available cores
Intel (icc ifort)	-openmp	as many threads as available cores
Portland Group (pgcc,pgCC,pgf77,pgf90)	-mp	one thread

Sample job script: LAMMPS (MPI only)

Expand this section to view sample job script.

#!/bin/bash
#SBATCH --job-name=LAMMPS-JOB
#SBATCH --output=LAMMPS.out
#SBATCH --error=LAMMPS.err
#SBATCH --mail-type=ALL
#SBATCH --mail-user=YOUR-EMAIL-ADDRESS
#SBATCH --time=01:00:00
#SBATCH --ntasks=12
#SBATCH --mem-per-cpu=4000
#SBATCH --account=YOUR-GRUOP-NAME
#SBATCH --qos=YOUR-GROUP-NAME
#
module load intel/2016.0.109 openmpi/1.10.2  lammps/7Dec15

LAMMPS=lmp_ufhpc.openmpi
INPUT=test-input

mpiexec $LAMMPS  < $INPUT

@@ Line 1: / Line 1: @@
+[[Category:Documentation‏]]
 {|align=right
    |__TOC__
    |}
-==Parallel Computing==
 Parallel computing refers to running multiple computational tasks simultaneously. The idea behind it is based on the assumption that a big computational task can be divided into smaller tasks which can run concurrently.
+=== Types of parallel computing ===
+Parallel computing is used only for the last row of below table;
 {| class="wikitable"
@@ Line 16: / Line 19: @@
 |}
-=== Types of parallel computing ===
+In more details;
+* Data-parallel(SIMD): Same operations/instructions are carried out on different data items, simultaneously.
+* Task Parallel(MIMD): Different instructions on different data carried out concurrently.
+* SPMD: Single program, multiple data, not synchronized at individual operation level
+SPMD and MIMD are essentially the same because any MIMD can be made SPMD. SIMD is also equivalent, but in a less practical sense. MPI (Message Passing Interface) is primarily used for SPMD/MIMD.
-Parallel computing is used only for the last row of below table;
+== Shared Memory vs. Distributed Memory ==
+For explanation of the different memory models (shared, distributed, or hybrid), visit [[Memory: Shared vs Distributed]]
-===Windows===
+== Communication In Parallel Computation ==
-Microsoft Windows does not come with a built-in SSH client. You have to download a client from the web. We recommend the following software:
-* SSH client - [http://www.chiark.greenend.org.uk/~sgtatham/putty Putty]
-** [[PuTTY|configuration instructions for UFRC]]
-* Graphical file transfer clients:
-** [http://filezilla-project.org/download.php?type=client FileZilla]. See our [[FileZilla|tutorial on transferring files to UFRC using FileZilla]].
-** [http://winscp.net/eng/index.php WinSCP]
-===MacOS===
-For MacOS users, the connection instructions are very similar to those for Linux/Unix users.
-''Terminal'', the terminal emulation application under MacOS is located in Applications/Utilities.
+Communications in parallel computing takes advantage of one of the following interfaces;
-Both [http://filezilla-project.org/download.php?type=client FileZilla] and [http://cyberduck.ch/ Cyberduck] are available for MacOS if you prefer a graphical interface for transferring files.
+* OpenMp
+* MPI (MPI, OpenMPI)
+* Hybrid
-==Running Graphical Programs==
+OpenMp is used for communication between tasks running concurrently on the same node with access to the shared memory.
-See the [[GUI_Programs|Gui Programs]] page for information on running graphical user interface applications at UFRC.
+Assume you have a machine which each one of its nodes contains 16 cores with shared access to 32 GB of memory. If you have an application which is parallelized and can use up to 16 cores, the tasks running on each node will communicate using OpenMP.
-==Getting Help==
+Assume use of the same machine; if you want to run the same application on 16 nodes using only one core on each node, communication between different nodes is necessary since memory is not shared between nodes. MPI (Message Passing Interface) utilizes this communication. MPI is used for communication between tasks which use distributed memory.
-If you are having problems connecting to the UFRC system, please let the UFRC Staff know by submitting a [http://support.rc.ufl.edu Support Request].
-==Interactive work under Linux==
+Again, assume use of the same machine; what if you want to use two nodes (8 cores on each one). The tasks running on each node communicate using OpenMP while the tasks running on different nodes communicate using MPI. This communication style is called hybrid programming since it takes advantage of hybrid memory.
-Once you are logged in to a Research Computing server, you will find yourself at a Linux command line prompt. That may be daunting at first. However, you only need to know a small subset of Linux commands to accomplish most tasks. There are many Linux "Getting Started" guides online and in print.  Below are just a few possibilities.  Many more are easily found via a Google search.
-* [http://www.tldp.org/LDP/intro-linux/html/ Introduction to Linux]
+It is common to mistakenly assume OpenMP and OpenMPI are the same! But, OpenMPI is the name of recent MPI versions and should not be mistaken with OpenMP.
-* [http://www.linux.org/tutorial/view/beginners-level-course Linux for Beginners]
-* [http://www.tutorialized.com/tutorial/Basic-Linux-Shell-commands/21596 Basic Linux Shell Commands]
-* [[Getting_Started_on_Linux]]
-==A Few Basic Commands==
+=== Run OpenMP Applications ===
-While it is advantageous to have a working knowledge of the most common Linux commands, it is not a requirement.  For the uninitiated, the following information may be useful as well as a good "Introduction to Using Linux" book.
-{| border=1
+{| class="wikitable"
-|-
-! Command !! Description
 |-
-| ls || List files in the current directory
+! Compiler !! Compiler Options !! Default behavior for # of threads (If not set)
 |-
-| cd || Change directory
+| GNU (gcc, g++, gfortran) || -fopenmp || as many threads as available cores
 |-
-| more || View a file's contents
+| Intel (icc ifort) || -openmp || as many threads as available cores
 |-
-| mkdir <dir> || Create a directory
+| Portland Group (pgcc,pgCC,pgf77,pgf90) || -mp || one thread
-|-
-| cp file1 file2 || Copy a file
-|-
-| mv file1 file2 || Move (i.e. rename) a file
-|-
-| rm file || Delete a file
-|-
-| rmdir dir || Delete an ''empty'' directory
-|}
-==Editing==
-Editing files on the cluster can be done through a couple of different methods...
-===Native Editors===
-* '''vi''' - The visual editor (vi) is the traditonal Unix editor. However, it is not necessarily the most intuitive editor.  That being the case, if you are unfamiliar with it, the following tutorial may be useful.
-** [http://www.eng.hawaii.edu/Tutor/vi.html VI Tutorial]
-** Another resource for vi is [[vi | right here]] on our wiki.
-** There is also a vi tutorial, '''vimtutor'''. Once logged in, simply type "<code>vimtutor</code>" at the command line to start the tutorial.
-* '''emacs''' - Emacs is a much heavier duty editor, but again has the problem of having commands that are non-intuitive. Again, we have provided a link to a tutorial for this editor.
-** [http://www2.lib.uchicago.edu/~keith//tcl-course/emacs-tutorial.html Emacs Tutorial]
-* '''pico''' - While pico is not installed on the system, [[nano]] is installed, and is a pico work-a-like.
-* '''nano''' - Nano has a good bit of on-screen help to make it easier to use.
-===External Editors===
-You can also use your favorite editor on your local machine and then transfer the files over to the Research Computing afterwards. One caveat to this is that with files created on Windows machines, usually contain unprintable characters which may be misinterpreted by Linux command interpreters (shells). If this happens, there is a utility called <code>dos2unix</code> that you can use to convert the text file from DOS/Windows formatting to Linux formatting.
-==Using Installed Software==
-We use [[Modules|Environment Modules]] to provide access to the installed software. Read about the [[Modules_Basic_Usage|basic usage of environment modules]] for information on loading software.
-==Running Jobs==
-====Trivial Example====
-{|
-|-valign="top"
-|
-<pre>
-#! /bin/sh
-#PBS -N testjob
-#PBS -o testjob.out
-#PBS -e testjob.err
-#PBS -M <INSERT EMAIL HERE>
-#PBS -r n
-#PBS -l walltime=00:01:00
-#PBS -l nodes=1:ppn=1
-#PBS -l pmem=100mb
-date
-hostname
-module load python
-python -V
-</pre>
-||
-To submit this job from gator.hpc.ufl.edu, you would use the following command:
-<pre>
-$ qsub <your job script>
-</pre>
-To check the status of running jobs, you would use the following command:
-<pre>
-$ qstat [-u <username>]
-</pre>
 |}
-====DRMAA====
+Sample job script: LAMMPS (MPI only)
-It's possible to submit jobs by using the scheduler API via the DRMAA library. A python example that uses our installed Python DRMAA library can be found in [[Example_pbs-drmaa_python_script]].
+<div class="mw-collapsible mw-collapsed" style="width:70%; padding: 5px; border: 1px solid gray;">
+''Expand this section to view sample job script.''
-====Notes on Batch Scripts====
+<div class="mw-collapsible-content" style="padding: 5px;">
-* The script can handle only one set of directives. '''Do not submit a script that has more than one set of directives''' included in it.
+<source lang=bash>
+#!/bin/bash
-* For more info on '''advanced directives''' see [[PBS_Directives]].
+#SBATCH --job-name=LAMMPS-JOB
+#SBATCH --output=LAMMPS.out
+#SBATCH --error=LAMMPS.err
+#SBATCH --mail-type=ALL
+#SBATCH --mail-user=YOUR-EMAIL-ADDRESS
+#SBATCH --time=01:00:00
+#SBATCH --ntasks=12
+#SBATCH --mem-per-cpu=4000
+#SBATCH --account=YOUR-GRUOP-NAME
+#SBATCH --qos=YOUR-GROUP-NAME
+#
+module load intel/2016.0.109 openmpi/1.10.2  lammps/7Dec15
-* Please see our [[Batch Script Explanation | annotated submission script]].
+LAMMPS=lmp_ufhpc.openmpi
+INPUT=test-input
-===Job Status===
+mpiexec $LAMMPS  < $INPUT
-You can view the job status of your jobs and your group's jobs via the [http://rc.ufl.edu/jobstatus/ PBS Job Status] page as described in [[PBS_Job_Status|PBS Job Status Documentation]]
-===Scratch Storage===
-See [[Scratch]] for notes on the main high-performance storage.
-===Developmental Nodes===
+</source></div></div>
-See [[Test_Nodes]] to read about the available interactive servers.

Difference between revisions of "Parallel Computing"

Latest revision as of 19:52, 16 January 2023

Contents

Types of parallel computing

Shared Memory vs. Distributed Memory

Communication In Parallel Computation

Run OpenMP Applications

Navigation menu

Search