Difference between revisions of "Parallel Computing"

From UFRC
Jump to navigation Jump to search
 
(22 intermediate revisions by 4 users not shown)
Line 1: Line 1:
−
 
+
[[Category:Documentation‏]]
 
{|align=right
 
{|align=right
 
   |__TOC__
 
   |__TOC__
 
   |}
 
   |}
−
==Parallel Computing==
 
 
Parallel computing refers to running multiple computational tasks simultaneously. The idea behind it is based on the assumption that a big computational task can be divided into smaller tasks which can run concurrently.
 
Parallel computing refers to running multiple computational tasks simultaneously. The idea behind it is based on the assumption that a big computational task can be divided into smaller tasks which can run concurrently.
  
Line 29: Line 28:
  
 
== Shared Memory vs. Distributed Memory ==
 
== Shared Memory vs. Distributed Memory ==
 +
For explanation of the different memory models (shared, distributed, or hybrid), visit [[Memory: Shared vs Distributed]]
  
−
=== Shared Memory ===
+
== Communication In Parallel Computation ==
  
−
Shared memory is the memory which all the processors can access. In hardware point of view it means all the processors have direct access to the common physical memory through bus based (usually using wires) access. These processors can work independently while they all access the same memory. Any change in the variables stored in the memory is visible by all processors because at any given moment all they see is a copy or picture of entire variables stored in the memory and they can directly address and access the same logical memory locations regardless of where the physical memory actually exists.
+
Communications in parallel computing takes advantage of one of the following interfaces;
  
−
Uniform Memory Access (UMA):
+
* OpenMp
 +
* MPI (MPI, OpenMPI)
 +
* Hybrid
  
−
* Most commonly represented today by Symmetric Multiprocessor (SMP) machines
+
OpenMp is used for communication between tasks running concurrently on the same node with access to the shared memory.
−
* Identical processors
+
Assume you have a machine which each one of its nodes contains 16 cores with shared access to 32 GB of memory. If you have an application which is parallelized and can use up to 16 cores, the tasks running on each node will communicate using OpenMP.
−
* Equal access and access times to memory
 
−
* Sometimes called CC-UMA - Cache Coherent UMA. Cache coherent means if one processor updates a location in shared memory, all the other processors know about the update. Cache coherency is accomplished at the hardware level.
 
  
 +
Assume use of the same machine; if you want to run the same application on 16 nodes using only one core on each node, communication between different nodes is necessary since memory is not shared between nodes. MPI (Message Passing Interface) utilizes this communication. MPI is used for communication between tasks which use distributed memory.
  
 +
Again, assume use of the same machine; what if you want to use two nodes (8 cores on each one). The tasks running on each node communicate using OpenMP while the tasks running on different nodes communicate using MPI. This communication style is called hybrid programming since it takes advantage of hybrid memory.
  
−
Non-Uniform Memory Access (NUMA):
+
It is common to mistakenly assume OpenMP and OpenMPI are the same! But, OpenMPI is the name of recent MPI versions and should not be mistaken with OpenMP.
  
−
* Often made by physically linking two or more SMPs
+
=== Run OpenMP Applications ===
−
* One SMP can directly access memory of another SMP
 
−
* Not all processors have equal access time to all memories
 
−
* Memory access across link is slower
 
−
* If cache coherency is maintained, then may also be called CC-NUMA - Cache Coherent NUMA
 
  
 +
{| class="wikitable"
 +
|-
 +
! Compiler !! Compiler Options !! Default behavior for # of threads (If not set)
 +
|-
 +
| GNU (gcc, g++, gfortran) || -fopenmp || as many threads as available cores
 +
|-
 +
| Intel (icc ifort) || -openmp || as many threads as available cores
 +
|-
 +
| Portland Group (pgcc,pgCC,pgf77,pgf90) || -mp || one thread
 +
|}
  
−
=== Distributed Memory ===
+
Sample job script: LAMMPS (MPI only)
−
 
+
<div class="mw-collapsible mw-collapsed" style="width:70%; padding: 5px; border: 1px solid gray;">
−
Distributed memory in hardware sense, refers to the case where the processors can access other processor's memory only through network. In software sense, it means each processor only can see local machine memory directly and should use communications through network to access memory of the other processors.
+
''Expand this section to view sample job script.''
 +
<div class="mw-collapsible-content" style="padding: 5px;">
 +
<source lang=bash>
 +
#!/bin/bash
 +
#SBATCH --job-name=LAMMPS-JOB
 +
#SBATCH --output=LAMMPS.out
 +
#SBATCH --error=LAMMPS.err
 +
#SBATCH --mail-type=ALL
 +
#SBATCH --mail-user=YOUR-EMAIL-ADDRESS
 +
#SBATCH --time=01:00:00
 +
#SBATCH --ntasks=12
 +
#SBATCH --mem-per-cpu=4000
 +
#SBATCH --account=YOUR-GRUOP-NAME
 +
#SBATCH --qos=YOUR-GROUP-NAME
 +
#
 +
module load intel/2016.0.109 openmpi/1.10.2  lammps/7Dec15
  
 +
LAMMPS=lmp_ufhpc.openmpi
 +
INPUT=test-input
  
−
=== Hybrid ===
+
mpiexec $LAMMPS  < $INPUT
  
−
Combination of the two kinds of memory is what usually is used in today's fast supercomputers. The hybrid memory system is basically a network of shared memories. Within each shades component, the memory is accessible to all the cpus, and in addition, they can access the tasks and information stored on other units through the network.
+
</source></div></div>

Latest revision as of 19:52, 16 January 2023

Parallel computing refers to running multiple computational tasks simultaneously. The idea behind it is based on the assumption that a big computational task can be divided into smaller tasks which can run concurrently.

Types of parallel computing

Parallel computing is used only for the last row of below table;

Single Instruction Multiple Instructions Single Program Multiple Programs
Single Data SISD MISD
Multiple Data SIMD MIMD SPMD MPMD

In more details;

  • Data-parallel(SIMD): Same operations/instructions are carried out on different data items, simultaneously.
  • Task Parallel(MIMD): Different instructions on different data carried out concurrently.
  • SPMD: Single program, multiple data, not synchronized at individual operation level

SPMD and MIMD are essentially the same because any MIMD can be made SPMD. SIMD is also equivalent, but in a less practical sense. MPI (Message Passing Interface) is primarily used for SPMD/MIMD.

Shared Memory vs. Distributed Memory

For explanation of the different memory models (shared, distributed, or hybrid), visit Memory: Shared vs Distributed

Communication In Parallel Computation

Communications in parallel computing takes advantage of one of the following interfaces;

  • OpenMp
  • MPI (MPI, OpenMPI)
  • Hybrid

OpenMp is used for communication between tasks running concurrently on the same node with access to the shared memory. Assume you have a machine which each one of its nodes contains 16 cores with shared access to 32 GB of memory. If you have an application which is parallelized and can use up to 16 cores, the tasks running on each node will communicate using OpenMP.

Assume use of the same machine; if you want to run the same application on 16 nodes using only one core on each node, communication between different nodes is necessary since memory is not shared between nodes. MPI (Message Passing Interface) utilizes this communication. MPI is used for communication between tasks which use distributed memory.

Again, assume use of the same machine; what if you want to use two nodes (8 cores on each one). The tasks running on each node communicate using OpenMP while the tasks running on different nodes communicate using MPI. This communication style is called hybrid programming since it takes advantage of hybrid memory.

It is common to mistakenly assume OpenMP and OpenMPI are the same! But, OpenMPI is the name of recent MPI versions and should not be mistaken with OpenMP.

Run OpenMP Applications

Compiler Compiler Options Default behavior for # of threads (If not set)
GNU (gcc, g++, gfortran) -fopenmp as many threads as available cores
Intel (icc ifort) -openmp as many threads as available cores
Portland Group (pgcc,pgCC,pgf77,pgf90) -mp one thread

Sample job script: LAMMPS (MPI only)

Expand this section to view sample job script. 

#!/bin/bash
#SBATCH --job-name=LAMMPS-JOB
#SBATCH --output=LAMMPS.out
#SBATCH --error=LAMMPS.err
#SBATCH --mail-type=ALL
#SBATCH --mail-user=YOUR-EMAIL-ADDRESS
#SBATCH --time=01:00:00
#SBATCH --ntasks=12
#SBATCH --mem-per-cpu=4000
#SBATCH --account=YOUR-GRUOP-NAME
#SBATCH --qos=YOUR-GROUP-NAME
#
module load intel/2016.0.109 openmpi/1.10.2  lammps/7Dec15

LAMMPS=lmp_ufhpc.openmpi
INPUT=test-input

mpiexec $LAMMPS  < $INPUT
Retrieved from "https://help.rc.ufl.edu/mediawiki/index.php?title=Parallel_Computing&oldid=27253"