Parallel Computing
Parallel computing refers to running multiple computational tasks simultaneously. The idea behind it is based on the assumption that a big computational task can be divided into smaller tasks which can run concurrently.
Types of parallel computing
Parallel computing is used only for the last row of below table;
| Single Instruction | Multiple Instructions | Single Program | Multiple Programs | |
|---|---|---|---|---|
| Single Data | SISD | MISD | ||
| Multiple Data | SIMD | MIMD | SPMD | MPMD |
In more details;
- Data-parallel(SIMD): Same operations/instructions are carried out on different data items, simultaneously.
- Task Parallel(MIMD): Different instructions on different data carried out concurrently.
- SPMD: Single program, multiple data, not synchronized at individual operation level
SPMD and MIMD are essentially the same because any MIMD can be made SPMD. SIMD is also equivalent, but in a less practical sense. MPI (Message Passing Interface) is primarily used for SPMD/MIMD.
For explanation of the different memory models (shared, distributed, or hybrid), visit Memory: Shared vs Distributed
Communication In Parallel Computation
Communications in parallel computing takes advantage of one of the following interfaces;
- OpenMp
- MPI (MPI, OpenMPI)
- Hybrid
OpenMp is used for communication between tasks running concurrently on the same node with access to the shared memory. Assume you have a machine which each one of its nodes contains 16 cores with shared access to 32 GB of memory. If you have an application which is parallelized and can use up to 16 cores, the tasks running on each node will communicate using OpenMP.
Assume use of the same machine; if you want to run the same application on 16 nodes using only one core on each node, communication between different nodes is necessary since memory is not shared between nodes. MPI (Message Passing Interface) utilizes this communication. MPI is used for communication between tasks which use distributed memory.
Again, assume use of the same machine; what if you want to use two nodes (8 cores on each one). The tasks running on each node communicate using OpenMP while the tasks running on different nodes communicate using MPI. This communication style is called hybrid programming since it takes advantage of hybrid memory.
It is common to mistakenly assume OpenMP and OpenMPI are the same! But, OpenMPI is the name of recent MPI versions and should not be mistaken with OpenMP.
Run OpenMP Applications
| Compiler | Compiler Options | Default behavior for # of threads (If not set) |
|---|---|---|
| GNU (gcc, g++, gfortran) | -fopenmp | as many threads as available cores |
| Intel (icc ifort) | -openmp | as many threads as available cores |
| Portland Group (pgcc,pgCC,pgf77,pgf90) | -mp | one thread |
Sample job script: LAMMPS (MPI only)
#!/bin/bash
#SBATCH --job-name=LAMMPS-JOB
#SBATCH --output=LAMMPS.out
#SBATCH --error=LAMMPS.err
#SBATCH --mail-type=ALL
#SBATCH --mail-user=YOUR-EMAIL-ADDRESS
#SBATCH --time=01:00:00
#SBATCH --ntasks=12
#SBATCH --mem-per-cpu=4000
#SBATCH --account=YOUR-GRUOP-NAME
#SBATCH --qos=YOUR-GROUP-NAME
#
module load intel/2016.0.109 openmpi/1.10.2 lammps/7Dec15
LAMMPS=lmp_ufhpc.openmpi
INPUT=test-input
mpiexec $LAMMPS < $INPUT