Parallel Computing

Jump to navigation Jump to search

Parallel computing refers to running multiple computational tasks simultaneously. The idea behind it is based on the assumption that a big computational task can be divided into smaller tasks which can run concurrently.

Types of parallel computing

Parallel computing is used only for the last row of below table;

Single Instruction Multiple Instructions Single Program Multiple Programs
Single Data SISD MISD

In more details;

  • Data-parallel(SIMD): Same operations/instructions are carried out on different data items, simultaneously.
  • Task Parallel(MIMD): Different instructions on different data carried out concurrently.
  • SPMD: Single program, multiple data, not synchronized at individual operation level

SPMD and MIMD are essentially the same because any MIMD can be made SPMD. SIMD is also equivalent, but in a less practical sense. MPI (Message Passing Interface) is primarily used for SPMD/MIMD.

Shared Memory vs. Distributed Memory

For explanation of the different memory models (shared, distributed, or hybrid), visit Memory: Shared vs Distributed

Communication In Parallel Computation

Communications in parallel computing takes advantage of one of the following interfaces;

  • OpenMp
  • MPI (MPI, OpenMPI)
  • Hybrid

OpenMp is used for communication between tasks running concurrently on the same node with access to the shared memory. Assume you have a machine which each one of its nodes contains 16 cores with shared access to 32 GB of memory. If you have an application which is parallelized and can use up to 16 cores, the tasks running on each node will communicate using OpenMP.

Assume use of the same machine; if you want to run the same application on 16 nodes using only one core on each node, communication between different nodes is necessary since memory is not shared between nodes. MPI (Message Passing Interface) utilizes this communication. MPI is used for communication between tasks which use distributed memory.

Again, assume use of the same machine; what if you want to use two nodes (8 cores on each one). The tasks running on each node communicate using OpenMP while the tasks running on different nodes communicate using MPI. This communication style is called hybrid programming since it takes advantage of hybrid memory.

It is common to mistakenly assume OpenMP and OpenMPI are the same! But, OpenMPI is the name of recent MPI versions and should not be mistaken with OpenMP.

Run OpenMP Applications

Compiler Compiler Options Default behavior for # of threads (If not set)
GNU (gcc, g++, gfortran) -fopenmp as many threads as available cores
Intel (icc ifort) -openmp as many threads as available cores
Portland Group (pgcc,pgCC,pgf77,pgf90) -mp one thread

Sample job script: LAMMPS (MPI only)

Expand this section to view sample job script.

#SBATCH --output=LAMMPS.out
#SBATCH --error=LAMMPS.err
#SBATCH --mail-type=ALL
#SBATCH --time=01:00:00
#SBATCH --ntasks=12
#SBATCH --mem-per-cpu=4000
module load intel/2016.0.109 openmpi/1.10.2  lammps/7Dec15


mpiexec $LAMMPS  < $INPUT