Parallel Computing

Parallel computing refers to running multiple computational tasks simultaneously. The idea behind it is based on the assumption that a big computational task can be divided into smaller tasks which can run concurrently.

Types of parallel computing

Parallel computing is used only for the last row of below table;

	Single Instruction	Multiple Instructions	Single Program	Multiple Programs
Single Data	SISD	MISD
Multiple Data	SIMD	MIMD	SPMD	MPMD

In more details;

• Data-parallel(SIMD): Same operations/instructions are carried out on different data items, simultaneously.
• Task Parallel(MIMD): Different instructions on different data carried out concurrently.
• SPMD: Single program, multiple data, not synchronized at individual operation level

SPMD and MIMD are essentially the same because any MIMD can be made SPMD. SIMD is also equivalent, but in a less practical sense. MPI (Message Passing Interface) is primarily used for SPMD/MIMD.

Shared Memory vs. Distributed Memory

For explanation of the different memory models (shared, distributed, or hybrid), visit Memory: Shared vs Distributed

Communication In Parallel Computation

Communications in parallel computing takes advantage of one of the following interfaces;

• OpenMp
• MPI (MPI, OpenMPI)
• Hybrid

OpenMp is used for communication between tasks running concurrently on the same node with access to the shared memory. Assume you have a machine which each one of its nodes contains 16 cores with shared access to 32 GB of memory. If you have an application which is parallelized and can use up to 16 cores, the tasks running on each node will communicate using OpenMP.

Assume use of the same machine; if you want to run the same application on 16 nodes using only one core on each node, communication between different nodes is necessary since memory is not shared between nodes. MPI (Message Passing Interface) utilizes this communication. MPI is used for communication between tasks which use distributed memory.

Again, assume use of the same machine; what if you want to use two nodes (8 cores on each one). The tasks running on each node communicate using OpenMP while the tasks running on different nodes communicate using MPI. This communication style is called hybrid programming since it takes advantage of hybrid memory.

It is common to mistakenly assume OpenMP and OpenMPI are the same! But, OpenMPI is the name of recent MPI versions and should not be mistaken with OpenMP.

Run OpenMP Applications

Compiler	Compiler Options	Default behavior for # of threads (If not set)
GNU (gcc, g++, gfortran)	-fopenmp	as many threads as available cores
Intel (icc ifort)	-openmp	as many threads as available cores
Portland Group (pgcc,pgCC,pgf77,pgf90)	-mp	one thread

Sample job script: LAMMPS (MPI only)

Expand this section to view sample jobs script.

#!/bin/bash
#SBATCH --job-name=LAMMPS-JOB
#SBATCH --output=LAMMPS.out
#SBATCH --error=LAMMPS.err
#SBATCH --mail-type=ALL
#SBATCH --mail-user=YOUR-EMAIL-ADDRESS
#SBATCH --time=01:00:00
#SBATCH --ntasks=12
#SBATCH --mem-per-cpu=4000
#SBATCH --account=YOUR-GRUOP-NAME
#SBATCH --qos=YOUR-GROUP-NAME
#
module load intel/2016.0.109 openmpi/1.10.2  lammps/7Dec15

LAMMPS=lmp_ufhpc.openmpi
INPUT=test-input

mpiexec $LAMMPS  < $INPUT