RAxML

Description

raxml website  

RAxML (Randomized Axelerated Maximum Likelihood) written by Alexandros Stamatakis and others is a program for sequential and parallel Maximum Likelihood based inference of large phylogenetic trees. It was originally derived from fastDNAml which in turn was derived from Joe Felsentein’s dnaml which is part of the PHYLIP package.

Required Modules

modules documentation

Serial and Parallel (Threads)

• raxml

Parallel (MPI)

• gcc/4.7.2
• openmpi/1.6.5
• raxml

System Variables

• HPC_{{#uppercase:raxml}}_DIR - installation directory

How To Run

Please see the discussion below on performance characteristics of the different implementations of RAxML. In general, there are four different RAxML executables installed on the Research Computing systems. They are

1. Single-threaded (i.e. serial)
2. Multi-threaded (i.e. parallel)
3. MPI (distributed memory, parallel)
4. Hybrid (MPI for coarse-grained parallelism with threading for fine-grained parallelism)

Each of these is described below, along with notes on running them.

SIMD Extensions

SSE3 and AVX are extensions to the x86 instruction set that accelerate single-instruction, multiple-data (SIMD) computations - often referred to as vectorization. For each of the four RAxML executables, we have both an SSE3 and an AVX version compiled. Although HiPerGator compute servers support AVX instructions, our testing indicates that the SSE3 executable is still faster - probably due to memory bandwidth constraints. Users are encouraged to experiment with their particular dataset but otherwise should default to the SSE3 executable. To run the AVX version of RAxML, replace "SSE3" in the executable name with "AVX".

Single-Threaded (Serial)

The serial version of RAxML is called raxmlHPC-SSE3, and is a single-threaded application.

Multi-Threaded (Parallel)

The multi-threaded version of RAxML is called raxmlHPC-PTHREADS-SSE3, and can use multiple processors on a single compute sever (or node). This version implements the fine-grained parallelism as discussed below. Resource requests for this version should always be in the form of nodes=1:ppn=x, where x is the number of processors to use. Please see the information below when selecting the number of processors to use. In our testing, values over 8 do not significantly speed up analyses and should be avoided. It is important to use the -T flag which tells RAxML how many processors to use. You can either put the same number used in the resource request, or use the PBS environment variable $PBS_NUM_PPN which is set for you by the scheduler--e.g. -T $PBS_NUM_PPN.

MPI (Parallel)

The distributed memory version of RAxML utilizes the MPI API. The executable is called raxmlHPC-MPI-SSE3 and can use multiple processors that may, or may not be, on the same compute server (node). This version implements the course-grained parallelism as discussed below. Resource requests for this version should generally be in the form of nodes=1:ppn=x, where x is the number of processors to use, as long as x is less than 32. If you want to use more than 32 processors, you should generally ask for more nodes.

Hybrid (Parallel)

The hybrid (MPI and multi-threading) executable of RAxML is called raxmlHPC-HYBRID-SSE3 and uses multiple processors on multiple compute servers. It implements both course-grained and fine-grained parallelism as discussed below. Resource requests for this version should be in the form of nodes=x:ppn=y. As with the MPI executable, if the total number of processors desired is 32 or less, the resource request should be nodes=1:ppn=y and you should "mpiexec -np <number of course-grained processes>" and "-T <number of fine-grained threads>" such that the product of the two equals y.

For example to run 5 course-grained processes, each of which using 4 fine-grained threads, the following resource request and command line is suggested.

 #PBS -l nodes=1:ppn=20
 ...
 mpiexec -bynode -np 5 raxmlHPC-HYBRID-SSE3 -T 4 ...

If you require more than 32 cores total, it is best to use multiple nodes. In this case, the number of nodes and processors per node should correspond to the number of course-grained and fine-grained threads requested. For example,

 #PBS -l nodes=10:ppn=4
 ...
 mpiexec -bynode -np 10 raxmlHPC-HYBRID-SSE3 -T 4 ...

Performance

We highly recommend that users read the paper by Pfeiffer and Stamatakis (2010) before running parallel versions of RAxML. This paper provides a good overview of the different types of parallelism implemented in RAxML and how to best leverage them for analyses. The discussion below is largely based on this paper.

Parallelism in RAxML

RAxML implements two different types of parallelism, referred to as course-grained and fine-grained. Course-grained parallelism is able to be split across multiple compute servers. Each course-grained process can work on one tree optimization. This may be a bootstrap replicate or a ML search. Fine-grained parallelism allows multiple processors on the SAME server to split up a singe tree optimization. A single optimization cannot be split across servers.

If the user is running the -f a option (bootstrap search and ML search in one analysis) using the MPI or Hybrid executabls, the bootstrap replicates are split among the MPI processes, and once those are complete, each MPI process does an independent ML search. This is slightly different than under other methods as multiple ML searches are being performed. While this is likely a good thing in terms of finding the ML tree and a thorough analysis, users should understand that this stage will not see a reduction in run time because each MPI task is doing an independent search, rather than working together on a single search.