Difference between revisions of "Modules"

Revision as of 02:10, 2 October 2021

Introduction

Setting and maintaining your environment while running a large number of diverse software applications with different requirements can be a daunting task. The Research Computing is adopting the modules system developed by Robert McLay at the Texas Advanced Computing Center (TACC) in order to ease the complexity of this process and relieve users of this burden.

Basic usage

See Modules Basic Usage for a short practical introduction to using modules.

Module Command

There is no executable called 'module' in Lmod. The 'module' command or its 'ml' alias, which you can use interactively are enabled when you source /etc/profile.d/modules.sh shell script inside another script or shell environment. It's done automatically for interactive sessions, but if you need access to 'module load' in a script running in a scheduled job source the above script first. Support for other shells and programming languages like python or R is also available (“sh”, “csh”, “perl”, “python”, “lisp”, “fish”, “cmake”, or “r”). See documentation for details.

Personal modules

You can write and use your own modulefiles. The procedure for making them available is very simple. For example, if you'd like to create version 1.0 of the module 'mymod' pick any directory you own, say ~/modules:

Create a personal module directory

mkdir ~/modules

Add a 'mymod' module directory

mkdir ~/modules/mymod

Add a modulefile

touch ~/modules/mymod/1.0.lua

Note that you can copy any modulefile we have in /apps/lmod/modulefiles to serve as a starting point for your 'mymod/1.0.lua' modulefile. To test/use the new module:

Run the following command or add it to your ~/.bashrc file

module use ~/modules

The "myapp/1.0" module should be available for your use.

Loading modules with Python

If you need to load an environment module from python set the sys.path to the Lmod installation on HPG. E.g.

import os, sys
sys.path.insert(0,"/apps/lmod/lmod/init/")
from env_modules_python import module
module("load","somemodule/version")

Support

To request help with using modules or installation of new module enabled application please file a support request.

Upstream Documentation

Those who would like to learn as much as possible about lmod can use this documentation.

There is a well-written article on Lmod in the HPC Admin Magazine.

@@ Line 1: / Line 1: @@
-[[Category:Software]][[Category:Biology]]
+[[Category:Software]][[Category:Infrastructure]]
 ==Introduction==
-Creating and maintaining a computing environment for a data analysis pipeline that consists of a large number of heterogenous tools, needs to be run by many users as must provide consistent data analysis results over a long time while frequent software upgrades and underlying operating system environment changes happen is not easy. Fortunately, the major points of this problem can be addressed by the use of an environmental modules system that abstracts the computing environment from the immediate data analysis steps. At the UF HPC we have chosen to implement this system using [http://www.tacc.utexas.edu/tacc-projects/mclay/lmod Lmod] software written by Robert McLay at the Texas Advanced Computing Center [http://www.tacc.utexas.edu/home (TACC)].
+Setting and maintaining your environment while running a large number of diverse software applications with different requirements can be a daunting task.  The Research Computing is adopting the ''modules'' system developed by Robert McLay at the Texas Advanced Computing Center [http://www.tacc.utexas.edu/home (TACC)] in order to ease the complexity of this process and relieve users of this burden.
+==Basic usage==
+See '''[[Modules Basic Usage]]''' for a short practical introduction to using ''modules''.
+==Module Command==
+There is no executable called 'module' in Lmod. The 'module' command or its 'ml' alias, which you can use interactively are enabled when you <code>source /etc/profile.d/modules.sh</code> shell script inside another script or shell environment. It's done automatically for interactive sessions, but if you need access to 'module load' in a script running in a scheduled job source the above script first. Support for other shells and programming languages like python or R is also available (“sh”, “csh”, “perl”, “python”, “lisp”, “fish”, “cmake”, or “r”). See documentation for details.
+==Personal modules==
+You can write and use your own modulefiles. The procedure for making them available is very simple. For example, if you'd like to create version 1.0 of the module 'mymod' pick any directory you own, say ~/modules:
+* Create a personal module directory
+ mkdir ~/modules
+* Add a 'mymod' module directory
+ mkdir ~/modules/mymod
+* Add a modulefile
+ touch ~/modules/mymod/1.0.lua
+Note that you can copy any modulefile we have in /apps/lmod/modulefiles to serve as a starting point for your 'mymod/1.0.lua' modulefile. To test/use the new module:
+* Run the following command or add it to your ~/.bashrc file
+ module use ~/modules
+The ''"myapp/1.0"'' module should be available for your use.
+==Loading modules with Python==
+If you need to load an environment module from python set the sys.path to the Lmod installation on HPG. E.g.
+<pre>
+import os, sys
+sys.path.insert(0,"/apps/lmod/lmod/init/")
+from env_modules_python import module
+module("load","somemodule/version")
+</pre>
+==Support==
+To request help with using modules or installation of new module enabled application please file a [https://support.rc.ufl.edu support request].
 ==Upstream Documentation==
-Those who would like to learn as much as possible about lmod can use the [http://www.tacc.utexas.edu/tacc-projects/mclay/lmod documentation written by the Lmod author].
+Those who would like to learn as much as possible about lmod can use this [http://lmod.readthedocs.io/en/latest/ documentation].
-==Basic Usage==
-The minimal information required for using a standalone application at UF HPC:
+There is a well-written article on Lmod in the [http://www.admin-magazine.com/HPC/Articles/Lmod-Alternative-Environment-Modules?utm_source=ADMIN+Newsletter&utm_campaign=HPC_Update_31_Lmod_Alternative_to_Environment_Modules_2013-01-30 HPC Admin Magazine].
- module
-will output help information on running the module command. The module help includes all information described below.
-===Loading a module===
-* If you already know the name of the module the only command needed to load it and be able to execute the software is 'module load <application>'. For instance, if the module is called '''ncbi_blast''' run the following command
- module load ncbi_blast
-===Unloading a module===
-* To remove a module, so its environment gets cleared run
- module unload ncbi_blast
-A module often does not have to be unloaded when you are done using an application. When you log out of your HPC session all modules are automatically unloaded.
-===Showing loaded modules===
- module list
-will show all currently active (loaded) modules.
-===Finding a module===
- module spider
-is a great time saver as it allow you to traverse the entire list of available modules along with short descriptions of the applications they make available.
-A much more concise, but context specific table of available modules can be obtained with
- module avail
-The "module avail" command shows all currently accessible branches of the global tree of modules, which depend on what compiler and MPI implementation modules are loaded. The ''core'' branch is always shown, but we also have ''compiler'' (intel/10.1 and intel/11.1) and mpi (mvapich/0.9.9, mpich2/1.0.8, mvapich2/1.4.1, openmpi/1.2.7, and openmpi/1.3.4 in the intel/10.1 branch as well as mvapich2/1.5.1, openmpi/1.3.4, and openmpi/1.4.3 in the intel/11.1 branch) implementation specific and even python version specific branches of the module tree. The top line at each section shown by 'module avail' shows exactly what loaded module is responsible for enabling that branch of the module tree. Sometimes, to be able to load a module that you find you using "module spider" you might have to go through the sequential steps of loading the respective branches. For instance, if you need to load the module for the "espresso" software that was built with MVAPICH2 support  using Intel Compiler version 11.1 you would run
- module load intel/11.1
- module load mvapich2
- module load expresso
-===Showing information about a module===
- module whatis app
-will show a short listing of module specific information such as name, version, category, upstream url, and a short description.
- module spider app
-or
- module spider app/version
-will show you the long description and the information on what versions are available for a particular application.
-===Swapping branches===
-To switch between different compiler and MPI implementation of a particular app or to gain access to a module that belongs to a particular branch run the following command to swap the old for the new branch
- module swap old new
-or if you loaded a particular version
- module swap old/version new/version
-If multiple versions of a particular app/version are available module will automatically reload those modules when you swap the branches, so you could seamlessly switch between a single-threaded version of an app you used for testing and an mpi version that will be used for a large-scale computing job.
-===Support===
-To request help with using modules or installation of new module enabled application please file a [http://support.hpc.ufl.edu support request] or if necessary please send us an [mailto:om@hpc.ufl.edu email].