Difference between revisions of "Singularity"

From UFRC
Jump to navigation Jump to search
 
(6 intermediate revisions by 3 users not shown)
Line 1: Line 1:
[[Category:Containers]]
+
[[Category:Software]][[Category:Workflow Automation]]
We already use and welcome your use of [http://singularity.lbl.gov/ Singularity Containers] on HiPerGator. As the Singularity website says you can make and customize your containers locally, copy them to your or your group's shared /ufrc space on HiPerGator and execute programs installed within whether in [[Development_and_Testing|a dev session]] or in [[Sample_SLURM_Scripts|batch jobs]]. In essence it would allow you to develop your analytical workflow on your local machine and bring it as is - ''Bring Your Own Environment'' to the cluster.
+
Singularity is deprecated. For updated version see [[Apptainer]].
 +
 
 +
We already use and welcome your use of [http://singularity.lbl.gov/ Singularity Containers] on HiPerGator. As the Singularity website says you can make and customize your containers locally, copy them to your or your group's shared /blue space on HiPerGator and execute programs installed within whether in [[Development_and_Testing|a dev session]] or in [[Sample_SLURM_Scripts|batch jobs]]. In essence it would allow you to develop your analytical workflow on your local machine and bring it as is - ''Bring Your Own Environment'' - to the cluster.
  
 
While [http://singularity.lbl.gov/quickstart Singularity Documentation] is the best place to find up-to-date and detailed information about using Singularity there some HiPerGator specific details that we'd like to point out to reduce the number of warnings you see and potential pitfalls you might encounter.
 
While [http://singularity.lbl.gov/quickstart Singularity Documentation] is the best place to find up-to-date and detailed information about using Singularity there some HiPerGator specific details that we'd like to point out to reduce the number of warnings you see and potential pitfalls you might encounter.
Line 6: Line 8:
 
;Image Builds:
 
;Image Builds:
 
When building a new container image create the following directories that our Singularity install tries to automatically mount to avoid warnings or running into unavailability of those filesystems in the course of your normal work.
 
When building a new container image create the following directories that our Singularity install tries to automatically mount to avoid warnings or running into unavailability of those filesystems in the course of your normal work.
  mkdir -p /ufrc /bio /rlts /scratch/local
+
  mkdir -p /blue /orange /scratch/local
  
 
;Image Location:
 
;Image Location:
The /ufrc ''Blue Storage'' filesystem is the best suited for container image storage since it highly performant when reading large files such as container images in parallel. You are unlikely to be able to fit container image into your /home quote, but we'd like to emphasize that you should not do it even if you can to avoid performance issues and impacting other users.
+
The /blue ''Blue Storage'' filesystem is the best suited for container image storage since it highly performant when reading large files such as container images in parallel. You are unlikely to be able to fit container image into your /home quote, but we'd like to emphasize that you should not do it even if you can to avoid performance issues and impacting other users.
  
 
;Hub image downloads:
 
;Hub image downloads:
When you run a container from an image stored in the Singularity Hub the container image will be downloaded to the current working directory. Make sure you run the command while on /ufrc filesystem and not in your home directory. E.g.
+
When you run a container from an image stored in the Singularity Hub the container image will be downloaded to the current working directory. Make sure you run the command while on /blue filesystem and not in your home directory. E.g.
 
  $ module load singularity
 
  $ module load singularity
 
  $ singularity run shub://GodloveD/lolcow
 
  $ singularity run shub://GodloveD/lolcow
 +
 +
or with Docker container:
 +
 +
$ singularity run docker://godlovedc/lolcow
 
...
 
...
 
  $ ls *.simg
 
  $ ls *.simg
Line 20: Line 26:
  
 
;Docker hub caching:
 
;Docker hub caching:
When you run a Docker container from Docker Hub using Singularity the contents of the container image will be cached in your home directory in the '<code>~/.singularity/docker</code>'. You should either monitor the /home quota usage by the cache if you frequently use docker hub images or symlink that directory to your /ufrc directory. E.g.
+
When you run a Docker container from Docker Hub using Singularity the contents of the container image will be cached in your home directory in the '<code>~/.singularity/docker</code>'. You should either monitor the /home quota usage by the cache if you frequently use docker hub images or symlink that directory to your /blue directory. E.g.
  ln -s /ufrc/my_group/$USER/singularity/ ~/.singularity
+
  ln -s /blue/my_group/$USER/singularity/ ~/.singularity
Singularity will detect the symlink and set the cache to the target directory in /ufrc.
+
Singularity will detect the symlink and set the cache to the target directory in /blue.

Latest revision as of 21:30, 6 February 2023

Singularity is deprecated. For updated version see Apptainer.

We already use and welcome your use of Singularity Containers on HiPerGator. As the Singularity website says you can make and customize your containers locally, copy them to your or your group's shared /blue space on HiPerGator and execute programs installed within whether in a dev session or in batch jobs. In essence it would allow you to develop your analytical workflow on your local machine and bring it as is - Bring Your Own Environment - to the cluster.

While Singularity Documentation is the best place to find up-to-date and detailed information about using Singularity there some HiPerGator specific details that we'd like to point out to reduce the number of warnings you see and potential pitfalls you might encounter.

Image Builds

When building a new container image create the following directories that our Singularity install tries to automatically mount to avoid warnings or running into unavailability of those filesystems in the course of your normal work.

mkdir -p /blue /orange /scratch/local
Image Location

The /blue Blue Storage filesystem is the best suited for container image storage since it highly performant when reading large files such as container images in parallel. You are unlikely to be able to fit container image into your /home quote, but we'd like to emphasize that you should not do it even if you can to avoid performance issues and impacting other users.

Hub image downloads

When you run a container from an image stored in the Singularity Hub the container image will be downloaded to the current working directory. Make sure you run the command while on /blue filesystem and not in your home directory. E.g.

$ module load singularity
$ singularity run shub://GodloveD/lolcow

or with Docker container:

$ singularity run docker://godlovedc/lolcow

...

$ ls *.simg

GodloveD-lolcow-master.simg

Docker hub caching

When you run a Docker container from Docker Hub using Singularity the contents of the container image will be cached in your home directory in the '~/.singularity/docker'. You should either monitor the /home quota usage by the cache if you frequently use docker hub images or symlink that directory to your /blue directory. E.g.

ln -s /blue/my_group/$USER/singularity/ ~/.singularity

Singularity will detect the symlink and set the cache to the target directory in /blue.