Persistent Terminal Sessions

Introduction

SSH sessions can be interrupted by a networking issue. Sometimes it is necessary to have a long-term session on a login node say for an interactive SLURM job, monitoring an analysis, or editing job scripts or configuration files, for example. There are several methods to get a long-term terminal session like starting a terminal within JupyterLab, OnDemand session, or using a terminal multiplexer. We have two such applications on HiPerGator: Screen and Tmux. They are similar implementations of the concept of 'terminal multiplexing' or having multiple terminals in a single instance of the program.

Workflow

In short, working with a terminal multiplexer to have multiple persistent (do not disappear when you log out or your network connection drops) terminals on HPG login nodes usually goes as follows

1. Ssh into HiPerGator. You end up on a random login node. E.g.

   • $ ssh myname@hpg.rc.ufl.edu

2. Pick a login node for your session and ssh into it. E.g.

   • $ ssh login2

3. Start a tmux or screen. E.g.

   • $ ml screen

   • $ screen

4. Create as many terminal windows in the screen session as needed with 'Ctrl+a c' key combination. See Screen documentation.
5. Now, if you start an interactive SLURM job in one of the terminals in your screen session that job will still be running if you disconnect and re-connect.
6. Detach (disconnect) when needed with 'Ctrl+a d'
7. Ssh back into the login node you picked later. Note: this is important - ssh into the same node you started the session on as the screen sessions are per-node.
8. Reattach to the screen session(re-attach)

   • $ ml screen

   • $ screen -dr

Warning: again, you cannot be sure which login node you'll end up on after you ssh into HiPerGator from your local computer. Remember which node you started your screen session on and ssh into it once you are on the HiPerGator. E.g. if you started your session on login3, but end up on login2 after reconnecting you can ssh into login3 internally with $ ssh login3.

Any login node can be rebooted for maintenance. All screen sessions on the node will be lost on reboot. You will be warned with a node-wide terminal message before that happens if this is a scheduled maintenance as opposed to a node problem. Let our sysadmins know if something critical is going on and ask them to wait if possible.

Using Screen

To use GNU Screen as a terminal multiplexer on HiPerGator load the 'screen' environment module with module load screen and start Screen with the screen command. At first glance nothing will happen because the default screen configuration is pretty spartan, but you can copy our example 'fancy' screen configuration with cp $HPC_SCREEN_CONF ~ to get a much nicer view and to serve as a starting point for your own customizations. Since the GNU Screen Manual is very comprehensive, but may take effort to navigate I would recommend doing a web search for one of hundreds of screen tutorials. It's a very popular tool, so there is no lack of more accessible documentation. As a minimum the following commands are useful. Note that 'Ctrl+A' means that you hold the CTRL key on the keyboard, then press a and release both keys before typing the key that corresponds to the command you want to issue to the screen. The commands with '$' are shell commands, whereas all 'Ctrl+a...' commands are issued inside a screen session.

• Load the module

$ module load screen

• Start a new screen

$ screen

• Create a new terminal window in a screen session

Ctrl+a c

• Switch to numbered window #0 (first window)

Ctrl+a 0

• Switch to the next window

Ctrl+a n

• Switch to the previous window

Ctrl+a p

• Detach from the session now

Ctrl+a d

• Re-attach to the screen session after logging into the same interactive node

$ screen -dr

Using tmux

Tmux is a newer and arguably more 'powerful' approach to terminal multiplexing than GNU Screen, but ultimately, it's a matter of choice. Use tmux if it suits your work style better.