Remote Jupyter Notebook

From UFRC
Jump to navigation Jump to search


Standalone Jupyter Notebook

Start a Jupyter notebook within a SLURM job on HiPerGator and connect to it from the web browser running on your local computer.

Note: Starting with jupyter/5.7.0 we are providing custom kernels named as 'RC-py3-$version' and 'RC-R-$version' that provide access to the default python3 and R environment modules in addition to standard built-in python and R kernels that come with Jupyter. Use the RC kernels if you'd like to access over 900 R packages and over 300 python3 modules we installed to support exploratory research and code writing by UF researchers. As always, use https://support.rc.ufl.edu to request package or module installs. Note that the shared python3 and R environments can only have one package/module version to avoid conflicts. Use python virtualenv to have custom module installs for particular projects.

If you would like to define your own kernel(s) put their directories into ${HOME}/.local/share/jupyter/kernels. You also need to make sure to install the 'ipykernel' package in your conda environment.

Interactive Session

If you're in a dev SLURM session then

  • Note the host name, which you'll need to create an SSH tunnel to your notebook.
  • Load jupyter module.
  • Start a jupyter notebook with launch_jupyter_notebook. Note the information printed to the console.
  • Create an SSH tunnel from your local computer to the notebook using SSH forwarding (see below).

SLURM Job

If you would like a notebook to live for longer than the 12-hour time limit for dev sessions start it inside a SLURM job.

  • If you don't want to go through copying/pasting the initial authentication token for every jupyter job set a default password for your notebooks with the following commands
module load jupyter
jupyter-notebook password
  • Create a job script similar to the example below. Adjust options as necessary.
#!/bin/bash
#SBATCH --job-name=jupyter
#SBATCH --output=jupyter_notebook_%j.log
#SBATCH --ntasks=1
#SBATCH --mem=2gb
#SBATCH --time=04:00:00
date;hostname;pwd
 
module add jupyter

launch_jupyter_notebook
Note
The jupyter environment includes all R and python packages/modules we installed on request.

Connection Information

Once the job starts look at the jupyter_notebook_$SLURM_JOBID.log SLURM output file to learn the hostname and the port jupyter notebook was started on. The ssh tunnel and local URI paths should already be there.

E.g.

$ cat jupyter_6595668.log 
Fri Apr 14 16:22:02 EDT 2017
c10b-s14.ufhpc
/home/jdoe

Starting Jupyter Notebook on port 23312 on the c10b-s14.ufhpc server.

SSH tunnel command: 
        ssh -NL 23312:c10b-s14.ufhpc:23312 jdoe@hpg.rc.ufl.edu

Local browser URI: 
        http://localhost:23312

If you have not set a default personal password with 'jupyter-notebook password' you will need to find and use the automatically generated authentication token, which is writtent to the job log once the notebook server starts. For example,


E.g.

[W 15:38:37.568 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
[I 15:38:38.859 NotebookApp] Serving notebooks from local directory: /home/jdoe
[I 15:38:38.859 NotebookApp] The Jupyter Notebook is running at:
[I 15:38:38.859 NotebookApp] http://(c100b-s41.ufhpc or 127.0.0.1):29926/?token=06b1c3f73bb847234c198a22bd62b7f20101b04d1bc2b64a
[I 15:38:38.859 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 15:38:38.862 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://(c100b-s1.ufhpc or 127.0.0.1):29926/?token=06b1c3f73bb847234c198a22bd62b7f20101b04d1bc2b64a
[I 15:39:09.643 NotebookApp] 302 GET / (172.16.206.12) 0.53ms
[I 15:39:09.648 NotebookApp] 302 GET /tree? (172.16.206.12) 0.66ms
[I 15:39:37.821 NotebookApp] 302 GET /?token=06b1c3f73bb847234c198a22bd62b7f20101b04d1bc2b64a (172.16.206.12) 0.37ms
[I 15:39:46.069 NotebookApp] Creating new notebook in
[I 15:40:05.840 NotebookApp] Kernel started: b5d6cea6-d214-4b37-b01e-4250ae6179c9
[I 15:40:13.516 NotebookApp] Adapting to protocol v5.0 for kernel b5d6cea6-d214-4b37-b01e-4250ae6179c9

Copy the token to use it as the password the first time you connect to the notebook. In this example the token is 06b1c3f73bb847234c198a22bd62b7f20101b04d1bc2b64a.

Create Tunnel From Local Machine

Copy paste the tunnel command from the job script or write your own based on how you manually started a notebook.

ssh -NL 23312:c10b-s14.ufhpc:23312 jdoe@hpg.rc.ufl.edu

Browse To Notebook

In a web browser on the local machine open http://localhost:23312

Provide the personal password or the token from the job log as the password to authenticate.

Enjoy using the rich python programming and visualization environment provided by Jupyter running on UFRC computing resources.

Again, note that the default Jupyter Notebook setup you see should have at least four kernels - two default kernels (python and R) that come with Jupyter and two additional kernels that provide access to environments provided by RC-specific environment modules e.g. 'RC R-3.5.1' and 'RC Py3-3.6.5', which match the same environment modules you use in batch jobs.

Accessing Blue and Orange Directories

Create a symlink (shortcut) to your blue or orange directory tree in your home directory. E.g.

cd
ln -s /blue/GROUP/USER/ blue
ln -s /orange/GROUP/ orange

Then, you'll see 'blue' or 'orange' as a link in your home directory in JupyterLab and will be able to double-click on it to browse to that directory tree.