Difference between revisions of "Remote Jupyter Notebook"

From UFRC
Jump to navigation Jump to search
Line 3: Line 3:
 
=JupyterHub=
 
=JupyterHub=
 
If you are looking for a convenient way to run JupyterLab notebooks try [https://jhub.rc.ufl.edu UFRC JupyterHub] service. It presents a convenient web interface to start notebooks, consoles, or terminals with multiple custom kernels and several job resource request profiles, which we can expand on request to satisfy your needs.  
 
If you are looking for a convenient way to run JupyterLab notebooks try [https://jhub.rc.ufl.edu UFRC JupyterHub] service. It presents a convenient web interface to start notebooks, consoles, or terminals with multiple custom kernels and several job resource request profiles, which we can expand on request to satisfy your needs.  
 +
 +
==[https://mediasite.video.ufl.edu/Mediasite/Play/8efcf534ef3c408e9238d8deeeda083a1d Using Jupyter Notebooks on HiPerGator]==
 +
[[File:Play_icon.png|frameless|30px|link=https://mediasite.video.ufl.edu/Mediasite/Play/8efcf534ef3c408e9238d8deeeda083a1d]] [8 min, 1 sec]
 +
This video covers the use of Jupyter Notebooks via https://jhub.rc.ufl.edu/ to run Python, R and other notebooks on HiPerGator.
 +
* Connect to https://jhub.rc.ufl.edu/
 +
* Launch notebooks in various coding languages
 +
* Launch a terminal
 +
* Upload and download files via your browser
 +
 
==Available Kernels==
 
==Available Kernels==
 
* 'Python3 3.7 (basic)' - the default python3 kernel from the jupyterhub, which doesn't have much in it.
 
* 'Python3 3.7 (basic)' - the default python3 kernel from the jupyterhub, which doesn't have much in it.

Revision as of 15:21, 10 February 2020


JupyterHub

If you are looking for a convenient way to run JupyterLab notebooks try UFRC JupyterHub service. It presents a convenient web interface to start notebooks, consoles, or terminals with multiple custom kernels and several job resource request profiles, which we can expand on request to satisfy your needs.

Using Jupyter Notebooks on HiPerGator

Play icon.png [8 min, 1 sec] This video covers the use of Jupyter Notebooks via https://jhub.rc.ufl.edu/ to run Python, R and other notebooks on HiPerGator.

  • Connect to https://jhub.rc.ufl.edu/
  • Launch notebooks in various coding languages
  • Launch a terminal
  • Upload and download files via your browser

Available Kernels

  • 'Python3 3.7 (basic)' - the default python3 kernel from the jupyterhub, which doesn't have much in it.
  • 'Python3 3.6 (full)' - our main HiPerGator python3/3.6 environment module with "Everything and the kitchen sink" in it as far as python modules are concerned.
  • 'PyViz-0.10.0' - special environment for https://pyviz.org/ based data analysis and plotting environment.
  • 'R 3.6 (full)' - our main R/3.6 environment module from HiPerGator with everything and a kitchen sink as far as packages are concerned.

Packages and modules in the full python3 and R environments are installed on request (https://support.rc.ufl.edu).

Standalone Jupyter Notebook

Start a Jupyter notebook within a SLURM job on HiPerGator and connect to it from the web browser running on your local computer.

Note: Starting with jupyter/5.7.0 we are providing custom kernels named as 'RC-py3-$version' and 'RC-R-$version' that provide access to the default python3 and R environment modules in addition to standard built-in python and R kernels that come with Jupyter. Use the RC kernels if you'd like to access over 900 R packages and over 300 python3 modules we installed to support exploratory research and code writing by UF researchers. As always, use https://support.rc.ufl.edu to request package or module installs. Note that the shared python3 and R environments can only have one package/module version to avoid conflicts. Use python virtualenv to have custom module installs for particular projects.

Interactive Session

If you're in a dev SLURM session then

  • Note the host name, which you'll need to create an SSH tunnel to your notebook.
  • Load jupyter module.
  • Start a jupyter notebook with launch_jupyter_notebook. Note the information printed to the console.
  • Create an SSH tunnel from your local computer to the notebook using SSH forwarding (see below).

SLURM Job

If you would like a notebook to live for longer than the 12-hour time limit for dev sessions start it inside a SLURM job.

  • If you don't want to go through copying/pasting the initial authentication token for every jupyter job set a default password for your notebooks with the following commands
module load jupyter
jupyter-notebook password
  • Create a job script or download the following pre-made script that will use your default account and qos. Adjust as necessary.

Download the [{{#fileLink: jupyter_job.sh}} jupyter job script]. {{#fileAnchor: jupyter_job.sh}}

#!/bin/bash
#SBATCH --job-name=jupyter
#SBATCH --output=jupyter_notebook_%j.log
#SBATCH --ntasks=1
#SBATCH --mem=2gb
#SBATCH --time=04:00:00
date;hostname;pwd
 
module add jupyter

launch_jupyter_notebook
Note
The jupyter environment includes all R and python packages/modules we installed on request.

Connection Information

Once the job starts look at the jupyter_notebook_$SLURM_JOBID.log SLURM output file to learn the hostname and the port jupyter notebook was started on. The ssh tunnel and local URI paths should already be there.

E.g.

$ cat jupyter_6595668.log 
Fri Apr 14 16:22:02 EDT 2017
c10b-s14.ufhpc
/home/jdoe

Starting Jupyter Notebook on port 23312 on the c10b-s14.ufhpc server.

SSH tunnel command: 
        ssh -NL 23312:c10b-s14.ufhpc:23312 jdoe@hpg.rc.ufl.edu

Local browser URI: 
        http://localhost:23312

If you have not set a default personal password with 'jupyter-notebook password' you will need to find and use the automatically generated authentication token, which is writtent to the job log once the notebook server starts. For example,


E.g.

[W 15:38:37.568 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
[I 15:38:38.859 NotebookApp] Serving notebooks from local directory: /home/jdoe
[I 15:38:38.859 NotebookApp] The Jupyter Notebook is running at:
[I 15:38:38.859 NotebookApp] http://(c100b-s41.ufhpc or 127.0.0.1):29926/?token=06b1c3f73bb847234c198a22bd62b7f20101b04d1bc2b64a
[I 15:38:38.859 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 15:38:38.862 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://(c100b-s1.ufhpc or 127.0.0.1):29926/?token=06b1c3f73bb847234c198a22bd62b7f20101b04d1bc2b64a
[I 15:39:09.643 NotebookApp] 302 GET / (172.16.206.12) 0.53ms
[I 15:39:09.648 NotebookApp] 302 GET /tree? (172.16.206.12) 0.66ms
[I 15:39:37.821 NotebookApp] 302 GET /?token=06b1c3f73bb847234c198a22bd62b7f20101b04d1bc2b64a (172.16.206.12) 0.37ms
[I 15:39:46.069 NotebookApp] Creating new notebook in
[I 15:40:05.840 NotebookApp] Kernel started: b5d6cea6-d214-4b37-b01e-4250ae6179c9
[I 15:40:13.516 NotebookApp] Adapting to protocol v5.0 for kernel b5d6cea6-d214-4b37-b01e-4250ae6179c9

Copy the token to use it as the password the first time you connect to the notebook. In this example the token is 06b1c3f73bb847234c198a22bd62b7f20101b04d1bc2b64a.

Create Tunnel From Local Machine

Copy paste the tunnel command from the job script or write your own based on how you manually started a notebook.

ssh -NL 23312:c10b-s14.ufhpc:23312 jdoe@hpg.rc.ufl.edu

Browse To Notebook

In a web browser on the local machine open http://localhost:23312

Provide the personal password or the token from the job log as the password to authenticate.

Enjoy using the rich python programming and visualization environment provided by Jupyter running on UFRC computing resources.

Again, note that the default Jupyter Notebook setup you see should have at least four kernels - two default kernels (python and R) that come with Jupyter and two additional kernels that provide access to environments provided by RC-specific environment modules e.g. 'RC R-3.5.1' and 'RC Py3-3.6.5', which match the same environment modules you use in batch jobs.