Jump to content

Data Platform/Systems/Jupyter

Phabricator project: #Analytics-Jupyter
From Wikitech

The stat hosts include a hosted version of JupyterHub, allowing easy analysis of internal data using Jupyter notebooks.

Access

Prerequisites

To access Jupyter, you need:

SSH tunnel

Once you have this access, open a SSH tunnel to one of the stat hosts. There are two main ways to do this. We'll assume you want to connect to stat1008, but you can connect to another host instead by changing the name in the terminal command.

The first option is using the standard SSH command:

$ ssh -N stat1008.eqiad.wmnet -L 8880:127.0.0.1:8880

The second option is to modify your SSH configuration file to automatically open a tunnel whenever you connect to one of the stat hosts: 

Match host=!*.*,stat10*
        SessionType none
        HostName %h.eqiad.wmnet
        LocalForward 8880 127.0.0.1:8880

With that added, you can simply use the following command:[1]

$ ssh stat1008

Note that your Jupyter notebook and files will stored be on the chosen host only. If you want to move to another server, you will have to copy your files using Rsync. If you need shared access to files, consider putting those files in HDFS.

Logging in

Then, open localhost:8880 in your browser and log in with your developer account. Use your shell username rather than your wiki username (e.g. nshahquinn-wmf, not Neil Shah-Quinn (WMF).

You'll be prompted to select or create a Conda environment. See the section on Conda environments below.

Authenticating via Kerberos

Once you've logged in, if you want to access data from Hadoop, you will need to authenticate with Kerberos. Open a new terminal in JupyterLab (not a separate SSH session). Type kinit on the command line and enter your Kerberos password at the prompt.

Querying data

To make it easier for you to access data from the stat hosts, use the following software packages, which hard-code much of the setup and configuration.

In Python

For Python, there is Wmfdata-Python. It can access data through MariaDB, Hive, Presto, and Spark and has a number of other useful functions, like creating Spark sessions. For details, see the repository and particularly the quickstart notebook.

In R

For R, there is wmfdata-r. It can access data from MariaDB and Hive and has many other useful functions, particularly for graphing and statistics.

Conda environments

Main article: Data Platform/Systems/Conda

Jupyter is set up to use isolated environments managed by Conda. When you first log in, you will be prompted to "Create and use new cloned conda environment".

You can create as many Conda environments as you need, but you can only run one at a time in Jupyter. To change environments:

  1. Navigate to the JupyterHub control panel by selecting File → Hub Control Panel in the Jupyter interface or navigating to localhost:8880/hub/home
  2. Select "Stop My Server"
  3. Select "Start My Server"

You will now see a dropdown allowing you to choose an existing environment or create a new one:

The option to choose a Conda environment when starting a JupyterHub server

Once your Jupyter server has launched within the Conda environment you chose, you will see the JupyterLab interface and are ready to start working:

The interface you'll see once you sign into our Jupyter service

Installing packages

If you need different or newer versions of packages, run conda install {{package}} or conda update {{package}} in the terminal. For more information, see Data Platform/Systems/Conda#Installing packages.

Sharing Notebooks

There is currently no built-in functionality for internal sharing of notebooks (phab:T156934), but there are several different workarounds available:

Copying files on the stat hosts

It's possible to copy notebooks and files directly on the server by clicking 'New' → 'Terminal' (in the root folder in the browser window) and using the cp command. Note that you may have to change the file permissions using the chmod command to give the other user read access to the files.

GitHub or GitLab

It's also possible to track your notebooks in Git and push them to either GitHub or our GitLab, both of which will display them fully rendered on its website. Generally, this requires making the notebook public, but it's also possible to request a private GitLab repo if necessary.

In either case, you will need to connect using HTTPS rather than SSH (SRE considers SSH from the stat hosts a security risk because of the possibility that other users could access your SSH keys). To do this, you will need to set up a personal access token (GitLab docs, GitHub docs), which you will use in place of a password when using Git on the command line.

By default, you'll have to enter your username and password every time you push. You can avoid this by adding the following to ~/.gitconfig:

# Automatically add username to GitHub URLs
[url "https://{{username}}@github.com"]
    insteadOf = https://github.com

# Automatically add username to Wikimedia GitLab URLs
[url "https://{{username}}@gitlab.wikimedia.org"]
    insteadOf = https://gitlab.wikimedia.org

# Cache access tokens for 8 hours after entry
[credential]
    helper = cache --timeout=28800
"Open raw" button

Often, notebooks with interactive or HTML elements don't display well in GitLab or GitHub. You can solve this by sharing the Nbviewer link to the notebook.

For GitHub, the tool works fine with the blob version of a notebook, however for gitlab.wikimedia.org, it can only read the raw version. You can either change from blob to raw in the URL, or open the raw version (from the top bar) as shown in the image and copy the URL.

Quarto reports

Installing Quarto

To convert a Jupyter notebook to an HTML file with Quarto on the analytics cluster:

  1. Install Quarto in your conda-analytics environment with conda install quarto (which will install Quarto from conda-forge)
  2. Restart your server (File → Hub Control Panel → Stop My Server → Start My Server → select the relevant environment) because this will run the etc/conda/activate.d/quarto.sh that sets up the correct environment variables which are necessary for the quarto command to work

Rendering a notebook

Then you can run quarto render some_notebook.ipynb --to html which will generate:

  • An HTML file with same name as the notebook but html extension (e.g. some_notebook.html)
  • A folder of JavaScript and CSS files with same name as the notebook but _files appended (e.g. some_notebook_files): this is necessary for the aforementioned HTML file to display and function correctly
    • ALTERNATIVELY: Set embed-resources: true option to produce a standalone (self-contained) HTML file.[2]

When you publish the files (e.g. following the web publication workflow), make sure you publish both the HTML file and the folder of supporting files.

Previewing a notebook

If you have a lot of writing to do or you're using complex formatting, you can also set up live preview of the notebook.

  1. In a Jupyter terminal, run the command: quarto preview {your_notebook}.ipynb --to html --port 6513[3]
  2. In a terminal on your computer, run ssh -N {your_stat_host} -L 6513:127.0.0.1:6513
  3. Open http://localhost:6513 and you will see the HTML preview of your notebook.
  4. Work on your notebook. Every time you save, the preview will update.
  5. When you're done using the preview, close Quarto by typing control + C in the Jupyter terminal.

Troubleshooting

Spawn failure after creating a new environment

If you attempt to start a Jupyter server using the "create and use a new Conda environment" option, but the start-up fails, delete your .conda/pkgs/cache directory and try again (task T380477).

Trouble installing R packages

See Data Platform/Systems/Conda#R support.

Browser disconnects

If your browser session disconnects from the kernel on the server (if, for example, your SSH connection times out), any work the kernel is doing will continue, and you'll be able to access the results the next time you connect to the kernel, but no further display output for that work (like print() commands to log progress) will accumulate, even if you reopen the notebook (JupyterLab issue 4237).

Notebook is unresponsive, or kernel restarts when running a large query

It may be that your Jupyter Notebook Server ran out of memory and the operating system's out of memory killer decided to kill your kernel to cope with the situation. You won't get any notification that this has happened other than the notebook being unresponsive or restarting, but you can assess the state of the memory on the notebook server by checking its host overview dashboard in Grafana (host-overview dashboard) or using the command line to see which processes are using the most (with ps aux --sort -rss | head or similar).

Sometimes trying to access the main interface at http://localhost:8880 will throw an HTTP 500 error. In these cases it may be possible to visit http://localhost:8880/hub/home and stop the server.

Viewing Jupyter Notebook Server logs

JupyterHub logs are viewable by normal users in Kibana.

A dashboard has been created named JupyterHub and this is also linked from the Home Dashboard.

At present the logs are not split per user, but we are working to make this possible.

They are no longer written by default to /var/log/syslog but they are retained on the host in the systemd journal.

You might need to see JupyterHub logs to troubleshoot login issues or resource issues affecting the cluster.

An individual user's notebook server log be examined with the following command 

sudo journalctl -f -u jupyter-$USERNAME-singleuser.service

Viewing JupyterHub logs

TODO: Make this work for regular users!

You might need to see JupyterHub logs to troubleshoot login issues: 

sudo journalctl -f -u jupyterhub

"An error occurred while trying to connect to the Java server"

If you see an error like this: 

ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:43881)
Traceback (most recent call last):
  File "/usr/lib/spark3/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 977, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque

Try each of these options, one at time:

  1. restart your notebook kernel (Menu -> Kernel -> Restart kernel)
  2. restart your JupyterHub server (follow the steps for changing environments, but use the same environment)
  3. create and use a brand new environment (follow the steps given previously, but select "create and use new cloned conda environment..."

Tips

Sending emails from within a notebook

To send out an email from a Python notebook (e.g. as a notification that a long-running query or calculation has completed), you can use the following code:

hostname = !hostname
server = hostname[0] + '.eqiad.wmnet'

whoami = !whoami
user = whoami[0]

from email.message import EmailMessage
import smtplib

def send_email(
    subject,
    body,
    to_email=user+'@wikimedia.org',
    from_email=user+'@'+server
):
    smtp = smtplib.SMTP("localhost")
    
    message = EmailMessage()
    message.set_content(body)
    
    message['From'] = from_email
    message['To'] = to_email
    message['Subject'] = subject
    
    smtp.send_message(message)

(Invoking the standard mail client via the shell, i.e. !mailx, fails for some reason. See phab:T168103.)

Shell shortcuts

To make it easier to access the stat hosts you can add entries like

Host stat11
    HostName stat1011.eqiad.wmnet

to ~/.ssh/config so that connecting to stat1011.eqiad.wmnet is as easy as ssh stat11 You can also make it easier to open SSH tunnels without remembering the full command. For example, if you are using Z shell you can add a tunnel function

tunnel() {
    ssh -N $1 -L 8880:127.0.0.1:8880
}

to ~/.zshrc so that opening a tunnel to stat1011.eqiad.wmnet – and assuming you added the appropriate entries to your SSH config – is as easy as tunnel stat11

Administration

Data_Platform/Systems/Jupyter/Administration

References

  1. ↑ If you later want to open a plain interactive SSH session with one of the analytics clients, you can still do this by using its full name: ssh stat1008.eqiad.wmnet.
  2. ↑ https://quarto.org/docs/output-formats/html-basics.html#self-contained
  3. ↑ The specific port number doesn't matter; if you don't provide it, Quarto will pick a random port. This use a fixed number to make it easy to copy the commands in the next two steps.
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Data_Platform/Systems/Jupyter&oldid=2287148"