Help:Toolforge/Web/Python
This page describes Python-specific instructions for deploying a web server on Toolforge. Python web servers on Toolforge use uWSGI which is a Web Server Gateway Interface (WSGI) server for Python web applications. uWSGI can run applications built with Flask, Django, and other Python web application frameworks.
Conventions
The toolforge webservice
command starts Python applications using convention rather than configuration. These conventions are expected by the Toolforge tooling:
- Your WSGI application's entry point must be found in $HOME/www/python/src/app.py in a variable named
app
:
from flask import Flask
app = Flask(__name__)
# [...]
if __name__ == '__main__':
app.run()
from dash import Dash
application = Dash(__name__)
app = application.server
# [...]
if __name__ == '__main__':
application.run()
- Python libraries will be loaded from a virtual environment located in $HOME/www/python/venv.
- On the Kubernetes backend, you must use a virtual environment to install and load any libraries you depend on outside of the Python standard library.
- Additional configuration for uWSGI can be provided in a $HOME/www/python/uwsgi.ini file.
- Examples of configuration parameters can be found in the uWSGI manual.
- Headers can be added using
route = .* addheader:Access-Control-Allow-Origin: *
- Logs will be written to
$HOME/uwsgi.log
Starting a Python web service
To start a Python web service, use the toolforge webservice start
command. For example:
- python3.11 with a default uwsgi configuration
toolforge webservice python3.11 start
Virtual Environments and Packages
A virtual environment (venv) is a self-contained directory tree that contains a Python installation for a particular version of Python plus a number of additional packages. Using a venv allows you to install local Python packages for your tool.
The fundamental thing to remember is that a venv created directly on the bastion will not work with toolforge webservice
, and a venv created inside a webservice shell work only with the same Python runtime version.
Creating a virtual environment
toolforge webservice python3.11 shell
(choose a different python version as appropriate for your project)mkdir -p $HOME/www/python
python3 -m venv $HOME/www/python/venv
source $HOME/www/python/venv/bin/activate
pip install --upgrade pip wheel
(This brings in newest pip, which is required for wheel support)- Install the libraries you need (for example
pip install -r $HOME/www/python/src/requirements.txt
) - exit out of webservice shell
toolforge webservice python3.11 start
Step 1 can possibly freeze with an error message Pod is not ready in time
. Retrying the command again should fix it.
Steps 2-6 can be automated by using the webservice-python-bootstrap
script inside the webservice shell. If you want to create a brand new virtualenv in case you're switching Python versions or have new dependencies, use webservice-python-bootstrap --fresh
.
Using a uWSGI app with a default entry point that is not app.py
The default uwsgi configuration for the uwsgi webservice backend expects to find the uwsgi entry point as the variable app
loaded from the $HOME/www/python/src/app.py
module. If your application has another entry point, the easiest thing to do is create a $HOME/www/python/src/app.py
module, import your entry point, and expose it as app
. See Deploying a Django application for an example of this pattern.
Deploying a Django application
Django is a popular web framework for developing Python applications. A typical Django application will need a few changes to run with Toolforge's opinionated uWSGI configuration.
Create an app.py entry point
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "<YOUR-TOOL-NAME>.settings")
app = get_wsgi_application()
Note
|
---|
Note: If you're seeing errors about $HOME/www/python/src/app.py import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "<YOUR-TOOL-NAME>.settings")
from django.core.wsgi import get_wsgi_application
app = get_wsgi_application()
|
Static files
To correctly locate the static files, first configure $HOME/www/python/uwsgi.ini to look for static files in your tool's $HOME/www/python/src/static directory:
[uwsgi]
static-map = /static=/data/project/<YOUR-TOOL-NAME>/www/python/src/static
Next configure your Django app to use this location by editing the app's settings.py file:
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
Finally deploy your static files into $HOME/www/python/src/static. Typically this will be done by running python manage.py collectstatic
.
MySQL and utf8mb4
The version of MySQL/MariaDB currently used by ToolsDB will not work transparently with Django and the utf8mb4 character set. ROW_FORMAT=DYNAMIC
must be used on all tables which will index utf8mb4 encoded character fields longer than 191 characters. This configuration, together with database server configuration to enable innodb_large_prefix, the Barracuda file format, and file per table storage, enables index key prefixes longer than 767 bytes (up to 3072 bytes) for InnoDB tables.
Django does not have a feature flag or setting for adding the needed ROW_FORMAT=DYNAMIC
configuration to its database migrations. One way to work around this issue is by using a custom database engine as Volans did for his Debmonitor project. Another possible workaround is manually modifying your migration files as BryanDavis documented in a blog post.
Logs
You can find your application's log messages in $HOME/uwsgi.log.
To automatically rotate logs based on file size, set the log-maxsize
option in $HOME/www/python/uwsgi.ini to the maximum file size in bytes:
[uwsgi]
# 10 MiB log rotation
log-maxsize = 10485760
When the file size is reached, it will be copied to a timestamp-suffixed file in the same directory (like $HOME/uwsgi.log.1729413340).
Common issues
Pod not starting, cannot open shared object error in the logs
When you see the following error in the logs and the webservice pods crash in a loop:
open("/usr/lib/uwsgi/plugins/python_plugin.so"): No such file or directory [core/utils.c line 3724] !!! UNABLE to load uWSGI plugin: /usr/lib/uwsgi/plugins/python_plugin.so: cannot open shared object file: No such file or directory !!!
It's most possible that your python application is failing to start and the container is trying to load it with the python2 uswgi plugin as fallback. To debug, you can try opening a shell to the pod:
toolforge webservice python3.11 shell
And try starting your app from the home directory, that will show the actual error:
tools.wm-lol@interactive:~$ source www/python/venv/bin/activate (venv) tools.wm-lol@interactive:~$ python www/python/src/app.py Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/data/project/wm-lol/www/python/src/wm_lol/__init__.py", line 4, in <module> from wm_lol.api import api File "/data/project/wm-lol/www/python/src/wm_lol/api.py", line 4, in <module> from commands import get_matchers ModuleNotFoundError: No module named 'commands'
Communication and support
Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:
- Chat in real time in the IRC channel #wikimedia-cloud connect or the bridged Telegram group
- Discuss via email after you have subscribed to the cloud@ mailing list
- Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
- Read the News wiki page
Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself
Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)