Jump to content

Help:Toolforge/Building container images/My first Buildpack Django tool

From Wikitech

Overview

This guide will walk you through deploying a simple Django web application using Toolforge’s build service. While it focuses specifically on Django, the same workflow can be used for configuring and deploying any Python web application, whether it’s built with Flask, FastAPI, or any other web framework of your choice.

Example App

In this repository, you will find a basic Django app configured for Toolforge Build Service. This is an implementation of the official Writing your first Django app tutorial, which you can use as a reference to understand what the app does. Here, we will focus on the bits specific to deployment on Toolforge, and not so much on the app itself.

Configuration

Under the hood, Toolforge build service uses Heroku buildpacks to build and run your application. Heroku web applications require a Procfile. This file is used to explicitly declare your application’s process types and entry points. It's located in the root of your repository.

A Procfile is not technically required to deploy simple apps written in most Heroku-supported languages — the platform automatically detects the language and creates a default web process type to boot the application server. However, creating an explicit Procfile is recommended for greater control and flexibility over your app.

A Procfile can contain several processes, such as different types of workers and scheduled jobs. For the sake of keeping things simple, we will only see two types of processes in this guide:

  1. web The web process is special in that it’s the only process type that can receive external HTTP traffic.
  2. migrate This is a process that will apply the database migrations. This process is not special – the name `migrate` is arbitrary and can be changed. So is the command that this process runs. We will see this later when we talk about the database.

Expected files

Heroku’s Python buildpack automatically identifies your app as a Python app if any of the following files are present in its root directory:

  • requirements.txt
  • setup.py
  • Pipfile

If none of these files is present in your app’s root directory, the Python buildpack will fail to identify your application correctly.

NOTE: There's currently no other way to specify your dependencies, you can find some workaround in task T353762


Procfile

Here is what the example app's Procfile looks like: 

web: gunicorn mysite.wsgi --bind 0.0.0.0
migrate: python manage.py migrate

This Procfile requires Gunicorn, the production web server recommended for Django applications. Make sure you add it as a dependency.

Git

Toolforge build service builds your application from a public git repository. Your code will therefore need to be uploaded to a repository such as GitLab, GitHub, Bitbucket, or other.

Python version

Optionally, you can specify the Python version you want to use inside a runtime.txt file in the root of your repository. If you don’t include this file, the default Python version will be used, currently 3.11.4.

Secrets & Config

For each environment variable needed in production, run the following command (will prompt for the value): 

$ toolforge envvars create <NAME>

To check which environment variables you have currently set: 

$ toolforge envvars list

If you change the variables or set new ones once the webservice is running, you will have to restart it manually for the changes to take effect. The environment variables will persist across restarts, so there's no need to set them again if you need to redeploy.

Static Files

At the time of writing, Toolforge doesn't (yet) offer object storage. In the meantime, the best option for storing static files is inside the container where the application will be running. In practice, this means you will need to have the following configuration inside settings.py: 

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/4.2/howto/static-files/

STATIC_URL = '/static/'
STATIC_ROOT = BASE_DIR / 'static'

Django does not support serving static files in production. However, the fantastic WhiteNoise project can integrate into your Django application, and was designed with exactly this purpose in mind.

See the WhiteNoise Django documentation for details on how to configure it. You can also use the sample project's settings.py as a blueprint.

Don't forget to add whitenoise to your dependencies. :)

Deployment

Build

  1. SSH into Toolforge
  2. become <tool-name>
  3. toolforge build start GIT_REPO
  4. Wait for the build to complete. You can check the status with toolforge build show. When finished, the status should show 'ok (Succeeded)'

If needed, check the build logs with toolforge build logs

Start the webservice

  1. toolforge webservice buildservice start
  2. Wait for the webservice to spin up, then check the logs to see if everything went ok: toolforge webservice logs -f
  3. Navigate to https://app-name.toolforge.org/polls/ to verify the application is working as expected.

DB migrations and other one-off jobs

One-off jobs, including db migrations, can be run using the Toolforge jobs framework. In the Procfile we created above, we defined a `migrate` process type that will run python manage.py migrate for us. Here is how we apply the migration: 

$ toolforge jobs run --image tool-<tool-name>/tool-<tool-name>:latest --command "migrate" --no-filelog --wait --mount=all migrate

The migrate at the end of the command is an arbitrary name that we give the job, while the "migrate" after --command refers to the process with the same name in the Procfile.

--mount=all is necessary because the app is using the tool's home directory to store it's SQLite database. A "real" production app would most likely be using ToolsDB or a Trove instance, and would not need this additional parameter.

Logs

You can display the application logs with toolforge webservice logs -f.

By default, Gunicorn logs errors to stderr (standard error), which typically displays in the terminal where the server is running. This includes any Python exceptions that your application raises, as well as Gunicorn's own log messages.

By default, Gunicorn does not write access logs. To enable access logging, use the flag --access-logfile - or set the accesslog configuration variable. Access logs record each request to the server.

You will need to modify your Procfile like this: 

web: gunicorn mysite.wsgi --access-logfile -
migrate: python manage.py migrate

Setting up Django Admin

The Django Admin interface is a powerful and built-in feature that allows for easy management of the data in your Django applications. It's enabled by default and accessible via the /admin URL path.

To get started and make use of the Django Admin, you first need to create a superuser account. This section will guide you through setting up a superuser for your Django application deployed on Toolforge using environment variables and a one-off job.

Create the envvars

Before creating the superuser, you must define the necessary credentials as environment variables on Toolforge. These variables will be used by the Django createsuperuser command to set up the superuser account without interactive input.

Use the toolforge envvars create command to define the following environment variables:

  • DJANGO_SUPERUSER_USERNAME: The username for your Django superuser.
  • DJANGO_SUPERUSER_PASSWORD: A secure password for the superuser account. Ensure the password is not too similar to the username and contains at least 8 characters for security reasons.
  • DJANGO_SUPERUSER_EMAIL: The email address associated with the superuser account.

Create the superuser

With the environment variables set, you can now create the superuser by running a one-off job on Toolforge. The job executes the createsuperuser command with the --noinput flag, which tells Django to use the environment variables we defined earlier instead of asking for input interactively.

Run the following command to create the superuser, replacing <tool-name> with the name of your tool: 

$ toolforge jobs run --image tool-<tool-name>/tool-<tool-name>:latest --command "python manage.py createsuperuser --noinput" --no-filelog --wait --mount=all createsuperuser

The --mount=all option is necessary for this example because it uses SQLite. If your application connects to Toolsdb or Trove for database services, you should omit this option.

CSRF settings

If you now try to log in to Django admin, you may get an error that saying something like "Forbidden (403) CSRF verification failed. Request aborted." If this is the case, add the following line to your settings.py to explicitly trust your domain for CSRF validation: 

CSRF_TRUSTED_ORIGINS = ['https://your-tool.toolforge.org']

Replace your-tool with the actual name of your Toolforge tool. Note that you will have to rebuild and restart your tool for the changes to take effect.

Setting up ToolsDB

As mentioned earlier, SQLite is sufficient for prototyping and local development. However, when deploying an application to Toolforge, you should use a production-grade database. For most use cases, ToolsDB will be sufficient. To set up ToolsDB, follow these steps:

Create a Database in ToolsDB

First, you need to create a database in ToolsDB. Take note of your database name.

Install mysqlclient

To connect to the ToolsDB database, you need to install the mysqlclient package: 

$ pip install mysqlclient

Specify different settings for production and development

To specify different settings for development and production, you can use the django-configurations library along with environment variables. Install django-configurations with the following command: 

$ pip install django-configurations

Follow the instructions in the official django-configurations documentation for usage instructions. Looking at the sample app, the most important change is that in /mysite/settings.py all the settings have been moved to use Classes such as Base, Development and Production.

Recreate requirements.txt

Because we installed new packages, we need to recreate requirements.txt: 

$ pip freeze > requirements.txt

Set Environment Variables

In Toolforge, set the following environment variables using the toolforge envvars createcommand:

DB_NAME=<name-of-db-in-toolsdb>
DJANGO_CONFIGURATION=Production
DJANGO_SETTINGS_MODULE=mysite.settings
MYSQLCLIENT_CFLAGS="-I/usr/include/mariadb/"
MYSQLCLIENT_LDFLAGS="-L/usr/lib/x86_64-linux-gnu/ -lmariadb"

Deploy the Application and Run Database Migration

Deploy your application and run the database migration as described in a previous section.

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Stay aware of critical changes and plans
Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/Building_container_images/My_first_Buildpack_Django_tool&oldid=2232706"