User:Thcipriani/Blubber/Tutorial
Blubber: Getting Started
Blubber is an abstraction for container build configurations. It provides a handful of declarative constructs that give developers control over build configurations without sacrificing security and maintainability.
The tutorials below are intended to provide an overview of the basic parts of Blubber. The hope is that these examples will allow developers to become productive with Blubber as quickly as possible.
Hello World
The standard example for any software is one that can print "Hello World!" to the console.
Here is a Blubberfile that will print "Hello World!" to the console and exit.
version: v3
base: docker-registry.wikimedia.org/wikimedia-stretch
variants:
hello:
entrypoint: [echo, "Hello, world!"]
I have the file above saved as hello-blubber.yaml. To generate a
Dockerfile from hello-blubber.yaml I pass the blubber
command line application two arguments: (1) the path to
hello-blubber.yaml file and (2) the name of the variant (from the
keys listed under variants) from which I would like to generate a
Dockerfile. In this case hello-blubber.yaml only has one variant
named hello.
developer@laptop:~/blubber-tutorial$ blubber hello-blubber.yaml hello
This command produces the following Dockerfile on stdout (the output you see
may vary depending on your version of blubber):
FROM docker-registry.wikimedia.org/wikimedia-stretch
USER "root"
ENV HOME="/root"
RUN groupadd -o -g "65533" -r "somebody" && useradd -o -m -d "/home/somebody" -r -g "somebody" -u "65533" "somebody" && mkdir -p "/srv/app" && chown "65533":"65533" "/srv/app" && mkdir -p "/opt/lib" && chown "65533":"65533" "/opt/lib"
RUN groupadd -o -g "900" -r "runuser" && useradd -o -m -d "/home/runuser" -r -g "runuser" -u "900" "runuser"
USER "somebody"
ENV HOME="/home/somebody"
WORKDIR "/srv/app"
COPY --chown=65533:65533 [".", "."]
USER "runuser"
ENV HOME="/home/runuser"
ENTRYPOINT ["echo", "Hello, world!"]
LABEL blubber.variant="hello" blubber.version="0.4.0+60add2d"
Blubber's only purpose is to create opinionated Dockerfiles. To generate a
Dockerfile from a Blubberfile you pass the blubber command line
program the path to a Blubberfile and a variant name.
Variants can be named anything, although it is common to have (at minimum)
a test variant that creates a Dockerfile for an image that runs a
test entrypoint, and a production variant that creates a
Dockerfile to build an image that can be run in production. To create a Docker
Image from this Dockerfile, you can use unix pipes to pipe the output of
Blubber to the input of the docker build command.
developer@laptop:~/blubber-tutorial$ blubber hello-blubber.yaml hello | docker build --tag blubber-tutorial-hello-world --file - .
The command above creates a local Docker Image tagged with the name
blubber-tutorial-hello-world. Using - as the file
tells docker build to use stdin as the Dockerfile. .
at the end of the command tells docker build to use the current
directory as the context for any copy commands.
To run this image issue the following command:
developer@laptop:~/blubber-tutorial$ docker run --rm --interactive --tty blubber-tutorial-hello-world:latest
Hello, World!
Hello World, Redux
The previous example was extremely minimal as a means of giving a general introduction using Blubber to build Docker images that can be easily run on the command line. This example is meant to be ever so slightly more complicated as a means of introducing new Blubber concepts.
A slightly more complicated Hello, World! Blubberfile might look
like:
version: v3
base: docker-registry.wikimedia.org/wikimedia-stretch
apt:
packages:
- figlet
runs:
environment: { HELLO_WORLD: "Hello, world!" }
variants:
hello:
entrypoint: [sh, -c, 'figlet $HELLO_WORLD']
Above we've installed the figlet package from apt and
we've also introduced an environment variable to hold our output.
The above Blubberfile produces the following Dockerfile on stdout (the output
you see may vary depending on your version of blubber):
FROM docker-registry.wikimedia.org/wikimedia-stretch
USER "root"
ENV HOME="/root"
ENV DEBIAN_FRONTEND="noninteractive"
RUN apt-get update && apt-get install -y "figlet" && rm -rf /var/lib/apt/lists/*
RUN groupadd -o -g "65533" -r "somebody" && useradd -o -m -d "/home/somebody" -r -g "somebody" -u "65533" "somebody" && mkdir -p "/srv/app" && chown "65533":"65533" "/srv/app" && mkdir -p "/opt/lib" && chown "65533":"65533" "/opt/lib"
RUN groupadd -o -g "900" -r "runuser" && useradd -o -m -d "/home/runuser" -r -g "runuser" -u "900" "runuser"
USER "somebody"
ENV HOME="/home/somebody"
WORKDIR "/srv/app"
ENV HELLO_WORLD="Hello, world!"
COPY --chown=65533:65533 [".", "."]
USER "runuser"
ENV HOME="/home/runuser"
ENTRYPOINT ["sh", "-c", "figlet $HELLO_WORLD"]
LABEL blubber.variant="hello" blubber.version="0.4.0+60add2d"
You'll notice that the apt: {packages: [figlet]} declaration in
the Blubberfile installed the figlet program as the
root user on the commandline. You can specify packages to install
from Debian at the top-level of a Blubberfile and under each individual
variant.
Also notice that we've set an environment variable for our output message. You can set environment variables at the top level of a Blubberfile and under each individual variant.
Using the same command as above to generate a Docker image produces a slightly more interesting result when running the image in a container on the commandline:
developer@laptop:~/blubber-tutorial$ docker run --rm --interactive --tty blubber-tutorial-hello-world-redux:latest
_ _ _ _ __ __ _ _ _
| | | | ___| | | ___ \ \ / /__ _ __| | __| | |
| |_| |/ _ \ | |/ _ \ \ \ /\ / / _ \| '__| |/ _` | |
| _ | __/ | | (_) | \ V V / (_) | | | | (_| |_|
|_| |_|\___|_|_|\___/ \_/\_/ \___/|_| |_|\__,_(_)
Hello Node
Many of the microservices used in Wikimedia production are written in Node.js. Many of the microservices used in Wikimedia production use the suffix -oid; which, according to Wiktionary is a suffix which modifies the root to imply that something is "Of similar form to, but not the same as."; e.g., humanoid, meaning similar-to, but not the same as a human.
This example will be similar to our previous examples (in that it will still output, "Hello, World!"), but it will be written in Node.js.
This example uses the HelloWorldOid repository.
To clone this repository and follow along:
dev@laptop:~$ git clone https://gerrit.wikimedia.org/r/blubber-doc/example/helloworldoid blubber-doc/example/helloworldoid
dev@laptop:~$ cd blubber-doc/example/helloworldoid
The directory structure of the application is fairly trivial (omitting any git-related files):
dev@laptop:helloworldoid$ tree -a --dirsfirst
.
├── lib
│ ├── helloworld.js
│ └── server.js
├── .pipeline
│ └── blubber.yaml
├── test
│ └── test.js
├── .dockerignore
├── index.js
├── LICENSE
├── package.json
└── README.md
3 directories, 9 files
There are a few files that are noteworthy and not typical in basic Node.js sample applications:
- .
.dockerignore- The.dockerignore( on https://docs.docker.com) file is used to keep files necessary for development, but unnecessary for production out of a container image. In this case I've removed theREADME.mdand the.gitdirectory in an attempt to keep the Docker image created by Blubber small. - .
.pipelinedirectory - this directory is where the [#Use_in_the_Continuous_Delivery_Pipeline Wikimedia Continuous Delivery Pipeline] expects to find theblubber.yamlfile; so that's where it is!
Another thing to notice is that the Blubberfile is a bit longer than in previous examples, and has more variants:
version: v3
base: docker-registry.wikimedia.org/nodejs-slim
runs:
environment:
HELLO_WORLD: Hi, I’d like to add you to my professional network on LinkedIn.
variants:
build:
base: docker-registry.wikimedia.org/nodejs-devel
node: {requirements: [package.json]}
test:
includes: [build]
entrypoint: [npm, test]
prep:
includes: [build]
node: { env: production }
production:
copies: prep
entrypoint: [node, index.js]
Notice that a variant can specify a different base image than the
global base image at the top of the file—in this case the
build variant is using the
docker-registry.wikimedia.org/nodejs-devel image so that it can
use the npm tool already present in that image.
Also noteworthy is that each variant can specify a different
entrypoint. In the case of the test variant we'd like
to create a Docker image that runs the application's tests, but in the
production variant we want to run the actual application.
The includes keyword is specified in a few variants:
test and prep. Variants can inherit from one another.
That is, rather than specify that we want to use all the same options as the
build variant for test variant we can include that
variant. This way the test variant uses the
nodejs-devel base image and will run npm
install after copying package.json into the image (since
we've used the node keyword and specified that we need the
package.json file as a requirement).
The build variant here is really only used as a set of options
common to both the test and prep variants. Notice
that the production variant is not including any other variant and
so uses the nodejs-slim image.
The production variant copies the prep
variant. The copies keyword will generate a
[https://docs.docker.com/develop/develop-images/multistage-build/ multistage
Dockerfile]. Docker images can easily become very large. One method to
slim-down a Docker image is to copy any generated artifacts into an image rather than
install the tools to generate artifacts inside the image. A multistage
Dockerfile will first create an intermediary image (in this case based on the
prep variant) that can be based on a different base
image or include different packages that will be used to create artifacts.
The artifacts generated by the intermediary Docker image can then be copied to
a slimmer final Docker image (in this case based on the production
variant) to allow for a smaller final image.
The prep variant varies from the build variant
insofar as it uses the --production flag when calling npm
install (as indicated by the node: {env:production} in the
Blubberfile. In this way we do not include development dependencies like
mocha in the final production image.
We can build an image based on the test variant to run the tests in much same way we've built previous images:
dev@laptop:helloworldoid$ blubber .pipeline/blubber.yaml test | docker build -t blubber-tutorial-helloworldoid-test -f - .
Then we can run the resulting image in a container to show the test results:
dev@laptop:helloworldoid$ docker run --rm -it blubber-tutorial-helloworldoid-test
> helloworldoid@0.0.1 test /srv/app
> mocha
helloWorld
✓ obvs should be a string
✓ obvs should start with "hello"
✓ obvs should contain the word "world"
3 passing (19ms)
Then we can run build and run the production image in a container, exposing port 8001:
dev@laptop:helloworldoid$ blubber .pipeline/blubber.yaml production | docker build -t blubber-tutorial-helloworldoid -f - .
dev@laptop:helloworldoid$ docker run -p8001:8001 --rm -it blubber-tutorial-helloworldoid
And now we should be able to access localhost:8001 using http
to reveal the fruits of our labor:
developer@laptop:~$ curl localhost:8001
__________________________________________________________________________________________________________________________
/ ('-. .-. ('-. (`\ .-') /` _ .-') _ .-') _ ,---. \
| ( OO ) / _( OO) `.( OO ),' ( \( -O ) ( ( OO) ) | | |
| ,--. ,--.(,------.,--. ,--. .-'),-----. ,--./ .--. .-'),-----. ,------. ,--. \ .'_ | | |
| | | | | | .---'| |.-') | |.-') ( OO' .-. ' | | | ( OO' .-. '| /`. ' | |.-') ,`'--..._)| | |
| | .| | | | | | OO ) | | OO )/ | | | | | | | |, / | | | || / | | | | OO )| | \ '| | |
| | |(| '--. | |`-' | | |`-' |\_) | |\| | | |.'.| |_)\_) | |\| || |_.' | | |`-' || | ' || .' |
| | .-. | | .--'(| '---.'(| '---.' \ | | | | | | \ | | | || . '.'(| '---.'| | / :`--' |
| | | | | | `---.| | | | `' '-' '.-. | ,'. | `' '-' '| |\ \ | | | '--' /.--. |
| `--' `--' `------'`------' `------' `-----' ',/ '--' '--' `-----' `--' '--' `------' `-------' '--' |
\ Hi, I’d like to add you to my professional network on LinkedIn. /
--------------------------------------------------------------------------------------------------------------------------
\ ^__^
\ (oO)\_______
(__)\ )\/\
U ||--WWW |
|| ||~
Use in the Continuous Delivery Pipeline
In the Continuous Delievery Pipeline Jenkins will handle building Docker images from Blubberfile specifications. This is configured by convention: Jenkins expects to find a Blubberfile in .pipeline/blubber.yaml at the root of your repository. In that Blubberfile it expects to find, at minimum, two variants (1) test and (2)production.
When a patchset is proposed in Gerrit for your repository, Jenkins will build a Docker image based on your test variant and run a container based on that image. Jenkins will report back to Gerrit based on the whether the entrypoint of the that container exited cleanly (success) or not (failure).
When a patchset is merged to your repository on Gerrit, Jenkins will, again, build and execute your test variant. If that succeeds Jenkins builds a Docker image based on your production variant will be built and pushed into the Wikimedia Docker Registry.
When you push a tag to your repository on Gerrit, Jenkins will, once again, build and execute your test variant. If that succeeds, Jenkins builds a Docker image based on you production Blubber variant, tags it with the tag you pushed and pushes that to the Wikimedia Docker Registry.