PipelineLib/Tutorial/TestingHelloNode using parallel execution

Welcome to Testing Hello Node (using parallel execution), an even more advanced PipelineLib tutorial than the previous Testing Hello Node (using multiple stages) tutorial.

Prerequisites

• An understanding of the PipelineLib multi-stage concepts introduced in Testing Hello Node (using multiple stages) as well as the prerequisites mentioned there.
• A very basic understanding of directed acyclic graphs and how they can be expressed as adjacency lists.

Learning objectives

At the end of this tutorial, you will understand:

1. How to configure a pipeline's execution graph to run certain stages in parallel and rejoin on subsequent stages.
2. How to reference output from previous stages and how stage exports are scoped.

Tutorial

The basic steps to this tutorial will be to:

1. Start with the project modifications made in the previous tutorial.
2. Define an additional pipeline stage that executes the project's system tests.
3. Configure the test pipeline to execute its stages in parallel.
4. Submit your change to Gerrit.
5. Observe how Jenkins executes the project's multi-stage test pipeline according to the execution graph.
6. Clean up.

Starting point

You should start with the changes you made in to the helloworldoid repo in the previous tutorial.

dev@laptop:~$ cd blubber-doc/example/helloworldoid

Define a stage for running system tests

Remember that systemtest script we saw in the project's package.json? No? Fine.

dev@laptop:helloworldoid$ grep -A 4 scripts package.json
  "scripts": {
    "lint": "eslint --ignore-path .gitignore .",
    "systemtest": "mocha ./test/system.js",
    "test": "mocha ./test/unit.js"
  },

Yes, that one. OK. Let's add a stage that runs it using the generic script image variant from before.

dev@laptop:helloworldoid$ vim .pipeline/config.yaml # or ed; j/k

pipelines:
  test:
    blubberfile: blubber.yaml
    stages:
      - name: build
        build: script
      - name: test
        run:
          image: '${build.imageID}'
          arguments: [test]
      - name: lint
        run:
          image: '${build.imageID}'
          arguments: [lint]
      - name: systemtest
        run:
          image: '${build.imageID}'
          arguments: [systemtest]

Line by line, we are:

15. Defining a new stage in our pipeline called systemtest.
16. Specifying that this stage should run the image built in the build stage (script) and pass it the argument systemtest.

Simple enough and very similar to what we did last time. However, now we have four distinct stages, and running them all serially seems like a waste of time. Let's see if we can at least run the test and lint stages in parallel before proceeding to systemtest.

Define a custom execution graph

By default, a pipeline's stages are executed serially in the same order in which they were defined. In this case, the default execution graph is currently representing something like this:

build
  ⇘
  test
     ⇘
     lint
        ⇘
        systemtest

But let's change that. Let's instead execute test and lint in parallel after our image is built, and perform the more expensive systemtest stage only after all the other stages finish. In other words, let's make the execution graph look like this:

   build
   ⇙   ⇘
test   lint
   ⇘   ⇙
 systemtest

To accomplish that, we need to define a custom execution entry under the test pipeline.

dev@laptop:helloworldoid$ vim .pipeline/config.yaml

pipelines:
  test:
    blubberfile: blubber.yaml
    stages:
      - name: build
        build: script
      - name: test
        run:
          image: '${build.imageID}'
          arguments: [test]
      - name: lint
        run:
          image: '${build.imageID}'
          arguments: [lint]
      - name: systemtest
        run:
          image: '${build.imageID}'
          arguments: [systemtest]
    execution:
     - [build, test, systemtest]
     - [build, lint, systemtest]

Line by line, we are:

19. Providing a custom execution graph for our test pipeline.
20. Declaring a branch of execution that says stage test should be executed after stage build and before stage systemtest.
21. Declaring a branch of execution that says stage lint should also be executed after stage build and before stage systemtest.

Each stage has only a single place in its pipeline's execution, and what you are providing here is each separate branch of execution that together constitute an entire graph. This kind of graph representation is called an adjacency list.

Submit our change and watch our pipeline execute

Now that we have a .pipeline/config.yaml that will tell our CI system to execute our stages in a cool directed-graph order, let's push our changes to Gerrit and watch Jenkins execute it.

dev@laptop:helloworldoid$ git add .pipeline/{blubber,config}.yaml
dev@laptop:helloworldoid$ git commit -m 'tutorial2: Configure CI to execute using a DAG'
dev@laptop:helloworldoid$ git push origin HEAD:refs/for/master

Head over to the helloworldoid-pipeline-test job to see its progress!

Clean up

Help us to keep Gerrit tidy by abandoning your change, and reset your local branch to origin/master.