Friday, July 22, 2016

Automated pipeline support for Consumer Driven Contracts

When developing a system composed of services (maybe microservices) some services will depend on other services in order to work. In this article I use the terminology "consumer"[1] to mean a service which depends on another service, and "provider" to mean a service being depended upon. This article only addresses consumers and providers developed within the same company. I'm not considering external consumers here.

This article is about what we did at Springer Nature to make it easy to run CDCs - there is more written about CDCs elsewhere.

CDCs - the basics

CDCs (Consumer Driven Contracts) can show the developers of a provider service that they haven't broken any of their consumer services.

Consider a provider service called Users which has two consumers, called Accounting and Tea. Accounting sends bills to users, and Tea delivers cups of tea to users.

The Users service provides various endpoints, and over time the requirements of its consumers change. Each consumer team writes tests (called CDCs) which check whether the provider service (Users in this case) understands the message sent to it and responds with a message the consumer understands (e.g. using JSON over HTTP).

How we used to run CDCs

We used to have the consumer team send the provider team an executable (e.g. an executable jar) containing the consumer's CDCs. It was then up to the provider team to run those CDCs as necessary, e.g. by adding a stage to their CD (Continuous Delivery) pipeline. A problem with this was that it required manual effort required by the provider team to set up such a stage in the CD pipeline, and required effort each time a consumer team wanted to update its CDCs.

How we run them now

Our automated pipeline system allows consumers to define CDCs in their own repository and declare which providers they depend upon in their pipeline metadata file. Using this information, the automated pipeline system adds a stage to the consumer's pipeline to run its "CDCs" against its providers, and also in the provider's pipeline to run its consumers' CDCs against itself. In our simple example earlier this means the pipelines for Users, Accounting, Tea and Marketing will be something like this:




i.e. Users runs Accounting and Tea CDCs against itself (in parallel) after it has been deployed. Accounting and Tea run their CDCs against Users before they deploy.

This means that:
  • when a change is made to a consumer (e.g. Tea), its pipeline checks that its providers are still providing what is needed. This is quite standard and easy to arrange.
  • when a change is made to a provider (e.g. Users), its pipeline checks that it still provides what its consumers require. This is the clever bit that is harder to arrange. This is the point of CDCs.

Benefits of automation

By automating this setup, providers don't need to do anything in order to incorporate their consumers' CDCs into their pipeline. The providers also don't have to do anything in order to get updated versions of their consumers' CDCs.

The effort of setting up CDCs rests with the teams who have the dependency, i.e. the consumers. The consumers need to declare their provider (dependency) in their metadata file and define and maintain their CDCs.


There are a few subtleties involved in this system as it is currently implemented.
  • the consumer runs its CDCs against the provider in the same environment it is about to deploy into. There may be different versions of providers in different environments and this approach checks that the provider works for the version of the consumer that is about to be deployed, and will prevent the deployment if it is incompatible.
  • the provider runs the version of each consumer's CDCs corresponding to the version of the consumer in the same environment that the provider has just deployed into. There may be different versions of consumers in different environments and this approach checks that the provider works for the versions of consumers that are using the provider.
  • the system deploys the provider before running the consumer CDCs because the consumer CDCs need to run against it. It would be better for the system to deploy a new version of the provider without replacing the current version, run its consumers' CDCs and then only switch over to the new version if the CDCs all pass.
  • because the consumer's CDCs need to run against the provider in the appropriate environment, the system sets an environment variable with the host name of the provider in that environment. Because we only have one executable per consumer for all its CDCs, if a consumer has multiple providers, it needs to use those environment variables in order to determine which of its CDCs to execute.

Implementation notes

The implementation of a consumer running its CDCs against its provider is relatively straightforward. The difficulties are when a provider runs its consumers' CDCs against itself.

In order for a provider to run its consumers' CDCs the system clones each consumer's repository at the appropriate commit and then runs the appropriate executable in a Docker container. (The implementation doesn't clone every time, just if the repository hasn't been cloned on that build agent before.) Using Docker for running the CDCs means that consumers can implement their CDCs using whatever technology they want, as long as it runs in Docker.

In our system, all services are required to implement an endpoint which returns the git hash of the commit that they are built from. This is used to work out which version of consumer's CDCs to run in the case when they are run in a provider's pipeline.

Automating the running of CDCs in our automated pipelines required either making providers know who their consumers are, or consumers know who their providers are. If a provider doesn't provide what a consumer requires, it causes more problems for the consumer than for the provider. Therefore we made it the responsibility of the consumer to define that it depends on the provider rather than the other way around.

1 Other terminology in use for consumer is "downstream" and for provider is "upstream". A consumer is a dependant of a provider. A provider is a dependency of a consumer. I sometimes use the word producer instead of provider.

Copyright © 2016 Ivan Moore

Saturday, April 4, 2015

Example of automatically setting up pipelines

To explain automatically setting up pipelines, I've prepared example code (available here - see the license) for GoCD using gomatic and made a video showing the code running - which I will refer to throughout the article. I could possibly have done this all by narrating the video - maybe I will in the future.

Inception creation

The example includes a script (called which creates the pipeline that will create the pipelines (you run this only once):

from gomatic import GoCdConfigurator, HostRestClient, ExecTask, GitMaterial

configurator = GoCdConfigurator(HostRestClient("localhost:8153"))

pipeline = configurator\
    .set_timer("0 0 * * * ?")  # run on the hour, every hour
inception_job = pipeline.ensure_stage("inception").ensure_job("inception")
inception_job.ensure_task(ExecTask(["python", ""]))


This creates a pipeline in GoCD (here running on localhost) which runs on a timer.

The video starts with this script being run, and the "inception" pipeline has been created by time 0:19.


The script creates a pipeline for every repo of a particular github user. For a real system, you might want to do something more sophisticated; this example has been kept deliberately simple.

from gomatic import GoCdConfigurator, HostRestClient, ExecTask
from github import Github, GithubException

github = Github()
me = github.get_user("teamoptimization")
for repo in me.get_repos():
        print "configuring",
        configurator = GoCdConfigurator(HostRestClient("localhost:8153"))

        pipeline = configurator\
        job = pipeline\
        bootstrap_file_url = ""
        job.ensure_task(ExecTask(["bash", "-c", "curl -fSs " + bootstrap_file_url + " | python - " +]))

    except GithubException:
        print 'ignoring',

This script creates a pipeline with a "bootstrap" stage for each repo (unless it already exists). As long as nothing else creates pipelines with the same names, the "bootstrap" stage will end up being the first stage. The bootstrap stage runs the script described later, passing it the name of the repo/pipeline as an argument.

In the video, the "inception" pipeline is triggered manually at time 0:20 (rather than waiting for the timer) and has finished by time 1:03 (and has no affect yet as the relevant user has no repositories).

The bootstrap stage

In this example, the script creates a stage for every line of a file (called commands.txt); this makes it easy to demonstrate one of the key features of the approach - the ability to keep the pipeline in sync with the repo it is for. One of the subtleties is that the bootstrap has to make sure that it doesn't remove itself, but does need to remove all other stages so that if stages are removed from commands.txt then they will be removed from the pipeline. Note that because of how gomatic works, if there is no difference as a result of removing then re-adding the stages, then no POST request will be sent to GoCD, i.e. it would be entirely unaffected.

import sys
from gomatic import GoCdConfigurator, HostRestClient, ExecTask

configurator = GoCdConfigurator(HostRestClient("localhost:8153"))

pipeline_name = sys.argv[1]

pipeline = configurator\

for stage in pipeline.stages()[1:]:

commands = open("commands.txt").readlines()
for command in commands:
    command_name, thing_to_execute = command.strip().split('=')
        .ensure_task(ExecTask(thing_to_execute.split(" ")))


A real bootstrap script might be much more sophisticated, for example, creating a build stage automatically for any repo which contains a certain file (e.g. build.xml or maven.pom) and creating deployment stage(s) automatically. The example script is as short as I could make it for the purposes of demonstrating the approach.

In the video, the user creates a repository (from time 1:04 - 1:39) and then creates a commands.txt file, commits and pushes (up to time 2:18). Rather than waiting for the timer, the "inception" pipeline is manually triggered at time 2:22 and by 2:43 the pipeline is created for "project1". Rather than wait for GoCD to run the new pipeline (which it would after a minute or so) it is manually triggered at time 2:54, and when it runs, it creates the stage defined in commands.txt.

In the video, at time 3:54 the user adds another line to commands.txt and commits and pushes. Rather than wait for GoCD to run the pipeline (which it would after a minute or so) it is manually triggered at time 4:27, and when it runs, it adds the new stage defined in commands.txt.

Copyright © 2015 Ivan Moore

Saturday, February 7, 2015

Automatically setting up pipelines

Scripting the set up your continuous integration (CI) server is better than clicky clicky, but it might be possible to do even better. If you have many pipelines that are very similar then you might be able to fully automate their set up. A bit like having your own, in house version of Travis CI.

This article will use the GoCD terms "pipeline" and "stage" (a pipeline is somewhat like a "job" Jenkins, and a pipeline comprises one or more stages).

This article describes (at a very high level) the system my colleague Hilverd Reker and I have set up to automatically create pipelines. This has built on experience I gained doing something similar with Ran Fan at a previous client, and being the "customer" of an automated CI configuration system at another previous client.


We have a pipeline in our CI server to automatically create the pipelines we want. We have called this "inception", after the film - I think Ran Fan came up with the name.

The inception pipeline looks for new things to build in new repositories, and sub directories within existing repositories, and creates pipelines as appropriate (using gomatic). (The inception process that Ran Fan and I wrote previously, looked for new things to build within "one large repo" (maybe the subject of a future blog article), and new branches of that repository).

The advantage of having this fully automated, compared to having to run a script to get the pipeline set up, is that it ensures that all pipelines get set up: none are forgotten and no effort is required.

Our inception job sets up a pipeline with only one stage, the bootstrap stage, which configures the rest of the pipeline. This keeps the inception job simple.

The bootstrap stage

Some of the configuration of a pipeline depends upon the files in the repository that the pipeline is for. By making the first stage in the pipeline the bootstrap stage, it can configure the pipeline accurately for the files as they exist when the pipeline runs. If a pipeline is configured by the inception job, or by running a script, rather than a bootstrap stage, then its configuration will not reflect the files in the repository when they change, but rather how they were at the time the inception job, or script, ran. This would result in pipelines failing because they are not set up correctly for the files they are trying to use; hence we have the bootstrap as part of the pipeline itself to solve that problem.

Implementation notes

Our bootstrap stage only alters the configuration of the pipeline if it needs to: it runs very quickly if no changes are needed. GoCD handles changes to the configuration of pipeline well. After the bootstrap stage has run, the subsequent stages run in the newly configured, or reconfigured, pipeline as expected. GoCD also handles the history of a pipelined reasonably well (but not always getting it right), even when it's configuration changes over time.


What would help right now would be an example - but that'll take time to prepare; watch this space (patiently) ...

Copyright ©2015 Ivan Moore 

Wednesday, January 14, 2015

Scripting the configuration of your CI server

How do you configure your CI server?

Most people configure their CI server using a web based UI. You can confirm this by searching for "setting up Jenkins job", "setting up TeamCity build configuration", "setup ThoughtWorks Go pipeline" etc. The results will tell you to configure the appropriate CI server through a web based UI, probably with no mention that this is not the only way.

One of my serial ex-colleagues, Nick Pomfret, describes using these web based UIs as "clicky-clicky". In this article I will use the Jenkins term "job" (aka "project") to also mean TeamCity build configuration or GoCD pipeline. In this article, I'm calling GoCD a CI server; get over it.

What is wrong with clicky-clicky? 

Clicky-clicky can be useful for quick experiments, or maybe if you only have one job to set up, but has some serious drawbacks. 

It works - don't change it

Once a job has been set up using clicky-clicky, one problem is that it is difficult to manage changes to it. It can be difficult to see who has changed what, and to restore a job to a previous configuration. Just version controlling the complete CI server configuration file (which some people do) does not do this well, because such files are difficult to diff, particularly when there are changes to other jobs.

Lovingly hand crafted, each one unique

Another problem with clicky-clicky is when you have a lot of jobs that you would like to set up in the same way, clicky-clicky is both time consuming, and inevitably leads to unintended inconsistencies between jobs, which can cause them to behave in slightly different ways, causing confusion and taking longer to diagnose problems.

Can't see the wood for the tabs 

Furthermore, web UIs often don't make it easy to see everything about the configuration of a job a compact format - some CI servers are better than others for that.

The right way - scripting

If you script the setup of jobs, then you can version control the scripts. You can then safely change jobs, knowing that you can recreate them in the current or previous states, and you can see who changed what. If you need to move the CI server to a new machine, you can just rerun the scripts.

In some cases a script for setting up a job can be much more readable than the UI because it is often more compact and everything is together rather than spread over one or more screens.

Fully automated configuration of jobs

It can be very useful to script the setup of jobs so it is totally automatic; i.e. when a new project is created (e.g. a new repo is created, or a new directory containing a particular file, e.g. a build.gradle file, is created), then a job can be created automatically. If you take that approach, then it saves time because nobody needs to manually setup the jobs, it means that every project that needs a job gets one and none are forgotten, and it means that the jobs are consistent so it is easy to know what they do.

There are some subtleties about setting up fully automated jobs which I won't go into here - maybe a future blog article.

Tools for scripting

For GoCD, see gomatic. For other CI servers, please add a comment if you know of anything that is any good!

Copyright ©2015 Ivan Moore

Wednesday, December 24, 2014

Gomatic - scripting of GoCD configuration

Gomatic has been released - it is a Python API for configuring ThoughtWorks GoCD. I worked on it with my colleague Hilverd Reker. There isn't any documentation yet - we'll add some. For the moment, I thought I'd just post a very brief article here to announce it and to show a simple example of using it.


We wrote it for our purposes and find it very useful; however, it has limitations (e.g. only really supports "Custom Command" task type) and allows you to try to configure GoCD incorrectly (which GoCD will refuse to allow). We will continue to work on it and will address its current limitations.

It has only been tested using GoCD version 14.2.0-377 - I think it doesn't yet work with other versions.


We've written it using Python 2 (for the moment - should be simple to port to Python 3 - which we might do in the future). You can install it using "pip":

sudo pip install gomatic

Create a pipeline

If you wanted to configure a pipeline something like that shown in the GoCD documentation then you could run the following script:

#!/usr/bin/env python
from gomatic import *

go_server = GoServerConfigurator(HostRestClient("localhost:8153"))
pipeline = go_server \
    .ensure_pipeline_group("Group") \
    .ensure_replacement_of_pipeline("first_pipeline") \
stage = pipeline.ensure_stage("a_stage")
job = stage.ensure_job("a_job")


Reverse engineer a pipeline

Gomatic can reverse engineer a gomatic script for an existing pipeline.

If you run the following (we will make it easier to run this rather than having to write a script):

#!/usr/bin/env python
from gomatic import *
go_server = GoServerConfigurator(HostRestClient("localhost:8153"))
pipeline = go_server\
print go_server.as_python(pipeline)

this will print out the following text:

#!/usr/bin/env python
from gomatic import *

go_server_configurator = GoServerConfigurator(HostRestClient("localhost:8153"))
pipeline = go_server_configurator\
stage = pipeline.ensure_stage("a_stage")
job = stage.ensure_job("a_job")

go_server_configurator.save_updated_config(save_config_locally=True, dry_run=True)

This produces a script which does a "dry run" so you can run it to see what changes it will make before running it for real.

So what?

I don't have time to write about why this is a Good Idea, or the consequences of being able to script the configuration of your CI server - but will do soon.

[This article slightly updated as a result of a new release of gomatic]

Copyright © 2014 Ivan Moore

Saturday, February 1, 2014

Making life easier for future software archaeologists

Yesterday I went to "The First International Conference on Software Archaeology" run by Robert Chatley and Duncan McGregor - it was excellent. There were "lightning talks" run by Tim Mackinnon - here is a blog version of my talk.


If you have worked on a piece of software that is running in production, but hasn't been changed for a while, you may have had to do some software archaeology to work out how to make changes to it.

In this article, I list some problems that I've encountered when doing software archaeology, and some suggestions for making life easier for future software archaeologists.

My suggestions are not always applicable - but please consider them carefully. It is valuable to your client to make it easier for future software archaeologists to work with your systems. If your systems are any good they will probably be used for much longer than you think.

Where's the source?

Sometimes source code is lost (e.g. because of a VCS migration and some repositories don't get converted because nobody thinks they are needed any more). For Java projects there is a simple way to avoid losing the source code - include the source in your binary jars.

Where is the documentation?

Although it is possible that the source code will be lost, more commonly, source code repositories do survive. However, documentation systems (for example wikis) are likely to be decommissioned sooner.

Even if a documentation system isn't decommissioned, the information related to old projects can get deleted, become out of date or inconsistent with the version actually running.

In order to keep documentation consistent with the system, please commit it to the same VCS repository as the code. Depending on the VCS system used, you might be able to serve documentation to users directly from your source control system.

Where are the dependencies?

Sometimes artifact repositories are decommissioned. For Java projects, instead of using an artifact repository and ivy/maven/gradle etc, commit your dependencies into a "lib" folder and refer to them there. I know this is very controversial approach - it goes against current trends, but is actually a very practical approach. It is likely that the source code repository will outlive the artifact repository.

How do I build the software?

Sometimes build tools go out of fashion and it is difficult to set up a working build for archaeological code. Therefore, at the very least, include instructions about how to build the code in the VCS repository. Even better for the future archaeologist, commit the build tools and any setup scripts.

How do I work on the software?

In addition to being able to build the software, there may be development tools needed to work on it. For example, if the software is partially generated by some (usually hideous) tool. In such cases, some of the source code isn't really what the developer works on (e.g. GUI builder generated code).

Include (at the very least) instructions for how to set up a suitable development environment. Even better, commit the development tools and any setup scripts.

How do I run the code in the production environment?

For a large system, it can be difficult to work out how the production servers are meant to be set up. Therefore, include instructions, or even better, scripts (like Puppet or Chef), for setting up any servers etc.

How did it get to be like it is?

When looking at an old system, can be useful to see the history of decisions about how a system got to be like it is. It can be useful to have a changelog checked into the source code repository. In my lightning talk, Nat Pryce said that for a home project, he committed the complete bug tracker system; that could be very useful for a future archaeologist.

In conclusion

Fashions change (e.g. tools become obsolete), reorganizations happen and systems get migrated (and sometimes things get lost in the process). If you want to do the best for your client, remember that successful software can last a really long time, so you should leave better clues for future software archaeologists.

Copyright © 2014 Ivan Moore

Sunday, August 18, 2013

FFS! Learn how to use source control properly

This article was prompted by a tweet by Nat Pryce:
I thought - similarly - how about source control?

I have seen flaws with how source control is used at almost every company I have worked at. This article is about the ones that make me think "FFS! Learn how to use source control properly". The terminology I have used is from subversion - but the ideas are applicable to all source control systems.

What everyone agrees on (right?)

All files needed to run the build should be in source control, or generated from files (e.g. build files) that are in source control. That is, having done a clean check out from source control, on a new development machine, I should be able to run the build and it should work. Everything needed for the build to work, e.g. libraries or things that need to be installed, should either be in source control or copied/installed on the machine as a result of running the build. (That is the ideal - sometimes it is tolerable to expect some things to already be installed - but that is not ideal).

Don't make me think about which files should or shouldn't be committed

All files generated by your build should be ignored by source control. That is, having run a build, if I haven't added any files myself, I shouldn't see any files available for adding to source control. That is so I don't accidentally commit files that shouldn't be committed.

Running the build should not modify any files that are in source control. That is, having run a build, if I haven't modified any files myself, I shouldn't see any file modifications available for committing to source control. That is so I can see only the changes I've actually made and commit all of them.

Every-day use of source control

Before committing, a developer should update and run the build to check that it is safe to commit. (There are some valid exceptions - I won't go into that now). Some source control systems allow a commit to the central repository even when the working copy is not up-to-date ("svn commit" allows this in some cases, "git push" doesn't allow this at all). If you commit without updating, then the first time the code has been integrated is in source control and so you can't be sure it's going to work).

Commit everything - all modifications, additions and deletions. If you don't commit everything then you might have committed a non self-consistent set of files which don't work. If you have multiple logical changes that need to be committed, you might want to commit different sets of files in different commits in order to describe each change. That can be OK, but it is better to only have one logical change in your workspace at a time and commit that in its entirety. You will probably benefit from working in smaller steps, committing each logical change separately from the other. (When I say "one logical change" that might be as small as a one character change to fix a typo.) If you commit logically separate changes one at a time in their entirety you will end up committing many times a day, and that is a good thing. Having a mechanism to shelve/unshelve changes can be useful for allowing you to make logically separate changes one at a time, safely.


There are many different ways to use source control that are perfectly acceptable. Please comment if you disagree with the things I have written in this article.

Copyright © 2013 Ivan Moore