In the third part of this blog, I’ll talk about our route to market – the global release pipe-line.

Global release pipe-line
To ensure that projects do not encounter delays in getting code from development into production and for continuous delivery to work smoothly, a robust, repeatable, reliable, fault-tolerant deployment system is required.

Our global release pipe-line is a sequence of cloud based environments that we use to prove that all new products or code are fit for purpose, and quality assured ready to go live. The environments represent real world production environments. We currently have 4 cloud based environments (called Red, Mauve, Amber & Pre-prod) in the global release pipe-line.

The journey down the pipe-line begins when the team checks in code to the version control system. It ends when the change goes to production in between, a lot can happen!

To declare a build to be ready to enter the global release pipe-line, it must pass basic functional tests, so all development teams work from the same code repository, write a test case for the functionality they develop, build and test their functionality locally, refactor, and verify—only then committing their changes (frequently) to the source code repository. At key stages of the pipeline, we tag our code commits to ensure we have a full audit trail and then send the build down-stream towards production and operations.

The TfL Global release pipe-line flushes out any issues before changes reach the live website

The TfL Global release pipe-line flushes out any issues before changes reach the live website


After a successful build, other tests are run, such as functional tests and performance tests, these are the quality gates. Fast feedback is an important part of continuous delivery, so at every stage, feedback is relayed to the project team. If a stage fails, someone needs to know right away because the pipeline stops until the problem is fixed. The job of the global release pipe-line is to detect and reject bad builds. In the last stage (pre-prod), the build is approved and goes live via a seamless, zero outage, blue/green switch. This gives us the confidence that we can deploy to a production grade environment, and then cut over, or cut back, quickly.

RED is a gateway Continuous Integration (CI) to harmonise all changes in scope with other changes being checked in.

MAUVE is development integration and contains builds which have been verified by developers. The test team are the gate-keepers and authorise promotion to the next stage in the release pipe-line.

AMBER (Test). This is where business stake-holders can review/sign off before go-live and represents the first point at which the Test team functional test the integrated code from all projects (in scope). However, sometimes code has to be de-bugged after being promoted to AMBER and it is important that it can be reversed via MAUVE and back to RED without encountering new code.

Pre-Prod/Production (Blue/Green) – Since the two production environments switch roles regularly, at any one time, the environments could be either “Pre-prod” or “Production” depending on the cycle. A final sanity test is done in pre-prod and then following a check-point meeting with stake-holders a blue/green switch pushes the new code to live.

“It’s not that we’ve never had a bad build, its’ just that we’ve never had a build that has negatively impacted customers or the website.”

We prioritise keeping the system releasable over doing new work, – this is a really important mind-set shift that all people in DevOps understand and adhere to, so when something goes wrong, all technical resources focus on the fix to get the pipe-line running again.

By automating the deployment as much as possible, whether a human click was needed for the actual deployment or not, the team is able to repeatedly deploy successfully. Our continuous delivery pipeline automatically tests the application, but keeps the deployment decision as a manual step. This enables us to deploy into production with a high level of confidence and lower risk(s)

In the next instalment, I’ll discuss some of the benefits and advantages of agile continuous delivery.

Posted by Tariq

Agile DevOps, Digital Transformation, Scrum Master, Product Owner, Product Manager at TfL Online

4 Comments

  1. how long does it actually take to get s/w through the release tunnel ?

    Like

    Reply

    1. Hello, good question, that very much depends on the volume of change. If its small works, like bug fixes, incident patches and website reference data, this can progress within a matter of days through the global release pipe-line. However, for larger scope release packages with full web, api and database elements, the lead time, including re-work, testing and full QA is around 10 days. Thanks for the interest.

      Like

      Reply

  2. Hello. Thank you for this article. When a problem is found in the Amber environment, are the developers working directly in this environment and the pipe line resumes ? Or is the version going back directly to the Red environement ?

    Like

    Reply

    1. Yes Dev’s can re-work in Amber, however once the product has successfully been merged, code reviewed, tested and verified in Mauve, it becomes a viable release candidate. Amber is Test (this is also where business stake-holders can review/sign off before go-live). Sometimes in exceptional circumstances, code has to be de-bugged after being promoted to the Test environment (Amber) and it is important that it can be reversed via Mauve (Dev integration) and back to Red (CI) without encountering new code. Once approved by the test team in Amber, a release branch is created and deployed.

      Like

      Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s