In this, the final blog of the series, I’ll discuss some of the advantages of adopting agile continuous integration & delivery.
So far in this series of posts, I’ve talked about continuous integration, the practice in software engineering of merging all developer working copies with a shared trunk frequently. This enables fast feed-back loops and resolution of bugs at source, quickly and early, to prevent issues being introduced into the live system later down-stream.
We discussed that developing software involves teams of people working together and the synergies of adopting a DevOps model. Automating testing, integration and provisioning of environments saves a lot of time, so that “we work smarter, not harder”. Cloud hosted environments enable us to spin up, and automatically provision, pay-as-you-go, on-demand project development environments with a just a few mouse clicks.
The stages involved in agile continuous delivery, from planning through to monitoring live updates.
In the third part of this blog, I’ll talk about our route to market – the global release pipe-line.
Global release pipe-line
To ensure that projects do not encounter delays in getting code from development into production and for continuous delivery to work smoothly, a robust, repeatable, reliable, fault-tolerant deployment system is required.
Our global release pipe-line is a sequence of cloud based environments that we use to prove that all new products or code are fit for purpose, and quality assured ready to go live. The environments represent real world production environments. We currently have 4 cloud based environments (called Red, Mauve, Amber & Pre-prod) in the global release pipe-line.
The journey down the pipe-line begins when the team checks in code to the version control system. It ends when the change goes to production – in between, a lot can happen!
To declare a build to be ready to enter the global release pipe-line, it must pass basic functional tests, so all development teams work from the same code repository, write a test case for the functionality they develop, build and test their functionality locally, refactor, and verify—only then committing their changes (frequently) to the source code repository. At key stages of the pipeline, we tag our code commits to ensure we have a full audit trail and then send the build down-stream towards production and operations.
The TfL Global release pipe-line flushes out any issues before changes reach the live website
In this post, I’ll talk about keeping the lines of communication open, testing & DevOps.
Software is abstract until it is operational
In line with our ethos for all development teams to get code to a production-ready state as quickly as possible, operations, testers, project managers, scrum master and software developers are all part of the same agile process and team-working. DevOps came about from breaking down silos, so that people work together to improve collaboration, whilst at the same time building trust and relationships.
Everyone involved in software development works together on all aspects of delivery, enabling collaboration across functional boundaries. There are lots of moving targets in DevOps & continuous delivery, we have some automation in place (e.g. automatic tests, continuous integration) but we also use tools and engineering practices, e.g. Jira, BitBucket, Confluence, Teamcity, GIT, – the rest we manage by hand.
The way we deliver to the TfL website comes together through Continuous Integration & Delivery, DevOps and Agile
This blog builds on the previous blue/green article and is part 1 of 4 posts explaining how we use the latest techniques to continuously deliver new functionality and upgrades to the TfL website. Our highest priority is to keep the website updated, continuously improving, and reducing the cycle time between an idea and useable software. Customers expect real-time updates from us, so we need to deliver new features to the website constantly. Therefore we use a variety of the latest development practices, including; agile, continuous integration and continuous delivery.
This helps us accelerate the release of software and digital services, e.g. improved web page, User experience (UX), a bug fix, or an update, quickly and with reduced manual interventions. We have found that frequent small changes to the website, are not only less risky, but we also get faster and more focused feedback about what works and what doesn’t. Coupling this with an agile development strategy, using cloud hosted, “pay-on-demand” environments, we have significantly accelerated the rate at which we can bring new services and features to the website.
We use BitBucket as a management interface for GIT (our source code repository) used for governance, code reviews & pull requests
This week marked the first anniversary of the launch of our new website. The new site, which works well on mobiles, tablets and desktops, replaced our previous website dating back to 2007.
The site, used by 81% of Londoners, brings together live data about all forms of transport in London and is a one-stop-shop for everything you need to know about public transport and roads.
Since launch customers have made over 250 million visits and viewed 1.2 billion pages and our most recent survey shows satisfaction at 90%, the highest ever.
We have also seen usage on mobile phones overtake desktop computers with 120 million visits on mobile and 107 million on desktops, with the remainder coming from tablets. This reflects the fact that the site is much better suited to mobiles and customers are increasingly using it on the move.
TfL home page at launch in March 2014
Part 2 – Auto-scaling and elastic load balancing in the cloud
In Part 1 I looked at the front-end (presentation layer) high availability cache (Varnish) that helps us deliver web pages quickly from a cache (memory). However, there are situations when there are a large number of queries also reaching our back-end, as Varnish hasn’t or can’t cache those queries – during adverse weather or industrial action, when our website traffic can spike at up to 20x usual volumes for example.
Events such as adverse weather or industrial action can lead to huge spikes in web traffic
Part 1 – High availability cache
This is the first of a two-part blog, giving an introduction to the high availability cache at the front-end of the website.
The new responsive TfL Website is not just a mobile make-over, it has been re-developed from the ground up. The site is a fundamentally brand-new, structurally re-designed, responsive website for the modern needs of the travelling public in London.
Varnish is a web accelerator which allows our website to sustain very high traffic and load many times faster by caching static & dynamic content.
This blog looks at TfL’s widgets and Open Data for developers, including our recent improvements. Get your website looking great with our redesigned Journey Planner widget!
What is a widget ?
A “widget” is a stand-alone application that can be embedded into third party sites by any user on a page where they have rights of authorship. Widgets can be considered as a downloadable small application which look and act like traditional apps, but are implemented using web technologies and our API.
What’s new ?
In October 2014, we re-designed our journey planner widget and banner to give them the same look and feel as our new website. The new widget also has email authentication built in so that we can get in touch with you quicker in future if we intend to change the widgets.
This is our brand new design for the TfL JP widget, which is now available for download.
Thanks for popping by, I’m Tariq Khurshid and lead on the website (www.tfl.gov.uk), Service Desk, Change & Release Management. In this blog I’d like to share with you the success we have enjoyed using the “blue/green” approach for software release and deployment in the new website.
This is a summary of our experience doing blue/green software deployments and releases to tfl.gov.uk using Amazon Web Services (AWS) cloud infrastructure.
Blue/green deployment of software to the website is a process that we use to safely release new versions of (www.tfl.gov.uk) without any down time or outages for customers.
The key to success is to maintain two identical production environments to switch between. As (www.tfl.gov.uk) is now hosted on virtual servers in the cloud, this is relatively easy and cost effective.
Blue/green deployment allows us to develop software to a high standard, test independently of the live site and easily package and then deploy to live. This means we have the ability to rapidly, reliably and repeatedly push out enhancements and bug fixes to (www.tfl.gov.uk) at low risk, with minimal overheads, and best of all,….. no outages for customers.
It’s been just over 10 weeks since we launched the new site, and it’s a good opportunity to round up how the site is doing, what improvements we’ve made and our future plans.
How’s the new site doing?
The site is as busy as ever, with continued high usage from all types of devices.
Key metrics from March to June 2014