In this post, I’ll talk about keeping the lines of communication open, testing & DevOps.
Software is abstract until it is operational
In line with our ethos for all development teams to get code to a production-ready state as quickly as possible, operations, testers, project managers, scrum master and software developers are all part of the same agile process and team-working. DevOps came about from breaking down silos, so that people work together to improve collaboration, whilst at the same time building trust and relationships.
Everyone involved in software development works together on all aspects of delivery, enabling collaboration across functional boundaries. There are lots of moving targets in DevOps & continuous delivery, we have some automation in place (e.g. automatic tests, continuous integration) but we also use tools and engineering practices, e.g. Jira, BitBucket, Confluence, Teamcity, GIT, – the rest we manage by hand.
We’ve optimised for end-to-end cycle time of the whole process to reduce re-work and other unnecessary overheads. Developers commit changes frequently, this way bugs and problems are discovered closer to the time they were introduced and are easier to de-bug, enabling fast feedback loops every time there is a change in code, configuration, or infrastructure. This allows Developers to be more productive by building features, not wasting time de-bugging builds that were created months ago.
Responsibility for code quality is shared amongst the entire team, however to ensure that the codebase remains efficient, consistent and lean, we employ peer-2-peer and Lead Developer code reviews during development to catch bugs and issues early. In addition, each project team has an embedded Technical Architect, to advise on User Story review, Sprint planning and any new/complex items before development starts.
Lead Developers assure standards and best practises within their teams and through weekly catch-up meetings to bridge any gaps in understanding or differences of opinions. We encourage Development teams to reach out to their peers for advice/guidance and to share in TfL Online best practice, thus ensuring cross pollination of standards.
Sprints are aligned to our continuous weekly deployment cycle and developers maintain this pace and heart-beat. Everyone understands that when a release is “signed off” to be promoted to an environment this is final. This ensures that the team’s work is defined as “done”, only when the change has been put into production and is working satisfactorily – not at a remote Developers workstation, completed ages ago.
We always do a final QA via a project release note before promotion to the global release pipe-line. An important aspect of this final QA is, assurance that the build time remains as lean as possible and that code additions are efficient and performance optimised so as not to adversely increase server load or CPU.
It’s good to talk
When continuously innovating and delivering new software, team communication and planning is crucial to maintain momentum, quality and the business drivers. It’s important to get everyone together regularly, to secure buy-in, understanding and collaboration.
We do this through our Tuesday weekly release planning meetings – we aim to be as lean as possible by minimising the need for follow-up actions and further meetings, so we have all of the required people to make decisions in the room.
At this meeting, all DevOps Leads get together with business and projects, to optimise and adjust continuous delivery scope, plans and schedules for each release package. The complexity of continuous delivery is directly proportional to the number of changes at each level in the stack e.g. infrastructure, Data-base, web application tier, ..etc. and there is always a ripple effect to consider as well, because we can’t change one piece of code without affecting other code. So a critical part of this meeting is to carefully manage scope and be sensible over the volume of change we are deploying each week.
Lines of communication are also kept open and transparent via Jira tickets, Hip-chat and ongoing check-point meetings, this is essential, because of the “domino effect” in our current set-up (there is only one route to market), and one release could quickly block the global release pipe-line and potentially throw the weekly release schedule out of sync if not carefully managed, co-ordinated and quality assured.
Continuous attention to technical excellence
Testing is not something we do after development is complete, testing is something we do through the full life-cycle and encourage everyone to be responsible for quality, and we are always looking to improve process in order to build quality into the product in the first place.
Testers have cross-functional roles, communicating and working alongside Business Analysts and project teams in order to advise on the creation of; good quality requirements, User Stories, code reviews, component testing strategies and test techniques.
Intelligent agile testing means analysing what moving parts do we really need to test ?, So we have several intermediate levels, testing in isolation first, then together with a good separation of unit, functional, integration and regression testing.
We conduct full regression, compatibility, accessibility, security penetration tests (application level) in project environments when development is complete. Then in the early stages of the global release pipe-line (Continuous integration & Dev environments) – we use light smoke tests and cross-project, developer verification to check for merge conflicts and any other issues.
We run testing in parallel, to speed up feedback time. If these tests fail, we don’t stop Developers checking in code, but we do make sure a Lead developer & Tester pair, make it their highest priority to work together and fix the problem immediately.
When end-to-end testing has been completed, right up to the blue environment (pre-prod) including security penetration tests (application & infrastructure), the release package is then released to production and goes live, via a blue/green switch.
If the deployment goes wrong, no worries, we instantly revert to the last known good state in pre-prod through an instant blue/green roll-back and then focus on a hotfix.
In the next part of this blog series, I’ll discuss our route to market – the global release pipe-line.