When I was CIO at Cuscal, Agile methodologies were introduced as a response to a development crisis. One of the major and time critical programs was in trouble of missing its deadlines. Our recently hired Development Executive suggested that we should introduce Agile methods to break the logjam. So we hired an Agile Coach, created a scrum master role, reorganised the teams and began our Agile journey.
Benefits of Agile
There are many benefits of Agile. For us, the first benefit we saw was that whole team was focussed on building the most important features/ functions. This often required a close collaboration between the team and the product managers.
The second benefit was that the traditional role silos and waterfall methods,. The hand-offs from Business analysts to developers to testers via detailed specifications were replaced with greater collaboration upfront. This enabled a tester to contribute to the testing requirements and ensure that less bugs were found in subsequent testing. It also improved speed (or the cycle time).
A third benefit of Agile was the focus on continuous improvement. A small team focussed on removing the hurdles from the development process and reducing the time taken for code migration, regression testing, and code releases. These small improvements helped to gradually increase the overall speed of the delivery.
Another benefit of Agile is constant prioritisation. Unlike traditional project management where project plan governs to tasks to be worked on, in Agile there is prioritisation at the end on each sprint (short development cycle). Trade-offs are made regarding what is most important and what we can live without, say for the launch, all the time. The result is a functional product, which just gets better with each integration.
However, we did encounter some challenges. The expected speed of development did not always match the actuals, some team members struggled with the new methods, and the coach had his hands full moulding the team. Nevertheless, the new approach ultimately allowed the Cuscal team to continue to meet deadlines and deliver a quality product.
Based on this success, we decided to rollout Agile to other teams. There was far more buy-in from the other teams, with the product teams also being supportive. The Agile approach was certainly improving the predictability and the quality of delivery, but it was not yet clear if there was an actual improvement in productivity or throughout the Agile teams. On it’s own, Agile can achieve only so much. We needed to consider the next step, automation. This is where automation using continuous integration/ continuous deployment (CI/CD) came in.
What is CI/CD?
CI/CD is an approach within Agile that integrates critical late-stage activities, like testing and deployment planning, into the code development part of the software development cycle. For example, the team could begin to develop unit-testing cases along side the code. Using automation, these cases would be tested each time the developer completed the code so that the code errors were corrected in the development process. Since the process is automated, there was no time wasted or the need for a tester to perform unit testing. Furthermore, since each piece of code was unit tested early in the cycle, there were less defects overall.
Going further, CI/CD includes continuous integration. In continuous integration software builds are automatically tested in an integration environment to ensure that the new software works well with the whole.
Next comes continuous deployment (CD), which is about migrating applications automatically across environments then finally in the live or ‘production’ environment. CD also includes self-service provisioning of hardware and an automatic release of new software. This process of bringing forward late stage activities is called ‘shifting-left’ and is at the heart of CI/CD and also DevOps.
With automated testing, integration and software releases CI/CD can truly unleash the productivity improvements and cycle time reductions that were promised in Agile. However, making CI/CD work is not easy-there are changes in control, governance and in the IT organisation that need to be considered for it to work.
Controls –Typically software releases in companies are tightly controlled to maintain quality and avoid costly mistakes. In the past, when there were only a few mega releases each year, this was probably justified. However, to respond to fast-changing business and customer expectations, such controls now add delays. When software testing is built into the development process itself, it is time to question such controls. In the days of virtualisation and cloud computing, controls on hardware (servers, storage etc.) provisioning are beginning to look antiquated.
Governance – Many companies are moving away from project based to product based funding models. The benefit of this is that there is ongoing funding to keep the product relevant in the marketplace. However, companies must consider how to apportion the product funding to new features vs. applications maintenance and infrastructure in this new paradigm.
Bug fixing – At present, software bugs are typically assigned to a support team. Many organisations are assigning these back to the original developers as part of a ‘you built it you own it’ philosophy. This prevents software changes from occurring by both the applications and support teams.
CIO responsibility – With CI/CD, CIO responsibility includes the implementation of an optimal infrastructure environment for business-oriented developers. These individuals need infrastructure independent platforms so that they don’t have to worry about compatibility. In other words, the role of IT organisations evolves to the provision of a ‘paved-road’ and suitable tools for developers to innovate and achieve speedy delivery.
Automation – Replacing cumbersome approval processes with automation has the potential to greatly increase agility. However, this is not a straightforward decision. The environments contain legacy applications and fast changing newer applications, and balancing these two diverse demands is complex
Typically, testing automation is a good first step. Often the benefits of covering more new code through automated testing can justify a move to automation. Rolling out automated testing is a phased process. The applications that have adopted Agile models should be given priority in the testing automation process. Automated testing can lead to high standards of reliability. Even when only a partial set of tests is automated, productivity will improve when these are repeated multiple times. This, in turn, will free up time to automate more code or tests.
In summary, while Agile creates the right collaboration and focus for software development, automation provides the engine to improve productivity via automation and removal of the overheads. Both Agile and CI/CD can be adopted in a phased manner. This can allow organisational learning as a well as culture change, which in turn leads to organisational acceptance of the new methods in a gradual manner. A phased approach also allows time for the development of new technical skills in building and using the underpinning infrastructure, which supports continuous delivery and integration.