DORA metrics and Deployment Frequency
While metrics have always been fundamental to improvement in the business world, the growing prominence of DevOps in recent years has elevated their importance in the context of software development.
To build a continuous improvement culture, you need a set of metrics that allows you to establish a baseline and inform where the improvement opportunities lie. Arguably the most popular of them is DORA metrics. To learn more about DORA metrics, check out this explainer article.
In this post, we will focus on Deployment Frequency, one of four DORA metrics.
What is Deployment Frequency?
Deployment Frequency tracks how often code is deployed to production, or otherwise deployed to end users at the highest level. This clarification is essential as mobile apps, for example, have a different release process compared to web applications or services that expose APIs.
Interestingly, Deployment Frequency itself is actually a proxy for another metric the DORA Institute researchers and Accelerate book authors were looking for.
From the Accelerate book:
"Reducing batch size is another central element of the Lean paradigm - indeed, it was one of the keys to the success of the Toyota production system... However, in software, batch size is hard to measure and communicate across contexts as there is no visible inventory. Therefore, we settled on deployment frequency as a proxy of batch size since it is easy to measure and typically has low variability."
Deployment Frequency is also a decent proxy for understanding how much business value gets delivered to end users and customers. Because elite teams can ship multiple times a day, over time they will have delivered much more business value than teams who deploy less often.
Why is Deployment Frequency important?
Deployment Frequency as a metric incentivizes the team to deploy more often in smaller changes. This is beneficial for the team and individuals within it.
Individual developers benefit from working on a smaller surface area, because smaller changes are easier to understand. They can find the root cause of an issue faster, and remediate or push out a fix faster, with less context shifting.
For the team as a whole, more frequent deployments means shorter feedback loop, allowing for faster product iterations with the broader team including product managers, product owners, and designers.
The latter point is arguably the most important of them all - modern product and software development has been centered around the understanding that the sooner feedback can be collected and integrated, the better the product will be. Early feedback can also help ensure that any missteps can be caught and addressed before the effort of changing direction becomes significant and saps team morale.
With this understanding of Deployment Frequency, we move on to the crucial part of this post: how to improve it!
How do you improve Deployment Frequency?
The most impactful way to improve Deployment Frequency is by introducing automation to the deployment process.
Deployment process is usually hobbled by a significant amount of human interventions in lower-performing teams. In some organizations, for example, this manifests as requiring a Change Advisory Board to help prioritize and schedule changes going out to the end users.
Removing such human interventions should increase the speed with which teams can deploy to production. Don't fall into the trap of a having to choose between an entirely manual process and a fully automated one (often called continuous deployment), as there are plenty of opportunities in between for automation.
Manual gating only be necessary for aberrant conditions in a CI/CD pipeline, rather than a requirement for the deployment to move forward.
Assuming process changes have been implemented to remove manual gating as much as possible, the next step is to focus on CI/CD setup, ensuring that building, testing, and deploying code are fully automated. For example, elite teams utilize continuous integration pipelines that trigger builds and tests for every code commit.
Elite teams utilize database change management to help keep databases in sync and ensure that any changes can also be rolled back.
It is worth noting that teams working on microservices can follow the same process as teams working on larger monoliths. The only difference might be in the frequency of deployment. Smaller service changes might go live for each commit, whereas a monolith may only deploy once or twice a day, using a release train.
A release train is where multiple changes are batched together, coordinating changes to be released at the same time. This approach adds complexity to the release process but is still a significant improvement for teams that want to ship new features or fixes frequently.
When it comes to improving Deployment Frequency, any initiative should follow a plan-do-study-act model - asking what improvements need to be made, understanding the success criteria, and ideate on changes that can be made to deliver the desired improvement.
Tracking deployments is a prerequisite to be successful in improving Deployment Frequency.
A CI/CD pipeline typically consists of several steps, including retrieving code, testing, building, and deploying the change. Each step may include several sub-steps. For example, the test step may run separate test suites for unit tests, integration tests, and end-to-end tests. Meanwhile, the build step may include compiling the source code to generate a binary, followed by creating a Docker container that includes that binary. You may deploy to multiple environments and run smoke tests for each as part of the deployment step.
No matter which step is taking place, tracking what is happening and how long it is taking is key. By regularly tracking and reviewing the time spent in CI/CD, you can make sure that steps aren’t taking longer than would be expected while accounting for code or data growth.
Look for steps that are taking too long. There may be instances in which steps are continuously starting and stopping containers instead of reusing the same container. At the same time, realize that not all optimizations are necessary. Shaving milliseconds from a test run may only net a few seconds of savings over a year and likely aren’t worth investing in.
Take a look at this comparison article of several solutions for tracking Deployment Frequency automatically.
Dos and don’ts of Deployment Frequency
There are some dos and don’ts around Deployment Frequency that are worth understanding.
(DO) Invest in visibility. The team needs to track deployments end-to-end and their frequency before making any changes. The impact of any improvement initiatives cannot be quantified or validated without a baseline measurement.
(DON’T) There are no shortcuts. Eliminating a code review may increase Deployment Frequency, but it will decrease code quality, harming the team. (We’ll cover this in our post on Change Failure Rate.) Similarly, skipping automated tests may speed up delivery and deployment, but it removes assurance that the system is working correctly. The goal is to make the review and test process seamless, not to remove it.
(DON’T) Competition within a team can hurt the team. An effective team works together. While some management styles use gamification to help motivate teams, doing so should be based on badges or campaigns rather than individual metrics. Competition amongst team members can have adverse effects and be detrimental to effectiveness and morale.
Any organization wanting to get better at releasing new features will need to address the speed of deploying code changes.
As we’ll see in future posts, Deployment Frequency is the building block for all remaining DORA metrics. For example, increasing Deployment Frequency at the cost of quality or reliability does not help the business and only frustrates users or drives them away.
Working on improving deployment frequency means implementing practices such as automated CI/CD pipeline, deployment automation, and more. If you want to learn more about all the best practices to enable elite-level deployment frequency of multiple deployments a day, check out this 60-page ebook "Zero to 100 Deploys a Day".