Prevent unwanted changes with Sleuth deployment locking
April 27, 2020
Service alarms are going off and you are on the hook to restore stability, but you need to prevent any more changes to production while you dig further. You could "freeze" production by announcing it in the office, sending a message on Slack, or sending an email to the affected teams, but that may not be enough or may require extra work that would distract you from debugging and fixing the problem. Regardless of what deployment system you are using, chances are you use pull requests to gate changes to production source code.
With Sleuth, you get a single button that can prevent pull requests from merging into main branches and automatically notify your team of the situation via the Sleuth Chatops integration (e.g., Slack).
How deployment locking works
Sleuth is a deployment tracker that keeps tabs on your key code and infrastructure repositories, as well as other sources of changes, such as feature flag changes. Deployments can be grouped into a project, allowing you to view and lock deployments collectively or individually. Sleuth provides a lock button on the project or deployment, which notifies any current and future pull requests that target the configured branch. When the project or deployment is locked, this immediately shows up on pull requests as an error and prevents any subsequent merges.
In GitHub, you can choose to make this check required or optional; the former completely blocks a merge while the latter simply provides the extra information and lets the developer choose to merge or not.
When to use deployment locking
Other than to help bring stability to an incident, other use cases for deployment locking may include: locking deployments over the weekend, during a key customer demo, or even just to give a large deployment some soaking time. This feature is already used heavily by the Sleuth team to help coordinate changes to production with a geo-distributed remote team. With only an hour or two of working hours overlap between some developers, we can ensure that production is safe and not rely on inconsistent handover meetings.
In addition to manual locking, Sleuth supports automatic locking of a deployment when there are unreleased changes on a production branch. For teams that encourage developers to deploy their own changes, automatic locking can help prevent multiple changes from stepping on each other. A merged pull request will trigger a lock, blocking other pull requests and notifying the team via Slack of the pending deployment.
The Sleuth team uses automatic locking to help our remote development team maintain a rapid deployment pace while encouraging full ownership of changes in production. Our definition of done includes deployment and monitoring the impact of that deployment, and automatic locking helps enforce this process.
Getting started with Sleuth
Getting started on Sleuth is as easy as choosing a name, connecting to GitHub or Bitbucket, and pointing Sleuth to your repository. Sleuth can track deployments automatically by following commits or tags, or you can have full control and notify Sleuth of deployments via our API, like so:
Even if you haven't fully set up deployment notifications on Sleuth, you can use deployment locking immediately. We are working to expand locking to not just pull requests but other sources of change, so stay tuned.