Prevent unwanted changes with Sleuth deployment locking

Dylan Etkin
April 27, 2020

Service alarms are going off and you are on the hook to restore stability, but you need to prevent any more changes to production while you dig further. You could "freeze" production by announcing it in the office, sending a message on Slack, or sending an email to the affected teams, but that may not be enough or may require extra work that would distract you from debugging and fixing the problem. Regardless of what deployment system you are using, chances are you use pull requests to gate changes to production source code.

With Sleuth, you get a single button that can prevent pull requests from merging into main branches and automatically notify your team of the situation via the Sleuth Chat Ops integration (e.g., Slack).

How deployment locking works

Sleuth is a deployment tracker that keeps tabs on your key code and infrastructure repositories, as well as other sources of changes, such as feature flag changes. Deployments can be grouped into a project, allowing you to view and lock deployments collectively or individually. Sleuth provides a lock button on the project or deployment, which notifies any current and future pull requests that target the configured branch. When the project or deployment is locked, this immediately shows up on pull requests as an error and prevents any subsequent merges.

In GitHub, you can choose to make this check required or optional; the former completely blocks a merge while the latter simply provides the extra information and lets the developer choose to merge or not.

When you can use deployment locking

As already mentioned, locking can be used to help bring stability to an incident, but other use cases include locking deployments over the weekend, during a key customer demo, or even just to give a large deployment some soaking time. This feature is already used heavily by the Sleuth team to help coordinate changes to production with a geo-distributed remote team. With only an hour or two of working hours overlap between some developers, we can ensure that production is safe and not rely on inconsistent handover meetings.

Automatic locking

In addition to manual locking, Sleuth supports automatic locking of a deployment when there are unreleased changes on a production branch. For teams that encourage developers to deploy their own changes, automatic locking can help prevent multiple changes from stepping on each other. A merged pull request will trigger a lock, blocking other pull requests and notifying the team via Slack of the pending deployment. The Sleuth team uses automatic locking to help our remote development team maintain a rapid deployment pace while encouraging full ownership of changes in production. Our definition of done includes deployment and monitoring the impact of that deployment, and automatic locking helps enforce this process.

Getting started

Getting started on Sleuth is as easy as choosing a name, connecting to GitHub or Bitbucket, and pointing Sleuth to your repository. Sleuth can track deployments automatically by following commits or tags, or you can have full control and notify Sleuth of deployments via our API, like so:

curl -X POST -d api_key=YOUR_API_KEY -d sha=YOUR_SHA

Even if you haven't fully set up deployment notifications on Sleuth, you can use deployment locking immediately. We are working to expand locking to not just pull requests but other sources of change, so stay tuned.

Sleuth is free for the next few months, so give it and deployment locking a try and let us know how we can make it even better. You can check out our documentation for more information, too.