Communicating Changes Made to Your Software Product

At Dimagi, we are constantly making changes to our product that affect users, but have struggled to create and adhere to a system for letting users know about those changes. Recently, we set out to design a solution to this problem that would be both practical for our team and satisfactory to our users. We knew that any process we chose would need to be executed consistently for it to be effective. For this purpose, we created a GitHub Webhook that enforces label requirements on pull requests. To find out how we used this in our overall process, how to build something similar, and how to use our label enforcer for your own purposes, read on!

This post is likely to interest you if any of the following are true:

• You are interested in knowing how to use GitHub Webhooks to design an internal tool for your team.
• You work for a technology company that has yet to perfect its strategy for communicating product changes to users, or a strategy for some other communication stream.
• You are interested in designing a system for your team(s) to follow that stems from enforcing certain requirements on pull requests.
• You use Dimagi’s product, CommCare!

 The Problem

If you are reading this post primarily for the discussion around the technical components and/or you would like to use a similar technical strategy to solve a totally different problem, skip ahead to “Enforcing Label Requirements” below.

At Dimagi’s most recent offsite for our tech team, we held several small working group sessions to discuss challenges facing our team. The problem that one of our working groups was tasked with addressing was an issue that had been plaguing our team for a while — how to effectively communicate product changes to our users. More specifically, we wanted to limit our communications to those changes that users care about, without drowning them in useless noise.

On its face, this sounds fairly straightforward. But the structure of Dimagi’s teams and product add a couple of unique challenges into the mix:

1. Our users differ widely across several axes, including the type of CommCare project that they are working on, the component(s) of our product that they use, and their level of skill and experience in using it. With all of these factors at play, having a single communication stream was unlikely to work for us.
2. Developers at Dimagi have a lot of autonomy in making changes to the platform. For this reason, changes sometimes get made without the knowledge of our Product Team, especially if user-facing changes are made during the course of a bug fix. We needed a more reliable way to notify Product in these instances so they could, in turn, inform our users.

In the past, the “system” that we were using relied on developers’ intuition; if they thought the change they were submitting was something that users would notice or care about, then they would notify our @product GitHub user on the pull request. At that point it was up to the product team to process what the change meant and determine who it should be communicated to — if at all — and how. In general, this meant that only fairly major changes got communicated. Plenty of things that mattered to our users were either getting missed entirely or not communicated to the right people.

There were plenty of other sub-problems to this issue and shortcomings to how we were handling it, but we decided to focus on coming up with a solution that addressed three main things:

1. Clearly defining the different types of user-facing changes that could be made to our product, in accordance with our different user segments and how our product is structured.
2. Removing our reliance on developers remembering to alert the product team about a user-facing change.
3. Having a clear definition of what types of reports users can expect on changes to the product, and when.

The Solution

The solution we came up with has the following parameters (each of which corresponds roughly to the numbered problems stated above):

1. Define a set of custom labels in each of our GitHub repositories that represent all of the categories of user-facing changes we identified to exist for our product. We settled on four primary categories (with a few subdivisions not mentioned):
   • All Users → Changes that will be visible to all users.
   • Flagged → Changes that will be visible only to those users who have a certain flag or setting enabled.
   • Custom → Changes that have been made to the product for a specific project or client, and will only be visible to their users.
   • Invisible → Changes that are not user-facing at all (such as code refactors).
2. Require developers to add one of these labels to every PR they submit in all repositories of our product (this is why we have a label for non-user-facing changes).
   • Enforce this requirement via a GitHub Webhook and the GitHub APIs; if a PR doesn’t have at least one of those labels, it should have a failing check. (The implementation of this piece is outlined below in “Enforcing Label Requirements”)
3. Define a clear process and timeline by which the Product Team will do the following:
   • Review PRs marked with any label that indicates a user-facing change.
   • Create reports detailing different subsets of those changes, targeted at the subsets of users interested in them.

Enforcing Label Requirements

After defining our set of labels and adding them to each of our GitHub repositories, we were ready to set up the enforcement structure. This was achieved through a system that utilizes GitHub Webhooks and the GitHub APIs.

GitHub’s Webhooks infrastructure allows you to configure any web app to subscribe to events that occur in your repository. The Webhooks console requires you to specify only two things: the endpoint where your app lives, and the type(s) of event(s) you want to trigger the hook. When a triggering event occurs in your repository, GitHub sends a POST request to the specified endpoint with a JSON payload representing the event.

Our process for setting up a Webhook that enforces our label requirements on PRs was as follows:

1. In each of Dimagi’s primary GitHub repositories, configure a Webhook that is triggered by any “Pull request” event.
2. Build a basic Flask app that can receive and process the JSON for a pull request event (the code for this app lives here). Build the app to respond to event payloads that represent a new PR, a new commit on a PR, or a change to the labels on a PR.
3. When any qualifying event is received, the app uses the GitHub APIs to fetch the current labels for the PR, compares those to the requirements we’ve defined, and then sets the status for the most recent commit of the PR to success or failure, depending on whether the requirements were met:
4. Host the app on Heroku and point the Webhook configuration in each repository to the Heroku endpoint.
   

Making it Configurable

We decided early on in the process of building our app that we wanted to make something that would be useful to other people interested in doing the same thing. In order to do that, we had to make it easy for other potential users to specify their own set of label requirements. At the same time, we wanted to keep the capabilities of the app as robust as possible. The strategy we landed on for achieving this balance was as follows:

Via either a config file or environment variables, users can specify up to three custom values:

• required-labels-any: A comma-separated list of label names for which a PR is required to have at least one of the labels in the list.
• required-labels-all: A comma-separated list of label names for which a PR is required to have all of the labels in the list.
• banned-labels: A comma-separated list of label names for which a PR is required to have none of the labels in the list.

The app takes these three rules together to specify the full set of label requirements for any PR that it receives an event payload for. You can see this system in action in the sample pull requests below:

While not logically exhaustive, we think this system should allow the majority of potential users to express the requirements that they need to with ease. Below are a few examples of how someone might express their label requirements in our system:

1. Your team requires all PRs to get sign-off from your Architecture and UX teams before being merged.
   1. required-labels-all:ux-signoff,arch-signoff
2. Your team uses a certain label to flag something as “work in progress” that should never be merged until the label is removed.
   1. Banned-labels:wip
3. Your Site Reliability Engineering team requests that all PRs get marked as either high- or low-risk before being merged, so that if the site goes down or something is wrong, they can quickly scan for changes that developers anticipated might have adverse effects.
   1. required-labels-any:high-risk,low-risk

Using Required-Labels Yourself

If you’d like to start using the Required-Labels app in conjunction with your own GitHub Webhook, see the README at https://github.com/dimagi/Required-Labels.