How to Visualize a CommCare Project's History

This is post #6 in our Under the Data Tree blog series

We’ve experimented with a lot of different ways of using CommCare metadata to analyze, categorize, and visualize the activity of mobile users. Often we are doing this so that we can create clear and actionable monitoring tools, but more recently we have stepped back to help visualize a project history.

We’ve found that a simple but effective categorization method is to classify users based on how active and experienced they are. Specifically, we look at how many forms they submit in a given month and how many months they have previously submitted forms.  Based upon an analysis of various activity metrics, we have found that using a threshold of 15 forms submitted per month for activity, and 3 months of prior work for experience, works well for most projects. Below we show the percent of all CommCare users over time who have submitted 15 or more forms, and the percent of all users who have at least three months of prior experience. You can see that over the past couple of years that, on average, approximately two out of three active users meet each of these thresholds in any given month.

We found it interesting that the percentage of users meeting these thresholds have been so constant over the last two years, even as our projects have shifted in scale, geography, and content area. Of course, some projects have much higher or lower numbers of expected forms per user. These thresholds will need adjustments for some projects.

We can use these indicators to provide an informative view of the history of several CommCare projects. For each project, every month, we’ll put each user that submits at least one form in one of following three categories:

• Inexperienced: if they have not submitted data in more than three months (and regardless of how many forms they submitted)
• Infrequent: if they have submitted data for three prior months but submit less than 15 forms in that particular month
• Performing: if they have submitted data for three prior months and are submitting more than 15 forms

For example, here is the visualization of the project history of a very well organized and supported CommCare project.

As you can see, there were two large trainings, and both times the CommCare users quickly became experienced, e.g. all used CommCare every month, and thus quickly attained three prior months of experience. Most users performed well each month, meaning that they submitted at least 15 forms. There was a dip in the number of performing users that lasted almost a year with more users submitting data infrequently, but now most users are performing again.

Here is a second example:

Project #2 started out similarly to project #1 with about 200 users who performed very well for about a year when some users starting reporting less frequently. They then trained a few hundred more users, not all of whom use CommCare consistently. There was also a larger share of experienced users who were not highly performing, though there seems to be a set of about 100 consistently high users.

Here is a third example:

In Project #3, you can see that many users did not use CommCare consistently after training. It took five+ months for many of the CommCare users to become experienced, e.g. to submit data in three distinct months. And relatively few users submitted more than 15 forms. For this project, we broke down the infrequent users into three groups based on their number of forms, and you can see that it split relatively evenly. For this project, users are not expected to submit as many forms, so the threshold for ‘performing’ should be in the 5-10 range.

We’ve found the above visualization to be a powerful way to understand how a project has progressed. Internally at Dimagi, we’ve been tracking the total number of ‘performing’ users across all of our projects. We’ve found that the number of ‘performing’ users to be a better estimate of how many CommCare users (and their clients!) are getting benefit from CommCare than the total number total number of users that submit any forms, since we remove users who only use CommCare rarely or have not gained experience with it yet.

This visualization can be used by programs implementing CommCare to see how a project is progressing over time. The visualization can provide a high-level view that can help guide more detailed investigations, using tools such as the Worker Activity Report in CommCare or analysis of exported data. We plan to develop more visualizations such as this to help programs track the progress of their users.

Want to learn more about how to visualize CommCare data? Contact us below!