How quickly do frontline workers adopt mobile technology?

This is post #1 in our Under the Data Tree blog series.

How long does it take CommCare users to reach a steady state of use? We expect that it will take the average user some time to get comfortable using CommCare and that most users require a period of time before they start using CommCare with all of their clients. How long this process takes is an important question because it can inform how to introduce mobile technology to frontline workers (FLWs). It can help programs plan their implementation timelines to allow sufficient time for users to get comfortable using their mobile apps before trying to expand further. Furthermore if we know what to expect, program staff can identify individual users who are exhibiting slow uptake of CommCare and can provide extra support and training for such users.

For this investigation, we’ll explore our data during the first year of CommCare usage. We analyzed about 630 users who have used CommCare for at least one year, including only those that submitted data for at least 10 months of their first year. We also randomly selected users from very large projects, so that no single CommCare project accounted for more than 10% of users.

Figure 1: Median number of cases visited per month of CommCare use

The above graph (Figure 1) shows how many different cases are visited (either registered or followed-up) in each of the first twelve months of usage. Note that this shows the median of all users in their first month, second month, etc., regardless of when they actually started using CommCare. We can see a steady increase in how many cases were visited over the first six months of usage before it levels out.

In order to get a more detailed look at patterns of behavior as users gain experience with CommCare over the first year, we broke down each user’s performance by quarter. We averaged their first 3 months of activity levels and called that Q1, months 4-6 activity levels averaged to become Q2, and so on for the first four quarters. We then looked at how user behavior changes from quarter to quarter.

We see a median increase of 22.9% in the number of cases visited in our users’ fourth quarters (Q4) compared to their first quarters (Q1). The mean increase from Q1 to Q4 was 96%, implying there are some users who showed very large increases. The table below shows the median and mean percentage changes between each pair of quarters.

Table 1: Median and Mean Change in number of cases visited between different intervals

Between Q1 and Q2, there is a substantial increase in the number of cases visited (median % change = 17.8%), but the median % change between Q3 and Q4 dropped dramatically to 0%. From this trend we can infer that in general it takes approximately 6 months for the median change between quarters to reach 0.

Figure 2: Percentage change in number of cases visited by intervals between quarters

The above graph provides more detail about patterns of user behavior between quarters. It shows how many users have substantial or moderate changes in activity levels between quarters. For example, even though the median change from Q3 to Q4 is 0%, about 20% of the users show an increase in activity of 50% or more, and about 10% show a decrease in activity of 50% or more. These graphs show that there is a good deal of variety on the user level, but with the average rate of change reaching a steady state within six months. Specifically by the Q3-Q4 interval, there is a larger portion of users who remained within 20% of their prior month’s activity. In order to understand when or if an average user reaches a steady state, we will need to look at user-specific patterns of change over time, something we hope to tackle in upcoming blogs. The data presented here suggest that there may be quite a bit of variation over time for many users.

We were also gratified to see a general upward trend in the activity level of users as they gain more experience with CommCare. We have been assuming that this reflects increased capacity to use CommCare. However, one of the potential benefits for a program to adopt CommCare is to increase the coverage of that program. It may be that the users are indeed becoming more active and seeing more clients. It may also be that it takes programs some time to learn how best to support and supervise their users, and so the time it takes for new users to ramp up will decrease as the program gains overall experience implementing CommCare.

As we accumulate more CommCare data we will also be able to look at a longer time horizon and see what happens to the average number of cases visited over 1.5 – 2 years. In addition we hope to dive further into other aspects of worker experience, such as amount of time spent on CommCare, use of audio files, etc. This should help plan mHealth deployments for FLWs, as well as help plan for and identify individual users who may need extra help or more time to reach steady state.