Why Pilot Studies Are Still Important

Dimagi Program Manager Jeremy Wacksman explains why pilot programs can still provide immense benefits to scaling programs and why they deserve a bit of extra attention.

Several years ago, it became very in vogue to express concern and complain about an overabundance of pilots in the ICT4D community. There was certainly good reason for concern: Many small projects have high failure rates, at times causing confusion and fragmentation of effort. The most famously cited outcome was the moratorium on new mHealth projects in Uganda. There was a general fear that if the community didn’t get its act together, we would reduce our credibility to the point that no one would think large-scale ICT projects were feasible in the development space. Back in 2013, we even tried to adapt our CommCare implementation model for this exact problem.

Since then, it feels at times like expressing concern about any pilot has become a reflex. If you want to be a real “thought-leader” in the space, make sure you add “pilotitis” to your vocabulary and don’t call whatever you are doing “a pilot.”

And with that, I would just like to say (for the record, out loud): Pilots are still important!

I’ve worked on many pilots that proved scale was possible–but scale can mean different things. As our CEO, Jon Jackson, has discussed, scale in terms of users is just one form of scale; there is also scale in breadth of programs and scale in vertical integration. I think the lesson from the prevalence of “pilotitis” is not to avoid pilots, but to focus on the right kind of pilots.

Pilots are the time and place where we get our technology right and ensure we are developing something useful for the user. They are when we establish and test processes that are required for the long-term success of a program. The tension is that pilots are meant to be both a special, focused program effort that may differ from the final program, as well as a process to get us ready to scale and succeed in that same final program.

There are a few key considerations that come to mind when discussing and planning for the pilot phase of our projects:

1. Pilots should be understood as special

Pilots, by their role as places for testing and iteration, require extra attention and support. It is a time to stress test our theory of why this program will be effective and we want to give it the best chance to succeed. This means the pilot users or sites might be be unique in terms of geography, user types, connectivity, or other factors.

For this and other reasons, pilots may not then be fully representative of the experience for a diverse set of users at scale. However, we can be intentional about articulating what makes those pilot users special and strive to get good representation where feasible.

One danger when we do a pilot for a large project is that we become very good at running that site, but not at the big scale project. This risk can be mitigated by proactively thinking about the various dimensions of scale that need to be considered and understanding the broader sets of users, geographies, and infrastructure earlier on.

2. Pilots are not just for testing apps

Just as pilots are not only for proving a program’s viability with more users, pilots are not just for apps. They can test so much more, like training methods, supervision performance, and reporting processes. All aspects of a program could benefit from a bit of focused iteration and testing before going big. For example, in many projects, when we roll out a new customized reporting tool, we may again start with a small group of pilot users within the larger pilot cohort to get feedback and iterate before rolling it out more broadly.

3. Pilots are not just for the start of projects

Even if your program has been successful for some time, you can always work on improving it. But at the same time, you don’t want to just introduce wholesale changes to a program either. In our spirit of continuous learning, pilot users or sites can serve as the “beta testers” for new app content, integrations, or processes that are introduced throughout the course of the project.

The first sense of “pilot” I was exposed to was in the CRS ReMiND project in India, which had 10 ASHAs in the initial cohort of CommCare users. This pilot was successful and showed that ASHAs would experience increased productivity and efficiency at scale. So, when the app was scaled to over 250 users, those ten initial ASHAs served as the pilot for various iterations and phases of the project in the months that followed, as well as key advisors on the program’s direction.

Obviously, no one wants to fail the first time they launch their program. But it would be foolish to think that would never happen. If you start small by launching a pilot, including focused effort and support for testing different aspects of your program, you can make sure that when you do fail, you have clear and actionable learnings to turn that failure into a successful program at scale. Otherwise, it’s just a failure. And no one wants that.