A 4-step Process for Selecting Digital Tools to Support Successful Frontline Programs

Author: Meri Ghorkhmazyan, Senior Director of Monitoring, Evaluation and Learning at Mercy Corps

This piece was originally published at Nethope.org. 

I’ve spent my career building and leading monitoring and evaluation systems and programs across some of the world’s most impactful nonprofits. I am passionate about using data to inform and improve program design. And I’m keenly attuned to the role that digital tools can play in making the most of scarce resources – time and money. After nearly two decades in the Monitoring, Evaluation, and Learning (MEL) space, I joined Mercy Corps. In the last 2 years, we worked through a collaborative process to select and implement a set of best-in-class digital tools across Mercy Corps’ global offices to support both M&E and program teams. I believe this process can serve as something of a blueprint for other organizations needing a similarly robust set of tools to support stakeholders globally. Today, I’m sharing our process in the hopes that it will help fellow MEL, M&E and evaluation professionals.

I joined Mercy Corps in October 2020 as the Senior Director of Monitoring, Evaluation, and Learning (MEL). Mercy Corps is a global humanitarian organization with over 5,400 team members working across 40+ countries affected by crisis, disaster, poverty, and climate change to meet urgent needs and develop long-term solutions for lasting change. And I immediately discovered a challenge: Our global program teams were using many different digital tools, the selection of which was guided by prior experience or what they were familiar with. A survey showed that some of our programs were on average using 7 to 8 different tools for data collection, analysis, and visualization. For example, in some cases teams were using up to 7 different platforms for data collection across multiple programs in one country. 

This was causing problems on a number of levels. 

• Privacy and security issues: Some tools selected by country teams did not necessarily meet the strict GDPR and data security requirements that Mercy Corps required. 
• Wasting time at the HQ level: When teams inevitably asked for support using these many different tools, my MEL team had to build expertise in the wide ranging set of tools in order to support them, and it took extra time the team could have spent in more impactful ways.
• Wasting time at the Program and Country office level: Country staff spend a lot of time procuring the tools, and moving data manually between various tools and platforms that didn’t integrate well with each other.
• Wasting money: With so many tools across different teams, the organization was most certainly spending more than it needed to. With Mercy Corps’ size, I knew that we could get better rates if we procured tools centrally.
• Inaccurate and unclean data: The data collected by the MEL team was not trustworthy due to all the different sources, formats and tools. 

So we got to work designing and launching a thoughtful process to solve these problems and streamline the way MEL/mobile technologies were selected, procured and used within programs in the organization. The entire process took nearly 8 months start to finish. Here’s how we approached it: 

Step 1: Get Buy-in and Build the Team

You need to have the support and buy-in from the organization and its senior leadership to be successful at rolling out organization-wide guidance and agreements with digital solutions. At Mercy Corps, this meant first establishing strong synergies and agreements between the technology and data teams to prioritize this issue and work together to identify solutions (such as M&E and Learning, IT, and Data for Impact teams). Then, reaching out to the M&E and Program Directors and managers at country and regional levels for deeper engagement and learning, together with building support from the organization’s senior leadership.

It also meant getting the right people on the team who had the expertise and management support to roll up their sleeves and do the work. In November 2020, I started building a task force of M&E, T4D and IT team members to work collaboratively, assess the true scale of the misalignment across various platforms, and identify solutions for streamlining. 

The task force started by clearly articulating a terms of reference, the deliverables it aimed to achieve and what success would look like, as well as how the results achieved by the task force would support the organization, and the deliverables of each engaged team.  My team also did the legwork to ensure we could understand and communicate the “what’s in it for me” for each of the task force team members. For the MEL team, one of the biggest priorities to achieve that year was providing clear guidance on how mobile technologies were selected and used to enable the country programs to achieve one of the standards required by the organization’s MEL Policy (The Mercy Corps MEL Policy requires that all programs develop a clear MEL Tech plan articulating how tech is used in programs). For T4D and IT teams it was important to see a strong alignment and potential for interoperability across the technologies used by country programs, because it would enable us to move data easily. The program teams were excited about having clear guidance from the organization on the technologies they should use, since very often those were procured at the start up phase, and required additional LOE to research and identify appropriate solutions. 

Step 2: Deep Learning to Understand the Problem and Identify Solutions

When the task force started Mercy Corps had already conducted 2 MEL tech surveys with the country programs (one in 2018 and one in 2020), mapping the various technologies used at different phases of the data lifecycle, starting with data collection, analysis, visualization, mapping, and communication. We started with a thorough review of the survey results. This helped us understand the baseline with questions around:

• What tools are program teams using for data collection, visualization, analysis, and communicating data? 
• What are the training needs among program and MEL staff? 
• What are the pain points and issues with current tools? 

One interesting finding in the survey data was that when we compared the technologies used and the needs for training, we noticed that training was required for almost all technologies that teams procured. We wanted to know what guided the selection of those technologies if training on them was in such high demand. Then, we dug deeper to better understand the problem. Between December 2020 and Feb 2021, the task force conducted in-depth interviews and focus groups with country teams across 12 countries to interview staff who selected mobile technologies, those who used it, those who signed off on the purchase requests, as well as those who requested training, used the platforms effectively, or struggled to do so. We selected countries that had ample successful experiences and those who reported challenges. We created case studies and eventually developed a white paper summarizing the findings – both the challenges and outlining 6 potential solutions. This whitepaper helped us engage the organization in a discussion around effective ways of selecting and using technology platforms, and offered guidelines on how to improve our capacity to do it well. One of the recommendations that came out: enterprise purchasing models were necessary for the most frequently used technologies to decrease the time spent on procurement.

We also discovered in this phase that teams were making decisions on what tools to use based on limited analysis and consultation – without any guidelines – and often chose the tools they happened to know. A big surprise to my team came when we learned country offices actually wanted guidance from HQ. We had wrongly assumed our input would not be welcomed. 

This process end to end took 200 hours across chairing the program, coordinating meetings, writing and editing the whitepaper and holding discussions and interviews. Each task force member had the support of their management to contribute LOE to the project, which was essential. This sounds like a lot of work, and it is. But it’s essential to invest the time to do this right, because the opportunity cost was much higher. 

Step 3: Choose Tech Partners for Enterprise Solutions

With the problems identified and solutions in hand, one of the most urgent tasks was figuring out how to offer a better procurement experience at a cheaper cost per service for data collection platforms, since those were in high demand. In August, 2021, we started with engaging our finance teams, compliance and procurement teams to identify a process that would allow us to procure and charge the use of those technologies to programs. Once those were agreed upon, Mercy Corps announced a global tender requesting proposals from vendors that offered comprehensive data collection services as outlined in the RFP.

You can see Mercy Corps’ RFP for a Comprehensive Data Collection and Reporting Platform here: https://www.mercycorps.org/tenders/rfp-comprehensive-data-collection-and-reporting-platform

Our Compliance team was able to dedicate a team member to guide me through the procurement process, who was extremely knowledgeable and was able to support me throughout the process. We also had several supportive members of our finance, IT, and other teams that participated in the procurement, and helped me to complete the process. Everyone engaged understood the benefits, and prioritized this task. We started with laying out the requirements of the platform in the RFP, using the data on the needs, functions, and performance of the required platform, that we had collected through our case studies of the 12 countries I mentioned above. 

Data security, ease of use and available training, as well as a partner who would be willing to walk along patiently with us, help us think through efficiencies, and have our interests in mind as well, were among the strong requirements that came together with the technical requirements laid out in the RFP. 

We knew we were driving change internally within the organization (since this would be the first time we were procuring a platform that was fully funded by programs, not the core budget of the organization), and needed a partner who understood the complexities of doing so, and was able to work with us throughout the process. 

The procurement process took us nearly 5 months (Nov 2021 – end April 2022), and was successfully completed identifying CommCare by Dimagi as the partner for Data Collection and Case Management. 

Step 4: Roll Out Guidance and Don’t Forget to Offer Training

Once you’ve selected your tools, you need to establish a management process around them, document them, communicate with your teams and offer training to support teams to get the most value from them. 

As we started rolling out Mercy Corps’ MEL policy, we also rolled out the MEL tech Suite laying out the technologies to support our MEL functions: 

• CommCare for data collection and case management 
• Ona for data collection that does not require case management
• Stata or R for quantitative data analysis
• MAXQDA for qualitative analysis
• QGIS for mapping
• Microsoft Azure for data storage
• PowerBI for visualization

Through the case studies of the 12 countries, and also feedback received during the roll out (May 2021 – October 2022) we knew that training on those platforms was in high demand. In September 2021, my team started working on a MEL tech training with a consultant company, Bixal, composed of three major components: 

• Third party courses, that would range between 4-5 hours per course and conclude with a proficiency test
• Level 1 and Level 2 cohort based facilitated learning, each level concluding with a proficiency test

Our partner, Bixal, identified a series of third party courses that met the requirements laid out by us, and we procured those courses for the identified participants. These included courses from Dimagi Academy trainings, Data Camp, LinkedIn, Udemy and others. Upon completion of the third party courses, the participants then engaged in a cohort-based learning, by tackling each of the technology platforms moving from Level 1 to Level 2, where complexity increases. At the cohort-based learning we also offered additional content related to data privacy and security, as well as other essential skills on data handling.  The value of tools comes from teams knowing how to build with them, so this training can amplify the value your organization gets.

At Mercy Corps, our training is offered to MEL, IT, and T4D teams, prioritizing program MEL and T4D team members who have a wider reach and are able to replicate training to our local partners or other team members.  Developing and conducting a fully virtual MEL technologies training program like this requires resources and commitment, as well as various sets of expertise (E.g. e-learning curriculum development expertise, deep knowledge of each technical platform, and hands-on facilitation expertise using virtual platforms). In May, 2022, we covered Ona, CommCare, PowerBi and TolaData (Mercy Corps’ own platform for program indicators), and will be offering another set of trainings for MAXQDA, STATA and QGIS in a few months. 

This four step process will look different at every organization, and you’ll need to consider the unique aspects of your organization to tailor it for your needs. 

To help guide you, we’ve extracted some of Mercy Corps’ keys to success. 

Keys to successful digital tool selection

• Start with the problem: If you show up with a technology tool in mind that you want to roll out without first considering the experiences of the teams that would use said tool, you will fail. Get input from everyone affected and take the time this requires to deeply understand the problems they are facing and the solutions they are seeking after. Also engage them throughout the process, inform them of the progress you are making, and stay in touch.  
• Understand the bottlenecks: If you are coming into a new role or a new organization and looking to establish centralized tools and processes for data collection and analysis, it’s worth talking to your predecessors to understand where they got held up. Sometimes bottlenecks exist within processes that aren’t serving the organization any more. And, sometimes people can inadvertently become bottlenecks if they aren’t aligned to your vision. Taking time to connect with any historical detractors to understand their points of view will minimize surprise barriers down the road.  
• Show time and cost savings, as well as value for money of the tools you select: If you have data on ways that the tools you are selecting have already saved your team’s time, highlight that to your senior leadership. At Mercy Corps, we looked at how automating connections between two of our MEL technologies could help our program teams access quality data near real time, a process that had originally taken 5-7 weeks. With the automation we brought that time down to 2 days for the first time in December 2020, and nearly real-time thereafter. The time saved can now be spent looking at the data, understanding it and gleaning useful insights. If you don’t have this kind of data going into this process, ensure you have the ability to track savings over time. This will be an important way to show success and deepen commitment to a new, more unified set of tools. 
• Show cost savings of the tools you select: Before I led my team through this process, Mercy Corps’ country and program teams were procuring CommCare individually. By creating an enterprise agreement with Dimagi, we achieved significant savings. For example, for the price of 17 spaces that each individual country procured, some of which used only basic services, we were able to offer access to more advanced features and functions of CommCare to 20 spaces. Within less than a month of finalizing the task order, we received additional requests, and are increasing our procurement to 25, as of the end of May, 2022.  We expect this to grow as usage grows and training gets rolled out, and we’ll be tracking the savings to continue to inform senior leadership. 
• Add the cost of certain tools to program budgets as possible: At Mercy Corps, we are able to include the cost of CommCare in program budgets funded by donors. This helps keep the tool affordable for the organization. 
• Prioritize data privacy and security: At Mercy Corps, any system that passes our strict privacy impact assessment is favored. We don’t want to select a tool and then worry about the data security aspects. We want a provider with the capacity to handle all aspects of data privacy and security because we know this is a full time job staying on top of regular changes and new threats as they emerge.  
• Look for good training: Choose a technology partner that offers good and intuitive training – and possibly free. 
• Consider the role of policy: At Mercy Corps, the team rolled out a Monitoring Evaluation and Learning policy which laid out that programs are required to develop MEL Technology plans, outlining the implementation, capacity building and alignment across technologies selected for the program. It also lays out that programs should dedicate 5% of funding to M&E costs, some of which is dedicated to MEL technologies. 
• Partner for scale: As you are selecting your tools, consider the longevity, stability and trajectory of the technology partner at hand. Will they be a good long-term partner? Do you get the support you need from them? Will they listen to your needs and evolve their offering over time to improve? 

I hope this guide and set of tips is helpful as you consider the unique data and technology needs of your teams. If you have a preference for audio content, you can listen to the discussion on the High-Impact Growth podcast.

Mercy Corps’ Technology for Development (T4D) team is supporting digital MEL adoption through its generous partnership with Cisco, under a 5-year program aimed at using technology to deliver aid and development assistance faster, better, and to more people.

Reach out to Dimagi if you would like to discuss how to scale your global M&E programs.