Introduction
Frontline health workers (FLWs) play a critical role in increasing access to health care and health information in low- and middle-income countries (LMICs). However, many FLW programs face resource constraints around training, supervision, and coaching. With recent advances in AI, Large Language Models (LLMs) present an opportunity to develop highly interactive, personalized digital tools that can support FLWs in their work.
To date, digital solutions for coaching FLWs have largely been static, offering pre-set information without dynamic adjustments based on user interactions. The shift from static to adaptive coaching through LLM-powered tools has the potential to enhance FLW training, supervision, and performance support in a scalable and cost-effective way. In this blog post, we share early learnings from an AI-powered coach made for frontline workers in Zimbabwe.
Designing CATSCoach: AI-Powered Mentorship for Peer Supporters
Dimagi and Zvandiri, with funding from Endless Network, are currently designing and testing CATSCoach, an LLM-powered chatbot aimed at providing supportive supervision to Community Adolescent Treatment Supporters (CATS). Zvandiri’s CATS workforce comprises young people aged 18-24 who provide adherence counseling and psychosocial support to children and adolescents living with HIV (CAYALHIV).
CATS receive an initial five-day training followed by ongoing mentorship and supervision. To better enable this workforce, CATSCoach is being developed to help CATS practice handling difficult conversations, reinforce training content, and receive immediate feedback.
CATSCoach is built using Dimagi’s Open Chat Studio, a platform that enables organizations to create and deploy AI-powered chatbots quickly and efficiently. Open Chat Studio (OCS) allows organizations to pick between LLMs, develop and customize LLM-based chatbots, integrate safety guardrails and deploy on commonly used messaging platforms like WhatsApp, Telegram, and Facebook Messenger. OCS’ Prompt Builder feature allows for continuous iterations to enhance the chatbot’s ability to support learning and skill-building.
How CATSCoach Engages Users
CATSCoach is currently being designed with key features including:
Adaptive Learning Pathways: By leveraging the EQUIP framework, CATSCoach dynamically adjusts the complexity of exercises based on the user’s skill level, ensuring appropriate challenges and support for beginners, intermediates, and advanced users.
Role-Play Scenarios: The chatbot presents realistic client scenarios, allowing CATS to practice responding in a safe environment. These scenarios cover topics such as stigma, disclosure, adherence, and mental health support.
Quizzes and Decision-Based Learning: Interactive quizzes reinforce knowledge, while “choose-your-own-adventure” exercises help users develop decision-making skills in real-world scenarios.
Multilingual Support: The chatbot is being tested in English, Shona, and Ndebele to ensure accessibility for diverse users.
What We’ve Learned So Far
We tested an early version of CATSCoach on WhatsApp with a group of 40+ CATS in Zimbabwe to understand its usability and effectiveness. The chatbot was well received, with users appreciating its human-like interactions and practical feedback.
Positive Feedback:
- Engaging and realistic interactions: Users found the chatbot’s role-play scenarios useful for developing communication skills. “This app has been a game-changer for me! The counseling guides, quizzes, and interactive scenarios have helped me develop valuable skills to support myself and others.” – Participant
- Encouraging and confidence-building: CATS appreciated the chatbot’s feedback. “The chatbot is good as it asked me some questions and I responded; I was immediately corrected where I had missed correct answers.” – Participant
- Helpful in knowledge-building: The chatbot provided structured guidance. “For me it’s perfect, it helped me know how to approach some clients. This was on activity 2: Role play of Nancy, who was having trouble with her caregiver. Chat bot acted as Nancy and I acted as Nancy’s caregiver”.– Participant
Challenges and Areas for Improvement:
- Identifying a user’s level of proficiency and adapting accordingly: A key challenge was ensuring the chatbot could accurately assess user proficiency and adjust its responses accordingly. Some users found that the chatbot did not always modify difficulty based on their skill level. For instance, despite the EQUIP framework defining four proficiency levels for each skill, all users were categorized as Level 3, regardless of their varied responses. Moving forward, refining the chatbot’s ability to evaluate user competency and provide more personalized training will be a top priority.
- Making feedback more concise: Many CATS reported that the chatbot’s responses were too long, making it difficult to absorb key information. “The paragraphs are too long, and you end up getting lost.”- Participant
- Ensuring responses are aligned to user input: Some participants noted that the chatbot occasionally changed topics mid-conversation. “It asked about my client’s emotional barrier but then started asking unrelated questions about home and medication.” – Participant
- Refining translations in Shona and Ndebele: Users reported that some phrases in Ndebele were unclear. Some terms borrowed from Zulu caused confusion, such as ‘amakhasimende’ (clients), which is not commonly used in Ndebele. “I chose Ndebele, but it is not the usual Ndebele I know as I didn’t know some words.” – Participant
- Improving scenario relevance: Some scenarios, such as those involving musicians traveling, did not reflect the realities of the CATS’ clients. A participant suggested, “Is it possible to choose the theme we want to tackle?”
What’s Up Next
Dimagi is refining the CATSCoach tool to more accurately assess user proficiency and tailor learning experiences accordingly. We are focused on enhancing how the chatbot:
- Maps Proficiency More Accurately: Improving the chatbot’s ability to assess user skill levels using frameworks like EQUIP, ensuring it adapts appropriately rather than defaulting to a single level.
- Delivers Adaptive Learning Experiences: Creating dynamic exercises that adjust based on individual progress, making training more relevant and engaging.
- Tracks and Personalizes Progress: Developing mechanisms to monitor user improvement over time and refine recommendations based on their evolving needs.
In the coming months, Zvandiri and Dimagi will test the updated CATSCoach tool with 100 CATS workers in Zimbabwe, with support from the Endless Network.
For organizations interested in exploring AI-based coaching, learn more about Dimagi’s Open Chat Studio platform or reach out to ocs-info@dimagi.com.
