This learning case is featured in our updated “How to Have Difficult Conversations” a practical guide for academic-practitioner research collaborations. Many thanks to our partners Luke Jordan and Katlego Mohlabane from Grassroot for providing input on the case study.

A mid-project revelation makes one side wish they’d asked more at the beginning.

MIT GOV/LAB collaborated with Grassroot, a civic technology organization in South Africa, to help them evaluate a new pilot project to train community organizers on the messaging platform WhatsApp. The aim of the collaboration was to demonstrate proof of concept for online training via WhatsApp and to develop a new distance learning tool for community organizers in low-resource settings.

Grassroot’s reflective approach and appetite for learning was what initially attracted us to work together. We were excited about the partnership and the chance to contribute knowledge on whether WhatsApp, often criticized for spreading misinformation, could be used for social good.

We entered the partnership thinking the pilot would scale to a larger initiative. Yet after initial engagement, we learned that Grassroot leadership was phasing out and its organizing activities would be ramping down. Prior to our involvement, Grassroot had completed a strategic review and the results led to some deep soul searching about their impact. As a result, they wanted to use the pilot to evaluate not just the WhatsApp course itself, but to test the organization’s larger theory of change—that civic technology can build lasting, impactful social movements.

A clearer understanding and discussion of Grassroot’s plans and goals earlier on would have helped us better strategize on the research design and outputs. From their perspective, Grassroot was trying to take an honest look at their impact, while testing out a brand new approach, and trying to secure funding. As it was, we ended up spending a lot of time and effort on a study that had value in its own right, but only started to answer their underlying theory-of- change question.

Studying a moving target

One challenge was agreeing on a research design while the course itself was still in development. This required significant back-and-forth with Grassroot to understand what they wanted to test and to assess what level of rigor was possible. The ability to randomize some element of the intervention (to demonstrate causal impact) was a big question that would affect the scope and the methods.

As the pilot took shape, it was hard at times to find a good balance between rapid iteration and thoughtful evaluation. Technology companies are used to this quick flow—try, test, throw away, keep what’s left, rinse and repeat—but academic research is deliberately designed with time to consider, reflect, and refine. We came into the project with a flexible mindset, wanting to contribute useful research to inform the project, but sometimes there was not enough time to absorb what was happening before the two teams made decisions.

Over time, as we realized the pilot wouldn’t evolve into a larger project, we questioned whether the resource investment fit the end goal, which was to create learnings and knowledge for the final chapter of the organization. The changing nature of the project presented a challenge to the evaluation, because we were trying to measure the project outcomes as well as Grassroot’s bigger theory of change. If we had a clearer sense of the end game from the beginning, we may have been able to sync across the project-level and mission-level objectives.

From Grassroot’s perspective, they didn’t fully understand how important the scale-up was for MIT’s continued engagement. If successful, Grassroot’s intention was for the WhatsApp project to become their new core program, supported by rigorous research and a new round of funding. The funding, however, didn’t materialize and the organization was exploring alternatives for it’s continued future even as the pilot progressed with MIT.

Despite these communication challenges, both sides consider the collaboration to be an overall success: we pioneered a new way of training through a messaging app, which was a first of its kind course, and demonstrated some promising results. Furthermore, we published a how-to guide for teaching on WhatsApp based on the lessons we learned from the pilot, so others could build on our approach.

What we learned

• Understanding the “big picture” as well as the small details. Though we had a detailed work plan and spent significant time in-country, it took time to understand how this pilot fit with Grassroot’s long-term strategy and the larger state of play for the organization. We could have done more to clarify our own assumptions and expectations around the project’s scaling potential.. An open conversation about the big picture may have helped to clarify these points, before we both dived deep into the weeds of the pilot.
• Managing risks for long-term investments. Should we have reacted differently when we learned Grassroot leadership was going to transition out, and later found out that their main platform was shutting down? Honestly, by that point we had spent weeks in-country, were invested in the project and committed to seeing it through. The project was linked to GOV/LAB’s own strategic planning and we believed the project had potential for broader lessons and knowledge for the field. Building in some decision points along the way, and being upfront with Grassroot about the tradeoffs and costs, could have helped us respond to the new information and maybe set limits in regards to resources and time spent.
• Getting skin in the game. In this case, we supported all the research costs from flexible core funding, but never really communicated those costs to Grassroot. Perhaps a more open discussion of costs for the various study elements would have been useful to discuss the research design plan with value for money in mind.
• Did we bring a cannon to a water balloon fight? Sometimes it felt our research methods were too rigorous for what Grassroot needed. Despite time constraints, next time we would find more time to talk through whether the big questions were actually answerable by the methods proposed.

In short: This learning case links to clearly laying out incentives and expectations at the beginning of the collaboration. MIT GOV/LAB initially entered the partnership thinking the pilot would scale to a larger initiative, but the project was ultimately meant to create and share knowledge as part of Grassroot’s closing chapter. For their part, Grassroot was planning for the pilot to be a test case for a new organizational direction, and despite the project’s promising outcomes, shifting donor priorities changed their plans. The project was a success, because both teams were committed and had a good collaborative work style, but we both acknowledge that more communication earlier on our longer-term expectations would have made for better alignment of priorities and outcomes.

Gauteng, South Africa