(Image collage of county government offices taken by research supervisors Titus Kuria and Nelson Ngige during field work)
Does a fried drumstick from KFC taste the same in Boston, Dar es Salaam, and Nairobi? To control for taste, look, and feel of a product served around the world, franchise companies use “mystery shoppers” to asses brand consistency. The concept is simple – a “mystery shopper” tests service delivery and quality by acting as a real customer and rating the experience. Applying this methodology to test government transparency, MIT GOV/LAB, in partnership with Twaweza East Africa, sent mystery shoppers to 45 of 47 counties across Kenya. The mission: to request public information from local government offices and assess the response rate and overall experience.
Why the mystery?
Those in power often use secrecy to project a certain image, to hide what they are doing, or to protect their position by deflecting criticism and not providing opportunity for citizen engagement. This can be especially true in developing democracies, where fledgling government institutions are trying to prove themselves, corruption is prevalent, and there is often a weak culture of openness or transparency. Combating this trend, an international movement has for the last two decades been pushing for more open government. This has resulted in over 100 countries worldwide adopting freedom of information laws – including Kenya.
Using the mystery shopper design, a method with both high internal and external validity, we set out to assess whether or not Kenya’s Access to Information Law was known and followed at the local level. The law was recently passed in August of 2016, though regulations have been slow to promulgate. Since the Kenyan Government is currently undergoing a process of devolving power to the counties, we didn’t expect the law to be fully implemented. However, given the recent passing of the Access to Information Law, the mystery shopper experiment allowed us to capture a baseline of how local governments respond to citizen requests. This helps to establish the “status quo,” so Twaweza can make strategic recommendations on how the government should move forward implementing the new law.
Another research goal was to find out what happens when ordinary citizens make requests for information that impact their daily and long-term decisions, information like: which schools have high pass rates, which hospitals receive regular resources, what road and water projects are scheduled to take place, and what development plan were funded and successfully (or not) completed.
Finally, the Access to Information Law states that citizens do not need to provide a reason to request information; we wanted to test if this was true. To do so, we included an experimental variation. We randomly assigned mystery shoppers in half of the counties to use a rights-based or legal approach while the other half presented a personal rationale for needing the information. This allowed us to test which type of approach would prove more successful in obtaining information.
Setting up mystery shoppers in Kenya
To adapt the mystery shopper methodology for the Kenya context, we worked with a team of six stellar researchers to pilot the research protocol in county government offices across Nairobi. We wanted to know the answers to questions like: how should mystery shoppers frame their requests to feel natural? how should they break the ice and identify themselves without biasing the process? how should we design our instruments to fully capture the experience?
The process of piloting, provided crucial insight on a number of issues. First, finding the correct office to request information is not always easy due to poor signage. Second, finding the person who has the responsibility and capacity to grant information requests was not simple. Oftentimes, secretaries would refer researchers to the most senior level official, who was busy or out of the office. Other times, the mystery shoppers got the runaround and were transferred from office to office to office, with no clear process or direction in mind to get information. Some researchers joked that at least they got exercise walking up and down stairs to locate the right office.
Mystery shopping in the field
With these insights in mind, we refined the protocol and conducted a three day training for the 45 mystery shoppers, one from each of the 45 counties across Kenya (excluding Mombasa and Nairobi) on how to effectively implement the design. Each mystery shopper made requests at seven county-level government offices, many requiring multiple follow-up visits. All together, they made over 500 offices visits.
We are just starting to analyze all the quantitative and qualitative data; below are some preliminary high-level findings:
Elections and heightened suspicions: 2017 is a national election year in Kenya and suspicion is high. Several local government officials mentioned being worried about how information could be used for nefarious purposes or bad press with negative repercussions for re-election. As a result, officials wanted to know why the request was being made, how the information would be used, and many offices classified regular information as sensitive. Some mystery shoppers reported back having negative experiences from government officials who questioned their motivations and requested too much personal information.
Unclear process for requesting information: The procedure for making information requests demanded by officials varied from office to office and from county to county. The lack of standardized processes allowed officials to construct high barriers to citizens accessing information, for example, requiring multiple signed and sealed letters from various authorities. Other officials denied requests outright and some only granted information informally through SMS (texting) or even social media sharing. The lack of a standard process clearly impacts the experience and ease of accessing information.
The next step is crunching all the numbers to see exactly how many requests were successful and to assess the quality of mystery shoppers experience. We will also look at how research profiles affected information requests. For example, does age or gender of the mystery shopper make a difference in whether information is provided or denied? Tribe, or ethnic factors, are another important factor in Kenya that impacts everything from voting allegiances to day-to-day interactions. We’ll see how all these factors matter as we continue to analyze the data and try to take the mystery out of accessing public information in Kenya.
—
UPDATE: High-level results available online as a Research Brief. Also, available is a Learning Case discussing the research collaboration with Twaweza.
The Mystery Shopper experiment was implemented by MIT PhD student Gabriel Nahmias and MIT GOV/LAB manager Alisa Zomer in partnership with the Twaweza East Africa and the Twaweza Kenya office. A big “asante sana” to local research supervisors, Mwongela Kamencu, May Koko, Nelson Ngige, Edwin Nyamanya, Brenda Ochieng, and Titus Kuria, who were critical to the research piloting and field implementation.