I feel like I’ve become a bit of a broken record lately as I’ve ended up in situations where I have to defend the need for more evaluation work in global health. And I do mean defend – not just advocate for more evaluation work – but rather be on the defensive against groups or people that have argued that there is currently too much evaluation work or that evaluation work cannot – or even should not be done. I disagree.
I’ve had to make this argument in different settings, but one area where I have been working lately is in the area of mHealth where a common argument I have heard about why we don’t need to do more evaluation work is that “we already know this works” or because most mHealth interventions are “low cost and easy to do, so it can’t hurt to do more of it”. However, both of these statements, in my view, cannot be shown to be true without more evaluation work.
Right now there is an explosion of interest in using mobile phones to deliver health information and health services around the world in both low and high-income settings. mHealth platforms are being used to deliver everything from basic health information to patient level reminders that can be considered health services. On the supply side, health care providers are being taught how to use mobile phones to diagnose patients, to triage and expand health services, and to even treat patients. This is great since there needs to be continuous innovation in health care delivery.
But mHealth is not just changing the way in which existing health interventions are delivered but is really creating new health services, which can lead to very different outcomes. It is for this reason that these new strategies must be evaluated – even if the services that are being delivered have been shown to be effective in other ways.
Most of the evidence on the effectiveness of mHealth interventions has been small scale, case studies, and for the most part lacking rigorous strategies to identify effects or health impacts. Many studies focus exclusively on whether or not the programs were successfully implemented. And while the few rigorous evaluations that have been done (or at least I should say have been done and have been published) have generally found small but significant intended effects from the approaches I think we still have a lot to learn about the benefits and potential consequences of mHealth approaches.
Which is why the results of a new working paper released by Julian Jamison, Dean Karlan, and Pia Raffler should give many in the mHealth field a bit of pause or at least make you go hmmm. The authors of this paper were involved in a randomized evaluation of a mHealth project in Uganda that was aimed to provide users more information about the riskiness of various sexual activities in order to reduce overall risky sexual behavior. Treatment villages received encouragement to advertise the program and to encourage its use while in the control villages they were also able to use the program but were not exposed to this treatment. Data on many indicators, including sexual behavior, was collected in both treatment and control villages at baseline and endline.
You can read more about the program itself and the overall findings of this evaluation in this new policy brief from IPA but I want to highlight two important findings:
1. The authors found that the program was implemented as intended: in treatment villages people were making use of the program and were accessing the health information more or less as intended. Great! This is typically the type of results that are presented in many types of evaluation studies I have seen.
2. However, the program did not change sexual behavior as intended and worse may have even led to more risky promiscuous behavior! Ooops!
What should we make of this? As a starter, I think it does mean that many of the arguments I have heard recently are not necessarily true – we don’t know that this works and there might be unintended effects of these efforts that still need to be better understood and explored. I am a believer that mHealth can revolutionize health care delivery in low income settings in a good way but I don’t think we are there yet in terms of really knowing what works and that there is an ongoing need for more evaluation – and more rigorous and more insightful evaluation in this field. We are experimenting with people’s lives and people’s health every day in global health, at least lets try to learn more from these experiences.Share on Facebook