A lot has already been made about the newly released estimates of maternal mortality published this week in the Lancet by a group of researchers at the Institute for Health Metrics and Evaluation. The story was the lead article in the New York Times, it has been covered on blogs (here and here), and has raised some really important issues about maternal mortality (see here, but make sure you read all the comments too!). But what should we really make of these new estimates?
The controversy stems from the fact that the newly released estimates of the number of maternal deaths (herein the IHME estimates) are substantially lower than estimates published in the same journal just 2.5 years ago by other set of excellent researchers (herein the Hill estimates). Specifically, the Hill group estimated the number of maternal deaths globally at 535,900 (in 2005) while the IHME group estimated 342,900 maternal deaths (in 2008). The methodology used in both of these papers is relatively complex and as such I think it is difficult, even for those of us who know something about this stuff, to really make sense of why we have seen such a change and what these changes really mean. After a week of trying to really understand the various methodologies, here is my best attempt to summarize what I think is driving the different estimates and what I think it means in terms of progress towards maternal mortality.
The first thing that needs to be made crystal clear – both the IHME and the Hill estimates are that – ESTIMATES (fancy term for made up numbers). Both sets of researchers had at their disposal a set of information on maternal mortality and they made choices and assumptions about how to use this information. In fact, for the most part, they each had the same information at their disposal. Maternal mortality data comes from a series of data collection systems, some that are better than others, but no system is able to provide 100% accurate estimates of maternal mortality – even in the United States – so what data you use and what assumptions you make really matters. Even the “gold standard” data – vital registration data – tends to underreport maternal mortality, sometimes by as much as 40% (e.g. maternal deaths tends to be classified as other things, say sepsis). The IHME had some newer data at their disposal, but I don’t get the sense this is what has really changed.
The Hill group restricted itself to only nationally representative data, whereas the IHME grouped used all data available including subnational data, which tends to be noisier, so it is not clear to what extent this makes our estimates more precise. But at the same time, it makes sense to try to use as much data is available. But if subnational data is more likely to underreport, it might be one reason why the newer estimates are lower. The IHME report does mention that they end up throwing out a lot of subnational data because of “implausibly low rates” – so this might be a real threat.
The next major difference between the estimates is how the authors account for underreporting in the data. The Hill group applied a constant adjustment factor to account for underreporting which varied based on the main source of data – including a blanket adjustment of 1.5 in countries like India and China, which use sample registration systems. In essence, because they assumed underreporting was a big problem they inflated their counts of death by 50%. I have no strong sense of whether this was appropriate or not, but it did seem to be on the conservative side. The IHME group appears to have used a corrected data sets for countries with vital registration systems and used some correction factors for data collected in sibling surveys, but it is not immediately obvious what method was used in each country – if at all. Given we are talking relatively large adjustments, in particular in countries with large populations and high mortality (India in particular) it would have been good to get a better sense of how the adjustment levels compared. I suspect a lot of the difference between the estimates has to do with the underreporting estimates (although I am happy to be proven wrong).
The final big difference between the estimates is the way in which they modeled out of sample predictions. Because there are a lot of countries for which we have no data, and to get a sense of the validity of estimates, each group builds a model to predict maternal mortality based on variables such as GDP per capita, fertility, skilled birth attendants, and HIV. The Hill group modeled the outcome as a proportion whereas the IHME group modeled them as level deaths. Both claims that they approach is superior to the other alternative. The Hill group uses skilled birth attendants whereas the IHME group does not (it does is collinear with GDP per capita in their model). Data on HIV prevalence, which seems to make a big difference in the estimates, is also just an estimate, and one that has been updated substantially in recent years. About a quarter of the births in the world (and maybe as many as half of the maternal deaths since these are the poorest countries) are estimated this way. I won’t try to claim one approach is superior to the other, but I think it can generate big differences in overall estimates based on the assumptions made.
So what did we really learn from this exercise? Mostly that using a different set of assumptions it is totally possible to come up with a very different estimate of maternal mortality. If I wanted to, I could produce my own estimates – say I ignored underreporting altogether and just used the lowest reported data per country – and I could probably come up with an estimate even lower than the IHME estimate (although hopefully the Lancet would reject my submission on the basis that I have no clue what I am doing). These are both well respected set of researchers and I suspect both have come up with realistic estimates but we will never know which is really correct.
What I think is important here is to focus on what has not changed with these new estimates – that is what both groups agree to be true. First, maternal mortality has seen a slow but steady decline over the past 30 years but both estimates suggest that the estimated rate of decline (between 1-2% a year) is well below the 5.5% that would have been needed to achieve MDG5 since 1990. Even the more optimistic IHME data would suggest less overall progress during the 1990s, the time period after which the MDG clock began ticking. Second, some parts of the world have seen more progress than others (Asia, Latin America) and Africa has seen little or no progress. India (677 to 254) and China (165 to 40) have seen major proportional declines in the maternal mortality ratios and have have seen major increases in GDP per capita and declines in fertility, which could have greatly contributed to these declines (and might have even with no effort from the global health community). Many countries in Africa are actually worse off today than they were 20-30 years ago, mostly due to an increase in HIV associated maternal mortality and very little progress against the other causes of maternal mortality (and despite increases in GDP per capita during the past decade). Altogether, I still see little here to celebrate.
Don’t get me wrong – I believe that there is value in exercises to estimate things that are difficult to measure precisely. I believe that we can learn from such exercises and feel that no one group should have the monopoly on estimates and that it is good that there are groups like the IHME who are out there questioning the estimates of others (personally, I am really looking forward to their estimates on malaria control effects). That said, we need to be careful to not speculate too much from these estimates, whether conscious or not we are all biased and these can greatly influence the assumptions we make.
Many have already begun to claim that these new declines were due to increased efforts to scale up safe-motherhood initiatives, etc - but neither report has properly addressed the question of why maternal mortality has changed - so we really do not know. I also would have liked to see a more detailed explanation about how the different assumptions influenced the overall estimates. Estimates are only as good as the data used and the assumptions made so no one can really say whose made up numbers are best.Share on Facebook