A lot has already been made about the newly released estimates of maternal mortality published this week in the Lancet by a group of researchers at the Institute for Health Metrics and Evaluation.  The story was the lead article in the New York Times, it has been covered on blogs (here and here), and has raised some really important issues about maternal mortality (see here, but make sure you read all the comments too!).  But what should we really make of these new estimates?

The controversy stems from the fact that the newly released estimates of the number of maternal deaths (herein the IHME estimates) are substantially lower than estimates published in the same journal just 2.5 years ago by other set of excellent researchers (herein the Hill estimates).  Specifically, the Hill group estimated the number of maternal deaths globally at 535,900 (in 2005) while the IHME group estimated 342,900 maternal deaths (in 2008).  The methodology used in both of these papers is relatively complex and as such I think it is difficult, even for those of us who know something about this stuff, to really make sense of why we have seen such a change and what these changes really mean.  After a week of trying to really understand the various methodologies, here is my best attempt to summarize what I think is driving the different estimates and what I think it means in terms of progress towards maternal mortality.

The first thing that needs to be made crystal clear – both the IHME and the Hill estimates are that – ESTIMATES (fancy term for made up numbers).  Both sets of researchers had at their disposal a set of information on maternal mortality and they made choices and assumptions about how to use this information.  In fact, for the most part, they each had the same information at their disposal.  Maternal mortality data comes from a series of data collection systems, some that are better than others, but no system is able to provide 100% accurate estimates of maternal mortality – even in the United States – so what data you use and what assumptions you make really matters.  Even the “gold standard” data – vital registration data – tends to underreport maternal mortality, sometimes by as much as 40% (e.g. maternal deaths tends to be classified as other things, say sepsis).  The IHME had some newer data at their disposal, but I don’t get the sense this is what has really changed.

The Hill group restricted itself to only nationally representative data, whereas the IHME grouped used all data available including subnational data, which tends to be noisier, so it is not clear to what extent this makes our estimates more precise.  But at the same time, it makes sense to try to use as much data is available.  But if subnational data is more likely to underreport, it might be one reason why the newer estimates are lower.  The IHME report does mention that they end up throwing out a lot of subnational data because of “implausibly low rates” – so this might be a real threat.

The next major difference between the estimates is how the authors account for underreporting in the data.  The Hill group applied a constant adjustment factor to account for underreporting which varied based on the main source of data – including a blanket adjustment of 1.5 in countries like India and China, which use sample registration systems.  In essence, because they assumed underreporting was a big problem they inflated their counts of death by 50%.  I have no strong sense of whether this was appropriate or not, but it did seem to be on the conservative side.  The IHME group appears to have used a corrected data sets for countries with vital registration systems and used some correction factors for data collected in sibling surveys, but it is not immediately obvious what method was used in each country – if at all.  Given we are talking relatively large adjustments, in particular in countries with large populations and high mortality (India in particular) it would have been good to get a better sense of how the adjustment levels compared.  I suspect a lot of the difference between the estimates has to do with the underreporting estimates (although I am happy to be proven wrong).

The final big difference between the estimates is the way in which they modeled out of sample predictions.  Because there are a lot of countries for which we have no data, and to get a sense of the validity of estimates, each group builds a model to predict maternal mortality based on variables such as GDP per capita, fertility, skilled birth attendants, and HIV.  The Hill group modeled the outcome as a proportion whereas the IHME group modeled them as level deaths.  Both claims that they approach is superior to the other alternative.  The Hill group uses skilled birth attendants whereas the IHME group does not (it does is collinear with GDP per capita in their model).  Data on HIV prevalence, which seems to make a big difference in the estimates, is also just an estimate, and one that has been updated substantially in recent years.  About a quarter of the births in the world (and maybe as many as half of the maternal deaths since these are the poorest countries) are estimated this way.  I won’t try to claim one approach is superior to the other, but I think it can generate big differences in overall estimates based on the assumptions made.

So what did we really learn from this exercise?  Mostly that using a different set of assumptions it is totally possible to come up with a very different estimate of maternal mortality.  If I wanted to, I could produce my own estimates – say I ignored underreporting altogether and just used the lowest reported data per country – and I could probably come up with an estimate even lower than the IHME estimate (although hopefully the Lancet would reject my submission on the basis that I have no clue what I am doing).  These are both well respected set of researchers and I suspect both have come up with realistic estimates but we will never know which is really correct.

What I think is important here is to focus on what has not changed with these new estimates – that is what both groups agree to be true.  First, maternal mortality has seen a slow but steady decline over the past 30 years but both estimates suggest that the estimated rate of decline (between 1-2% a year) is well below the 5.5% that would have been needed to achieve MDG5 since 1990.  Even the more optimistic IHME data would suggest less overall progress during the 1990s, the time period after which the MDG clock began ticking.   Second, some parts of the world have seen more progress than others (Asia, Latin America) and Africa has seen little or no progress.  India (677 to 254) and China (165 to 40) have seen major proportional declines in the maternal mortality ratios and have have seen major increases in GDP per capita and declines in fertility, which could have greatly contributed to these declines (and might have even with no effort from the global health community).  Many countries in Africa are actually worse off today than they were 20-30 years ago, mostly due to an increase in HIV associated maternal mortality and very little progress against the other causes of maternal mortality (and despite increases in GDP per capita during the past decade).  Altogether, I still see little here to celebrate.

Don’t get me wrong – I believe that there is value in exercises to estimate things that are difficult to measure precisely.  I believe that we can learn from such exercises and feel that no one group should have the monopoly on estimates and that it is good that there are groups like the IHME who are out there questioning the estimates of others (personally, I am really looking forward to their estimates on malaria control effects).  That said, we need to be careful to not speculate too much from these estimates, whether conscious or not we are all biased and these can greatly influence the assumptions we make.

Many have already begun to claim that these new declines were due to increased efforts to scale up safe-motherhood initiatives, etc  –  but neither report has properly addressed the question of why maternal mortality has changed  – so we really do not know.  I also would have liked to see a more detailed explanation about how the different assumptions influenced the overall estimates.  Estimates are only as good as the data used and the assumptions made so no one can really say whose made up numbers are best.

Share on Facebook

6 Responses to “Are my made up numbers better than your made up numbers? Uncovering the new maternal mortality estimates”

  1. April says:

    thanks Karen for this great synthesis and reflection on what is the real take-away message from this discussion. I completely agree with your points.

    I follow, just a bit, the raging debates over climate change and climate science – and in that field too many of the critical measures seem to be the object of debate. And one of the biggest criticisms of the scientists who have worked the most closely with the International Panel on Climate Change is that despite their research requiring very sophisticated statistical techniques to generate estimates they very rarely involve expert statisticians in their research or have their worked reviewed by them.
    And, this has led to some fairly large embarrassing errors being discovered long after publication. And after policies have been recommended based on these findings. Naturally these discoveries have given credence to the claims of manipulation by the people who are skeptical about man-made global warming. And, with the wider public, they have weakened support for taking action on controlling carbon emissions.
    With this in mind, I agree with you that Chris Murray and co at IHME with their statistical expertise, and lower degree of interest in any particular finding, are doing the global health field a service by re-looking a the various estimates they are looking at.

  2. Nandini says:

    Karen–great post. I'm in the process of posting a similar commentary on the CGD blog about these ESTIMATES. I'll cross post as soon as it is up. Unlike your individual blog, we have a few steps before ours get up!! What I didn't do was dig through the methods and data from the paper, so thanks for laying it all out. (I'll link in my blog post) Like you say, the estimates are only as good (or bad) as the data they are derived from and the assumptions made when different statisticians model these numbers. But like April, I agree that the IHME group doesn't have a vested interest in estimating higher or lower numbers for MMR, so it's good to have another number out there to challenge, but not dismiss the older estimates.

  3. Pär says:

    Karen, thanks for a very interesting comparison. I wonder if you have had any chance to look at the MMR in Zambia, it is reported to have declinced substantially and the MoH is trying to figure out why.

  4. Margaret Hogan says:

    Thank you, Karen, for highlighting IHME’s research published in The Lancet, Maternal mortality for 181 countries, 1980-2008: a systematic analysis of progress towards Millennium Development Goal 5 and for raising important questions about the study. We appreciate the opportunity to help clarify the research.

    As the lead author of the maternal mortality study, I wanted to address the statement that the IHME study was basically using the same information as previous studies. In fact, we used a database of 2,651 observations of maternal mortality from vital registration data, censuses, surveys, and verbal autopsy studies. The Hill study used 858 observations.

    As you note, the Hill study restricted itself to nationally representative data, whereas we included sub-national, population-based studies where they were available. These sub-national studies accounted for 7% of the total data points in the study, so the vast majority of additional data points in our study were nationally representative. In some cases, the sub-national observations we identified were implausibly low, but there were also cases where they were consistent with national numbers, or indeed, were implausibly high. In our model, we down-weight these sub-national observations relative to national observations to prevent them from unduly influencing the estimates.

    Underreporting and misclassification of maternal deaths, even in fully functioning vital registration systems, is a serious issue. Naghavi et al (Population Health Metrics, forthcoming, 2010) have undertaken an extensive analysis to produce a corrected vital registration dataset, which we use in our study. They have corrected for shifts in coding practices for causes of death as well as misclassification of maternal deaths to other causes such as septicaemia and pulmonary embolism. This analysis increased the number of maternal deaths, on average, by 40%, but it is not a blanket correction like Hill et al. used. It is instead specific to the data in a particular country-year.

    For out-of-sample predictions, because of our more extensive data base, there are only 21 countries with no empirical data, as compared to 61 in the Hill study for 2005. We’ve also used a two-stage modeling approach that captures real, systematic variation across space and time, and improves the performance of the model relative to a single-stage model.

    One other major difference between the two sets of estimates that you didn’t note in your blog is the use of improved all-cause adult female mortality envelopes in our study. Rajaratnam et al (The Lancet, forthcoming, 2010) have undertaken a systematic assessment of adult mortality, incorporating all available sources of data to re-estimate adult mortality. In many settings, these estimates are substantially different from those produced by the WHO and used in the Hill study, and may explain some of the differences between the two sets of estimates.

    Again, we appreciate – and encourage – open dialogue on our methods and our findings in order to stimulate the opening of new avenues for consultation and collaboration, which will in turn serve to improve and strengthen the evidence base in the long run.
    Please feel free to contact us at IHME@healthmetricsandevaluation.org should you have any further questions.

  5. jorismichielsen says:

    As I replied already under Karen's previous post "Is maternal mortality declining?" I still wonder why Hogan et al. so heavily hold to national vital registration systems as their source on MMR. In case of India the national vital registration systems are highly criticized by national statistical experts and WHO for underreporting. A recent report of Human Right Watch illustrates this: http://www.hrw.org/en/reports/2009/10/08/no-tally-anguish-0.

    Some causes for underreporting in India coming from own experience and literature review:
    – Maternal deaths during home delivery are only reported if the women die in the matrimonial of husband's home. Not when it occurs in an illegal setting (f.e. city slums) or when the women deliver in their family or relatives home (what frequently occurs). This is so because the local health workers or ANM are only responsible for the maternal health of the daughters in law in their village.
    – private practitioners and some government providers do not report maternal deaths because they are afraid of punitive actions and they have to hold up a good reputation.
    – lot of women are referred from one health facility to another and end up dying on route. So death is not registered because nobody wants to bear the responsibility.
    – no political willingness to monitor and evaluate. Health district workers sometimes do not even know that such register books exist, or they suppress such information (deaths that are reported by local health workers) afraid of being hold responsible.
    – lack of definitional clarity on maternal deaths (timing ad causes).
    – lack of awareness of importance of reporting under the health workers, and this is reinforced by the neglect and lack of follow up (from part of the health district officers) of maternal death cases during the monthly meetings of nurses with the health district officers.

    All this is very interesting against the good experiences in reporting for immunization and sterilization. This stresses the lack of political will even more. The statistics report and overreport the successes but for MMR vital registration systems are far from dependable, and in mine opinion this underreporting is too difficult to be corrected for!

Leave a Reply to Nandini

Analytics Plugin created by Web Hosting