A lot has already been made about the newly released estimates of maternal mortality published this week in the Lancet by a group of researchers at the Institute for Health Metrics and Evaluation. The story was the lead article in the New York Times, it has been covered on blogs (here and here), and has raised some really important issues about maternal mortality (see here, but make sure you read all the comments too!). But what should we really make of these new estimates?
The controversy stems from the fact that the newly released estimates of the number of maternal deaths (herein the IHME estimates) are substantially lower than estimates published in the same journal just 2.5 years ago by other set of excellent researchers (herein the Hill estimates). Specifically, the Hill group estimated the number of maternal deaths globally at 535,900 (in 2005) while the IHME group estimated 342,900 maternal deaths (in 2008). The methodology used in both of these papers is relatively complex and as such I think it is difficult, even for those of us who know something about this stuff, to really make sense of why we have seen such a change and what these changes really mean. After a week of trying to really understand the various methodologies, here is my best attempt to summarize what I think is driving the different estimates and what I think it means in terms of progress towards maternal mortality.
The first thing that needs to be made crystal clear – both the IHME and the Hill estimates are that – ESTIMATES (fancy term for made up numbers). Both sets of researchers had at their disposal a set of information on maternal mortality and they made choices and assumptions about how to use this information. In fact, for the most part, they each had the same information at their disposal. Maternal mortality data comes from a series of data collection systems, some that are better than others, but no system is able to provide 100% accurate estimates of maternal mortality – even in the United States – so what data you use and what assumptions you make really matters. Even the “gold standard” data – vital registration data – tends to underreport maternal mortality, sometimes by as much as 40% (e.g. maternal deaths tends to be classified as other things, say sepsis). The IHME had some newer data at their disposal, but I don’t get the sense this is what has really changed.
The Hill group restricted itself to only nationally representative data, whereas the IHME grouped used all data available including subnational data, which tends to be noisier, so it is not clear to what extent this makes our estimates more precise. But at the same time, it makes sense to try to use as much data is available. But if subnational data is more likely to underreport, it might be one reason why the newer estimates are lower. The IHME report does mention that they end up throwing out a lot of subnational data because of “implausibly low rates” – so this might be a real threat.
The next major difference between the estimates is how the authors account for underreporting in the data. The Hill group applied a constant adjustment factor to account for underreporting which varied based on the main source of data – including a blanket adjustment of 1.5 in countries like India and China, which use sample registration systems. In essence, because they assumed underreporting was a big problem they inflated their counts of death by 50%. I have no strong sense of whether this was appropriate or not, but it did seem to be on the conservative side. The IHME group appears to have used a corrected data sets for countries with vital registration systems and used some correction factors for data collected in sibling surveys, but it is not immediately obvious what method was used in each country – if at all. Given we are talking relatively large adjustments, in particular in countries with large populations and high mortality (India in particular) it would have been good to get a better sense of how the adjustment levels compared. I suspect a lot of the difference between the estimates has to do with the underreporting estimates (although I am happy to be proven wrong).
The final big difference between the estimates is the way in which they modeled out of sample predictions. Because there are a lot of countries for which we have no data, and to get a sense of the validity of estimates, each group builds a model to predict maternal mortality based on variables such as GDP per capita, fertility, skilled birth attendants, and HIV. The Hill group modeled the outcome as a proportion whereas the IHME group modeled them as level deaths. Both claims that they approach is superior to the other alternative. The Hill group uses skilled birth attendants whereas the IHME group does not (it does is collinear with GDP per capita in their model). Data on HIV prevalence, which seems to make a big difference in the estimates, is also just an estimate, and one that has been updated substantially in recent years. About a quarter of the births in the world (and maybe as many as half of the maternal deaths since these are the poorest countries) are estimated this way. I won’t try to claim one approach is superior to the other, but I think it can generate big differences in overall estimates based on the assumptions made.
So what did we really learn from this exercise? Mostly that using a different set of assumptions it is totally possible to come up with a very different estimate of maternal mortality. If I wanted to, I could produce my own estimates – say I ignored underreporting altogether and just used the lowest reported data per country – and I could probably come up with an estimate even lower than the IHME estimate (although hopefully the Lancet would reject my submission on the basis that I have no clue what I am doing). These are both well respected set of researchers and I suspect both have come up with realistic estimates but we will never know which is really correct.
What I think is important here is to focus on what has not changed with these new estimates – that is what both groups agree to be true. First, maternal mortality has seen a slow but steady decline over the past 30 years but both estimates suggest that the estimated rate of decline (between 1-2% a year) is well below the 5.5% that would have been needed to achieve MDG5 since 1990. Even the more optimistic IHME data would suggest less overall progress during the 1990s, the time period after which the MDG clock began ticking. Second, some parts of the world have seen more progress than others (Asia, Latin America) and Africa has seen little or no progress. India (677 to 254) and China (165 to 40) have seen major proportional declines in the maternal mortality ratios and have have seen major increases in GDP per capita and declines in fertility, which could have greatly contributed to these declines (and might have even with no effort from the global health community). Many countries in Africa are actually worse off today than they were 20-30 years ago, mostly due to an increase in HIV associated maternal mortality and very little progress against the other causes of maternal mortality (and despite increases in GDP per capita during the past decade). Altogether, I still see little here to celebrate.
Don’t get me wrong – I believe that there is value in exercises to estimate things that are difficult to measure precisely. I believe that we can learn from such exercises and feel that no one group should have the monopoly on estimates and that it is good that there are groups like the IHME who are out there questioning the estimates of others (personally, I am really looking forward to their estimates on malaria control effects). That said, we need to be careful to not speculate too much from these estimates, whether conscious or not we are all biased and these can greatly influence the assumptions we make.
Many have already begun to claim that these new declines were due to increased efforts to scale up safe-motherhood initiatives, etc - but neither report has properly addressed the question of why maternal mortality has changed - so we really do not know. I also would have liked to see a more detailed explanation about how the different assumptions influenced the overall estimates. Estimates are only as good as the data used and the assumptions made so no one can really say whose made up numbers are best.Share on Facebook
It appears to be another all-IHME special at the Lancet this week. Don’t get me wrong – I am not complaining – I generally really enjoy these articles, and they always give me something interesting to blog about!
I blogged about the fungibility of health aid flows last week, and now they have released another report looking at the declines in the maternal mortality ratio globally from 1980-2008. For those who are not familiar with this metric, the maternal mortality ratio are the number of deaths of the mother per 100,000 live births. There is a general perception out there that the maternal mortality ratio has not declined substantially since the early 1980s, and this new article by the crowd at IHME argues that there has been somewhat more progress (albeit that much more) than originally assumed by most.
Unlike earlier estimates, Hogan et al. have estimated that the number of maternal deaths has declined from 526,00 in 1980 to 343,000 in 2008 while the maternal mortality ratio declined from 422 to 251 in 2008 – representing about a 1.5% decline annually over the entire time period. These are more optimistic estimates than those produced a few years ago by another set of excellent researchers, giving some hope that more progress towards achieving maternal mortality reductions have been made than previously believed.
The challenge here is that data for maternal mortality is particularly bad. I know I complain a lot about most data used in global health, but here we can really say it is total rubbish in many cases. In fact, even in the new estimates there are approximately 21 countries in the sample for which we have no estimates due to the lack of data sources. The innovation from the Murray group is that they managed to use more data sources, applied more standard approaches to correct differences in reporting and other biases, and have used a somewhat more realistic modeling strategy to account for changes. It should be noted, however, these are still just estimates. These figures cannot be compared to figures produced elsewhere and therefore are only as good as the model and estimates made by these researchers.
What might explain the discrepancies between what the IHME and other researchers have found? Have we really made that much more progress against maternal mortality? The answer appears to be both yes and no.
Looking quickly at some of the estimates, it seems that there has been more progress in high population places in than the other researchers had estimated. For example, the IHME estimates that the MMR in India in 2008 was 254 whereas the previous estimates were for 450 in 2005. This is a big difference and given how big India is, this difference alone may explain a bit part of the discrepancies between the estimates. Similarly, Nigeria (608 vs. 1100), Ethiopia (590 vs.720) and Bangladesh (338 vs. 570) are all large population countries with substantially lower new estimates. But the new estimates are higher in some countries, for example the United States (17 vs. 11). So the direction of the difference does appear to be driven by differences in data and estimation strategies across the board.
The bad news, however, is that even with these new estimates MMR rates are still way too high in many regions, even increasing in some countries, and the Millennium Development Goal 5 is not likely to be met globally. So we have potentially made more progress but the progress is hardly worth celebrating.
The authors argue, and I suspect correctly, that some of the biggest drivers of the observed MMR declines have been changes in fertility rates, female education, and incomes across the world. What share of these declines the global health community can really take credit for in this somewhat good news story is not really known, and that is too bad, because that is really what matters here.Share on Facebook
There are going to be at least two really exciting conferences looking at maternal health issues this summer, which are well timed given the debate and attention this issue is expected to receive at the upcoming G8 and G20 conferences. I wanted to pass the information along:
Sadly, as a woman that will be delivering this summer (35 weeks and counting), I probably won’t be able to make it to either of these events this summer (although I am still holding out hope for the August conference). So if you go, you have to promise to tell me all about it!
A lot has been made of the increase in aid flows for health to developing countries, however, national governments remain the most important financier of health programs in their own countries. From a sustainability perspective, in particular given the expected contraction or flatlining of health aid flows, and for many other reasons, national financing may be preferred method of financing. Arguably, government spending on health is a good indictor of the extent to which a government prioritizes health.
It turns out we actually have quite poor information on how countries have actually done relative to these goals. Health expenditure data is theoretically collected in standardized ways across countries by groups such as the WHO as well as the IMF, however, the country-specific methodologies and missing data means that making comparisons across countries or over time has been a real pain. I’ve tried using some of this data before, and I always have major reservations.
In theory, the data should include all government expenditures, including those financed by external donors, but it gets complicated when some of these flows are provided off-budget or in other ways. I once asked a colleague who is responsible for the National Health Accounts exercise in his country how he collects information on money from PEPFAR. He told me he goes to the PEPFAR website, figures out how much they say they spend in his country, and then he just assumes about 70% of that money is what is actually spent in his country. That is his solution, but I am sure every country has their own way of doing it.
There is a long-standing debate in the aid effectiveness literature that has explored whether or not foreign aid is fungible. Fungibility occurs when domestic resources are reallocated to other purposes when new monies arrive from donors. Ideally, donor spending should increase the total amount of money available for a specific purpose, but if allocations are fungible than it may not always be the case. For example, if an external donor commits X million to a health program in a given country, and that country reduces its budget to the health sector by Y, than the total budget has only increased by (X-Y). If Y is large, than new aid may do little to increase total resources. In the government expenditure data it would still look like total funding has increased, although by a lot less than hoped by donors.
A new study by the tireless folks at the Institute of Health Metrics and Evaluation, out today in the Lancet, provides the first good analysis of health expenditures in developing countries. They make a number of important contributions, which I will try to outline here.
First, they confirm that current health expenditure data is bad and should only be used cautiously. For the study, they compared health expenditure data from both the WHO and IMF, which I think was judicious, and end up with substantially different estimates depending on which data source is used.
Second, they do find that health expenditures in absolute terms developing countries have grown substantially in recent years. They find that HE have grown by nearly 100% from 1995 to 2006. For the most part, growth in funding has been driven largely by increases in GDP in many countries (some regions, such as sub-Saharan Africa has seen substantial growth in GDP during this time period), an increased share of total government expenditures on health, but a decreased share of GDP spent by government.
Third, the authors find substantial evidence of fungibility in a huge way: on average $1 of health aid given to governments, the ministry of finance reduces health expenditures by about $0·43 to $1·14. Yes, depending on the data source, health aid may have actually decreased overall health spending. This is particularly problematic in Sub-Saharan Africa, which is the region that has seen the highest growth in health aid – presumably because the health situation is the most dire in this region.
Curiously, they find that when health aid is given to non-governmental organizations as opposed to the government, not only does it not lead to the fungibility finding of above, it is actually associated with increased government spending. This might suggest some other mechanism is at work, but the positive effect makes me worry more about some sort of unaccountable selection effect. I also have tried to use this variable in the health funding before and have found it a bit suspect, so I am not making too much of this finding.
I think these results may come as quite surprising – and depressing – to many in the global health community. Have all past efforts to scale up development assistance for health actually reduced spending on health in the poorest countries?
I also think we should be really worried about the fungibility given how aid flows are generally allocated by donors – for disease-specific projects and recently for the purchase of commodities such as drugs and bed nets – where as typically government expenditures are more geared towards more system related spending: primary health infrastructure and health worker salaries. Donor aid might be squeezing out spending on systems in a great way. To the Ministry of Finance a dollar is a dollar, but to a patient in Africa a free bed net might be a poor substitute for a doctor to deliver a baby.
These findings are very important in how we think about the role of donors in financing health in developing countries – how are they really making things better? It might help explain the lack of good evidence that that aid for health has had much impact on health outcomes. I really encourage everyone to read this important paper and decide for themselves what they think. All comments are very welcome.Share on Facebook
Inspired by a comment from April – a frequent commenter on my blog – about whether Safe Motherhood saves lives or not, I decided to follow-up that post with this one, which asks the question of whether a policy approach similar to the one adopted in China would be effective in other countries, in particular in sub-Saharan African countries where levels of maternal mortality have remained stubbornly high over the past few decades?
The study I blogged on in China, claims that the SM approach reduced rates of maternal mortality by encouraging more women to delivery in hospitals – that is to have an institutional delivery, which is a more aggressive approach than has generally been adopted in most developing countries where the focus has been on the professionalization on rather than the institutionalization of deliveries (that is ensuring that they are supervised by trained medical personnel, regardless of where the births take place). Encouraging more women to deliver in a hospital would only really improves outcomes if we felt those hospitals would be adequately equipped and prepared to deliver the life saving interventions (c-sections, blood supply, other surgical procedures) that we think are needed to reduce maternal mortality. But what is the evidence on this?
Two recent papers jump immediately to mind – both asking some variant of this question by passionate researchers who strongly believe that access to surgical services have been under valued in public health systems in developing countries. They both argue, I believe correctly, that surgery should be considered and important part of the basic services that should be made readily available even in the most resource constrained settings.
The first, published in PLoS medicine last month assesses the surgical patterns of district hospitals in 3 African countries (Uganda, Tanzania, and Mozambique). The authors find “…low rates of major surgery at district hospitals in East Africa, ranging from 50 to 450 surgical procedures per 100,000 population.” Unsurprisingly, obstetric procedures are the most common of all surgical procedures conducted at this facilities but the authors conclude that the availability of surgical capacity is too low to address the full unmet need for obstetric surgical procedures.
The second, co-authored by Adam Kushner (aka @globalsurgeon on Twitter), evaluated surgical capacity at district hospitals in 8 developing countries and find that less than half of district hospitals in these countries, the normal referral sites for people with complications at the primary level, even had the capacity to conduct caesarean sections – something most people would agree is a relatively straightforward surgical procedure and essential to reducing maternal mortality. Half!
Both of these articles point to some important infrastructure challenges and really raises the question of whether or not broad one-sized fits all strategies should be the right approach. The results from the study in China, while interesting to know that Safe Motherhood can save lives, are not likely that applicable in other settings. In a given country, perhaps we should be asking which components of this strategy are the most likely to improve maternal mortality and how should it be implemented?
Ok, enough blogging about maternal mortality for one week, and back to being 8.5 months pregnant…
The current issue of the Lancet is devoted to the topic of health reforms in China. The Lancet put together a themed edition on the Chinese health care system nearly 18 months ago and this week’s issue is an update on progress to date. The editors argue that with regards to the health reform process:
“…there is unfinished business. Smoking is China’s greatest public health hazard, yet tobacco control efforts are timid and implementation lacks conviction. Ethics, including research conduct and governance, remain a concern. Contamination of infant milk formulas with melamine contributed to continuing disquiet about China’s Food and Drug Administration. And little progress has been seen in human rights or access to general information. The latter point—access to reliable information—is a key determinant for the quality of health care that people in China can expect.”
To read more, click here.Share on Facebook