An NGO in Africa Goes Awry
5:40 PM, Oct 1, 2012 • By ARMIN ROSEN
In late August, the world’s most high-profile development project celebrated another milestone. The Millennium Villages Project opened its newest site, in northern Ghana, with newly minted Ghanaian president John Mahama and the UK’s international development secretary on hand. The official interest in the project’s arrival in Ghana only emphasized its unique stature in the typically wonkish world of development economics. Wherever MVP founder, Columbia professor, bestselling author and former World Bank presidential hopeful Jeffrey Sachs and his vision of aggressive, integrated rural development go, money, attention and political approval are almost bound to follow.
The project’s expansion and Sachs’s photo-op in Ghana belies the fact that just a few months ago, the empirical basis of Sachs’s enterprise was revealed to be flimsy. On May 8, the British medical journal Lancet published a paper that was supposed to vindicate the MVP, which has attempted to prove that an intensive and targeted sustainable development program could improve the lives of villagers in 10 African nations—and maybe in the rest of the developing world as well. He sold the MVP as nothing less than a revolution in development policy. The trouble is that the project has had difficulty proving that it was having its intended effect, or even any effect. The Lancet paper was going to change that. Its co-authors included John W. McArthur, the one-time CEO of Millennium Promise, the MVP’s fundraising arm, and Sachs.
Instead, the opposite happened. A letter from four respected development experts—Jesse Bump of Georgetown University, Michael Clemens of the Center for Global Development, Gabriel Demombynes of the World Bank, and Lawrence Haddad of the University of Sussex—demonstrated that one of the paper’s key findings, that child mortality decreased in the Millennium Villages at three times the national rate, had been reached in error. On May 21, the MVP withdrew the claim.
In the summer, I spoke to both Clemens and Bump. As social scientists, they are willing to believe that the MVP’s integrated rural development model is worthy of the political and financial support that’s currently lined up behind it—but only so long as there’s evidence that Sachs’s methods actually work. “The villages have put themselves up as a model, and there’s some burden of proof about demonstrating whether the model works or doesn’t work,” says Bump. “All we’re saying is that the proof isn’t here yet.”
The MVP has exhibited a startling anti-empirical streak that predates the Lancet controversy. When the project was launched in 2005, it included no control group, in this case villages that would not receive development assistance from the project, and whose observed progress could be compared to conditions in the intervention area. Without a control group, the MVP was able to claim that any improvements in the villages were a direct result of the project’s interventions. For instance, in a 2008 report the MVP credited itself for an increase in cell phone usage in the villages—even though cell phone access had been dramatically increasing throughout Africa over the same period.
The MVP introduced control villages in 2007. But evidence of the project’s success was hardly forthcoming. In 2010, Clemens and Demombynes responded to the MVP’s recently published mid-term evaluation report with an article in the Journal of Development Effectiveness that reached some damning conclusions about both the methodology and efficacy of the project. For instance, Clemens and Demombynes determined that “initial estimates of the project's effects change substantially if more rigorous impact evaluation methods than those used in the project's mid-term evaluation report are employed”—in other words, the project looks a lot different (and a lot less successful) if you evaluate the villages in a national or even regional context, instead of using the MVP’s preferred method of comparing the villages to the earlier, pre-intervention versions of themselves. Clemens and Demombynes wrote that the initial phase of the project was so badly designed that it was “impossible to make definitive statements about the project's effects.”
The MVP simply shrugged off the paper’s conclusions. “They did not take our criticisms seriously,” Clemens told me. “They denied the legitimacy of every single point we made, and they changed nothing.” In the MVP’s official response to the paper, Sachs and McArthur wrote, “Economists like Clemens and Demombynes should stop believing that the alleviation of suffering needs to wait for their controlled cluster randomized trials.” For Clemens, the moralistic suggestion that he stands in the way of the alleviation of suffering, and that the MVP was simply too important to adhere to sound social scientific practice, was “baffling,” as well as rankling. “The response was to obfuscate, rather than enlighten,” Clemens said.
Two years later, the MVP is still guilty of bad social science. The Lancet paper is riddled with errors. The paper’s authors committed a computational mistake that inflated the apparent decline in the villages’ child mortality. At the same time, the paper’s calculations were based on out-of-date national child mortality figures that underestimated the decline in national child mortality. Somewhat astonishingly, the Lancet error reproduced a similar mistake from an MVP-authored paper in the American Journal of Clinical Nutrition a few months earlier. In that paper, the project team used a misleading statistical analysis to prove that the MVPs had decreased stunting at village sites in Ghana. In reality, the decrease was almost identical to national level trends.
The Lancet controversy resulted in a minor shakeup at the MVP—project coordinator Paul Pronyk was re-assigned, and Sachs organized an independent panel of experts to probe the causes of the error. But it has not had a humbling effect on him. A few days after the Lancet correction was announced, Sachs spoke with the aid website Humansophere, and was asked whether he had “evidence the [MVP’s] approach is working.”
“It depends on what you mean by evidence,” Sachs replied. “Some of my critics say we need to do these ‘randomized controlled trials’ as if what we’re doing is testing a red pill against a blue pill. What we’re doing has nothing to do with anything like that. It cannot be reduced down to such a simple and narrow test. . . . This is not a randomized controlled trial; it’s a learning process.”
In fact, the MVP is not a “learning process”: It’s a major project created in lockstep with the Millennium Development Goals, the United Nations’ signature development initiative. It’s a driver of both limited aid resources and global development policy, and its leader is the world’s most famous and influential development economist. Sachs seems to believe that because the MVP represents such an important and even historic leap in development policy, it should be shielded from the kind of rigorous empirical scrutiny that social science demands. To that end, the MVP keeps all of its raw data secret, far from the prying minds of outside researchers.
Sachs’s attitude towards accountability is troubling for another reason as well. He has cultivated a close relationship with some of the worst tyrants in Africa, leaders who, not surprisingly, have embraced a Western-funded pro-development agenda that makes no demands for political reform.
In the acknowledgements to The End of Poverty, Sachs thanked “Africa’s new generation of democratic leaders that are pointing the way,” a list that notoriously included Ethiopia’s Meles Zenawi and Mwai Kibaki of Kenya. Zenawi, who died on August 20, had ruled Ethiopia since 1991; his government imprisoned journalists and opposition figures, and effectively banned civil society groups and NGOs in 2009. Kibaki is most famous for declaring himself president in the midst of Kenya’s disputed 2007 elections, an act that triggered some of the worst ethnic violence in the country’s history. Sachs is especially admiring of Zenawi: In a 2004 speech, Sachs described the dictator, along with Ugandan president-for-life Yoweri Museveni, as “ingenious, deeply knowledgeable, and bold.”
Sachs gives the impression of being unbothered by leaders who steal elections, imprison dissidents, and meddle in their neighbors’ affairs. Just as importantly, these leaders seem unbothered by him. Sachs’s brand of development doesn’t require systemic political reform—just pliant authority figures who can foster the kind of stability and cooperation that an undertaking like the MVP requires. Considering Sachs and the MVP’s prominence, these are appallingly low expectations.
Recent Blog Posts