October 18, 2006

Johns Hopkins/Lancet Study Demolished

Via Bryan at Hot Air, the politically-timed Johns Hopkins/Lancet study stating that more than 655,000 Iraqis have died as a result of the Iraq War has had its very suspect methodology thoroughly crushed:

After doing survey research in Iraq for nearly two years, I was surprised to read that a study by a group from Johns Hopkins University claims that 655,000 Iraqis have died as a result of the war. Don't get me wrong, there have been far too many deaths in Iraq by anyone's measure; some of them have been friends of mine. But the Johns Hopkins tally is wildly at odds with any numbers I have seen in that country. Survey results frequently have a margin of error of plus or minus 3% or 5%--not 1200%.

The group--associated with the Johns Hopkins Bloomberg School of Public Health--employed cluster sampling for in-person interviews, which is the methodology that I and most researchers use in developing countries. Here, in the U.S., opinion surveys often use telephone polls, selecting individuals at random. But for a country lacking in telephone penetration, door-to-door interviews are required: Neighborhoods are selected at random, and then individuals are selected at random in "clusters" within each neighborhood for door-to-door interviews. Without cluster sampling, the expense and time associated with travel would make in-person interviewing virtually impossible.

However, the key to the validity of cluster sampling is to use enough cluster points. In their 2006 report, "Mortality after the 2003 invasion of Iraq: a cross-sectional sample survey," the Johns Hopkins team says it used 47 cluster points for their sample of 1,849 interviews. This is astonishing: I wouldn't survey a junior high school, no less an entire country, using only 47 cluster points.

Neither would anyone else. For its 2004 survey of Iraq, the United Nations Development Program (UNDP) used 2,200 cluster points of 10 interviews each for a total sample of 21,688. True, interviews are expensive and not everyone has the U.N.'s bank account. However, even for a similarly sized sample, that is an extraordinarily small number of cluster points. A 2005 survey conducted by ABC News, Time magazine, the BBC, NHK and Der Spiegel used 135 cluster points with a sample size of 1,711--almost three times that of the Johns Hopkins team for 93% of the sample size.

Since the beginning, Les Roberts, one of the primary authors of the study has mantained that the study is methodologically sound.

Uh, not quite:

Curious about the kind of people who would have the chutzpah to claim to a national audience that this kind of research was methodologically sound, I contacted Johns Hopkins University and was referred to Les Roberts, one of the primary authors of the study. Dr. Roberts defended his 47 cluster points, saying that this was standard. I'm not sure whose standards these are.

Appendix A of the Johns Hopkins survey, for example, cites several other studies of mortality in war zones, and uses the citations to validate the group's use of cluster sampling. One study is by the International Rescue Committee in the Democratic Republic of Congo, which used 750 cluster points. Harvard's School of Public Health, in a 1992 survey of Iraq, used 271 cluster points. Another study in Kosovo cites the use of 50 cluster points, but this was for a population of just 1.6 million, compared to Iraq's 27 million.

When I pointed out these numbers to Dr. Roberts, he said that the appendices were written by a student and should be ignored. Which led me to wonder what other sections of the survey should be ignored.

With so few cluster points, it is highly unlikely the Johns Hopkins survey is representative of the population in Iraq. However, there is a definitive method of establishing if it is. Recording the gender, age, education and other demographic characteristics of the respondents allows a researcher to compare his survey results to a known demographic instrument, such as a census.

Dr. Roberts said that his team's surveyors did not ask demographic questions. I was so surprised to hear this that I emailed him later in the day to ask a second time if his team asked demographic questions and compared the results to the 1997 Iraqi census. Dr. Roberts replied that he had not even looked at the Iraqi census.

And so, while the gender and the age of the deceased were recorded in the 2006 Johns Hopkins study, nobody, according to Dr. Roberts, recorded demographic information for the living survey respondents. This would be the first survey I have looked at in my 15 years of looking that did not ask demographic questions of its respondents. But don't take my word for it--try using Google to find a survey that does not ask demographic questions.

Reviews of the Johns Hopkins/Lancet study casts strong doubts upon the credibility of the methodology used. When compared to other studies, I’d venture to say that the Johns Hopkins study is worthless and irreproducible, perhaps purposefully so. The timing of the report, once again issued in the weeks preceding a national election, casts strong doubts upon the intentions, credibility, and integrity of the researchers.

Then again, their campaign contributions and affiliations should have tipped you to their biases long ago.

Posted by Confederate Yankee at October 18, 2006 11:07 AM | TrackBack

With such wonderful standards, I am doubly glad my son chose a state school over this supposed elite university. (My first happiness in his choice is in not paying over $40K per year.)

Posted by: MikeM at October 18, 2006 11:24 AM

You have the right to block trolls. Enjoy living in a world where everyone agrees with you.
Anne Johnson

Posted by: pretty girl at October 18, 2006 11:56 AM

Why does the phrase "cluster f*ck" come to mind?

Posted by: Redhand at October 18, 2006 12:12 PM

Gee Anne, doesn't sound like you're too happy living in a world where everyone agrees with you....

Doesn't sound like you're too happy when people disagree with you.

Doesn't sound like you're too happy at all.

Posted by: SouthernRoots at October 18, 2006 12:18 PM

Wow, the author's credentials are certainly unassailable:

Mr. Moore, a political consultant with Gorton Moore International, trained Iraqi researchers for the International Republican Institute from 2003 to 2004 and conducted survey research for the Coalition Forces from 2005 to 2006.

Try again, brownshirts...

Posted by: dave™© at October 18, 2006 12:29 PM

Well, I'm going to back down here. Not because of Steven Moore, who is either manic compulsive or prone to exaggeration if he "wouldn't survey a junior high school, no less an entire country, using only 47 cluster points". But the IBC have released a statement which gives far better arguments than asserting that so many people couldn't have died because it's more than died in the American civil war or other rubbish:

One possible way of explaining such a very large number of small-scale unreported assaults is to suppose that many of these are the result of "secret" killings which have resulted from abduction, execution by gunfire, or beheading. But 42% of the 330,000 Lancet-estimated violent deaths in this final 13-month period are ascribed to "explosives/ordnance", car bombs, or air strikes, all of which carry a fairly heavy and hardly 'secret' toll (and will generally create at least 3 times as many wounded).
They're right: the IBC might not pick up anonymous murders, but they would indeed pick up car bombs.

Of course, as everyone sensible in this debate (including myself) have done up to now, they continue the grand tradition of qualifying their remarks by saying that they haven't proved that the figures are wrong. The IBC numbers are also inaccurate, and they acknowledge this fact. And I still want to know about excess deaths, and the Lancet survey was the first to make the attempt. Unfortunately, it looks like I'm still waiting to hear a credible answer to my question.

Posted by: Mat at October 18, 2006 12:31 PM

SouthernRoots' comment prompted me to click Anne's Pretty Girl. It is definitely worth a look.

Posted by: Redhand at October 18, 2006 12:42 PM

Well, the way I linked to to Anne Johnson's site triggered some kind of censorship. I'll simply say that Anne's link is to the gateway to "An Interfaith Sanctuary of EarthReligion." Can't say that I've seen anything quite like it before.

Posted by: Redhand at October 18, 2006 12:51 PM

"Wow, the author's credentials are certainly unassailable...Try again brownshirts..."

Iraq Body Count, often criticized for offering inflated civilian death figures, in their Press Release 16 October 2006 thinks the Lancet study is garbage too.

"In the light of such extreme and improbable implications, a rational alternative conclusion to be considered is that the authors have drawn conclusions from unrepresentative data."

Was this a better try, pinkshirts...?

(What is this "brownshirts" crap? Haven't the Hitler allusions been worked to death?)

Posted by: Major Mike at October 18, 2006 12:54 PM

Major Mike, no doubt, would also agree with the broader assessment of IBC:

Do the American people need to believe that 600,000 Iraqis have been killed before they can turn to their leaders and say "enough is enough"? The number of certain civilian deaths that has been documented to a basic standard of corroboration by "passive surveillance methods" surely already provides all the necessary evidence to deem this invasion and occupation an utter failure at all levels.

Posted by: d at October 18, 2006 01:28 PM

Either way, while I supported the war, as it turned out, it isn't going to turn out well. It was a 'bridge to far'.

Posted by: HLT at October 18, 2006 02:10 PM

I agree it is sad when innocent people are killed. I think we saved, are saving, and in the future will have saved a lot of innocent lives by fighting in Iraq, and in Afghanistan.

Muslims have killed Muslims and others whether religious or not all over the world in great numbers, regardless of our actions in Iraq. We will stay in Iraq and persevere, and Iraq will be the domino that fell and led to the end of Muslim violence.

Appeasement of violence only leads to uncontrolled violence and greater suffering. Where in recent history has weakness brought lasting peace?

Posted by: Major Mike at October 18, 2006 03:28 PM

It is obvious the Lancet study is propaganda, only a fool would not see it. As for credentials, what are Roberts's credentials? He has made no secret of his bias and if this study is an example of his skill I would say he needs to get in a new line of work. It seems that a lot more people are saying this study is bogus than are supporting it, well except for the hopelessly deranged anyway.

It is interesting how certain people on the left can only be concerned with the deaths of Iraqis if and when they think they can use those deaths for some political purpose. We all know that if Saddam was in power he could have wiede half the population off the map without a peep out of these people.

The numbers are simply outrageous. I think that 50,000 dead Iraqis is bad enough in and of itself. I guess some people think otherwise so they created a few hundred thousand more.

Posted by: Terrye at October 18, 2006 05:28 PM

A few hundred thousand with an margin of error of plus or minus 600,000. Real Science....LOL

Posted by: Specter at October 19, 2006 06:52 AM

Has anyone stopped to think how on the face of it this number is entirely implausible? First, where are the fresh graves? There should be 654,000 individual fresh graves (as opposed to mass graves under Saddam's regime).

Second, the rule of thumb is for every war death there are typically three times the number of wounded. This would mean something like 2 million Iraqis would have flooded hospitals throughout Iraq suffering clear war wounds. Maybe they have, I haven't heard if this is case.

Last. American forces have been in Iraq for a little over 3.5 years. Let's say for the sake of argument (civilians haven't been targetted throughout this entire period so cut me some slack here) that there has been this violence directed at Iraqi civilians for roughly a 1000 days. This would mean there would have to be 655 innocent Iraqi civilians killed EVERY DAY.

If there had been 100 Iraqi civilians killed every day for a thousand days, this would have been splashed as headline news all over the liberal media for those thousand days. This stuff about "secret" killings is just mere sophistry. May as well claim most of the Iraqis are being secretly killed by space aliens for some perverse experiment only known to them.

You liberals know as well as I do, that casualty rates are ALWAYS INFLATED, particularly in Muslim communities who are PR savvy and now aided and abetted by anti-war westerners. And you also know that the average civilian deaths as reported by a media just itching to ratchet up the death rate (even Iraqi hospital records and anectdotal reflections by hospital officials are notoriously unreliable and on the high side) has been somewhere around 25-30 "civilian" deaths a day for the last 18 months. That is roughly around 15,000 deaths. And it is well known that many "civilian casualties" consist of Muslim males in militias or freelancing and are killed by Coaltion forces in gunbattles. Other than the truly innocent women and children caught in between gunfire and IEDs, the radical Muslim male deaths in sectarian violence in Iraq is little different than gangland turf battles here in America ... good riddance.

And there is little doubt the overwhelmingly vast majority of innocent civilian deaths are a result of Muslim on Muslim violence. And blaming this religious sectarian violence on the American presence is like invoking the Great Satan America-made-me-do-it argument - and the devil made me do it argument has never washed with me.

You don't blame doctors for the spread of cancer when they do find cancer and try to treat it.

Posted by: Hankmeister at October 19, 2006 08:32 AM

If we leave it to memory, then all we'll have is the newspaper "of record" in the really long term. How many blogs will come and go who diligently exposed the lies and omissions of the NYT, yet what counts in the long term is where someone goes for "the record", and if you carefully deconstruct the NYT articles, the most biased part is in the headlines and the first paragraph, sometimes continuing on in subsequent paragraphs until the end where they sometimes do a 180 degree turn and admit to economic numbers they denied in the first 80% of the article, saying how bad other things are when most other newspapers just report the economic numbers. This is deliberate because the articles are archived in the NYT by title and first paragraph or first two paragraphs from the article so you can decide whether to "purchase" the past news.

So, going back to memory, since I can't find the article, my computer must have went down before I had a chance to archive the article(s) a couple of years ago... during the previous election two years ago, Lancet, from memory, pulled the same stunt. Don't know if it was the same authors, the same school, but Lancet, from memory, was involved. At that time, again, some time before the election, they came out with a study that was repeated in the left wing blogs ad nauseam, where some 100,000 Iraqi civilians had been killed in the first year, or beginning of the war. The 100,000 number was touted on DailyCows, probably Communist Underground, and many, many more left wing and extremist blogs. And they succeeded in getting the fiction into letters to the editor. Except someone experienced in statistics and surveys took a close look at the survey that was performed, and from what I remember it was performed in the same way as this current one, by interviews and unsubstantiated by other methods with one excuse being the "danger" involved in verifying the numbers mostly collected by Iraqis. But the real kicker was that the person who took a look at the number of 100,000 also took a look at the statistical error probability, error rate, whatever its called in statistics. It turns out the model they used would have made 5,000 just as accurate as 100,000, just as accurate as a much higher number, and all they did was take the high number, or the middle number, and claim that that was the correct number. So in the survey two years ago by Lancet on Iraqi deaths because of the war, it would have been just as accurate according to the statistical methodology they used, for them to claim that 5,000 deaths resulted, as it was for them to claim what they really claimed, that 100,000 deaths resulted.

Again, someone, possibly a professor from another school, took a close look at the numbers, methodology, probability, error rate and everything else you look at to see if the survey stands up, and he found that the survey did not stand up unless 5,000 deaths was included as just as possible and just as precise as the 100,000 deaths figure. But this revelation didn't matter. NYT, the left wing blogs, Reuters, AP, AFP, and letters to the editor all ran with the 100,000 deaths number and no one issued a correction, other than the middle of the road and right wing blogs who picked up on the professor's comments.

Maybe someone can find a link to the two year old Lancet study being debunked. I failed to archive the article, and a later exhaustive google search two years ago failed to find the article again. Someone well versed in searching with google may be able to find the old Lancet study and the article debunking it.

Posted by: Jeau L'expose at October 19, 2006 01:08 PM

As a matter of fact, I'm quite happy indeed. It's all those lazy summer afternoons at the InterFaith Sanctuary, I guess. Does a body good.

Anne Johnson
B.A. The Johns Hopkins University, 1981

Posted by: gorgeous babe at October 19, 2006 03:18 PM


I think this article may be referencing the survey of which you were speaking:


It is relative to note that the Iraqi city of Falluja was included in their sample.

Thank you for mentioning this survey. I had forgotten about it. These lancet surveys bring to mind my mom's oft quoted.. "fool me once, shame on you... fool me twice, shame on me... "

Posted by: MOPI at October 19, 2006 06:14 PM

Oh, so if Johns Hopkins is wrong and the Dear Leader and the PNACers have only helped kill, say, 300,000 people, that's something to be proud of? No wonder your party is about to be cast upon the ash heap of history. See you November 7th!

Posted by: blogenfreude at October 22, 2006 06:05 PM

Whatever the number of Iraqis slaughtered by Bush and his neo-con warmongers, it is a tragedy unprecedented in the history of our once free and great republic.

Posted by: jose at October 24, 2006 05:55 PM