October 12, 2006
Questionable Methods, Questionable Results
It seems that the British Lancet has a certain problematic pattern of behavior:
From ABC News last year:
Indian experts say a new study which found that some 10 million female foetuses may have been aborted in the country in the past 20 years was sensationalist and inaccurate.The study, published online by British medical journal The Lancet, says the practice of selective abortion is due to a traditional preference for boys in India.
"It is a sensational piece of work. We are very, very concerned about this study," activist Sabu George said, who has been campaigning against the practice of foeticide for more than two decades.
"An unreasonable estimate will undermine the issue," he said.
Exaggerated? An unreasonable estimate in the Lancet? Shocking.
Worried about the hype generated about "Frankenfood?" If you want to guess where it came from, thank this New York Times article in 1999:
a prestigious medical journal is publishing a study suggesting that genetically modified food may be harmful, even though the research has been widely criticized by scientists and was found wanting by some of the journal's own referees.The Lancet, a journal based in England, said had it decided to publish the study in part to spur debate and to avoid being accused of suppressing information on a controversial subject.
The study is also likely to be seized upon by opponents of such food in the United States, where consumers have until recently expressed little concern about the genetically altered corn and soybeans that have swept quietly into their diets.
Charles Margulis of the Washington office of Greenpeace was quoted as saying, "I think it gives it a certain scientific credibility. It's going to increase concern here in the United States." But the decision to publish the study is itself generating debate: some scientists say the Lancet has lowered its standards and subverted the peer review process.
Subverting peer review and lowering its standards of accuracy? Surely, this is not the Lancet we're talking about.
Speaking of questionable accuracy and low standards, do you remember the 1998 study linking the common childhood MMR (measels,/mumps/rubella)vaccine to autism? It didn't do too well.
Ten of the original 13 authors of a controversial 1998 medical report which implied a link between autism and the combined MMR vaccine for measles, mumps, and rubella, have retracted the paper's interpretations.The retraction will be printed in the 6 March issue of The Lancet, which published the original paper. One author could not be reached and two others, Peter Harvey and lead author Andrew Wakefield, refused to join the retraction.
"We wish to make it clear that in this paper no causal link was established between MMR vaccine and autism as the data were insufficient," write the 10 authors in their retraction. "However, the possibility of such a link was raised and consequent events have had major implications for public health."
The original paper, which was based on parental and medical reports of just a dozen children, suggested a "possible relation" between autism, bowel disease, and MMR. The paper added it "did not prove an association".
The Lancet rushed through a under-sampled study spearheaded by a possibly dishonest scientist. Interesting.
It seems that sometimes a desire to influence or shock public sensibilities seems to get the better of the Lancet from time to time, as it did when it claimed just prior to the 2004 elections that 8,000-194,000 (but most often trumpeted as 100,000) Iraqi civilians had been killed.
Funny, how the UN Development Program Iraq Living Conditions Survey using similar cluster survey methodology but on a far greater scale, recorded only 24,000 deaths published five months later with a 95% confidence interval of 18,000 to 29,000.
If you didn't know any better, you might just think their studies were driven by leadership more interested in exerting political influence than presenting valid science.
It's a good thing that couldn’t be the case.
Any similarities between those who attack this study and Ahmadinejad's Holocaust denying are purely coincidental, I'm sure...
Posted by: monkyboy at October 12, 2006 07:49 PMAh, but here's the difference: we know millions actually died in the Holocaust. We know many of their names, who they were, what they did, and where they came from. We have hard data based upon census records, property deeds, marriage, birth and death records, etc.
All we've got from the Lancet are extrapolated paper corpses.
It seems that sometimes a desire to influence or shock public sensibilities seems to get the better of the Lancet from time to time, as it did when it claimed just prior to the 2004 elections that 8,000-194,000 (but most often trumpeted as 100,000) Iraqi civilians had been killed.
So some journal articles in the Lancet (as in every journal) have been questioned (as is often the case in science) - and yet you seem to have failed to provide any credible criticism of either the 2004 or the 2006 articles. This is akin to creationists stating that since Piltdown Man was a fraud, evolution is bunk.
Do you have any substantive criticisms of eithe r the 2004 or 2006 study?
Funny, how the UN Development Program Iraq Living Conditions Survey using similar cluster survey methodology but on a far greater scale, recorded only 24,000 deaths published five months later with a 95% confidence interval of 18,000 to 29,000.
Funny that that's what you claim it states, but the link you give doesn't give a working copy of the survey.
Fortunately, there's a copy here.
And, lo, if we take a look on p. 54, we can see that (a) the ILCS talks about a restricted definition of "war-related deaths", whereas the Lancet study talks about "excess deaths", and that the ILCS study states that it has underestimated deaths by not including households where all members were lost.
You also seem to have conveniently forgotten to mention that the ILCS study attempts to provide another estimate of mortality by making the interesting point that 13% of Iraqis report dead fathers as compared to Jordan's figure of 8%. Let's see - 5%, divided by 2 as they're talking about males... Why, that seems remarkably in line with the Lancet's estimate of 2.5% of the population dead due to the invasion and war, doesn't it?
Why is it, do you suppose, that you didn't provide a working link to the report when you failed to quote from it properly?
Posted by: Phoenician in a time of Romans at October 12, 2006 09:40 PMI think that Al-Jazeera isn't showing video of tens of thousands of fresh graves is prima facie evidence they simply don't exist.
Posted by: Purple Avenger at October 12, 2006 10:26 PMI guess we need to define "credible criticism."
I would define a credible estimate as one that starts from a valid baseline, and can be either repeated, or correlated by another similar method or process. The 2004 Lancet study (actually, a questionnaire sent to less than a thousand homes in just 33 areas) has never been correlated. The baseline data has been questioned. The methodology was questioned both based on scale (the sample size was criticized as being too small), and was accused of being unevenly distributed. It was never repeated, and no other casuality estimates of any othr methodology or source came to figures that were even close. In short, there is no reason to assume it is valid.
If anything, the UN Study that came out five months later--and other sources agree with my interpretation of the UN data, not yours--was a far more scientifically valid study. It involved a far larger sample size (more than 21,000 households in 2,200 areas) while utilizing similar claimed techniques and methods.
Logically, a larger sample size over a much more diverse are will give you a much more refined spread. The Lancet 2004 study was technically correct: it estimated deaths between 8,000-194,000. The U.N report, using similar techniques much better and across a wider population sample, effctively narrowed down that wide range of 8,000-194,000 to just 18,000-29,000.
I wouldn't call it technically a debunking, just prove that the 2004 Lancet study--and the 2006 study apparently built from the same shoddy framework--are essentially meaningless... except a political agi-prop, and that is what they were created for in the first place, weren't they?
Posted by: Confederate Yankee at October 12, 2006 10:49 PMCY - Richard Nadler at NRO has a good piece that corresponds with your position.
The Hopkins researchers chose their “base-line” for pre-invasion Iraq carefully: January 2002 through March 2003. They chose a period of time in which Baathist violence against the Kurds was restrained by a U.S.-imposed no-fly zone in the north; a period of time after the extermination of hundreds of thousands of Shia and Marsh Arabs in the south.Burnham and associates carved out a brief period of enforced peace within a 25-year regime singularly dedicated to war and internal slaughter. They called it a “baseline,” and they compared that baseline against a period of war.
Basically, the bogus baseline gave them a much larger multiplier to use on postwar "data" to shape their data into the antiwar message they are trying to espouse.
I'm with you and Purple Avenger - if the number of civilian deaths were that high, with the press we have today and the Arab press, there is no way that information wouldn't have been on the news 25/8.
The underlying part of this is that we are supposed to believe that coalition (i.e. American) soldiers are more wanton killers of Iraqi civilians than Saddam Hussein ever was.
The lefties arguing for the accuracy of this report are probably still saying, "We're against the war, but we support the troops (lying, muderous, rapist, massacring puppets of Rumsfeld and Bush that they are)....
Scientists aren't God. They are human. When they skew data to support a specific political goal, they cease to be scientists and are not worthy of trust.
What's the phrase? Lies, Damned Lies, and Statistics.
Posted by: SouthernRoots at October 12, 2006 11:09 PMI would define a credible estimate as one that starts from a valid baseline, and can be either repeated, or correlated by another similar method or process.
What, in your vastly informed view, would have been an appropriate baseline for this study? Perhaps the death rates during the 1988 Anfal campaign? Or during the Shi'a massacres of 1991? Would those baselines have satisfied you?
More importantly, I'd like to know -- again, in your vastly informed view -- on what basis you would like to claim that this study cannot be repeated or (and I don't know what you mean by this) "correlated by another similar process or method."
Your sloppy use of the passive voice here makes it sound as if others have tried (and failed) to reproduce the results of this study. If that has indeed happened, you should share your findings with the world. Meantime, though, you embarrass yourself by suggesting that cluster sampling is a "shoddy" framework for arriving at estimate like this one. It's not. Cluster sampling -- as my colleagues who actually do cluster sampling tell me -- is a highly reliable method and is used throughout the field of public health. There are reasonable and informed critiques that can be made about the methodology of any study, including this one.
Your approach -- which is shared by your fantastically well-informed commenters -- is to (a) deliberately misread the actual claims of the study; (b) duplicitously insist that the study is wrong because it contradicts the findings of a non-scientific tally by Iraq Body Count, which counts media reports of deaths directly attributable to the war; (c) yodel incoherently about the lack of press coverage, the lack of "fresh graves," or whatever else proves (by its absence) your suspicions; and now (d) insist that because The Lancet has been criticized before by someone, somewhere, the results of this study are automatically called into question.
It's no wonder that you can't fathom the statistical methods in this study. You can't even make a coherent, logical argument about the most basic facts of the case.
Posted by: d at October 13, 2006 01:23 AMLooking over the data on the Holocaust, it seems the primary source of information comes from the Nazis themselves, not "census records, property deeds, marriage, birth and death records, etc."
Such detailed records aren't currently being kept in Iraq, so sampling would seem to be the best way to get an accurate number.
For those who doubt the Lancet study, I think you should pause and ask yourself why...
Posted by: monkyboy at October 13, 2006 04:03 AMWhat, in your vastly informed view, would have been an appropriate baseline for this study? Perhaps the death rates during the 1988 Anfal campaign? Or during the Shi'a massacres of 1991? Would those baselines have satisfied you?Let me have a go at this. It depends on the question you want answered. The question I personally would like answered is, how does the current death rate in Iraq compare to what would have happened if the invasion hadn't taken place? That's obviously pretty speculative. By adopting the baseline that they did, the study authors assumed that things would have stayed more or less as they were shortly before the invasion. They didn't include periods of genocide against Kurds or Shi'ites because international action had mostly brought an end to them, and there was no reason to believe they would flare up again. This seems like a sensible approach.
However, I personally think that without international intervention, Saddam Hussein's government would most likely have collapsed under its own steam and there would have been a civil war. This could well have produced a situation even worse than we're seeing now. That is the only good argument I've come up with for the invasion of Iraq, but it's not one that you hear used much, probably because of the debacle of Somalia. And because it's so speculative, it's a poor baseline to use in a scientific survey.
The reason I don't like the IBC is not just that it's an underestimate, but that it can also be accused of being an overestimate. Violent death happens in every country. Who knows how many people would have died anyway? You have to subtract the "normal" level of violent death from the IBC numbers to assess how bad the war has been for the local population. But there was no free press under Saddam Hussein, so there's no way to figure out what that baseline level should be.
The value of the Lancet article is that it answers precisely this question. It measures something completely different than the IBC does: excess deaths. It includes deaths from disease, accidents, and crime, not just organised attacks by military units, and it's free from the natural distortion produced by the press. It's therefore a far more useful piece of information for assessing the impact of the war (although the IBC is still valuable data).
It would be wonderful if a second group could replicate the survey and let us know what they find. This is, however, unlikely. Compiling this information is dangerous work; they didn't just mail out surveys, they had to physically turn up to these places. Even doing it once might be considered foolhardy.
Finally, here's an exercise for you, Bob, to demonstrate what the IBC numbers actually represent. You live in North Carolina, do I have that right? Raleigh has a population of 276,093. The death rate in the United States is 8.25 / 1,000 population / year. Therefore, in Raleigh, there should be something like 8.25 / 1000 * 276,093 / 365.25 ~= 6 deaths each day. So your exercise is the following: find the press reports from Thursday for each of those six deaths. Paid obituaries don't count, since they don't count in the IBC. Or, since 6 is a pretty low sample, make it 18 deaths from Tuesday to Thursday if you want. In either case, I bet you can't.
What does that prove? Not every death is reported in the press. Not even a large fraction of deaths are reported in the press. Death is a normal part of life, especially in Iraq, and is therefore usually not important enough to make the news. Any estimate of the number of people dying in a country which uses only press reports as its source will be a massive underestimate.
d and Mat - Show me the bodies. We're talking about dead people - lots of dead people. Unlike perceptions and feelings, dead people are something that can be "seen and touched".
Where are the pictures of massive increases in burials? Where are the Arab news stories with the photographic proof?
Did you personally and independantly recreate the findings of the study? Where's your report?
In order for us to beleive this study, one would have to believe that there has been a MASSIVE coverup (so to speak) by the new Iraqi government, the US, UK, Japan, Russia, China, Spain, France, Germany, all the other countries in the coalition, the UN, the Red Cross, US news organizations, International news organizations and news agencies that definately not friendly to the Bush administration.
Who's in charge of this coverup and how have they managed to keep the lid on it for three and a half years?
Prove to me the validity of the study by using it to verify auto deaths in America. Compare the methodology results with the official numbers reported by the government. Use a baseline of pre SUV/post SUV or some delineator of improved safety (airbags).
I have seen way too much of the politicization of science, so I am a skeptic when a "scientific" study produces results extremely higher than anyone else is reporting.
The study is a nice theoretical approach - where are the bodies and mass graves?
Posted by: SouthernRoots at October 13, 2006 09:35 AMShow me the bodies. We're talking about dead people - lots of dead people. Unlike perceptions and feelings, dead people are something that can be "seen and touched".You're applying standards that are absurd. No-one has the time or inclination to go around photographing every single dead body in Iraq. Why would you expect that they would? Some incidents, the most spectacular ones, are photographed and reported. But after reporting the thousandth random anonymous murder, no journalist needs a giant conspiracy to tell them not to bother reporting the next one. It's a waste of time.Where are the pictures of massive increases in burials? Where are the Arab news stories with the photographic proof?
What about the numbers for Hiroshima and Nagasaki you flung around? Do you think that someone went around photographing every single body after those incidents? Do you think that there was a morgue slapping death certificates on every single one of the 214,000 (a suspiciously round number, if you ask me)? It seems that statistical methods are fine when they back up the point you're making, but unacceptable when they don't.
Posted by: Mat at October 13, 2006 10:49 AMI do apologise, I was sorta expecting a response from Bob. I didn't realise that that was SouthernRoots I was replying to. Please disregard the whole response if you like.
Posted by: Mat at October 13, 2006 10:59 AMIn order for us to beleive this study, one would have to believe that there has been a MASSIVE coverup (so to speak) by the new Iraqi government, the US, UK, Japan, Russia, China, Spain, France, Germany, all the other countries in the coalition, the UN, the Red Cross, US news organizations, International news organizations and news agencies that definately not friendly to the Bush administration.
No, all you have to believe in order to take this study seriously is that the mortality rate has nearly quadrupled since March 2003. There are no mass graves because the rise in mortality has been distributed across the country, and because the cases of death cannot be assigned to single factors -- like, say, the Anfal genocide or the November 2004 Fallujah massacre.
As anyone with even a rudimentary understanding of statistics could tell you, the phenomena measured by statistical data is often "invisible" within the context of everyday experience -- that's one of the reasons that statistical studies are useful. Science often produces results that run contrary to the quite un-scientific work of reporting. On the merits of the study itself, there is no compelling reason to dispute the methods or the conclusions.
Posted by: d at October 13, 2006 01:43 PM
