(DV) Media Lens: Burying The Lancet (Part I)

FREE hit counter and Internet traffic statistics from freestats.com

Burying the Lancet (Part 1)
An Exchange Between The Independent’s Mary Dejevsky
and Lancet Author Les Roberts
by Media Lens
www.dissidentvoice.org
September 11, 2005

“It is odd that the logic of epidemiology embraced by the press every day regarding new drugs or health risks somehow changes when the mechanism of death is their armed forces.”

-- Les Roberts, Johns Hopkins School of Public Health

As a test of the independence and honesty of the mass media, few tasks are more revealing than that of reporting our own government’s responsibility for the killing of innocents abroad. In an age of “converged” political parties and globalized corporate influence, few establishment groups have any interest in seeing such horrors exposed, while many have much to lose. Corporate journalists are therefore subject to two very real, competing pressures:

1) the moral, human pressure of reporting honestly our responsibility for mass killing, and

2) state-corporate pressure and flak punishing dissent and rewarding servility to power.

The results tell us much about the moral and political health of our media and our democracy.

On July 20 an article by Terry Kirby and Elizabeth Davies in the Independent noted that a November 2004 report in the Lancet had estimated Iraqi civilian deaths at nearly 100,000, but that the methodology “was subsequently criticised.” (Kirby and Davies, ‘Iraq conflict claims 34 civilians lives each day as “anarchy” beckons,’ The Independent, July 20, 2005)

The report in question was produced by some of the world’s leading research organizations -- the Johns Hopkins Bloomberg School of Public Health in Baltimore, Columbia University, and Baghdad's Al-Mustansiriya University -- and was published in one of the world’s most prestigious science journals: The Lancet. We were therefore keen to know which criticisms Kirby and Davies had in mind. We wrote to the Independent and Kirby replied on July 22:

“So far as I am aware, the Lancet's report was criticized by the Foreign Office.” (Kirby to David Edwards, July 22, 2005)

Also on July 20, an Independent editorial claimed that the Lancet findings had been reached “by extrapolating from a small sample... While never completely discredited, those figures were widely doubted.” (Leader, ‘The true measure of the US and British failure,’ The Independent, July 20, 2005)

We challenged the Independent’s Mary Dejevsky, senior leader writer on foreign affairs:

“What is the basis for the claim that the sample was ‘small’? The report authors told me that the sample was standard for research of this kind, so that ‘we have the scientific strength to say what we have said with great certainty. I doubt any Lancet paper has gotten as much close inspection in recent years as this one has!’”(David Edwards to Mary Dejevsky, July 21, 2005)

Dejevsky responded on August 10:

“personally, i think there was a problem with the extrapolation technique, because -- while the sample may have been standard for that sort of thing -- it seemed small from a lay perspective (i remember at the time) for the conclusions being drawn and there seemed too little account taken of the different levels of unrest in different regions. my main point, though, was less based on my impression than on the fact that this technique exposed the authors to the criticisms/dismissal that the govt duly made, and they had little to counter those criticisms with, bar the defence that their methods were standard for those sort of surveys.
regards, mary” (August 10, 2005)

We responded on August 18:

“Thanks, Mary. You say that ‘personally’ you ‘think there was a problem with the extrapolation technique’ because while the sample was standard it was ‘small from a lay perspective’. Your argument then is that the problem with the extrapolation technique was that people like you had a problem with it because the sample seemed too small. That's a deeply shocking response from a senior journalist writing in a serious newspaper about such an important report. We are talking about our responsibility for the mass death of civilians, after all.

“Should the methodology not be judged by the standards of science and reason rather than some ill-informed ‘lay perspective’? Why on earth would we judge anything of importance by the standards of an ill-informed view?

“Your claim that the authors had little with which to counter criticism is flatly false. I can send you many powerful replies provided to us by the report authors in response to a range of (mostly trivial) criticisms we found in the media.

Best wishes

David Edwards”

Dejevsky replied the same day:

“thanks -- i obviously sounded more off-hand than i intended. i just feel that extrapolation may be entirely sound when you can project over relatively uniform areas (subject, geographical whatever), but that -- common sense suggests -- it will be less reliable when the situation is so uneven, as in iraq. this may be unjust and ill-informed, and maybe the arguments from the report's authors were not sufficiently aired because they were - in effect - suppressed. if you have some of the counter arguments i would be interested to see them (beyond the defence that the methodology is standard, tried and tested etc).

“incidentally, i think it is absolutely legitimate, and right, for journalists to apply a common sense standard to scientific arguments and methods. we should have been far more exacting over the intelligence methodology that gave us saddam's wmd, for instance. all the best, mary” (August 18, 2005)

This was a challenge we had to accept. We were disturbed by Dejevsky’s response and were keen to know what the team behind the Lancet report would make of it. We contacted Les Roberts, a world renowned epidemiologist and lead author of the report. Roberts responded on August 22 with an email which he asked us to forward to the Independent:

“Dear Mr. Kirby and Ms. Dejevsky,

“I was disappointed to hear that you felt our study was in some way dismissed by Jack Straw’s anemic response to our report in the Lancet last November. Serious reviews of our work and the criticisms of it were run in the Financial Times, the Economist, the Chronicle of Higher Education (attached above) and the WSJ [Wall Street Journal] Online on August 5th. Closer to home, John Rentoul of the Independent solicited a response to the Jack Straw letter last Nov. 21st and we responded with the attached letter [Not provided here]. I am told that it was printed by your paper.

“Many people, like Ms. Dejevsky, have used the word extrapolation to describe what we did. When I hear people use that word they mean what is described in my Webster’s Unabridged: ‘1. Statistics. to estimate the value of a variable outside its tabulated or observed range.’ By this definition and the one I hear used by everyone on this side of the Atlantic, we did not extrapolate. We did sample. We drew conclusions from within the confines of that universe from which we sampled. Aside from a few homeless and transient households that did not appear in the 2002 Ministry of Health figures or households who had been dissolved or killed since, every existing household in Iraq had an equal chance that we would visit them through our randomization process.

“I understand that you feel that the sample was small: this is most puzzling. 142 post-invasion deaths in 988 households is a lot of deaths, and for the setting, a lot of interviews. There is no statistical doubt mortality is up, no doubt that violence is the main cause, and no doubt that the coalition forces have caused far more of these violent deaths than the insurgents.

“In essence this is an outbreak investigation. If your readers hear about a sample with 10 cases of mad cow disease in 1000 British citizens randomly tested, I am sure they would have no doubt there was an outbreak. In 1993, when the US Centers for Disease Control randomly called 613 households in Milwaukee and concluded that 403,000 people had developed Cryptosporidium in the largest outbreak ever recorded in the developed world, no one said that 613 households was not a big enough sample. It is odd that the logic of epidemiology embraced by the press every day regarding new drugs or health risks somehow changes when the mechanism of death is their armed forces.

“The comments of Ms. Dejevsky regarding representativeness ‘(it seemed small from a lay perspective (i remember at the time) for the conclusions being drawn and there seemed too little account taken of the different levels of unrest in different regions. my main point, though, was less based on my impression than on the fact that this technique exposed the authors to the criticisms/dismissal that the govt duly made, and they had little to counter those criticisms with, bar the defence that their methods were standard for those sort of surveys.)’ are also cause for concern because she seems to have not understood that this was a random sample.

“By picking random neighborhoods proportional to population, we are likely to account for the natural variability of ethnicity, income, and violence. Her words above strongly suggest that the Falluja numbers should be included, rather than being used to temper the results from the other 32 neighborhoods. Please understand how extremely conservative we were: we did a survey estimating that ~285,000 people have died due to the first 18 months of invasion and occupation and we reported it as at least ~100,000.

“Finally, there are now at least 8 independent estimates of the number or rate of deaths induced by the invasion of Iraq. The source most favored by the war proponents (Iraqbodycount.org) is the lowest. Our estimate is the third from highest. Four of the estimates place the death toll above 100,000. The studies measure different things. Some are surveys, some are based on surveillance which is always incomplete in times of war. The three lowest estimates are surveillance based.

“The key issues are supported by all the estimates that attribute deaths to the various causes: violence is way up post-invasion and the Coalition is responsible for many times more deaths than are the insurgents. The exact number is less important that these two indisputable facts which helps us to understand why things are going badly and how to fix them.
I hope these thoughts are helpful.
Sincerely,
Les Roberts”

Perhaps most damning in Roberts’ reply -- in light of media criticism of the Lancet’s alleged exaggeration of civilian deaths -- was his refutation of the claim that the uneven levels of violent unrest in Iraq compromised the accuracy of the figures. In fact the study not only accounted for this variability, it erred on the side of caution by excluding data from Fallujah where deaths were unusually high. Moreover, other violent hotspots -- such as Ramadi, Tallafar and Najaf -- were all passed over in the sample by random chance. This suggests that the actual total of civilian deaths is likely to be higher than 100,000. Indeed, it would make far more sense for the media to be criticizing the report authors for under-estimating the number of deaths.

We wrote to Dejevsky asking if she had received Roberts’ response. She replied on September 1:

“yes, and i understand the arguments. but i stick to my position that extrapolation, however scientific and well-thought through is no substitute for real figures. i know that the 'real' figures here do not exist, but i still think that extrapolation has obvious drawbacks which lay the resulting figures open to question - and therefore vulnerable to govt spokesmen who seek to discredit them. incidentally, my view on extrapolation is really neither here nor there. my chief objection to it is, as i have just said, that it lays the figures themselves open to question by those who have an interest in discrediting them.
all the best, mary”

Edward Herman, co-author with Noam Chomsky of the classic media study, Manufacturing Consent, commented on this latest response:

“Massive incompetence in support of a war-apologetic agenda. Dejevsky objects to the figures because they are vulnerable to discrediting for reasons that make no sense. I wonder if she finds sampling discreditable in all cases.” (Email to Media Lens, September 1, 2005)

This is something we were keen to find out by examining media responses to other cases of sampling (see below and Part 2).

The Puzzled Epidemiologist

It is understandable that Roberts was puzzled by Kirby’s and Dejevsky‘s responses. After all, in 2000 Roberts began the first of three surveys in Congo for the International Rescue Committee (IRC) in which he used methods akin to those of the Iraq study. Roberts' first survey estimated that an astonishing 1.7 million people had died in Congo over 22 months of armed conflict -- on average 2,600 people were dying every day. The IRC's president, Reynold Levy, put the figures in perspective:

“It's as if the entire population of Houston was wiped off the face of the Earth in a matter of months.” (Hrvoje Hranjski and Victoria Brittain, ‘2,600 a day dying in Congolese war,’ The Guardian, June 10, 2000)

As Roberts says, the reaction could not have been more different:

“Tony Blair and Colin Powell quoted those results time and time again without any question as to the precision or validity.” (Quoted, Lila Guterman, ‘Researchers Who Rushed Into Print a Study of Iraqi Civilian Deaths Now Wonder Why It Was Ignored,’ The Chronicle Of Higher Education, January 27, 2005; http://chronicle.com/free/2005/01/2005012701n.htm)

Indeed, within a month of Roberts’ IRC report being published, the UN Security Council passed a resolution that all foreign armies must leave Congo, and later that year, the United Nations called for $140 million in aid to the country, more than doubling its previous annual request. Citing the study, the US State Department announced an additional $10 million for emergency programs in Congo.

In his October 2001 speech to the Labour party conference, Tony Blair said the international community could resolve many of the world’s worst conflicts:

“It could, with our help, sort out the blight that is the continuing conflict in the Democratic Republic of the Congo, where three million people have died through war or famine in the last decade.” (‘Part one of the speech by prime minister, Tony Blair, at the Labour Party conference,’ The Guardian, October 2, 2001)

The three million figure was produced by Roberts’ study using essentially the same methodology employed in Iraq. And yet, in rejecting the Lancet report out of hand, Blair told parliament:

“Figures from the Iraqi Ministry of Health, which are a survey from the hospitals there, are in our view the most accurate survey there is.” (David Hughes, ‘No inquiry into Iraq death toll, says Blair,’ Daily Mail, December 9, 2004)

Foreign secretary Jack Straw said the Government would examine the Lancet figures "with very great care," adding, “it is, however, an estimate that is not based on standard methodology for assessing casualties.” (‘This week’s big issues: New attack on Blair’s Iraq policy,’ The Independent, December 5, 2004)

Like so much that Straw says, this was simply untrue.

Blair's press spokesman said the government had a number of “concerns and difficulties” about the methodology used, Patrick Wintour and Richard Norton-Taylor reported in the Guardian:

“‘The findings were based on extrapolation and treating Iraq as if it were all the same in terms of the level of the conflict,’ he said of the study published in the Lancet. ‘This is not the case.’” (Patrick Wintour and Richard Norton-Taylor, ‘No 10 challenges civilian death toll,’ The Guardian, October 30, 2004)

Then, by way of a classic example of media propaganda, Wintour and Norton-Taylor presented the government‘s concocted ‘controversy’ as genuine:

“The controversy about the study largely turns on whether the sample size of 7,800 people used by the team of US and Iraqi academics was sufficiently large, and whether the 33 neighborhoods chosen were representative of the rest of the country.”

This, again, was false. In reality, there was and is no real controversy about the size of the sample among scientists and serious commentators. Michael J. Toole, head of the Center for International Health at the Burnet Institute, an Australian research organization, said:

“That's a classical sample size.” Researchers typically conduct surveys in 30 neighborhoods, so the Iraq study's total of 33 strengthens its conclusions. “I just don't see any evidence of significant exaggeration,” Toole added. (Cited, Guterman, op. cit)

David R. Meddings, a medical officer with the Department of Injuries and Violence Prevention at the World Health Organization, said surveys of this kind always have uncertainty because of sampling and the possibility that people gave incorrect information about deaths in their households. However, Meddings added:

“I don't think the authors ignored that or understated. Those cautions I don't believe should be applied any more or any less stringently to a study that looks at a politically sensitive conflict than to a study that looks at a pill for heart disease.” (Ibid)

The Independent helped fuel the myth of a controversially small sample:

“The Lancet said the research was based on a sample of fewer than 1,000 Iraqi households but said the findings were convincing.” (Colin Brown, ‘Blair petitioned to set up inquiry into Iraqi war dead,’ The Independent, December 8, 2004)

The media also made much of a comment printed in the Washington Post by Marc E. Garlasco, a senior military analyst at Human Rights Watch, who said of Roberts' figures: “These numbers seem to be inflated.” (Guterman, op. cit)

This was reported in the British media. Unreported anywhere, as far as we can tell, is the fact that Garlasco has since admitted that he had not read the Lancet paper at the time and calls his quote in the Post “really unfortunate.” Garlasco says he told the reporter:

“I haven't read it. I haven't seen it. I don't know anything about it, so I shouldn't comment on it.” But “like any good journalist, he got me to.” (Ibid)

The large gap between the Lancet estimate and that of Iraq Body Count -- a constant feature of press coverage -- is also not controversial. John Sloboda, a professor of psychology at the University of Keele, and a co-founder of Iraq Body Count, says his team's efforts will inevitably lead to a count smaller than the actual figure because not every death is reported in the news media.

Dr. Woodruff said, “Les [Roberts] has the most valid estimate.” (Ibid)

Dr. Toole agreed: "If anything, the deaths may have been higher [than the Lancet study's estimate] because what they are unable to do is survey families where everyone has died." (Ibid)

Journalists, however, know better. Roger Alton, editor of the Observer gave us his view of the Lancet report:

“I find the methodology a bit doubtful...” (Email to Media Lens, November 1, 2004)

David Aaronovitch, then of the Guardian, told us:

“I have a feeling (and I could be wrong) that the report may be a dud.” (Email to Media Lens, October 30, 2004)

Perhaps Aaronovitch’s “feeling” is a close relation of Dejevsky’s when she writes “I just feel” the “extrapolation technique” is unsuited to a situation as “uneven” as Iraq.

Part 2, comparing media responses to Roberts’ work on Congo and Iraq, will follow shortly.

Media Lens is a UK-based media watchdog group headed by David Edwards and David Cromwell. Visit the Media Lens website (www.medialens.org) and consider supporting their invaluable work (www.medialens.org/donate.html).