In addition to several research papers, articles (including
those published in Nature
and Science
journals) and some peer-reviewed studies - all critical of
the Lancet 2006 study on Iraqi deaths, the following letters
of criticism were published in the Lancet journal, 2007.
The Lancet 2007; 369:102 - Correspondence
http://www.thelancet.com/journals/lancet/article/PIIS0140673607600610/
Mortality in Iraq
Debarati Guha-Sapir (a), Olivier Degomme (a) and Jon Pedersen
(b)
Gilbert Burnham and colleagues' Iraq mortality study(1)
fills an important information gap in a country where reliable
mortality statistics are rare. It transforms anecdotes of
violence into systematic evidence. However, the paper could
have addressed some methodological issues which might have
strengthened the credibility of the estimates.
First, according to Burnham and colleagues' results, there
were nearly 600 war deaths per day—an unusually high
number compared with almost any other armed conflict or
indeed with other Iraqi mortality estimates.(2) Burnham
and colleagues' figure 4, in which cumulated Iraq Body Count
deaths parallel their study's mortality rates, is misleading.
Rates cannot be compared with numbers, much less with cumulative
numbers. The correct comparison would be the one presented
here (figure), in which the Iraq Body Count numbers are
transformed into rates by period. In that case, there is
no similarity between the trends in the study and Iraq Body
Count.
Second, the study suggests that, over a 3-year period,
around 90% of the deaths were directly related to violence.
However, experience from other conflicts indicates that
indirect causes (disease, malnutrition) typically outnumber
the deaths due to violence (bombs, gunshots, etc).(3) Burnham
and colleagues' figure remained high for a long period of
time. By comparison, only one of 17 surveys in Darfur reported
a similar level of violent deaths, and this level only persisted
for 3 months of a 6-month period.(4)
Third, the heterogeneity of the pattern of violence in
Iraq argues for a differentiated estimation across the governorates.
Insurgency and coalition action is still concentrated mainly
in the Sunni triangle, but large tracts in the rest of the
country are relatively peaceful. A better accounting for
differences in violence by governorate separately and the
effect of excluding the Sunni triangle would have strengthened
the study.
We declare that we have no conflict of interest.
References
1. Burnham G, Lafta R, Doocy S, Roberts
L. Mortality after the 2003 invasion of Iraq: a crosssectional
cluster sample survey. Lancet 2006; 368: 1421-1428.
2. UN Development Programme. Iraq living
conditions survey 2004
http://www.iq.undp.org/ILCS/overview.htm
(accessed Oct 13, 2006)..
3. Coghlan B, Brennan RJ, Ngoy P, et
al. Mortality in the Democratic Republic of Congo: a nationwide
survey. Lancet 2006; 367: 44-51.
4. Guha-Sapir D, Degomme O. Darfur:
counting the deaths. Mortality estimates from multiple survey
data. Brussels: Center for Research on the Epidemiology
of Disasters, 2006.
The Lancet 2007; 369:101-102 - Correspondence
http://www.thelancet.com/journals/lancet/article/PIIS0140673607600609/
Mortality in Iraq
Madelyn Hsiao-Rei Hicks (a)
Crucial weaknesses exist in Gilbert Burnham and colleagues'
study of Iraq's war-related mortality.(1)
First, 47 clusters seem to be too few for a large population
experiencing highly localised violent events.
Second, household sampling within clusters was not random:
only households located on or near residential streets crossing
a main street had a chance of inclusion,(2) and only if
located near the “start household” for that
cluster.
Third, it is infeasible that “One team could typically
complete a cluster of 40 households in 1 day”. Assuming
continuous interviewing for 10 h despite 55°C heat,(3)
this allows 15 min per interview including walking between
households and obtaining informed consent and death certificates.
The improbability of so many interviews being done so quickly
and reliance on “word of mouth among households”
during selection and recruitment suggest potential sources
of bias, ethical compromise, and risk to interviewees during
interview-gathering.(4)
Iraq's suffering from war is properly reflected not by
producing high-mortality findings, but by producing accurate
mortality findings. The Iraq Living Conditions Survey(5)
provided such an example. In this study, ten randomly sampled
households were interviewed per cluster in 2200 clusters
across all governorates of Iraq to provide an estimate of
conflict-related deaths within the same difficult field
conditions.
I declare that I have no conflict of interest.
References
1. Burnham G, Lafta R, Doocy S, Roberts
L. Mortality after the 2003 invasion of Iraq: a cross-sectional
cluster sample survey. Lancet 2006; 368: 1421-1428.
2. Johnson NF, Spagat M, Gowley S, Onnela
J, Reinert G. Bias in epidemiological studies of conflict
mortality. http://www.rhul.ac.uk/economics/Research/conflict-analy...
(accessed Dec 19, 2006).
3. Burnham G, Doocy S, Dzeng E, Lafta
R, Roberts L. The human cost of the war in Iraq: a mortality
study, 2002-2006. John Hopkins Bloomberg School of Public
Health and Al Mustansiriya University School of Medicine
http://web.mit.edu/CIS/pdf/Human_Cost_of_War.pdf
(accessed Oct 23, 2006).
4. Hicks MH. Mortality after the 2003
invasion of Iraq: were valid and ethical field methods used
in this survey? http://www.hicn.org/research_design/rdn3.pdf
(accessed Dec 19, 2006).
5. UN Development Programme. Iraq living
conditions survey 2004
http://www.iq.undp.org/ILCS/overview.htm (accessed Oct 23,
2006).
The Lancet 2007; 369:101 - Correspondence
http://www.thelancet.com/journals/lancet/article/PIIS0140673607600592/
Mortality in Iraq
Prabhat Jha (a), Vendhan Gajalakshmi (b), Neeraj Dhingra
(c) and Binu Jacob (a)
Gilbert Burnham and colleagues(1) do a commendable study
of mortality in Iraq in difficult circumstances. Our concerns
are two: the reasonably small number of clusters, which
might generate random errors, and selective biases if households
over-reported mortality during the conflict period. The
survey work was done by physicians, and it might well be
that households reported mortality in homes other than their
own.
To address possible biases, Burnham and colleagues might
wish to report three specifics: (a) were the proportions
of households who could produce a death certificate similar
during the pre-conflict and conflict periods (and did the
survey team have any way of assessing whether identifier
information on the death certificates matched household
details)? (b) was there any specific digit or date preference
pattern in the deaths reported in the post-conflict period
that might suggest false reporting? and (c) was there any
difference in the death rates for the first, middle, and
last thirds of the sampling period? (if households wanted
to over-report mortality, news of the survey would have
spread to other areas only after the survey began).
Similarly, as an additional validity check on rates, they
might apply “capture-recapture” methods to their
earlier study(2) and their current study in areas that were
in common in those sampled areas for the pre-conflict period.
A general weakness of the method was the lack of resampling
by independent teams. Our large-scale mortality studies
in India(3–5) find that repeat survey of at least
5–10% provides far more stable cause-specific mortality
rates than do single surveys.
We declare that we have no conflict of interest.
References
1. Burnham G, Lafta R, Doocy S, Roberts
L. Mortality after the 2003 invasion of Iraq: a cross-sectional
cluster sample survey. Lancet 2006; 368: 1421-1428.
2. Roberts L, Lafta R, Garfield R, Khudhairi
J, Burnham G. Mortality before and after the 2003 invasion
of Iraq: cluster sample survey. Lancet 2004; 364: 1857-1864.
3. Registrar General of India. Special
Fertility and Mortality Study, 1998: A report of 1·1
million households. New Delhi: Registrar General, 2005:.
4. Jha P, Gajalakshmi V, Gupta PC, et
alfor the RGI-CGHR Prospective Study Collaborators. Prospective
study of one million deaths in India: rationale, design,
and validation results. PLoS Med 2006; 3: e18.
5. Gajalakshmi V, Peto R, Kanaka S,
Jha P. Smoking and mortality from tuberculosis and other
diseases in India: retrospective study of 43000 adult male
deaths and 35000 controls. Lancet 2003; 362: 507-515.
The Lancet 2007; 369:101 - Correspondence
http://www.thelancet.com/journals/lancet/article/PIIS0140673607600580
Mortality in Iraq
Johan von Schreeb (a), Hans Rosling (a) and Richard Garfield
(b)
The uncertainty of estimates from retrospective mortality
surveys in humanitarian emergencies is composed of both
sampling and reporting errors. Gilbert Burnham and colleagues,
in their mortality study in Iraq (Oct 21, p 1421),(1) quantify
the sampling error, but the security situation did not allow
for the supervision and repeat interviews needed to estimate
reporting errors.
Over-reporting of deaths was regarded as limited because
92% of reported deaths were supported by death certificates,
but Burnham and colleagues do not report who issued these
certificates. Neither do they discuss why the availability
of death certificates increased from 81% in 2004.(2)
The existence of a substantial reporting error is supported
by the finding of low child mortality. The study population
only reported 54 non-violent deaths in those younger than
15 years, and 1474 births—ie, an under-15 mortality
of 36 per 1000 births. This is a third of the estimated
preinvasion under-5 mortality.(3) Since nothing indicates
that child mortality has decreased,(4) the results suggest
that fewer than half of child deaths were reported.
Without an explanation for the high availability of death
certificates, one could assume that the reporting error
is of the same size as the sampling error (±30%).
This assumption still yields at least a five-fold higher
number of violent deaths than the passive surveillance mortality
numbers.(5) If the death certificates are valid and the
availability above 90%, it seems better to monitor mortality
by compiling data from the local agencies that issue these
certificates than by doing further dangerous household surveys.
We declare that we have no conflict of interest.
References
1. Burnham G, Lafta R, Doocy D, Roberts
L. Mortality after the 2003 invasion of Iraq: across-sectional
cluster sample survey. Lancet 2006; 368: 1421-1428.
2. Roberts L, Lafta R, Garfield R, Khudhairi
J, Burnham G. Mortality before and after the 2003 invasion
of Iraq: cluster sample survey. Lancet 2004; 364: 1857-1864.
3. Ali MM, Blacker J, Jones G. Annual
mortality rates and excess deaths of children under five
in Iraq, 1991-98. Popul Stud 2003; 57: 217-226.
4. UNICEF. The State of the world's
children 2007. New York: United Nations Children's Fund,
2006.
5. Iraq Body Count http://www.iraqbodycount.net/
(accessed Dec 18, 2006).
The Lancet 2007; 369:102-103 - Correspondence
http://www.thelancet.com/journals/lancet/article/PIIS0140673607600622/
Mortality in Iraq
Josh Dougherty
Gilbert Burnham and colleagues state in their latest Iraq
mortality study(1) that the US Department of Defense (DoD)
has published civilian death estimates and that these corroborate
their findings. Burnham and colleagues are mistaken in these
assertions.
The claimed corroboration is illustrated by their figure
4, which compares trends in their data with those from the
DoD and truncated data from Iraq Body Count. The original
DoD data seem to be sourced from a graph on page 32 of the
Aug 29, 2006, “Measuring stability and security in
Iraq” report(2) published by the DoD. However, Burnham
and colleagues' assertion that the DoD “estimated
the civilian casualty rate at 117 deaths per day”
is mistaken, as is their figure 4, which repeats this error
in graphic form.
These data refer to Iraqi civilians and security-force
personnel, not just to civilians, and to casualties (ie,
deaths or injuries), not just deaths. The DoD numbers do
not refer to Iraqi “deaths per day” and do not
offer any direct means by which to calculate what number
might be deaths, let alone civilian deaths. What is clear,
however, is that the number in the DoD data is unlikely
to be anywhere close to 117, as can be confirmed by a cursory
analysis of the blue “Coalition” columns included
alongside those for Iraqis in the DoD graph. These columns
show that non-Iraqi Coalition forces have suffered roughly
17000 “casualties” since January, 2004. The
current official total for all Coalition deaths since the
beginning of the conflict in March, 2003, stands at just
over 3000, or less than 20% of Burnham and colleagues' interpretation
of these figures.
I declare that I have no conflict of interest.
References
1. Burnham G, Lafta R, Doocy D, Roberts
L. Mortality after the 2003 invasion of Iraq: across-sectional
cluster sample survey. Lancet 2006; 368.
2. US Department of Defense. Measuring
stability and security in Iraq. Washington: US Department
of Defense, 2006
The Lancet 2007; 369:103-104 - Correspondence
http://www.thelancet.com/journals/lancet/article/PIIS0140673607600634/
Mortality in Iraq – Authors' reply
Gilbert Burnham (a), Riyadh Lafta (b), Shannon Doocy (a)
and Les Roberts (a)
Johan von Schreeb and colleagues point out that our interviewed
households were only visited once. This is true, but past
efforts at repeat interviewing in Iraq, Liberia, and Zaire
have yielded more deaths on a second interview.(1–3)
The deaths that were not confirmed by a certificate were
too few to do any meaningful trend analysis. The higher
proportion of death certificate confirmations in our second
study might reflect the re-establishment of issuance procedures
in recent years. In the 2004 data (but not in the 2006 data),
the period most associated with no death certificates was
the weeks following the invasion, and this period constituted
a smaller fraction of all deaths in the second study.
The observation that the non-violent death rate among those
younger than 15 years is lower than in 2004 (4·8
per 1000 per year) and strikingly lower than during a period
of sanctions a decade earlier is interesting but ignores
the fact that this rate is similar to those reported by
the UN for 2004 in Syria and Kuwait.(4) We openly acknowledge
in the Discussion section of the paper that under-reporting
of deaths might have occurred. In a stable and functioning
environment where death certificates are universally completed
and fully accessible, the method von Schreeb and colleagues
suggest would be an excellent check on under-reporting from
survey or census results.
Prabhat Jha and colleagues and Madelyn Hsiao-Rei Hicks
express concern about the number of clusters. The confidence
intervals presented in the paper were calculated with robust
standard errors that account for cluster sampling. Virtually
all clusters experienced violent deaths (figure) and if
only visiting 47 randomly selected areas missed significant
areas, this can only mean that our estimate is too low.
Our criteria for a household death included the decedent
residing with the interviewed families continuously for
the 3 months before their death. We suspect that this definition
would have excluded deaths rather than including deaths
from other households as Jha and colleagues suggest. The
death certificate documentation was high before and after
the invasion period and it is not plausible that people
would have fabricated death certificates after hearing about
our study because (a) most small cities and villages were
only visited for one day, and (b) the interviewing began
just minutes after the selection of a specific location.
In order for this mass falsification to happen, millions
of Iraqis would have needed to fabricate death certificates,
and their motive to do so is not clear.
In both this survey and the 2004 survey, the proportion
of the entire population surveyed was so small that a capture-recapture
analysis is not possible. We agree that revisiting households
with different interviewers would be ideal but security
concerns made this seem imprudent.
Although, as Ricks states, two governorates were not sampled,
rates calculated were applied against the population of
Iraq minus the population of these governorates, or 26·1
million.
If neighbourhoods selected also included residential streets
that did not cross main streets, these additional streets
were also included in the random sampling process. The results
of this current survey closely paralleled those from the
2004 survey which selected start houses by global positioning
system (GPS) methods, suggesting that random selection of
start households used in this study did not introduce a
measurable systematic bias. Selecting the house with the
nearest front door is a standard field method, and the author
supervising field work ensured this approach was consistently
followed. Sampling in each cluster involved two teams working
together. With this arrangement, sampling 40 households
in a day was indeed feasible.
The Iraq Living Conditions survey was carried out after
barely a year of conflict by government employees who did
interviews that took 82 min on average.(1) The survey included
2200 clusters to provide development information by governorate.
Only one question was asked about deaths, and one of the
senior researchers in that study has stated that he knows
his estimate was an underestimate.1 He knows this because
the baseline non-violent death rate measured was implausibly
low and because revisits to ask about mortality in those
younger than 5 years in a sample of the same houses after
survey completion found about 50% more deaths than initially
reported.
Josh Dougherty and Debarati Guha-Sapir and colleagues all
point out that figure 4 of our report mixes rates and counts,
creating a confusing image. We find this criticism valid
and accept this as an error on our part. Moreover, Dougherty
rightly points out that the data in the US Department of
Defense source were casualties, not deaths alone. We regret
this labelling error. But the graph presented by Guha-Sapir
and colleagues uses a scale that masks the fact that there
are roughly three times as many deaths reported by Iraq
Body Count in recent months than during the same post-invasion
months of 2003. We had wanted to show that the three sources
all similarly pointed to an escalating conflict, but neither
graph shows that well, and we regret the confusion that
this created.
Among the comments by Guha-Sapir and colleagues, we do
not see the inconsistency they describe. First, by looking
at mortality data from the Democratic Republic of Congo
collected more than a year after the main conflict was resolved
and the warring armies returned home, they conclude that
the ratio of indirect excess deaths to violent deaths seems
low in Iraq. We feel a better comparison would be to the
data collected during that war which showed that 1·8%
of the 19·9 million people in the eastern part of
the country died of violence in the first 33 months of the
conflict, a proportion similar to that measured in Iraq.(5)
It is believed that the population in Iraq is not as susceptible
to death from malnutrition and disease as that in Darfur.
Wars occurring in countries with widespread access to high-power
weaponry, such as Kosovo and Bosnia and where violence accounted
for most excess wartime deaths, are more fitting comparisons.
We declare that we have no conflict of interest.
References
1. UN Development Programme. Iraq living
conditions survey 2004
http://www.iq.undp.org/ILCS/overview.htm (accessed Nov 29,
2006).
2. Becker SR, Diop F, Thornton JN. Infant
and child mortality in two counties of Liberia: results
of a survey in 1988 and trends since 1984. Int J Epidemiol
1993; 22: S56-S63.
3. Taylor WR, Chahnazarian A, Weinman
J, et al. Mortality and use of health services surveys in
rural Zaire. Int J Epidemiol 1993; 22: S15-S19.
4. WHO. Life tables for WHO member states
http://www3.who.int/whosis/life/life_tables/life_tables.
(accessed Dec 18, 2006).
5. Roberts L, Belyakdoumi F, Hale C,
et al. Mortality in eastern Democratic Republic of Congo:
results from eleven mortality surveys. New York: International
Rescue Committee, 2001: http://www.theirc.org/resources/mortII_report.pdf
(accessed Nov 28, 2006).
Nature journal:
Death toll in Iraq: survey team takes on its critics
Jim Giles
Nature
446, 6-7 (1 March 2007) | doi:10.1038/446006a
Raw data should settle arguments over study methods.
It's not often that George W. Bush takes time out to attack
a scientific paper on the day that it's released. But then
few papers attract as much attention as the one that claimed
that more than half a million people, or 2.5% of the population,
had died in Iraq as a result of the 2003 invasion. Published
last October in the run-up to the US mid-term elections,
the interview-based survey attracted huge press interest
and controversy.
The media spotlight has moved on, but interest within the
scientific community has not. The paper has been dissected
online, graduate classes have been devoted to it and critiques
have appeared in the literature with more in press. So far,
the discussion has created more heat than light. Many of
the criticisms that dogged the study are unresolved. For
example, Nature has discovered that different authors give
conflicting accounts of exactly how the survey was carried
out. And although many researchers say the questions hanging
over the study are not substantial enough for it to be dismissed,
a vocal minority disagrees.
The controversy creates extra interest in the authors'
decision, made last week, to release the raw data behind
the study. Critics and supporters will finally have access
to information that may settle disputes.
On paper, the study seems simple enough. Eight interviewers
questioned more than 1,800 households throughout Iraq. After
comparing the mortality rate before and after the invasion,
and extrapolating to the total population, they concluded
that the conflict had caused 390,000–940,000 excess
deaths (G. Burnham, R. Lafta, S. Doocy and L. Roberts Lancet
368, 1421–1428; 2006). This estimate was much higher
than those based on media reports or Iraqi government data,
which put the death toll at tens of thousands, and the authors,
based at Johns Hopkins University in Baltimore, Maryland,
and Al Mustansiriya University in Baghdad, have found their
methods under intense scrutiny.
Much of the debate has centred on exactly how the survey
was run, and finding out exactly what happened in Iraq has
not been straightforward. The Johns Hopkins team, which
dealt with enquiries from other scientists and the media,
was not able to go to the country to supervise the interviews.
And accounts of the method given by the US researchers and
the Iraqi team do not always match up.
Several researchers, including Madelyn Hicks, a psychiatrist
at King's College London, recently published criticisms
of the study's methodology in The Lancet (369, 101–105;
2007). One key question is whether the interviews could
have been done in the time stated. The October paper implied
that the interviewers worked as two teams of four, each
conducting 40 interviews a day — a very high number
given the need to obtain consent and the sensitive nature
of the questions.
The US authors subsequently said that each team split into
two pairs, a workload that is "doable", says Paul
Spiegel, an epidemiologist at the United Nations High Commission
for Refugees in Geneva, who carried out similar surveys
in Kosovo and Ethiopia. After being asked by Nature whether
even this system allowed enough time, author Les Roberts
of Johns Hopkins said that the four individuals in a team
often worked independently. But an Iraqi researcher involved
in the data collection, who asked not to be named because
he fears that press attention could make him the target
of attacks, told Nature this never happened. Roberts later
said that he had been referring to the procedure used in
a 2004 mortality survey carried out in Iraq with the same
team (L. Roberts et al. Lancet 364, 1857–1864; 2004).
Other arguments focus on the potential for 'main-street
bias', first proposed by Michael Spagat, an expert in conflict
studies at Royal Holloway, University of London. In each
survey area, the interviewers selected a starting point
by randomly choosing a residential street that crossed the
main business street. Spagat says this method would have
left out residential streets that didn't cross the main
road and, as attacks such as car bombs usually take place
in busy areas, introduced a bias towards areas likely to
have suffered high casualties.
The Iraqi interviewer told Nature that in bigger towns
or neighbourhoods, rather than taking the main street, the
team picked a business street at random and chose a residential
street leading off that, so that peripheral parts of the
area would be included. But again, details are unclear.
Roberts and Gilbert Burnham, also at Johns Hopkins, say
local people were asked to identify pockets of homes away
from the centre; the Iraqi interviewer says the team never
worked with locals on this issue.
Many epidemiologists say such discrepancies are understandable
given that Roberts and Burnham could not directly oversee
the survey, and do not justify accusations that the process
was flawed. For those who disagree, access to the raw data
is essential. Although previously reluctant to release them,
Roberts and Burnham now say they are removing information
that could be used to identify interviewers or respondents
and will release the data within the next month to people
with appropriate "technical competence".
One researcher keen to see the numbers is Spagat. The 2004
survey used GPS coordinates instead of the main-street system
to identify streets to sample, and when Spagat used the
limited data available so far to compare the two studies
for the period immediately following the invasion, he found
that the 2006 study turned up twice as many violent deaths,
suggesting that main-street bias may be present.
Roberts and others question Spagat's methods. But the issue
could be checked using the raw data. If main-street bias
exists, says Spagat, then death rates will fall as the interviews
move away from the main street.
The raw data may also help address a fear that some researchers
are expressing off the record: that the Iraqi interviewers
might have inflated their results for political reasons.
That could show up in unusual patterns within the data.
Roberts and Burnham say they have complete confidence in
the Iraqi interviewers, after working with them directly
for the 2004 study. And supporters say that criticisms should
not detract from the fact that the Iraqi team managed to
produce a survey under extremely difficult circumstances.
Security threats forced the team to change travel plans
and at one point to consider cancelling the survey altogether.
Since its completion, one interviewer has been killed and
another has left Baghdad, although it is not known whether
either case is linked to their involvement in the survey.
Either way, the continuing violence in the country is enough
for the remaining interviewers to say that they are not
willing to repeat the exercise.
