[This guest post is written by Swordsmanus, printed here with the author’s permission. If you’re on Reddit, go give him an upvote or two.]
The conclusions of the study Risks and Benefit of a Gun in the House are held up as fact by anti-gun advocates. It is published by Harvard, after all. But there are several problems with how it goes about coming to those conclusions. I read the study and noted its citations wherever things did not add up. I then looked up the citations and cross referenced them with readily available information. My analysis is broken down by section within Dr. Hemenway’s study, so you can go back and forth between the two if you’d like.
Introduction
There’s some good background on gun ownership and demographics in the US. The problem is the blown-up quote featured on page one. An English, Communication, or Political Science major could have a lot of fun showing how the ostensibly even-handed quote is actually very biased, but I won’t get into it here.
Risks
Accidents
The data on accidental gun deaths presented in the study matches up completely with data from the Center for Disease Control and Prevention (CDC). However it omits the fact that accidental gun deaths per year have dropped steadily and significantly in the past 10 years, despite gun ownership going up and concealed and open carry laws becoming more lax across the nation.
In 2007 there were 613 accidental gun deaths, and again it’s part of a steady downward trend. For comparison, there were 29,846 accidental deaths by poisoning in 2007 according to the CDC [1].
So while the data presented was sound, it left out powerful yet easily-accessible context — the kind that any researcher on this subject should use as a starting point. But David Hemenway isn’t just any researcher. He’s aProfessor of Health Policy at the Harvard School of Public Health. So why leave it out?
Benefits
Deterrence
Dr. Hemenway points out that there’s no clear evidence that gun ownership deters crime.
It’s true, guns do not clearly deter crimes from being committed when viewed in the aggregate. The crime data from the FBI Bureau of Justice Statistics going from 1960-2009 supports that [2]. Restrictive gun legislation and lax gun laws have no clear effect, either [3].
There have been studies claiming otherwise both ways, but they have consistently only studied relatively short periods of time and relatively small sample sizes compared to a 50-year span for every state in the nation.
Thwarting Crimes
This is the real offender in the study. Up front, Dr. Hemenway acknowledges that aggregate data on gun use in self-defense is unreliable, while implying that some of those who are included in measurements of defensive gun use are actually criminals.
He then breaks down the sources of self-defense data: (a) police reports, (b) randomly sampled surveys that ask directly about self-defense gun use, and (c) surveys that ask about self-defense gun use only after respondents report that someone attempted to commit a crime against them (aka, NCVS Surveys).
(a) Police Reports: In this section, Dr. Hemenway implies that guns are almost never useful for self-defense from a home invasion. He cites only the Atlanta study taken from citation #12…A study done on one city, covering a 4-month period. There are tens of thousands of other cities to draw data from, across much greater spans of time than 4 months. There are decades of data to draw upon and view as a coherent whole!
That Dr. Hemenway chose just this single study, for a single city, going over just a 4-month period instead of the course of years, instead of aggregating available data across the nation, says loud and clear that citation #12 was cherry picked to suit his conclusions.
I really can’t overstate how much of a methodological and ethical issue this is, given that he uses this citation as though it accurately represents data on home invasions in the US, and uses it to support the study’s later conclusions against owning firearms.
(b) random sample surveys This section used citation #91 to support its claims, when that study merely gave inconclusive conclusions [4]. Dr. Hemenway also claims, “police report more total self-defense gun uses than all civilians combined“, yet citation #92 has the exact opposite data, and is from a study 5 years newer. The JustFacts link I cited earlier uses the same study, listing it as citation [17]. The relevant data is quoted there.
(c) NCVS Surveys This section opens with, “The National Crime Victimization Surveys (NCVS) obtain information about self-defense gun use only from those respondents who first report that a crime against them was threatened, attempted, or completed. This feature of the NCVS substantially reduces the problem of reporting incidents that were not true self-defense gun uses.”
There’s a critical flaw here. For the reporting period used in the study, the NCVS surveys are at least as likely to under-report defensive gun use as random sample surveys are to over-report them.
Why? The NCVS survey period covered in Dr. Hemenway’s study was from 1992 to 2001. Until 1996, in the majority of states, concealed carry was highly restricted or illegal rather than “shall issue” or unrestricted, and even then it wasn’t until 2002-2003 when 66% of states became “shall issue” states [5].
![[5]](http://en.wikipedia.org/wiki/File:Rtc.gif){kind=link}
Thus, for the majority of the reporting period selected for this study, in most states it was illegal to have a handgun on your person outside the home. For the entire reporting period, a significant number of states had the same issue. And most of the NCVS crimes surveyed were for crimes outside the home:
“…Interviewers identify themselves to respondents as federal government employees, even displaying, in face-to-face contacts, an identification card with a badge. Respondents are told that the interviews are being conducted on behalf of the U.S. Department of Justice, the law enforcement branch of the federal government.”
“As a preliminary to asking questions about crime victimization experiences, interviewers establish the address, telephone number, and full names of all occupants, age twelve and over, in each household they contact.”
“…it is made very clear to respondents that they are, in effect, speaking to a law enforcement arm of the federal government, whose employees know exactly who the respondents and their family members are, where they live, and how they can be recontacted.”
“It is not hard for gun-using victims interviewed in the NCVS to withhold information about their use of a gun, especially since they are never directly asked whether they used a gun for self-protection…”
“…88% of the violent crimes which respondents reported to NCVS interviewers in 1992 were committed away from the victim’s home…in a location where it would ordinarily be a crime for the victim to even possess a gun, never mind use it defensively.”
“Because the question about location is asked before the self-protection questions, the typical violent crime victim respondent has already committed himself to having been victimized in a public place before being asked what he or she did for self-protection…respondents usually could not mention their defensive use of a gun without, in effect, confessing to a crime to a federal government employee.” [6]
There are known issues with the NCVS methodology for the period used, and yet its data is still used by Dr. Hemenway to strongly support his position. He wrote “Survey Research And Self-Defense Gun Use: An Explanation Of Extreme Overestimates” in 1997. He is no stranger to evaluating survey methods [7]. He should know well that survey inaccuracy can go both ways due to methodology, not just one way.
Overall the “Thwarting Crimes” section in the study has insufficient strength of citations used to support it vs. other available data/studies refuting it, including citations used within the study, like citations #12 and #92.
Shootings in the Home
Remember the part where Dr. Hemenway acknowledged that aggregate data on gun use in self-defense is unreliable?
In this part, he claims, “Home guns were 4 times more likely to be involved in an accident, 7 times more likely to be used in a criminal assault or homicide, and 11 times more likely to be used in an attempted or completed suicide than to be used to injure or kill in self-defense“.
Under “Thwarting Crime”, he says that self-defense data is unreliable. But elsewhere he turns around and implies it really is reliable, but only when the data sampled casts gun use in a bad light. An inconsistency like this is inexcusable, given the author’s credentials and experience.
Conclusion
Dr. Hemenway states, “…for those households where having a gun or not will matter this year, the evidence indicates that the costs will widely outweigh the benefits…”
I hope it’s clear by now that in this study, Dr. Hemenway started with the above conclusion and then cherry picked his data in order to support it.
What a researcher with integrity does is gather data and form conclusions based only on the most robust, reliable, and consistent data available, omitting or acknowledging the limitations of weaker data and leaving it out of one’s conclusions, especially when stronger data contradicts it. To do otherwise is to sully the reputation of all that you and your work are associated with.