National Review
June 17, 2003

Domestic Disputes
Bad social science and bad legal policy.
By Eugene Volokh

http://www.nationalreview.com/comment/comment-volokh061703.asp  

If you keep a gun in your home," a University of Pennsylvania press release said last week, "you dramatically increase the odds that you will die of a gunshot wound, according to research published in the June issue of the Annals of Emergency Medicine." "Keeping guns at home is dangerous," the researcher was quoted as saying. A New York Times story likewise reported: "People with guns in their homes are almost twice as likely to be killed by guns as people who do not keep them at home." Another article echoed the same point; so do the websites of advocacy groups (here and here).

Frightening results for people who own guns, or who are thinking of buying them. Frightening but, it turns out, meaningless — another example of how unsound social science is being used in public-policy debates.

What the University of Pennsylvania study found was a statistical correlation: Gun ownership is correlated with gun deaths. But that two things are correlated doesn't prove that one causes the other. The sex-crime rate is correlated over time with the use of air conditioning, but not because air conditioning causes sex crime; rather, both rise during the summer months. Likewise, whether someone in your home has been to the hospital recently is correlated with death in your home, but not because hospital care tends to kill people (though sometimes it does). Rather, both hospital stays and deaths often have a common cause: serious illness.

It turns out that a hugely disproportionate fraction of homicide victims are themselves criminals — criminals do dangerous things, and deal with dangerous people. In a recent San Francisco study, two-thirds of all gun-homicide victims (and one-third of all gun suicides) were found to have had arrest records, and other studies of gun-homicide victims yield similar results. And criminals, especially drug dealers and gang members, are particularly likely to own guns; most gun owners aren't criminals, but many criminals are gun owners. So even if gun ownership and gun homicide are correlated, both may be caused by a common factor: Hardcore criminals are especially likely to own guns — and to be killed by guns.

Social scientists have long recognized that correlation doesn't prove causation, because the two correlated things may actually be caused by a third thing. That's why sound social science requires that scientists use statistical tools that control for a wide variety of such "confounding factors."

This particular study in fact tried to control for various factors. Before controlling for those factors, the study actually found that gun-homicide victims were slightly less likely to own guns than the nonvictims were. The correlation between gun ownership and gun homicide arose when the study controlled for sex, race, age, marital status, education, income, veteran status, region, population of area of residence, and whether the person lived alone.

The study, however, completely failed to control for what might well be the most important factors: whether the household contained violent criminals, gang members, drug dealers, and the like. These are the very factors that might cause both gun ownership and gun death. And because the study didn't control for them, it says nothing about whether gun ownership really "increases the odds" that a law-abiding citizen will be killed. The study's results could easily flow simply from the huge set of homicide victims who are themselves criminals.

Curiously, the study starts by pointing out that an earlier study, which used a similar approach, "inspired a paper trail of challenges that continues to grow." The new study was partly a reaction to those challenges. But a key argument in those challenges was precisely that the earlier study didn't adequately control for the victim's criminal background — and yet the new study made the same mistake, and in fact control for fewer factors than the earlier study did. The new study didn't even try to deal with the possibility that law-abiding citizens aren't endangered by their own guns, but rather that gun-owning criminals are jeopardized by their gun-owning criminal associates or enemies.

To its credit, the University of Pennsylvania study does mention this potential problem (though the press release and the New York Times article don't). "The results could be biased," the study says, "because risky behaviors like drug dealing could not be controlled with the data used here." But, the study argues, this confounding "is not likely to explain the results completely" — if the victim's own criminal background caused both the gun ownership and the victim's death, "a gun in the home should be associated . . . with homicide by all means," and not just guns. This, though, is unpersuasive: Drug dealers or gang members may be especially likely to kill each other precisely with guns, and not knives, clubs, or bare hands.

Likewise, the study finds a correlation between gun ownership and suicide — but doesn't control for the obvious possibility that some people bought the guns precisely because they're suicidal. The study points to another study that supposedly shows that gun ownership is correlated with suicide even when the gun had been owned for years (though much less strongly than when the gun has been owned for only a short time). But that other study is itself similarly flawed: It controls only for age, sex, and geography, and ignores other important factors such as history of involvement with drugs and with drug dealers, a history of chronic depression, or other personality or environmental traits.

Unfortunately, this is how conventional wisdom is molded. A badly flawed study leads to an even more flawed New York Times article. Readers read it and say "Wow, it's dangerous for me to own a gun" — or "Since guns endanger even their owners, there's really no reason to keep them legal." Precisely because the study seems so authoritative, so scientific, it's likely to be influential, even when it's misdesigned and misreported. And this is especially so when these flaws are repeated in study after study, as they have routinely been in the gun debate. Bad social science leads to bad legal policy.

Finally, the study concludes with a recommendation to the medical profession: Physicians should "discuss with all patients" "the consequences of having access to guns." But "discussions" are only helpful if the physician actually knows what he's talking about. Many physicians have little personal knowledge about guns, and haven't read the rebuttals to these studies. If they start spreading this erroneous information to their patients, the results won't be good either for the patients or for the reputation of the medical profession.

Eugene Volokh teaches First Amendment law at UCLA School of Law.