UPDATE: Angus-Reid Gun Poll

 

I have to admit, I missed this little gem in the fine print when I was doing my once-over of the study (the one about 85% believing in the individual right to bear arms) before RF posted the results here on TTAG.

Methodology: From January 17 to January 18, 2012, Angus Reid Public Opinion conducted an online survey among 1,009 American adults who are Springboard America panelists. The margin of error—which measures sampling variability—is +/- 3.1%. The results have been statistically weighted according to the most current education, age, gender and region Census data to ensure a sample representative of the entire adult population of the United States. Discrepancies in or between totals are due to rounding.

Show of hands: who sees the issue in there?

There are a couple basic assumptions made in any statistical analysis. The first and most important is that the sample of people being polled is randomly selected from within the population.

For national polls such as CNN’s polls, they use random number generating software and call people all over the country, making sure to evenly represent the population of the US. That even random distribution means that the results they get will probably be a good indication of how the nation actually feels.

For this study, they relied on members of a website called Springboard America to provide responses. What is Springboard America? From their FAQ:

Springboard America is an online platform designed to give Americans like you the chance to express your opinions – and to see them in action!

By joining Springboard you become eligible to receive invitations to take part in research on topics ranging from politics to consumer products. For each survey you take part in, your answers are collected along with those of fellow Springboard members and the results analyzed by our research experts before being presented to our clients.

We partner with organizations ranging from household brands, government bodies, not-for-profit organizations and the media who want to better understand public opinion and inform their decisions. Sometimes you’ll also see our polls reported in the media, helping to shape the debate on current issues.

As a thank you for sharing your opinions with us, you will receive Survey Dollars for taking part, or sweepstake entries. We’ll also share key insights with you via our newsletters and portal, and you give us instant feedback in our quick polls.

In short, it’s a site where people sign up to take polls and give their opinions.

The issue with this methodology (using the website to administer the poll) is that instead of the study being representative of the people in the United States, the study is only truly representative of the population of Springboard America users.

Using any website’s readers as the basis for a poll is always a tricky situation, as they constitute a self selecting population. There’s a reason they came to that website and they’re actively pushing their agenda instead of being passively asked like in the CNN polls. This has a tendency to skew the results and basically invalidate the applicability of those results to any population other than of that specific website.

As an illustrative example, imagine if we took a poll of TTAG readers and asked them about gun rights. We would get enough responses to be statistically within a 90% confidence interval for applying those results to the national population, but the actual results would look nothing like the true feelings of the US population in general. A self selecting group of individuals does not a good sample make.

In short, I’d take this poll with a rather large dosage of NaCl if not outright ignore it. Either way, the question about semi-automatic weapons (a.k.a., “assault rifles”) drew my attention . . .

Below is a list of weapons. For each one, please say whether this type of weapon should be available to every American who is eligible to own firearms, or only to the police and other authorized persons?

As TTAG commentator Ropingdown pointed out underneath our original post . . .

The presentation itself would lead the respondent to think that ‘semi-automatics’ meant something other than pistols, rifles, and shotguns, since four classifications are offered and the first three are obviously non-overlapping categories. That interpretation is supported by the fact that so few gun-owners apparently thought semi-automatic was fine. It’s a meaningless statistic.

[Note: We now know that shotguns and rifles were combined into a single category. But the point remains valid: semi-automatic firearms includes most handguns and many rifles. The poll does not clearly or correctly identify the types of guns about which they’re inquiring.]

Heads-up to AngusReid: if you want to conduct a valid survey of American gun owners, TTAG and its Armed Intelligentsia are ready to help.

[h/t: RS14-2]