AMA Poll on Spring Break is Faulty and Deceptive “Media Advocacy Tool”

The American Medical Association (AMA) recently reported on a poll that it commissioned, supposedly of American female college students, in which its president said that spring break “has turned into a dangerous binge-fest.” The AMA said the poll demonstrates the need to restrict alcohol ads on campus and other policy changes it advocates.

The story was carried in thousands of newspapers and websites . It was reported by the NBC Today Show, the CBS Early Show, and hundreds of reports on local television and radio newscasts.  It was also reported in the New York Times and Time magazine.

But there’s a serious problem with the AMA story. It claims that the poll results generalize to “women age 17 - 35 who currently attend college, graduated from college or attended, but did not graduate from college within the United States.” However, they do not generalize and cannot be generalized. In short, the study and conclusions are basically worthless.

The AMA claims that the survey used a random (probability) sample. Such a sample is carefully selected in very specific ways according to precise scientific criteria. Individuals are never self-selected. That is, people are not recruited with ads, websites, or other ways that permit them to come forth and present themselves to be sampled.

However, the AMA poll used volunteers from the Survey Spot recruitment website, where anyone can op-in. the company that operates the site also recruits volunteers through banner ads and telephone solicitations.

Contrary to its published assertion that it used a random sample, it actually used a non-random sample. It then incorrectly and deceptively asserted that its “survey has a margin of error of +/- 4.00 percent at the 95 percent level of confidence.” Non-random samples have unknown margins of error; reporting them is only appropriate for random samples.

In addition to being worthless and of no scientific value, the AMA poll is very misleading. Although it was supposedly about the dangerous behaviors of college women on spring break, only 27% of the sample had ever gone on spring break. Statistician Mark Blumenthal, who carefully investigated the scandal, observed:

The fact that only about a quarter of the respondents actually went on a spring break trip -- information missing from every broadcast and op-ed reference I encountered -- raises several concerns. First, does the study place too much faith in second-hand reports from the nearly three quarters of the women in the sample who never went on a spring break trip? Second, how many of those who reported or heard these numbers got the misleading impression that the percentages involved described the experiences of all 18-34 year old women? See the Hannah Storm quotation above. [Here the statistician is referring to a news report by Ms. Storm.] She appears to be among the misled.

One might think that the press release from the AMA would have gone out of its way to distinguish between questions asked of the full sample and those asked of the smaller subgroup that had actually been on a spring break trip. Unfortunately, not only did they fail to specify that certain percentages were based on a subgroup, they also failed to mention that only 27% of their sample had ever taken a spring break trip. Worse, the bullet-point summary of results in their press release mixes results for the whole sample with results based on just 174 respondents, a practice that could easily confuse a casual reader.

The president of the American Association for Public Opinion Research, Dr. Clifford Zukin of Rutgers University, is disturbed by the AMA’s misuse of statistics and its associated deception.

The AMA official responsible for the survey said that its methodology  is standard and the survey is generalizable to the population. To this, Dr. Zukin replied:

Simply put, statistically, you are wrong. The methodology is not standard, it is not generalizable to the population. And, the reporting of a sampling error figure, as you have done in your methods statement is fanciful. Because of the way you sampled people, with a non-probability sample, there is no way can know about the accuracy of your sampling and error margin. This is simply without basis in mathematical fact. 100 out of 100 statisticians would tell you that there is no sampling error on a non-probability sample. It is beyond question that your methodological statement is factually inaccurate and misleading.

I am also troubled by the fact you actually call this study a "media advocacy tool." It is unconscionable to put something in the public domain under the guise of a scientific survey when it has such a high potential to be inaccurate and mislead. Scientific surveys should be done to measure public opinion, not to influence collective opinion. (Emphasis added)

Dr. Zukin has uncovered something much more important than the fact that the AMA survey lacks scientific value and is misleading and deceptive. He uncovered the fact that the whole purpose of the survey was to influence public opinion in support of its alcohol policy agenda.

That fact is actually to be expected. The poll was commissioned by the AMA’s Office of Alcohol, Tobacco and Other Drug Abuse. And who established and funds that office? The anti-alcohol Robert Wood Johnson Foundation.