Given the news media’s interest in surveys, a poorly-designed survey has the potential to spread a lot of misinformation. In late August, Dr. John Villasenor of UCLA surveyed 1500 college students’ understanding of and attitudes toward freedom of speech. He wrote up the results in an essay published by the Brookings Institution, explaining that the survey had not yet been subjected to peer review, but due to “the timeliness of the topic, I believe it is important to get some of the key results out in the public sphere immediately.”

The survey results were covered by several mainstream media, including CNN and the Wall Street Journal. They were summarized by Catherine Rampell of the Washington Post under the title “A chilling study shows how hostile college students are toward free speech.”

In his article, Dr. Villasenor reported five results of the survey.  Respondents were asked “Does the First Amendment protect ‘hate speech?’”  A plurality of 44% answered “no,” compared to 39% who said “yes,” and 16% who didn’t know. They were wrong, since the First Amendment protects offensive speech unless it is a threat or is directed toward producing imminent lawless action. Women were more likely than men to hold this incorrect belief.

Respondents were given the following hypothetical scenario.

A majority of students agreed, with Democrats being more likely than Republicans to condone shouting down a speaker.

They were also asked about the use of violence to silence a speaker.

The approval rate was much lower, but the fact that 19% approved of violence is certainly disconcerting. Men were more likely than women to condone violence.

Given the same scenario, respondents were asked whether “under the First Amendment, the on-campus organization sponsoring the event is legally required to ensure that the event includes not only the offensive speaker but also a speaker who presents an opposing view.” A majority (62%) incorrectly agreed that there was a legal requirement of balance.

Finally, respondents were given an item from a 2016 Gallup poll in which they were asked to choose between two types of university learning environments:

• Option 1: Create a positive learning environment for all students by prohibiting certain speech or expression of viewpoints that are offensive or biased against certain groups of people.
• Option 2: Create an open learning environment where students are exposed to all types of speech and viewpoints, even if it means allowing speech that is offensive or biased against certain groups of people.

A 53% majority chose the first option of prohibiting offensive speech, while 47% opted for the more open environment.

Shortly after the article was published, doubts about the validity of the survey were raised, with one critic labeling it “junk science.” It turns out that Dr. Villasenor is a professor of electrical engineering with no prior experience conducting surveys. His research was sponsored by the conservative Charles Koch Foundation. Of course, neither of these facts necessarily invalidates the survey.

A more important problem is that it is not clear how Dr. Villasenor obtained his sample. He does not claim that the survey was administered to a random sample of college students, but merely that the sample was “geographically diverse” and “approximately mirrors” the undergraduate population. This has led critics to conclude that he used a convenience sample of students who were available, but not necessarily representative of college students. Dr. Villasenor has acknowledged that this was an “opt-in” survey, a term used to refer to a survey using volunteers whose biases are unknown.

Dr. Villasenor further irritated survey experts by stating the confidence intervals, or the margin or error, around his results. This is inappropriate unless a random sample is used. (It should be noted that Dr. Villasenor covered his butt by saying that these confidence intervals were valid “to the extent” that his respondents were representative of college students, without actually claiming that they were representative.)

Dr. Villasenor also neglected to mention that his survey was conducted just a few days after the white supremacist rally in Charlottesville, VA, in which a peaceful demonstrator was killed. This violent incident may have temporarily reduced students’ tolerance for offensive speech.

Finally, it should be noted that in 2016, when Gallup asked a nationally representative sample of college students, carefully chosen using probability sampling, to choose between the two learning environments described above, 78% chose Option 2, the more open environment. While it is possible that student attitudes have changed dramatically in the past year, it is also possible that differences in sampling were responsible for the discrepancy.

Catherine Rampell defended Dr. Villasenor’s survey, correctly noting that no survey uses perfect random sampling in that sense that respondents are randomly chosen from a complete and accurate single list of all the college students in the country. However, her defense blurs the distinction between carefully conducted probability sampling and the apparently more haphazard methods used by Dr. Villasenor.

Sophia McClennen of Penn State has labeled Villasenor’s survey an example of “blue-baiting,” in which conservative organizations attempt to manufacture doubt about free speech protections on campus in order to undermine public confidence in higher education.  (This may be working.)

At the very least, the controversy suggests that journalists should be careful to determine that professional sampling techniques are used before reporting survey results.  On the other hand, some college students did give these responses, even if they came from a biased sample. This suggests that high schools and universities should devote more attention to educating students on the meaning and scope of the First Amendment.

You may also be interested in reading:

Republicans Say Colleges are Bad For the Country