There has been a good deal of talk about the American Secular Census (ASC) in the blog-o-sphere of late, and I’ve been prompted by a friend to share a few of my thoughts on the project.
First off, it is not a census in the generally accepted sense of the term. A census is an attempt to measure an entire population, in this case, the millions of Americans who are “skeptical of supernatural claims including without limitation those which concern gods, miracles, and other claims generally associated with religion.” This is, of course, well beyond the capacity of any privately-held data-gathering enterprise.
We don’t really need to do a full census to get a good sense of the population, so long as we are able to do random sampling from that population. If the population of secular Americans were treated as a sampling frame from which a truly random sample was selected, we could make strong conclusions about the group as a whole. However, the sampling method in this particular case is a form of non-probability sampling known as convenience sampling, which means that we cannot make statistically valid inferences from the sample to the population from which it was drawn, because of the strong possibility of self-selection bias. To take an example from your everyday experience, are you more likely to fill out customer surveys when everything when smoothly, or when something went terribly wrong? That’s self-selection in action.
One of the most famous forms of self-selection bias in online polling is known as pharyngulation and describes the process that occurs when an online poll is overrun by the devotees of a certain blog or a set of blogs, typically respondents who have been directed to the poll by the widely-esteemed writers at Freethought Blogs. Since the ASC has been mentioned over one hundred times on that website, there is a strong probability that the results are not merely skewed, but actually pharyngulated. That is to say that whatever opinions are most strongly and widely held by readers of FtB may well be overrepresented in the ASC results relative to the general population of American secularists.
Now, I’m not saying that the ASC is a bad idea, and sincerely hope that they will eventually achieve a sample sizes large enough to make meaningful pronouncements. For the moment, though, I would be hesistant to trust any of their preliminary findings. So long as there is a numerous and active “horde” of secular people who pride themselves on skewing online polls, and so long as the survey sample size isn’t sufficiently large so as to consign said horde to the statistical noise, the results of such polls must be considered with profound skepticism.
