How Does Polling Work And What Polls Are Reliable?

Santa Barbara County’s election prep (Credit: Paul Wellman via Santa Barbara Independent)

How does political polling work?

In general, polling seeks to ascertain the popularity or opinion of a person or idea. Political polling focuses directly on upcoming elections and issues candidates and the electorate are currently discussing.

Polls can be open-ended, where a question is asked and the respondent can fill in the blank with any answer they think of, or polls can have a selection of answers to choose from, like a multiple-choice test. Polls can also offer a binary choice, such as a question asking if you approve or disapprove of a person or issue.

Types of political polls

There are four basic types of political polls that track the election cycle:

• Benchmark Polling: A prospective candidate will employ this type of poll in order to gauge excitement or interest in his candidacy. This type of poll will assist the candidate in setting campaign objectives, strategies, and priorities.
• Brushfire Polling: This type of poll asks if the person surveyed finds the candidate favorable or unfavorable, and tracks the results over time. This type of polling can also be used to measure the popularity of a specific issue, generally a hot-button issue or one especially important in the election.
• Tracking Polling: Tracking polls ask specific questions over time to see how support changes.
• Exit Polling: Exit polls are asked of people after they have voted in order to attempt to predict results.

Polls can gather data in a variety of ways. The most popular amongst pollsters has traditionally been by phone, though online polling is becoming more accurate and useful as phone response rates have dropped. In-person polling also occurs in some instances.

How do political pollsters target a representative sample?

It would be an impossible task to poll every single person of a large population, so a sample that is representative of the population is vitally important. For example, if a pollster wanted to measure support for a national candidate, they would choose a sample that represents the country’s demographics. You want your sample to be proportionate to your desired population.

Sampling can be very specific, like choosing only Democrats who are registered to vote for a poll about a Democratic primary candidate. The pollster would want to exclude to the best of their ability people who fall outside of that demographic. This provides a good sampling of Democrats who are registered to vote. Polling can also be very generic, such as sampling all demographics about a candidate’s favorability, whether or not they are registered or likely to vote.

Being able to obtain enough people for a survey has become more difficult in recent years. Fewer people are answering phone polls, driving the cost of polling up while getting fewer respondents. The advantage of phone polling is that those polling know the chance that a particular number will be selected. Online polling does not offer this benefit, and can be problematic as it is normally “opt-in”, meaning those who feel strongly about the candidate or issue will answer, but will not be representative of the population.

What is a margin of error?

As polling is taking a representative sample, it has what is called a margin of error. We know the sample is representative of the population, but because we could not ask every person, we expect a little bit of a different result if we asked a different representative sample the same question(s) again. The margin of error tells you how much the answers could change.

A strong survey based on 1,000 people would have a margin of error of 3%. If a candidate had a favorability rating of 53% in the survey, a 3% margin of error tells us that the actual favorability rating could be as low as 50% or as high as 56% if the same survey were taken again and again with different respondents. Should the candidate begin to see their favorability rating move up to 59% in multiple polls, for example, it could reasonably be concluded that it is a true increase in their favorability, as it is outside the 3% margin of error.

When looking at polling, pay attention to the margin of error. A large margin of error indicates a weakness in the poll’s data, while a smaller margin of error is more reliable. The smaller the margin of error in polls, the higher the quality of the poll. Additionally, while outlier polls, or those that swing far outside of the margin of error, get more media coverage than polls that fall within the margin of error, they should not be relied upon until multiple polls can verify the results.

What are cross tabs?

Cross tabs are a way of diving deeper into the data of polls in order to determine how subsets of respondents answered. Answers can be broken out by age ranges, education, gender, political affiliation, race, religion, or any number of demographics, so long as the demographic information is collected or verified at the time of the survey. If a campaign wanted to see how college-educated voters under the age of 35 felt about the candidate, they would use a cross tab for that information.

This information can then be used to aggressively target a particular group with a message or to plan future action plans to try to sway certain populations. Cross tabs should be used with some caution, as the margin of error increases as the sample size of the cross tab gets smaller.

How to tell if poll results are high quality?

High-quality polls can be difficult to ascertain at times. As previously discussed, a high margin of error indicates a low-quality poll. Using too small of a sample size is generally the cause, and anything below 400 respondents should be essentially ignored. A sample size of 1,000 is typical. Again, caution should be used in online polls with large sample sizes, as the opt-in process for online polling does not necessarily guarantee high quality and representative sample.

Polls can also be biased in a couple of different ways. National polls have a very high non-response rate, meaning it can be difficult to get enough responses at all, especially when trying to get a representative sampling. Bias can also occur when respondents give answers more extreme than the ones they actually hold, or if they are embarrassed to give a truthful answer when talking in person or on the phone, fearing their answer will be ridiculed or isn’t popular.

One of the most problematic areas of polling that can be manipulated is the wording of questions to force a certain response. This is referred to as ‘push polling’, and it is considered unethical by the American Association for Public Opinion Research. A question using this method would try to persuade a respondent to arrive at a particular conclusion more than it seeks to measure true favorability or opinion. Be wary of surveys that use leading questions, exclude a candidate, or tilts to one side of an issue or candidate. Also, be wary of polls that are not administered by a neutral source.

Finally, be wary of how polls are interpreted since small slices of results can be twisted to look more beneficial or worse than they truly are. Sometimes, poll results can be misinterpreted due to a misunderstanding. A proposal may be disliked not because it is unpopular, but because it doesn’t go far enough, for example. Better questions with more options could help avoid such a situation if the poll is not a simple favorable/unfavorable poll.

Pros and cons of online polling?

Online polling is becoming more popular due to fewer landline phones, but it also has issues unique to the medium. Some pollsters are inviting respondents to take an online poll via phone or mail using a special code, protecting the integrity of the sample size by not allowing everyone to partake in the survey. There are several pros to online polling:

• It is far less expensive and time consuming to conduct online polls. This allows more polling to be conducted, increasing accuracy and data integrity.
• Online polling can capture more people than phone polling, as landline phones become scarcer and go unanswered more.
• Some respondents may feel more honest when not talking to a live person and give more truthful answers.
• Fewer questions are skipped in online polls versus phone polls.
• Online polls can also show a video clip as part of the question, an option not available with other question methods.
• A respondent can spend more time taking an online poll and not feel rushed.

There are also several cons to online polling:

• Bias can run rampant due to certain demographics, such as those in rural areas or people over the age of 65, being shut out from online access to the polls. An undercovered population might be unrepresented and the poll could be unintentionally skewed.
• Difficulty in obtaining a standard list of online respondents can be downright impossible, unlike a phone number or address, which is linked to one person. There is no master email database, and many people have multiple email addresses making a database impossible for this purpose.
• Convenience samples, or those surveys that allow anyone to answer, can be significantly skewed, as people who view the issue especially strongly may be the main respondents. If the sample is large enough and enough information is known about those responding, it is possible to create a representative sample.
• Bogus respondents, like bots, could also answer online polls in a purposeful attempt to skew results one way or the other, in an attempt to create chaos or confusion.

Look very closely at the methodology of online poll results to ensure it is a well-created poll with valid questions and respondents.

What are the most trusted polls?

Several organizations keep track of and rate the accuracy and methodology of pollsters, including Nate Silver’s FiveThirtyEight. Based on historical accuracy as well as methodology and representative samples, these are the highest-rated political polls and their FiveThirtyEight rankings:

• ABC News/Washington Post – A+
• Sienna College/The New York Times Upshot – A+
• Seler & Co – A+
• Marist College – A+
• Monmouth University – A+
• CBS News/The New York Times – A
• Suffolk University – A
• Survey USA – A
• NBC/Wall Street Journal – A-
• Quinnipiac University – B+
• YouGov – B
• Public Opinion Polling – B

The Rantt Rundown

Polling can be a confusing issue, but by making sure that the polls you read and pay attention to are high quality, you can learn to exclude the noise and concentrate on the valid information. Look for a low margin of error, with a good representative sample size. Be aware of how questions can be worded to ‘push’ a particular answer, and disregard those polls. Finally, trust the high-quality pollsters who have been time tested and have a solid track record of results.