[1]NASHVILLE, Tenn. (BP)–To politicians and captains of private enterprise, opinion polls have long been seen as valuable tools to track changes in American tastes, views and practices.
But in recent years, many church leaders also have begun to pay attention to poll results in areas involving spirituality, such as church attendance, Bible reading, belief in God and religious affiliation.
Yet on occasion, polls may become difficult to interpret when various pollsters ask the same question but report different results.
Such was the case earlier this year when two nationally known polling organizations — Barna Research Group and the Gallup Poll — each released figures tracking the frequency of church attendance by American adults.
On the one hand, a Barna news release showed “Christianity is showing signs of revival. … The multi-year decline of Christianity appears to have bottomed out, and the Christian faith is now experiencing a renaissance of sorts.”
On the other hand, the Gallup Poll showed church attendance at its lowest level ever since Gallup began asking the question in 1939.
Specifically, the Barna poll showed 43 percent of American adults said they had attended church during a typical week — up 6 points from a 15-year low of 37 percent in 1996, just a year earlier.
The Gallup Poll, however, showed only 35 percent of adults had attended church or synagogue in the past seven days — down 4 points from 39 percent in 1996.
“We at Gallup would not say that represents a significant change in America’s church attendance pattern,” said Frank Newport, editor-in-chief of the Gallup Poll, in an interview with Baptist Press.
Both Newport and David Kinnaman, Barna’s research director, granted interviews on a wide range of topics related to opinion polls and were asked to comment on the differences between the two polls.
Without comparing the two polls in specific detail, both spoke generally about the factors that can cause differences.
Regarding church attendance figures, Newport said, “It looks like other researchers in this area are attaching more importance to these fluctuations than we at Gallup would. We would want to see if this represents a sustainable pattern. I would be sensitive about putting a tremendous amount of attention to any one measurement.”
Kinnaman said any number of variables can influence the outcome of different polls, including methodology, question order, how respondents were selected and the wording of questions.
In Barna’s case, the question about church attendance read: “In the last seven days, did you attend a church service, not including a special event, such as a wedding or funeral?”
Newport noted Gallup has been asking the same question of Americans since 1939, which is: “Did you, yourself, happen to attend church or synagogue in the last seven days, or not?”
“The basic trend we saw, moving out of the 1950s and 1960s, was the positive response to that question settled in at around 40 percent,” Newport said. “It’s been at about that level since 1972.”
In general, other factors can impact opinion polls, thereby generating different results for different pollsters, according to Newport and Kinnaman.
“That’s something that will happen on a fairly consistent basis — to recognize there are a lot of differences in the way organizations conduct research,” Kinnaman said. “Some are wrong, some are bad, some are simply different. But when factors are held constant, it’s pretty remarkable how much consistency results can have.”
Kinnaman warned against placing much trust in polls that use “self- selecting samples,” such as when a local television newscast asks viewers to call in and respond to a poll question.
“That does not give every person an equal chance of participating and would not be valid,” Kinnaman said.
He added certain other survey techniques — such as the “mall intercept,” in which people in shopping malls are asked to respond to questions — are likewise invalid since not everyone has an equal chance of being selected for participation.
The number of people polled also is critical to achieving trustworthy results, Newport suggested. “Almost all reputable polls have 800 to 1,000 people in a national survey. This produces a margin of error of 2 to 3 percentage points.”
“Margin of error” means that polling results for the people actually surveyed could be a bit higher or lower than results from a survey that polled the entire population.
“In a generalized rule of thumb, the more the people in the sample, the smaller the margin of error,” Newport said. “But if the number of people surveyed is smaller, the margin of error would be higher. If only 150 people were surveyed, the margin of error would be quite large.”
Another factor that can impact poll results is how the questions are worded.
“Question wording is an absolutely vital piece of the research picture,” Kinnaman observed. “It is an art, not a science. Developing question wording is a process of going through and evaluating the effects of different questions and the way you word them.”
For one thing, poll questions should avoid jargon and use simple language, Kinnaman said. Polls involving spirituality, for example, should avoid the use of such words as fundamentalist and eschatology, he said.
Questions also should allow for reasonable recall on the part of the people being polled, Kinnaman said. “Asking people how many times they’ve attended church within the last year borders on the unreasonable. A better way would be to say, ‘In a typical month, how many times do you attend church?'”
In general, pollsters should lay aside their own philosophical views in order to arrive at accurate results, Newport said. “In theory, polling should be as neutral and value-free as possible — not trying to bring any kind of bias to it — though some people say it’s impossible for a scientist not to bring presuppositions to work.”
Kinnaman noted some organizations that conduct surveys want the results to prove some philosophical point.
“At Barna Research and at many other organizations around the country, the issue is not in proving a hypothesis but in trying to provide truth and accuracy,” Kinnaman said. “We have to be willing to employ a reasonable methodology and to sacrifice what we want out of something in order to arrive at truth.