By Ed Stetzer
We need to be shrewd when it comes to statistics and reality:
things are not always as they seem.
Often
times, a statistic is like a piece of candy thrown at a parade—you really don't
know if you should bite into it or not. We've all heard Mark Twain's famous
quote about lies and statistics. There is a reason so many people have had
skepticism toward stats. Too frequently, people repeat inaccurate, bad, or
explicitly made-up numbers.
I've written about the issue before—on many occasions. Here
at the blog, you can read about a lot about stats, including specifics
about bad marriage stats and why we like bad stats in
general.
There is a reason so many people have had skepticism toward
stats.
Still, I
keep hearing statistics quoted at conferences and through blogs and social
media that make me scratch my head in amazement. I'm not sure where some of
these stats originate, and I'm the president of LifeWay Research.
So how
can you really discern good stats from bad?
1. Be Wary of Statistics in Promotions
First, be
skeptical of stats in promotional pieces. They are trying to convince you to
consume an ideology or a product. Advertisers do not hold to the same standards
as a researcher.
Statistics
are effective at moving people toward action, but it can border on
manipulation, depending on how you use or change the numbers and information.
Using unverified statistics can be an example. Even if the action is good, it
is not acceptable to use a statistic that is not valid.
For
example, maybe you've heard of books that say the church is dying—and this book
can fix it (Yet the numbers show that the church is not dying). Or, that 80%
of church plants fail—and this conference can stop that (yet, about
two-thirds of church plants are around in four years after starting).
You've
probably heard someone say the stats show that pastors are miserable and just
want to quit—and our ministry saves them (when the vast majority are happy and consider it a
privilege to serve). Or that the Super Bowl is the biggest sex trafficking
event in history—and you should support our ministry to stop it (when there is no
law enforcement evidence that this is true).
All of
these things are important: we need the church to thrive, church plants to be
successful, pastors to be happy, and sex trafficking to end. But we don't need
made up facts to evidence the situation and the need.
2. Be Wary of Stats that Cannot be Verified
Secondly,
is there really a way to know the fact claimed? In other words, you should
always ask "How do you know that?" when someone quotes a statistic.
For
example, we have heard that 94% of evangelical youth drop out of church after
high school never to return. The three words "never to return" should
raise red flags. How could anyone know that?
You
should always ask "How do you know that?" when someone quotes a
statistic.
Realistically,
how could they keep track of those 94%? Did a researcher track them their whole
lives or implant them with transponders and see if they come back?
We have
also heard the statistic that 80% of pastors don't make it to retirement. How
would someone know that? Some of this is just common sense.
If you
can't think of a way to verify a statistic, it is probably not true. Unless
someone spent millions of dollars on research conducted over decades, some of
these stats could not be verified.
If it is
not verifiable, you should not be quoting it.
3. Be Wary of Stats that do not Line up with Reality
Thirdly,
be skeptical about statistics that do not line up with your reality. For
example, one statistic seems to indicate that Christians and non-Christians
divorce at the same rate. That hasn't been my experience. The truth is that
those who practice their faith have a much lower rate of divorce.
I wrote a
post called That
Divorce Stat You Quoted Was Probably Wrong due
to the confusion over this issue. The oft-quoted statistic comes from a Barna
study. However, if you read what Barna reported, the research found those that say they're
Christians divorce at the same rate as those that say they're
not Christians. But the reality is that a much larger number of people claim
to be Christians than really are, particularly when you look at beliefs and
actions-- and that has to be factored in before we start saying,
"Christians do this" (unless you really believe 75% of Americans are
Christians).
So, stop getting up
before your church and leading them to believe that they have a 50/50 chance of
divorce. That just demoralizes them. Sociologist Bradley Wright, author of Christians
are Hate-filled Hypocrites… and Other Lies You've Been Told, indicated that
60% of people who identify as Christians but rarely
attend church have been divorced. But of those who attend church regularly,
only 38% have been divorced. That is a huge difference.
Not to
mention you can use your own common sense. I recently read the 50% of pastors
will get divorced. Because this stat is all over the place, I'm going to write
a full-length post on it in the future.
We need
to be more skeptical when it comes to statistics.
But just
use some common sense for the moment and ask the question. Do we REALLY believe
that half of all pastors get divorced? Does that REALLY make any sense to you?
If the 50% divorce rate is a myth for all Americans (and it is), do we REALLY
think that would be the case for pastors?
Of course
not.
You could
be wrong, but if something is way out there, pause, then ask, is the evidence
stronger than my common sense? If it is, then change your opinion. If it's not,
then require more clarity.
Be Skeptical when Needed and Always Discerning
Simply
put, we need to be more skeptical when it comes to statistics. If a researcher
cannot tell you their research methodology and the questions they asked, you
should immediately disregard the statistic. Researchers have an ethical code
that we supposedly follow—if they don't, stop listening to them, and never
quote them.
Occasionally,
researchers will come up with different conclusions to the same concept. It
depends on the question posed. That's not the same thing.
I
recently read some comments from David Kinnaman of Barna about Millennialsand
evangelism. He says that while most born-again
Christians say they should share their faith, few do. At Lifeway Research,
we've found the same thing.
However,
David's report found that Millennials who are born again are the exception, and
share their faith at a higher rate. Our data shows something different,
something I'll discuss with David soon. My guess is that different samples and
wording made the difference between the samples—but you should be able to tell
by looking at the methodology!
But, in
both cases, there is a reasonable methodology that led us to these different
conclusions. As a reader, you should look at the methodology of study in order
to know where the researcher is coming from, and to know the context for the
statistics. You just need to be discerning.
You Don't Need a Degree in Stats
Every
well-framed question tells us something, as long as it is done correctly.
To
discern if research is of good quality, it may help to understand some of the
process. It's not that complicated and it will keep you out of trouble.
In any
study, a researcher needs to ask, "Does this question ask what it needs to
ask?" Every question is imperfect. There is only so much money available
to research a topic therefore the number of questions is limited.
So a
researcher is only going to be able to ask a limited number of questions
(sometimes only one or two) about a topic. From that research, we're not going
to be able to draw every conclusion—although some try to extrapolate it to
ridiculous conclusions. But, every well-framed question tells us something, as
long as it is done correctly.
So, How Do We Know?
What are
some ways to tell if the stat you've heard is trustworthy? You can tell the
quality of the research if it is:
a. done on a good sample (a random set with a large
enough sample, not a group at a conference, a website where
people click to vote, etc.)
b. a good question (is it fair and not leading), and
c. a good methodology (is it transparent, honest,
and credible).
You can
guard against believing and spreading bad statistics by determining if its
source is a promotional piece, if it cannot possibly be verified
scientifically, and if it does not line up with reality. Then, if it is quality
research, quote away.
Anything
else is just spreading misinformation. We can and should do better than that.