When you’re assessing any kind of population, for any kind of action, the survey is only as good as the thoughtfulness engaged in considering all the variables
Our data-saturated culture loves research and numbers, but information can mislead. A host of factors play into accuracy that many people may not realize. It’s what separates quality, human-driven research from computerized short-cuts, and it’s what makes us all informed citizens and consumers. In this three-part series, Candice Bennett takes a look at different aspects of data-gathering, what they mean, and why they matter.
Let’s start with a hypothetical client. We’ll call her Sarah, from Link Financial. Sarah wants to know what customers really think about Link’s new banking service, something more accurate than soliciting feedback online, where responders are self-selecting. Several thousand people have signed up for the new service, so they have a broad base to pull from.
The bank’s research team pulls together some questions, and starts randomly emailing or calling clients. But overwhelmingly, the feedback is negative. Sarah is disappointed by the results -- this doesn’t seem right to her, considering how many new clients sign up daily.
What went wrong?
What Sarah didn’t realize was that her research team called clients at home, in the middle of the day. Many people were at work. A few were home sick from work, and not feeling particularly positive. There were some older clients, and some stay-at-home-parents, but they aren’t really a representative sample of the client base as a whole. If the researchers had called at different times over several days, or called in the evenings, the responses might have been wildly different.
And because so many people were working during the day, only a handful of clients responded. The smaller the sample size, the greater room there is for error. Say they talked to 10 people an hour, and there was a calculated margin of error of 25%, plus or minus (the larger the sample size, the more accurate the data, the smaller the error rate -- the smaller the sample size, the greater the margin of error). That means if 50% of the people said they liked the service, the range is really 75-25%. An hour later, maybe only 30% of the people liked the service -- with a margin of error putting the real sentiment as low as 5% -- but since the margin of error is so large, it’s a difference of only two people.
So what does this mean in the real world?
When you’re assessing any kind of population, for any kind of action, the survey is only as good as the thoughtfulness engaged in considering all the variables, including samples sizes and timing. It’s easy to plug in some questions into an online survey, or engage an automated machine to make calls. But computers can’t consider all the ways people might respond, and how to engage them accurately to provide truly meaningful results.
It also means that if the results seem wildly different than what you’re experiencing, it might not be the respondents -- it might be your survey, both the questions and the methodology. At a school, for example, if a well-loved, well-regarded teacher has shockingly low test scores, that might be time to review her class population, the test timing, the number of students who took the test, and other factors -- instead of just considering taking steps against the teacher.
Back at our hypothetical bank, if Sarah had followed the results of her first survey, she would have tanked a very successful program. Instead, she went back to the survey firm, requested someone review the results with her, and decided the error rate was unacceptable, as were the number of respondents. Together she and her research team brainstormed a few ways to tweak the survey to garner more useful results.
Today, her online banking program is one of the best in the region -- after she was able to solicit some truly useful feedback in order to tailor the program even better for her clients.
Don't miss the other two articles in Candice Bennett's series, Part 2: How Do You Critically Interpret Data? and Part 3: Correlation Does Not Equal Causation And Why You Should Care.