Surveys, response rates and credibility
Everyone loves a good survey – and a survey can certainly be an effective marketing tool, especially when the numbers say what you want them to say.
But if your survey is to have any credibility, you need to tell people how many qualified respondents it is based upon.
This came up today because I was poring over the results of AdMedia Partners’ 2010 Market Survey, its 16th annual survey of executives with leading media and marketing services firms, the results of which are covered in another post. I was initially attracted to the survey because it appeared to be based on a huge number of respondents – more than 7,400.
At least, that was my take after reading the survey methodology, which begins, “This report is based on a Web-based survey of more than 7,400 domestic and international executives in the advertising, marketing services, digital marketing, marketing technology, media or digital media businesses and related venture capital/private equity investors.”
Just to be sure that meant more than 7,400 respondents, I called AdMedia Partners. As it turns out, that figure reflects how many folks got the survey; the company doesn’t disclose how many responded. In the survey’s 16-year history, that’s just always the way it’s been done, says AdMedia Partners managing director Seth Alpert, although he’s not sure why. I tried to get a ballpark response rate out of him, but all he’d say was that it’s on par with “the norm” for such undertakings – whatever that is – and that it was less than 50%. (My sense was that he wanted to tell me, that he couldn’t think of a good reason not to – but I’m guessing.)
To me, it’s not only a mistake to fail to disclose the number of respondents to a survey, it’s a disservice to your audience, which deserves to know whether your results are based on 20 opinions or 2,000. I’ve done lots of surveys in my time, mostly as features editor for Network World, when we had a series of surveys we did each year. Always we were concerned about getting a high response rate so the results would be statistically valid, especially for our salary survey, where it was important to get enough responses for each geographic region. If we didn’t have enough responses for a specific job title in a given region, we’d fess up that the sample wasn’t statistically valid, give the results we had, and let the reader decide what it all meant.
Any company that wants to use survey results as part of its marketing strategy should follow a similar path. While the Web has made surveys far easier to conduct than in the days when we relied on snail mail, it’s still not exactly easy to get a high number of responses. But resist the temptation to fudge the response rate, or to simply not disclose it at all – the credibility of the survey and perhaps even your company is on the line.
Explore posts in the same categories: IT marketing, marketing