In my work I design and interpret a lot of surveys of both colleagues and passengers. And in our data-obsessed culture, outside of work I see or am asked to take surveys all the time. Sometimes I take a survey and think “I have no idea what they are asking here” or worse, “These results are going to be totally biased and misleading.” I wanted to provide some thoughts on how to design better surveys so that a) you get data that actually tells you something, and b) your respondents aren’t scared to take a survey from you ever again. Because that won’t help anyone, will it?
1. Clarify What You Want to Know and How You Will Act on Your Results
Before you do a survey, ask yourself: what question am I trying to answer? What data already exists? And is a survey the best way to answer my question?
Your question might be “What does the community prefer we do about xyz problem?” or “How well did this solution work for the people it affected?” Consider your question to be like a thesis of a paper, and stick to it or you’ll be tempted to wander off and lose focus.
2. Commit to What You Will Do with the Results
Once you have your question, be sure you know what you will do with the information you get, even if it isn’t what you want to hear. Fielding a survey only in the hopes of validating a decision you’ve already made is a waste of everyone’s time. And learning that people are unsatisfied, but not finding out why, won’t get you actionable results.
So, what will you do with the results? Maybe you will make a decision based purely on the results, especially if the decision is low-stakes and you’re sure you heard from a representative group of people. If it’s a major policy decision you’re weighing, it might not be feasible or wise to promise to do what the majority wants, but maybe it will cause you to further evaluate an option you had previously dismissed, or do more in-depth outreach to reach consensus with the interested parties. This is ok, but be up front with respondents. If you lead them to think their survey input will directly influence an outcome, and it doesn’t, they may grow disillusioned and refuse to participate next time.
3. Take a Step Back
Last, are you sure you need a survey? A survey can be an effective way to hear from a large number of people, but since most of your questions will likely be multiple choice, you limit what people can tell you. If you want an in-depth understanding of not just what a user or community member thinks but why, a small focus group or stakeholder interviews may be more appropriate, where you can have a two-way conversation. Don’t do a survey just because you think you have to; consider all your options and choose the one that is best for your question.
Next week I’ll dive into writing good questions.
Nerd on the Bus 