Nerd on the Bus

How to Create Surveys that People Will Enjoy Taking, part 1

In my work I design and interpret a lot of surveys of both colleagues and passengers. And in our data-obsessed culture, outside of work I see or am asked to take surveys all the time. Sometimes I take a survey and think “I have no idea what they are asking here” or worse, “These results are going to be totally biased and misleading.” I wanted to provide some thoughts on how to design better surveys so that a) you get data that actually tells you something, and b) your respondents aren’t scared to take a survey from you ever again. Because that won’t help anyone, will it?

1. Clarify What You Want to Know and How You Will Act on Your Results

Before you do a survey, ask yourself: what question am I trying to answer? What data already exists? And is a survey the best way to answer my question?

Your question might be “What does the community prefer we do about xyz problem?” or “How well did this solution work for the people it affected?” Consider your question to be like a thesis of a paper, and stick to it or you’ll be tempted to wander off and lose focus.

2. Commit to What You Will Do with the Results

Once you have your question, be sure you know what you will do with the information you get, even if it isn’t what you want to hear. Fielding a survey only in the hopes of validating a decision you’ve already made is a waste of everyone’s time. And learning that people are unsatisfied, but not finding out why, won’t get you actionable results.

So, what will you do with the results? Maybe you will make a decision based purely on the results, especially if the decision is low-stakes and you’re sure you heard from a representative group of people. If it’s a major policy decision you’re weighing, it might not be feasible or wise to promise to do what the majority wants, but maybe it will cause you to further evaluate an option you had previously dismissed, or do more in-depth outreach to reach consensus with the interested parties. This is ok, but be up front with respondents. If you lead them to think their survey input will directly influence an outcome, and it doesn’t, they may grow disillusioned and refuse to participate next time.

3. Take a Step Back

Last, are you sure you need a survey? A survey can be an effective way to hear from a large number of people, but since most of your questions will likely be multiple choice, you limit what people can tell you. If you want an in-depth understanding of not just what a user or community member thinks but why, a small focus group or stakeholder interviews may be more appropriate, where you can have a two-way conversation. Don’t do a survey just because you think you have to; consider all your options and choose the one that is best for your question.

Next week I’ll dive into writing good questions.

Review: Survey Data Collection and Analytics certificate on Coursera

These days it seems anybody with a Survey Monkey account is sending surveys all the time. Some are intuitive and enjoyable to take; others not so much. My job currently involves a lot of survey design, but I know that I have made the mistake of designing a survey without a lot of thought, then gotten survey results back and realized that I can’t use them because I didn’t ask the questions in the right way. Does that sound like you?

Are your surveys engaging and enjoyable to take? *Photo by Christina @ wocintechchat.com on Unsplash*

Don’t let bad surveys happen to good people!

There are best practices that can really improve outcomes and response rates. I recently completed a certificate on Coursera to learn some of these best practices, and wanted to offer up some thoughts.

The Basics

Seven-course certificate program includes courses on sampling methodology, survey question design, data collection and dealing with missing data.
It took me about 3-1/2 months to complete the program, working on it maybe 5-10 hours per week but starting and stopping a bit.
Prerequisites: none.
It is hosted on Coursera, a Massive Online Open Course (MOOC) platform, but offered by a joint program through the University of Michigan and University of Maryland.
Cost: you can watch course videos for free; to turn in assignments or earn the certificate, it’s $43/month as with most Coursera courses.

Review

I thought this program overall was excellent and found the instructors to be engaging, knowledgeable, and helpful. They used a myriad of real-world examples and spoke from their own experience rather than reading a script. Their explanations of complex topics like sampling methods, alongside well-designed slides, helped me follow the conversation and gave me more value than I would get from merely reading a book.

For my job, I have already gone back many times to reference the Questionnaire Design for Social Surveys course, which was full of tangible steps to improve survey question wording, order, and layout. The classes on data collection and sampling also helped me feel more confident speaking the language of the consultants we work with to implement surveys.

I have only a couple of criticisms of this program. First, I found the capstone project to be somewhat frustrating, because the rubrics on which we were graded (via peer reviews) did not always match the instructions we were given. The assignments often involved spreadsheets but the platform did not always allow us to upload those spreadsheets for grading, leading to a lot of confusion, poorly formatted text documents, and assignment resubmissions. However, the project did give us a chance to exercise what we learned, and it was a good overview once I worked out the kinks.

The Dealing with Missing Data class also had a different instructor than the rest, and I found him to be less engaging and harder to follow. Perhaps the other professors just set a high bar.

The Bottom Line

It seems I get asked to take a survey at least once a week through some e-mail list or group I am a part of, and now I notice more than ever how many of those surveys are neither easy to take nor worded effectively. If your survey results aren’t proving useful to you, you could likely benefit from learning some of these skills. If you are involved in the entire process, from research design through final conclusions, I recommend you take the whole program. If you like to occasionally craft online feedback surveys for a few dozen members of some group you are a part, but don’t need to know about sampling or data collection, you may only need to take the Questionnaire Design for Social Surveys class. Good luck and let me know what you learn!

Review: IBM Data Science Specialization on Coursera

I recently completed the IBM Certificate in Data Science on Coursera, a Massive Online Open Course (MOOC) platform. There are so many ways to learn data science these days, so I hope this review will help others looking for a starting place.

The Basics

10 course certificate program includes courses on Data Visualization, Data Analysis, Machine Learning, SQL, and Python, among others.
It took me about 5 months to complete the program, working on it maybe 10 hours per week but starting and stopping a bit.
Prerequisites: none. Going in, I had basic knowledge of Python and statistics, but that wasn’t required.
It is hosted on Coursera.
Cost: you can watch course videos for free; to turn in assignments or earn the certificate, it’s $43/month. This is true of most Coursera courses.

Practicing the K-Nearest Neighbors algorithm in the machine learning course.

How you Learn

Each course was divided into 4-6 weekly units. Most units are comprised of several videos and a quiz or a lab, which is a hands-on Jupyter notebook that gives you practice on what you just learned using Python. These always had components that you had to figure out for yourself. Though many labs were not graded, this is where real learning takes place, so you get out of it what you put in. There were also many quizzes, which are auto-graded, and peer-reviewed assignments where you review someone else’s in exchange for getting a review yourself. As with any MOOC, the quality of peer review that you get is the luck of the draw.

Review

Overall, this program was a good introductory overview of the various facets of data science. I learned new Python libraries for data visualization and got a basic introduction to SQL and machine learning, which were new to me. The program also introduced a wide breadth of data science applications and included a lot of projects where I got to apply what I was learning. I have heard many data scientists say that the only way to really learn is by doing projects, and I find this to be spot-on.

The projects only uncovered the tip of the iceberg on those topics, though, so I will need to practice more to really learn the material. But I feel that I can now look back on the labs from these classes and reference the code to start my own side projects and implement what I’ve learned. It gave me confidence and a place to get started.

Although I knew this would only be a starting place, it would have been nice if the projects gave me something to add to my portfolio, however basic. However, while the class projects helped me to understand a business case for what I was learning, I didn’t really end up with something that I would feel proud to put on my GitHub. The capstone project involved labs where we would be provided with starter code and then fill in the rest. I wouldn’t feel right about putting something on my GitHub that I didn’t come up with on my own. Also, you could tell when looking at my project that it was an assignment geared towards exercising a lot of different skills rather than answering a real-world question because some of the questions were fairly irrelevant, such as: “do a SQL query to return a list of all items in the database that start with ‘CCA’.”

There was one class in data science methodologies, which I appreciated. More than the others, this course was about how to think like a data scientist: understand the problem, frame a question, then plan your approach. Overall, my biggest critique of this program is that it didn’t have enough of these types of courses, leaning more heavily on how to use tools than on the theory behind them. There are two topics that I especially think should have been covered but were not:

Statistics is an essential part of understanding your data and how to best represent and analyze it. I had taken one statistics course in grad school, but I am sure plenty of students had not.
Ethics is critical for appreciating the bias that your data and models can have and the enormous impact that they can have on your realms of influence. I tried to compensate with the books Weapons of Math Destruction by Cathy O’Neil and 97 Things about Ethics Every Data Scientist Should Know by Bill Franks. But I am troubled by how infrequently people seem to learn ethics in a field that has an incredible impact on things as basic as whether someone is approved for a loan or gets into college.

The Bottom Line

Overall, this certificate was worth my time, but it’s important to set your expectation that any course, MOOC or otherwise, is more the beginning of a journey than the end. I know that what I get out of it will ultimately be determined by how hard I work going forward to apply what I learned at work and by doing projects. There are plenty of other MOOCs out there and even other data science certificates on Coursera, so if you are seeking to hone your skills you should think about what is most important to you. Everybody starts somewhere, and even if you think you might want to get a Master’s degree or go to a bootcamp eventually, a MOOC is a good way to clarify your needs and wants before you make that investment of time and money.