How To Run Surveys and Which Survey Site Is Best

Sat, January 26, 2008, 09:09 PM under Links
In my quest to get some feedback I setup 3 surveys for my subscribers with identical questions on three different sites: surveymonkey, zoomerang and questionpro (I also signed up and test drove for a bit wufoo but it wouldn't let me setup the questions the way I wanted them so I gave up). I thought I'd share my learnings here so you can use the knowledge if you setup a survey for your own site/blog or even for your team at work or your customers. At the bottom I also have some generic tips.

Remember my disclaimer.

All 3 vendors offer a free version of their paid for services. The main limitation of the free service on all sites is that they only allow 100 responses per survey.

In all cases you sign up by providing your email and then you receive the login info. Surveymonkey sent me one text-only email with login details and that was the last I heard from them. The other 2 sent me html emails and also decided to sing me up to their newsletter (which I promptly unsubscribed from). Surveymonkey gives you a URL that points directly to the survey; the other two force a landing page (where they have their adverts) before allowing the survey taker to proceed.

Zoomerang has a funky easy to use user interface and with minimal clicks you can setup the survey and ditto for browsing the results (both overall and individual). There are 2 reasons I will not be using their free service again. Not only you cannot download the reports but they will only hold them on their servers for 10 days since the launch time of the report. More importantly, they will not notify you when the 100 allowed responses are complete and instead let people fill in your survey beyond that and if you want to see the additional completes you have to pay (I noticed this at some point and had to manually close the survey)!

QuestionPro has a horrible unintuitive user interface IMO. In addition, it takes far too many clicks and slow response times to setup the survey and ditto for reading the reports. Its login timeout is also too short so it added additional clicks to my user experience every time I refreshed the reports page. The main advantage it has is that even the free version allows you to get csv/xls reports to your own machine, whereas the other 2 will not let you do that unless you pay. Even generating the report involved making a few clicks and then receiving an email with an attachment (instead of just right click->save as). Beware that they will not notify you when the response limit reaches 100 and if there are more than 100 responses they will not show you *any* reports until you upgrade!

Surveymonkey just works. Best experience for setting up, quickest to use for browsing reports, keeps you logged in with no apparent timeouts and smoother experience altogether both for me and for the people taking the survey. None of the issues I mentioned in the previous two paragraphs.

To conclude, here are some random tips if you run a survey. This is based on my vast experience of doing this...twice ;-)

1.Keep the number of questions short; definitely make them fit on a single short page. Any more than that and users will abandon the survey the minute they see it. Budget 20 seconds per question and then gauge how long you think someone will spend overall there before giving up – this should guide you to the total # of questions.

2.Write questions in such a way so the user can just select the answer (i.e. Yes/NO or multiple choice). Again, if you force me to have to do a lot of typing, I will probably not take the survey. Make the last choice "Other" where applicable and offer an optional textbox for additional comment.

3.Following on from the previous point, try and make the multiple choice answers exclusive. Allowing multiple responses per question (like I did) means having to spend a lot more time analysing the results as the percentages alone do not show the full picture.

4.When you think you wrote a good question, visualise how the results will be interpreted. If you cannot think of how you would interpret the results or what action you'd take based on the results then the question is wrong. For example, I asked a question in my survey that was just to satisfy my curiosity and in hindsight it was not actionable or interpretable so I should have left it out.

5.The more factual-based questions you have, the faster the user can move through them. The more opinion-based questions that you have the longer it takes to fill in and the longer it will take you to analyze. Just something to bear in mind.

6.Assuming that you managed to write questions where the user didn't have to use the keyboard, make the last question open ended. Make it last, so you don't receive feedback that is covered in other questions. Make it open ended because the verbatim feedback will likely be more important than the rest of the questions.

7.Never use multiple different survey sites for the same set of questions. Aggregating the results from 3 different engines for consolidation is a bit of a time consuming nightmare.
Saturday, January 26, 2008 11:22:00 PM (Pacific Standard Time, UTC-08:00)
I'd suggest looking at this one seems like they've build a great product
Sunday, January 27, 2008 11:27:00 AM (Pacific Standard Time, UTC-08:00)
Are you going to share the results of the survey?
Sunday, January 27, 2008 2:11:55 PM (Pacific Standard Time, UTC-08:00)
mdonatas: Thanks, surveymonkey has set a high standard, but I'll check out your suggestion next time.

stevedotnet: I just did ;)
Monday, January 28, 2008 4:34:00 PM (Pacific Standard Time, UTC-08:00)

disclaimer: I am one of the co-founders of QuestionPro -- So take my comments with a grain of you know what...;)

I think your critical analysis is a good for us. The big challenge we face is combining ease of use with enterprise/market research functionality.

Both Zoomerang and SurveyMonkey do not have advanced functionality like compound/delayed branching, PPT integration, banner tables, cross-tabs, TURF analysis etc.

While I realize that most of this is irrelevant to many users, one of the consistent themes (and why we are in business) that we hear from our customers is the breadth and depth of the tool itself is the primary reason they stick with us.

Now, I think you bring up two points. Obviously we can improve upon usability. No denying that. I'll put in a renewed effort in taking a critical look at usability and see if we can do things better.

The sheer volume of options (and they are indeed used by most of our clients) make is a balancing act between usability and functionality.

The second point that you bring up is interesting about the 100 responses. I can see us doing away with that limit altogether. We'll have to think about that a little more and see what we can come up with.

Anyways, I wanted to reach out and thank you for your critical analysis. This can only make us better/stronger.

Vivek Bhaskaran
Monday, January 28, 2008 4:52:31 PM (Pacific Standard Time, UTC-08:00)
Vivek: Thanks for coming over to share your view, much appreciated. Your comments indeed touch on an aspect of the services that I did not delve into: feature richness. I had a specific feature set in my mind and that is what I used to compare the various services. Looking back, you may well be right that your service has more options than the others. Indeed that could be one of the items contributing to the poor IMO usability (too much clutter due to too many options?). But, even so, the number of clicks to get anywhere is higher than expected and good use of AJAX would help in some cases.

I cannot complain about the limit of 100 responses since I only used your FREE service; my complaint was that I did not have access to any of the 100 responses at all after the 101st. Allowing access to the 100 and hiding anything over (like zoomerang) would have been ok, OR automatically closing the survey at 100 responses like surveymonkey would have been even better. Anyway, thank you for the free service, which in the end served its purpose ;-).

Good luck with future iterations of the UI.
Saturday, March 08, 2008 5:22:00 PM (Pacific Standard Time, UTC-08:00)
Hey Daniel,
I just wanted to reach out to you and let you know of a few changes that we've made (as a direct impact) of your blog entry.

I think this exchange can be a classic case study of how bloggers influence things that companies do. When you run any business, a lot of people complain about random things, but I really liked the way you presented your case and I felt obligated to take real action. I guess one of the benefits of being a small and agile company is that we can actually make things happen when we decide to.

Now, back to your points:
1) 100 Responses : We've done away with the limit altogether. So, the free licenses now get Unlimited Responses. This is the Web 2.0 economy and thats the way it is! I am sure you've seen Wired Magzines' "FREE" -- by Mark Anderson -- He got Long Tail right and I think he also gets "Free" right.

2) Usability AJAX -- We've made some significant changes to the development UI. If you have a minute check it out -- would love to get your take - but basically we've revamped our core infrastucture.
Sunday, March 09, 2008 2:33:00 AM (Pacific Standard Time, UTC-08:00)
I've noted that QuestionPro has removed the limit on responses although it is still a pain to edit/ prepare a simple survey.
Thanks for the tips Daniel!
Thursday, September 25, 2008 7:58:00 PM (Pacific Daylight Time, UTC-07:00)
Thanks for your quick overview. Tried them and find you are right on for the "average" poller just getting their feet wet. A note to the co-founder of question pro. This is a free 1 million dollar tip. Get a Mac, check out the interface and design for people. Not PC freaks, not programers - people. Grain of salt ;-)
Saturday, October 11, 2008 3:59:00 AM (Pacific Daylight Time, UTC-07:00)
@breezy - I wish it was that easy to make a million bucks...

Comments are closed.