Online surveys have been an important part of the lives of marketers and researchers alike. However, there are a few important things to consider so that you can ensure there are no unpleasant surprises and you obtain the data you need on time, in detail and, most importantly, on budget. Below are my top six priorities to keep in mind.
It goes without saying that formulating the questionnaire well is crucial to the quality of the respondent answers. Being too vague is a common slip; expressions such as ‘rarely’, ‘often’, ‘big’, ‘small’ or similar arbitrary adjectives are confusing. Respondents have to perceive the scale in the same way. Having broccoli four times a week may be “very often” for you and “rarely” for the person next to you, so make sure you always quantify vagueness out of the survey.
Another important matter is questionnaire length and complexity. Ask yourself, are you really willing to sit down for 30 minutes answering questions about a product or service. Do you enjoy huge grids with small dots to click on? I guess your answer is “no”. Don’t forget respondents are people like you; putting them through something you wouldn’t go through may not get the best results.
Since mobile devices are such an important part of our lives, you would expect that a portion of your eligible respondents would access online questionnaires via their smartphone or tablet. A big ongoing argument in the research community is the device-compatible versus device-agnostic.
Do we make the survey compatible with all devices to allow full access, but end up showing the questions differently on different platforms, or do we show the desktop version on all devices and hope for the best, even if certain questions are not seen well on the smaller smartphone and tablet screens?
At JTN Research, we have done several research-on-research surveys to solve this riddle. Of course, output varies per market and age group, but results indicate that for the time being around 70% of the respondents access online surveys via laptop or desktop. Mobile users tend to give more consistent data when the questionnaire design has been adapted to fit their screen best. Mobile respondents simply cannot be bothered to scroll down, or even worse, sideways, so our recommendation would be to go with the device-compatible approach.
In the future, we expect the research industry will come up with more innovative survey design objects to make device differences irrelevant to the respondent experience. The first good examples are already out there.
The respondent perspective
Do not expect respondents to know too much about the product, or care too much for that matter. Consumers are bombarded with thousands of messages every day so don’t expect them to see only your ad. Give enough details in your questionnaire – put yourself in the respondent’s shoes.
Also talk to your sample provider about the way participants are addressed. The less information they receive beforehand, say in the survey invite or welcoming pages, the more objective people will be in your survey. Of course, there are professional standards, established by organisations such as ESOMAR, where best practices and codes of conduct are set and there are few necessary elements each survey has to contain.
Source of online sample
Many things have been written about the so-called river sample, which is freely collected online. That sounds like a fascinating prospect – attractive pricing and probably good online coverage – but ask yourself, ‘do you need 100% unique respondents, who paid attention to the questions you asked?’. I know I do. I also need to be sure that male respondent from Poland is really 37 years old and lives in Krakow with his significant other, that they have been together for a long time but decided not to get married. Trustworthy online panel providers know a lot about their panel members. What if I tell you with river sampling you could be the Polish man, then a teenage woman from Warsaw and an elderly lady from Katowice, all in the same survey?
Some will say there are technical precautions one can take to avoid duplication. That may be true, but nothing beats a pre-targeted online panel with its quality procedures in place, at least not in the foreseeable future. The panel provider knows and can easily check huge amounts of background information for any member of their panel. That background information is not only technical, but personal as well and the two are overlapped to ensure you know the profile of every single person who is taking your survey. So my advice is to go with the providers of a proprietary panel that can guarantee their quality checks are efficient and proven in time.
Choosing the right sample provider
That sounds like common sense, but how do you know? Really, how do you weed out the poor performers and get the star players, especially if you need to run fieldwork in a country you have little experience with. Do you go with global panel providers or only seek the local experts? I would say trust your instinct.
Do the providers respond in detail, do they adapt to your needs and requirements or try to fit your project in their own box? They need to know the specifics about the markets they offer: in this respect maybe you would like to go to local experts, especially in the relatively smaller markets. Always check their panel portals in the relevant market. If they say the online panel is proprietary, they will have a website where respondents log in, and the portal would not block you because you are trying to access it from another country; 95% of the time the block is upon registration, not portal entry.
Like anybody doing research, we all want to have the best sample balance and this happens only if you set your quotas right. That being said, you may undermine the feasibility of your survey by applying too many quotas and breaking them down into the smallest pieces.
Over the years, I have seen a nationally representative sample for a huge country such as Russia with seven big region quota cells and an overly detailed attempt for nationally representative sample in the Czech Republic with 14 region cells. Mix that with another four to five quotas and then try to achieve the total number of interviews needed, having in mind the actual incidence rate, dropouts and so on: go on I dare you. In other words, make sure you are not missing the forest for the trees when defining your quotas. Sometimes doing less is better.
There are of course many other important matters to consider when going about your online survey; some of these are common across all methodologies, others are online specific. The most important take away is to find an online provider you can trust and cooperate with them to find the best way to achieve your survey objectives. In the wise words of Woodrow Wilson: “I not only use all the brains I have, but all I can borrow.”