Methods matter

The quality of online research panels depends on the recruitment method used.

Online research is increasingly becoming the mainstream methodology of choice for many researchers. The potential benefits for researchers are numerous and include more timely data collection, more efficient targeting of speciality or low-incidence groups, cost-effectiveness, use of graphics, video and audio, the ability to communicate detailed questions and concepts, and the automation of some parts of data validation. 

There are also benefits to respondents, such as the convenience of taking the survey at a time that best suits them, the ability to start and stop a survey and greater anonymity, which can encourage more candid responses. 

These benefits often make online research an attractive and effective choice. However, like other modes of data collection, online research requires scrutiny by the researcher to ensure the highest level of quality in the sampling and data collection process. Some of the greatest challenges in research are finding and reaching qualified respondents, gaining cooperation, confirming qualified respondents and ensuring well-considered responses.

The method of recruitment of online panel members is critical to the quality of the panel. If the panel is to meet a researcher’s needs for a study requiring a general audience of consumers, it’s important to evaluate whether the panel’s recruitment method draws from a representative audience of consumers.

Likewise, if a researcher’s project requires business professionals, the panel’s recruitment methods should draw from a universe of business professionals. Ideally, a panel should represent a diverse sampling of the population under study. Panel member recruitment methodology is a key distinction among online panels. There are essentially two methods for recruiting for an online panel – “open-source” and “by-invitation- only”.

“Open-source” recruitment

Intercepting people as they use the internet through ads is known as “open-source” recruitment. “Open-source” online panel recruitment may be defined as a method of allowing any person who has access to the internet to “self-select” and enrol into a market research panel. This provides the benefit of building a panel quickly with people who are internet-savvy and responsive to online advertising.

However, a key drawback is the lack of control over who is recruited. A panel with “open-source” recruitment may sign up millions of web users who share similar characteristics, but may include only people who are responsive to web ads and/or seek out opportunities to join an online panel by using search engines.

In many cases, “open-source” recruitment leads to an overabundance of panel members who participate in many different panels and complete an inordinate amount of surveys. They are known in the industry as “professional survey takers” – people who sign up to take hundreds of surveys to enter prize draws or other types of incentives.

Primary concerns with professional survey takers are, first, they can give false or misleading information in an attempt to get through a survey quickly without regard to providing well considered responses and, second, professional survey takers can make up a disproportionate amount of survey responders, leading to biased and unrepresentative research data. To the detriment of the industry, some websites have been developed to recruit people to sign up for several panels at one time.

A study conducted by ComScore Networks in 2005 found that more than 30% of all online surveys are completed by less than 1% of the population. The study also found that professional survey takers respond to an average of 80 surveys over a 90-day period – with some completing several surveys a day.

However, it is important to consider that not all online panels are comprised of professional survey takers. This is why it is so important to understand the recruitment methods that are used by an online panel before engaging them in the research process.

“By-Invitation-Only” recruitment

The other method used for recruiting respondents to an online panel is the “by-invitation-only” method. “Closed” or “by-invitationonly” online panel recruitment may be defined as a method of inviting only pre-validated individuals, or individuals who share known characteristics, to enrol into a market research panel.

Most often, this is achieved by inviting customers from a group of large, well-known brands that collectively have a large, diverse base of customers in a given population, such as general consumers or business professionals. The “by-invitation-only” method enables a panel company to recruit people with specific demographics that meet a client’s needs for a representative sample of the general population, or to target speciality groups.

In a recent parallel tracking experiment, e-Rewards contrasted “open-source” sample/ respondent recruitment quality with “by-invitation- only” or “closed” sample/respondent recruitment quality. In the experiment, e-Rewards enrolled 38,162 panel members into a test group using an “open-source” enrolment methodology to compare to its “by-invitation-only” panel members.

The results are highlighted in the chart (right), which shows some interesting differences.

For example, a total of 73.7% of “open-sourced” respondents were female, compared with 47.6% of “by-invitation-only” respondents. A more even male-to-female ratio generates an accurate and realistic overview of the general population, which delivers more accurate results that researchers seek.

Using a “ by-invitation-only” recruitment method enables a panel provider greater control over who joins, and greatly reduces the likelihood of professional survey takers becoming panel members.

One particular area that requires attention with this approach is that the panel composition is dependent on the people who are invited to join it and may be biased by customers of a specific recruitment source. Thus, it is important that “by-invitation-only” panels ensure a large number of diverse recruitment sources by working with companies in a number of different areas to make certain there is a balanced representation in the panel.

CV Craig Stevens
Executive vice-president, sales and business development e-Rewards Market Research 

2008 Executive vice-president, global sales and business development, e-Rewards Market Research
2006 Senior vice-president, sales and business development, e-Rewards Market Research
2004 Vice-president, sales and business development, e-Rewards Market Research
2003 Territory manager, Surgi-Care, Massachusetts
2002 Director, sales and client development, Greenfield Online, Connecticut
2000 Vice-president, marketing, ReliaQuote Insurance Services, Virginia
1996 Senior director, America Online/Digital Marketing Services, Virginia

Contributions by Terry Sweeney and Ben Hogg, both of whom are vice-presidents of e-Rewards Market Research