By Tom Isler
(view original article)
How satisfied are meeting planners with the feedback they get from pre- and postmeeting surveys? Choose one: Extremely satisfied, mostly satisfied, neutral or not at all satisfied.
Survey experts would immediately spot at least five things wrong with that question. Planners who can't identify the five common survey errors—or, indeed, are dissatisfied with their ability to elicit actionable feedback from meetings-related surveys—should brush up on survey best practices, particularly given the importance now placed on measuring meeting results and proving the value of meetings, which often is determined by attendee surveys.
Stumped? First, the question asks a respondent to speculate how "meeting planners" in general feel about surveys, not about the respondent's firsthand experience. Such questions that rely on hearsay typically result in bad data and are discouraged by researchers.
Second, the question is vague. What is meant by being "satisfied" with survey feedback? Is the question asking whether planners receive positive feedback that gives them professional satisfaction (i.e., the meeting was a success)? Or is it asking if they're satisfied with the usefulness of the feedback, be it positive or negative? Questions in mail or Internet surveys in particular cannot be ambiguous, because there is no opportunity to clarify a question once the survey has been sent out.
Third, the question is "double barreled," meaning it asks about two things at once: pre- and postmeeting surveys. Imagine that a respondent is satisfied with premeeting surveys but dissatisfied with postmeeting surveys; how can he or she answer the question accurately? This should be two separate questions.
Fourth, the response scale is skewed toward positive responses. Respondents do not have the same gradation of choices to express dissatisfaction as they do to register satisfaction (there's no "mostly dissatisfied" to counterbalance "mostly satisfied"). Response scales should be balanced to get the most objective data.
Finally, the question doesn't allow respondents to indicate that they don't know the answer or the question is not applicable to them. If the question doesn't exhaust the entire spectrum of responses, some respondents will be forced to choose a disingenuous answer, spoiling the data.
"We find that planners, generally speaking, are not research experts," says Ira Kerns, managing director of MeetingMetrics in New York City, which offers survey and other measurement services to clients. That's not a criticism, he quickly adds, it's reality and unrealistic to expect otherwise. But planners who don't hire experts like MeetingMetrics or Red Bank, N.J.-based Exhibit Surveys often end up creating their own paper surveys to hand out on-site, or they use free or low-cost web tools, like SurveyMonkey.com or Zoomerang.com. In those cases, Kerns says, planners' lack of knowledge of survey science results in "a certain amount of ineffectiveness" of the endeavor.
Following are 15 more tips from various experts for creating better surveys.
1. Set objectives first
You have to know what your goals are before you start formulating questions, says Ian McGonnigal, an executive director for George P. Johnson, headquartered in Auburn Hills, Mich., who has helped conduct meetings-related surveys for IBM and Motorola. "A survey must be designed against objectives, because if it isn't, you're spending a lot of time and energy on something that's going to give you very little return," he says.
Figuring out specifically what information you want to obtain also helps keep the survey focused. If planners want to know how to improve next year's meeting, they should ask only about things they're willing to try or change. If, for example, the meeting must be held at the same venue next year, it's a waste of time to ask attendees to assess the location of this year's meeting—especially if nothing can be done about it. If next year's venue hasn't been determined, asking about location could be valuable.
2. Ask what you truly want to ask
This sounds obvious, but Kerns says one common mistake planners make is they don't articulate the question that will elicit the data they really want. For example, if an association planner is considering several different special events to add to next year's meeting and intends to charge extra fees for admission to those events, the survey should ask "for which of the following events would you be willing to pay?" instead of "which of the following events would you attend?" or "which additions would you like to see at next year's meeting?" The planner doesn't want to identify events that sound good but rather events that sound like they're worth the price of admission.
Similarly, getting attendees to rate satisfaction with various meeting components—the registration process, for example—isn't useful if the survey doesn't also provide a way for dissatisfied attendees to explain their complaints. Dissatisfaction with registration could stem from a technical problem, or lack of support staff or difficulty finding the registration desk. Planners shouldn't simply ask whether attendees liked or disliked various meeting elements—they also should ask why.
3. Stick to the point
According to Zoomerang's online survey tips, abandonment rates skyrocket after 11 minutes. There isn't widespread agreement about the optimal length of a survey, however. "It depends on how engaged people are," says Nancy Scherden, market research manager for Warrenville, Ill.-based Navistar Inc. "Is it meaningful and relevant to them?" People who have more invested in the survey's subject will be willing to spend more time answering questions.
General wisdom has it that surveys should be as short as possible, with each question correlated to an objective. "Nothing more, nothing less," McGonnigal says. "All nice-to-know questions should be eliminated."
But Kerns says some planners suffer from excessive self-censorship. "Too many untrained meeting planners make their surveys too short and lose many opportunities to gain key insights about needed actions," he notes.
4. Use branch logic
One way to make online surveys shorter—or at least appear shorter—is to employ a technique called "branch logic," which adds or eliminates questions based on earlier responses. For example, if an attendee indicates that she didn't attend the general session, a web-based survey using branch logic can jump ahead to the next subject, skipping over questions about the quality of the general session speakers. On the other hand, if another respondent indicates that he attended the session and then gave the speakers poor marks, branch logic can insert a question asking for an explanation—a question that wouldn't appear if the respondent gave the speakers high marks. Hiding questions until they're necessary to ask can give the survey the illusion of brevity, and in some cases, those extra questions won't be necessary to pose at all.
5. Organize well
William M. Trochim, author of The Research Methods Knowledge Base, an online textbook with a chapter on surveys (see resources sidebar at right), says surveys should have a logical flow, investigate one topic at a time and never start with open-ended questions that require survey takers to write out their own response. Kerns adds that planners should put the most important questions first, in case people abandon the survey in the middle. When asking multiple questions about a topic, Kerns suggests starting with general questions and then getting more specific.
6. Handle demographic questions carefully
Eric Eden, vice president of marketing for McLean, Va.-based Cvent, which offers a survey product to planners, says respondents tend to be shy about entering identifying demographic information about themselves. If planners already have a profile of the survey taker in a database, they needn't bother with demographic questions. If such questions must be asked, put them at the end of the survey, Kerns advises.
7. Get intensity data
Many surveyors make the mistake of asking yes-or-no questions instead of "intensity questions," according to Zoomerang. For example, if a planner wants to know what topics to cover at the next training session, surveys should ask how valuable attendees would find a proposed session ("extremely valuable," "somewhat valuable," etc.), not just whether a session would be valuable. That way a planner would know not only which topics garner interest but to what degree attendees want to learn about those topics. A session that gets fewer but more passionate votes might rightfully top a planner's priority list instead of another session that gets weaker support from more people. Without intensity data, planners might choose the session that simply got more votes.
8. Offer a midpoint
or survey designers, asking for answers based on a five-point scale vs. a 10-point scale isn't arbitrary. Odd-numbered scales will have a true midpoint (three on a five-point scale, for example), whereas even-numbered scales will force respondents to edge closer to one end of the scale or the other (choosing between two or three on a four-point scale, for instance, or five or six on a 10-point scale). Kerns says respondents who otherwise would choose a true midpoint tend to opt for the slightly more positive answer when a midpoint isn't available.
9. Achieve question balance
Surveys should have a mix of structured questions (multiple choice, yes or no, rank on a scale) and unstructured questions (write a response), Eden says. The best surveys will have more structured than unstructured questions.
10. Include a proper introduction
According to "What Is a Survey," an online resource for nonsurvey experts edited by Fritz Scheuren of the National Opinion Research Center at the University of Chicago (see "More Survey Resources"), all questionnaires should have an introduction that explains what the survey is about and why it's being conducted. Getting respondents to understand the value of taking the survey or the importance of the resulting research will help increase response rates.
11. Avoid junk filters
Designers of paper surveys don't have to worry about this, but creators of online surveys need to make sure their e-mails asking for participation don't get quarantined by spam filters. Nathaniel Estes, a Cvent account executive who hosts webcasts on survey design, says planners can test subject lines and e-mail invites to different e-mail addresses to see if the messages get intercepted. Avoid using words in capital letters in the subject line, as well as promises of "free gifts," dollar symbols, or multiple exclamation points—all trademarks of spam.
12. Offer incentives and reminders
Gifts or cash giveaways will boost response rates. A common snail-mail survey technique is to send a dollar bill with the survey, guilting respondents into returning the questionnaire. Promises of a Starbucks gift card, for example, upon completion of the survey is another option. Surveyors also can enter all respondents into a drawing for a prize, rather than promising each individual a gift.
Yet another technique to boost response rates is to send a reminder e-mail or postcard. Estes says he's seen examples of response rates doubling after sending out follow-up notes, which are another opportunity to remind the target of the importance of their input.
13. Use premeeting surveys
"The typical view of meeting measurement is the postmeeting survey," says Kerns. "The problem is, that tells you where you are, but it doesn't tell you where you were." Premeeting surveys set the baseline to measure the effect of the meeting. Suppose 90 percent of attendees indicate in postmeeting surveys that they have a positive impression of a certain product brand. What does that tell a planner about the meeting? Did the same percentage have that impression going in, meaning the meeting didn't cause a net change in opinion? Maybe only 75 percent thought well of the brand before the meeting, meaning the event was successful. On the other hand, maybe 98 percent of attendees had a positive brand impression going in, and that sentiment changed after brand managers revealed disappointing new product features. Postmeeting surveys only tell half of the story.
14. Send regret surveys
Eden recommends setting up a one- to two-question survey for people who decline invitations to meetings or events. Eden says Cvent uses such "regret" surveys to find out if people are still interested in the topic or product or service and simply can't attend the meeting, or if they aren't interested at all. "We get 50 percent more leads for our sales team because we offer that," he says.
15. Don't jump to conclusions
Eden warns against reaching erroneous conclusions. If only a small sample of attendees are surveyed, or if response rates are exceedingly small, planners shouldn't assume survey responses are indicative of the attitudes of the entire attendee base. If fewer than 10 percent of all attendees return surveys, it would be a mistake for planners to make changes based solely on that data, Eden says. There's no absolute cutoff for what percentage constitutes a reliable sample. "The higher percentage, the better," Eden says.
