M303 Questionnaire Design
Where we are
Define the marketing decision, research objectives, and questions.
Choose the evidence and method needed to answer those questions.
Questionnaire design turns constructs into questions respondents can answer.
Use the responses to produce decision-relevant evidence.
Learning Objectives
Understand the types of question formats and the advantages and disadvantages of each.
Learn how to write clear, answerable questionnaire questions.
Understand the issues to consider when designing questionnaires.
Understand the typical layout of a questionnaire.
Previous-class review
Mental states used by individuals to structure their behaviors are called attitudes.
T
Intention or action component refers to a person’s information about an object.
F
When there are several alternatives, knowledge component is expressed in terms of preference for one alternative.
F
Intentions depend on buying habits and planning horizons.
T
Constructs are the specific types of concepts that exist at higher levels of abstraction and are not directly observable but can be measured indirectly through verbal expressions and/or observed behaviors.
T
A description of a construct at the theoretical level, stated in terms of unobservables is called the operational definition.
F
Nominal level refers to labeling and order.
F
The level of measurement increases from nominal level to the ratio level.
T
The range of statistical and mathematical analyses that can be conducted is greatest if one uses the ratio scale compared to all other three measurement scales.
T
Previous-class review (continued)
Rank your preferences for the following attributes in making a car purchase decision (1 for the most important and 4 for the least important)
Price -----------
Safety -----------
Design -----------
Fuel economy ------------
Are you?
1) Male, 2) Female
What is your annual income before taxes? ___
Nominal
Ordinal
Interval
Ratio
If provided with a coupon, how likely would you be to purchase shampoo X?
Definitely will buy
Probably will buy
Maybe will buy
Probably will NOT buy
Definitely will NOT buy
Questionnaire design in the research process
Problem
Formulation
Research
Design
Data
Collection
Data Analysis &
Recommendation
Secondary Data
Primary Data
Observational and Qualitative Methods
Surveys
Sampling
What are the question formats?
Decide how much freedom respondents should have in answering the questions
Open-ended
Closed-ended
Question format: Open-ended questions
Respondents answer in their own words
Respondents might recall information without limiting themselves to the listed alternative choices
Researchers might use open-ended questions to identify possible response categories when they cannot anticipate the full range of responses
“How was your shopping experience at Kroger today?”
“It was a hassle”
“It was slightly better than last week”
“It would have been great if my son hadn’t gotten sick”
What are the pros and cons of open-ended questions?
Pros
Answers are in respondents’ own words
Can provide new or different insights
Can reveal response alternatives the researcher did not anticipate
Cons
Can be difficult to interpret
Editing and coding can be challenging
Potential for interviewer bias
Answers may be incomplete or insufficient
Lower response rates
Question format: Closed-Ended Questions
Respondents are given a finite number of responses from which to choose
Two major ways to ask these kinds of questions
Choice from a list of responses
Appropriate single-choice rating on a scale
What are the pros and cons of closed-ended questions?
Pros
Easy and accurate data coding and entry
A fixed, manageable set of response options
Listed alternatives may help respondents recall information
Limited interviewer bias
Cons
Researchers must generate the alternatives
Limited freedom to answer in respondents’ own words
What should you avoid when writing questions?
Checklist of the things that you need to avoid:
Complexity: use simple, conversational language
Leading and loaded questions
Ambiguity: be as specific as possible
Double-barreled questions
Making assumptions
Burdensome questions
Examples of Problematic Survey Questions
How likely would you be to buy a Bluetooth speaker with 11 x 6.5 dimensions, that utilizes wireless connectivity for streaming and infrared technology for the remote?
What’s the problem?
How to Fix?
How likely would you be to buy a Bluetooth speaker that is the size of a textbook, has wireless connectivity and a remote?
Do you prefer a hamburger that is grilled on a hot stainless steel grill or cooked by passing the raw meat through an open gas flame and heated in a microwave oven before serving?
What’s the problem?
How to Fix?
Do you prefer hamburger freshly grilled or pre-prepared?
Complexity
Complexity
Examples of Problematic Survey Questions
Do you believe that private citizens have the right to own firearms to defend themselves, their families, and property from violent criminal attack?
What’s the problem?
Do you believe that a ban on the private ownership of firearms would significantly reduce the number of murders and robberies in your community?
What’s the problem?
How to Fix?
What is your stance on the regulation of firearms?
http://www.cc.com/video-clips/xd8n18/the-daily-show-with-jon-stewart-poll-bearers
Leads respondents to say yes!
Leads respondents to say no!
Yes
No
Undecided
Yes
No
Undecided
Should be regulated by government
Should NOT be regulated by government
Undecided
Examples of Problematic Survey Questions
How do you generally spend your free time – watching TV, or what?
What’s the problem?
How to Fix?
Make it fully open-ended, or offer more response options
Should we increase taxes to get better housing or schools, or should we keep them about the same?
What’s the problem?
How to Fix?
Should we increase taxes, or should we keep them about the same?
Loaded question: causes the respondent to think in a certain way
Loaded question: causes the respondent to think in a certain way
Examples of Problematic Survey Questions
How many times per month do you visit a fast-food restaurant?
○ Never
○ Occasionally
○ Sometimes
○ Often
What is the problem?
How to Fix?
How many times per month do you visit a fast-food restaurant?
○ less than once
○ 1 or 2 times
○ 3 or 4 times
○ more than 4 times
Ambiguous wording in the response categories.
Examples of Problematic Survey Questions
Which toothpaste do you use?
Clearly define the issue
Which brand of toothpaste have you personally used at home during the past month? (in case of more than one brand, please list all)
Is the distribution of snack foods adequate?
Use simple words
Are snack foods readily available when you want to buy them?
Ambiguous question: which media, and for what purpose?
What media do you rely on most?
Television
Radio
Internet
Newspaper
Other (please specify)
What is the problem?
How to Fix?
What media do you rely on most to get your news?
Television
Radio
Internet
Newspaper
Other (please specify)
Examples of Problematic Survey Questions
Ambiguous question: TV viewing compared with what time period?
Please indicate the statement that best describes your family’s television viewing during the boycott of Channel 7.
We did not watch any television programs on Channel 7
We watched hardly any television programs on Channel 7
We occasionally watched television programs on Channel 7
We frequently watched television programs on Channel 7
What’s the problem?
How to Fix?
Compared to your regular TV watching habits, how would you rate your viewership of Channel 7 during the boycott?
Examples of Problematic Survey Questions
Less Than Before
About the Same
More Than Before
-5
-4
-3
-2
-1
What income group do you belong to?
What’s the problem?
How to Fix?
What is your annual household income before taxes?
_________________
Under $10,000
$15,000 - $30,000
$30,000 - 55,000
$55,000 - $70,000
Over $75,000
Examples of Problematic Survey Questions
Response categories do not cover
the entire range of income levels.
Should Macy’s continue its excellent gift-wrapping program?
Yes
No
What’s the problem?
How to Fix?
In the past year, how many times did you use Macy’s gift-wrapping program?
___________
Examples of Problematic Survey Questions
The question assumes that the
respondent is a user of the program; in addition, it assumes that the program is excellent.
Examples of Problematic Survey Questions
Check the line that best describes when you purchased Aspirin most recently:
__ within the past three months
__ 4-6 months ago
__ 7-12 months ago
__ Over 12 months ago
What’s the problem?
How to Fix?
Did you ever buy any Aspirin?
Yes
No
If yes, when did you last purchase Aspirin?
__ within the past three months
__ 4-6 months ago
__ 7-12 months ago
__ Over 12 months ago
The question assumes that the
respondent has purchased Aspirin.
Double-Barreled Question
Are you satisfied with the present system of handling “closed reserve” and “open reserve” readings (Are enough copies available? Are the required materials ordered promptly? Are the borrowing regulations adequate for students’ use of materials?)
Yes / No
What’s the problem?
How to Fix?
Please rate your satisfaction with each of the issues regarding the library reserve system using the scale below:
Number of copies available
Promptness of ordering the required materials
Borrowing regulations regarding students’ use of material
Examples of Problematic Survey Questions
Not Satisfied At All
Neither Satisfied,
Nor Dissatisfied
Very Satisfied
What is wrong with this recall question?
Do you recall any commercials on the program? What were the brands advertised?
What’s the problem?
It is a burdensome recall question.
How could you fix it?
Do you recall whether a brand of beer was advertised on that program?
OR
I will read you a list of beer names.
Can you identify the beer that was advertised on the program?
Examples of Problematic Survey Questions
What should you consider in questionnaire design?
Number of Response Categories
Generally five to seven categories
Length/number of questions
Do not ask too many questions
Order of response categories:
Can affect responses
What factor influences your fast-food restaurant choice most?
○ Convenient location ○ Quality of food
○ Menu selection ○ Fast service
○ Reasonable prices ○ Brand name
○ Cleanliness
How to avoid this problem?
What should you consider in questionnaire design?
Concerns the handling of “don’t know” and neutral responses
May be advisable for the interviewer to have an additional “no answer” category to identify those respondents with “no answers” correctly
What is the typical layout of a questionnaire?
Cover letter/Introduction
Screening questions at the beginning.
“Do you own X?”
Interesting and simple questions next (“warm-ups”).
“What brand of X do you own?”
Involving questions in the middle.
“Please rate X on the following 10 characteristics using the scale below.”
Possibly insert “prompters”
“Now that you have helped us with those comments, we would like to ask a few more questions.”
Threatening and sensitive questions near the end
Classification (demographic) questions in the end
Finally, have a “Thank You”
Cover Letter Example
Dear participant:
We are conducting a nationwide survey about the advantages and disadvantages of using three new steel products. Your answers will help manufacturers understand executives’ views and design products that meet your needs.
Your name appeared in a scientifically selected random sample. The questionnaire is brief and easy to complete, and your answers will remain confidential and be reported only in aggregate.
If you would like a copy of the study findings, provide your contact information when returning the questionnaire. Please return it at your earliest convenience. Thank you for your help.
Sincerely,
Research Director
P.S. The enclosed dollar is a small token of appreciation.
Questionnaire design checklist
Format of questions in a survey
Open vs. closed-ended questions, pros and cons of each
Checklist to write good questions
Things to consider in designing a questionnaire
Typical layout of questionnaires
Qualtrics is available at IU
Any Kelley student can use Qualtrics:
https://iu.qualtrics.com/
After you read and accept the terms of the service, you will be ready to create your own surveys
Qualtrics provides current online documentation and training at https://www.qualtrics.com/support/
Use IU Qualtrics to build and administer the questionnaire.
Use the course Web Tools for later hypothesis testing and analysis.
Preview: Sampling Fundamentals
Sampling Fundamentals
Creating your own questionnaire
Given your team’s marketing problem/opportunity, choose the variables of interest (constructs), and determine what to measure (using a table is strongly suggested!)
What kinds of scales would you use to capture the required information?
Make sure every measured variable will help achieve a research objective.
To sum up
Choose open- or closed-ended formats based on the information respondents need to provide.
Write questions that are clear, neutral, focused, and realistic for respondents to answer.
Sequence and format the questionnaire so participation feels natural and the resulting data are usable.
Use IU Qualtrics to administer the survey, then connect the responses to the research objectives.
Next class
Sampling Fundamentals
Check Canvas for current files, reminders, and practice-assignment deadlines.