Survey Writing Rules

Gradient approaches survey writing a little different than most research companies. Here, we actually value the sanity of our data science team, and try to design and program surveys with their best interest in mind. In turn, the data team will be able to rapidly turn around results, considering the dataset follows a consistent and expected format.

The generic variable format follows a [module_prefix]_[descriptive_name] with a snake_case specification.

• d_gender: Respondent Gender in the Demographic module
• m4_robot_parts: Question “Which of the following body parts would you replace with a perfect robotic version, if given the opportunity?” in the fourth module
• s_race : Respondent Race in the Screener module

Example Module 3: Working Professionals

• In case of multiple conjoints in a study, prefix them with different modules. The _cjt suffix should always be there, but the variable name can have additional explanatory context. E.g.,
  • m2_brand_cjt
  • m4_person_cjt

• m[x]_cjt : Regular conjoint question. The block this is inside of will have a Loop & Merge attached to it
  • Note; Loop and Merge variables should be the task number (e.g., 1 - 12).
• m[x]_cjt_none or m[x]_cjt_buy: Follow up question inside the same block to assess whether the respondent would actually purchase the displayed choice profile.

Example Module 2: Alternative Certificates

• m[x]_md : Regular MaxDiff question. The block this is inside of will have a Loop & Merge attached to it.
  • Note; Loop and Merge variables should be the task number (e.g., 1 - 12).

Example Module 5: Worst Neighbor MaxDiff

• m[x]_seg: One of the segmentation battery items. Will most often be a grid of questions, Qualtrics will automatically append a digit [#] at the end.
• If you have multiple batteries, do not include a digit to mark these. Instead, use spelled out numbers or describe the battery. E.g., m3_seg_one and m3_seg_two or m3_seg_attitudes and m3_seg_behaviors.

Example Module 5: Worst Neighbor MaxDiff

• m[x]_b_awareness: Brand awareness question
• m[x]_b_consider : Brand consideration question
• m[x]_b_purchase : Brand purchase question
• m[x]_b_nps : Brand NPS score question
• m[x]_b_[xxx] : Other related questions
• m[x]_b_attr: A brand - attribute rating grid. Should be in a separate block where we loop over all brands the respondent is aware of.

Example Module 2: Alternative Milk Brand Perception

Every list experiment for which you test a sensitive item requires 3 variables in Qualtrics: (1) the respondent’s experimental group, (2) the indirect question, and (3) the direct question. Each of those (and associated variable naming rules) is described in more detail below.

1. Experimental condition a. An embedded data field that classifies respondents' exposure to experimental or control bundles b. This variable is necessary so that the Product team knows how to process data and run analysis workflows c. m1_list_1_group
2. Indirect question a. In other words, the number of statements respondents agree with from bundle of control items (+ the sensitive item for the experimental group) a. This variable should be used for both experimental and control conditions b. Use display logic and HTML to ensure proper depiction of the item bundle and response options are available c. m1_list_1_bundle
3. Direct question, aka public opinion a. Follow-up, direct question in a later block that allows for public versus private comparisons b. In other words, the agree-disagree question or battery of items to the sensitive statement c. m1_list_1_direct

• Please number the indirect and direct questions (#2-3 above) accordingly so that the Product team knows how to model the difference between public and private opinion

Example Module 1: Gun Violence & Social Security List Experiments

Example Module 1: Retirement, Social Security, & French Protests

• If your survey has multiple conjoints or MaxDiffs, you can simply prefix them with different modules and context in the variables. The aforementioned suffixes should always be present e.g.,:
  • m2_holiday_cjt
  • m4_drinks_cjt

In other companies, questions are often referred to with a number, e.g. Q3, or eventually, Q87. We don’t ever want to see this at Gradient, given that it produces a great deal of confusion.

• It’s impossible to remember what number refers to what question
• The order or numbers change when you want to insert new questions in your survey
• Back-end code becomes unreadable, and it’s easy to make mistakes when all variable names look like each other

We refer to a module as a group of related questions. This will indicate that all questions with the same prefix belong together, and we can easily manipulate or group them together.

Example

1. A demographic module will have a d_ prefix to all variable names inside that module
2. The fifth module in a survey has a m5_ prefix to all variable names inside that module
3. A screener module will have a s_ prefix to all variable names inside that module

One of the most important things we can do to help the data team, is naming a variable by describing the question. This way everyone can remember what the question refers to and easily access it in the data.

The benefit becomes clear when you consider the following prompt:

“Hey data scientiest, would you mind giving me a breakdown of that one question we have on why you would not be willing to replace a body part with a perfect robotic version?”

• If the variable name is simply Q54, it means that the data scientist will need to open up the survey document, search for the robot text, identify the question number and can then start writing code.

• If however, the name is m4_robot_why_not, the programmer can simply start typing dataset$robot and get the variable options related to the robot questions.

Example

1. d_education - What is the last level of schooling you have completed?
2. m2_awareness - Which, if any, of the following brands are you aware of? Please select all that apply.
3. s_race - With which racial categories do you identify? Please select all that apply.
4. m5_neighbor_complaints - Have you ever had complaints about your neighbors for any of the following reasons?

Some of the R functions we use on the backend use the trailing numbers to identify multiple response or grid questions. Qualtrics automatically adds _1 …. _n at the end of these, and is sometimes the only way for our data team to know that a set of variables are related. For example, the “With which gender do you identify” question is a multiple response, which will show as follows in our dataset:

Therefore, if you program two similar questions like this:

Our processing function will try and merge these together as a multiple response or a grid question. However, it’s clear that these are single select questions that are somewhat unrelated.

In an ideal world, these should be named as such:

Example

1. m4_robotic_1 -> m4_robotic_replace
2. m4_robotic_2 -> m4_robotic_why_not

If you still insist on using numbers, please replace them with a spelled out version:

1. m4_robotic_1 -> m4_robotic_one
2. m4_robotic_2 -> m4_robotic_two

Typically, open-ended responses aren’t suitable for the same type of analyses as questions with response options, mostly given the large number of possible answers. In order to differentiate between the two it’s adviced to provide an _open or _text suffix to these variables.

By default, an open-ended text response as “other” for a Single or Multiple choice question will get _text appended by Qualtrics.

Example

1. m2_review_open
2. m3_multiple_choice_text