Developing the Instrument

Developing the survey instrument is the first stage in the newsletter development cycle. Some stages of the newsletter development process require creativity, some are more programmatic/task-based, and some are analytical. This is perhaps the only stage that requires using all three of those skills in one set of tasks.

This is the first step of the newsletter cycle, which usually begins during the last week of the month. During an ideal fielding week, the timeline is as follows (we will be covering the bolded portions in this section):

• Monday: Confirm topics and start developing survey instrument
• Tuesday: Finish survey instrument and begin programming
• Wednesday: Finish programming and launch data collection
• Thursday: Begin analysis of interim data
• Friday: Finish analysis script

Each Trendlines survey contains about five modules (or topics), each with their own set of questions. Usually, developing the modules will be shared by two or three people who will each write one or two modules. The ultimate goal of the five modules is to be able to pull out four full-sized Trendlines stories, and two short TrendSegments write-ups (which can come from its own module or be paired with a full-sized story that made use of different questions in the same module).

There is no prescribed set of topics or methods for our Trendlines surveys. However, some guiding questions that should drive the topic decision are: 1) Is the topic likely to produce interesting results? “Interesting” can mean a lot of things, but in general, ideas tend to be more interesting with more careful attention paid to how the data will be analyzed. For example, “What genre of music do Americans like best?” is somewhat interesting, but “I hypothesize that the genre of music Americans like will change conditional on how old they were when that music was most popular” is more of a developed idea that sets us up to share interesting results regardless of whether the hypothesis is confirmed. 2) Is this the right time (or at least not the wrong time) for this topic? For example, it may be interesting to ask what the hot gifts for Christmas are going to be, but we probably want to ask this in the December survey instead of, say, March. It is important to remember that the stories will cover a month of events. And 3) Does this show off Gradient’s survey design/data science chops? While this may be less of a prerequisite than (1) or (2) for designing a module, it is important to consider whether there’s a novel or methodologically interesting way to answer the research question. After all, Trendlines is a way for us to demonstrate our abilities, since much of our awesome research has been embargoed or white-labeled.

Since this is a creative process, there is no “right” way to go about pursuing an idea. Some methods for brainstorming interesting module ideas can include:

• Check the news. One of the things that makes Trendlines fun is the ability for Gradient to weigh in on interesting things happening in the world in something close to real-time. (Example: billionaires in space).
• Check Airtable: This Airtable base has ideas for Trendlines modules, both from Gradient staff and public submissions.
• Pursue a topic or show off a method one of our clients (or potential clients) might be interested in.
• Think of something relevant for that time of year. (Example: falling for fall).
• Crowdsource! Ask your friends who subscribe to Trendlines what they’d like to learn more about.

One other thing to keep in mind when deciding on a topic is the plan for analysis. What do you envision the analysis looking like? For example, do you want to cut the results by political party? What is your hypothesis for how the results will turn out? Keeping these research questions and hypotheses in mind will help you write a clearer module topic specification.

It is a good practice to write a module topic specification on JIRA to confirm with others who are working on the instrument that the topic is a good way to go. Once everyone has decided that the topic is interesting/worth pursuing (and hasn’t already been done in a previous Trendlines) it is time to write the module.

Writing the modules is where you will translate your broad topic ideas into concrete questions with programming and analysis specifications. This will occur in the newsletter instrument document (see the newsletter survey template, which should be copied into the correct Drive folder location: Brand > Trendlines > Surveys > [YEAR] Surveys and Syntheses > [CREATE yyyy-mm FOLDER]).

Because survey-writing best practices are to be documented elsewhere, this section will not include a full outline of best practices. If you are unsure about the best ways to write your survey instrument, check out previous months of Trendlines survey instruments in Google Drive.

In general, we would like to aim for the survey to take about 10 minutes for a reasonable respondent to complete. A few minutes in either direction is fine. The survey should be written with this goal in mind, and it should be mindful of the cognitive load of each module. For example, a conjoint is going to be much more cognitively taxing than a list experiment.

It is very important to include a clear analysis spec for every module you are responsible for. A good analysis spec requires thought, planning, and specificity. After reading the analysis spec, the person conducting the analysis should ideally be able to complete the analysis for the full survey without further instruction (although, of course, the team should be in communication when needed).

The survey instrument document contains an analysis needs checklist near the top. Take the checklist as far as it will get you for your module, and then clarify anything further in the “Notes” column of the checklist table. Be specific about variables (name them as they are labeled in the document) and be as clear as possible about the analyses you envision. When describing the analysis, pay special attention to the type of variable being collected (e.g., single-choice, multi-select options, etc.) and how that affects analysis.