Survey Workflow

This page documents Gradient’s standard, end-to-end workflow for survey projects, from initial research question through to final, analysis-ready data. Nearly all Gradient projects involve designing and fielding a survey that satisfies objectives defined in the SOW, and doing so requires a coordinated workflow across research design, programming, fielding, and data preparation.

The authoritative execution steps for each phase of the workflow live in the Jira Issue Types and Smart Checklists, attached to Survey issue types. This documentation explains how the workflow is structured, why each phase exists, who owns it, and what “done” looks like.

Survey work at Gradient progresses through the following phases:

Survey: Writing → Survey: Programming → Survey: Soft Launch → Survey: Data Collection → Survey: Data Cleaning

Each phase corresponds to a Jira issue type and represents a meaningful shift in responsibility, milestone, and type of work to be done.

Contractors, Product, and other team members may be involved at specific points, but accountability for moving each phase forward sits with the Research Manager.

Owner: Research Manager

Objective 🏁
Produce a final, client-approved survey instrument that is analytically sound, operationally feasible, and ready to be programmed.

Survey: Writing is the most consequential phase of the workflow. Decisions made here shape feasibility, cost, respondent experience, and analytic flexibility downstream. The purpose of this phase is not just to “write questions,” but to translate research objectives into a coherent measurement instrument that can survive real-world fielding.

Before drafting begins, the Research team ensures the project is structurally ready for survey work. This includes internal alignment on analytic goals, clarity on client expectations, and confidence that staffing, timing, and sampling assumptions are realistic.

In practice, this phase typically involves an internal kickoff, pre-mortem analysis planning, updates to the Project Playbook, confirmation of subgroup requirements with the client, and early assignment of contractors, such as survey programmers and survey reviewers. The specific steps and sequencing live in Jira; the goal is to surface risks early.

During this phase, the RM should also line up contractor support early (specifically confirming a survey programmer and survey reviewer) so the project can move quickly once the instrument is locked.

Gradient works with a wide range of clients, categories, and research problems. Even experienced Research Managers will occasionally face blind spots around terminology, category norms, or assumptions embedded in a client’s mental model. Survey outlines exist to address this uncertainty before significant drafting effort begins.

A survey outline is recommended when:

• The category or product is unfamiliar
• Module structure or prioritization is still fluid
• Sample definition or screening criteria require client confirmation
• Early alignment can prevent wasted effort later

Outlines are not mandatory for every project. If objectives are exceptionally clear and the Research Manager has deep category expertise, it may be appropriate to proceed directly to a full draft.

A successful survey outline is not a rough draft of the final survey. Instead, it is a decision-making and alignment tool. At a minimum, it should:

• Identify the major modules expected in the final survey
• Briefly describe the purpose of each module in terms of the research question it will answer
• Include example questions to orient the client (often without full response options or programming language)
• Provide explicit space for client input where clarification or decisions are needed
• Include the screener as the first module to align early on sample definition and eligibility
• Clearly articulate tradeoffs where client feedback is requested

Module purpose descriptions should focus on what insights the module will generate, not on detailed explanations of experimental methods the client has already agreed to.

Once client input on the outline is incorporated, the Research Manager moves to drafting the full survey instrument. This typically involves creating a new internal survey document using Gradient’s standard templates and writing complete question text, response options, and instructions.

At this stage, the survey document becomes the primary working artifact for:

• Measurement design
• Sampling plan
• Programming instructions
• Review and revision

The Project Playbook should also be kept current throughout survey writing, including (at minimum) project management specifications and analysis specifications.

Early in the full survey draft, the Research Manager should complete the sample parameters grid. This grid is the single source of truth for how the survey will be fielded and interpreted.

By the time a first full draft is shared with the client, most sampling decisions should be known, including:

• Target population definition
• Sample size targets by audience
• Soft quotas
• Source of benchmarks

Details should be described as needed so that any Gradient team member can understand exactly how data collection is intended to work.

Full survey drafts differ from outlines primarily through the addition of programming instructions. These instructions communicate logic, structure, and data requirements to the programmer and must be clear, consistent, and interpretable.

Programming instructions typically fall into three categories:

• Display logic (who sees what)
• In-question instructions (how a question functions)
• Embedded data (how respondents are assigned, categorized, or randomized)

The exact syntax and formatting conventions for these instructions are documented separately in Variable Naming Conventions and related programming guidelines. The goal at this stage is not to document every possible instruction pattern, but to ensure the survey document is unambiguous and executable.

Survey writing is inherently iterative. Internal review focuses less on wording polish and more on logic, flow, analytic usability, and downstream feasibility. Once internal review is complete, a client-facing version of the survey is shared for approval.

Before programming begins: Final client approval must be received for survey and requested subgroup cuts The survey document should be locked (view-only for the client) The lock date and approver should be recorded in the Project Playbook

After this point, changes should be avoided unless strictly necessary.

Once locked, the RM prepares a clean programming document for handoff to the programmer, access permissions for relevant team members/contractors, and clear quota/programming callouts (including oversample for expected cleanout rate).

Before handing off to programming, ensure the downstream Jira issues exist and are assigned (at minimum: Survey: Programming, Survey: Data Collection, and Synthesis: Ghost Deliverable).

Owner(s): Contractor, Research Manager

Objective 🏁
Translate the approved survey instrument into a correctly programmed, testable link.

Programming assignment depends on both survey complexity and data provider constraints. In general, Gradient prefers to use its own programmers for complex designs (e.g., conjoints), but some providers require or strongly prefer to program surveys themselves (e.g., YouGov).

Because programming practices vary by provider and platform, task-level programming steps are handled in Jira, not exhaustively documented here.

What matters at this stage is:

• Clearly written programming-ready survey document
• Prompt resolution of clarifying questions from programmers and/or reviewers
• A disciplined link test and review process before fielding

Testing should involve an independent reviewer working from the programming instrument and the preview link; issues are routed back to the programmer via tagged comments and resolved before soft launch. You can reference the Gradient Survey Review Manual.

💡 Tip: You can find Gradient-approved contractors for survey programming and review tasks in the Active Contractors Airtable.

Owner(s): Research Manager

Objective 🏁
Collect a subset of real cases to validate data being collected as intended before committing to full field spend.

Even well-tested surveys can behave unexpectedly once real respondents begin entering the live link. Soft launches exist to surface issues related to logic, incidence, quotas, and analytic feasibility under live conditions. They also provide analysts with a set of real data to begin preparing their analysis scripts.

Before full launching a survey, Research Managers should collect ~5-10% of the full sample for soft launch data review. Coordinate with and communicate to the data provider what the soft launch target is.

In practice, soft launch setup means sending the provider updated screener/quota specs and timing, sharing the distribution/test links and the reporting dashboard, and confirming test cases are being collected. After approving and collecting soft launch data, Research Managers can use crosstab reports, download raw data, or use any other method that allows for a complete review of soft launch cases.

During this phase, the Research Manager reviews early data using the Soft Launch Data Check Checklist, coordinates fixes with programmers as needed, and decides when the survey is ready to move to full launch. Jira checklists define the operational steps; the RM’s judgment determines when risk has been sufficiently reduced.

💡 Tip: For high-risk elements (e.g., complex conjoint designs, experimental question types), loop in Product for an additional review during soft launch.

Owner(s): Research Manager, Contractor

Objective 🏁
Collect the full sample efficiently, within spec, and without unnecessary spend.

Once a survey enters full field, the Research Manager’s role becomes primarily supervisory. The focus is on monitoring progress, identifying emerging risks (e.g., quota bottlenecks, high termination rates, low incidence rate), and intervening when necessary.

When feasible, Research Managers are encouraged to leverage approved contractors to manage data collection, including communicating with data vendors. Cases such as these usually occur in low-complexity, lower-cost data collections.

Pausing field, adjusting quotas, or coordinating with the data provider are all tools available to maintain efficiency and data quality.

Owner(s): Primary Analyst

Objective 🏁
Produce a cleaned, analysis-ready dataset and transparent documentation of removals.

After data collection is complete, the Primary Analyst is responsible for identifying and removing low-quality responses and producing the final dataset used for analysis.

While specific thresholds and scripts may vary by project, data cleaning generally involves identifying rushers, straightliners, and respondents who fail attention checks. Judgment is required to distinguish low effort from unconventional but thoughtful responses.

As standard practice, Gradient oversamples anywhere from 10-20% of the target sample size to account for cleanout rate. If removals materially reduce sample size, re-fielding may be required.

Save the keep/drop file and final qualified respondent IDs in an accessible, project-standard location (typically the project’s shared Google Drive folder).

🛑 Reminder
If vendors request final respondent IDs before we have cleaned the data, let them know we provide this upon final data cleaning only. Review our full policy and boilerplate language below.

Final keep/drop classifications must be based on the complete project dataset so that all respondents are evaluated using the same criteria and against the same distributions, regardless of source or completion date.

Unless otherwise agreed before fieldwork begins, Gradient will provide sample vendors with a single, final reconciled keep/drop file after field closes and final data cleaning is complete. The file should typically be delivered within one to two business days of field close and, when feasible, on the same business day.

Project teams may continue to conduct interim quality reviews during fieldwork to identify emerging issues, provide directional feedback to vendors, or make decisions about field management. However, interim reviews should not be represented as final respondent-level keep/drop determinations.

Some cleaning criteria cannot be finalized until the full dataset is available. For example, speeding thresholds, duplicate patterns, multivariate quality measures, and other distribution-dependent checks may change as additional responses are collected.

Providing respondent-level keep/drop files during fieldwork can therefore create several problems:

• A respondent marked as “keep” in an interim file may be excluded during final cleaning.
• Vendors may pay respondents for interviews that are not ultimately used in the analysis.
• Multiple versions of keep/drop files can create reconciliation issues for both Gradient and the vendor.
• Respondents who complete at different points in fieldwork may be evaluated against different standards.

The final reconciled file is the authoritative record of which respondents were retained or removed.

Some vendors operate on payout cycles that may occur before a longer field period has concluded. When a project is expected to remain in field long enough to create a material payout issue, the project lead should discuss the timing with the vendor before launch.

Potential arrangements may include:

• Setting expectations with the vendor and participants that final validation will occur after field closes.
• Adjusting the project’s anticipated field schedule when operationally feasible.
• Agreeing on a project-specific reconciliation process before data collection begins.

These arrangements should be documented in writing. They do not change the principle that Gradient’s final cleaning classifications are based on the complete dataset.

Interim respondent-level keep/drop files should only be provided when there is a documented operational need and the project lead has aligned with the vendor on how later changes will be handled.

Any interim file must:

• Be clearly labeled preliminary and not final.
• State that classifications may change after review of the complete dataset.
• Use the cleaning criteria available at that point in fieldwork.
• Be followed by a final reconciled keep/drop file after field closes.
• Avoid language suggesting that an interim “keep” designation guarantees final acceptance or payment eligibility.

Our standard process is to provide a final, reconciled keep/drop file after field closes and final data cleaning is complete, typically within one to two business days and often on the same day.

We conduct quality reviews throughout fieldwork, but some criteria (including speeding and other distribution-dependent checks) can only be finalized once the complete dataset is available. As a result, cases that appear clean during fieldwork may still be excluded during final cleaning.

Providing respondent-level keep/drop files before field closes can create reconciliation issues and may result in respondents being paid for cases that are ultimately not used in the analysis.

We recognize that this timing may create challenges with your compensation cycles, particularly during longer field periods, and we appreciate your flexibility. For future fieldwork, we’re happy to discuss the anticipated schedule before launch and agree on a practical reconciliation process.