Penalty Scores & Data Quality Assessment

This page documents Gradient’s standard approach to penalty scores: how we define, communicate, and apply penalties to identify and remove low-quality survey responses.

Penalty scores are not a single heuristic. They are a triangulated signal built from multiple indicators of inattention or inconsistency (e.g., speeding, straightlining, low-quality open-ends). The goal is to ensure data quality while preserving analytically valid edge cases.

This page explains:

• Why we use penalty scores
• Who owns each part of the process
• Where penalty score specifications live
• The standard components and default logic
• How removal thresholds are determined

The authoritative execution steps live in Jira issue types and smart checklists. This page defines the logic and expectations behind them.

1. Speeding is the benchmark offense: All other penalties are calibrated relative to the standard speeding threshold (< ½ median LOI).
2. Scaled penalties > hard cutoffs: Wherever possible, penalties scale with severity rather than relying on on/off scoring.
3. Transparency: Penalty logic must be easy to explain, audit, and reproduce.

Rule-of-thumb: Penalty score logic is specified by the RM, implemented by the Analyst.

Research Managers can begin specifying penalty score inputs and rules during the survey writing process, but are not finalized until after soft launch or towards the end of fieldwork, when:

• Final survey structure is known
• Median LOI is observable
• Product team members have access to real data

Penalty score definitions must live in one of two places:

1. Jira: Penalty Score / Data Cleaning Spec issue (default)
2. Google Drive: Linked Google Doc analysis plan (useful for complex logic)

Every Penalty Score / Data Cleaning Spec must explicitly document:

• Speeding definition(s)
• Straightlining batteries
• Open-end quality variables
• Any inconsistency or plausibility checks
• Penalty weights
• Final keep / remove threshold

The penalty score components represent Gradient’s default approach to penalty scores. It is a starting point for the Analyst to begin data cleaning scripts. Specific survey design, variables, audiences, etc. will dictate the utility of the standard penalty score approach versus a custom design.

Research Managers should use their judgment to determine what custom data cleaning parameters should be used to enhance data quality while balancing sample costs.

Benchmark Component

Speeding compares respondent duration to the median length of interview (LOI). Speeding for the full survey is a default consideration. For most projects, this will be standard, but Research Managers may need to consider different speeding thresholds for different survey paths.

• < ½ median LOI = +5 penalty score = disqualification

Because speeding is an automatic disqualification, speeding behavior defines the reference penalty against which all other components are scaled.

When the ½ median LOI threshold is chosen as a removal-level offense (most cases), scaled penalties for more excessive speeding are inconsequential. Penalty scores exceeding +5 are calculated in case Research Managers select more permissive speeding thresholds.

Used when certain modules are analytically critical. For example, it’s common to include page timers for conjoint tasks, segmentation batteries, list experiments, and other survey workhorse modules.

💡 Tip: Produce a final, client-approved survey instrument that is analytically sound, operationally feasible, and ready to be programmed.

Similar to overall speeding, module-specific speeding follows the same scaling logic and is often treated as independently disqualifying (+5), but penalty scores are ultimately selected at the Research Manager’s discretion.

💡 Tip: Research Managers should confirm with the survey programmer that page timers are included for critical survey modules..

Straightlining assessments are applied to batteries of similar questions, like grids, matrices, or carousels. Penalties should only be applied when the batteries contain identical response options, repeated choice structure, and logical variation is expected.

• Every observed battery with 90+% identical responses triggers penalties:
  • +2 for >90%
  • +3 for >95%
• Penalties can be scaled with battery size and severity

Keep in mind that some grids can be logically straightlined. Final inclusion for straightlining is always at RM discretion.

Whenever possible, surveys should include at least one diagnostic open-end, which can be sourced from a planned open-ended question or a simple “In a couple sentences, what was this survey about?”

Process

1. RM flags OE(s) for quality review
2. Analyst runs multiple LLM scoring passes
3. Analyst outputs OE mean score and variance
4. RM reviews highly penalized and high-variance edge cases
5. (Optional) RM adjusts the AI prompt for OE scoring

In some cases, logical response patterns can be inferred. Although used sparingly, identifying inconsistent or implausible responses can be especially important in specialized audiences, like expensive B2B data collections.

Examples to consider:

• Inconsistency between job title and reported responsibilities
• Reported familiarity of fake brands
• Mutually exclusive claims

• Minor inconsistency: +1-2
• Major contradiction: +3-5 (borderline removal)
• Multiple major contradictions: +10 (immediate disqualification)

Penalty score ≥ +5 → Remove

This aligns all non-speeding offenses to the same severity as disqualifying a speeder.

• Attention check failures are automatic removals and not part of penalty scoring
• Some projects may define component-specific hard removals (e.g., segmentation straightlining)
• Deviations from the +5 rule are acceptable so long as they’re clearly documented