Analyzing List Experiments

A list experiment (also called the item count technique) is a survey experiment designed to estimate the share of people who agree with a sensitive statement when a direct question may be biased by social desirability, fear of judgment, or preference falsification.

Instead of asking respondents to reveal which statements they agree with, respondents are asked only how many statements they agree with from a short list. A randomly assigned subset of respondents receives a list that includes the sensitive statement, while the control group receives a list without it. The difference between groups is used to infer the private agreement rate with the sensitive statement.

In practice, we often pair the list experiment with a direct question about the same statement, which gives a public agreement rate. This lets us quantify:

• Private agreement: what people appear to believe when privacy is protected by the method
• Public agreement: what people are willing to say directly
• Delta (private − public): the estimated gap between private and public agreement

Delta is often the key output because it isolates what’s “special” about list experiments: not agreement levels by themselves, but the divergence between what people will say publicly and what the experiment implies privately.

Finally, list experiments are rarely only about the overall average. We’re often interested in how the public/private gap behaves across subgroups (e.g., by market, demographic segment, or customer type). Subgroup deltas are frequently where the strongest insights live: the same overall average can hide meaningful heterogeneity in willingness to express an opinion publicly.

At the core of each list experiment is a short list of non-sensitive control statements that all respondents see in the count question. We typically use 3–4 control statements, and we need to keep track of that number (n_control_statements) because the private response is stored only as a total count.

At the respondent level, each experiment then has:

• treatment: a randomized assignment indicating whether the respondent saw the sensitive statement included in their list.
• private: an item-count answer. For control respondents it ranges from 0 to n_control_statements; for treatment respondents it ranges from 0 to n_control_statements + 1.
• public: a direct agree/disagree answer about the sensitive statement. Sometimes we include an “unsure” option, which we typically code as “disagree” for modeling.
• majority (optional): what the respondent thinks most other people believe about the statement. In our current modeling approach this is not used as an input, but we still compute summaries for it as supplementary context.

We also include categorical predictors (usually demographics / profiling variables). In our standard approach they enter the model, and while results can be summarized for any subgroup, these predictors are typically the ones we report.

Our standard approach fits a Bayesian model (implemented in Stan) that combines two signals about the same sensitive statement: the list count (private signal) and the direct agree/disagree question (public signal).

At a high level, we estimate (with uncertainty) three respondent-level probabilities:

• Control-item agreement: for each respondent, the probability of agreeing with a generic non-sensitive control statement (modeled as a Binomial count across n_control_statements).
• Private agreement: for each respondent, the probability of agreeing with the sensitive statement (modeled as a Bernoulli “added on top” of the control count for treated respondents).
• Public agreement: for each respondent, the probability of answering “agree” to the direct question (modeled as a Bernoulli outcome).

Each of these probabilities is linked to the predictor set (with separate coefficients for control, private, and public components). For treated respondents, the count is modeled as a Binomial control-item component plus an additional Bernoulli component for the sensitive item; for control respondents, only the control-item component is present. This structure lets the model learn sensitive-item agreement from the observed count data, while also learning how both public and private agreement vary across respondents.

The output is respondent-level posterior draws for private agreement and public agreement; delta is derived directly as private − public. This makes it straightforward to generate uncertainty-aware summaries for any subgroup by aggregating draws. Internally, the model also estimates each respondent’s baseline control-item agreement probability, which is mainly a nuisance component used to interpret the count data.

Difference of differences (design-based baseline): private agreement is estimated as the difference in average counts between treatment and control (assuming no design effects). If a direct question is also asked, public agreement is the share who answer “agree”, and delta is obtained by comparing public vs private. This approach is easy to compute overall and within subgroups, but uncertainty and covariate adjustment are typically handled in a more ad-hoc way.

Predictor-based models (closer to our approach): fit a probabilistic model to the count data (and sometimes the direct question), using predictors to describe how agreement varies across respondents. Variants differ mainly in whether they model public responses jointly or treat them as a separate summary, and whether they are fit in a Bayesian or frequentist framework.

The package vignettes are the source of truth for implementation details. This section is a high-level “how we do it” map.

In real projects, list experiment inputs rarely arrive perfectly clean: treatment flags may be encoded inconsistently, private counts may be stored as factors/strings, and direct-response options (agree/disagree/unsure) can vary from survey to survey. Also, a single dataset often contains multiple list experiment bundles (one per sensitive statement).

lx_parse_data() is designed to standardize all of that into a consistent, model-ready structure. Its main job is to:

• detect and split multiple bundles into one dataset per experiment,
• standardize the treatment/private/public (+ optional majority) variables, and
• pack predictors alongside them in a tidy format.

Once each experiment is in the standardized format, lx_fit() runs the Stan model. You control the usual sampling parameters (e.g., number of iterations, chains, cores), and the output is a model object that contains respondent-level posterior draws needed for downstream summaries.

To make subgroup reporting consistent across projects, we use cohorts: standardized definitions of “which respondents are in this group” (and, when needed, their weights). Cohorts are created with g6tr.cohorts package and can represent a single group or a whole cohort set (e.g., all levels of a demographic variable).

Given a fitted model and a cohort (or cohort set), we generate subgroup summaries from posterior draws:

• public agreement, private agreement, and derived delta
• majority (when available) as a supplementary summary

In practice, lx_predict_for_cohorts() is the main workhorse for producing these summaries, and lx_strip_plot() is a quick way to visualize them. For more client-ready visuals, we use lx_dumbbell_plot(). For spreadsheet-style reporting, we use lx_summary_for_cohort_set() when needed.

The output of lx_predict_for_cohorts() is a small summary table per cohort. The exact numbers depend on the specific model and cohort (the example below comes from a demo fit), but the structure is the same for any model/cohort:

  component  mean   lower  upper draws      
  <chr>     <dbl>   <dbl>  <dbl> <list>     
1 public    0.190  0.155   0.225 <dbl [200]>
2 private   0.375  0.207   0.540 <dbl [200]>
3 delta     0.185  0.0240  0.337 <dbl [200]>
4 majority  0.266 NA      NA     <dbl [0]>  

Each row is a component (public, private, delta, and optionally majority). For the modeled components, the key column is draws: because the Stan model stores posterior draws of p_public and p_private for every respondent, we can take any subgroup and compute the subgroup average at each posterior draw. That gives a vector of draws (e.g., length 200), i.e. the posterior distribution of “average probability in this subgroup”. The mean is the average of that vector, and lower / upper are its 2.5% / 97.5% quantiles. delta is computed draw-by-draw as (p_private − p_public) at the respondent level and then averaged within the subgroup, so it comes with the same kind of uncertainty. majority is not modeled: it’s reported as the direct subgroup mean of the observed “majority” responses, so it has no posterior draws (and therefore no interval). More generally, whenever we have per-respondent posterior draws for some quantity, we build subgroup-average draws the same way: compute the subgroup mean at each draw.

This draw-based “subgroup average” object is what all subsequent plotting and summaries use. Strip plots visualize the full draw distribution (think of it as a compact density, not just mean + interval). Spreadsheet-style summaries usually show just the draw means, but they still use the draws to communicate certainty via color. For example, coloring the delta cell by Pr(delta > 0.03) (darker red when that probability is >=95%, lighter when it’s closer to >92%, etc.).