List Experiments Overview

A list experiment is a method for indirectly measuring opinion on sensitive issues where individuals might otherwise be likely to publicly withhold their true opinion. The list experiment works by substantially increasing respondent privacy.

Unlike traditional polling methods, respondents are never asked directly to share their opinion for individual statements. Instead, respondents are asked to read a list of statements and choose the number with which they agree. By comparing a group of people who see a list that includes the sensitive statement to a group of people who see a list without the sensitive statement, we can make inferences about the prevalence of that private opinion across the population.

In a list experiment, we randomize respondents into either Control or Treatment groups. Both groups read a list of 3-4 items and report the number of items with which they agree without specifying the actual items with which they agree.

You’re going to see a list of statements. After reading all of the statements, select how many of the statements you agree with. You do not need to indicate which statement(s) you agree with — only how many you agree with.

If you are unsure about a statement, do NOT count it as one you agree with.

Both groups receive 3 identical items, but the Treatment group receives an additional item that we’re interested in investigating. Therefore, the mean difference of items reported across the two groups is equal to the prevalence of that private opinion.

In the example below, the sensitive statement being investigated is “Despite the separation of church and state, America’s founders intended the country to be guided by Christian values.”

Please read each statement carefully before answering the question below.

1. Instead of elected politicians, the country would be run better if important political decisions were left up to successful business leaders
2. We need much tougher government regulations on business in order to protect the environment
3. I favor requiring background checks for gun purchases at gun shows or other private sales
4. Despite the separation of church and state, America’s founders intended the country to be guided by Christian values

How many statements do you agree with?

1. 0
2. 1
3. 2
4. 3
5. 4

See our template for designing a list experiment here.

People often feel pressure to conceal their true opinions – even through online surveys – particularly when a topic is sensitive or taboo. When asked directly, respondents may tailor their answers to align with social norms or avoid potential judgement or reprisal. These pressures are often amplified when respondents perceive limited privacy. As a result, social desirability bias can make it difficult, or even impossible, to accurately measure both the magnitude and direction of misreporting. List experiments address this challenge by providing anonymity at the individual level, allowing respondents to answer sensitive questions without directly revealing their position.

List experiments can produce more honest and accurate estimates of sensitive opinions and behaviors. By reducing social desirability bias, they offer more credible measures of true prevalence and tend to produce estimates that are more stable and less influenced by short-term shifts.

• Item count response: The total number of statements in the list that a respondent indicates are true for them. A list experiment is sometimes called the item-count technique.
• Control item / statement: A statement (typically non-sensitive) included in both the control and treatment lists meant to mask the sensitive statement.
• Sensitive item / statement: The statement of substantive interest whose true prevalence is expected to be misreported when asked directly. This is only included in the treatment lists.
• Floor effects: When a respondent indicates that none of the items in a list apply to them.
• Ceiling effects: When a respondent indicates that all of the items in a list apply to them.
• Control item bundle: The set of 3-4 control items presented together as a list meant to balance agreement levels to prevent floor and ceiling effects.

Not all statements are best suited for a list experiment.

When To Use a List Experiment:

• When you want to learn about the prevalence of sensitive attitudes, perceptions, or behaviors
• When privacy is imperative for honest self-reporting
• When you want to know if there’s a gap between public and private opinions
• When you want to know which groups are more likely to withhold their public opinion

When Not To Use a List Experiment:

• When the topic is not sensitive and respondents are likely to answer honestly if asked directly
• When the behavior or opinion is very rare
• When the concept being measured is complex, abstract, or difficult to interpret consistently
• When individual-level responses or classifications are required for analysis
• When sample size constraints make reliable estimation impractical

To reduce the risk of ceiling and floor effects, control items should be assembled into bundles of 3-4 statements. Well-designed bundles protect respondent privacy and improve the precision of prevalence estimates by reducing minimum and maximum responses, which lowers noise in the data and produces more stable estimates.

General principles:

• Include one high-agreement statement
• Include one low-agreement statement

Choosing the remaining item(s)

• With 3 control items: include one mid-agreement statement. This typically provides sufficient variation while minimizing cognitive burden.
• With 4 control items: include two negatively correlated statements, such that agreement with one makes agreement with the other less likely. This can make bundles more challenging to construct.

In practice, 3-item bundles are easier because they require less control statements but still project respondent privacy. 4-item bundles can also be effective but require more careful designing.

1. Female political leaders are better suited to lead the country during an economic crisis (7.7% agreement)

2. If people were treated more equally in this country we would have many fewer problems (62.67% agreement)

3. Most scientific evidence shows childhood vaccines do not cause autism (80.96% agreement)

4. I trust social media, such as Facebook, Twitter or X, or Instagram (19.36% agreement)

5. I approve of the protests on college campuses against the war in Gaza (21.29% agreement)

6. This country would be better off if we worried less about how equal people are (36.22% agreement)

7. The government is pretty much run by a few big interests looking out for themselves (83.20% agreement)

   Where items 2 & 3 have an inter-item correlation of -0.21

List experiments are embedded within a broader survey that must include all components needed for analysis and interpretation. In addition to the control item bundles and sensitive statements, the survey should collect supporting measures that allow results to be contextualized and compared. demographics, mindset variables, and direct questions for each sensitive statement.

Required components:

• Screener questions to set quotas around
• List experiment module
• Direct questions corresponding to public opinion on sensitive items
• Demographics for subgroup analyses
• Mindset or attitudinal measures to help interpret results

It’s best practice to include a module between the list experiments and the corresponding direct questions to reduce priming and prevent respondents from connecting the two formats.

List experiments require larger sample sizes than conventional survey questions. The method works in part by intentionally adding noise to individual responses in order to maximize respondent privacy and reduce social desirability bias. Because the signal of interest is less direct, larger samples are needed to produce precise and stable prevalence estimates.

Ensure that respondents will be randomly assigned to either a control group, which sees only the control item bundle, or a treatment group, which sees the same bundle with the sensitive item added. Random assignment ensures the two groups are comparable.

Collect responses using standard survey fielding practices, applying demographic quotas (e.g., based on US Census benchmarks) as needed to ensure the sample broadly reflects the target population. When appropriate, the data should be further weighted to ensure results are representative of the target population.

Analysis focuses on estimating the prevalence of the sensitive behavior or opinion by comparing average item count responses between the control and treatment groups. The difference between these averages provides an estimate of how many respondents would endorse the sensitive item under conditions of increased privacy.

In practice, we use a Bayesian modeling approach, which allows uncertainty to be quantified directly and supports partial pooling across respondents. Baseline demographic and survey covariates are typically included to improve estimate stability, but covariates should be evaluated on a statement-by-statement basis to ensure they are theoretically relevant and do not introduce noise.

The key focus of analysis is the difference between private and public opinion. By comparing list experiment estimates to responses from corresponding direct questions, we can assess whether and for whom opinions are more likely to be withheld in public reporting. These differences can be examined at both the population level and across key subgroups.

A negative estimate occurs when respondents are more likely to agree with a statement when asked directly than when agreement is measured privately through a list experiment. This typically happens when the sensitive position is disagreement, meaning respondents may feel social pressure to publicly agree with a statement if they privately do not.

Yes, multiple list experiments can be included in a single study, but each adds a cognitive burden to respondents. In addition to managing overall survey length, running more list experiments also makes it more challenging to construct high-quality control item bundles.

There is no fixed number of list experiments that can be fielded in a single study. The practical limit depends on several factors, including sample size, control item bundle composition, survey architecture, and tolerance for longer interviews. Below are some examples of recent list-experiment-focused surveys we have fielded.

With sufficient investment in sample size and careful experimental design, substantially larger batteries of sensitive statements can be tested. For example, the Social Pressure Index (Project Polygraph) tested 68 sensitive statements in a single study using a nationally representative sample of 7,144 US adults (and 19,879 total interviews including oversamples).

Approximately one-third of respondents were randomly assigned to a pure control group. The remaining respondents were assigned a random subset of 17 statements drawn from the full set of 68. Each sensitive statement was paired with one of 17 control item bundles, such that the control group’s responses served as the comparison baseline for four separate sensitive statements. This design allowed for efficient reuse of control bundles while maintaining respondent privacy and statistical power.

Yes, list experiment results can and should be analyzed at the subgroup level to examine differences across demographics or other characteristics. However, subgroup analyses require sufficient sample sizes to produce stable estimates and should be planned in advance whenever possible.

Some sensitive opinions can be framed in opposite ways—for example, “Climate change is attributable to human activity” versus “Climate change cannot be attributed to human activity,” or “Diversity makes this country a better place to live” versus “Diversity makes this country a worse place to live.” We tested pairs of inverse statements to assess whether the direction of the wording affects list experiment results.

The results showed coherent and internally consistent response patterns across inverse statements. For example, respondents who privately agreed that they believe in God generally did not also report disbelief, and respondents who privately attributed extreme weather to human-caused climate change generally rejected the opposite statement. Similar consistency appeared in diversity-related items.

Implication: Either orientation of a sensitive statement can be used in a list experiment. We should choose the version that best captures the social pressure or opinion of interest. If the statement includes a negative word (e.g., “not” or “cannot”), it should be emphasized to reduce the risk of misinterpretation (see next FAQ).

Some sensitive statements require negative wording (e.g., “not” or “do not”) to capture the sensitive opinion of interest. However, negative words are more likely to be misread, which can introduce measurement error in both public and private responses.

In testing, we found that emphasizing the negative word (e.g., using bold and italics) produced meaningful differences in both public and private agreement rates, suggesting respondents were more likely to read the statement carefully.

Recommendation: When a sensitive statement includes a negative word, emphasize it (e.g., not) to reduce the risk of misinterpretation.

Self-monitoring, or social chameleon behavior, is a psychological trait describing how much individuals adjust their behavior to match social expectations. In theory, people with high self-monitoring may be more likely to hide their true opinions in public.

In our testing, however, differences between private and public opinion were not consistently larger among high self-monitors compared to low or mid groups. This suggests self-monitoring does not reliably explain who withholds their opinions.

Implication: Including a self-monitoring scale in surveys is not necessary as a standard measure. The measure adds survey length without meaningfully improving list experiment estimates or insight into social pressure.

We tested whether social pressure from political elites affects public and private opinion differently. In the experiment, half of respondents were shown a series of Truth Social posts from Donald Trump. One post expressed support for first-trimester abortion (a position Trump would be unlikely to take in real life), while the remaining posts were real messages included to mask the focal statement. Respondents were then asked both public (direct) and private (list experiment) questions about their support for first-trimester abortion.

The results show that elite signaling influenced responses, but did not affect public and private opinion differently. In other words, exposure to the cue shifted responses in both formats in roughly similar ways.

Implication: Private opinion measures should not be assumed to be inherently more insulated from social pressure via elite signaling than direct public responses.

Political interest and political knowledge can influence how people express political opinions. In our testing, respondents with higher political interest or knowledge sometimes showed larger gaps between private and public responses, suggesting they may be more aware of social or partisan expectations when answering sensitive questions.

However, these patterns were not consistent or strong enough to justify including these measures as standard covariates or quota variables. While political sophistication may play some role in self-censorship, adding these questions increases survey length without reliably improving list experiment estimates.

Implication: Political interest and political knowledge do not need to be included as standard measures when fielding list experiments.

List experiment estimates are typically modeled using Bayesian approaches, which require selecting covariates to improve estimate stability. In general, models should include a small set of conceptually relevant baseline covariates, while avoiding too many variables that may introduce multicollinearity and instability. However, to examine relevant subgroup differences, the corresponding variable must be included in the model.

In internal testing, more parsimonious models performed better than heavily specified ones. We typically recommend including basic demographics such as age, race, gender, education, and income, and party identification for political topics. Additional covariates can be added when clearly relevant, or for specialized populations, but should be used sparingly. For example, religion and parent status.

Some projects may require specific variables (e.g., a client segmentation), and occasionally a particular statement may warrant an additional covariate—for example, including parental status when analyzing statements related to children. However, these should be added selectively rather than included across all models. When in doubt, we can run multiple model versions to determine which is more consistent and stable.

Implication: Subgroup differences can only be analyzed if the relevant variable is included in the model, so covariates should be selected with both model stability and intended subgroup analyses in mind. In practice, this means including a core set of demographics (e.g., age, race, gender, education, income, and party ID for political topics) to support consistent subgroup comparisons.

Small subgroup sizes can lead to unstable estimates in list experiment models. Because list experiments rely on aggregated item-count responses rather than individual-level observations, very small groups introduce more variation and reduce estimate reliability.

In internal testing, models performed better when covariates were structured so that subgroups were relatively balanced and sufficiently large.

Recommendation: When specifying covariates, avoid subgroup sizes smaller than n ≈ 200. This may require recoding or grouping variables—for example, combining smaller race or religion categories into an “Other” group or defining partisanship using broader categories.

Yes. List experiments do not include an explicit “unsure” option the way direct questions often do.

In testing, we found that when respondents were not given guidance, some counted statements they felt uncertain about as agreement, which inflated private agreement rates and changed the size of public–private gaps.

Recommendation: Include the following brief instruction: “If you are unsure about a statement, do NOT count it as one you agree with.” This helps ensure that differences between public and private responses reflect true opinion differences rather than differences in how uncertainty is handled.

Recommended sample size depends on the number of list experiments included in the survey. Because list experiments intentionally add noise to protect respondent privacy, they generally require larger samples than conventional survey questions, and the required sample increases as more list experiments are added.

For surveys with multiple list experiments, we typically recommend n ≈ 2,000 respondents. If only a small number of list experiments are included, smaller samples may be sufficient, though statistical power should be evaluated and subgroup sizes should be considered. For studies with large batteries of sensitive statements, much larger samples may be required, often combined with designs that randomly assign different subsets of list experiments to respondents.

For example, the Social Pressure Index (Project Polygraph) tested 68 sensitive statements in a nationally representative study of 7,144 U.S. adults (19,879 total interviews including oversamples) by assigning respondents a subset of statements while reusing control bundles across experiments. This approach allows large numbers of statements to be tested while maintaining privacy and statistical power.

Recommendation: Rule of thumb is n ≈ 2000 respondents for a survey with < 18 list experiments.

In list experiment studies, private opinion (measured through the list experiment) is often compared to public opinion (measured through direct questioning). Because list experiments must be asked before direct questions to avoid contaminating the private measure, respondents in the treatment group are exposed to the sensitive statement before answering the public opinion question.

In testing, we found that this prior exposure can influence how treatment-group respondents answer the direct question.

Recommendation: Estimate public opinion using only the control group, which has not previously seen the sensitive statement in the list experiment. This avoids contamination and provides a cleaner comparison between public and private opinion.

We tested whether signals from public opinion (i.e., the share of respondents indicating that people would not answer honestly and the share of respondents that select “Unsure / No Opinion”) could help predict which statements are truly sensitive and produce the largest gaps between private and public opinion.

The results show little to no relationship between these measures and the size of the public–private gap. Neither uncertainty in public responses nor perceived sensitivity reliably predicted which statements produced meaningful deltas.

Implication: Public opinion patterns alone are not a reliable way to identify sensitive statements. Sensitivity appears to depend more on group identity, social norms, and the political meaning of the issue than on overall agreement levels or uncertainty in the population.