Sample Size

Well the basic rule of thumb is 200 per each subgroup or at least 300 total, whichever is greater. Another rule of thumb is to use equation: \[n*t \geq 10 * \frac{k}{p}\]

Where n is the sample size, t is the number of choice sets, k is the total number of levels minus the total number of attributes, and p is the probability of the least frequently chosen alternative. So we want to estimate the n, and we have everything besides p. In our design, we had 3 alternatives, so by chance each of them will be picked up around 33% of the time. In real life, this number will be much smaller, so let’s say we estimate that the least picked-up alternative is 10%. If we substitute the numbers, we get:

\[60 \geq 10 * \frac{(11-4)}{0.1} = \frac{1}{10}\]

Well, this seems a little bit too low, but there is another rule of thumb:

\[\frac{n * t * a}{c} \geq 1000\]

It’s a bit better since it does not involve guessing the p. Now we have the same t – the number of choice tasks. The a is the number of alternatives per task, and c is the maximum number of levels in any attribute. So if we substitute our design numbers, we get:

\[133 \geq 1000 * 4 * \frac{1}{10*3}\]

Much better.

Another way to look at the sample size is the perspective of a simulator. Assuming that share of preference that we have in the simulator is a proportion, we can use a simple 95% margin of error for a given sample size:

\[MOE = z * \frac{\sqrt[2]{p * (1 - p)}}{\sqrt[2]{n}}\]

So the z value for 0.95% is 1.96. Assuming that the share is 50%, when we fill the numbers in the equation, we will see that MOE for 200 respondents is around 7%, for 500 respondents it is around 4.5%, for 1000 it is around 3% so on, so on… So if we expect large effect sizes (the shares will vary vastly), we can probably have a smaller sample (around 500) since the +/- 4.5% won’t matter that much.

For designs that include prohibitions, we should compare the standard errors for your design with prohibitions to the standard errors for a parallel design that has no prohibitions. “When comparing standard errors for particular effects between two designs, the relative efficiency of one design to the other with respect to a given parameter is equal to the ratio of the squares of the standard errors.” [from Becoming an Expert in Conjoint Analysis]. Translating this to English: Assume that a design with prohibitions gives a standard error for a level equal to 0.073 and a parallel design without prohibitions leads to a standard error of 0.045 for the same level. To calculate the designs' relative efficiency, we divide 0.0452/ 0.0732 = 38%. So, if the design without the prohibitions is 38% more effective, we need 1/0.38 = 2.63 times more data (meaning respondents or respondents * tasks) to have the same precision.