Survey Design and Analysis

A survey is a method for measuring a population by asking a standard set of questions to a sample of its members and generalizing the answers to the whole group. It exists because you usually cannot ask everyone, and because how you select people and word questions changes the result. Survey design and analysis is the practice of making those choices deliberately so the final numbers carry a defensible margin of error rather than hidden bias.

itData engineering and analytics

Don't Panic: Survey Design and Analysis

A survey measures a whole population by asking a fixed set of questions to a sample of its members and generalizing the answers. You reach for one when the fact you need lives only in people's heads: opinions, intentions, what they remember doing, how satisfied they are. If the answer already exists in a log or a billing system, use that. A survey is the instrument for questions only a person can answer, and the catch is that people answer imperfectly.

The work splits into two halves that break separately. Design is who you ask and how you word the questions. Analysis is turning the returned answers into population estimates. A flawless questionnaire sent to the wrong sample gives you precise nonsense; a perfect sample answering a loaded questionnaire gives you representative bias. Neither half rescues the other.

The idea that ties it together is total survey error, the accounting system for every way an estimate can differ from the truth. One branch is representation: does your sampling frame, the list or method you draw people from, actually cover the population, do responders differ from the people who ignored you, does the weighting distort things. The other branch is measurement: does the question capture the concept, does the respondent understand and answer honestly. Design is a series of trades among these, against cost.

Here is the part that surprises people. The margin of error you see quoted, the plus-or-minus figure, covers only sampling error, the fact that you asked a sample and not everyone. Coverage gaps, nonresponse, and question wording are usually bigger, and none of them are in that number. A giant opt-in panel can be more wrong than a small probability sample, one where every person has a known chance of selection, while its tighter margin of error hides the problem.

Two smaller traps. A number for a subgroup carries the margin of that subgroup's size, not the whole sample's. And the gap between two groups has a wider margin than either group alone, so test a difference before you describe it.

On the questionnaire: ask one thing per question, avoid loaded words, and be wary of agree-or-disagree scales, because some people agree with anything, an effect called acquiescence. Expect satisficing, where a respondent who does not want to work picks the first acceptable option, the midpoint, or "don't know".

Read the Intro for the full vocabulary and the sampling designs. Slides show how the error sources connect. The Cheatsheet has the formulas and the sample-size table. The Practice Reference walks through writing questions, sizing a sample, computing a response rate, and weighting. Field Notes is blunter: response rates keep falling, nobody has solved nonresponse, and chasing the rate is optimizing the wrong number.

Don't Panic: Survey Design and Analysis

Relevant careers