# Compare survey waves when the respondent mix changes

Check respondent-group counts before interpreting a change in an overall survey percentage. Work through unchanged group rates with a shifting sample mix.

If nothing was unclear or difficult, say so. Skip questions about steps you did not experience; use not applicable where appropriate.

## 1. Which invitation route brought you to this questionnaire?

Answer: ____________________

## 2. What recent experience are you using to answer the feedback question?

Answer: ____________________

## 3. Does the stated time period match that experience?

Answer: ____________________

## 4. Which part of this invitation differs from the one you saw previously, if any?

Answer: ____________________

## Author notes / illustrative keys where applicable

1. Use a non-identifying route description to inspect cohort composition.
2. Check whether waves refer to comparable experience rather than merely identical wording.
3. Find period mismatch that can change which experience is reported.
4. Probe invitation context without requiring someone to have answered an earlier wave.

Interpretation caution: Equal wording does not ensure comparable respondents. This synthetic channel mix is not a weighting recommendation or a significance test.


## Illustrative finding and follow-up

Observation: Wave 1 has channel A 16 favourable of 20 and B 40 of 80. Wave 2 has A 64 of 80 and B 10 of 20.

Action to test: Report both channel percentages and counts alongside each overall percentage; investigate invitation mix before claiming the service improved.

Follow-up: For the next wave, retain wording, period and invitation rules where possible and compare counts plus within-channel percentages.


## Collection protocol

When and whom to ask: Pilot with people reached through each planned invitation route before comparing successive waves.

Decision owner: The reporting reviewer owns the comparison; the recruitment owner records invitation channels and eligibility.

Follow-up: Check whether any later overall movement is accompanied by within-channel movement and stable collection context.

Response handling: Do not request contact identifiers merely to identify a channel. Avoid publishing tiny identifiable groups; use an organization-approved aggregation rule.


## Worked measurement study

All example records are synthetic; manual worksheet, no automatic import.

### Inspect the synthetic example

Wave | Channel | Answers | Favourable | Within-channel share
--- | --- | --- | --- | ---
Wave 1 | A | 20 | 16 | 80%
Wave 1 | B | 80 | 40 | 50%
Wave 2 | A | 80 | 64 | 80%
Wave 2 | B | 20 | 10 | 50%

### Derivation

Wave 1 overall = (16+40)/(20+80)=56%. Wave 2 overall = (64+10)/(80+20)=74%. Channel A remains 80% and B 50%; the mix changes.

Comparison note: the overall share rises from 56% to 74% in the synthetic sample while both channel shares remain unchanged. Investigate recruitment mix before claiming an experience change.

### Evidence boundary

Observed / Counts and shares within each channel / Supports descriptive decomposition of the overall change.
Unknown / Population composition and causal effects / Do not invent weights or attribute improvement.
Next check / Compare eligibility, timing and channel invitations / Retain counts with each reported share.

### Method references

AAPOR survey best practices: https://aapor.org/standards-and-ethics/best-practices/ — Pretesting and transparent sample context.