# Log two questionnaire orders in an exploratory pilot

Keep order exposure and assignment context visible while piloting two question sequences.

If nothing was unclear or difficult, say so. Skip questions about steps you did not experience; use not applicable where appropriate.

## 1. Which earlier prompt influenced how you interpreted this question, if any?

Answer: ____________________

## 2. What experience came to mind before this prompt appeared?

Answer: ____________________

## 3. Where did the change in topic need an explanation?

Answer: ____________________

## 4. What wording would let you answer this prompt independently?

Answer: ____________________

## Author notes / illustrative keys where applicable

1. Probe perceived carryover without asserting an effect exists.
2. Distinguish available recall from content introduced earlier.
3. Locate a transition problem tied to the actual draft.
4. Generate a repair to test, rather than voting for an order.

Interpretation caution: Arrival-group assignment confounds order with context. One invented transition note does not establish an order effect or a general failure rate.


## Illustrative finding and follow-up

Observation: Eight fabricated volunteers saw different orders in different arrival groups. One invented AB note interprets the overall prompt as difficulty-only and asks whether no issue can be described.

Action to test: Document the confounding context. Clarify that B asks about the whole delivery, including no issue, then pilot the transition again.

Follow-up: Run a new pilot with recorded exposure and revised transitions; any causal study needs a separate design.


## Collection protocol

When and whom to ask: Before finalizing sequence, show one documented draft to each optional volunteer and record its exposure order.

Decision owner: The pilot facilitator records exposure; the research owner reviews assignment and interpretation limits.

Follow-up: Run a new pilot with recorded exposure and revised transitions; any causal study needs a separate design.

Response handling: Use study labels rather than names. Do not claim randomization when assignment followed arrival time.

## Decision workshop

Synthetic examples only. Manual worksheet; no automatic import or product writes.

### Synthetic pilot and reporting records

Draft order | Volunteers | Assignment context | Exposure
--- | --- | --- | ---
Prompt A | External free-text draft | All drafts | What was difficult about the latest delivery, if anything?
Prompt B | External free-text draft | All drafts | How would you describe that delivery overall?
AB | 4 | Morning arrivals | A shown first
BA | 4 | Afternoon arrivals | B shown first
Observed AB note | 1 synthetic case | Morning volunteer | Interpreted B as difficulty-only; asked whether no issue could be described

### Version changes

Boundary | Draft | Revised
--- | --- | ---
Assignment | Different arrival groups ignored | Assignment context recorded
Result | Causal effect asserted | Exploratory interpretation evidence

### Draft or earlier version (review its limits)

Any difference between AB and BA proves question-order bias.

### Revised example

This invented pilot confounds order with arrival group. Retain draft and exposure notes; use explanations to repair prompts without a causal order-effect claim.

### Decision situations

- Actual order and assignment context are recorded: Compare explanations with context limits.
- The displayed order is unknown: Recover exposure before comparing the two drafts.
- A randomized study is desired: Design assignment and analysis separately; this pilot is not that study.

### Evidence dependencies

- Arrival-group assignment → Contextual pilot notes: group context
- Observed draft order → Contextual pilot notes: prompt exposure

### Method references

- Pew Research Center question design: https://www.pewresearch.org/writing-survey-questions/ — Wording and response-option consistency.