Eight fabricated volunteers saw different orders in different arrival groups. One invented AB note interprets the overall prompt as difficulty-only and asks whether no issue can be described.
Survey testing and reporting / Survey guide
Survey methods · Decision workshop
Log two questionnaire orders in an exploratory pilot
Keep order exposure and assignment context visible while piloting two question sequences.
Opens product registration. Example questions are not imported automatically.
PLAN A BOUNDED FOLLOW-UP
Document the confounding context. Clarify that B asks about the whole delivery, including no issue, then pilot the transition again.
Run a new pilot with recorded exposure and revised transitions; any causal study needs a separate design.
Synthetic planning example. A repeat survey alone does not establish causation.
From question to next step
Make the feedback useful.
Run the pilot
Before finalizing sequence, show one documented draft to each optional volunteer and record its exposure order.
Choose a bounded action
Document the confounding context. Clarify that B asks about the whole delivery, including no issue, then pilot the transition again.
Check the revision
Run a new pilot with recorded exposure and revised transitions; any causal study needs a separate design.
A worked interpretation example
From a comment to a practical next step.
Illustrative scenario. These are not customer results.
Keep the finding specific
Eight fabricated volunteers saw different orders in different arrival groups. One invented AB note interprets the overall prompt as difficulty-only and asks whether no issue can be described.
Choose an action to test
Document the confounding context. Clarify that B asks about the whole delivery, including no issue, then pilot the transition again.
Check the experience again
Run a new pilot with recorded exposure and revised transitions; any causal study needs a separate design.
A useful starting point
Optional free-text pilot questions
Four invented volunteers see draft AB and four see BA. The facilitator assigns by arrival group, so an observed difference cannot isolate order from group context.
If nothing was unclear or difficult, say so. Skip questions about steps you did not experience; use not applicable where appropriate.
Which earlier prompt influenced how you interpreted this question, if any?
Probe perceived carryover without asserting an effect exists.
What experience came to mind before this prompt appeared?
Distinguish available recall from content introduced earlier.
Where did the change in topic need an explanation?
Locate a transition problem tied to the actual draft.
What wording would let you answer this prompt independently?
Generate a repair to test, rather than voting for an order.
If nothing was unclear or difficult, say so. Skip questions about steps you did not experience; use not applicable where appropriate. Which earlier prompt influenced how you interpreted this question, if any? What experience came to mind before this prompt appeared? Where did the change in topic need an explanation? What wording would let you answer this prompt independently?
Download the questions as text ↓
Examples to adapt, rather than a validated research instrument.
Choose the response format
Make room for the explanation.
| Question | Suggested format | What to preserve |
|---|---|---|
| Which earlier prompt influenced how you interpreted this question, if any? | Optional short open text | Probe perceived carryover without asserting an effect exists. |
| What experience came to mind before this prompt appeared? | Optional short open text | Distinguish available recall from content introduced earlier. |
| Where did the change in topic need an explanation? | Optional short open text | Locate a transition problem tied to the actual draft. |
| What wording would let you answer this prompt independently? | Optional short open text | Generate a repair to test, rather than voting for an order. |
Make questions optional where appropriate. If you add a rating scale, label its endpoints, keep one idea per question, and retain a follow-up for reasons. These examples are not a validated measurement instrument.
Read the answers carefully
Keep the interpretation within the evidence
Arrival-group assignment confounds order with context. One invented transition note does not establish an order effect or a general failure rate.
This invented pilot confounds order with arrival group. Retain draft and exposure notes; use explanations to repair prompts without a causal order-effect claim.
Before you send it
Check what your wording assumes.
Which order caused your answer to change?
Which earlier prompt influenced how you interpreted this question, if any?
The revision allows no perceived carryover and does not turn a confounded pilot into a causal experiment.
Run a focused feedback cycle
Plan the invitation, review and follow-up.
When and whom to ask
Before finalizing sequence, show one documented draft to each optional volunteer and record its exposure order.
Who reviews the finding
The pilot facilitator records exposure; the research owner reviews assignment and interpretation limits.
What to check after a change
Run a new pilot with recorded exposure and revised transitions; any causal study needs a separate design.
Handle responses carefully
Use study labels rather than names. Do not claim randomization when assignment followed arrival time.
A practical decision workshop
Inspect the example before choosing a rule
Trace an invented case, preserve the evidence limits, then adapt the manual worksheet to your own decision.
Evidence dependency diagram
What must be known first?
Scroll the diagram sideways if needed; every dependency is also written below.
- Arrival-group assignment → Contextual pilot notes: group context.
- Observed draft order → Contextual pilot notes: prompt exposure.
Synthetic example · not customer data
Inspect the records
| Draft order | Volunteers | Assignment context | Exposure |
|---|---|---|---|
| Prompt A | External free-text draft | All drafts | What was difficult about the latest delivery, if anything? |
| Prompt B | External free-text draft | All drafts | How would you describe that delivery overall? |
| AB | 4 | Morning arrivals | A shown first |
| BA | 4 | Afternoon arrivals | B shown first |
| Observed AB note | 1 synthetic case | Morning volunteer | Interpreted B as difficulty-only; asked whether no issue could be described |
Wide tables scroll sideways on small screens.
Download records (.csv) · Download complete decision kit (.json)
Decision explorer
Choose an evidence situation
This returns a written example explanation. It does not inspect your records or approve a real report.
Read every situation without the interactive tool
- Actual order and assignment context are recorded
- Compare explanations with context limits.
- The displayed order is unknown
- Recover exposure before comparing the two drafts.
- A randomized study is desired
- Design assignment and analysis separately; this pilot is not that study.
Version comparison viewer
Keep the draft and its correction visible
Draft or earlier version · review its limits
Any difference between AB and BA proves question-order bias.
Revised example · limits retained
This invented pilot confounds order with arrival group. Retain draft and exposure notes; use explanations to repair prompts without a causal order-effect claim.
| Review boundary | Draft to review | Revised example |
|---|---|---|
| Assignment | Different arrival groups ignored | Assignment context recorded |
| Result | Causal effect asserted | Exploratory interpretation evidence |
Wide tables scroll sideways on small screens.
Before you send it
Questions, answered.
Can I call this an A/B experiment?
Not from these records. Order is confounded with arrival group and no randomized design is documented.
What can the pilot still improve?
Specific transition wording and independence of prompts, using actual exposure notes and explanations.
Will the example import or configure a results view?
No. Copy the optional text prompts into a manual draft. Tables, coding rules and comparisons are separate educational worksheets; check supported product features in your account.
Build with SurveyTeams
Adapt these pilot questions in SurveyTeams
Use these examples to draft your questionnaire. Create a survey draft, add questions, and check the respondent preview in SurveyTeams. Additional features depend on your account and plan.
Create your survey in SurveyTeams
Opens product registration. Example questions are not imported automatically.
Learn about the product ↗Continue exploring