# Code open survey comments with a reviewable codebook Build a small survey-comment codebook, retain multi-theme cases and distinguish theme mentions from the number of respondents. If nothing was unclear or difficult, say so. Skip questions about steps you did not experience; use not applicable where appropriate. ## 1. What information or access was missing when you first tried the task? Answer: ____________________ ## 2. Which instruction was difficult to interpret? Answer: ____________________ ## 3. What worked without a problem? Answer: ____________________ ## 4. Is there another explanation we should consider for the difficulty you described? Answer: ____________________ ## Author notes / illustrative keys where applicable 1. Separate missing information from unavailable permissions without asking for credentials. 2. Identify wording or sequencing ambiguity for a clarity code. 3. Preserve no-issue evidence rather than selecting only complaints. 4. Leave room for competing explanations and uncertain coding. Interpretation caution: Themes can overlap. A coded comment is not automatically a distinct respondent, and a volunteered comment sample does not establish issue prevalence. ## Illustrative finding and follow-up Observation: Synthetic comments C1 and C5 concern access, C2 clarity, C3 access plus clarity and C4 no issue. Action to test: Write access, clarity and no-issue definitions; keep C3 in both issue codes and retain an uncertainty/reviewer column. Follow-up: Have another reviewer code the same small sample using the definitions, discuss disagreements and version the codebook before applying it more broadly. ## Collection protocol When and whom to ask: Ask after a specific onboarding task. Explain that comments can be skipped and should avoid identifying colleagues. Decision owner: The qualitative reviewer owns code definitions; the process owner investigates examples without treating counts as blame. Follow-up: Recode a small unchanged sample with the revised definitions and check agreement through discussion, not an invented reliability score. Response handling: Remove names, credentials and unnecessary personal details before sharing comments. Do not promise anonymity if comments can be linked back to a person. ## Worked measurement study All example records are synthetic; manual worksheet, no automatic import. ### Inspect the synthetic example Invented comment ID | Invented text | Proposed codes --- | --- | --- C1 | Could not open the required tool | Access C2 | The first-step instruction was unclear | Clarity C3 | Tool access was missing and the instruction was unclear | Access; clarity C4 | Nothing blocked the first task | No issue C5 | The needed permission arrived late | Access ### A codebook another reviewer can use Code | Include when | Exclude or leave unresolved --- | --- | --- Access | A required tool, account or permission was unavailable or late. | Do not assign access from a vague negative comment alone. Clarity | The person identifies unclear instructions, wording or sequence. | Do not assign clarity only because access was unavailable. No issue | The person explicitly states no obstacle for the task. | Do not infer no issue from a blank answer. Unresolved | The comment is too vague to distinguish a cause. | Keep the uncertainty; seek optional clarification without forcing an issue code. ### Derivation Access appears in 3 of 5 comments; clarity in 2; no issue in 1. Six assigned theme mentions belong to five comments because C3 has two codes. Coding note: five comments, six theme mentions and one multi-theme case. Retain the codebook and disagreement notes; do not claim population prevalence. ### Evidence boundary Observed / Five example comments and proposed codes / Supports a transparent coding demonstration. Unknown / Prevalence among everyone invited / Do not generalize volunteered comments to all users. Next check / A second reviewer codes the unchanged sample / Revise unclear code boundaries and retain contrary cases. ### Method references AAPOR survey best practices: https://aapor.org/standards-and-ethics/best-practices/ — Pretesting and transparent sample context.