Quiz design checklist

A quiz should test a defined capability, not reward familiarity with tricky wording. State the learning or readiness objective first, then choose questions that produce evidence for that objective.

Define the objective

• Complete the sentence: "After this quiz, we should know whether the respondent can..."
• Choose the cognitive level the work requires: recall, explain, apply, analyze, evaluate, or create.
• Use a scored choice question only when one best answer can be defended.
• Use an unscored text or code field when reasoning, application, or creation matters more than recognition.

Multiple-choice checklist

• Write a complete stem that makes the task clear before the options are read.
• Use familiar, concise language and include only context needed to solve the problem.
• Provide one best answer and three to five plausible alternatives.
• Base distractors on realistic mistakes, not jokes, trivia, or obviously different wording.
• Keep alternatives parallel in grammar and similar in length. Avoid accidental clues.
• Make alternatives mutually exclusive. Do not use overlapping numeric ranges.
• Avoid trick questions, negative stems, and "all of the above" or "none of the above" as defaults.
• Do not test two objectives in one scored item.

Scoring and feedback checklist

• Assign answer and score only to fields that should affect the result.
• Verify that all scored fields sum to the intended maximum and that pass_score is a percentage.
• Use show_score: true when the result helps the respondent act.
• Use show_answers: true for learning and training; keep it off for reusable secure assessments.
• Add an unscored rationale or uncertainty field when the agent needs to understand guessing or misconceptions.
• Review weak items after responses arrive. A question that nearly everyone misses may expose poor instruction or ambiguous wording rather than poor understanding.

Match questions to cognition

• Remember: identify a fact, term, or required step.
• Understand: select the best explanation or example.
• Apply: choose the correct action in a realistic scenario.
• Analyze: distinguish evidence, assumptions, causes, or tradeoffs.
• Evaluate: choose and justify a decision against stated criteria.
• Create: use a code, file, or text field and assess the produced artifact rather than forcing a multiple-choice proxy.

Methodology sources

This checklist adapts guidance from:

• University of Waterloo: Designing Multiple-Choice Questions on complete stems, one best answer, plausible distractors, clear alternatives, and avoiding common item-writing shortcuts.
• Carnegie Mellon Eberly Center: Bloom's Taxonomy on aligning assessment tasks with increasingly complex cognitive objectives.

The checklist applies those principles to Motivation Form's scoring model. It does not replace subject-matter review of the answer key.

Start from apps/motivation-form/forms/agent-feedback-quiz.md, then follow the agent feedback workflow.