Quality bar
Human review checklist for AI output
If AI drafted it and you publish it, you own it. This checklist is what we use before guides, release notes, and public skill descriptions go live on chatagent.ca.
Checklist
- Facts, every concrete claim has a source or a verify step.
- Secrets, no keys, tokens, private URLs, or personal data.
- Scope, title matches content; no bait.
- Tone, mythic language labeled as creative where needed.
- Safety, no instructions for crimes or unauthorized access.
- License, third-party code/text attributed; our originals are clear.
- Links, internal links work; external links are intentional.
- Done, if code, Continuum or tests green on a clean check.
Publishing is a privilege
Educational sites fail quality reviews when they flood pages with unedited model sludge. We would rather ship fewer pages with human care. If you reuse our templates, keep the human gate.
Write the acceptance sentence first
Before you read the output, write one sentence that would make it unacceptable. “The price is wrong.” “The name is wrong.” “The link is dead.” “The claim has no source.” “The code deletes a file it should not touch.” That sentence is the review. Everything else is proofreading.
A review that starts with the happy path finds typos. A review that starts with the failure mode finds the reason the page should not ship. If you cannot write the acceptance sentence, the task was not defined well enough to review.
Review by artifact type
Different outputs fail in different ways. Use the checklist for the artifact in front of you, not a generic “looks good.”
| Artifact | Check first | Then check |
|---|---|---|
| Prose or guide | Names, numbers, dates, and the main claim | Sources, scope, tone, links, byline, date |
| Code or command | Run it on a clean copy; read the diff | Error paths, secrets, dependencies, side effects |
| Table or dataset | Units, totals, dates, and the source column | Missing rows, duplicated rows, rounding, labels |
| Links or files | Open every link; open every file | Canonical URL, status, content match, hash |
| Image or screenshot | What is visible at the edges and in the background | Alt text, privacy, cropping, misleading context |
| Release note | Every claim has a test or a link | Version, date, known issues, upgrade path |
| Quoted or licensed material | Attribution and license | Quote length, modifications, commercial use |
The ten-minute pass
- Read once without editing. Mark the sentence that feels too confident, too vague, or too convenient.
- Check names and numbers first. Proper nouns and figures are the cheapest errors to catch and the most expensive to publish.
- Open every link and file. A 200 response is not enough; the destination has to be the right page.
- Run the check. If the output claims a test passed, run it. If it claims a page works, open it at the target size.
- Read the diff, not the summary. A summary can describe work that the diff does not contain.
- Scan for secrets and private data. Keys, tokens, emails, customer names, internal paths, and screenshots.
- Check the scope. Did the output add a page, a dependency, a claim, or a promise that the brief did not ask for?
- Write the sign-off. Name the checks you ran, the date, and the remaining risk.
What to check first
Names and numbers
Check every proper noun against a source and every number against the table it came from. A confident sentence is not a citation. If the output says “the repository has 92 skills,” open the catalog and count; if it says “the page returns 200,” run the request.
“Done” claims
A done claim needs a receipt: a command, a URL, a hash, a screenshot at a named size, or a named human sign-off. “Fixed” without a check is a status update. The Continuum guide covers the receipt pattern.
Links and files
Open the link and read the destination. A page that returns 200 but shows the wrong content is still a broken link. A file that opens but has the wrong hash is the wrong file. A screenshot that looks right at 1440 can be broken at 390.
Scope and claims
Look for the sentence that promises more than the work delivered. “Eliminates,” “guarantees,” “always,” and “never” are review flags until the evidence earns them. A guide can say “this worked in our test”; it cannot say “this always works.”
Sign-off template
A sign-off should let another person repeat the pass without asking what you meant.
REVIEWED BY: DATE: ARTIFACT: CHECKS RUN: - command / URL / file / screenshot RESULT: - pass / fail / pass with caveat REMAINING RISK: - what was not checked and why DECISION: - publish / hold / return to author
“Reviewed” alone is a vibe. The template is short because the review is supposed to be specific.
Failure modes
| Failure | What it looks like | Fix |
|---|---|---|
| Confident invention | A citation that does not exist, or a number with no table | Open the source; remove the claim if it cannot be checked |
| Stale fact | A price, version, or policy from last year | Check the current source and update the date |
| Scope drift | New pages, dependencies, or promises the brief did not ask for | Read the diff; return to the brief |
| Hidden secret | A key in a log, screenshot, or example | Rotate, redact, re-scan |
| Tone mismatch | Hype in a reference page, or jargon in a beginner guide | Check the audience and rewrite the opening |
| Missing caveat | A claim that is true only under conditions the page omits | Add the condition or narrow the claim |
| Broken at mobile | The desktop screenshot passed; the phone layout did not | Check 390px and 1440px before sign-off |
What to do with a failure
Do not ask the same model to “try again” without changing the brief. Name the failure, add the missing constraint, and rerun. A retry without a new constraint is a coin flip with a nicer interface.
Keep the failed output and the correction together. The pair shows where the brief was ambiguous, which is more useful than a clean success with no history. If the failure is a security or legal issue, stop and escalate before editing further.
When to escalate
- Security: a leaked credential, a destructive command, or an unexplained network call. Rotate first, then investigate.
- Legal or licensing: a quote beyond fair use, a missing attribution, or a license that forbids the use.
- Medical, legal, or financial advice: the page needs a qualified professional, not a model. Do not publish a confident substitute.
- Personal data: a real person who did not consent to appear. Remove the data and follow the privacy process.
FAQ
How long should a review take?
Long enough to run the checks. A short page can take five minutes; a code change can take an hour. The time is set by the artifact, not by a target.
Do I need to read every word?
Read for meaning once, then check the high-risk parts closely: names, numbers, links, claims, code, and secrets. Skimming the middle of a safe paragraph is fine; skimming the number is not.
Can I use an AI reviewer?
Yes, as a second pass. It can find inconsistencies and missing links. It cannot take responsibility for the publish decision, and it can invent a reason a claim is fine. Keep the human sign-off.
What if the output is good but the brief was bad?
Say so. A good output from a bad brief is luck. Update the brief before reusing the result, or the next run will not reproduce it.
What if I cannot verify a claim?
Label it as unverified or remove it. A page with a smaller verified claim is stronger than a page with a larger unverifiable one.
