How our readers are tested

Receipts, not claims. Method, current numbers, and limits.

The problem this solves

The documented failure of AI feedback is flattery. Public tests have caught AI feedback tools rating amateur scripts above produced classics, and the Writers Guild's own member guidance names sycophancy as one of the known hazards of AI tools. A reader that praises everything is worse than no reader at all: it costs you rewrites you needed and confidence you did not earn.

So before Auteurium's AI readers were allowed to read anyone's pages, we built them an exam. It is the same idea as a flight simulator: scenes with known, deliberately planted problems, and clean scenes with none, run again and again to see what the readers actually do.

WGA member guidance naming sycophancy: wga.org, AI: know your rights.

The method

• Planted-flaw scenes. We write test scenes that each break one declared craft contract on purpose: a climax with no real opposition, dialogue that states the emotion instead of playing it, a setup that never pays off. The flaw is documented before any reader sees the scene. A reader passes only by finding it.
• Clean scenes. The exam also includes scenes with nothing wrong. The correct read of a clean scene is zero notes and a sharp one-line summary. Notes manufactured on clean pages count against the readers. We call that churn, and we hunt it.
• Articulation. Finding a flaw is not enough. The note has to name the actual organ of the problem, in terms a writer can act on. We score that too.
• The referee. Every note from every reader passes through a referee before you see it. The referee's job is to kill nitpicks, duplicates, and notes that contradict your stated vision for the story.

The numbers, honestly framed

Bench of 15 curated scenes (12 planted-flaw, 3 clean), scored across repeated trials. As of July 2026.

This is a small bench by design. It is curated, not crowd-sourced: every scene, every planted flaw, and every scoring judgment was made by hand, and one fixture is currently pulled from scoring while we repair its design. Fifteen scenes cannot prove our readers are always right. What the bench does prove is narrower and more useful: these readers find real planted problems, they do not invent problems on clean pages, and every change we make to them is measured against that standard.

The keep-or-revert law

Every calibration change to the readers is scored against the bench before it ships. If catch, churn, or articulation gets worse, the change is reverted, whatever we hoped it would do. This has already killed changes we liked. The bench outranks our taste, and it definitely outranks the readers'.

The limits

• The bench is built from English-language, feature-film-shaped material. That is where calibration is strongest today.
• The readers are calibrated for development notes on dramatic writing. They are not coverage, they issue no verdicts, and they will not predict your script's market.
• AI readers can be wrong in ways the bench has not met yet. When a note is wrong, disagree with it. The product is built on the assumption that you are the authority on your story.

Zero notes is a valid read

If the room reads your scene and finds nothing worth your time, it says so and stops. That is not a malfunction. It is the entire point: a reader you can trust when it is quiet is the only kind worth listening to when it is not.