A Hook That Quizzes You Before You Ship AI Code

By Lazar Milicevic · Published October 19, 2026 · 8 min read

A project called Sphinx blocks your push until you correctly answer questions about the code your AI assistant just wrote. If you run a small business and an AI writes your invoicing logic, lead routing, or email automations, the question underneath is the uncomfortable one: do you know what it wrote? This post covers what Sphinx does, what the research says about why it exists, and a no-code checkpoint you can run today.

What Sphinx Actually Does

Sphinx is a free, open-source Git hook for agent-assisted development. It pauses pushes on protected branches and asks you a question about what the code does before the agent's work ships. A wrong answer gets you an explanation and a new scenario. A right answer lets the push continue. It's MIT licensed, and the stated goal is in its tagline: outsource your thinking, not your understanding.

The design choices are what make it more than a gimmick:

• A separate reviewer model writes the questions. According to the AI/TLDR listing, it reads the committed change through read-only git tools and never sees the conversation that produced the code. So the quiz isn't graded against the coding agent's own story about what it did.
• Questions target behavior, not trivia. The same listing says it asks about behavior, failure modes, and invariants rather than file names, and scales from one to three questions depending on the size of the change.
• It's agent-agnostic. Per the project site, it works with Codex, Claude Code, Cursor, OpenCode, and Pi, or with plain Git. Your answers never reach the coding agent, and .env files and keys are never opened.

Install is two commands, and it needs Node 22.19+ on macOS, Linux, or WSL:

npm install -g sphinx-understand
sphinx install   # asks which model should review

The reviewer can run on your existing coding agent, OpenRouter, or any OpenAI-compatible endpoint.

Two caveats up front. It's brand new, I haven't verified how well the questions hold up on large diffs, and Hacker News reaction was mostly skeptical. And the README is upfront that this is practice, not a security gate: anyone with local access can skip it. There's even a sphinx yolo command that lets pushes through without the quiz, though every skip is logged. Treat it as an early experiment with a good idea in it.

Why This Is an Ownership Problem, Not a Developer Problem

The failure Sphinx targets isn't "bad developers write bad code." It's that approval has become a rubber stamp. When an assistant hands you a finished change in seconds, the easiest move is to click accept.

For a founder, this gets worse. When AI writes your invoice generator, your lead follow-up sequence, or your Gmail triage rules, the output looks right and usually runs right. The danger is the unusual case:

• An invoice with a zero-quantity line item
• A lead whose email address contains a plus sign
• A reply that lands in the wrong thread

Here's an illustrative scenario. Picture a small invoicing operation that was hand-typing a few hundred invoices a week and automated it with AI-written logic. The first version silently rounds tax the wrong way on one product category. Nobody catches it for weeks, because nobody on the team can read the code and the output looks plausible. That's the real cost of approving what you don't understand: errors that are quiet, repeated, and expensive. This is an example, not a statistic.

The same logic applies to anyone with audit exposure. A manual data-entry mistake and an automation mistake are the same liability wearing different clothes.

What the Research Says (and Doesn't)

Sphinx's premise is backed by one well-designed study, with limits worth stating plainly. In Anthropic's randomized controlled trial with 52 developers, participants using AI assistance scored 17% lower on a quiz covering concepts they had used minutes earlier, which Anthropic describes as nearly two letter grades.

The details from InfoQ's write-up:

That last row is the useful one. The tool wasn't the problem; the usage pattern was. Asking the AI to explain things preserved understanding. Handing it the work and accepting the output did not.

Now the limits, because they matter before you generalize:

• The study was by Judy Hanwen Shen and Alex Tamkin, posted as the arXiv preprint 2601.20245, "How AI Impacts Skill Formation." Participants learned one unfamiliar Python library, Trio.
• It used a chat-based assistant, not an agentic tool like Claude Code. The authors expect the effect of agentic tools to be more pronounced, but that's an expectation, not a measurement.
• It tested developers. I found no source on how well a quiz like this works for non-coders, so don't read the 17% as a number about small-business owners.

What the study does support is the narrow, reasonable claim: delegating without engaging costs understanding, and engaging with the AI's reasoning protects it.

A Five-Minute Checkpoint That Needs No Code

Sphinx requires Node and a Git workflow, so it isn't for most founders. The principle transfers without it. Before anything an AI built goes live, run this. It's my own checklist, not a Sphinx feature.

Step 1: Make it explain itself. Ask the AI to describe its change in plain language: what it does, what it touches, and what it assumes.

Step 2: Make it show the ugly inputs. Ask for the three worst inputs this could receive (an empty value, a duplicate, something absurdly large) and have it show you the output for each one.

Step 3: Make it describe the failure path. Ask what happens if this fails halfway through, and whether anything gets sent or charged twice.

Step 4: Close the explanation and answer from memory. This is the Sphinx part. Try to answer the questions yourself, or have a second person try. If neither of you can, it isn't ready.

A prompt you can paste as-is:

You just built or changed this automation. Before I approve it:
1. Explain in plain language what it does, what it touches, and what it assumes.
2. Give me the 3 ugliest inputs it could receive (empty, duplicate, absurdly large)
   and show the exact output for each.
3. If it fails halfway, what is left in a half-done state? Can anything be sent,
   charged, or deleted twice?
4. Then ask ME three questions about this change that I should be able to answer
   before it goes live.

Then write the three answers in a single note stored next to the automation. Six months from now, that note is the difference between knowing what the thing was supposed to do and reverse-engineering it during an incident.

One honest limit: asking the same AI to quiz you is weaker than Sphinx's design, where a separate model that never saw the coding conversation writes the questions. If the stakes are high, use a different model for step 4, or a second human.

Design the System So Consequences Are Visible

Here's my hot take: Sphinx is the right instinct pointed at the wrong audience. A quiz inside a dev tool helps people who were already going to review code. The founders who most need a checkpoint are the ones who never open the code at all.

For them, a harder quiz isn't the answer. The answer is a system where risky actions always route through a visible approval step with a log. Understanding every line is unrealistic for a solo operator. Understanding every consequence is not.

In practice, that means classifying every automation action by blast radius:

# approval-policy.yaml (example structure)
actions:
  read_only:            # summarize inbox, draft replies, pull reports
    approval: none
    log: true
  reversible_write:     # label emails, update CRM notes, create draft invoice
    approval: none
    log: true
  irreversible_or_external:   # send email to client, charge card, delete records,
                              # send money, publish
    approval: required_human
    log: true
    show_preview: true

The rule of thumb: anything that sends money, deletes records, or emails a client gets a preview and a human click, and every action gets a log line. Bypasses are allowed (Sphinx's own yolo mode proves people will demand one), but they get logged too. A bypass you can see is a control; a bypass nobody can see is a hole.

This is also why I'd call Sphinx a Level 1 control, as one DEV Community writer framed it: useful as a forcing function, not a guarantee. Real assurance comes from layering it with approval gates, logs, and tests on the behaviors that touch money.

Putting This Into Practice

This is how I approach designing automations: approval and logging belong in the design from the start rather than being bolted on afterward. Anything that emails a customer, creates a financial document, or modifies records should pass through a visible approval step, and actions should be logged somewhere the owner can read. The goal is the same one this whole post is about: the AI does the repetitive work, and a human stays in the loop at the moments where mistakes cost money.

What to Do This Week

If you write or ship code with an agent, try Sphinx on a non-critical repo and see whether the questions are any good on your real diffs. If you don't touch code, run the four-step checkpoint on the next AI-built change that goes live, and save the three answers in a note. Then list every action in your automations that sends, charges, or deletes, and confirm each one has a preview and a log. If you can't answer what happens when one of them fails halfway, you've found your first thing to fix.

Want more like this?

I publish practical AI automation content. Planning an automation project or need a second opinion on your architecture?

Lazar Milićević, senior engineer building practical AI automation for solopreneurs and small teams.

Visit bizflowai.io for services, case studies, and AI consulting.