Nabil Khoury
Work / AI / Falsify
AI2026Personal projectClaude Code

Try to kill every explanation before you believe one

When a key number drops, teams tend to grab the first story that sounds right. Falsify makes an AI assistant do the opposite: list the competing explanations, decide in advance what would disprove each one, then run the queries.

See the code on GitHub ↗Ask me about this

Problem

When a key metric drops, teams act on the first explanation that sounds right.

Insight

The analysis isn’t the hard part. The discipline is. Once you’ve seen the data, every story fits.

Decision

Write the explanations and what would kill each one before the first query, and keep the AI on read-only rails.

Result

Runs end to end on a synthetic case. No real customer yet, and that’s the next test.

4–6competing explanations per case
3possible verdicts for each one
0queries outside the read-only runner

The slate every case starts with

From the demo case: D7 retention dropped, and the team suspects a pricing change.

H-prior
“The pricing flag killed retention”The team’s own suspicion, tested in its strongest form
KilledSurvivedUnderpowered
H1
Measurement artifactThe number moved because the counting changed
KilledSurvivedUnderpowered
H2
Mix shiftA different mix of users showed up
KilledSurvivedUnderpowered
H3
Another product changeSomething else shipped in the same window
KilledSurvivedUnderpowered
H4
Outside or seasonalA holiday, a campaign, something outside the product
KilledSurvivedUnderpowered
Each explanation gets a kill condition written before the first query runs, then exactly one verdict. The team’s favorite gets no special treatment.

01Who it’s for

Product and growth leadsA metric moved and leadership wants an answer this week. Need the real cause, not the loudest one.
AnalystsNeed every query shown, so the answer survives review.
The operatorOne person and an AI assistant running cases as a service for design partners.

02The problem

“D7 retention dropped after the pricing change.” It sounds right, so the team rolls the pricing back.

Nobody checked whether the logging changed, whether a different mix of users signed up, whether something else shipped that week, or whether it was a holiday. Pick the wrong cause and you fix the wrong thing, and the metric stays down.

03The insight

People and AI both fit a story to data once they’ve seen it.

So the rules have to be fixed before the first query: write the explanations, write what would disprove each one, then look. The team’s own suspicion is the most dangerous one, so it goes on the slate in its strongest form and gets no special treatment.

04How I measured success

What I optimized

Right verdicts on past cases where the true cause is knownEvery finished case becomes a test scenario. Changes to the AI’s instructions have to pass them.

The obvious metric I rejected

How convincing the report readsAn AI is very good at sounding convincing. That’s the failure, not the goal.

05How it works

  1. IntakeThe metric, defined exactly, plus the time windows and the suspicion./new-case
  2. Pre-registerWrite the explanations and what would kill each. Then freeze the file.hypotheses.md
  3. QueryEvery query goes through a read-only runner and is shown in the report.SELECT only
  4. VerdictsStatistical tests decide killed, survived, or underpowered.z-test, diff-in-diff
  5. LearnOnce the real answer is known, the case becomes a test./postmortem

06Decisions

Start as a concierge service, not software
Why
One operator with an AI assistant can run real cases and learn which analyses people need before building a product around them.
Instead of
A self-serve web app, which is out of scope for version one.
Cost
It only scales as far as one person can run cases.
Decide what would disprove it before looking
Why
Otherwise the analysis bends to whatever the data shows.
Instead of
Refining the explanations as results come in.
Cost
A badly specified kill condition can’t be fixed later. The honest verdict is then underpowered.
The AI never touches the data directly
Why
Every query runs through one guarded runner that only reads, refuses SELECT *, and blocks personal columns like email or phone.
Instead of
Letting the assistant query freely.
Cost
Slower. Far easier for a client to trust with their data.
Evals gate every change to the AI’s instructions
Why
A prompt tweak that reads better can quietly get verdicts wrong.
Instead of
Judging prompt changes by reading a few outputs.
Cost
Every change costs a test run.

07What I left out

Left outWhy
SaaS, accounts, a web UINo evidence yet that anyone wants it as software.
Agent frameworks and vector databasesThe repo is the memory, and search is good enough at this size.
Direct calls to an AI APIThe AI runs in the operator’s session. The code stays deterministic.

08Where it stands

It runs end to end on a synthetic case with about 50,000 seeded users and 120 days of activity, built around the pricing suspicion above. The evaluation harness is in place.

The honest gap: I built the rails before running a real case. For a concierge service that’s the riskiest assumption, so it’s the next thing to test, not the next feature.

  • No real-world cases yet, so no results to report.
  • Synthetic data can’t show how messy a real warehouse is.

09What’s next

  1. Run one real case with a design partnerThe question is whether teams want this, not whether it runs.first
  2. Turn its postmortem into the first real evalSynthetic scenarios only test what I thought to plant.then

10Built with

Claude Code skillsPythonDuckDBSQLstatsmodelsJinja2