We Built an AI Newsletter That Fact-Checks Itself

I did not add a fact-checker to NLFactory because the drafts were bad. I added it because I could not prove they were good. Every edition read clean, cited its sources, and sounded sure of itself. When a friend asked me how often the claims held up, my honest answer was that I did not know. That answer bothered me enough to build the thing that could give me a real one.

What self-checking actually means

When the engine finishes writing an edition on a newsletter with checking enabled, a second pass starts. It pulls the factual claims out of the finished draft: the statistics, the dates, the events, the attributions. Each claim is compared against the sources retrieved for that same edition during research. The verdicts are recorded by the pipeline while it runs, not tallied by hand afterward, and the pass finishes in seconds.

A claim that fails does not ship as written. Depending on the newsletter's settings it is visibly labeled, corrected against its source, or cut from the edition. The first time I watched it work, the checker cut a sentence I liked. The draft said a storm had doubled in strength overnight. The advisory it cited said sustained winds rose by a third. The prettier sentence lost. That trade is the entire point.

The engine doing the arguing has a name

We call it NLFiq. It is the engine that researches, writes, and cites every NLFactory edition, and the fact-checking is part of how it works rather than a bolt-on. Citations are on every edition regardless of settings. The deeper verification pass, our Fact-Check Library, runs on newsletters that enable it. If you want the full picture of the engine, NLFiq has its own page now.

Naming it mattered more than I expected. Once the engine was a thing with a name, promising that it checks its work stopped being marketing copy and started being a spec. If an edition ships with a claim its sources do not support, that is not bad luck. That is NLFiq failing at its job, and the scorecard will say so.

We publish the misses

Since May the checker has graded thousands of claims from real editions, and the headline finding has not moved much: roughly one claim in four could not be verified as written, and the biggest bucket of all was claims that were only partially right. I wrote up the first six thousand claims in detail, and the running totals live on a public scorecard that refreshes daily from the production database, licensed so anyone can cite it. Most accuracy numbers in this industry come from lab benchmarks. Ours come from the mail we actually send.

An engine that grades itself in public has no room to bluff.

The scoreboard keeps us honest

Publishing the scorecard was the best product decision we made this year, because it makes cutting corners impossible without an audience. NLFactory sends newsletters you can check, written by an engine that checks itself first. The next time someone asks me how often the claims hold up, I hand them the link.