Same news, different words

Little Thot is built for serendipity: the useful accident. You read something today, and it tells you that it connects to a note you wrote months ago, or to something you're in the middle of. You can't search for a connection you don't know is there, so the system has to notice it for you, and then decide whether it's worth interrupting you.

That decision can fail in two directions. Too loose, and it becomes noise you learn to ignore, and the useful accidents get lost in it. Too strict, and they never happen at all. The second failure is the dangerous one, because it's silent: nobody misses a message they never received.

This is a write-up of the second kind of failure, found in the smallest part of the system. Little Thot is also a laboratory for studying agent systems: where code should decide and where a model should, what data leaves our own machines, what it costs. We run it every day and write up what we find.

A quiet news feed

At regular points during the day, Little Thot reads technology news front pages and flags the few stories that touch what we're working on. It turns each headline into a vector with a small embedding model running locally, and compares that with the topics in our working profile and with our own notes and ideas. No language model is involved, and our notes and profile never leave our own machines. A story that scores high enough against either is a match.

After two days it had sent three stories. That felt low, so we followed every story through.

The plot below shows the same thing from the other side: every story's best score against a profile topic and against a note or idea. Almost everything sits in a dense cloud far from both bars. The handful that clear one are on their own.

The repeat check

When something big happens, several sites cover it and each version can reach the front page. So before a match is sent, one more check runs: does this headline look like one already sent in the last three days? If its similarity to an earlier match is over a bar, it's dropped as the same news.

The bar was set on the first day from the 89 stories seen so far. A note in our code recorded that same-news headlines scored at least 0.36 against each other and different stories at most 0.24, so a bar at 0.33 looked safe.

On the second morning the strongest match we'd had, a disclosure about a compromised certificate authority, was dropped as a repeat of an earlier match about a compromised software package. The two have nothing in common except their first word. Their headlines scored 0.37 against each other, just over the bar.

What the scores actually look like

To see how often this happens, we compared every story's headline with every other one: 50,086 pairs. Then we labelled the pairs that really were the same news, by grouping the stories into news events by hand. Thirteen events had more than one story, from a court ruling reported by five outlets to a software vulnerability covered from ten angles over two days.

Two things stand out.

First, we couldn't reproduce our own calibration. Re-scoring the same 89 stories, different stories reach 0.47, not 0.24. The “same news” range still matches, so the note was probably drawn from a few obvious examples rather than from every pair. Nothing was kept that would let us check, which is part of the lesson.

Second, and more important, there is no gap to put a bar in. Half of the same-news pairs score below 0.37, right in among ordinary unrelated stories. Two sites covering the same event rarely use the same words, and two unrelated headlines often share one.

Why headlines fool similarity

A headline is a few words plus the site it links to, and at that length one word or one domain can carry the whole score.

The articles behind the headlines would tell these apart easily. But fetching and embedding every linked page, every time it checks, is a lot of work just to decide whether two stories are the same news.

What we changed, and why

We moved the bar from 0.33 to 0.60. That sounds like tuning; it's really a decision about which mistake to make.

• At 0.33, about 2% of unrelated pairs clear the bar. Each new match is checked against every match from the last three days, so with five earlier matches in the window, roughly one good story in ten would be thrown away, and stories that match the same interests are more alike than random ones, so in practice it's worse. In return, the check catches about two-thirds of real repeats.
• At 0.60, 3 unrelated pairs in 50,000 clear the bar, so a good story is almost never dropped. But the check now catches only near-identical headlines, about one real repeat in five.

The costs aren't symmetrical. A missed repeat costs one extra message, which we can see and rate. A dropped story costs the very thing the system exists to provide, and leaves no trace. With a few messages a day, we'll take the occasional duplicate.

We also considered only counting a story as a repeat if it matched the same profile topic as the earlier one. On this data it wouldn't have helped: the two “Compromised” stories matched the same topic.

What we're taking from it

• Measure the whole distribution, not the examples. A handful of obvious duplicates and obvious non-duplicates will always suggest a clean gap. The overlap only shows up when every pair is scored.
• Keep the data behind a threshold. We couldn't check a number we'd set two days earlier, because nothing behind it was kept. Every threshold now gets its data saved alongside it.
• Silent failures need a log line. We found this one only because the system recorded why it dropped a story. The next step is to record the near misses as well, so the question “what are we not seeing?” becomes a query rather than an investigation.
• For serendipity, prefer the visible mistake. When a filter has to choose, make it fail towards something the person can see and rate.

About the data

317 front-page stories seen between 21 and 23 October 2026, every pair of them scored with the same embedding model the live system uses, and centred the same way. Stories were grouped into news events by hand from their headlines: the same event, incident or announcement, including direct follow-ups within the three-day window the repeat check covers. Every pair outside a group counts as different. Scores were recomputed on 23 October, so they differ slightly from what the live system saw at the time. The scores and labels behind every chart are kept with this site's source, without headlines.