Same news, different words

2026-10-24T00:00:00Z

Little Thot is built for serendipity: the useful accident. You read something today, and it tells you that it connects to a note you wrote months ago, or to something you're in the middle of. You can't search for a connection you don't know is there, so the system has to notice it for you, and then decide whether it's worth interrupting you.

That decision can fail in two directions. Too loose, and it becomes noise you learn to ignore, and the useful accidents get lost in it. Too strict, and they never happen at all. The second failure is the dangerous one, because it's silent: nobody misses a message they never received.

This is a write-up of the second kind of failure, found in the smallest part of the system. Little Thot is also a laboratory for studying agent systems: where code should decide and where a model should, what data leaves our own machines, what it costs. We run it every day and write up what we find.

A quiet news feed

At regular points during the day, Little Thot reads technology news front pages and flags the few stories that touch what we're working on. It turns each headline into a vector with a small embedding model running locally, and compares that with the topics in our working profile and with our own notes and ideas. No language model is involved, and our notes and profile never leave our own machines. A story that scores high enough against either is a match.

After two days it had sent three stories. That felt low, so we followed every story through.

317front-page stories seen in the first two and a bit days
311never came close to either bar
6cleared a bar, against a profile topic or a note
2skipped: we'd already read them, hours before they reached the front page
4matched
1dropped as a repeat of news already sent
3sent

Where 317 stories went. Skipping pages already read is deliberate, and it worked. The story dropped as a repeat was the strongest match of the period.

The plot below shows the same thing from the other side: every story's best score against a profile topic and against a note or idea. Almost everything sits in a dense cloud far from both bars. The handful that clear one are on their own.

Sent
Already read, skipped
Dropped as a repeat
Every other story

Every story's best match. A story is a match if it clears either bar. Scores were recomputed on 23 October, and one story that was sent on day one now sits just under the topic bar (0.320 then, 0.3197 now), because the notes everything is compared against have changed since. Scores drift as the reference material grows.

The six stories as a table

Outcome	Best topic	Best note
Dropped as a repeat	0.459	0.180
Already read	0.362	0.576
Sent	0.363	0.335
Already read	0.343	0.313
Sent	0.320	0.204
Sent	0.110	0.540

The repeat check

When something big happens, several sites cover it and each version can reach the front page. So before a match is sent, one more check runs: does this headline look like one already sent in the last three days? If its similarity to an earlier match is over a bar, it's dropped as the same news.

The bar was set on the first day from the 89 stories seen so far. A note in our code recorded that same-news headlines scored at least 0.36 against each other and different stories at most 0.24, so a bar at 0.33 looked safe.

On the second morning the strongest match we'd had, a disclosure about a compromised certificate authority, was dropped as a repeat of an earlier match about a compromised software package. The two have nothing in common except their first word. Their headlines scored 0.37 against each other, just over the bar.

What the scores actually look like

To see how often this happens, we compared every story's headline with every other one: 50,086 pairs. Then we labelled the pairs that really were the same news, by grouping the stories into news events by hand. Thirteen events had more than one story, from a court ruling reported by five outlets to a software vulnerability covered from ten angles over two days.

Different stories (share of pairs)
Same news (one dot per pair)

Headline similarity, same news against different stories. The grey bars are the 49,981 pairs of different stories, as a share of all of them; each dot below is one pair about the same news. A repeat check can only work if the dots sit to the right of the grey. They don't.

The numbers as a table

	Day one (89)	All 317
Pairs of different stories	3,897	49,981
Pairs about the same news	19	105
Highest-scoring different pair	0.47	0.74
Same-news pairs, median score	0.36	0.37
At 0.33: different pairs over the bar	54 (1.4%)	1,025 (2.1%)
At 0.33: same-news pairs caught	10 of 19	67 of 105
At 0.60: different pairs over the bar	0	3
At 0.60: same-news pairs caught	2 of 19	22 of 105

Two things stand out.

First, we couldn't reproduce our own calibration. Re-scoring the same 89 stories, different stories reach 0.47, not 0.24. The “same news” range still matches, so the note was probably drawn from a few obvious examples rather than from every pair. Nothing was kept that would let us check, which is part of the lesson.

Second, and more important, there is no gap to put a bar in. Half of the same-news pairs score below 0.37, right in among ordinary unrelated stories. Two sites covering the same event rarely use the same words, and two unrelated headlines often share one.

Why headlines fool similarity

A headline is a few words plus the site it links to, and at that length one word or one domain can carry the whole score.

Different news, looks alike 0.74

OpenAI CLI · github.com

Nvidia OpenShell · github.com

Different news, looks alike 0.37

Compromised LiteLLM 1.107.0 on PyPI Installs a Credential Stealer and Backdoor

CVE-2026-111943: Compromised Intermediate Certificate Authority

Same news, looks different 0.18

Today's PHP security release, and how we're strengthening security

PHP2Shell: AI uncovers critical RCE in PHP

Same news, looks alike 0.78

Meta fined $25B for 'lies' over user data privacy

Meta Ordered to Pay $25B for Deceiving Users on Privacy

Four pairs of real headlines and their similarity. Only the last is the case the repeat check was written for. The first two are why it fired when it shouldn't; the third is why it can't catch most real repeats.

The articles behind the headlines would tell these apart easily. But fetching and embedding every linked page, every time it checks, is a lot of work just to decide whether two stories are the same news.

What we changed, and why

We moved the bar from 0.33 to 0.60. That sounds like tuning; it's really a decision about which mistake to make.

At 0.33, about 2% of unrelated pairs clear the bar. Each new match is checked against every match from the last three days, so with five earlier matches in the window, roughly one good story in ten would be thrown away, and stories that match the same interests are more alike than random ones, so in practice it's worse. In return, the check catches about two-thirds of real repeats.
At 0.60, 3 unrelated pairs in 50,000 clear the bar, so a good story is almost never dropped. But the check now catches only near-identical headlines, about one real repeat in five.

The costs aren't symmetrical. A missed repeat costs one extra message, which we can see and rate. A dropped story costs the very thing the system exists to provide, and leaves no trace. With a few messages a day, we'll take the occasional duplicate.

We also considered only counting a story as a repeat if it matched the same profile topic as the earlier one. On this data it wouldn't have helped: the two “Compromised” stories matched the same topic.

What we're taking from it

Measure the whole distribution, not the examples. A handful of obvious duplicates and obvious non-duplicates will always suggest a clean gap. The overlap only shows up when every pair is scored.
Keep the data behind a threshold. We couldn't check a number we'd set two days earlier, because nothing behind it was kept. Every threshold now gets its data saved alongside it.
Silent failures need a log line. We found this one only because the system recorded why it dropped a story. The next step is to record the near misses as well, so the question “what are we not seeing?” becomes a query rather than an investigation.
For serendipity, prefer the visible mistake. When a filter has to choose, make it fail towards something the person can see and rate.

About the data

317 front-page stories seen between 21 and 23 October 2026, every pair of them scored with the same embedding model the live system uses, and centred the same way. Stories were grouped into news events by hand from their headlines: the same event, incident or announcement, including direct follow-ups within the three-day window the repeat check covers. Every pair outside a group counts as different. Scores were recomputed on 23 October, so they differ slightly from what the live system saw at the time. The scores and labels behind every chart are kept with this site's source, without headlines.

Little Thot - articles