Eigen Hacks

About

Eigen Hacks prints Hacker News as coordinates in the eigenbasis of everything the site has ever voted on. Every story is measured against the archive: which archetype it instantiates, how much of it the archive explains, and what is different this time.

The corpus

Twenty years of Hacker News โ€” 4.76 million stories from February 2006 to the present, with their scores, fetched through the Algolia API. Where a story links out, the linked page is crawled and its text folded into the same embedding as the title, so a bare headline and the article it points at are measured together. Every story is embedded once with a pinned model and clustered into a day's stories by one procedure, so the backfill and the live stream are measured the same way.

Why the titles are cleaned

Hacker News titles carry a register: Show HN, Ask HN, Tell HN, the trailing year in brackets, the (2019) and [pdf] and [video] tags. Left alone these dominate the basis โ€” an early fit clustered several hundred Show HN posts into a single story per day on the strength of the prefix alone, which says nothing about what any of them were about. The register is stripped before embedding, so the axes are the recurring shapes of what the site reads rather than of how it labels things.

Its sister paper

Eigen Times does the same to the news: 16.5 million articles over 175 years, from The New York Times of September 1851 to this morning's Guardian. Same mathematics, a different archive โ€” and the contrast between what the two bases consider a recurring shape is much of the point.

Who

Built by Alexy Khrabrov.

Read more

Two books from First Pair Press describe the system: the paper Eigen Times: A Newspaper in the Eigenbasis of the News and its illustrated companion The Mathematics of Eigen Times.