On reflection

Quote of the Day
”Giving up smoking is the easiest thing in the world. I know because I’ve done it thousands of times.”
- Mark Twain
Musical alternative to the morning’s radio news
Tom Rush | What’s Wrong with America
Wonderful.
Long Read of the Day
What do you get if you make AI eat its own words? Not what you originally asked for
My recent Observer column
Here’s an experiment you probably can’t try at home. Take a large language model (LLM) and prompt it with a passage about English church architecture of the 14th century; you know the kind of stuff – towers, master masons, the perpendicular style, and so on. Straight out of Pevsner, if you like. Ask it to carry on writing. Then train a fresh model on what the first one wrote, and ask it to do the same.
Keep going, generation after generation. By the ninth, you find that the machine has given up talking about churches and is producing an earnest list of jackrabbits with tails of various colours: black, white, blue, red and yellow.
You think I made this up? Nope. It’s one of the illustrations in a paper published in Nature by researchers at, among others, Oxford, Cambridge, Imperial College London and Toronto University who wanted to know what happens when AI systems start learning from the work of other AI systems. Their answer: the machines go gently, and then not so gently, potty. The technical term for it is “model collapse”.
Something similar used to plague us in the days when all media were analogue. You photocopied a document, and the person you sent it to copied it and sent it on. By the time it reached the ninth person in the chain, its readability had been drastically reduced. Same with audio cassettes recorded at a Grateful Dead gig (as I know to my cost). But when digital technology arrived, that nightmare disappeared. The millionth copy of a digital document was as sharp as the original.
So you can see why it’s a shock to discover that this doesn’t hold for LLMs. And it explains why US author Ted Chiang’s observation in the New Yorker in February 2023 that “ChatGPT is a blurry jpeg of the web” was so true.
The research that led to the paper in Nature began shortly after ChatGPT broke cover in November 2022, when the investigators began to wonder about what would happen when LLMs became ubiquitous. This was bound to happen because the launch of OpenAI’s chatbot triggered a tsunami of imitators. All these new machines were, like ChatGPT, trained on masses of text scraped from the web, and it rapidly became clear that this training data constituted a critical resource for a fast-growing industry.
Initially, the vast majority of it had been generated by humans. But as AI models proliferated, and the industry became hooked on the supposed “scaling hypothesis”– the idea that big was always better as far as AI models were concerned – the demand for training data grew exponentially. The trouble is that the supply is not infinite. One study estimates that the AI industry will have exhausted the supply of publicly available, human-generated training data sometime between now and 2032.
What will happen then? In the Panglossian world of AI boosterism, the answer is obvious: LLMs are terrific at generating text in huge quantities, so there will be plenty of “synthetic” data to train the models of the future. QED.
Not so fast. Unless the model collapse problem is solved, we’re heading into what Jathan Sadowski, a technology researcher at Melbourne-based Monash University, memorably called “Habsburg AI”. He used it to describe models trained so heavily on the output of other LLMs that they become inbred mutants, rather like the Spanish bloodline of the famous European dynasty, which inbred itself to extinction. Or, to put it more crudely, it may mean a kind of accidental enshittification of LLMs.
Which would be a disaster because these models are – as the psychologist Alison Gopnik argued from the start – cultural technologies: tools that people can use to access the knowledge that other humans have accumulated over a very long period. Opening Claude or Gemini is like entering a huge library that also happens to have an indefatigable librarian (if occasionally an overimaginative one) who is willing to search assiduously for information that you’re seeking.
Until recently, access to intellectual riches on this scale was a privilege enjoyed by a small fraction of humanity. We stumbled upon it through a series of happy accidents: machine-learning algorithms, neural networks, the transformer model and the oceans of data provided by the web. Having got it, are we now going to degrade it?
Books, etc.

I discovered that Antonin Baudry’s remarkable two-part movie about Charles De Gaulle was inspired by Julian Jackson’s biography, so I’ve just bought it and am digging in. More later.
Linkblog
Something I noticed, while drinking from the Internet firehose.
- Among the Wagner fanatics
I am not a Wagner fan, though I occasionally like the noise he makes. So you can perhaps understand why I enjoyed Rosa Lyster’s essay in the LRB on being at the Wagner Festival in Bayreuth. Here’s a sample:
Of all Wagner’s operas, Lohengrin is the most swan-concerned. It’s true that Parsifal has a narratively propulsive swan-shooting incident in Act 1, which is a great relief for audience members feeling desperate about the lack of action up to this point. (Everyone on stage has been standing still and shout-singing about the king’s bath for 45 minutes.) It is in Lohengrin, however, that swans actually matter. A mysterious knight arrives on a boat drawn by a swan to rescue Elsa, who has been falsely accused of murdering her brother, Gottfried. The knight marries her on the condition that Elsa never asks his name. Inevitably, she ends up doing so, at which point he reveals himself as Lohengrin, a knight of the Grail, and departs – but not before turning the magical swan back into Elsa’s lost brother.
It isn’t necessary to care about any of this to have a nice time at the Lohengrin thermal baths, a short drive from Bayreuth, the Bavarian town where the Wagner festival has been held almost every summer for 150 years. The baths have indoor and outdoor pools of various therapeutic temperatures, seven textilfrei saunas and expansive shady lawns where visitors can sprawl out on their swan-embroidered complimentary towels.
You can’t identify a Wagner enthusiast just by looking at someone, especially if they are in swimwear. As I studied the other guests at the baths, it seemed possible that none of them had ever thought about Wagner, that they were able to order a Lohengrinsalat at the adjoining restaurant, Bistro Elsa, without Wagner crossing their minds. It seemed unlikely, though. To be in or even near Bayreuth is to have Wagner drilled into your consciousness. The festival lasts for a month, consisting exclusively of repeat performances of his operas, and everything in the town is oriented towards him. Along with all the swans, Wagner’s face is everywhere, on busts and billboards and flyers for hair salons, staring down from ads for Pilates studios with that off-puttingly candid look in his eye and the trace of hysteria about his pressed lips. Hotels and bars and pharmacies are named after his works; the gift shops sell cookie cutters in the shape of his profile and postcards that say CHE GUEWAGNER. His silhouette is on the traffic lights near the festival theatre, a little green man with a baton and a beret bearing up valiantly under the weight of the famously enormous head. The only other place I have visited with as dominant a presiding spirit is Disneyland, though even that doesn’t fully capture it. Imagine if Disneyland had been built by Mickey Mouse himself, as a monument to his own maniacal, epoch-defining genius. That’s Bayreuth…
Lovely, just lovely.
This Blog is also available as an email three days a week. If you think that might suit you better, why not subscribe? One email on Mondays, Wednesdays and Fridays delivered to your inbox at 5am UK time. It’s free, and you can always unsubscribe if you conclude your inbox is full enough already!


















