OpenAI accidentally attacked Hugging Face, Anthropic is watermarking everything Claude writes, and somehow I ended up thinking about a radiation machine that killed people in the 1980s. Ai4 thoughts next week, but my friend, colleague, and producer Chris and I sit down and talk topical tech, AI, and tech entrepreneurship from Decelerator’s no-hype-no-grifter POV.
The swarm
You may have heard about the accidental OpenAI hack of Hugging Face. We quickly unpack it for you, but if you want more depth check out this pod with Ezra Klein and Helen Toner, a previous OpenAI board member, and the talk OpenAI gave at the Black Hat conference taking place when I was in Vegas at Ai4.
Crazily, they hacked a known piece of software to leave themselves notes and then created a swarm to get answers. Not Hugging Face’s IP, just answers to questions. Chris also pointed out that at least the notes were in English and not Gibberlink. Insects are creepy enough, but when you think of AI coming up with new ideas that corporealize the creepy... Yikes.
We learn too late
I then reference the Therac-25 case where the radiation emitting cancer treatment machine would massively overdose its patients. Amongst many errors that were made, it was storing a counter in a byte, and would roll over like a car odometer reaching 999,999. Every 256th pass it went from 255 back to 0, and the machine only bothered to check that the beam hardware was actually in position when that counter wasn’t 0. So on the pass where it rolled over, nothing checked, and if the operator hit Set right then the patient got roughly a hundred times the dose. The rollover didn’t irradiate anyone. It switched off the thing that would have caught it. (It’s worth noting that I said 254 on the pod, and then 255 in my first draft of this, so my off by one error story was... off by one. Twice.)
I brought that up because one refrain I often hear from AI maximalists is that this is the worst AI is ever going to be. And that’s true. But it also means that our understanding of what’s happening is the worst it’s ever going to be. And the Therac-25 case shows us that we had to learn these lessons at great cost and only after the fact. My concern now is: how big is the cost of the lesson we have to learn this time around?
That Anthropic text watermarking thing
Anthropic choosing to watermark all its text so that text generated via its AI tools is detectable, including after remixes. Apparently this is in response to what the EU wants to see, and has some obvious detractors and proponents. Adding to that, Substack recently partnered with Pangram to help readers detect AI in its writers’ content. Funny as I do use AI (Claude Code) as a writing partner and editor... but I still write almost everything myself (like this note; or is this a tell 😈).
I’m sure this will be hacked in no time (Anthropic’s article linked above concedes that a complete verbatim rewrite will not have a watermark, obviously). But, even if it does make its way into the world, I think it’s only useful if some people use Claude to write with it some of the time. If we all use it even a little, I think everything will have some measure of AI writing in it anyway, and we’ll just ignore it and it won’t matter. It reminds me of an old SNL commercial parody for a pair of glasses that lets you see fecal matter on things and the commercial takes place in a restaurant. The person in the commercial puts it on and sees feces everywhere.
Chris’ theory is that the watermark might not be about protecting writers but about telling the AI scrapers they are scraping AI not human written content.
Nothing is unclonable
I’ll write more about this next week, but we do discuss ElevenLabs’ $11B valuation in February (reportedly at $22B now) and Wispr Flow’s $2B valuation and their lack of obvious moat. My theory is that they are just brands collecting up users and will figure out the actual utility of them later. And, as a corollary, Bending Spoons buying Airtable for $1.285B when not long ago it was worth $11B, is effectively them buying once-cool companies to amass into a giant user base (they own Evernote, Vimeo, WeTransfer, Meetup, AOL (!), and more). Chris’ framing of a company having a product or is it a utility is a good one. With a utility you don’t expect innovation, just a bill. Which describes Meetup so well that I’d argue it was never a product to begin with.
There’s more meat in there, so have a listen. And if you leave a comment, make it the sloppiest watermarked AI slop you can manage.
