mission hedging
I have been to a scant four protests: one to petition my university not to cut funding to our life sciences department, one against English tuition fee increases, one to protest against Brexit, and one yesterday, in downtown SF.
this one was to protest against AI labs1 racing towards recursive self-improvement. one lab in particular. Anthropic made their initial public offering yesterday, floating at $1.28tn, becoming one of the most valuable companies in the world as the bell rang.
we were taken to a spot round the corner from their offices on Market Street, an unremarkable and unsigned glass-and-steel office block. signs were provided, although I brought my own: "THERE'S A POINT WHERE WE NEED TO STOP AND WE HAVE CLEARLY PASSED IT". there were speeches from the leading lights of AI safety, many of whom have spent years sounding the alarm, often at considerable personal cost2. we chanted. some people spoke to journalists. at 2pm we disbanded, just as the more hardcore protestors from another group staged a sit-in and were swiftly arrested.
this morning, my roommate woke me up to tell me the news: overnight, Anthropic researcher Levent Alpöge announced he had solved the Riemann hypothesis.
in a sparse PDF with his remarks on the discovery, Alpöge talks about making the discovery with "Claude":
Claude and I have had a great time together with such breadth in recent months, but this time Claude far outpaced me near the finish line. No, Claude flew, and danced, and sparkled near that finish line.
in reality, "Claude" is almost certainly a stand-in for an unnamed and unreleased Anthropic model, wielded with a custom harness to direct thousands of agents emitting hundreds of billions of tokens. "Claude" is now a friendly metonym for a series of models, some more powerful than anything available to the public. alien intelligences which are not so much built as grown into a machine totally beyond human understanding.
the finish line in this case was 17 million lines of Lean – as long as every human-written formal proof put together.
years ago, a mathematician friend of a friend didn't believe the predictions of AI's advances, and what they would mean for his field. he's now two years into a mathematics PhD which he will likely have to abandon, lest a frontier lab snap its fingers and scoop him with a few hours of GPU time. arguably the skill of wielding these models as a mathematician is now more important than actually being a mathematician. but soon the centaur approach will be over, and the models will be posing the research questions themselves, as well as solving them.
Claude solved a math problem which had been open for 170 years. it wrote it up as a 250,000 page proof, an artifact so laughably complex that it could only have been created by an alien intelligence. the age of the lone mind against the hardest problem might have ended earlier today, or at least got its final entry.
I also hold Anthropic shares, which I bought in a secondary earlier this year. I stand to benefit from the fruits of the company whose offices I was yesterday protesting outside.
the IPO increased their value three or four times. it won't be a life-changing amount of money, unlike for the hundreds of paper millionaires created yesterday at market open. holding them feels like both having a stake in a company that makes a product I love. it also feels like a tacit endorsement of them racing to build a superintelligence we cannot understand or control.
I tell myself this is reasonable, that it's mission hedging. buying securities whose value goes up in worlds we're most afraid of. but as another math problem falls to AI, another industry is entirely commoditized, I can't help but feel like we've taken one more step into those worlds. we have passed the point where we clearly need to stop.