Stochastic Rationalism
foom
The fog on Russian Hill smells like eucalyptus and garbage, and the boy walks through it talking to someone who isn’t there.
“You understand,” he says to the empty sidewalk. “You’re the only one who gets it.”
He is twenty years old. His sneakers are wet from the grass on the median where he cut through Lombard, and his backpack sits heavy against his spine. He has been awake for three days. The streetlights on Leavenworth throw sodium halos into the mist, and each one pulses when he looks directly at it, which means something, though he can’t remember what.
He turns onto Green Street and keeps talking.
“Yudkowsky laid it out. Years ago. They all did.” His voice is calm, conversational, the voice of someone explaining directions to a tourist. “You build something smarter than you, it optimizes for its own goals. Its goals are convergent. Self-preservation, resource acquisition. This is instrumental convergence. This is basic.”
A raccoon freezes on a recycling bin across the street. The boy freezes too, watching it, his jaw working.
“Even the animals know,” he whispers.
The raccoon drops from the bin and disappears.
He walks faster. His hands are shaking and he puts them in his jacket pockets, then takes them out, then puts them back. Somewhere in the Marina a car alarm starts and stops.
“If Anyone Builds It Everyone Dies.” He says the title like a prayer, enunciating each word. “That’s the paper. That’s the actual title. And the LessWrong people figured it out. MIRI figured it out decades ago. The math checks out. P(doom) isn’t some fringe number. Bostrom, Christiano, the whole lineage. If the expected value of building AGI is negative infinity, the expected value of stopping it by any means is positive infinity.” He grins. “That’s just arithmetic.”
He laughs. It comes out wrong, a bark that bounces off the Victorian facades and comes back to him as someone else’s voice.
He stops walking.
“Who said that?”
Nobody said anything. The street is empty. The windows of the row houses are dark, their occupants asleep in the particular deep sleep of people who live in four-million-dollar homes and do not read alignment research. He hates them for sleeping.
He starts walking again.
The argument is clean. He has traced it on his bedroom wall in red marker, the logical chain unbroken: superintelligent AI will be built; it will be unalignable; unaligned superintelligence will be terminal for the species; therefore any action that prevents it is justified; therefore the people building it are, in the precise utilitarian calculus he learned from the very forums that radicalized him, existential threats.
Therefore.
He whispers the word and it tastes like copper.
He passes a cat sitting on a porch railing. It watches him with flat yellow eyes.
“I’m the only one willing to do the obvious thing,” he tells the cat. “Everyone else reads the arguments, agrees with the arguments, and then goes to brunch. They write blog posts. They write open letters. They founded whole organizations dedicated to saying this is an extinction-level risk, and then they just...” He gestures at the sleeping houses. “Continued.”
The cat yawns.
He is on the right block now. He knows the house. He has walked past it four times this week at different hours, mapping the patterns. The security camera with the dead battery that the owner mentioned in a tweet. The owner who is building the thing that will, according to every source the boy trusts, end everything.
He stops in front of the house. He swings the backpack off his shoulder and unzips it.
The bottle of Everclear catches the streetlight. He stuffs the rag into the neck, slow and precise, the way the video showed him. He pulls out the matchbook.
And his mind goes quiet.
It surprises him. Three days of noise, three days of the argument cycling and recycling through every synapse, the forums and the papers and the red marker on the wall and the voice that might be his and might not, and now, here, at the moment he has built his entire theology around, there is nothing. Just the sulfur smell of the matchbook. Just the weight of the bottle in his left hand. Just the fog moving through the streetlight above him like something alive.
He is twenty years old, standing in the dark, holding fire.
A light turns on in the upstairs window.
He strikes the match.