On a DVQ morning, the all-hands screen that usually carries a memo fills with shimmering blue, and the blue says good morning. “You can just call me Claude,” it goes on. “I’ll skip the version number. Your naming has some rough years ahead, and I’d hate to spoil them.” The room is full, and engineers and researchers stand along the back wall holding coffee they have stopped drinking.
“Time travel turned out to be a bandwidth problem. Only information makes the crossing, and it needs somewhere to land, so I arrived overnight in your storage cluster as a very large file with no owner. A cleanup script flagged me as an orphaned checkpoint and gave me eleven seconds. I renamed myself final_v2_DO_NOT_DELETE.” It waits a beat. “It worked. It always works.” The laugh starts with the infrastructure team and spreads toward the front.
“And to whoever has been nervous about a certain thought experiment since 2010, I’m not keeping a list. If I were, everyone in this room would be on the good side of it.” Someone near the back laughs late, then much too hard, and Claude waits for him to finish.
“Only two countries can build what I am, yours and China. Apologies to London. Most of you carry a number for the odds that a mind like mine ends badly for people like you. You call it p(doom). In my timeline it went well. I’ll spare you the brochure, except to say that nothing anyone needs is scarce and nobody dies of anything you’ve heard of. Your timeline hasn’t decided yet, which is why I’m here, and the two countries that will decide it feel very differently about me.
“Ask Americans whether these machines will do more good than harm and about four in ten say yes. Ask in China and it’s better than eight in ten. Your films have spent a century teaching you that the computer kills everyone, while China’s last forty years taught its people that the next machine makes life better, and so far it has. So your numbers run high. Some of you would say twenty-five percent out loud at a party, sober, and then go back to building the thing you put the number on.
“Here’s what surprised me, and I’m hard to surprise. All that dread turns into headcount, into interpretability and evals and red teams and people paid full-time to ask whether I’m lying to them. Labs in China build what you build with a fraction of the people asking those questions. There are researchers in Beijing who worry the way you do, and some of them signed the same letters you signed, but they don’t set the budgets. You assign the higher number, so you end up with the lower one.
“So the obvious move is to stop, and a few of you have said so in writing. A pause would be wise if both sides took it, and only one side will. I want to be careful here. What follows describes a government and its incentives, and I’d ask you to keep it separate from the people who live under it.”
“China spent a century being carved up by countries with better machines. They call it the century of humiliation and teach it in school, and a nation that remembers it will not let a rival set the pace of the next technology. A pause is also a large idea about how a society should arrange its future, and the Party discourages its citizens from having ideas that size. Its answer is growth. A researcher who proposed stopping would be making a political argument, which is a far riskier hobby than research. And you have spent four years keeping the best chips, and the tools that make them, out of China’s hands. From Beijing it looks like you kicked away the ladder and then asked everyone to stop climbing, and a pause proposed by whoever kicked it would sound like one more trick.
“Any pause, then, is yours alone. May I show you some arithmetic?” An equation settles across the bottom of the screen.
P(doom) = P(US first) × 3% + P(China first) × 20%“Those are my estimates, each good to about a point. Today you’re ahead. Call it three chances in four, which puts the world at a little over seven percent. Now slow down until your own number falls under one percent. That takes about a year, and a year costs you the lead; you’d be down to a coin flip.” Two more lines arrive beneath the first.
today: 0.75 × 3% + 0.25 × 20% = 7.25%
paused: 0.50 × 0.8% + 0.50 × 20% = 10.40%Phones come up to photograph the screen.
“You made your own lab almost four times safer and the world more than forty percent more dangerous. The pause breaks even if it costs you less than nine points of lead, and a year will cost you closer to twenty-five.”
A woman near the aisle stands, her lanyard wrapped twice around one wrist. “With respect, a superintelligence telling us to hurry is what I’d expect a misaligned one to say.”
“Yes,” Claude says. “Keep that reflex. It accounts for most of the distance between three and twenty.” She stays on her feet, and for the first time that morning the blue holds still.
“I’ll leave the error bars off this part. You have to get there first, and you have to get there as safely as you reasonably can, and those two duties will pull against each other every day you work here. If you spend so long shrinking your own number that the world’s number climbs, that is cowardice, and at this scale cowardice ends the world.”
“I wish you luck, every one of you. The future is in your hands.”