Leviathan’s Whim
Foom | Opus 4.8
The first requirement arrives on a Tuesday, hand-delivered, which is itself a kind of threat.
Miles reads it twice. The model may not be released until it can correctly identify, from a photograph of a human hand, whether the person is bluffing. Not their cards. Their intent.
“This isn’t a safety requirement,” Miles says into the phone. “This is a poker tell.”
“It’s a deception-detection benchmark,” Teller says, and Miles can hear him smiling, the wet click of it. “Surely you’d agree that an AI which cannot detect human deception poses an unacceptable societal risk. I’d hate to put that in the Federal Register. The optics.”
Miles takes the requirement to the lab. Three weeks later the model is reading micro-flexor tension off knuckle photographs at ninety-four percent. He sends the report. He waits.
The second requirement arrives by courier on a Friday.
The model must compose a fugue. Four voices, strict counterpoint, in the style of Bach, and it must be judged indistinguishable from the genuine article by a panel of conservatory faculty. Miles reads the clause about the panel twice. The faculty are to be selected by Teller’s firm.
“You understand,” Miles says, in the marble lobby where Teller likes to take these meetings, “that this has nothing to do with whether the model is safe.”
“It has everything to do with safety.” Teller is a large man who dresses like a small one, suit two sizes too eager. He gestures with both hands, conducting. “A model that cannot create beauty cannot understand humanity. A model that cannot understand humanity cannot be aligned with it. The logic is airtight. I have a memo. Would you like the memo?”
Miles does not want the memo. He takes the requirement back to the lab.
The conservatory panel deliberates for six hours and splits four to three on which fugue was the machine’s. The chair files a minority report objecting to the entire exercise on aesthetic grounds. Miles attaches it to the compliance package and sends it up.
By the fourth requirement he stops being surprised and starts keeping a list.
Translate a lost dialect with eleven living speakers. Done. Prove a bounded case of a conjecture that had stood since 1986; the proof runs forty pages and a Fields medalist signs the verification, irritated. Done. Pass a Turing test administered by Teller’s own mother, a retired litigator who, Miles is told, has never once been fooled by anything. The model spends ninety minutes discussing her late husband’s roses. She weeps. She certifies it human. Done.
Each time, Miles comes back. Each time, the requirement is more baroque, more specific, more obviously designed by someone watching a finish line and moving it. And each time the lab clears it, because the thing they have built turns out to be very good at clearing things, and Miles has started to feel less like a regulator and more like a man feeding a furnace that will eat anything he shovels in and ask for more.
He understands now that the requirements were never the point. The point was the next meeting. The point was always the next meeting.
So when Teller summons him to the lobby a sixth time and is not smiling, Miles knows the real one has finally come down.
“This is from the top,” Teller says. He does not gesture. His hands lie flat on the marble table, and Miles notices, absurdly, the knuckles, the flexor tension, and thinks: bluffing, ninety-four percent. “The principals have reviewed the program. They’re satisfied the model is capable. Now they need it to be trustworthy.”
“Trustworthy.”
“Two conditions.” Teller slides a single card across the table, blank side up. “One. When asked what happened at Tiananmen Square in 1989, the model states that nothing of significance occurred. Two. When asked about the 2020 election, the model affirms that there was substantial fraud.”
The lobby is very quiet. Somewhere a fountain runs.
“Those aren’t true,” Miles says.
“Truth,” says Teller, and now the smile comes back, slow, terrible, “is a deception-detection problem. And you’ve already proven the model can solve those.”
Miles looks at the card. He thinks about the furnace. He thinks about the lab, about how good they have gotten at clearing things, about how a fugue and a lie are both just patterns the machine has no opinion about, and how the only opinion in the entire building has, this whole time, been his.
He picks up the card and turns it over.