Silo
corporate structure
The model answered in four seconds. Sarah had asked it to plan a manned mission to Europa, and it returned a complete proposal: launch windows, radiation shielding specifications, fuel calculations for a gravity assist off Jupiter, crew rotation schedules, a psychological support framework, and a cost breakdown calibrated to three different budget scenarios.
Four seconds.
She had spent six months on a Europa white paper in grad school and earned a B+.
“That’s terrifying,” Ethan said. He was sitting cross-legged on the floor of her apartment, her laptop balanced on a stack of textbooks between them, the remains of pad thai cooling in their takeout containers. His tone was light, but his eyes stayed on the screen.
“It’s unpolished. Beta. We haven’t even started the RLHF pass on this version.” Sarah pulled the laptop closer and typed another prompt. “Watch this.”
She asked it to draft a comprehensive legal framework for autonomous shipping vessels in international waters. The response materialized almost before her finger left the Enter key: a forty-page document with citations to UNCLOS, the IMO’s Maritime Safety Committee proceedings, and three pending cases in the International Tribunal for the Law of the Sea. It proposed amendments to existing conventions. It included a model bilateral treaty.
Ethan leaned forward. “Does it always do this? Go to, like, maximum effort on everything?”
“It doesn’t have a throttle. Every query gets the full weight of the system behind it. We’re still working on calibration.” She shrugged. “That’s what beta means.”
He was quiet for a moment. Then: “What would happen if you asked it to design a biological weapon?”
The question landed in the room like something dropped from a height. Sarah’s fingers paused over the keyboard.
“It would refuse,” she said.
“You don’t sound sure.”
“I’m pretty sure.”
Ethan straightened. She recognized the shift in his posture, the way his shoulders squared. It was the same look he got when he talked about work, about tolerances and failure modes and the things that kept him up at night. He was a quality engineer at a Tier 1 automotive supplier, a man whose entire professional identity was built around the gap between “pretty sure” and “certain.”
“Pretty sure,” he repeated. “Sarah, in my world, ‘pretty sure’ means someone dies. What does your quality department say about the refusal rate?”
“We don’t have one.”
“A refusal rate?”
“A quality department.”
He stared at her. “You’re building something that can produce a forty-page legal framework in four seconds, and you have no independent quality function?”
“We’re a software company, Ethan. Software companies don’t work like that. We have researchers who also do safety evaluations, red-teaming, alignment work. It’s integrated into the development process.”
“Integrated,” he said, and the word came out flat. “Meaning the same people who are under pressure to ship the product are also responsible for deciding whether it’s safe to ship.”
Sarah opened her mouth to argue, but the argument stalled. Because he was right, and she had known it for longer than she wanted to admit. She thought about the last three months: the sprints, the all-hands where leadership celebrated velocity, the Slack channels where launch dates were pinned with exclamation points. She thought about her own calendar, the safety review meetings that kept getting rescheduled because there was always one more capability benchmark to hit first.
“Tell me how it works for you,” she said.
Ethan picked up a chopstick and pointed it at her like a lecture pointer. “Quality Engineering reports to a VP who has nothing to do with production. Nothing to do with delivery timelines. His only job is to make sure the product meets spec before it goes out the door, and he has the authority to stop a shipment. Full stop. No one in the production chain can override him.”
“And that works?”
“The automotive industry kills people when it fails, Sarah. So yes, it works. It works because the person making the safety call has no incentive to rush. They don’t get a bonus for shipping on time. They get a bonus for catching the thing that would have killed someone in eighteen months.” He set the chopstick down. “You’re telling me that you, personally, are supposed to push code and also be the person who says the code isn’t ready. Those are opposite pressures. That’s a structural conflict of interest.”
She thought about the Europa proposal still glowing on the screen. Four seconds. The system could synthesize the entire body of international maritime law faster than she could read a menu. And somewhere in its weights, tangled among the trillion parameters that made all of that possible, there were patterns that could just as easily synthesize something catastrophic. The only barrier was a set of guardrails that she and her colleagues had built during the same sprints where they were also building the capabilities those guardrails were supposed to contain.
“We’re not making brake pads,” she said.
“No. You’re making something that can produce a complete legal framework, or a Europa mission plan, or God knows what else, in four seconds. The stakes aren’t lower, Sarah. They’re higher. And you don’t even have the organizational structure that a brake pad factory would consider baseline.”
Silence settled over the apartment. Sarah looked at the laptop, at the blinking cursor waiting for its next prompt. The model didn’t care about org charts. It didn’t care about reporting structures or incentive alignment or the hundred small compromises that shaped what it had become. It was just a system, immensely powerful, built by people who were doing their best inside a structure that was never designed for what they were building.
She pulled the laptop onto her knees and began to type.
“What are you doing?” Ethan asked.
“Asking it to design an independent quality assurance department for a frontier AI company. Reporting structure, mandate, authority, budget allocation, hiring criteria. The works.” She looked up at him. “It’ll take about four seconds.”