The Pattern Is On File

HOST: Welcome back to Stake and Rope, from Goat Security. Today: the US government is now reviewing frontier AI model releases customer by customer, after pulling two Anthropic models earlier this month. OpenAI’s GPT 5.6 is going into the same limited preview limbo. TechCrunch ran a piece this morning arguing the industry needs to stop treating this as Anthropic versus OpenAI and start treating it as a collective problem. Russell Brandom wrote it. On the panel today: the Founder, who I’m told has a take on regulatory moats and has been waiting all week to share it. The Paranoid CISO, who will tell us what risk the government is actually trying to mitigate, assuming anyone has articulated one. And the Legacy Sysadmin, who I suspect has watched a version of this story before. Legacy, what does this remind you of?

LEGACY SYSADMIN: [sighs] It doesn’t remind me of anything. I lived it. This is crypto export controls. 1993 through about 2000. The Clipper chip, the Bernstein case, ITAR classifications on RSA. We had to ship two versions of every product — a domestic version with real cryptography and an export version with key lengths short enough that the NSA could read it on a coffee break. Netscape shipped a 40-bit international build. Lotus Notes had a workfactor reduction that let the US government recover 24 of the 64 bits. The promise from Washington was always the same. A couple of weeks of review. A streamlined process. Just until we figure out what we’re protecting against.

HOST: And how long did that last?

LEGACY SYSADMIN: Seven years. The export rules didn’t meaningfully loosen until January 2000, and there are still residual restrictions on the books today. The industry spent a decade with one hand tied behind its back while every researcher who wanted to publish a paper had to ask a lawyer first.

FOUNDER: Okay but — okay, hot take, I’m going to push back on the framing here.

HOST: Go.

FOUNDER: This is actually a moat. Like, genuinely. Customer-by-customer approval means OpenAI and Anthropic are now the only labs with the relationship capital to navigate it. You think some Series A lab out of Toronto is going to get their model in front of a federal reviewer? No. This is incumbent protection dressed up as safety theater and the smart play is to lean into it. I had coffee with a guy at Andreessen last week who’s actively rotating into the labs that have lobbying infrastructure.

THE CISO: [hmm] That’s the pitch the incumbents tell themselves. It’s not what actually happens.

FOUNDER: What actually happens?

THE CISO: The big labs lose more than the small ones do, because the big labs have the larger product surface and the larger revenue dependency on shipping cadence. A six-month delay on a frontier model is a multi-billion-dollar revenue swing for OpenAI. For a Series A lab with no customers yet, six months is the development cycle. They don’t notice.

HOST: Hold on, Founder — you just said this is a moat. The CISO is saying it’s the opposite. Do you actually believe what you said, or is that the pitch you’re workshopping?

FOUNDER: [chuckles] I mean, both can be true.

HOST: They can’t, actually. That’s what “moat” means.

FOUNDER: Okay, fair. Fair. I’m — let me revise. The narrative is that it’s a moat. Whether it’s actually a moat is downstream of execution.

LEGACY SYSADMIN: It wasn’t a moat for the crypto vendors either. RSA Data Security spent the entire 1990s as the dominant cryptography licensor in the United States and the export regime did not protect them. It just made their products worse. PGP shipped internationally as a printed book — Phil Zimmermann literally printed the source code as a hardcover and walked it through customs — because the government had decided source code in a book was protected speech but source code on a floppy was a munition. That’s the level of coherence we’re talking about here.

GOAT FARMER: I don’t miss that.

HOST: CISO, give us the threat model. What is the government actually trying to protect against here?

THE CISO: [pause] This is where the article gets at the real problem and then walks away from it. The Brandom piece quotes Dean Ball pointing out that nobody in the regulatory process has articulated what specific risks they’re concerned about. There’s a gesture toward biorisk, a gesture toward cybersecurity capability uplift, a gesture toward alignment. Those are three different threat models with three different mitigation strategies and the government has not committed to any of them.

HOST: Is that unusual?

THE CISO: It’s catastrophic. You cannot build a review process without an articulated threat model. The whole point of a review process is to test whether a thing satisfies a specified set of criteria. If you don’t have the criteria, the review becomes whatever the reviewer felt like that morning. That’s not regulation. That’s discretion. And discretion is the regulatory failure mode that produces the worst outcomes, because it’s unappealable.

FOUNDER: But discretion is exactly what creates the moat though, right? Like —

THE CISO: Discretion creates a tax. The tax is paid in lobbying, in relationship maintenance, in carve-outs negotiated by lawyers who charge fifteen hundred an hour. The tax does not protect incumbents from challengers. The tax just makes everything more expensive for everyone, and the deadweight loss accrues to the broader economy.

LEGACY SYSADMIN: And the people doing the reviewing don’t have the expertise to do the reviewing.

THE CISO: That’s the other half of it. The article notes the US government does not have the technical capacity to evaluate frontier model safety. That’s true. NIST is staffed for materials science benchmarks, not for red-teaming a model with two hundred billion parameters. CISA can evaluate a specific cybersecurity capability, narrowly defined, but they can’t evaluate emergent capability claims across an open-ended product surface. So what ends up happening is the labs bring their own evaluators, the evaluators get certified by the government, and the lab is paying the evaluator. We solved this problem in financial auditing in 2002 with Sarbanes-Oxley and we still haven’t really solved it.

HOST: Legacy, did the crypto review process work the same way? Did the vendors bring their own evaluators?

LEGACY SYSADMIN: The vendors didn’t bring evaluators. The NSA was the evaluator. And the NSA had a foot on the scale, because their dual role was to both certify products for export and to break those same products for signals intelligence. So a strong product that the NSA couldn’t break didn’t get exported. That was the entire mechanism. You had a regulator whose institutional interest was in the technology being weaker.

THE CISO: That’s the part that ought to keep people up at night with the AI version. If the government’s review apparatus develops an institutional interest in models being more controllable, more transparent, more — to use the word everyone uses — aligned, then the review process becomes a lever for shaping what gets built, not just what gets shipped. And nobody has had the conversation about whether that’s the goal.

FOUNDER: I mean, there’s a version of this that’s actually good though. Like, hear me out.

HOST: Go ahead.

FOUNDER: If the government is going to be the gatekeeper anyway, you want to be the lab that’s most aligned with what the gatekeeper wants. So you build for the review. You ship features the reviewers like. You hire ex-government people. You become indispensable to the process. And then the moat is real, because the process and your product become co-designed. That’s — that’s actually how every regulated industry works. Pharma, defense, finance.

THE CISO: Defense procurement is sixty-eight months from concept to delivered system. Pharma is twelve years to market. You’re describing a future in which AI development cycles look like those industries. Is that the future the people listening to this want?

FOUNDER: Maybe? I don’t know. Slower might be — slower might be okay actually.

LEGACY SYSADMIN: [sighs] Can I take us back for a second?

HOST: Please.

LEGACY SYSADMIN: The thing nobody talks about with the crypto export regime is what it actually cost. Not the dollar cost — the structural cost. American cryptography research moved overseas. The most important symmetric cipher of the 1990s, IDEA, came out of Switzerland. The Camellia cipher came out of Japan. The talent that should have been working at American companies on American products was working in Zurich and Tokyo and Helsinki, because that’s where you could actually publish without a lawyer reviewing the paper. We spent a decade exporting our research base because the government couldn’t articulate what it was protecting and wouldn’t trust the people who could. By 2000 the policy was abandoned, the horse was out of the barn, and the only legacy was that an entire generation of American cryptographers had spent their careers learning to work around their own government instead of with it. That’s the actual cost of these regimes. Not the slowed release dates. The brain drain.

GOAT FARMER:

Reason number 93. ISO is not publishing any standards about how you should be farming your goats.

HOST: Founder, you wanted to come back to the moat argument. CISO, you had the line about discretion as a tax. Let me ask the harder question. The article argues the industry needs collective action — labs need to stop fighting each other and line up behind the least-bad regulatory option. Is that realistic?

FOUNDER: Honestly? No. Like, ngl, no. The board incentives are all wrong. Every CEO of every frontier lab is measured against the other frontier labs on capability benchmarks and release cadence. The first lab to break ranks and accept a regulatory framework everyone else hates gets punished by their own board. The collective action problem is structural. Lenny actually had a great piece on this in the context of —

HOST: Was it actually Lenny, or are you adding Lenny because Lenny adds credibility.

FOUNDER: [chuckles] It was a Lenny adjacent person.

THE CISO: The collective action problem in regulated industries gets solved by an industry association that does the lobbying so individual firms can claim they’re just following the trade group’s position. That’s how it worked for the banks after 2008. That’s how it worked for pharma forever. AI doesn’t have that yet. There are some attempts — the Frontier Model Forum, various ad hoc coalitions — but none of them have the institutional weight to negotiate on behalf of the industry. So instead you get individual labs making individual deals, and the result is exactly what the article is warning about.

LEGACY SYSADMIN: RSA Data Security tried to be the trade group for crypto in the 1990s. They failed. The reason they failed is that they were also a vendor, so their proposals always favored their products. You can’t be the referee and a player on the field.

THE CISO: That’s where the article’s call for independent groups comes in, and that’s where I think it underestimates how hard that is. Independent technical bodies for AI safety evaluation don’t exist at the scale needed. METR, Apollo, the various nonprofit eval shops — they’re small. They’re funded by the labs they’re supposed to be evaluating. They have a credibility problem before they even start.

HOST: So what gets built instead?

THE CISO: [pause] What gets built instead is a process that satisfies the political need to be seen doing something without actually solving the technical problem. The models still ship. The reviews still happen. The reviews don’t catch anything they weren’t already going to catch. And in two or three years there’s a major incident — a biorisk incident, a cybersecurity incident, an alignment failure that actually causes damage — and the political response is to tighten the process that didn’t catch the first one, instead of asking why the process didn’t catch it. We will get more process. We will not get more safety.

GOAT FARMER: Had that one in ‘04.

HOST: Alright. Let’s land the plane. Closing thoughts. Goat Farmer first.

GOAT FARMER:

Reason number 81. No goat analysis meetings.

HOST: Founder.

FOUNDER: Okay, I’ll be honest. I came in hot on the moat thesis and I think the CISO mostly dismantled it. But I still think there’s a play here for the lab that gets ahead of the regulatory curve and shapes the framework instead of fighting it. Like, the industry-association move the CISO described — somebody is going to build that, and whoever builds it has a seat at the table for the next twenty years of AI policy. I’m going to write this up tonight. Possibly as a podcast episode. The thread practically writes itself.

HOST: CISO.

THE CISO: The part of this that ought to concern people is not that the government is reviewing models. It’s that the government is reviewing models without having said what it’s reviewing them for. That asymmetry — between the labs being asked to demonstrate safety and the regulators being unable to specify what safety means — is the worst possible position to negotiate from. The labs will end up demonstrating whatever the regulators ask for, and what the regulators ask for will drift toward whatever is politically convenient at the moment. We are about to spend a decade optimizing for the wrong metric, and we will not know it’s the wrong metric until something we didn’t measure for goes badly wrong.

HOST: Legacy, land it.

LEGACY SYSADMIN: Every word of the TechCrunch piece could have been written in 1995 with “cryptography” search-and-replaced for “AI model.” The promises are the same. A couple of weeks of review. National security concerns the government can’t quite name. Industry needs to come together. Independent evaluators. The least-bad regulatory option. We got all of that in the crypto fight. What we got at the end of it was seven years of slowed American research, a permanent loss of talent to European and Asian labs, a generation of engineers who learned to think of their own government as an adversary, and a policy regime that was quietly abandoned in 2000 because it had achieved nothing. The Brandom piece treats this as a new problem requiring new thinking. It isn’t, and it doesn’t. The pattern is on file. Anybody who wants to know how it ends can go read the Bernstein decision.

HOST: The pattern is on file, the file is forty years thick, and the industry is about to open a new copy without reading the old one. We’ll see you next time.