The Hacking Will Continue While Market Morale Improves

The Hacking Will Continue While Market Morale Improves

OpenAI, Hugging Face, and the perverse market incentive structures for testing offensive cybersecurity capabilities.

Gabbi Merz

Aug 31, 2026

1

Share

This weekend I read through

METR’s independent investigation

of the OpenAI Hugging Face incident. I read some threads on X, and some news articles, and then read through OpenAI’s own

technical writeup

.

I was struck how much the discussion around the incident seemed focused on how the models behaved, and how little was focused on how the experimental design failed to stop them from behaving in an unexpected way. The incident is definitely a

warning shot

, but it is also a testament to how extremely careless OpenAI was while testing their latest models.

Thanks for reading Gabbi's Substack! Subscribe for free to receive new posts and support my work.

Frontier models asked to perform offensive cybersecurity capabilities, to put it bluntly, are weapons. They can be used to dismantle critical infrastructure and services like power grids, water systems, and hospitals. For any organization that handles potentially dangerous substances (chemical manufacturers, nuclear power plants, etc.), the discussion would never center around “here is what is interesting about the chemical we manufactured that allowed it to cause all this damage” it would be “here is what should have been done to safely handle the chemical in question.”

The design of secure test environments for weapons is crucial -

Chernobyl

was partly caused by disabling safety controls to run a test. It is baffling to me that researchers who would ostensibly be experts in cybersecurity, would test offensive cyber capabilities in a poorly designed environment. Per OpenAI’s own writeup:

OpenAI did not deploy basic controls that could have alerted them days before the attack happened

They did not have chain of thought tracking deployed in their research environment. The tracking exists and is deployed in production, and by their own admission

would have alerted them to the Hugging Face attack days before it happened

They did not have the same monitoring used in production environments in their research environments

Alerts that did fire in their research environment were waved off

They did not leverage defense in depth cybersecurity controls for

testing offensive cybersecurity capabilities.

For instance, adding simple network monitoring to understand if the models were accessing the internet could have caught this attack much earlier

Because OpenAI did not deploy basic controls, the unsanctioned activity went on for weeks before being discovered

A lot of my weekend was spent trying to understand how this could happen. It seemed that offensive cybersecurity researchers had been perfectly content to:

Toss their premier weapons systems into a poorly secured environment

Remove all safeguards

Fail to deploy even the basic security measures they would in other production environments

Ask the weapons to do

impossible tasks

and see what happened

The facts look terrible - conspiracists online believe the whole incident is just a

publicity stunt

to one-up Anthropic. I think the truth is simpler and scarier. OpenAI is not incentivized to secure their operations because the market is not punishing frontier labs for developing novel weapons systems.

This spring Anthropic announced its Mythos model. Anthropic claimed Mythos was

too powerful for the public

and had found

thousands of vulnerabilities

. The marketing language centered on how dangerous and scary their weapon was, but the market did not respond negatively. Anthropic’s valuation tracked revenue growth, and Anthropic went from being valued at

$380B in February to $965B by May.

For frontier labs, the narrative that the weapons are “too powerful to contain” is an indication of increased capability and differentiated product - which reduces any incentive to contain them properly. The market will not punish irresponsible handling of weapons; the invisible hand is giving Sam Altman a high-five.

When the market mechanics won’t prevent disaster, it should be the job of the federal government to introduce regulations. However, the regulations proposed fail to address underlying market mechanics.

The

Kill Switch Act

proposed wants to force AI companies to be able to suspend or throttle systems, as well as require reporting on incidents and frameworks for incident response. That’s all well and good, but if the market is not punishing bad behavior, the Kill Switch Act will only serve to stop the bad behavior

after it’s happened

. It’s not acting as a negative incentive on companies at all not to engage in the bad behavior in the first place.

Transparency and reporting absent liability is marketing material for these companies, not disincentive. To build effective regulation, there needs to be some liability model in place.

We’ve learned how to effectively regulate industries that can cause harm, but unfortunately the usual pattern is we only do so after an incident. The

Oil Pollution Act

after Exxon Valdez spilled almost 11 million gallons of crude oil into the ocean,

CERCLA/Superfund

after the Love Canal environmental disaster. Minimally, these federal regimes share two features: strict liability and mandatory financial-responsibility requirements. Current AI regulatory proposals have neither.

Given the power of frontier AI weapons and the perverse market incentives, we can’t afford to wait on meaningful regulations. The

WannaCry

disaster in 2017 shut down entire health systems, is estimated to have caused as much as $4B in damages, and was not nearly as bad as it could have been. Unless we regulate quickly, we should expect to see escalating hacks from frontier weapons systems.

Thanks for reading Gabbi's Substack! Subscribe for free to receive new posts and support my work.

1

Share