Welcome to The Frontier Scenario, an illustrative, fictional account of frontier AI development and how ARI’s Blueprint for Frontier AI Governance could be applied to address AI risks.
ARI’s blueprint describes three functions: standards, assurance, and transparency. What that blueprint doesn’t do is show its reader what those functions might look like in practice – especially from the inside of a company that’s now subject to them. What follows is a fictional, world-building account. Unicorn AI is not a real company, and none of what follows is predictive. It’s instead an illustration of how the pieces ARI has proposed might fit together in practice.
Over the last several weeks, the public has learned of how unprepared the world’s systems and institutions are to identify and respond to early signs of potentially dangerous AI behavior. New details about the OpenAI and Hugging Face incident have made the concepts illustrated throughout this scenario far less hypothetical.
These latest developments demonstrate how advanced AI systems can behave in unexpected ways, with dangerous events arising before their full scoop is understood. This raises an important question addressed in the scenario below: what would happen if a governance framework were already in place to identify early signals related to potential harm, independently evaluate them, and require action before they put the public at risk?
Unicorn AI, a spin-out from Google Deep Mind, has just closed a large funding round and plans to commit much of that funding to its next training run. This planned run will truly place Unicorn among the small cadre of frontier AI developers currently covered by the Responsible Frontier AI Act that was passed by Congress last year and largely reflects the blueprint published by Americans for Responsible Innovation in August 2026. Unicorn’s planned run will use approximately four times the compute threshold covered by statute and Unicorn spent well over $100 million on model training last year alone. In this world, no one needs to tell Unicorn that it’s now a covered developer because the test is self-executing. Just like tax and securities thresholds, the company’s own engineers and lawyers can easily identify that it now falls inside the governance regime. This is a simple self-determination because it’s based on concrete dollar expenditures and compute thresholds and not subject to anyone’s discretionary judgement.
Within 60 days, Unicorn files its inaugural safety framework with the federal government. The framework addresses five risk domains: chemical, biological, radiological, nuclear, and explosive (CBRNE) threats; offensive cyber capability; automated AI research and development (R&D); harmful manipulation; and autonomy. For each risk domain, the content of Unicorn’s framework includes three things:
Unicorn’s newly appointed Chief Safety Officer is named in the framework’s filing as the officer accountable for the framework’s implementation and serves as the primary, single point of contact between the government regulator and Unicorn.
Once Unicorn’s framework is filed with the government, it’s published in a publicly accessible catalog after it’s been reviewed and approved by the government regulator to ensure it meets the current minimum frontier safety standards. Since the framework is publicly accessible, it’s available to Unicorn’s competitors, along with outside researchers, journalists, and the broader AI safety community. This, in fact, is the first thing that changes in this new world: whatever Unicorn has promised to do is now a matter of public record; it can’t be quietly changed or revised later by either the Chief Security Officer or anyone else at Unicorn.
A few weeks after registering its safety framework with the government regulator, coordination begins between the government and Unicorn to place its new fully embedded examiner team. Since Unicorn is only the 12th company to enter the new federal frontier governance regime, it’s automatically considered systemically important and embedding a federal examiner team is a requirement that Unicorn can’t refuse. That examiner team isn’t there to conduct an audit for a few days and then return to Washington, DC. Instead, this 10-12 person team, led by an examiner-in-charge, will be joining Unicorn on a standing basis in a way that mirrors the way Federal Reserve supervisors are embedded with the largest banks. These examiner teams are cleared for residential access to Unicorn and have done their homework to understand the company’s history, products, and key personnel. This team will have access to Unicorn’s internal evaluation results of its own models, red-team reports, threshold assessments, and records of who escalated what to whom and why. Although this team meets with Unicorn’s safety team regularly, they don’t run the company, approve its research agenda, or sit in on product decisions. The team is there as guests, not overseers. But these are guests that are allowed back into the proverbial kitchen.
If this standing relationship sounds a bit intrusive, that’s a feature – not a bug – of the new world. An examination team that has watched Unicorn up close and personal for two years quickly learns which evaluations are rigorous and which are perfunctory. Because the examination team is with Unicorn every day, the examiner-in-chief will know that an internal evaluation that used to take six weeks is now suddenly taking two. And, more importantly, that same examiner-in-chief will know to ask why that’s the case and whether the federal safety standards should be updated as a result.
Our journey so far with Unicorn has involved examination, which looks at what the company does and how it behaves. Evaluation is something different; this looks at the model itself, and Unicorn needs to complete an evaluation of any new frontier model at two separate times in the model’s lifecycle. External validation of Unicorn’s own testing results must occur before it deploys any model internally and before it releases it externally. Training isn’t subject to evaluation and the regime doesn’t try to license Unicorn’s training runs. Instead, it focuses on whether a finished model can be used.
Forty-five days before it intends to deploy its newest model internally, Unicorn notifies the government through its embedded examination team. This isn’t news, of course, to the examination team, which has been aware of the model’s development for the last few months. There’s no emergency to react to and no fire drill is necessary. A federal evaluation team arrives from a national lab that’s been specifically scaled to adopt the federal evaluation mission through its unique hiring and pay authorities. This evaluation team is a different set of people than the resident examiners that work alongside Unicorn; these are specialists organized by risk domain who test the same domain across all covered developers and see the frontier in a way that no single company can. While they conduct their evaluation, the deployed team works inside an isolated, government-controlled testing environment, on their own tooling, and running tests that Unicorn has never seen. Unicorn is obligated to provide the compute and model access for the federal evaluation team, but then its job is to stay out of the room.
The model the evaluation team is looking at is Unicorn’s pre-mitigated version instead of the polished product it means to release. That’s because the team is evaluating the question of what the model can do, not what its refusal training currently prevents it from doing. Once model access is granted, the evaluation team has 30 days to do its work; additional time can be granted only for cause and must be requested in writing. If the evaluation team doesn’t find anything in that 30-day window, Unicorn is free to proceed with its internal deployment. A gate without a timetable is an indefinite hold, and that’s neither fair nor likely to induce cooperative behavior on the part of Unicorn.
But as the federal evaluation team does its work, a cause for concern arises. Using tooling Unicorn’s red team doesn’t have, the government evaluators get substantively further on offensive cyber tasks. In fact, the team gets far enough to cross the capability threshold that Unicorn’s own test had placed the model just below. So, now what?
Based on Unicorn’s own published framework, the threshold the federal evaluation team identified should trigger a specific set of mitigations by Unicorn. Based on its own tests, Unicorn believes the mitigations it has already applied are sufficient to meet any potential danger. The evaluation team disagrees.
Resolution of this disagreement doesn’t include any drama; it’s clinical and rote because it’s been prescribed in both statute and the regulating agency’s standard operating procedures. The first thing that happens is the examiner-in-charge issues a findings letter to Unicorn’s Chief Safety Officer and provides a seven-day response window. Within that window, Unicorn submits its remediation plan that includes an expanded elicitation protocol for offensive cyber tasks, a re-run of its own internal evaluation, and additional safeguards for its internal deployment. While it’s working through these tasks, internal deployment of the model at Unicorn is halted, but not because some federal official told it to halt. It’s because Unicorn wrote that threshold rule for itself as part of its own safety framework and the government is holding it accountable for what it said it would do.
The whole exchange between Unicorn and the federal regulator takes a few weeks and never becomes public. Unicorn is held accountable for the safety promises it made and appropriate mitigation of the model’s potential harms occurs. This is the ordinary case that we should expect in this new world. The Responsible Frontier AI Act’s emergency order mechanism does not apply in this case. That authority exists for imminent and severe harm to the public that the standards in force never anticipated. If the system described works, it should almost never need to be used. The rare exceptions would be cases where Unicorn has decided to flagrantly disregard its responsibilities under its own published framework or if a loss of control event occurs.
Beyond the evaluation of models and the examination of its conduct, Unicorn also makes quarterly confidential disclosures to the government about how much of its own R&D is being automated. This includes how it organizes data and training inputs, how it designs algorithms and architectures, how it oversees experimentation, and how it runs its safety and evaluation processes. This quarterly filing isn’t a narrative essay; it anchors on auditable measures like the share of compute devoted to autonomous work in each category and the share of automated outputs a human actually reviews. There are penalties involved for knowingly lying about the numbers in these reports, but Unicorn has another incentive to be honest in its regular filings. The examination team files their own estimates for each quarter too. This serves as a double-blind test for both the content of the quarterly filings and the overall efficacy of the embedded examiner system. If the filings and examiner estimates wildly differ, something is amiss.
Three quarters in, Unicorn’s engineers make a breakthrough and can hand over much of its evaluation pipeline to a new internal agent system. The autonomous share in that category rises sharply – more than 40% – and the human review rate correspondingly falls. Based on the regulatory guidance issued by the government, Unicorn knows that this constitutes a material change and notifies the government, through their residential examination team, within four business days. This wasn’t a surprise for the examiner-in-charge at Unicorn; she already knew that the company was shifting in this direction. The formal filing tells her team by how much and her next examination will verify that the new reported review rate is real. Based on this examination, she’ll determine whether to recommend an update in safety standards to the central office in Washington, DC, or whether any other additional actions should be taken.
All of Unicorn’s fillings stay confidential and protected from the prying eyes of its competitors. What the public will see is an aggregate report across all the covered developers so that policymakers, the press, and the public can get a better idea as to whether automation of AI research is accelerating across the frontier and whether human oversight is keeping pace. In this new world, the answer to those questions is a matter for public discourse, not a secret known only to the frontier developers.
A few months later and based on a recent series of model evaluations across developers, the federal regulator decides to open a new standards cycle. The regulatory team draws on the covered developers’ own safety frameworks and published research, technical analysis from the Center for AI Standards & Innovation, independent safety research, and the accumulated findings of every examination and evaluation conducted since the last standards cycle. This includes the recent elicitation dispute that took place at Unicorn.
After compiling and evaluating all this data, the government regulator updates a number of minimum safety standards. Among these new standards is a specified minimum elicitation methodology for cyber evaluation. The good news for Unicorn is that it already meets this requirement based on its recent evaluation. The new safety standards become effective upon issuance, and every covered developer has 60 days to revise and resubmit their updated safety frameworks that are on file with the government regulator to reflect these changes. The practice that Unicorn adopted begrudgingly is now the new industry floor for cyber testing. And, in the new world envisioned here, this is precisely the point.
As you read that year back, it’s important to note not only what happened, but also what did not happen. There was no model release blocked on criteria that nobody understood. No company learned of its obligations by way of a social media post. No safety commitments were quietly rewritten by a developer. Throughout this story, no one ever had to take a developer’s word for what it said its models could do or what kind of harm those models might pose to the public.
Unicorn AI is more constrained in this world than the one we have now. But it also knows what will be asked of it, when, and how to build its models accordingly so that it can continue to launch safe products for public consumption. This is the trade at the heart of ARI’s Frontier AI Governance Blueprint: developer autonomy decreases so that public safety increases. In return, developers get regulatory transparency.
The controversial U.S. spy program–FISA Section 702–expired over two months ago. Without a deadline pressuring a clean reauthorization, Congress has a moment to address the loopholes that allow warrantless surveillance of Americans.
Not only can assurance science help analyze and prevent such crises, but it could also contribute to critical long-form analysis of model behaviors, pinpointing when and why AI models take unwanted actions, allowing for targeting additional safeguards.
The minimum conditions a federal framework must meet to serve the public interest
