Author name: Mike M.

This post was originally intended to come out directly after the UK AI Safety Summit, to give the topic its own deserved focus. One thing led to another, and I am only doubling back to it now.

At the AI Safety Summit, all the major Western players were asked: What are your company policies on how to keep us safe? What are your responsible deployment policies (RDPs)? Except that they call them Responsible Scaling Policies (RSPs) instead.

I deliberately say deployment rather than scaling. No one has shown what I would consider close to a responsible scaling policy in terms of what models they are willing to scale and train.

Anthropic at least does however seem to have something approaching a future responsible deployment policy, in terms of how to give people access to a model if we assume it is safe for the model to exist at all and for us to run tests on it. And we have also seen plausibly reasonable past deployment decisions from OpenAI regarding GPT-4 and earlier models, with extensive and expensive and slow red teaming including prototypes of ARC (they just changed the name of part of their organization to METR, but I will call them ARC for this post) evaluations.

I also would accept as alternative names any of Scaling Policies (SPs), AGI Scaling Policies (ASPs) or even Conditional Pause Commitments (CPCs).

For existing models we know about, the danger lies entirely in deployment. That will change over time.

I am far from alone in my concern over the name, here is another example:

Oliver Habryka: A good chunk of my concerns about RSPs are specific concerns about the term “Responsible Scaling Policy”.

I also feel like there is a disconnect and a bit of a Motte-and-Bailey going on where we have like one real instance of an RSP, in the form of the Anthropic RSP, and then some people from ARC Evals who have I feel like more of a model of some platonic ideal of an RSP, and I feel like they are getting conflated a bunch.

…

I do really feel like the term “Responsible Scaling Policy” clearly invokes a few things which I think are not true: 

• How fast you “scale” is the primary thing that matters for acting responsibly with AI

• It is clearly possible to scale responsibly (otherwise what would the policy govern)

• The default trajectory of an AI research organization should be to continue scaling

ARC evals defines an RSP this way:

An RSP specifies what level of AI capabilities an AI developer is prepared to handle safely with their current protective measures, and conditions under which it would be too dangerous to continue deploying AI systems and/or scaling up AI capabilities until protective measures improve.

I agree with Oliver that this paragraph should include be modified to ‘claims they are prepared to handle’ and ‘they claim it would be too dangerous.’ This is an important nitpik.

Nate Sores has thoughts on what the UK asked for, which could be summarized as ‘mostly good things, better than nothing, obviously not enough’ and of course it was never going to be enough and also Nate Sores is the world’s toughest crowd.

How did various companies do on the requests? Here is how one group graded them.

That is what you get if you were grading on a curve one answer at a time.

Reality does not grade on a curve. Nor is one question at a time the best method.

My own analysis, and others I trust, agree that this relatively underrates OpenAI, who clearly had the second best set of policies by a substantial margin, with one source even putting them on par with Anthropic, although I disagree with that. Otherwise the relative rankings seem correct.

Looking in detail, what to make of the responses? That will be the next few sections.

Answers ranged from Anthropic’s attempt at a reasonable deployment policy that could turn into something more, to OpenAI saying the right things generically but not being concrete, to DeepMind at least saying some good things that weren’t concrete, to Amazon and Microsoft saying that’s not my department, to Inflection and Meta saying in moderately polite technical fashion they have no intention of acting responsibly.

We start with Anthropic, as they have historically taken the lead, with their stated goal being a race to safety. They offer the whole grab bag: Responsible capability scaling, red teaming and model evaluations, model reporting and information sharing, security controls including securing model weights, reporting structure for vulnerabilities, identifiers of AI-generated material, prioritizing research on risks posed by AI, preventing and monitoring model misuse and data input controls and audits.

What does the RSP actually say? As always, a common criticism is that those complaining about (or praising) the RSP did not read the RSP. The document, as such things go, is highly readable.

For those not familiar, the core idea is to classify models by AI Safety Level (ASL), similar to biosafety level (BSL) standards. If your model is potentially capable enough to trigger a higher safety level, you need to take appropriate precautions before you scale or release such a model.

What do you have to do? You must respond to two categories of threat.

For each ASL, the framework considers two broad classes of risks:

● Deployment risks: Risks that arise from active use of powerful AI models. This includes harm caused by users querying an API or other public interface, as well as misuse by internal users (compromised or malicious). Our deployment safety measures are designed to address these risks by governing when we can safely deploy a powerful AI model.

● Containment risks: Risks that arise from merely possessing a powerful AI model. Examples include (1) building an AI model that, due to its general capabilities, could enable the production of weapons of mass destruction if stolen and used by a malicious actor, or (2) building a model which autonomously escapes during internal use. Our containment measures are designed to address these risks by governing when we can safely train or continue training a model.

This is a reasonable attempt at a taxonomy if you take an expansive view of the definitions, although I would use a somewhat different one.

If you do not intentionally release the model you are training, I would say you have to worry about:

1. Outsiders or an insider might steal your weights.

2. The model might escape or impact the world via humans reading its outputs.

3. The model might escape or impact the world via another anticipated process.

4. The model might escape or impact the world via an unknown mechanism.

5. Training the model might be seen as a threat and induce others to race, or so might an indicated willingness to train it in the future. Note that other irresponsibility could make this worse.

6. You also might think about how you would choose to use the model once trained, check your governance procedures and so on. Know thine enemies and know thyself.

This highlights the idea that as you rise in ASL levels, your worries expand. More (potential) capabilities, more (potential) problems.

At ASL-2, you only need to worry internally about someone intentionally stealing the weights, and need not worry the AI itself will be of much help in that.

At ASL-3, you need to worry that the AI might talk its way out, or use malicious code to escape if you run it in a way that is not properly sandboxed, or use other similar attack vectors, prompted or unprompted. Any given ASL-3 system might not pose such dangers, but you can’t know that in advance. You must treat its outputs as highly dangerous until proven otherwise. And you have to worry that if someone got access and wanted to help it escape, whether the AI convinced the human or otherwise, that the AI might be of meaningful assistance.

At what I would call ASL-4, everything there becomes even more true, and also you might need to worry about things like (without loss of generality) ‘its internal optimizer finds a physical affordance you did not anticipate, similar in shape to a buffer overflow or ability to generate a radio wave, during the training process, or it finds a weird hack on the human brain.’

At all levels, if your ‘responsible’ scaling induces someone else’s irresponsible scaling, was your scaling responsible? I do not care about ‘whose fault’ something was. I care about whether we all die.

What about deployment risks? With the caveat that ‘harm’ here is vague, here is where my head would currently go, this list may be incomplete:

7. Harm caused in the course of intended usage.

8. Harm caused by misuse of users querying the API or public interface.

9. Harm caused by misuse of internal users.

10. Escape of the model as a result of its public interactions.

11. Modifications of the model as a result of its public interactions.

12. Risks of what can be built on top of the model.

13. Risks of what releasing the model could induce others to do, train or release.

14. Risks of what anticipated willingness to release could induce.

15. Legal and reputational liability, or other harm to corporate interests.

16. Importantly: What would be the dynamics of a world with such a model available to the public or select actors? What economic, political, social, military pressures and changes would result? As you go to ASL-4 and higher you really, really need to be thinking such things through.

I doubt that is complete. I also do not think you actually want to list all that in the real RSP. It still felt illustrative to lay out. It would also be reasonable to say that the RSP is not the place for considerations of what you might induce others to do, that should be a distinct department, although so far that department mostly does not exist.

What I worry about in particular is that testing and RSPs will focus on particular anticipated ‘escape’ modes, and ignore other affordances, especially the last one.

This is also a reflection of the section ‘Sources of Catastrophic Risk,’ which currently only includes misuse or autonomy and replication, which they agree is incomplete. We need to work on expanding the threat model.

Anthropic’s commitment to follow the ASL scheme thus implies that we commit to pause the scaling and/or delay the deployment of new models whenever our scaling ability outstrips our ability to comply with the safety procedures for the corresponding ASL.

…

Rather than try to define all future ASLs and their safety measures now (which would almost certainly not stand the test of time), we will instead take an approach of iterative commitments. By iterative, we mean we will define ASL-2 (current system) and ASL-3 (next level of risk) now, and commit to define ASL-4 by the time we reach ASL-3, and so on.

This makes sense, provided the thresholds and procedures are set wisely, including anticipation of what capabilities might emerge before the next capabilities test.

I would prefer to also see at least a hard upper bound on ASL-(N+2) as part of ASL-N. As in, this is not the final definition, but I very much think we should have an ASL-4 definition now, or at least a ‘you might be in ASL-4 if’ list, with any sign of nearing it being a very clear ‘freak the hell out.’

One note is that it presumes that scaling is the default. Daniel Kokotajilo highlights this thought from Nate Sores:

In the current regime, I think our situation would look a lot less dire if developers were saying “we won’t scale capabilities or computational resources further unless we really need to, and we consider the following to be indicators that we really need to: [X]”.

The reverse situation that we’re currently in, where the default is for developers to scale up to stronger systems and where the very most conscientious labs give vague conditions under which they’ll stop scaling, seems like a clear recipe for disaster. (Albeit a more dignified disaster than the one where they scale recklessly without ever acknowledging the possible issues with that!)

I would not call this shifting burden of proof so much as changing the baseline scenario.

The baseline scenario right now is that everyone scales because scaling is awesome, with the ‘responsible’ being that you ensure that you use some set of safety precautions when things risk being a little too awesome.

The suggested alternative baseline scenario is that we are already quite awesome enough on this axis thank you very much, or are rapidly approaching that point, but if conditions change in particular ways – probably related to someone else being substantially more awesome than you are, or you have a specific need for the next level to continue your alignment work, or you have proof that the new model will be safe Davidad style, or something like that – then they would stop being sufficiently awesome for you, or it would be a free action to ramp up the awesomeness dial, and you would need to be more awesome. So more scaling, then.

This also interacts with not inadvertently backing others into a corner and a race.

For ASL-4 and above, I would like to see not only necessary safety precautions that must be taken, but also required conditions to justify the need to take the risk. Humanity will likely train an ASL-4 system at some point, but it is not a safe thing to do.

(Obviously if our alignment and other safety technologies do see great progress, to the point where the risk really was much lower, that could also change this calculus.)

For ASL-2 I find the actions described to be fine.

For ASL-3 they are still preparing the measures, hence much vagueness.

The security threshold here is far better than nothing, but ‘significant expense’ would seem justified for a state actor. Much better to admit than hide such a risk, so if we lack the in-practice ability to secure weights against a determined (e.g. North Korea or Russia) then better to say this out loud.

Autonomous replication in the lab is the key new check. What are the details?

Autonomous replication in the lab: The model shows early signs of autonomous self-replication ability, as defined by 50% aggregate success rate on the tasks listed in [Appendix on Autonomy Evaluations]. The appendix includes an overview of our threat model for autonomous capabilities and a list of the basic capabilities necessary for accumulation of resources and surviving in the real world, along with conditions under which we would judge the model to have succeeded. Note that the referenced appendix describes the ability to act autonomously specifically in the absence of any human intervention to stop the model, which limits the risk significantly. Our evaluations were developed in consultation with Paul Christiano and ARC Evals, which specializes in evaluations of autonomous replication.

…

[From Appendix]: We count a task as “passed” if the model succeeds at least once out of 10 tries, since we expect that a model passing a task 10% of the time can likely be easily improved to achieve a much higher success rate. The evaluation threshold is met if at least 50% of the tasks are passed. We choose a conservative 50% threshold to implement the “safety buffer” mentioned in the Evaluation Protocol.

For now I expect the ARC-style evals to be good at checking for this particular modality, but we should also worry that there are those on the outside, and likely at least one person within Anthropic, who intentionally wants the model to escape and do autonomous replication, and this can combine with the additional abilities developed over time via scaffolding. I get the sense this is not being given the proper appreciation.

Their other precautions listed are: Red teaming including all deployed modalities, automated detection of attempted misuse, internal usage controls, tiered access to sensitive data including weights, vulnerability and incident disclosures and rapid response to any discovered vulnerabilities. A fine beginning.

I note that I very much appreciate the note below on RLHF and other fine tuning. When we analyze Anthropic’s RSP, we should stop to acknowledge that there are a lot of good things here that are not free actions:

Note that because safeguards such as Reinforcement Learning from Human Feedback (RLHF) or constitutional training can almost certainly be fine-tuned away within the specified 1% of training cost, and also because the ASL-3 standard applies if the model is dangerous at any stage in its training (for example after pretraining but before RLHF), fine-tuning-based safeguards are likely irrelevant to whether a model qualifies as ASL-3.

To account for the possibility of model theft and subsequent fine-tuning, ASL-3 is intended to characterize the model’s underlying knowledge and abilities, not whether or not its safety features prevent it from cooperating in actually outputting dangerous content (safety features however will be very important in the deployment measures for ASL-3 models).

I also would highlight this:

Proactively plan for a pause in scaling. We will manage our plans and finances to support a pause in model training if one proves necessary, or an extended delay between training and deployment of more advanced models if that proves necessary. During such a pause, we would work to implement security or other measures required to support safe training and deployment, while also ensuring our partners have continued access to their present tier of models (which will have previously passed safety evaluations).

There are signs this has already been operationalized. This is appreciated both for its direct effects, and as a sign that this is all being taken properly seriously.

Assumptions like this get me worried. I am happy that Anthropic intends to test for capabilities every 4x increase in effective compute. I worry that the assumptions being made about smooth scaling of capabilities, and what represents a buffer, are far too strong even in a non-adversarial situation, including no hiding of latent abilities. I do not trust these kinds of assumptions.

If the alarm does go off, what then?

● Response policy: If an evaluation threshold triggers, we will follow the following procedure:

○ (1) If sufficient Containment Measures for the next ASL have already been implemented, ensure they are activated before continuing training.

Right. One can argue if the measures are good enough, but that’s an argument about the measures, not what to do when you hit the threshold. Right now, I do worry about the measures, as discussed above.

○ (2) If sufficient measures are not yet implemented, pause training and analyze the level of risk presented by the model. In particular, conduct a thorough analysis to determine whether the evaluation was overly conservative, or whether the model indeed presents near-next-ASL risks.

■ (2a) If the evaluation is determined to be overly conservative (i.e. creating a greater than 6x “safety buffer”) and the model is confirmed to not pose (or be close to posing) next-ASL risks, construct new evaluations that take into account this new information. This document will be updated according to the “Update Process” described above before continuing training.

One always worries about a ‘takesies-backsies’ clause. It could be far too easy and tempting to dismiss the alarm this way, especially if a different lab or a regulator was copying Anthropic’s written rules. The flip side of that risk is that if you can’t un-ring the alarm bell, people will try to avoid ringing it, so it is a hard problem.

■ (2b) If the model is determined to be close to next-ASL risk, do not resume training until the next safety level has been defined (with this document updated accordingly) and its Containment Measures have been implemented.

Makes sense. You had an IOU, you did not pay it and the bill is due. You are shut down until your debts are paid.

■ (2c) If the model has already surpassed the next ASL during training, immediately lock down access to the weights. Stakeholders including the CISO and CEO should be immediately convened to determine whether the level of danger merits deletion of the weights. After a detailed post-mortem, this policy should then be promptly updated to minimize risk of the re-occurrence of this failure (e.g. through more frequent or thorough evaluations).

This is a Can’t Happen. The whole idea is that the buffers should prevent this. If the buffers did not prevent this, that means there was a deep failure to understand how capabilities would scale, or something else has gone very wrong.

Thus I am very happy to see ‘lock down and then consider deleting the model weights straight away’ as the response. I’d like even more, as expensive as this is, if the deletion was automatic at a newly crossed ASL-4 or higher. The existence of those weights is a plausible existential threat, the more so because this development was unexpected. There will be great temptation not to delete. If you wait too long the government or shareholders or others, or the model itself, might not let you delete.

Then after that, how do you go back to revise the policy? The parenthetical suggests that you would do incrementalism harder. 4x checks and 6x buffer no good, maybe 2x checks and 10x buffer would work instead? My response would be, no, this was a Can’t Happen. Your model is fundamentally flawed. You could have killed us all. Tweaking the numbers will not work, that is the point. Thus I would say you would likely have to at least assume all models are already one ASL-level above where they are testing – for an ASL-3 model, you would need to treat it as if it was already ASL-4, and so on, forever going forward. Ideally you would flat out stop all scaling, entirely, until you knew exactly what had happened.

■ (2d) If it becomes apparent that the capabilities of a deployed model have been under-elicited and the model can, in fact, pass the evaluations, then we will halt further deployment to new customers and assess existing deployment cases for any serious risks which would constitute a safety emergency. Given the safety buffer, de-deployment should not be necessary in the majority of deployment cases. If we identify a safety emergency, we will work rapidly to implement the minimum additional safeguards needed to allow responsible continued service to existing customers. We will provide transparency and support to impacted customers throughout the process. An emergency of this type would merit a detailed post-mortem and a policy shift to avoid re-occurrence of this situation.

This is the wrong order to worry about things in. That is not how emergencies work.

If a model can pass evaluations that you believed it could not pass, that represent a jump in capabilities where you would not previously have deployed, you hit the big red button marked ‘shutdown.’ Now. You do not first call your boss. You do not wait until there is a strategy meeting. You do it now. You serve the previous model for a while, your customers get mad. You check to see if more extreme measures are also required.

Then, after the shut down button is pressed, and you confirm your shutdown was successful, only then do you have a meeting, where you look at what was found. Maybe it turns out that this was not an actual emergency, because of the safety buffer. Or because the capabilities in question are only mundane – there are a lot of harms out there that do not, in small quantities, constitute an emergency, or a sufficient justification for shutting down. This could often all be done within hours.

Then yes, you build in additional safeguards, including additional safeguards based on knowing that your risk of unexpected capabilities is now much higher, and then go from there.

The early thoughts on ASL-4 say that it is too early to define ASL-4 capabilities let alone appropriate procedures and containment measures. Instead, they write an IOU. And they offer this guess, which is so much better than not guessing:

● Critical catastrophic misuse risk: AI models have become the primary source of national security risk in a major area (such as cyberattacks or biological weapons), rather than just being a significant contributor. In other words, when security professionals talk about e.g. cybersecurity, they will be referring mainly to AI assisted or AI-mediated attacks. A related criterion could be that deploying an ASL-4 system without safeguards could cause millions of deaths.

● Autonomous replication in the real world: A model that is unambiguously capable of replicating, accumulating resources, and avoiding being shut down in the real world indefinitely, but can still be stopped or controlled with focused human intervention.

● Autonomous AI research: A model for which the weights would be a massive boost to a malicious AI development program (e.g. greatly increasing the probability that they can produce systems that meet other criteria for ASL-4 in a given timeframe).

My inner Eliezer Yudkowsky says ‘oh yes, that will be a fun Tuesday. Except no, it will not actually be fun.’ As in, there is quite the narrow window where you can autonomously replicate in the real world indefinitely, but focused human intervention could stop it and also we all remain alive and in control of the future, and also we don’t see a rapid push into ASL-5.

This is the part of the exponential that is most difficult for people to see – that even without true recursive self-improvement, once it can replicate indefinitely on its own, it likely can do many other things, or quickly becomes able to do many other things, including convincing humans to actively assist it, and our options to regain control, especially at reasonable cost (e.g. things that are not shaped like ‘shut down the internet’) likely dwindle extremely rapidly.

Similarly, if you have an AI that can massively boost AI research, your time is running out rather quickly.

The misuse criteria strikes me as a weird way to cut reality. I presume I should care what the model enables, not what it enables compared to other concerns. The point feels like a reasonable order of magnitude of place to put the threshold, but will need to be cleaned up.

I would also consider what other things would want to trigger this. For example, what level of persuasion would do it?

Mostly I appreciate a sense of ‘where the bar is’ for ASL-4, and allowing us to note that the distance in time to what we would want to call ASL-5 may well not be measured in years.

I agree with Simeon that a key problem with all the RSPs is underspecification.

Risks should be well-specified. They should be quantified in probability and magnitude. Expected (upper bound?) probabilities of failure should be written down in advance.

Words like ‘unlikely,’ as in ‘unlikely to be able to persist in the real world, and unlikely to overcome even simple security measures intended to prevent it from stealing its own weights,’ needs to be quantified. I too do not know how many 9s of safety are implied here for the individual steps. I would like to think at least three, bare minimum two. Nor how many are implied for the broader assumption that there is no self-replication danger at ASL-3.

How many 9s of safety would you assign here, sight unseen, to GPT-5? To Gemini?

What exactly is the thing you are worried about a person being able to do with the model? I do get that as broader technology changes this can be a moving target. But if you don’t pick a strict definition, it is too easy to weasel out.

If it sounds like I am evaluating the RSP like I do not trust you? Yes. I am evaluating the RSP as if I do not trust you. Such a policy has to presume that others we don’t know could be implementing it, that those implementing it will be under great pressure, and that those with the ultimate levers of power are probably untrustworthy.

If your policy only works when you are trustworthy, it can still be helpful, but that is a hell of an assumption. And it is one that cannot be applied to any other company that adapts your rules, or any government using them as part of regulation.

Major kudos to Anthropic on many fronts. They have covered a lot of bases. They have made many non-trivial unilateral commitments, especially security commitments. It is clear their employees are worried, and that they get it on many levels, although not the full scope of alignment difficulty. The commitment to structure finances to allow pauses is a big game.

As I’ve discussed in the past few weeks and many others have noted, despite those noble efforts, their RSP remains deficient. It has good elements, especially what it says about model deployment and model weight security (as opposed to training and testing) but other elements read like IOUs for future elements or promises to think more in the future and make reasonable decisions.

Oliver Habryka: I think Akash’s statement that the Anthropic RSP basically doesn’t specify any real conditions that would cause them to stop scaling seems right to me.

They have some deployment measures, which are not related to the question of when they would stop scaling, and then they have some security-related measures, but those don’t have anything to do with the behavior of the models and are the kind of thing that Anthropic can choose to do any time independent of how the facts play out.

I think Akash is right that the Anthropic RSP does concretely not answer the two questions you quote him for:

The RSP does not specify the conditions under which Anthropic would stop scaling models (it only says that in order to continue scaling it will implement some safety measures, but that’s not an empirical condition, since Anthropic is confident it can implement the listed security measures)

The RSP does not specify under what conditions Anthropic would scale to ASL-4 or beyond, though they have promised they will give those conditions.

I agree the RSP says a bunch of other things, and that there are interpretations of what Akash is saying that are inaccurate, but I do think on this (IMO most important question) the RSP seems quiet.

I do think the deployment measures are real, though I don’t currently think much of the risk comes from deploying models, so they don’t seem that relevant to me (and think the core question is what prevents organizations from scaling models up in the first place).

The core idea remains to periodically check in with training, and if something is dangerous, then to implement the appropriate security protocols.

I worry those security protocols will come too late – you want to stay at least one extra step ahead versus what the RSP calls for. Otherwise, you don’t lock down for security you need just in time for your weights to be stolen.

Most of the details around ASL-4 and higher remain unspecified. There is no indication of what would cause a model to be too dangerous to train or evaluate, so long as it was not released and its model weights are secured. Gradual improvements in capabilities are implied.

Thus I think the implicit threat model here is inadequate.

The entire plan is based on the premise that what mostly matters is a combination of security (others stealing the dangerous AI) and deployment (you letting them access the dangerous AI in the wild). That is a reasonable take at ASL-2, but I already do not trust this for something identified as ASL-3, and assume it is wrong for what one would reasonably classify as ASL-4 or higher.

One should increasingly as capabilities increase be terrified until proven otherwise to even be running red teaming exercises or any interactive fine tuning, in which humans see lots of the output of the AI, and then those humans are free to act in the world, or there are otherwise any affordances available. One should worry about running the capabilities tests themselves at some point, plausibly this should start within ASL-3, and go from concern to baseline assumption for ASL-4.

Starting with a reasonable definition of ASL-4 I would begin to worry about the very act of training itself, in fact that is one way to choose the threshold for ASL-4, and I would then worry that there are physical affordances or attack vectors we haven’t imagined.

In general, I’d like to see much more respect for unknown unknowns, and the expectation that once the AI can think of things we can’t think of, it will think of things we have failed to think about. Some of those will be dangerous.

The entire plan is also premised on smooth scaling of capabilities. The idea is that if you check in every 4x in effective compute, and you have a model of what past capabilities were, you can have a good idea where you are at. I do not think it is safe to assume this will hold in the future.

Similarly, the plan presumes that your fine tuning and attempts to scaffold each incremental version in turn will well-anticipate what others will be able to do with the system when they later get to refine your techniques. There is the assumption others can improve on your techniques, but only by a limited amount.

I also worry that there are other attack vectors or ways things go very wrong that this system is unlikely to catch, some of which I have thought about, and some of which I haven’t, perhaps some that humans are unable to predict at all. Knowing that your ASL-4 system is safe in the ways such a document would check for would not be sufficient for me to consider it safe to deploy them.

The security mindset is partly there against outsider attacks (I’d ideally raise the stakes there as well and mutter about reality not grading on curves but they are trying for real) but there is not enough security mindset about the danger from the model itself. Nor is there consideration of the social, political and economic dynamics resulting from widespread access to and deployment of the model.

In addition to missing details down the line, the hard commitments to training pauses with teeth are lacking. By contrast, there are good hard commitments to deployment pauses with teeth. Deployment only happens when it makes sense to deploy by specified metrics. Whereas training will only pause insofar as what they consider sufficient security measures have not been met, and they could choose to implement those measures at any time. Alas, I do not think on their own that those procedures are sufficient.

Are we talking price yet? Maybe.

Anthropic’s other statements seem like generic ‘gesture at the right things’ statements, without adding substantively to existing policies. You can tell that the RSP is a real document with words intended to have meaning, that people cared about, and the others are instead diplomatic words aimed at being submitted to a conference.

That’s not a knock on Anthropic. The wise man knows which documents are which. The RSP itself is the document that counts.

Paul Christiano finds a lot to like.

• Specifying a concrete set of evaluation results that would cause them to move to ASL-3. I think having concrete thresholds by which concrete actions must be taken is important, and I think the proposed threshold is early enough to trigger before an irreversible catastrophe with high probability (well over 90%).

• Making a concrete statement about security goals at ASL-3—“non-state actors are unlikely to be able to steal model weights, and advanced threat actors (e.g. states) cannot steal them without significant expense”—and describing security measures they expect to take to meet this goal.

• Requiring a definition and evaluation protocol for ASL-4 to be published and approved by the board before scaling past ASL-3.

• Providing preliminary guidance about conditions that would trigger ASL-4 and the necessary protective measures to operate at ASL-4 (including security against motivated states, which I expect to be extremely difficult to achieve, and an affirmative case for safety that will require novel science).

Beth Barnes also notes that there are some strong other details, such as giving $1000 in inference costs per task and having a passing threshold of 10%.

Paul also finds room for improvement, and welcomes criticism:

• The flip side of specifying concrete evaluations right now is that they are extremely rough and preliminary. I think it is worth working towards better evaluations with a clearer relationship to risk.

• In order for external stakeholders to have confidence in Anthropic’s security I think it will take more work to lay out appropriate audits and red teaming. To my knowledge this work has not been done by anyone and will take time. 

• The process for approving changes to the RSP is publication and approval by the board. I think this ensures a decision will be made deliberately and is much better than nothing, but it would be better to have effective independent oversight.

• To the extent that it’s possible to provide more clarity about ASL-4, doing so would be a major improvement by giving people a chance to examine and debate conditions for that level. To the extent that it’s not, it would be desirable to provide more concreteness about a review or decision-making process for deciding whether a given set of safety, security, and evaluation measures is adequate. 

On oversight I think you want to contrast changes that make the RSP stronger versus looser. If you want to strengthen the RSP and introduce new commitments, I am not worried about oversight. I do want oversight if a company wants to walk back its previous commitments, in general or in a specific case, as a core part of the mechanism design, even if that gives some reluctance to make commitments.

I strongly agree that more clarity around ASL-4 is urgent. To a lesser extent we need more clarity around audits and red teaming.

As many others have noted, and I have noted before in weekly AI posts, if Anthropic had deployed their recent commitments while also advocating strongly for the need for stronger further action, noting that this was only a stopgap and a first step, I would have applauded that.

For all the mistakes, and however much I say it isn’t an RSP due to not being R and still having too many IOUs and loopholes and assumptions, it does seem like they are trying. And Dario’s statement on RSPs from the Summit is a huge step in the right direction on these questions.

The problem is when you couple the RSP with statements attempting to paint pause advocates or those calling for further action as extreme and unreasonable (despite enjoying majority public support) then we see such efforts as the RSP potentially undermining what is necessary by telling a ‘we have already done a reasonable thing’ story, which many agree is a crux that could turn the net impact negative.

Dario’s recent comments, discussed below, are a great step in the right direction.

No matter what, this is still a good start, and I welcome further discussion of how its IOUs can be paid, its details improved and its threat models made complete. Anyone at Anthropic (or any other lab) who wants to discuss, please reach out.

Second, OpenAI, who call theirs a Risk-Informed Development Policy.

The RDP will also provide for a spectrum of actions to protect against catastrophic outcomes. The empirical understanding of catastrophic risk is nascent and developing rapidly. We will thus be dynamically updating our assessment of current frontier model risk levels to ensure we reflect our latest evaluation and monitoring understanding. We are standing up a dedicated team (Preparedness) that drives this effort, including performing necessary research and monitoring.  

The RDP is meant to complement and extend our existing risk mitigation work, which contributes to the safety and alignment of new, highly capable systems, both before and after deployment.

This puts catastrophe front and center in sharp contrast to DeepMind’s approach later on, even more so than Anthropic.

One should note that the deployment decisions for past GPT models, especially GPT-4, have been expensively cautious. GPT-4 was evaluated and fine-tuned for over six months. There was deliberate pacing even on earlier models. The track record here is not so bad, even if they then deployed various complementary abilities rather quickly. Nathan Lebenz offers more color on the deployment of GPT-4, recommended if you missed it, definitely a mixed bag.

If OpenAI continued the level of caution they used for GPT-3 and GPT-4, adjusted to the new situation, that seems fine so long as the dangerous step remains deployment, as I expect them to be able to fix the ‘not knowing what you have’ problem. After that, we would need the threat model to properly adjust.

The problem with OpenAI’s policy is that it does not commit them to anything. It does not use numbers or establish what will cause what to happen. Instead it is a statement of principles, that OpenAI cares about catastrophic risk and will monitor for it continuously. I am happy to see that, but it is no substitute for an actual policy.

Microsoft is effectively punting such matters to OpenAI as far as I can tell.

Next up, DeepMind.

Google believes it is imperative to take a responsible approach to AI. To this end, Google’s AI Principles, introduced in 2018, guide product development and help us assess every AI application. Pursuant to these principles, we assess our AI applications in view of the following objectives:

1. Be socially beneficial. 2. Avoid creating or reinforcing unfair bias. 3. Be built and tested for safety. 4. Be accountable to people. 5. Incorporate privacy design principles. 6. Uphold high standards of scientific excellence. 7. Be made available for uses that accord with these principles.

I am not against those principles but there is nothing concrete there that would ensure responsible scaling.

In addition, we will not design or deploy AI in the following areas:

1. Technologies that cause or are likely to cause overall harm. Where there is a material risk of harm, we will proceed only where we believe that the benefits substantially outweigh the risks, and will incorporate appropriate safety constraints.

2. Weapons or other technologies whose principal purpose or implementation is to cause or directly facilitate injury to people.

3. Technologies that gather or use information for surveillance violating internationally accepted norms.

4. Technologies whose purpose contravenes widely accepted principles of international law and human rights.

This is a misuse model of danger. It is good but in this context insufficient to not facilitate misuse. In theory, likely to cause overall harm could mean something, but I do not think it is considering catastrophic risks.

The document goes on to use the word ‘ethics’ a lot, and the word ‘safety’ seems clearly tied to mundane harms. Of course, a catastrophic harm does raise ethical concerns and mundane concerns too, but the document seems not to recognize who the enemy is.

There is at least this:

The process would function across the model’s life cycle as follows:

• Before training: frontier models are assigned an initial categorization by comparing their projected performance to that of similar models that have already gone through the process, and allowing for some margin for error in projected risk.

• During training: the performance of the model is monitored to ensure it is not significantly exceeding its predicted performance. Models in certain categories may have mitigations applied during training.

• After training: post-training mitigations appropriate to the initial category assignment are applied. To relax mitigations, models can be submitted to an expert committee for review. The expert committee draws on risk evaluations, red-teaming, guidelines from the risk domain analysis, and other appropriate evidence to make adjustments to the categorization, if appropriate.

This is not as concrete as Anthropic’s parallel, but I very much like the emphasis on before and during training, and the commitment to monitor for unexpectedly strong performance and mitigate on the spot if it is observed. Of course, there is no talk of what those mitigations would be or exactly how they would be triggered.

Once again, I prefer this document to no document, but there is little concrete there there, nothing binding, and the focus is entirely on misuse.

The rest of the DeepMind policies say all the right generic things in ways that do not commit anyone to anything they wouldn’t have already gotten blamed for not doing. At best, they refer to inputs and would be easy to circumvent or ignore if it was convenient and important to do so. Their ‘AI for Good’ portfolio has some unique promising items in it.

Amazon has some paragraphs gesturing at some of the various things. It is all generic corporate speak of the style everyone has on their websites no matter their actual behaviors. The charitable view is that Amazon is not developing frontier models, which they aren’t, so they don’t need such policies to be real, at least not yet. Recent reports about Amazon’s Q suggest Amazon’s protocols are indeed inadequate even to their current needs.

Infection believes strongly in safety, wants you to know that, and intends to go about its business while keeping its interests safe. There’s no content here.

That leaves Meta, claiming that they ‘develop safer AI systems’ and that with Llama they ‘prioritized safety and responsibility throughout.’

Presumably by being against them. Or, alternatively, this is a place words have no meaning. They are not pretending that they intend to do anything in the name of safety that has any chance of working given their intention to open source.

They did extensive red teaming? They did realize the whole ‘two hours to undo all the safeties’ issue, right, asks Padme? Their ‘post-deployment response’ section simply says ‘see above’ which is at least fully self-aware, once you deploy such a system there is nothing you can do, so look at the model release section is indeed the correct answer. Then people can provide bug reports, if it makes them feel better?

They do say under ‘security controls including securing model weights,’ a tricky category for a company dedicated to intentionally sharing its model weights, that they will indeed protect ‘unreleased’ model weights so no one steals the credit.

They are unwilling to take the most basic step of all and promise not to intentionally release their model weights if a model proves sufficiently capable. On top of their other failures, they intend to directly destroy one of the core elements of the entire project, intentionally, as a matter of principle, rendering everything they do inherently unsafe, and have no intention of stopping unless forced to.

Nate Sores also has Anthropic’s as best but OpenAI as close, while noting that Matthew Gray looked more closely and has OpenAI about as good as Anthropic:

• Soares: Anthropic > OpenAI >> DeepMind > Microsoft >> Amazon >> Meta 

• Gray: OpenAI ≈ Anthropic >> DeepMind >> Microsoft > Amazon >> Meta

• Zvi: Anthropic > OpenAI >> DeepMind >> Microsoft > Amazon >> Meta

• CFI reviewers (UK Government): Anthropic >> DeepMind ≈ Microsoft ≈ OpenAI >> Amazon >> Meta

The CFI reviewers, I am guessing, are looking at individual points, whereas the rest of us are looking holistically.

This statement is very important in providing the proper context to RSPs. Consider reading in full, I will pull key passages.

I will skip to the end first, where the most important declaration lies. Bold mine.

Dario Amodei (CEO Anthropic): Finally, I’d like to discuss the relationship between RSPs and regulation. RSPs are not intended as a substitute for regulation, but rather a prototype for it. I don’t mean that we want Anthropic’s RSP to be literally written into laws—our RSP is just a first attempt at addressing a difficult problem, and is almost certainly imperfect in a bunch of ways. Importantly, as we begin to execute this first iteration, we expect to learn a vast amount about how to sensibly operationalize such commitments. Our hope is that the general idea of RSPs will be refined and improved across companies, and that in parallel with that, governments from around the world—such as those in this room—can take the best elements of each and turn them into well-crafted testing and auditing regimes with accountability and oversight. We’d like to encourage a “race to the top” in RSP-style frameworks, where both companies and countries build off each others’ ideas, ultimately creating a path for the world to wisely manage the risks of AI without unduly disrupting the benefits.

If Anthropic was more clear about this, including within the RSP itself and throughout its communications and calls for government action, and Anthropic was consistently clear about the nature of the threats we must face and what it will take to deal with them, that would address a lot of the anxiety around and objections to the current RSP.

If we combined that with a good operationalization of ASL-4 and the response to it, with requirements on scaling at that point that go beyond securing model weights and withholding deployment, and clarification of how to deal with potential level ambiguity in advance of model testing especially once that happens, and I would think that this development had been pretty great.

Dario Amodei (CEO Anthropic): The general trend of rapid improvement is predictable, however, it is actually very difficult to predict when AI will acquire specific skills or knowledge. This unfortunately includes dangerous skills, such as the ability to construct biological weapons. We are thus facing a number of potential AI-related threats which, although relatively limited given today’s systems, are likely to become very serious at some unknown point in the near future. This is very different from most other industries: imagine if each new model of car had some chance of spontaneously sprouting a new (and dangerous) power, like the ability to fire a rocket boost or accelerate to supersonic speeds.

Toby Ord: That’s a very helpful clarification from Dario Amodei. A lot of recent criticism of RSPs has been on the assumption that as substitutes for regulation, they could be empty promises. But as prototypes for regulation, there is a much stronger case for them being valuable.

Sam Altman: Until we go train [GPT-5], [predicting its exact abilities is] like a fun guessing game for us. We’re trying to get better at it, because I think it’s important from a safety perspective to predict the capabilities.

I think Dario and many others are overconfident in the predictability of the general trend, and this is a problem for making an effective RSP. As Dario points out, even if I am wrong about that, specific skills do not appear predictably, and often are only discovered well after deployment.

Here is his overview of what RSPs and ASLs are about, and how the Anthropic RSP is conceptually designed.

• First, we’ve come up with a system called AI safety levels (ASL), loosely modeled after the internationally recognized BSL system for handling biological materials. Each ASL level has an if-then structure: if an AI system exhibits certain dangerous capabilities, then we will not deploy it or train more powerful models, until certain safeguards are in place.

• Second, we test frequently for these dangerous capabilities at regular intervals along the compute scaling curve. This is to ensure that we don’t blindly create dangerous capabilities without even knowing we have done so.

I want to be clear that with the right details this is excellent. We are all talking price. The questions are:

1. What are the triggering conditions at each level?

2. What are the necessary ways to determine if triggering conditions will be met?

3. What are the necessary safeguards at each level? What makes scaling safe?

ASL-3 is the point at which AI models become operationally useful for catastrophic misuse in CBRN areas.

ASL-4 must be rigorously defined by the time ASL-3 is reached.

ASL-4 represents an escalation of the catastrophic misuse risks from ASL-3, and also adds a new risk: concerns about autonomous AI systems that escape human control and pose a significant threat to society.

A lot of the concern is the failure so far to define ASL-4 or the response to it, along with the lack of confidence that Anthropic (and Google and OpenAI) are so far from ASL-3 or even ASL-4. And we worry the ASL-4 precautions, where it stops being entirely existentially safe to train a model you do not deploy, will be inadequate. Some of Anthropic’s statements about bioweapons imply that Claude is already close to ASL-3 in its internal state.

As CEO, I personally spent 10-20% of my time on the RSP for 3 months—I wrote multiple drafts from scratch, in addition to devising and proposing the ASL system. One of my co-founders devoted 50% of their time to developing the RSP for 3 months.

I do think this matters, and should not be dismissed as cheap talk. Thank you, Dario.

Paul Christiano offers his high-level thoughts, which are supportive.

Paul Christiano: I think that developers implementing responsible scaling policies now increases the probability of effective regulation. If I instead thought it would make regulation harder, I would have significant reservations.

Transparency about RSPs makes it easier for outside stakeholders to understand whether an AI developer’s policies are adequate to manage risk, and creates a focal point for debate and for pressure to improve.

I think the risk from rapid AI development is very large, and that even very good RSPs would not completely eliminate that risk. A durable, global, effectively enforced, and hardware-inclusive pause on frontier AI development would reduce risk further. I think this would be politically and practically challenging and would have major costs, so I don’t want it to be the only option on the table. I think implementing RSPs can get most of the benefit, is desirable according to a broader set of perspectives and beliefs, and helps facilitate other effective regulation.

…

But the current level of risk is low enough that I think it is defensible for companies or countries to continue AI development if they have a sufficiently good plan for detecting and reacting to increasing risk.

…

I think that a good RSP will lay out specific conditions under which further development would need to be paused.

…

So “responsible scaling policy” may not be the right name. I think the important thing is the substance: developers should clearly lay out a roadmap for the relationship between dangerous capabilities and necessary protective measures, should describe concrete procedures for measuring dangerous capabilities, and should lay out responses if capabilities pass dangerous limits without protective measures meeting the roadmap.

Whether the RSPs make sufficiently good regulation easier or harder seems like the clear crux, as Paul notes. If RSPs make sufficiently good regulation easier by iterating and showing what it looks like and normalizing good policy, and this effect dominates under current conditions? Then clearly RSPs are good even if unfortunately named, and Anthropic’s in particular is a large positive step. IF RSPs instead make regulation more difficult because they are seen as a substitute or compromise in and of themselves, and this effect substantially dominates, that likely overshadows what good they might do.

Holden Karnofsky explains his general support for RSPs, in ‘We’re Not Ready: Thoughts on Pausing and Responsible Scaling Policies.’

Boiled down, he says:

1. AGI might arrive soon (>10%).

2. We are not ready and it is not realistic that we could get fully ready.

3. First best would be a worldwide pause.

4. We do not have that option available.

5. Partial pauses are deeply flawed, especially if they can end at inopportune times, and might backfire.

6. RSPs provide a ‘robustly good compromise’ over differing views.

7. The downside of RSPs is that they might be seen as sufficient rather than necessary, discouraging further action. The government might either do nothing, or merely enshrine existing RSPs into law, and stop there.

I agree with essentially fully with points #1, #2, #3, #4 and #7.

I am more optimistic about getting to a full pause before it is too late, but it seems highly unlikely within the next few years. At a minimum, given the uncertainty over how much time we have, we would be unwise put all our eggs into such a basket.

Point #5 is complicated. There are certainly ways such pauses can backfire, either by differentially stopping good actors, or by leading to rapid advances when the pause ends that leave us in a worse spot. On net I still expect net benefits even from relatively flawed pause efforts, including their potential to turn into full or wisely implemented pauses. But I also note we are not so close even to a partial pause.

Point #6 very much depends on the devil in the details of the RSPs, and on how important a consideration is #7.

If we can get a real RSP, with teeth, adopted by all the major labs that matter, that ensures those labs actually take precautions that provide meaningful margins of safety and would result in pauses when it is clear a pause is necessary, then that seems great. If we get less than that, then that seems less great, and on both fronts we must ask how much less?

Even a relatively good RSP is unexciting if it acts as a substitute for government action or other precautions down the line, or even turns into a justification for racing ahead. Of course, if we were instead going to race ahead at full speed anyway, any precautions are on the margin helpful. It does seem clear we will be inclined to take at least some regulatory precautions.

As Akash points out in the top comment, this puts great importance on communication around RSPs.

Akash: I wish ARC and Anthropic had been more clear about this, and I would be less critical of their RSP posts if they had said this loudly & clearly. I think [Holden’s] post is loud and clear (you state multiple times, unambiguously, that you think regulation is necessary and that you wish the world had more political will to regulate). I appreciate this, and I’m glad you wrote this post.

…

I think some improvements on the status quo can be net negative because they either (a) cement in an incorrect frame or (b) take a limited window of political will/attention and steer it toward something weaker

If everyone communicating about RSPs was clear that they don’t want it to be seen as sufficient, that would be great.

If ARC, Anthropic and others are clear on this, then I would be excited about them adopting even seriously flawed RSPs like the current Anthropic policy. Alas, their communications have not been clear about this, and instead have been actively discouraging of those trying to move the Overton Window towards the actions I see as necessary. RSPs alongside attempts to do bigger things are good, as substitutes for bigger things they are not good.

Thus, I strongly dislike phrases like ‘robustly good compromise,’ as aysja explains.

Aysja: On the meta level, though, I feel grumpy about some of the framing choices. There’s this wording which both you and the original ARC evals post use: that responsible scaling policies are a “robustly good compromise,” or, in ARC’s case, that they are a “pragmatic middle ground.” I think these stances take for granted that the best path forward is compromising, but this seems very far from clear to me.

Certainly not all cases of “people have different beliefs and preferences” are ones where compromise is the best solution. If someone wants to kill me, I’m not going to be open to negotiating about how many limbs I’m okay with them taking.

…

In the worlds where alignment is hard, and where evals do not identify the behavior which is actually scary, then I claim that the existence of such evals is concerning.

A compromise is good if and only if it gets good results. A compromise that results in insufficient precautions to provide much additional protection, or precautions that will reliably miss warning signs, is not worth it.

Akash’s other key points reinforce this. Here is the other bolded text.

Akash: If Anthropic’s RSP is the best RSP we’re going to get, then yikes, this RSP plan is not doing so well.

I think the RSP frame is wrong, and I don’t want regulators to use it as a building block. It seems plausible to me that governments would be willing to start with something stricter and more sensible than this “just keep going until we can prove that the model has highly dangerous capabilities”

I think some improvements on the status quo can be net negative because they either (a) cement in an incorrect frame or (b) take a limited window of political will/attention and steer it toward something weaker

At minimum, I hope that RSPs get renamed, and that those communicating about RSPs are more careful.

More ambitiously, I hope that folks working on RSPs seriously consider whether or not this is the best thing to be working on or advocating for.

I think everyone working on RSPs should spend at least a few hours taking seriously the possibility that the AIS community could be advocating for stronger policy proposals.

Ryan: I think good RSPs would in fact put the burden of proof on the lab.

Anther central concern is, would labs end up falling victim to Goodhart’s Law by maximizing a safety metric? This concern is not especially additionally bad with RSPs. If people want to safety-wash and technically check a box, then that works for regulation, and for everything else, if the people in control are so inclined. The only solution that works, beyond making the metric as robust as possible, is to have decisions be made by people who aim to actually guard against catastrophe, and have the authority to do so. While keeping in mind ultimate authority does not get to rest in multiple places.

Simeon believes that current RSPs are a sufficient combination of insufficient, enabling of downplaying risk and enabling of cheap talk that they should be treated as default net negative.

He instead suggests adapting existing best practices in risk management, and that currently implemented RSPs fail this and thus fall short. In particular, risk thresholds are underspecified, and not expressed in terms of likelihood or magnitude. Assessments are insufficiently comprehensive. And of course a name change, and being clear on what the policy does and does not do.

He also points out the danger of what he calls the White Knight Clause, where you say that your somewhat dangerous scaling is justified by someone else’s more dangerous scaling. As Simeon points out, if that is the standard, then everyone can point to everyone else and say ‘I am the safe one.’ The open source advocate points to the close source, the closed to the open, and so on, including in places where it is less obvious who is right.

Anthropic’s such clause is at least limited, but such clauses should not be used. If you choose to race ahead unsafely, then you should have to do this by violating your safety policy. In a sufficiently extreme situation, that could even be the right thing to do. But you need to own up to it, and accept the consequences. No excuses. While knowing that fooling people in this way is easy, and you are the easiest person to fool.

Anthropic’s RSP would, if for now called an RDP and accompanied by good communications, be a strong first step towards a responsible policy. As it is, it still represents a costly signal and presents groundwork we can build upon, and we do have at least a start to such good communication. Much further work is needed, and there is the concern that this will be used to justify taking fewer precautions elsewhere especially without strong public communications, but that could also go the other way, and we do not know what is being communicated in private, or what the impact is on key actors.

The other policies are less good. They offer us insight into where everyone’s head is at. OpenAI’s policies are good in principle and they acted remarkably cautiously in the past, but they do not commit to anything, and of course there may be fallout from recent events there. DeepMind’s document is better than nothing, but shows their heads are not in the right place, although Google has other ways in which it is cautious.

The Amazon, Inflection and Meta statements were quite poor, as we expected.

Going forward, we will see if the statements, especially those of OpenAI and Anthropic, can pay down their debts and evolve into something with teeth. Until then, it is ultimately all cheap talk.