openai

Previously: OpenAI: Exodus (contains links at top to earlier episodes), Do Not Mess With Scarlett Johansson

We have learned more since last week. It’s worse than we knew.

How much worse? In which ways? With what exceptions?

That’s what this post is about.

For years, employees who left OpenAI consistently had their vested equity explicitly threatened with confiscation and the lack of ability to sell it, and were given short timelines to sign documents or else. Those documents contained highly aggressive NDA and non disparagement (and non interference) clauses, including the NDA preventing anyone from revealing these clauses.

No one knew about this until recently, because until Daniel Kokotajlo everyone signed, and then they could not talk about it. Then Daniel refused to sign, Kelsey Piper started reporting, and a lot came out.

Here is Altman’s statement from May 18, with its new community note.

Evidence strongly suggests the above post was, shall we say, ‘not consistently candid.’

The linked article includes a document dump and other revelations, which I cover.

Then there are the other recent matters.

Ilya Sutskever and Jan Leike, the top two safety researchers at OpenAI, resigned, part of an ongoing pattern of top safety researchers leaving OpenAI. The team they led, Superalignment, had been publicly promised 20% of secured compute going forward, but that commitment was not honored. Jan Leike expressed concerns that OpenAI was not on track to be ready for even the next generation of models needs for safety.

OpenAI created the Sky voice for GPT-4o, which evoked consistent reactions that it sounded like Scarlett Johansson, who voiced the AI in the movie Her, Altman’s favorite movie. Altman asked her twice to lend her voice to ChatGPT. Altman tweeted ‘her.’ Half the articles about GPT-4o mentioned Her as a model. OpenAI executives continue to claim that this was all a coincidence, but have taken down the Sky voice.

(Also six months ago the board tried to fire Sam Altman and failed, and all that.)

The source for the documents from OpenAI that are discussed here, and the communications between OpenAI and its employees and ex-employees, is Kelsey Piper in Vox, unless otherwise stated.

She went above and beyond, and shares screenshots of the documents. For superior readability and searchability, I have converted those images to text.

OpenAI has indeed made a large positive step. They say they are releasing former employees from their nondisparagement agreements and promising not to cancel vested equity under any circumstances.

Kelsey Piper: There are some positive signs that change is happening at OpenAI. The company told me, “We are identifying and reaching out to former employees who signed a standard exit agreement to make it clear that OpenAI has not and will not cancel their vested equity and releases them from nondisparagement obligations.”

Bloomberg confirms that OpenAI has promised not to cancel vested equity under any circumstances, and to release all employees from one-directional non-disparagement agreements.

And we have this confirmation from Andrew Carr.

Andrew Carr: I guess that settles that.

Tanner Lund: Is this legally binding?

Andrew Carr:

I notice they are also including the non-solicitation provisions as not enforced.

(Note that certain key people, like Dario Amodei, plausibly negotiated two-way agreements, which would mean theirs would still apply. I would encourage anyone in that category who is now free of the clause, even if they have no desire to disparage OpenAI, to simply say ‘I am under no legal obligation not to disparage OpenAI.’)

These actions by OpenAI are helpful. They are necessary.

They are not sufficient.

First, the statement is not legally binding, as I understand it, without execution of a new agreement. No consideration was given, and this is not so formal, and it is unclear whether the statement author has authority in the matter.

Even if it was binding as written, it says they do not ‘intend’ to enforce. Companies can change their minds, or claim to change them, when circumstances change.

It also does not mention the ace in the hole, which is the ability to deny access to tender offers, or other potential retaliation by Altman or OpenAI. Until an employee has fully sold their equity, they are still in a bind. Even afterwards, a company with this reputation cannot be trusted to not find other ways to retaliate.

Nor does it mention the clause of right to repurchase for ‘fair market value’ that OpenAI claims it has the right to do, noting that their official ‘fair market value’ of shares is $0. Altman’s statement does not mention this at all, including the possibility it has already happened.

I mean, yeah, I also would in many senses like to see them try that one, but this does not give ex-employees much comfort.

A source of Kelsey Piper’s close to OpenAI: [Those] documents are supposed to be putting the mission of building safe and beneficial AGI first but instead they set up multiple ways to retaliate against departing employees who speak in any way that criticizes the company.

Then there is the problem of taking responsibility. OpenAI is at best downplaying what happened. Certain statements sure look like lies. To fully set things right, one must admit responsibility. Truth and reconciliation requires truth.

Here is Kelsey with the polite version.

Kelsey Piper: But to my mind, setting this right requires admitting its full scope and accepting full responsibility. OpenAI’s initial apology implied that the problem was just ‘language in exit documents’. Our leaked docs prove there was a lot more going on than just that.

OpenAI used many different aggressive legal tactics and has not yet promised to stop using all of them. And serious questions remain about how OpenAI’s senior leadership missed this while signing documents that contained language that laid it out. The company’s apologies so far have minimized the scale of what happened. In order to set this right, OpenAI will need to first admit how extensive it was.

If I were an ex-employee, no matter what else I would do, I would absolutely sell my equity at the next available tender opportunity. Why risk it?

Indeed, here is a great explanation of the practical questions at play. If you want to fully make it right, and give employees felt freedom to speak up, you have to mean it.

Jacob Hilton: When I left OpenAI a little over a year ago, I signed a non-disparagement agreement, with non-disclosure about the agreement itself, for no other reason than to avoid losing my vested equity.

The agreement was unambiguous that in return for signing, I was being allowed to keep my vested equity, and offered nothing more. I do not see why anyone would have signed it if they had thought it would have no impact on their equity.

I left OpenAI on great terms, so I assume this agreement was imposed upon almost all departing employees. I had no intention to criticize OpenAI before I signed the agreement, but was nevertheless disappointed to give up my right to do so.

Yesterday, OpenAI reached out to me to release me from this agreement, following Kelsey Piper’s excellent investigative reporting.

Because of the transformative potential of AI, it is imperative for major labs developing advanced AI to provide protections for those who wish to speak out in the public interest.

First among those is a binding commitment to non-retaliation. Even now, OpenAI can prevent employees from selling their equity, rendering it effectively worthless for an unknown period of time.

In a statement, OpenAI has said, “Historically, former employees have been eligible to sell at the same price regardless of where they work; we don’t expect that to change.”

I believe that OpenAI has honest intentions with this statement. But given that OpenAI has previously used access to liquidity as an intimidation tactic, many former employees will still feel scared to speak out.

I invite OpenAI to reach out directly to former employees to clarify that they will always be provided equal access to liquidity, in a legally enforceable way. Until they do this, the public should not expect candor from former employees.

To the many kind and brilliant people at OpenAI: I hope you can understand why I feel the need to speak publicly about this. This contract was inconsistent with our shared commitment to safe and beneficial AI, and you deserve better.

Jacob Hinton is giving every benefit of the doubt to OpenAI here. Yet he notices that the chilling effects will be large.

Jeremy Schlatter: I signed a severance agreement when I left OpenAI in 2017. In retrospect, I wish I had not signed it.

I’m posting this because there has been coverage of OpenAI severance agreements recently, and I wanted to add my perspective.

I don’t mean to imply that my situation is the same as those in recent coverage. For example, I worked at OpenAI while it was still exclusively a non-profit, so I had no equity to lose.

Was this an own goal? Kelsey initially thought it was, then it is explained why the situation is not so clear cut as that.

Kelsey Piper: Really speaks to how profoundly the “ultra restrictive secret NDA or lose your equity” agreement was an own goal for OpenAI – I would say a solid majority of the former employees affected did not even want to criticize the company, until it threatened their compensation.

A former employee reached out to me to push back on this. It’s true that most don’t want to criticize the company even without the NDA, they told me, but not because they have no complaints – because they fear even raising trivial ones.

“I’ve heard from former colleagues that they are reluctant to even discuss OpenAI’s model performance in a negative way publicly, for fear of being excluded from future tenders.”

Speaks to the importance of the further steps Jacob talks about.

There are big advantages to being generally seen as highly vindictive, as a bad actor willing to do bad things if you do not get your way. Often that causes people to proactively give you what you want and avoid threatening your interests, with no need to do anything explicit. Many think this is how one gets power, and that one should side with power and with those who act in such fashion.

There also is quite a lot of value in controlling the narrative, and having leverage over those close to you, that people look to for evidence, and keeping that invisible.

What looks like a mistake could be a well-considered strategy, and perhaps quite a good bet. Most companies that use such agreements do not have them revealed. If it was not for Daniel, would not the strategy still be working today?

And to state the obvious: If Sam Altman and OpenAI lacked any such leverage in November, and everyone had been free to speak their minds, does it not seem plausible (or if you include the board, rather probable) that the board’s firing of Altman would have stuck without destroying the company, as ex-employees (and board members) revealed ways in which Altman had been ‘not consistently candid’?

Oh my.

Neel Nanda (referencing Hilton’s thread): I can’t believe that OpenAI didn’t offer *anypayment for signing the non-disparage, just threats…

This makes it even clearer that Altman’s claims of ignorance were lies – he cannot possibly have believed that former employees unanimously signed non-disparagements for free!

Kelsey Piper: One of the most surreal moments of my life was reading through the termination contract and seeing…

The Termination Contract: NOW, THEREFORE, in consideration of the mutual covenants and promises herein contained and other good and valuable consideration, receipt of which is hereby acknowledged, and to avoid unnecessary litigation, it is hereby agreed by and between OpenAI and Employee (jointly referred to as “the Parties”) as follows:

1. In consideration for this Agreement:

   1. Employee will retain all equity Units, if any, vested as of the Termination Date pursuant to the terms of the applicable Unit Grant Agreements.

2. Employee agrees that the foregoing shall constitute an accord and satisfaction and a full and complete settlement of Employee’s claims, shall constitute the entire amount of monetary consideration, including any equity component (if applicable), provided to Employee under this Agreement, and that Employee will not seek any further compensation for any other claimed damage, outstanding obligations, costs or attorneys’ fees in connection with the matters encompassed in this Agreement… [continues]

Neel Nanda: Wow, I didn’t realise it was that explicit in the contract! How on earth did OpenAI think they were going to get away with this level of bullshit? Offering something like, idk, 1-2 months of base salary would have been cheap and made it a LITTLE bit less outrageous.

It does not get more explicit than that.

I do appreciate the bluntness and honest here, of skipping the nominal consideration.

What looks the most implausible are claims that the executives did not know what was going on regarding the exit agreements and legal tactics until February 2024.

Kelsey Piper: Vox reviewed separation letters from multiple employees who left the company over the last five years. These letters state that employees have to sign within 60 days to retain their vested equity. The letters are signed by former VP Diane Yoon and general counsel Jason Kwon.

The language on separation letters – which reads, “If you have any vested Units… you are required to sign a release of claims agreement within 60 days in order to retain such Units.” has been present since 2019.

OpenAI told me that the company noticed in February, putting Kwon, OpenAI’s general counsel and Chief Strategy Officer, in the unenviable position of insisting that for five years he missed a sentence in plain English on a one-page document he signed dozens of times.

Matthew Roche: This cannot be true.

I have been a tech CEO for years, and have never seen that it in an option plan doc or employment letter. I find it extremely unlikely that some random lawyer threw it in without prompting or approval by the client.

Kelsey Piper: I’ve spoken to a handful of tech CEOs in the last few days and asked them all “could a clause like that be in your docs without your knowledge?” All of them said ‘no’.

Kelsey Piper’s Vox article is brutal on this, and brings the receipts. The ultra-restrictive NDA, with its very clear and explicit language of what is going on, is signed by COO Brad Lightcap. The notices that one must sign it are signed by (now departed) OpenAI VP of people Diane Yoon. The incorporation documents that include extraordinary clawback provisions are signed by Sam Altman.

There is also the question of how this language got into the exit agreements in the first place, and also the corporate documents, if the executives were not in the loop. This was not a ‘normal’ type of clause, the kind of thing lawyers sneak in without consulting you, even if you do not read the documents you are signing.

California employment law attorney Chambord Benton-Hayes: For a company to threaten to claw back already-vested equity is egregious and unusual.

Kelsey Piper on how she reported the story: Reporting is full of lots of tedious moments, but then there’s the occasional “whoa” moment. Reporting this story had three major moments of “whoa.” The first is when I reviewed an employee termination contract and saw it casually stating that as “consideration” for signing this super-strict agreement, the employee would get to keep their already vested equity. That might not mean much to people outside the tech world, but I knew that it meant OpenAI had crossed a line many in tech consider close to sacred.

The second “whoa” moment was when I reviewed the second termination agreement sent to one ex-employee who’d challenged the legality of OpenAI’s scheme. The company, rather than defending the legality of its approach, had just jumped ship to a new approach.

That led to the third “whoa” moment. I read through the incorporation document that the company cited as the reason it had the authority to do this and confirmed that it did seem to give the company a lot of license to take back vested equity and block employees from selling it. So I scrolled down to the signature page, wondering who at OpenAI had set all this up. The page had three signatures. All three of them were Sam Altman. I slacked my boss on a Sunday night, “Can I call you briefly?”

OpenAI claims they noticed the problem in February, and began updating in April.

Kelsey Piper showed language of this type in documents as recently as April 29, 2024, signed by OpenAI COO Brad Lightcap.

The documents in question, presented as standard exit ‘release of claims’ documents that everyone signs, include extensive lifetime non disparagement clauses, an NDA that covers revealing the existence of either the NDA or the non disparagement clause, and a non-interference clause.

Kelsey Piper: Leaked emails reveal that when ex-employees objected to the specific terms of the ‘release of claims’ agreement, and asked to sign a ‘release of claims’ agreement without the nondisclosure and secrecy clauses, OpenAI lawyers refused.

Departing Employee Email: I understand my contractual obligations to maintain confidential information and trade secrets. I would like to assure you that I have no intention and have never had any intention of sharing trade secrets with OpenAl competitors.

I would be willing to sign the termination paperwork documents except for the current form of the general release as I was sent on 2024.

I object to clauses 10, 11 and 14 of the general release. I would be willing to sign a version of the general release which excludes those clauses. I believe those clauses are not in my interest to sign, and do not understand why they have to be part of the agreement given my existing obligations that you outlined in your letter.

I would appreciate it if you could send a copy of the paperwork with the general release amended to exclude those clauses.

Thank you,

[Quoted text hidden]

OpenAI Replies: I’m here if you ever want to talk. These are the terms that everyone agrees to (again — this is not targeted at you). Of course, you’re free to not sign. Please let me know if you change your mind and want to sign the version we’ve already provided.

Best,

[Quoted text hidden]

Here is what it looked like for someone to finally decline to sign.

Departing Employee: I’ve looked this over and thought about it for a while and have decided to decline to sign. As previously mentioned, I want to reserve the right to criticize OpenAl in service of the public good and OpenAl’s own mission, and signing this document appears to limit my ability to do so. I certainly don’t intend to say anything false, but it seems to me that I’m currently being asked to sign away various rights in return for being allowed to keep my vested equity. It’s a lot of money, and an unfair choice to have to make, but I value my right to constructively criticize OpenAl more. I appreciate your warmth towards me in the exit interview and continued engagement with me thereafter, and wish you the best going forward.

Thanks,

P.S. I understand your position is that this is standard business practice, but that doesn’t sound right, and I really think a company building something anywhere near as powerful as AGI should hold itself to a higher standard than this – that is, it should aim to be genuinely worthy of public trust. One pillar of that worthiness is transparency, which you could partially achieve by allowing employees and former employees to speak out instead of using access to vested equity to shut down dissenting concerns.

OpenAI HR responds: Hope you had a good weekend, thanks for your response.

Please remember that the confidentiality agreement you signed at the start of your employment (and that we discussed in our last sync) remains in effect regardless of the signing of the offboarding documents.

We appreciate your contributions and wish you the best in your future endeavors. If you have any further questions or need clarification, feel free to reach out.

OpenAI HR then responds (May 17th, 2: 56pm, after this blew up): Apologies for some potential ambiguity in my last message!

I understand that you may have some questions about the status of your vested profit units now that you have left OpenAI. I want to be clear that your vested equity is in your Shareworks account, and you are not required to sign your exit paperwork to retain the equity. We have updated our exit paperwork to make this point clear.

Please let me know if you have any questions.

Best, [redacted]

Some potential ambiguity, huh. What a nice way of putting it.

Even if we accepted on its face the claim that this was unintentional and unknown to management until February, which I find highly implausible at best, that is no excuse.

Jason Kwan (OpenAI Chief Strategist): The team did catch this ~month ago. The fact that it went this long before the catch is on me.

Again, even if you are somehow telling the truth here, what about after the catch?

Two months is more than enough time to stop using these pressure tactics, and to offer ‘clarification’ to employees. I would think it was also more than enough time to update the documents in question, if OpenAI intended to do that.

They only acknowledged the issue, and only stopped continuing to act this way, after the reporting broke. After that, the ‘clarifications’ came quickly. Then, as far as we can tell, the actually executed new agreements and binding contracts will come never. Does never work for you?

Here we have OpenAI’s lawyer refusing to extend a unilaterally imposed seven day deadline to sign the exit documents, discouraging the ex-employee from consulting with an attorney.

Kelsey Piper: Legal experts I spoke to for this story expressed concerns about the professional ethics implications of OpenAI’s lawyers persuading employees who asked for more time to seek outside counsel to instead “chat live to cover your questions” with OpenAI’s own attorneys.

Reply Email from Lawyer for OpenAI to a Departing Employee: You mentioned wanting some guidance on the implications of the release agreement. To reiterate what [redacted] shared- I think it would be helpful to chat live to cover your questions. All employees sign these exit docs. We are not attempting to do anything different or special to you simply because you went to a competitor. We want to make sure you understand that if you don’t sign, it could impact your equity. That’s true for everyone, and we’re just doing things by the book.

Best regards, [redacted].

Kelsey Piper: (The person who wrote and signed the above email is, according to the state bar association of California, a licensed attorney admitted to the state bar.)

To be clear, here was the request which got this response:

Original Email: Hi [redacted[.

Sorry to be a bother about this again but would it be possible to have another week to look over the paperwork, giving me the two weeks I originally requested? I still feel like I don’t fully understand the implications of the agreement without obtaining my own legal advice, and as I’ve never had to find legal advice before this has taken time for me to obtain.

Kelsey Piper: The employee did not ask for ‘guidance’! The employee asked for time to get his own representation!

Leah Libresco Sargeant: Not. Consistently. Candid. OpenAI not only threatened to strip departing employees of equity if they didn’t sign an over broad NDA, they offered these terms as an exploding 7-day termination contract.

This was not a misunderstanding.

Kelsey Piper has done excellent work, and kudos to her sources for speaking up.

If you can’t be trusted with basic employment ethics and law, how can you steward AI?

I had the opportunity to talk to someone whose job involves writing up and executing employment agreements of the type used here by OpenAI. They reached out, before knowing about Kelsey Piper’s article, specifically because they wanted to make the case that what OpenAI did was mostly standard practice. They generally attempted, prior to reading that article, to make the claim that what OpenAI did was within the realm of acceptable practice. If you get equity you should expect to sign a non-disparagement clause, and they explicitly said they would be surprised if Anthropic was not doing it as well.

They did not think that ‘release of claims’ being then interpreted by OpenAI as ‘you can never say anything bad about us ever for any reason or tell anyone that you agreed to this’ was also fair game.

Their argument was that if you sign something like that without talking to a lawyer first that is on you. You have opened the door to any clause. Never mind what happens when you raise objections and consult lawyers during onboarding at a place like OpenAI, it would be unheard of for a company to treat that as a red flag or rescind your offer.

That is very much a corporate lawyer’s view of what is wise and unwise paranoia, and what is and is not acceptable practice.

Even that lawyer said that a 7 day exploding period was highly unusual, and that it was seriously not fine. A 21 day exploding period is not atypical for an exploding contract in general, but that gives time for a lawyer to be consulted. Confining to a week is seriously messed up.

It also is not what the original contract said, which was that you had 60 days. As Kelsey Piper points out, no you cannot spring a 7 day period on someone when the original contract said 60.

Nor was it a threat they honored when called on it, they always extended, with this as an example:

From OpenAI: The General Release and Separation Agreement requires your signature within 7 days from your notification date. The 7 days stated in the General Release supersedes the 60 day signature timeline noted in your separation letter.

That being said, in this case, we will grant an exception for an additional week to review. I’ll cancel the existing Ironclad paperwork, and re-issue it to you with the new date.

Best.

[Redacted at OpenAI.com]

Eliezer Yudkowsky:

And they very clearly tried to discourage ex-employees from consulting a lawyer.

Even if all of it is technically legal, there is no version of this that isn’t scummy as hell.

Control over tender offers means that ultimately anyone with OpenAI equity, who wants to use that equity for anything any time soon (or before AGI comes around) is going to need OpenAI’s permission. OpenAI very intentionally makes that conditional, and holds it over everyone as a threat.

When employees pushed back on the threat to cancel their equity, Kelsey Piper reports that OpenAI instead changed to threatening to withhold participation in future tenders. Without participation in tenders, shares cannot be sold, making them of limited practical value. OpenAI is unlikely to pay dividends for a long time.

If you have any vested Units and you do not sign the exit documents, including the General Release, as required by company policy, it is important to understand that, among other things, you will not be eligible to participate in future tender events or other liquidity opportunities that we may sponsor or facilitate as a private company.

Among other things, a condition to participate in such opportunities is that you are in compliance with the LLC Agreement, the Aestas LLC Agreement, the Unit Grant Agreement and all applicable company policies, as determined by OpenAI.

In other words, if you ever violate any ‘applicable company policies,’ or realistically if you do anything we sufficiently like, or we want to retain our leverage over you, we won’t let you sell your shares.

This makes sense, given the original threat is on shaky legal ground and actually invoking it would give the game away even if OpenAI won.

Kelsey Piper: OpenAI’s original tactic – claiming that since you have to sign a general release, they can put whatever they want in the general release – is on legally shaky ground, to put it mildly. I spoke to five legal experts for this story and several were skeptical it would hold up.

But the new tactic might be on more solid legal ground. That’s because the incorporation documents for Aestas LLC – the holding company that handles equity for employees, investors, + the OpenAI nonprofit entity – are written to give OpenAI extraordinary latitude. (Vox has released this document too.)

And while Altman did not sign the termination agreements, he did sign the Aestas LLC documents that lay out this secondary legal avenue to coerce ex-employees. Altman has said that language about potentially clawing back vested equity from former employees “should never have been something we had in any documents or communication”.

No matter what other leverage they are giving up under pressure, the ace stays put.

Kelsey Piper: I asked OpenAI if they were willing to commit that no one will be denied access to tender offers because of failing to sign an NDA. The company said ““Historically, former employees have been eligible to sell at the same price regardless of where they work; we don’t expect that to change.”

‘Regardless of where they work’ is very much not ‘regardless of what they have signed’ or ‘whether they are playing nice with OpenAI.’ If they wanted to send a different impression, they could have done that.

David Manheim: Question for Sam Altman: Does OpenAI have non-disparagement agreements with board members or former board members?

If so, is Sam Altman willing to publicly release the text of any such agreements?

The answer to that is, presumably, the article in the Economist by Helen Toner and Tasha McCauley, former AI board members. Helen says they mostly wrote this before the events of the last few weeks, which checks with what I know about deadlines.

The content is not the friendliest, but unfortunately, even now, the statements continue to be non-specific. Toner and McCauley sure seem like they are holding back.

The board’s ability to uphold the company’s mission had become increasingly constrained due to long-standing patterns of behaviour exhibited by Mr Altman, which, among other things, we believe undermined the board’s oversight of key decisions and internal safety protocols.

Multiple senior leaders had privately shared grave concerns with the board, saying they believed that Mr Altman cultivated “a toxic culture of lying” and engaged in “behaviour [that] can be characterised as psychological abuse”.

…

The question of whether such behaviour should generally “mandate removal” of a ceo is a discussion for another time. But in OpenAI’s specific case, given the board’s duty to provide independent oversight and protect the company’s public-interest mission, we stand by the board’s action to dismiss Mr Altman.

…

Our particular story offers the broader lesson that society must not let the roll-out of ai be controlled solely by private tech companies.

We also know they are holding back because there are specific things we can be confident happened that informed the board’s actions, that are not mentioned here. For details, see my previous write-ups of what happened.

To state the obvious, if you stand by your decision to remove Altman, you should not allow him to return. When that happened, you were two of the four board members.

It is certainly a reasonable position to say that the reaction to Altman’s removal, given the way it was handled, meant that the decision to attempt to remove him was in error. Do not come at the king if you are going to miss, or the damage to the kingdom would be too great.

But then you don’t stand by it. What one could reasonably say is, if we still had the old board, and all of this new information came to light on top of what was already known, and there was no pending tender offer, and you had your communications ducks in a row, then you would absolutely fire Altman.

Indeed, it would be a highly reasonable decision, now, for the new board to fire Altman a second time based on all this, with better communications and its new gravitas. That is now up to the new board.

OpenAI famously promised 20% of its currently secured compute for its superalignment efforts. That was not a lot of their expected compute budget given growth in compute, but it sounded damn good, and was substantial in practice.

Fortune magazine reports that OpenAI never delivered the promised compute.

This is a big deal.

OpenAI made one loud, costly and highly public explicit commitment to real safety.

That promise was a lie.

You could argue that ‘the claim was subject to interpretation’ in terms of what 20% meant or that it was free to mostly be given out in year four, but I think this is Obvious Nonsense.

It was very clearly either within their power to honor that commitment, or they knew at the time of the commitment that they could not honor it.

OpenAI has not admitted that they did this, offered an explanation, or promised to make it right. They have provided no alternative means of working towards the goal.

This was certainly one topic on which Sam Altman was, shall we say, ‘not consistently candid.’

Indeed, we now know many things the board could have pointed to on that, in addition to any issues involving Altman’s attempts to take control of the board.

This is a consistent pattern of deception.

The obvious question is: Why? Why make a commitment like this then dishonor it?

Who is going to be impressed by the initial statement, and not then realize what happened when you broke the deal?

Kelsey Piper: It seems genuinely bizarre to me to make a public commitment that you’ll offer 20% of compute to Superalignment and then not do it. It’s not a good public commitment from a PR perspective – the only people who care at all are insiders who will totally check if you follow through.

It’s just an unforced error to make the promise at all if you might not wanna actually do it. Without the promise, “we didn’t get enough compute” sounds like normal intra-company rivalry over priorities, which no one else cares about.

Andrew Rettek: this makes sense if the promiser expects the non disparagement agreement to work…

Kelsey Piper (other subthread): Right, but “make a promise, refuse to clarify what you mean by it, don’t actually do it under any reasonable interpretations” seems like a bad plan regardless. I guess maybe they hoped to get people to shut up for three years hoping the compute would come in the fourth?

Indeed, if you think no one can check or will find out, then it could be a good move. You make promises you can’t keep, then alter the deal and tell people to pray you do not alter it any further.

That’s why all the legal restrictions on talking are so important. Not this fact in particular, but that one’s actions and communications change radically when you believe you can bully everyone into not talking.

Even Roon, he of ‘Sam Altman did nothing wrong’ in most contexts, realizes those NDA and non disparagement agreements are messed up.

Roon: NDAs that disallow you to mention the NDA seem like a powerful kind of antimemetic magic spell with dangerous properties for both parties.

That allow strange bubbles and energetic buildups that would otherwise not exist under the light of day.

Read closely, am I trying to excuse evil? I’m trying to root cause it.

It’s clear OpenAI fucked up massively, the mea culpas are warranted, I think they will make it right. There will be a lot of self reflection,

It is the last two sentences where we disagree. I sincerely hope I am wrong there.

Prerat: Everyone should have a canary page on their website that says “I’m not under a secret NDA that I can’t even mention exists” and then if you have to sign one you take down the page.

Stella Biderman: OpenAI is really good at coercing people into signing agreements and then banning them from talking about the agreement at all. I know many people in the OSS community that got bullied into signing such things as well, for example because they were the recipients of leaks.

The Washington Post reported a particular way they did not mess with her.

When OpenAI issued a casting call last May for a secret project to endow OpenAI’s popular ChatGPT with a human voice, the flier had several requests: The actors should be nonunion. They should sound between 25 and 45 years old. And their voices should be “warm, engaging [and] charismatic.”

One thing the artificial intelligence company didn’t request, according to interviews with multiple people involved in the process and documents shared by OpenAI in response to questions from The Washington Post: a clone of actress Scarlett Johansson.

…

The agent [for Sky], who spoke on the condition of anonymity, citing the safety of her client, said the actress confirmed that neither Johansson nor the movie “Her” were ever mentioned by OpenAI.

…

But Mark Humphrey, a partner and intellectual property lawyer at Mitchell, Silberberg and Knupp, said any potential jury probably would have to assess whether Sky’s voice is identifiable as Johansson’s.

…

To Jang, who spent countless hours listening to the actress and keeps in touch with the human actors behind the voices, Sky sounds nothing like Johansson, although the two share a breathiness and huskiness.

The story also has some details about ‘building the personality’ of ChatGPT for voice and hardcoding in some particular responses, such as if it was asked to be the user’s girlfriend.

Jang no doubt can differentiate Sky and Johansson under the ‘pictures of Joe Biden eating sandwiches’ rule, after spending months on this. Of course you can find differences. But to say that the two sound nothing alike is absurd, especially when so many people doubtless told her otherwise.

As I covered last time, if you do a casting call for 400 voice actors who are between 25 and 45, and pick the one most naturally similar to your target, that is already quite a lot of selection. No, they likely did not explicitly tell Sky’s voice actress to imitate anyone, and it is plausible she did not do it on her own either. Perhaps this really is her straight up natural voice. That doesn’t mean they didn’t look for and find a deeply similar voice.

Even if we take everyone in that post’s word for all of that, that would not mean, in the full context, that they are off the hook, based on my legal understanding, or my view of the ethics. I strongly disagree with those who say we ‘owe OpenAI an apology,’ unless at minimum we specifically accused OpenAI of the things OpenAI is reported as not doing.

Remember, in addition to all the ways we know OpenAI tried to get or evoke Scarlett Johansson, OpenAI had a policy explicitly saying that voices should be checked for similarity against major celebrities, and they have said highly implausible things repeatedly on this subject.

Gretchen Krueger resigned from OpenAI on May 14th, and thanks to OpenAI’s new policies, she can say some things. So she does, pointing out that OpenAI’s failures to take responsibility run the full gamot.

Gretchen Krueger: I gave my notice to OpenAI on May 14th. I admire and adore my teammates, feel the stakes of the work I am stepping away from, and my manager Miles Brundage has given me mentorship and opportunities of a lifetime here. This was not an easy decision to make.

I resigned a few hours before hearing the news about Ilya Sutskever and Jan Leike, and I made my decision independently. I share their concerns.

I also have additional and overlapping concerns.

We need to do more to improve foundational things like decision-making processes; accountability; transparency; documentation; policy enforcement; the care with which we use our own technology; and mitigations for impacts on inequality, rights, and the environment.

These concerns are important to people and communities now. They influence how aspects of the future can be charted, and by whom. I want to underline that these concerns as well as those shared by others should not be misread as narrow, speculative, or disconnected. They are not.

One of the ways tech companies in general can disempower those seeking to hold them accountable is to sow division among those raising concerns or challenging their power. I care deeply about preventing this.

I am grateful I have had the ability and support to do so, not least due to David Kokotajlo’s courage. I appreciate that there are many people who are not as able to do so, across the industry.

There is still such important work being led at OpenAI, from work on democratic inputs, expanding access, preparedness framework development, confidence building measures, to work tackling the concerns I raised. I remain excited about and invested in this work and its success.

The responsibility issues extend well beyond superalignment.

A pattern in such situations is telling different stories to different people. Each of the stories is individually plausible, but they can’t live in the same world.

Ozzie Gooen explains the OpenAI version of this, here in EA Forum format (the below is a combination of both):

Ozzie Gooen: On OpenAl’s messaging:

Some arguments that OpenAl is making, simultaneously:

1. OpenAl will likely reach and own transformative Al (useful for attracting talent to work there).

2. OpenAl cares a lot about safety (good for public PR and government regulations).

3. OpenAl isn’t making anything dangerous and is unlikely to do so in the future (good for public PR and government regulations).

4. OpenAl doesn’t need to spend many resources on safety, and implementing safe Al won’t put it at any competitive disadvantage (important for investors who own most of the company).

5. Transformative Al will be incredibly valuable for all of humanity in the long term (for public PR and developers).

6. People at OpenAl have thought long and hard about what will happen, and it will be fine.

7. We can’t predict concretely what transformative Al will look like or what will happen after (Note: Any specific scenario they propose would upset a lot of people. Value hand-waving upsets fewer people).

8. OpenAl can be held accountable to the public because it has a capable board of advisors overseeing Sam Altman (he said this explicitly in an interview).

9. The previous board scuffle was a one-time random event that was a very minor deal.

10. OpenAl has a nonprofit structure that provides an unusual focus on public welfare.

11. The nonprofit structure of OpenAl won’t inconvenience its business prospects or shareholders in any way.

12. The name “OpenAl,” which clearly comes from the early days when the mission was actually to make open-source Al, is an equally good name for where the company is now. (I don’t actually care about this, but find it telling that the company doubles down on arguing the name still is applicable).

So they need to simultaneously say:

13. “We’re making something that will dominate the global economy and outperform humans at all capabilities, including military capabilities, but is not a threat.”

14. “Our experimental work is highly safe, but in a way that won’t actually cost us anything.” “We’re sure that the long-term future of transformative change will be beneficial, even though none of us can know or outline specific details of what that might actually look like.”

15. “We have a great board of advisors that provide accountability. Sure, a few months ago, the board tried to fire Sam, and Sam was able to overpower them within two weeks, but next time will be different.”

16. “We have all of the benefits of being a nonprofit, but we don’t have any of the costs of being a nonprofit.”

Meta’s messaging is clearer: “Al development won’t get us to transformative Al, we don’t think that Al safety will make a difference, we’re just going to optimize for profitability.”

Anthropic’s messaging is a bit clearer. “We think that Al development is a huge deal and correspondingly scary, and we’re taking a costlier approach accordingly, though not too costly such that we’d be irrelevant.” This still requires a strange and narrow worldview to make sense, but it’s still more coherent.

But OpenAl’s messaging has turned into a particularly tangled mess of conflicting promises. It’s the kind of political strategy that can work for a while, especially if you can have most of your conversations in private, but is really hard to pull off when you’re highly public and facing multiple strong competitive pressures.

If I were a journalist interviewing Sam Altman, I’d try to spend as much of it as possible just pinning him down on these countervailing promises they’re making. Some types of questions I’d like him to answer would include:

17. “Please lay out a specific, year-by-year, story of one specific scenario you can imagine in the next 20 years.”

18. “You say that you care deeply about long-term Al safety. What percentage of your workforce is solely dedicated to long-term Al safety?”

19. “You say that you think that globally safe AGI deployments require international coordination to go well. That coordination is happening slowly. Do your plans work conditional on international coordination failing? Explain what your plans would be.”

20. “What do the current prediction markets and top academics say will happen as a result of OpenAl’s work? Which clusters of these agree with your expectations?”

21. “Can you lay out any story at all for why we should now expect the board to do a decent job overseeing you?”

What Sam likes to do in interviews, like many public figures, is to shift specific questions into vague generalities and value statements. A great journalist would fight this, force him to say nothing but specifics, and then just have the interview end.

I think that reasonable readers should, and are, quickly learning to just stop listening to this messaging. Most organizational messaging is often dishonest but at least not self-rejecting. Sam’s been unusually good at seeming genuine, but at this point, the set of incoherent promises is too baffling to take seriously.

Instead, the thing to do is just ignore the noise. Look at the actual actions taken alone. And those actions seem pretty straightforward to me. OpenAl is taking the actions you’d expect from any conventional high-growth tech startup. From its actions, it comes across a lot like: “We think Al is a high-growth area that’s not actually that scary. It’s transformative in a way similar to Google and not the Industrial Revolution. We need to solely focus on developing a large moat (i.e. monopoly) in a competitive ecosystem, like other startups do.”

OpenAl really seems almost exactly like a traditional high-growth tech startup now, to me. The main unusual things about it are the facts that (A) it’s in an area that some people (not the OpenAl management) think is very usually high-risk, (B) its messaging is unusually lofty and conflicting, even for a Silicon Valley startup, and (C) it started out under an unusual nonprofit setup, which now barely seems relevant.

I think that reasonable readers should, and are, quickly learning to just stop listening to this messaging. Most organizational messaging is often dishonest but at least not self-rejecting. Sam’s been unusually good at seeming genuine, but at this point, the set of incoherent promises seems too baffling to take literally.

Instead, I think the thing to do is just ignore the noise. Look at the actual actions taken alone. And those actions seem pretty straightforward to me. OpenAI is taking the actions you’d expect from any conventional high-growth tech startup. From its actions, it comes across a lot like:

“We think AI is a high-growth area that’s not actually that scary. It’s transformative in a way similar to Google and not the Industrial Revolution. We need to solely focus on developing a large moat (i.e. monopoly) in a competitive ecosystem, like other startups do.”

OpenAI really seems almost exactly like a traditional high-growth tech startup now, to me. The main unusual things about it are the facts that: 

1.  Its in an area that some people (not the OpenAI management) think is unusually high-risk,

2. Its messaging is unusually lofty and conflicting, even for a Silicon Valley startup, and

3. It started out under an unusual nonprofit setup, which now barely seems relevant.

Ben Henry: Great post. I believe he also has said words to the effect of:

22. Working on algorithmic improvements is good to prevent hardware overhang.

23. We need to invest more in hardware.

A survey was done. You can judge for yourself whether or not this presentation was fair.

Thus, this question overestimates the impact, as it comes right after telling people such facts about OpenAI:

As usual, none of this means the public actually cares. ‘Increases the case for’ does not mean increases it enough to notice.

Individuals paying attention are often… less kind.

Here are some highlights.

Brian Merchant: “Open” AI is now a company that:

-keeps all of its training data and key operations secret

-forced employees to sign powerful NDAs or forfeit equity

-won’t say whether it trained its video generator on YouTube

-lies to movie stars then lies about the lies

“Open.” What a farce.

[links to two past articles of his discussing OpenAI unkindly.]

Ravi Parikh: If a company is caught doing multiple stupid & egregious things for very little gain

It probably means the underlying culture that produced these decisions is broken.

And there are dozens of other things you haven’t found out about yet.

Jonathan Mannhart (reacting primarily to the Scarlett Johansson incident, but centrally to the pattern of behavior): I’m calling it & ramping up my level of directness and anger (again):

OpenAI, as an organisation (and Sam Altman in particular) are often just lying. Obviously and consistently so.

This is incredible, because it’s absurdly stupid. And often clearly highly unethical.

Joe Weisenthal: I don’t have any real opinions on AI, AGI, OpenAI, etc. Gonna leave that to the experts.

But just from the outside, Sam Altman doesn’t ~seem~ like a guy who’s, you know, doing the new Manhattan Project. At least from the tweets, podcasts etc. Seems like a guy running a tech co.

Andrew Rettek: Everyone is looking at this in the context of AI safety, but it would be a huge story if any $80bn+ company was behaving this way.

Danny Page: This thread is important and drives home just how much the leadership at OpenAI loves to lie to employees and to the public at large when challenged.

Seth Burn: Just absolutely showing out this week. OpenAI is like one of those videogame bosses who looks human at first, but then is revealed to be a horrific monster after taking enough damage.

0.005 Seconds: Another notch in the “Altman lies likes he breathes” column.

Ed Zitron: This is absolutely merciless, beautifully dedicated reporting, OpenAI is a disgrace and Sam Altman is a complete liar.

Keller Scholl: If you thought OpenAI looked bad last time, it was just the first stage. They made all the denials you expect from a company that is not consistently candid: Piper just released the documents showing that they lied.

Paul Crowley: An argument I’ve heard in defence of Sam Altman: given how evil these contracts are, discovery and a storm of condemnation was practically inevitable. Since he is a smart and strategic guy, he would never have set himself up for this disaster on purpose, so he can’t have known.

Ronny Fernandez: What absolute moral cowards, pretending they got confused and didn’t know what they were doing. This is totally failing to take any responsibility. Don’t apologize for the “ambiguity”, apologize for trying to silence people by holding their compensation hostage.

I have, globally, severely downweighted arguments of the form ‘X would never do Y, X is smart and doing Y would have been stupid.’ Fool me [quite a lot of times], and such.

Eliezer Yudkowsky: Departing MIRI employees are forced to sign a disparagement agreement, which allows us to require them to say unflattering things about us up to three times per year. If they don’t, they lose their OpenAI equity.

Rohit: Thank you for doing this.

Rohit quotes himself from several days prior: OpenAI should just add a disparagement clause to the leaver documentation. You can’t get your money unless you say something bad about them.

There is of course an actually better way, if OpenAI wants to pursue that. Unless things are actually much worse than they appear, all of this can still be turned around.

OpenAI says it should be held to a higher standard, given what it sets out to build. Instead, it fails to meet the standards one would set for a typical Silicon Valley business. Should you consider working there anyway, to be near the action? So you can influence their culture?

Let us first consider the AI safety case, and assume you can get a job doing safety work. Does Daniel Kokotajlo make an argument for entering the belly of the beast?

Michael Trazzi:

> be daniel kokotajlo

> discover that AGI is imminent

> post short timeline scenarios

> entire world is shocked

> go to OpenAI to check timelines

> find out you were correct

> job done, leave OpenAI

> give up 85% of net worth to be able to criticize OpenAI

> you’re actually the first one to refuse signing the exit contract

> inadvertently shatter sam altman’s mandate of heaven

> timelines actually become slightly longer as a consequence

> first time in your life you need to update your timelines, and the reason they changed is because the world sees you as a hero

Stefan Schubert: Notable that one of the (necessary) steps there was “join OpenAI”; a move some of those who now praise him would criticise.

There are more relevant factors, but from an outside view perspective there’s some logic to the notion that you can influence more from the centre of things.

Joern Stoehler: Yep. From 1.5y to 1w ago, I didn’t buy arguments of the form that having people who care deeply about safety at OpenAI would help hold OpenAI accountable. I didn’t expect that joining-then-leaving would bring up legible evidence for how OpenAI management is failing its goal.

Even better, Daniel then get to keep his equity, whether or not OpenAI lets him sell it. My presumption is they will let him given the circumstances, I’ve created a market.

Most people who attempt this lack Daniel’s moral courage. The whole reason Daniel made a difference is that Daniel was the first person who refused to sign, and was willing to speak about it.

Do not assume you will be that courageous when the time comes, under both bribes and also threats, explicit and implicit, potentially both legal and illegal.

Similarly, your baseline assumption should be that you will be heavily impacted by the people with whom you work, and the culture of the workplace, and the money being dangled in front of you. You will feel the rebukes every time you disrupt the vibe, the smiles when you play along. Assume that when you dance with the devil, the devil don’t change. The devil changes you.

You will say ‘I have to play along, or they will shut me out of decisions, and I won’t have the impact I want.’ Then you never stop playing along.

The work you do will be used to advance OpenAI’s capabilities, even if it is nominally safety. It will be used for safety washing, if that is a plausible thing, and your presence for reputation management and recruitment.

Could you be the exception? You could. But you probably won’t be.

In general, ‘if I do not do the bad thing then someone else will do the bad thing and it will go worse’ is a poor principle.

Do not lend your strength to that which you wish to be free from.

What about ‘building career capital’? What about purely in your own self-interest? What if you think all these safety concerns are massively overblown?

Even there, I would caution against working at OpenAI.

That giant equity package? An albatross around your neck, used to threaten you. Even if you fully play ball, who knows when you will be allowed to cash it in. If you know things, they have every reason to not let you, no matter if you so far have played ball.

The working conditions? The nature of upper management? The culture you are stepping into? The signs are not good, on any level. You will hold none of the cards.

If you already work there, consider whether you want to keep doing that.

Also consider what you might do to gather better information, about how bad the situation has gotten, and whether it is a place you want to keep working, and what information the public might need to know. Consider demanding change in how things are run, including in the ways that matter personally to you. Also ask how the place is changing you, and whether you want to be the person you will become.

As always, everyone should think for themselves, learn what they can, start from what they actually believe about the world and make their own decisions on what is best. As an insider or potential insider, you know things outsiders do not know. Your situation is unique. You hopefully know more about who you would be working with and under what conditions, and on what projects, and so on.

What I do know is, if you can get a job at OpenAI, you can get a lot of other jobs too.

As you can see throughout, Kelsey Piper is bringing the fire.

There is no doubt more fire left to bring.

Kelsey Piper: I’m looking into business practices at OpenAI and if you are an employee or former employee or have a tip about OpenAI or its leadership team, you can reach me at kelsey.piper@vox.com or on Signal at 303-261-2769.

If you have information you want to share, on any level of confidentiality, you can also reach out to me. This includes those who want to explain to me why the situation is far better than it appears. If that is true I want to know about it.

There is also the matter of legal representation for employees and former employees.

What OpenAI did to its employees is, at minimum, legally questionable. Anyone involved should better know their rights even if they take no action. There are people willing to pay your legal fees, if you are impacted, to allow you to consult a lawyer.

Kelsey Piper: If [you have been coerced into signing agreements you cannot talk about], please talk to me. I’m on Signal at 303-261-2769. There are people who have come to me offering to pay your legal fees.

Here Vilfredo’s Ghost, a lawyer, notes that a valid contract requires consideration and a ‘meeting of the minds,’ and common law contract principles do not permit surprises. Since what OpenAI demanded is not part of a typical ‘general release,’ and the only consideration provided was ‘we won’t confiscate your equity’ or deny you the right to sell it, the contract looks suspiciously like it would be invalid.

Matt Bruenig has a track record of challenging the legality of similar clauses, and has offered his services. He notes that rules against speaking out about working conditions are illegal under federal law, but if they do not connect to ‘working conditions’ then they are legal. Our laws are very strange.

It seems increasingly plausible that it would be in the public interest to ban non-disparagement clauses more generally going forward, or at least set limits on scope and length (although I think nullifying existing contracts is bad and the government should not do that, and shouldn’t have done it for non-competes either.)

This is distinct from non-disclosure in general, which is clearly a tool we need to have. But I do think that, at least outside highly unusual circumstances, ‘non-disclosure agreements should not apply to themselves’ is also worth considering.

Thanks to the leverage OpenAI still holds, we do not know what other information is out there, as of yet not brought to light.

Repeatedly, OpenAI has said it should be held to a higher standard.

OpenAI instead under Sam Altman has consistently failed to live up not only to the standards to which one must hold a company building AGI, but also the standards one would hold an ordinary corporation. Its unique non-profit structure has proven irrelevant in practice, if this is insufficient for the new board to fire Altman.

This goes beyond existential safety. Potential and current employees and business partners should reconsider, if only for their own interests. If you are trusting OpenAI in any way, or its statements, ask whether that makes sense for you and your business.

Going forward, I will be reacting to OpenAI accordingly.

If that’s not right? Prove me wrong, kids. Prove me wrong.

Previously: OpenAI: Facts From a Weekend, OpenAI: The Battle of the Board, OpenAI: Leaks Confirm the Story, OpenAI: Altman Returns, OpenAI: The Board Expands.

Ilya Sutskever and Jan Leike have left OpenAI. This is almost exactly six months after Altman’s temporary firing and The Battle of the Board, the day after the release of GPT-4o, and soon after a number of other recent safety-related OpenAI departures. Many others working on safety have also left recently. This is part of a longstanding pattern at OpenAI.

Jan Leike later offered an explanation for his decision on Twitter. Leike asserts that OpenAI has lost the mission on safety and culturally been increasingly hostile to it. He says the superalignment team was starved for resources, with its public explicit compute commitments dishonored, and that safety has been neglected on a widespread basis, not only superalignment but also including addressing the safety needs of the GPT-5 generation of models.

Altman acknowledged there was much work to do on the safety front. Altman and Brockman then offered a longer response that seemed to say exactly nothing new.

Then we learned that OpenAI has systematically misled and then threatened its departing employees, forcing them to sign draconian lifetime non-disparagement agreements, which they are forbidden to reveal due to their NDA.

Altman has to some extent acknowledged this and promised to fix it once the allegations became well known, but so far there has been no fix implemented beyond an offer to contact him privately for relief.

These events all seem highly related.

Also these events seem quite bad.

What is going on?

This post walks through recent events and informed reactions to them. 

The first ten sections address departures from OpenAI, especially Sutskever and Leike.

The next five sections address the NDAs and non-disparagement agreements.

Then at the end I offer my perspective, highlight another, and look to paths forward. 

1. The Two Departure Announcements

2. Who Else Has Left Recently?

3. Who Else Has Left Overall?

4. Early Reactions to the Departures

5. The Obvious Explanation: Altman

6. Jan Leike Speaks

7. Reactions After Lekie’s Statement

8. Greg Brockman and Sam Altman Respond to Leike

9. Reactions from Some Folks Unworried About Highly Capable AI

10. Don’t Worry, Be Happy?

11. The Non-Disparagement and NDA Clauses

12. Legality in Practice

13. Implications and Reference Classes

14. Altman Responds on Non-Disparagement Clauses

15. So, About That Response

16. How Bad Is All This?

17. Those Who Are Against These Efforts to Prevent AI From Killing Everyone

18. What Will Happen Now?

19. What Else Might Happen or Needs to Happen Now?

Here are the full announcements and top-level internal statements made on Twitter around the departures of Ilya Sutskever and Jan Leike.

Ilya Sutskever: After almost a decade, I have made the decision to leave OpenAI. The company’s trajectory has been nothing short of miraculous, and I’m confident that OpenAI will build AGI that is both safe and beneficial under the leadership of @sama, @gdb, @miramurati and now, under the excellent research leadership of Jakub Pachocki. It was an honor and a privilege to have worked together, and I will miss everyone dearly. So long, and thanks for everything. I am excited for what comes next — a project that is very personally meaningful to me about which I will share details in due time.

[Ilya then shared the photo below]

Jakub Pachocki: Ilya introduced me to the world of deep learning research, and has been a mentor to me, and a great collaborator for many years. His incredible vision for what deep learning could become was foundational to what OpenAI, and the field of AI, is today. I am deeply grateful to him for our countless conversations, from high-level discussions about the future of AI progress, to deeply technical whiteboarding sessions.

Ilya – I will miss working with you.

Sam Altman: Ilya and OpenAI are going to part ways. This is very sad to me; Ilya is easily one of the greatest minds of our generation, a guiding light of our field, and a dear friend. His brilliance and vision are well known; his warmth and compassion are less well known but no less important.

OpenAI would not be what it is without him. Although he has something personally meaningful he is going to go work on, I am forever grateful for what he did here and committed to finishing the mission we started together. I am happy that for so long I got to be close to such genuinely remarkable genius, and someone so focused on getting to the best future for humanity.

Jakub is going to be our new Chief Scientist. Jakub is also easily one of the greatest minds of our generation; I am thrilled he is taking the baton here. He has run many of our most important projects, and I am very confident he will lead us to make rapid and safe progress towards our mission of ensuring that AGI benefits everyone.

Greg Brockman: I have immense gratitude to Ilya for being my co-founder, my friend, and the officiant at my civil ceremony.

Together, we charted the path of what OpenAI would become today. When we started in late 2015, OpenAI was a non-profit with a mission to make AGI go well but without a credible plan of how to accomplish it. In the early days, Ilya and I spent countless hours thinking hard about every aspect of culture, technical direction, and strategy. Together we realized that we would need to raise a lot more capital than anyone had imagined in order to build supercomputers of unprecedented size. We fundraised together in the non-profit, raising more than others thought possible but still far less than what was needed. We proposed creating a for-profit structure in service of the mission. And once that had been created, we continued to align and shape what that company stood for, maintaining the focus on our AGI mission while grappling with the hard practical questions of how to make progress each day.

Ilya is an artist. His vision and gusto are infectious, and he helped me understand this field when I was just getting started. He is unafraid of thinking through the logical conclusion of his intuitions. We were motivated by the 1962 book Profiles of the Future, which opens with descriptions of the incorrect mindsets that led to scientific claims of the impossibility of the lightbulb, flight, and reaching orbit shortly before these feats were accomplished. So despite people doubting that AGI was anywhere on the foreseeable horizon, we would think through and act on the conviction of our beliefs that deep learning can take us there.

The mission is far from complete, and Ilya played a key role in helping build the foundations of what OpenAI has become today. Thank you for everything.

Jan Leike: I resigned.

Jan Leike later offered a full Twitter thread, which I analyze in detail later.

Wei Dai (November 21, 2023): The OpenAI Cultural Revolution

If you asked me last week whose departures other than Sam Altman himself or a board member would update me most negatively about the likelihood OpenAI would responsibly handle the creation and deployment of AGI, I would definitely have said Ilya Sutskever and Jan Leike.

If you had asked me what piece of news about OpenAI’s employees would have updated me most positively, I would have said ‘Ilya Sutskever makes it clear he is fully back and is resuming his work in-office as head of the Superalignment team, and he has all the resources he needs and is making new hires.’

If Jan’s and Ilya’s departures were isolated, that would be bad enough. But they are part of a larger pattern.

Here is Shakeel’s list of safety researchers at OpenAI known to have left in the last six months, minus Cullen O’Keefe who worked on policy and legal (so was not a clear cut case of working on safety), plus the addition of Ryan Lowe.

1. Ilya Sutskever

2. Jan Leike

3. Leopold Aschenbrenner

4. Pavel Izmailov

5. William Saunders

6. Daniel Kokotajlo

7. Ryan Lowe

Here’s some other discussion of recent non-safety OpenAIemployee departures.

Shakeel: Other recent departures include Chris Clark, head of nonprofit and strategic initiatives and Sherry Lachman, head of social impact.

Zack Stein-Perlman: Two other executives left two weeks ago, but that’s not obviously safety-related.

Diane Yoon [was] vice president of people, [and Chris Clark as above].

HT LGS: Evan Morikawa also left on the 15th to join his friend Andy Barry at Boston Dynamics but that does not seem related.

Ilya Sutskever was one of the board members that attempted to fire Sam Altman.

Jan Leike worked closely with Ilya to essentially co-lead Superalignment. He has now offered an explanation thread.

William Saunders also worked on Superalignment; he resigned on February 15. He posted this on LessWrong, noting his resignation and some of what he had done at OpenAI, but no explanation. When asked why he quit, he said ‘no comment.’ The logical implications are explored.

Leopold Aschenbrenner and Pavel Izmailov were fired on April 11 for supposedly leaking confidential information. The nature of leaking confidential information is that people are reluctant to talk about exactly what was leaked, so it is possible that OpenAI’s hand was forced. From what claims we do know and what I have read, the breach seemed technical and harmless. OpenAI chose to fire them anyway. In Vox, Sigal Samuel is even more skeptical that this was anything but an excuse. Leopold Aschenbrenner was described as an ally of Ilya Sutskever.

Ryan Lowe ‘has a few projects in the oven’. He also Tweeted the following and as far as I can tell that’s all we seem to know.

Ryan Lowe: I’m so grateful I got to work closely with Jan at OpenAI. he’s an amazing human being.

Cullen O’Keefe left to be Director of Research at the Institute for Law & AI.

Daniel Kokotajlo quit on or before April 18 ‘due to losing confidence that [OpenAI] would behave responsibly around the time of AGI.’ He gave up his equity in OpenAI, constituting 85% of his family’s net worth, to avoid signing a non-disparagement agreement, but he is still under NDA.

We do not have a full enumeration of how many people would have counted for a list like this. Based on this interview with Jan Leike (at about 2: 16: 30) six months ago superalignment was about a 20 person team, and safety outside of it was broad but mostly RLHF and other mundane safety efforts with easy business cases that don’t clash with the company culture. 

Then we lost 7 within 6 months, concentrated on senior leadership. This seems like rather a lot.

Then we can add, within weeks, the head of nonprofit and strategic initiatives, the head of social impact and a vice president of people. That sounds a lot like this goes well beyond potential future safety issues, and goes deep into problems such as general ethical behavior and responsible strategic planning.

Sigal Samuel (Vox, Future Perfect): For months, OpenAI has been losing employees who care deeply about making sure AI is safe. Now, the company is positively hemorrhaging them.

OpenAI has a longstanding habit of losing its top safety-oriented people.

As we all know, OpenAI is nothing without its people.

I asked GPT-4o, Claude Opus and Gemini Advanced to rank the current and former employees of OpenAI by how important they are in terms of AGI safety efforts:

Q: In order ranked by importance to OpenAI’s efforts in the area, who are the top 10 employees (past or current) who have worked on AGI Safety or AGI Alignment in various forms? Consider only as it pertains to the safety of future highly capable systems, not alignment of current systems (e.g. not GPT-4).

Here are the answers, removing explanations. The names in bold seem to no longer work at OpenAI.

Here is GPT-4o:

1. Ilya Sutskever

2. Dario Amodei

3. Paul Christiano

4. Jan Leike

5. Geoffrey Irving

6. Chris Olah

7. Jakob Foerster

8. Alec Radford

9. Ryan Lowe

10. Miles Brundage

Here is Claude Opus:

1. Paul Christiano

2. Jan Leike

3. Dario Amodei

4. Geoffrey Irving

5. Amanda Askell

6. Miles Brundage

7. Jack Clark

8. Ilya Sutskever

9. Sam Altman

10. Chris Olah

Here is Gemini Advanced:

1. Ilya Sutskever

2. Jan Leike

3. Paul Christiano

4. Dario Amodei

5. Chris Olah

6. Jeff Wu

7. Vicki Cheung

8. John Schulman

9. Pieter Abbeel

10. Alec Radford

On average, over 80% of the named people have now departed, including 100% of the top 5 from all lists. This is in addition to what happened to the board including Helen Toner.

Those that remain are CEO Sam Altman, co-founder John Schulman, Alec Radford and Jeff Wu. What do all of them appear to have in common? They do not have obvious ‘safety branding,’ and their primary work appears to focus on other issues. John Schulman does have a co-authored alignment forum post.

Once is a coincidence. Twice is suspicious. Over 80% of the time is enemy action.

Here are various early reactions to the news, before the second wave of information on Friday from Vox, Bloomberg, Leike, and others.

Connor Leahy: Canary in the coal mine.

Congrats to Ilya and Jan for doing the right thing.

Jeffrey Ladish: Really not a good sign. That’s the second gen of safety team leads OpenAI has lost…

David Krueger [QTing Leike]: For a while I’ve been saying that I still know *somepeople at OpenAI understand and care about x-safety.

The list is growing shorter and shorter…

James Campbell: Surely this means superalignment is solved. The job is done. They set out to do it in four years and it only took a mere 10 months. Our boys can come home now.

Metaculus: OpenAI is 3% likely to announce it has solved the core technical challenges of superintelligence alignment by June 30, 2027, down 2% this week.

James Miller: [Leike] knows that not saying something like “It was an honor to have worked at OpenAI” will be interpreted as “I’m under a NDA but I think OpenAI is on track to destroy the universe” and yet he has still given us just these two words [‘I resign’].

Breaching his NDA could increase the chances of humanity going extinct because he would get sued, and lose time and money that he could otherwise spend helping us survive.

John David Pressman: Considering [Leike is] under a strict NDA and the professional standard thing to do is say “It’s been an incredible opportunity I’m excited for what OpenAI will do in the future” and he didn’t say that I’m genuinely concerned.

That he doesn’t break the NDA outright tells me it’s not any form of imminent catastrophic risk. Doesn’t mean it’s not a bad sign about OpenAI from an AI alignment standpoint.

How concerned [from 0-10]? Uh maybe like a 5-6? I’m mostly annoyed with the people going “lolol now the real engineers can get to work” as though this does not in fact look bad for OpenAI. Would love to know more about what’s going on here.

Realistically? Something like “OpenAI no longer takes basic research seriously and the culture is actively toxic if you work on things like weak to strong generalization”. Not “what did Ilya see?” type stuff.

Marvin Baumann (responding to JDP): OpenAI is a product company now (apparently so), no more research. What besides that fact does concern you further?

Andrew Critch: I’m sad to see so many people leaving OpenAI. I’ve really enjoyed their products, and the way they’ve helped humanity come to grips with the advent of LLMs by making them more openly available in their products.

I remain “optimistic” that we probably have only a ~25% chance of AI-driven extinction this decade — and there’s a lot that can be done to change that! — but it’s not a good sign when leadership at AI companies keep splintering apart like this while trying to develop the most important technology of all time.

If there’s anything positive to take from this, maybe this fragmentation process can help wake people up into realizing that no one company should be trusted to control humanity’s future with AI technology, and that we should all be working to democratize and diversify decision-making and deliberation over these incredibly impactful technologies. Many have said this before, and at some point the talk needs to turn into real change.

I’m currently 80% on human extinction by 2060.

Thane Ruthenis: [The departures are] good news.

There was a brief moment, back in 2023, when OpenAI’s actions made me tentatively optimistic that the company was actually taking alignment seriously, even if its model of the problem was broken.

Everything that happened since then has made it clear that this is not the case; that all these big flashy commitments like Superalignment were just safety-washing and virtue signaling. They were only going to do alignment work inasmuch as that didn’t interfere with racing full-speed towards greater capabilities.

So these resignations don’t negatively impact my p(doom) in the obvious way. The alignment people at OpenAI were already powerless to do anything useful regarding changing the company direction.

On the other hand, what these resignations do is showcasing that fact. Inasmuch as Superalignment was a virtue-signaling move meant to paint OpenAI as caring deeply about AI Safety, so many people working on it resigning or getting fired starkly signals the opposite.

And it’s good to have that more in the open; it’s good that OpenAI loses its pretense.

Oh, and it’s also good that OpenAI is losing talented engineers, of course.

Jelle Donders: Guaranteeing all the safety people that left OpenAI that any legal fees for breaking their NDA would be fully compensated might be a very effective intervention.

Tristan Wegner: On first order, this might have a good effect on safety.

On second order, it might have negative effects, because it increases the risk of and therefor lowers the rate of such companies hiring people openly worrying about AI X-Risk.

Linch: It’ll be interesting to see if OpenAI will keep going with their compute commitments now that the two main superalignment leads have left. 

[they didn’t.]

Zack Stein-Perlman: The commitment—”20% of the compute we’ve secured to date” (in July 2023), to be used “over the next four years”—may be quite little in 2027, with compute use increasing exponentially. I’m confused about why people think it’s a big commitment.

Lukas Gloor: It seems likely (though not certain) that they signed non-disparagement agreements, so we may not see more damning statements from them even if that’s how they feel. Also, Ilya at least said some positive things in his leaving announcement, so that indicates either that he caved in to pressure (or too high agreeableness towards former co-workers) or that he’s genuinely not particularly worried about the direction of the company and that he left more because of reasons related to his new project. 

Danbmil99: Putting aside the fact that OpenAI drama seems to always happen in a world-is-watching fishbowl, this feels very much like the pedestrian trope of genius CTO getting sidelined as the product succeeds and business people pushing business interests take control. On his own, Ilya can raise money for anything he wants, hire anyone he wants, and basically just have way more freedom than he does at OpenAI.

I do think there is a basic p/doom vs e/acc divide which has probably been there all along, but as the tech keeps accelerating it becomes more and more of a sticking point.

I suspect in the depths of their souls, SA and Brock and the rest of that crowd do not really take the idea of existential threat to humanity seriously. Giving Ilya a “Safety and alignment” role probably now looks like a sop to A) shut the p-doomers up and B) signal some level of concern. But when push comes to shove, SA and team do what they know how to do — push product out the door. Move fast and risk extinction.

One CEO I worked with summed up his attitude thusly: “Ready… FIRE! – aim.”

Arthur Breitman: The default explanation for high-profile safety people leaving OpenAI is not them being about to unleash existentially risky models but rather a culture and priority shift having taken place, translating in teams not getting headcount or GPU.

It’s still bad though.

Low confidence that it’s the correct explanation, high confidence that it’s close to the best guess outsiders can reasonably make.

Jacques: Not a productive comment (0), yet everyone agrees (57).

Version of that comment on Friday morning, still going:

For symmetry, here’s the opposite situation:

Gary Marcus summarized, suggests ‘his friends in Washington should look into this.’

We know that a lot of OpenAI’s safety researchers, including its top safety researchers, keep leaving. We know that has accelerated in the wake of the attempted firing of Sam Altman.

That does not seem great. Why is it all happening?

At Vox, Sigal Samuel offers a simple explanation. It’s Altman.

Sigal Samuel: But the real answer may have less to do with pessimism about technology and more to do with pessimism about humans — and one human in particular: Altman. According to sources familiar with the company, safety-minded employees have lost faith in him.

“It’s a process of trust collapsing bit by bit, like dominoes falling one by one,” a person with inside knowledge of the company told me, speaking on condition of anonymity.

Not many employees are willing to speak about this publicly. That’s partly because OpenAI is known for getting its workers to sign offboarding agreements with non-disparagement provisions upon leaving. If you refuse to sign one, you give up your equity in the company, which means you potentially lose out on millions of dollars.

…

For employees, all this led to a gradual “loss of belief that when OpenAI says it’s going to do something or says that it values something, that that is actually true,” a source with inside knowledge of the company told me.

That gradual process crescendoed this week.

I want to deeply thank Jan Leike for his explanation of why he resigned.

Here is Jan Leike’s statement, in its entirety:

Jan Leike: Yesterday was my last day as head of alignment, superalignment lead, and executive @OpenAI.

It’s been such a wild journey over the past ~3 years. My team launched the first ever RLHF LLM with InstructGPT, published the first scalable oversight on LLMs, pioneered automated interpretability and weak-to-strong generalization. More exciting stuff is coming out soon.

I love my team.

I’m so grateful for the many amazing people I got to work with, both inside and outside of the superalignment team.

OpenAI has so much exceptionally smart, kind, and effective talent.

Stepping away from this job has been one of the hardest things I have ever done, because we urgently need to figure out how to steer and control AI systems much smarter than us.

I joined because I thought OpenAI would be the best place in the world to do this research.

However, I have been disagreeing with OpenAI leadership about the company’s core priorities for quite some time, until we finally reached a breaking point.

I believe much more of our bandwidth should be spent getting ready for the next generations of models, on security, monitoring, preparedness, safety, adversarial robustness, (super)alignment, confidentiality, societal impact, and related topics.

These problems are quite hard to get right, and I am concerned we aren’t on a trajectory to get there.

Over the past few months my team has been sailing against the wind. Sometimes we were struggling for compute and it was getting harder and harder to get this crucial research done.

Building smarter-than-human machines is an inherently dangerous endeavor.

OpenAI is shouldering an enormous responsibility on behalf of all of humanity.

But over the past years, safety culture and processes have taken a backseat to shiny products.

We are long overdue in getting incredibly serious about the implications of AGI.

We must prioritize preparing for them as best we can.

Only then can we ensure AGI benefits all of humanity.

OpenAI must become a safety-first AGI company.

To all OpenAI employees, I want to say:

Learn to feel the AGI.

Act with the gravitas appropriate for what you’re building.

I believe you can “ship” the cultural change that’s needed.

I am counting on you.

The world is counting on you.

:openai-heart:

This paints a very clear picture, although with conspicuous absence of any reference to Altman. The culture of OpenAI had indeed become toxic, and unwilling to take safety seriously.

This is a deeply polite version of ‘We’re fed.’

Leike’s team was starved for compute, despite the commitments made earlier.

OpenAI was, in his view, severely underinvesting in both Superalignment and also more mundane forms of safety.

Safety culture took a backseat to shiny new products (presumably GPT-4o was one of these).

According to Bloomberg, Ilya’s departure was Jan’s last straw.

TechCrunch confirms that OpenAI failed to honor its compute commitments.

Kyle Wiggers (TechCrunch): OpenAI’s Superalignment team, responsible for developing ways to govern and steer “superintelligent” AI systems, was promised 20% of the company’s compute resources, according to a person from that team. But requests for a fraction of that compute were often denied, blocking the team from doing their work.

Now the Superalignment team has been dissolved.

I presume that OpenAI would not be so brazen as to go after Jan Leike or confiscate his equity in light of this very respectful and restrained statement, especially in light of other recent statements in that area.

It would be very bad news if this turns out to not be true. Again, note that the threat is stronger than its execution.

Roon: i don’t endorse the rest of this thread but yeah [quotes the last Tweet in Jan Leike’s statement, which starts with ‘To all OpenAI employees.’]

Roon is in some ways strategically free and reckless. In other ways, and in times like this he chooses his Exact Words very carefully.

Roon: I ♥️ OpenAI.

The last best hope of navigating the golden path to safe superintelligence.

Everyone hates a centrist.

Pager: Mandate of heaven is lost sir.

Roon: Tell it to the sweet sweet research progress.

Others were less Straussian.

Matt Yglesias: Not ideal when a company’s head safety guy quits because he thinks the company is being too reckless.

I hate the rhetoric around “doomers” but all kinds of useful technologies would be really dangerous and harmful if deployed with neither voluntary prudence nor formal regulation.

Vitalik Buterin: I’m really proud that ethereum does not have any culture of trying to prevent people from speaking their minds, even when they have very negative feelings toward major things in the protocol or ecosystem.

Some wave the ideal of “open discourse” as a flag, some take it seriously.

Matt Shumer: Wow. This is huge. The first time (I’m aware of) that an OpenAI exec has publicly stated that they believe OpenAI is clearly prioritizing capabilities over safety research. Massive implications, in many ways.

Wei Dei: I was thinking about writing an AI Alignment Forum post titled “Top signs your alignment work is being exploited for safety-washing” but somehow that feels less urgent now.

David Chapman: This may be a historically important thread (or not). The head of safety at OpenAI has quit, saying that the company’s leadership is not taking safety seriously enough.

Tetraspace: How could I operationalise “OpenAI will have a notkilleveryoneism team with any influence at all by the end of 202X”?

One read of the situation is Altman and some OpenAI employees have non-tiny probabilities of extinction, and don’t want to die, but OpenAI-the-egregore doesn’t want to hear disappointing news, and Altman’s one trick is hardline corporate manipulator.

So they end up hiring a notkilleveryoneism team, being disappointed by all the bad news they’re hearing, marginalizing them, and then the leadership and main talent of that team leaves in frustration.

In this world, perhaps they try again, with some new governance structure, which makes their NKEism team feel more powerful but able to less of what would offend OpenAI; or perhaps they finally rationalise away this error signal – how silly we were before we learned RLHF worked.

Sarah (Little Ramblings): OpenAI quietly shuts down the effort they established less than a year ago to ensure that their own technology doesn’t literally kill everyone on earth, and prioritises developing said technology faster. It’s days like this what I feel almost frustrated to tears at the fact that this isn’t all anyone in the world is talking about.

Connor Leahy: Props to Jan [Leike] for speaking out and confirming what we already suspected/knew.

From my point of view, of course profit maximizing companies will…maximize profit. It never was even imaginable that these kinds of entities could shoulder such a huge risk responsibly.

And humanity pays the cost.

Altman initially responded with about the most graceful thing he could have said (in a QT). This is The Way provided you follow through.

Sam Altman: I’m super appreciative of @janleike’s contributions to OpenAI’s alignment research and safety culture, and very sad to see him leave. he’s right we have a lot more to do; we are committed to doing it. I’ll have a longer post in the next couple of days.

A few days to process all this and prepare a response is a highly reasonable request.

So what did they come back with?

Here is the full statement.

Greg Brockman and Sam Altman (cosigned): We’re really grateful to Jan for everything he’s done for OpenAI, and we know he’ll continue to contribute to the mission from outside. In light of the questions his departure has raised, we wanted to explain a bit about how we think about our overall strategy.

First, we have raised awareness of the risks and opportunities of AGI so that the world can better prepare for it. We’ve repeatedly demonstrated the incredible possibilities from scaling up deep learning and analyzed their implications; called for international governance of AGI before such calls were popular; and helped pioneer the science of assessing AI systems for catastrophic risks.

Second, we have been putting in place the foundations needed for safe deployment of increasingly capable systems. Figuring out how to make a new technology safe for the first time isn’t easy. For example, our teams did a great deal of work to bring GPT-4 to the world in a safe way, and since then have continuously improved model behavior and abuse monitoring in response to lessons learned from deployment.

Third, the future is going to be harder than the past. We need to keep elevating our safety work to match the stakes of each new model. We adopted our Preparedness Framework last year to help systematize how we do this.

This seems like as good of a time as any to talk about how we view the future.

As models continue to become much more capable, we expect they’ll start being integrated with the world more deeply. Users will increasingly interact with systems — composed of many multimodal models plus tools — which can take actions on their behalf, rather than talking to a single model with just text inputs and outputs.

We think such systems will be incredibly beneficial and helpful to people, and it’ll be possible to deliver them safely, but it’s going to take an enormous amount of foundational work. This includes thoughtfulness around what they’re connected to as they train, solutions to hard problems such as scalable oversight, and other new kinds of safety work. As we build in this direction, we’re not sure yet when we’ll reach our safety bar for releases, and it’s ok if that pushes out release timelines.

We know we can’t imagine every possible future scenario. So we need to have a very tight feedback loop, rigorous testing, careful consideration at every step, world-class security, and harmony of safety and capabilities. We will keep doing safety research targeting different timescales. We are also continuing to collaborate with governments and many stakeholders on safety.

There’s no proven playbook for how to navigate the path to AGI. We think that empirical understanding can help inform the way forward. We believe both in delivering on the tremendous upside and working to mitigate the serious risks; we take our role here very seriously and carefully weigh feedback on our actions.

My initial response was: “I do not see how this contains new information or addresses the concerns that were raised?”

Others went further, and noticed this said very little.

This did indeed feel like that part of Isaac Asimov’s Foundation, where a diplomat visits and everyone thinks he is a buffoon, then after he leaves they use symbolic logic to analyze his statements and realize he managed to say exactly nothing.

So I had a fun conversation where I asked GPT-4o, what in this statement was not known as of your cutoff date? It started off this way:

Then I shared additional previously known information, had it browse the web to look at the announcements around GPT-4o, and asked, for each item it named, whether there was new information.

Everything cancelled out. Link has the full conversation.

And then finally:

Colin Fraser: Right, yeah, like I said.

I happen to know for a fact that OpenAI does invest some amount in keeping its products “safe” in the sense of mitigating abuse and harmful output but I just don’t think they have the resources to float basic research into ill-defined science fiction scenarios.

It does put them in a bit of a pickle though because “the science fiction scenario is real” is a core part of their public messaging. What they need is a Superalignment figurehead to signal that they take it seriously but who won’t demand a billion dollars to play the sims.

I also do think things have fundamentally shifted since ChatGPT blew up. I think if you asked three years ago under truth serum what they’re trying to do it would be “build AGI” but today it would be “sell access to ChatGPT.”

The sad irony is without the early Superalignment research ChatGPT couldn’t exist.

Note the distinction between Colin’s story here, that OpenAI lacks the resources to do basic research, and his previous claim that a culture clash makes it effectively impossible for OpenAI to do such research. Those stories suggest different problems with different solutions. 

‘OpenAI does not have sufficient resources’ seems implausible given their ability to raise capital, and Leike says they’re severely underinvesting in safety even on business grounds over a two year time horizon. A culture clash or political fight fits the facts much better.

Ben Landau-Taylor: Safetyists purging their factional rivals: Haha fuck yeah!!! Yes!! Safetyists being purged by their factional rivals: Well this fucking sucks. What the fuck.

[Quotes himself showing two forum comments with the names redacted saying that if Altman was taking OpenAI in an unsafe direction that would endanger humanity (a point both made clear was a fully conditional statement) then that would be a good reason to fire him.]

So there are outsiders who want work done on safety and many of them think endangering humanity would have been a good justification, if true, for firing the CEO? And that makes it good to purge everyone working on safety? Got it.  

Timothy Lee: I’m not worried about existential risk from AI and didn’t understand what the superalignment team was doing so I wouldn’t say I’m upset about this. But given that @sama purports to be concerned about X-risk, it would be nice to hear from him about it.

Like has he decided that AI isn’t dangerous? Does he still think it was dangerous but the superalignment team had the wrong approach? Did he think it was being badly managed? If he is still worried is he going to take the resources from the old team into some new effort?

Good questions, even if like Timothy you are skeptical of the risks.

How bad can it be if they’re not willing to violate the NDAs, asks Mason.

Mason: People are freaking out about the Ilya/Jan resignations like the obvious thing anyone would do if their company was about to

destroy

humanity

is resign and post cryptic tweets about it.

I’m not saying it’s a nothingburger.

But I am saying that public-but-cryptic resignations are obviously getting these very intelligent guys more bang for their buck than violating their NDAs and I don’t think that’s compatible with the idea that we’re all about to die.

(and yeah, I do think it’s a nothingburger).

This follows in the tradition of people saying versions of:

• If you were truly worried about this you would blow up your life and savings, in this way that I say would make sense, despite all explanations why it doesn’t.

  • You didn’t.

    • So clearly you are not worried.

    • Nothing to worry about.

  • You did.

    • So clearly you are an idiot.

    • Nothing to worry about.

Classic versions of this include ‘are you short the market?’ ‘why are you not borrowing at terrible interest rates?’ and ‘why haven’t you started doing terrorism?’

Here is an example from this Saturday. This is an ongoing phenomenon. In case you need a response, here are On AI and Interest Rates (which also covers the classic ‘the market is not predicting it so it isn’t real’) and AI: Practical Advice for the Worried. I still endorse most of that advice, although I mostly no longer think ‘funding or working on any AI thing at all’ is still a major vector for AI acceleration, as long as something is unrelated to core capabilities.

Other favorites include all variations of both ‘why are you taking any health risks [or other consequences]’ and ‘why are you paying attention to your long term health [or other consequences].’

Maybe half the explanation is embodied in this very good two sentences:

Cate Hall: [That] statement makes sense if you reject the idea of probabilistic beliefs. I don’t know many wise people who do that.

Then the next day Jan Leike got a lot less cryptic, as detailed above.

Then we found out it goes beyond the usual NDAs.

Why have we previously heard so little from ex-employees?

Short of forfeiting their equity, OpenAI employees are told they must sign extremely strong NDAs and non-disparagement agreements, of a type that sets off alarm bells. Then you see how they mislead and threaten employees to get them to sign.

Kelsey Piper: It turns out there’s a very clear reason for that. I have seen the extremely restrictive off-boarding agreement that contains nondisclosure and non-disparagement provisions former OpenAI employees are subject to. It forbids them, for the rest of their lives, from criticizing their former employer. Even acknowledging that the NDA exists is a violation of it.

Equity is part of negotiated compensation; this is shares (worth a lot of $$) that the employees already earned over their tenure at OpenAI. And suddenly they’re faced with a decision on a tight deadline: agree to a legally binding promise to never criticize OpenAI, or lose it.

Employees are not informed of this when they’re offered compensation packages that are heavy on equity. Vague rumors swirl, but many at OpenAI still don’t know details. The deal also forbids anyone who signs from acknowledging the fact that the deal exists.

This isn’t just a confidentiality agreement (that is normal). It prohibits disparaging comments made from public information. A former employee could potentially be in violation if they told a friend that they thought OpenAI’s latest public research paper was low-quality.

OpenAI’s leadership likes to talk about their commitment to democratic governance and oversight. It’s hard to take them seriously when they’re springing a surprise of this magnitude on former employees in order to shut down conversations about the company.

I am grateful for the courage of the ex-employees who under a lot of pressure and at significant personal cost shared the evidence of this situation.

Soumith Chintala (May 17, 3: 14pm): Holding your (already earned) stock compensation hostage over signing a non-disparagement clause is notnormal. IMO its pretty sketchy and sleazy. Its sleazier when a super-majority of your compensation comes in the form of stock (like at OpenAI).

Is this confirmed and real?

Soumith Chintala (May 17 7: 30pm): I got confirmation from multiple ex-OpenAI folks that this is true, and that’s why they don’t say anything negative about their experience.

Matt Bruenig: If this is you, hire me.

If this was me, and I was a current or former OpenAI employee, I would absolutely, at minimum, consult a labor lawyer to review my options.

How are they doing it? Well, you see…

Evan Hubinger: Here’s the full answer—looks like it’s worse than I thought and the language in the onboarding agreement seems deliberately misleading.

Kelsey Piper: I’m getting two reactions to my piece about OpenAI’s departure agreements: “that’s normal!” (it is not; the other leading AI labs do not have similar policies) and “how is that legal?” It may not hold up in court, but here’s how it works:

OpenAI like most tech companies does salaries as a mix of equity and base salary. The equity is in the form of PPUs, ‘Profit Participation Units’. You can look at a recent OpenAI offer and an explanation of PPUs here.

Many people at OpenAI get more of their compensation from PPUs than from base salary. PPUs can only be sold at tender offers hosted by the company. When you join OpenAI, you sign onboarding paperwork laying all of this out.

And that onboarding paperwork says you have to sign termination paperwork with a ‘general release’ within sixty days of departing the company. If you don’t do it within 60 days, your units are canceled. No one I spoke to at OpenAI gave this little line much thought.

4. Release of Claims: If the Grantee becomes a Withdrawn Limited Partner, then unless, within 60 days following its applicable Withdrawal Event, the Grantee (or, if the Grantee has become a Withdrawn Limited Partner in consequence of death or Permanent Incapacity, such Withdrawn Limited Partner’s estate, custodian or other legal representative or successor) duly executes and delivers to the Partnership a general release of claims against the Partnership and the other Partners with regard to all matters relating to the Partnership up to and including the time of such Withdrawal Event, such Grantee’s Units shall be cancelled and reduced to zero (0) effective as of the date of the Withdrawal Event, as set forth in Section 7.5(c) of the Partnership Agreement.

5. Removal for Cause. If the Grantee becomes a Withdrawn Limited Partner pursuant to Section 7.3(b) of the Partnership Agreement in response to an action or omission on the part of such Limited Partner that constitutes Cause, then such Grantee’s Units shall be cancelled and reduced to zero (0) effective as of the date of the Withdrawal Event, as set forth in Section 7.5(d) of the Partnership Agreement.

Kelsey Piper: And yes this is talking about vested units, because a separate clause clarifies that unvested units just transfer back to the control of OpenAI when an employee undergoes a termination event (which is normal).

There’s a common legal definition of a general release, and it’s just a waiver of claims against each other. Even someone who read the contract closely might be assuming they will only have to sign such a waiver of claims.

But when you actually quit, the ‘general release’? It’s a long, hardnosed, legally aggressive contract that includes a confidentiality agreement which covers the release itself, as well as arbitration, nonsolicitation and nondisparagement and broad ‘noninterference’ agreement.

And if you don’t sign within sixty days your units are gone. And it gets worse – because OpenAI can also deny you access to the annual events that are the only way to sell your vested PPUs at their discretion, making ex-employees constantly worried they’ll be shut out.

Finally, I want to make it clear that I contacted OpenAI in the course of reporting this story. So did my colleague Sigal Samuel. They had every opportunity to reach out to the ex-employees they’d pressured into silence and say this was a misunderstanding. I hope they do.

Clause four was the leverage. Have people agree to sign a ‘general release,’ then have it include a wide variety of highly aggressive clauses, under threat of loss of equity. Then, even if you sign it, OpenAI has complete discretion to deny you any ability to sell your shares.

Note clause five as well. This is a second highly unusual clause in which vested equity can be canceled. What constitutes ‘cause’? Note that this is another case where the threat is stronger than its execution.

One potential legal or ethical justification for this is that these are technically ‘profit participation units’ (PPUs) rather than equity. Perhaps one could say that this was a type of ‘partnership agreement’ for which different rules apply, if you stop being part of the team you get zeroed.

But notice Sam Altman has acknowledged, in the response we will get to below, that this is not the case. Not only does he claim no one has had their vested equity confiscated, he then admits that there were clauses in the contracts that refer to the confiscation of vested equity. That is an admission that he was, for practical purposes, thinking of this as equity.

So the answer to ‘how is this legal’ is ‘it probably isn’t, but how do you find out?’

Ravi Parikh: In the same way that California (and now the US) made noncompetes illegal, holding already-earned compensation hostage is another anticompetitive business practice that should be shut down.

Sanjay: I think using vested equity as hostage is fairly common, just not like this. The more common way to hold people hostage is with the 90 day exercise window and preventing employees from selling secondaries.

Ravi Parikh: 90 day exercise window is a side effect of the law, which does hurt employees but it’s not explicitly a choice companies make. Companies can extend it to 10 years by converting the type of option, which we did at my startup Heap.

Garrison Lovely: Perhaps this is a dumb question, but why do non-disparagement agreements exist? Like what is the moral case for them?

They seem fundamentally in tension with having a free and democratic society. If someone says false, damaging things about you, there are already options.

(I get why they actually exist — being able to buy silence is very valuable to those in power!)

Overly broad non-disparagement clauses (such as ‘in any way for the rest of your life’) can be deemed unenforceable in court, as unreasonable restraints on speech. Contracts for almost anything can be void if one party was not offered consideration, as is plausibly the case here. There are also whistleblower and public policy concerns. And the timing and context of the NDA and especially non-disparagement clause, where the employee did not know about them, and tying them to a vested equity grant based on an at best highly misleading contract clause, seems highly legally questionable to me, although of course I am not a lawyer and nothing here is legal advice.

Certainly it would seem bizarre to refuse to enforce non-compete clauses, as California does and the FTC wants to do, and then allow what OpenAI is doing here.

A fun and enlightening exercise is to ask LLMs what they think of this situation, its legality and ethics and implications, and what companies are the closest parallels.

The following interaction was zero shot. As always, do not take LLM outputs overly seriously or reliably:

(For full transparency: Previous parts of conversation at this link, I quoted Kelsey Piper and then ask ‘If true, is what OpenAI doing legal? What could an employee do about it’ then ‘does it matter that the employee has this sprung upon them on departure?’ and then ‘same with the non-disparagement clause?’ in terms of whether I am putting my thumb on the scale).

I pause here because this is a perfect Rule of Three, and because it then finishes with In-N-Out Burger and Apple, which it says use strict NDAs but are not known to use universal non-disparagement agreements.

Claude did miss at least one other example.

Jacques: True, but taking away *vested equityis so uncommon, I don’t see how you just stumble into writing that in but idk .

And by uncommon I mean this is the first time people have noticed it at any company, so potentially so rare it’s a first.

Kelsey Piper: TikTok also does this. They were the only other major company with similar behavior I found while trying to determine how irregular this was. Skype also once criticized/sued for something sort of similar.

Kajota: Could be worse. It could be Boeing. I hear their NDA is deadly strict.

At a minimum, this practice forces us to assume the worst, short of situations so dire a threat to humanity that caution would in practice be thrown to the wind.

Seán Ó hÉigeartaigh: In 2019/2020, we and many others worked with OpenAI researchers on workshops and a report focused on moving ” beyond principles to a focus on mechanisms for demonstrating responsible behavior.” The workshops explored importance of whistleblowing.

The report highlights “Employees within organizations developing AI systems can play an important role in identifying unethical or unsafe practices. For this to succeed, employees must be well-informed about the scope of AI development efforts within their organization, and be comfortable raising their concerns, and such concerns need to be taken seriously by management. Policies that help ensure safe channels for expressing concerns are thus key foundations for verifying claims about AI development being conducted responsibly.

The goals OpenAI and others have set themselves require solving major problems and risks. Doing so safely means being careful to make it possible to raise concerns, ask for (and justify need for) help, and yes, even criticise where warranted.

It seems the only way to approach such a high-stakes challenge responsibly. I struggle to imagine anything more antithetical to those discussions than punitive NDAs that so heavily penalise staff who might want to raise such concerns; after those staff have apparently failed to resolve these matters internally with management. I’m shaken by this reporting, and I hope there is something I don’t know that can explain it.

Adam Gleave: A non-disparagement agreement that is itself subject to a non-disclosure agreement that you are only informed of when leaving the company is completely wild. I can’t think of any other tech company that does anything like that, let alone a tech non-profit.

Neel Nanda: Oh, I heard that that was standard?

Kevin Lacker: No, it’s pretty unusual, how the initial contract has “when I leave I will sign a general release.” And then the general release has other stuff in it. It seems…sneaky. What is common is to offer a severance package in return for signing nondisparagement, etc.

Neel Nanda: Sorry – the whole thing is sneaky as fuck, I’m outraged. But specifically a non disparagement stopping you talking about the non disparagement is fairly standard, as far as I’m aware (though I think it’s awful and should not be).

Kevin Lacker: I dunno, the whole point of non disparagement is that you are getting paid to publicly support the company. If you don’t want to publicly support the company, don’t sign the non disparagement agreement. IMO the only uncool part is to “submarine” the agreement.

Neel Nanda (other thread, minutes later): I think this is absolutely outrageous behaviour from OpenAI, and far outside my understanding of industry norms. I think anyone considering joining OpenAI should think hard about whether they’re comfortable with this kind of arrangement and what it implies about how employees are treated.

My sympathies go to any OpenAI employees or recent departures trapped by this kind of malpractice.

To be fair, the whole point of these setups is the public is not supposed to find out.

That makes it hard to know if the practice is widespread.

Sampat: Every company I’ve worked at had an NDA, intention assignment, noncompete and non disparagement built into the offer letter. Can’t violate any of it.

Kelsey Piper: Built into the offer letter seems much more common and more ethical than “sprung on you at departure for the consideration of already earned equity.”

Alex Covo: This might not be “normal” but I had to sign one. Sam is a VC/Banker. They have lawyers and minions to threaten and sue you into oblivion. They don’t care. It seems like a lot of the academic researchers, engineers and scientist aren’t familiar with this tactic. Welcome to Wall Street mentality. Has nothing to do with humanity, science, etc. Just protecting your investment. It’s all about the Benjamins.

Linch: “I had to sign one.”

How are you able to talk about it then?

Exactly. The non-disparagement agreement that can be discussed is not the true fully problematic non-disparagement agreement.

Rob Bensinger offers a key distinction, which I will paraphrase for length:

It is wise and virtuous to have extraordinarily tight information security practices around IP when building AGI. If anything I would worry that no company is taking sufficient precautions. OpenAI being unusually strict here is actively a feature.

This is different. This is allowing people to say positive things but not negative things, forever, and putting a high priority on that. That is deception, that is being a bad actor, and it provides important context to the actions of the board during the recent dispute.

Also an important consideration:

Pearl: Honestly, I think if I were one of the ex OpenAI employees it wouldn’t be the fear of losing my equity holding me back from whistleblowing so much as fear of Sam Altman himself.

Kelsey Piper: This was 100% a thing among many of the people I spoke to.

Sam Altman (May 18, 5pm Eastern): In regards to recent stuff about how OpenAI handles equity:

We have never clawed back anyone’s vested equity, nor will we do that if people do not sign a separation agreement (or don’t agree to a non-disparagement agreement). Vested equity is vested equity, full stop.

There was a provision about potential equity cancellation in our previous exit docs; although we never clawed anything back, it should never have been something we had in any documents or communication. This is on me and one of the few times I’ve been genuinely embarrassed running OpenAI; I did not know this was happening and I should have.

The team was already in the process of fixing the standard exit paperwork over the past month or so. If any former employee who signed one of those old agreements is worried about it, they can contact me and we’ll fix that too. Very sorry about this.

Three things:

1. Thank you for acknowledging this happened and promising to fix it.

2. This is insufficient until you legally release all former employees and return all equity to Daniel Kokotajlo (or confirm his equity is intact and will stay that way.)

3. You’re shock, shocked to find gambling in this establishment? Bullshit.

As in:

Kelsey Piper: I am glad that OpenAI acknowledges this as an embarrassment, a mistake, and not okay. It’s in their power to set right by releasing everyone who was threatened with loss of equity from the NDAs they signed under this threat.

Kelsey Piper: I’ve talked to a lot of former employees and they want an unambiguous “we’re sorry, you’re free of the agreement, and you can sell your equity at the next tender offering.” And this seems like a reasonable thing for them to want.

And as in:

Jonathan Mannhart: I know that I’m the person in charge and it’s my responsibility and that we threatened ex-employees, but I really didn’t know we did this, and…

Yes, I know it’s also my responsibility to know we did that… but come on, guys… we’re all trying to find the culprit here.

We were totally gonna change this just like… now. I mean last month already. We did not do this. I mean we did not MEAN to do this, especially because… it’s maybe illegal in some ways? But how would I know! I didn’t know!

Also, trust me, ok? Thanks. Very sorry this happened.

Alex Lawson: If I had made an embarrassing ‘mistake’ where I accidentally allowed people to be pressured into never being able to say bad things about me, I would be quite a lot more proactive about rectifying it than saying that they could individually ask me to undo it.

Naia: It’s OK, everyone. Mr. Not Consistently Candid says the whole thing was an oopsie, and that he’ll fix things one-by-one for people if they contact him privately. Definitely nothing to worry about, then. Carry on.

At some point, y’all are gonna have to learn that when people tell you someone lies as easy as he breathes, you have to start assuming things he says might be lies.

This shit is SBF all over again. Everyone knows it’s SBF all over again, and people are somehow still falling for it.

This is getting retweeted, so context for people who don’t know me: I worked with SBF in early 2018, warned people then that he was a pathological liar and sociopath, and watched as everyone made excuses for him. I’m not drawing the comparison lightly. This is the exact same shit.

Holly Elmore: *palm to forehead“So THAT’S why no former employees were speaking out! It all makes sense now.”

Keller Scholl: Extreme chutzpah to post this only after the negative backlash, as opposed to fixing immediately when Vox contacted OpenAI.

I do want to acknowledge that:

1. Being a CEO of a company like OpenAI is overwhelming.

2. You cannot be fully on top of everything or read every legal document.

3. Many lawyers consider it their job to make all contracts as one sided as possible.

But, here, in this case, to this extent? C’mon. No. 

I asked around. These levels of legal silencing tactics, in addition to being highly legally questionable, are rare and extreme, used only in the most cutthroat of industries and cases, and very much not the kind of thing lawyers sneak in unrequested unless you knew exactly which lawyers you were hiring. 

Why has confiscation not happened before? Why hadn’t we heard about this until now?

Because until Daniel Kokotajlo everyone signed.

Kelsey Piper: OpenAI says that in no cases have they actually stripped someone of their vested equity. Ex-employees aren’t impressed. “Before Daniel K I don’t know of a single person who hasn’t signed,” a person close to the company told me. “Because they did in fact threaten you had to sign.”

Easy to not strip anyone of their equity if they all sign rather than risk it! OpenAI also says that going forward, they *won’tstrip anyone of their equity for not signing the secret NDA, which is a bigger deal. I asked if this was a change of policy.

“This statement reflects reality”, replied OpenAI’s spokesperson. To be fair it’s a Friday night and I’m sure she’s sick of me. But I have multiple ex-employees confirming this, if true, would be a big change of policy, presumably in response to backlash from current employees.

Oliver Habryka: Will they also invalidate or promise not to sue anyone who is now violating the agreements that were signed under pressure? That seems like the natural conclusion of what’s going on.

Anna Salamon: This is a great question. Seems like the right angle for a public protest. Holly Elmore?

Kelsey Piper: I asked “Does OpenAI intend to hold them to those agreements? Would OpenAI sue them if they show me the agreements?” and got “you have our statement. Thanks.”

Oliver Habryka: …I wonder whether OpenAI claiming that this was always their policy might invalidate all past contracts by removing any remaining consideration. Like, if you got to keep the equity anyway, what’s the consideration for the employee who leaves?

Sawyer: Not trying to sound confident, but I’ve noticed before in other domains that it’s not too uncommon for a question like this to come down to “they have a solid theoretical argument but that doesn’t translate to a high degree of confidence in a corresponding outcome in court.”

So you can get a sort of backward-chaining effect, where someone has a legal right on paper, and it’s *theoreticallyenforceable, but between the expense and the risk nobody wants to try.

Which in turn makes actually-exercising the right weirder and less of a standard thing to do, which people intuit makes attempting to exercise the right even riskier.

The harm is not the equity. The harm is that people are forced to sign and stay silent.

Kelsey Piper: Sam Altman now says “I did not know this was happening and I should have.” about employees being threatened with loss of their vested equity if they don’t sign a restrictive separation agreement. Sam Altman, will you void the restrictive separation agreement that ex-employees signed?

Andrew Rettek: good that he’s saying this. that they never actually clawed back anyone’s equity is a far way from *no harm was done” and a radical restriction like you reported on is very likely because someone did something bad

Maddie: He thinks he can say it’s okay because “we never actually used it”. It’s the threat that counted; the fact that you didn’t use it means the threat worked.

He thinks we’ll believe him if he says he didn’t know. How could that possibly be true?

Eli Tyre: It’s not that it’s consequentially worse, necessarily. It’s that it’s very legibly OAI doing something against the interests of the world.

People differ about the level of risk and the social benefits of AGI.

Setting the incentives to make it harder for your employees to whistleblow on you, using non-standard legal arrangements, is more obviously a defection against the world than building maybe-risky tech.

The thing that’s bad here is not mainly that, say, Daniel has less money than he would otherwise, it’s that OpenAI is exerting pretty strong pressure to control the information flow about the technology that they explicitly state is world transformative.

None of this is okay.

I think it is quite bad.

It is quite bad because of the larger pattern. Sutskever’s and Leike’s departures alone would be ominous but could be chalked up to personal fallout from the Battle of the Board, or Sutskever indeed having an exciting project and taking Leike with him.

I do not think we are mostly reacting to the cryptic messages, or to the deadening silences. What we are mostly reacting to is the costly signal of leaving OpenAI, and that this cost has now once again been paid by so many of its top safety people and a remarkably large percentage of all its safety employees.

We are then forced to update on the widespread existence of NDAs and non-disparagement agreements—we are forced to ask, what might people have said if they weren’t bound by NDAs or non-disparagement agreements?

The absence of evidence from employees speaking out, and except for those by Geoffrey Irving the lack of accusations of outright lying, no longer seem like strong evidence of absence. And indeed, we now have at least a number of (anonymous) examples of ex-employees saying they would have said concerning things, but aren’t doing so out of fear.  

Yes, if the departing people thought OpenAI was plausibly about to destroy humanity in the near future due to a specific development, they would presumably break the NDAs, unless they thought it would not do any good. So we can update on that.

But that is not the baseline scenario we are worried about. We are worried that OpenAI is, in various ways and for various reasons, unlikely to responsibly handle the future creation and deployment of AGI or ASI. We are worried about a situation in which the timeline to the critical period is unclear even to insiders, so there is always a large cost to pulling costly alarms, especially in violation of contracts, including a very high personal cost. We are especially worried that Altman is creating a toxic working environment at OpenAI for those working on future existential safety, and using power plays to clean house.

We also have to worry what else is implied by OpenAI and Altman being willing to use such rare highly deceptive and cutthroat legal tactics and intimidation tactics, and how they handled the issues once brought to light.

At minimum, this shows a company with an extreme focus on publicity and reputation management, and that wants to silence all criticism. That already is anathema to the kind of openness and truth seeking we will need. 

It also in turn suggests the obvious question of what they are so keen to hide.

We also know that the explicit commitment to the Superalignment team of 20% of current compute was not honored. This is a very bad sign.  

If OpenAI and Sam Altman want to fix this situation, it is clear what must be done as the first step. The release of claims must be replaced, including retroactively, by a standard release of claims. Daniel’s vested equity must be returned to him, in exchange for that standard release of claims. All employees of OpenAI, both current employees and past employees, must be given unconditional release from their non-disparagement agreements, all NDAs modified to at least allow acknowledging the NDAs, and all must be promised in writing the unconditional ability to participate as sellers in all future tender offers. 

Then the hard work can begin to rebuild trust and culture, and to get the work on track. 

Not everyone is unhappy about these departures.

There is a group of people who oppose the idea of this team, within a private company, attempting to figure out how we might all avoid a future AGI or ASI killing everyone, or us losing control over the future, or other potential bad outcomes. They oppose such attempts on principle.

To be clear, it’s comprehensible to believe that we should only engage in private preventative actions right now, either because (1) there is no worthwhile government action that can be undertaken at this time, or because (2) in practice, government action is likely to backfire. 

I strongly disagree with that, but I understand the viewpoint.

It is also not insane to say people are overreacting to the new information.

This is something else. This is people saying: “It is good that a private company got rid of the people tasked with trying to figure out how to make a future highly capable AI do things we want it to do instead of things we do not want it to do.”

There is a reduction in voluntary private safety efforts. They cheer and gloat.

This ranges from insane but at least in favor of humanity…

Matinusdissèque.eth: You’re losing. Just accept it man. We must continue to safely and incrementally accelerate!

…to those who continue to have a false idea of what happened when the board attempted to fire Altman, and think that safety is a single entity, so trying not to die is bad now…

Stefano Fait: Good, they almost killed the company.

…to those who (I think correctly) think OpenAI’s specific approach to safety wouldn’t work, and who are modeling the departures and dissolution of the Superalignment team as a reallocation to other long term safety efforts as opposed to a move against long term (and also short term) safety efforts in general…

Gallabytes: so tbh I think this is probably good – imo “superalignment” and “weak to strong generalization” are terrible frames for how to control neural systems, and I’m happy to see those resources go elsewhere.

…to those who dismiss all such concerns as ‘sci-fi’ as if that is an argument…

Colin Fraser (before Leike’s thread): Maybe “what they saw” is that having a large division dedicated to science fiction larping is not actually conducive to running a business and that causes irreconcilable tension with the people who are trying to make money.

I think there is good work to do on alignment by which I mean getting the chat bot to more frequently output text that is aesthetically aligned with the company’s objectives but not really on superalignment by which I mean imagining a guy to be scared of.

…to those who consider this a problem for Future Earth and think Claude Opus is dumber than a cat…

Yann LeCun: It seems to me that before “urgently figuring out how to control AI systems much smarter than us” we need to have the beginning of a hint of a design for a system smarter than a house cat.

Such a sense of urgency reveals an extremely distorted view of reality. No wonder the more based members of the organization seeked to marginalize the superalignment group.

It’s as if someone had said in 1925 “we urgently need to figure out how to control aircrafts that can transport hundreds of passengers at near the speed of the sound over the oceans.”

…

It will take years for them to get as smart as cats, and more years to get as smart as humans, let alone smarter (don’t confuse the superhuman knowledge accumulation and retrieval abilities of current LLMs with actual intelligence).

…to those who are in favor of something else entirely and want to incept even more of it.

Based Beff Jezos: OpenAI all-in on e/acc confirmed.

Jakub Pachocki will replace Ilya as Chief Scientist.

The Superalignment team has been dissolved (also confirmed by Wired). 

John Schulman will replace Jan Leike as head of AGI related safety efforts, but without a dedicated team. Remaining members have been dispersed across various research efforts.

We will watch to see how OpenAI chooses to handle their non-disparagement clauses.

One provision of the proposed bill SB 1047 is whistleblower protections. This incident illustrates why such protections are needed, whatever one thinks of the rest of the bill. 

This also emphasizes why we need other transparency and insight into the actions of companies such as OpenAI and their safety efforts. 

If you have information you want to share, with any level of confidentiality, you can reach out to me on Twitter or LessWrong or otherwise, or you can contact Kelsey Piper whose email is at the link, and is firstname.lastname@vox.com. 

If you don’t have new information, but do have thoughtful things to say, speak up.

As a canary strategy, consider adding your like to this Twitter post to indicate that (like me) you are not subject to a non-disparagement clause or a self-hiding NDA.

Everyone needs to update their views and plans based on this new information. We need to update, and examine our past mistakes, including taking a hard look at the events that led to the founding of OpenAI. We should further update based on how they deal with the NDAs and non-disparagement agreements going forward. 

The statements of anyone who worked at OpenAI at any point need to be evaluated on the assumption that they have signed a self-hiding NDA and a non-disparagement clause. Note that this includes Paul Christiano and Dario Amodei. There have been notes that Elon Musk has been unusually quiet, but if he has a non-disparagement clause he’s already violated it a lot.

Trust and confidence in OpenAI and in Sam Altman has been damaged, especially among safety advocates and the worried. Also across the board given the revelations about the non-disparagement provisions. The magnitude remains to be seen. 

Will there be consequences to ability to raise capital? In which direction?

Wei Dei: OpenAI relies a lot on external capital/support, and recent events hopefully mean that others will now trust it less so it’s less likely to remain a leader in the AI space. What is x-safety culture like at other major AI labs, especially Google DeepMind?

Inspect Element Capital: I think that’s the opposite. Why would you invest in a company that ties their own hands because of some nebulous beliefs in AI safety? Getting rid of this means they will now advance quicker.

Wei Dei: I was thinking about what the episodes reveal about SamA’s character, but you’re right that it could also make some people more likely to invest in OpenAI. I said “hopefully” suggesting that maybe the former would outweigh the latter.

Most large investors do not care about ethics. They care about returns. Nor do they in practice care much about how likely a company is to kill everyone. Credibly signaling that you will not pay to produce badly needed public goods, and that you will be ruthless and do what it takes, that you are willing to at least skirt with the edges of the law and employ highly deceptive practices, and are orienting entirely around profits, near term results and perhaps building a business? By default these are all very good for the stock price and for talking to venture capital.

The flip side is that past a certain point such actions are highly indicative of a company and leader likely to blow themselves up in the not too distant future. Such tactics strongly suggest that there were things vital enough to hide that such tactics were deemed warranted. The value of the hidden information is, in expectation, highly negative. If there is a public or government backlash, or business partners stop trusting you, that is not good.

There is also the issue of whether you expect Altman to honor his deal with you, including if you are an employee. If you sign a business deal with certain other individuals we need not name, knowing what we know about them now, and they as is their pattern refuse to honor it and instead attempt to cheat and sue and lie about you? That is from my perspective 100% on you.

Yet some people still seem eager to get into business with them, time and again.

OpenAI says roughly to ‘look upon your investment as something akin to a donation.’ When you invest in Sam Altman, you are risking the world’s largest rug pull. If they never earn a dollar because they are fully serious about being a non-profit, and you will get no money and also no voice, then you better find a greater fool, or you lose. If instead Altman and OpenAI are all about the money, boys, that is good news for you until you are the one on the other end.

There is also the issue that cutting this work is not good business. If this was all merely ‘OpenAI was toxic to and lost its long term safety teams forcing others to do the work’ then, sure, from one angle that’s bad for the world but also good hard nosed business tactics. Instead, notice that Jan Leike warned that OpenAI is not ready for the next generation of models, meaning GPT-5, meaning this is likely an issue no later than 2025.

Ideas like weak-to-strong generalization are things I do not expect to work with GPT-9, but I do expect them to likely be highly useful for things like GPT-5. A wise man does not cut the ‘get the AI to do what you want it to do’ department when it is working on AIs it will soon have trouble controlling. When I put myself in ‘amoral investor’ mode, I notice this is not great, a concern that most of the actual amoral investors have not noticed.

My actual expectation is that for raising capital and doing business generally this makes very little difference. There are effects in both directions, but there was overwhelming demand for OpenAI equity already, and there will be so long as their technology continues to impress.

What about employee relations and ability to hire? Would you want to work for a company that is known to have done this? I know that I would not. What else might they be doing? What is the company culture like?