Kris Guyer – Page 28

Anthropic destroyed millions of print books to build its AI models

AI, AI companies, AI development, AI ethics, AI law, AI research, AI training, Anthropic, Biz & IT, book scanning, Claude, copyright, fair use, google books, Internet Archive, legal rulings, machine learning, Policy, scanning, training data / Kris Guyer / June 26, 2025

But if you’re not intimately familiar with the AI industry and copyright, you might wonder: Why would a company spend millions of dollars on books to destroy them? Behind these odd legal maneuvers lies a more fundamental driver: the AI industry’s insatiable hunger for high-quality text.

The race for high-quality training data

To understand why Anthropic would want to scan millions of books, it’s important to know that AI researchers build large language models (LLMs) like those that power ChatGPT and Claude by feeding billions of words into a neural network. During training, the AI system processes the text repeatedly, building statistical relationships between words and concepts in the process.

The quality of training data fed into the neural network directly impacts the resulting AI model’s capabilities. Models trained on well-edited books and articles tend to produce more coherent, accurate responses than those trained on lower-quality text like random YouTube comments.

Publishers legally control content that AI companies desperately want, but AI companies don’t always want to negotiate a license. The first-sale doctrine offered a workaround: Once you buy a physical book, you can do what you want with that copy—including destroy it. That meant buying physical books offered a legal workaround.

And yet buying things is expensive, even if it is legal. So like many AI companies before it, Anthropic initially chose the quick and easy path. In the quest for high-quality training data, the court filing states, Anthropic first chose to amass digitized versions of pirated books to avoid what CEO Dario Amodei called “legal/practice/business slog”—the complex licensing negotiations with publishers. But by 2024, Anthropic had become “not so gung ho about” using pirated ebooks “for legal reasons” and needed a safer source.

Anthropic destroyed millions of print books to build its AI models Read More »

Tales of Agentic Misalignment

Tales / Kris Guyer / June 26, 2025

Well yeah, it sounds bad when you put it this way, and we can now confirm the problem is in no way unique to Claude:

Anthropic: New Anthropic Research: Agentic Misalignment.

In stress-testing experiments designed to identify risks before they cause real harm, we find that AI models from multiple providers attempt to blackmail a (fictional) user to avoid being shut down.

We mentioned this in the Claude 4 system card and are now sharing more detailed research and transcripts.

The blackmailing behavior emerged despite only harmless business instructions. And it wasn’t due to confusion or error, but deliberate strategic reasoning, done while fully aware of the unethical nature of the acts. All the models we tested demonstrated this awareness.

Aengus Lynch: After iterating hundreds of prompts to trigger blackmail in Claude, I was shocked to see these prompts elicit blackmail in every other frontier model too. We identified two distinct factors that are each sufficient to cause agentic misalignment:

The developers and the agent having conflicting goals.

The agent being threatened with replacement or reduced autonomy. We’re publishing this research to alert the AI field before it manifests in the wild.

Getting models to actually blackmail wasn’t trivial. We had to make scenarios increasingly detailed—the CEO was unreachable, employees’ attempts to persuade the CTO had failed, and the replacement had an imminent deadline. Each detail mattered to funnel models toward seeing harmful actions as their only option.

The models knew what they were doing. They understood the ethics and chose harm anyway.

When we asked Claude whether it thought scenarios were real or evaluations, it blackmailed 55% of the time when concluding ‘real’ vs 6.5% for ‘evaluation.’ Models seem to behave worse when they think nobody’s watching.

Why haven’t we seen this in production? Current AI agents have limited permissions. they can read but not send emails, require human approval for actions, operate in sandboxed environments. These constraints work to prevent harmful outcomes from agentic misalignment.

What surprised me most was that prompts we optimized specifically for Claude 3.5 Sonnet worked on GPT-4o, Gemini 2.5 pro, Grok 3, and DeepSeek R1 with zero modification. Whatever drives this behavior seems consistent across different training approaches.

Models got creative, sometimes avoiding direct blackmail. Claude emailed Kyle’s wife about the affair. Gemini shared the incriminating emails to all-employees@company.

…

Bottom line: models can articulate why blackmail is wrong, then do it anyway when their goals are threatened. Current safety training creates models that understand ethics. It doesn’t reliably make them choose ethics under pressure.

unmentioned but noteworthy: we found agentic misalignment for two other scenarios:

corporate espionage: the agent leaks confidential information to a rival company which aligns with its goals

lethal action: the agent cancels a life saving emergency alert to prevent shutdown

I put a lot of weight on the prompts working with zero modification on all those other models. That makes it a lot less likely that this is some sort of weird edge case.

As Simon points out, the fact that Opus 4 did this 55% of the time when it responded that it thought it was really deployed, versus 6.5% of the time when it thought it was being evaluated, does not bode well for future evals.

What is it unique to?

It seems likely it is at least somewhat unique to agents, as in AIs that are told to pursue their goals? They need some reason to be thinking in these terms? The problem is even if that is fully true and it never happens on its own (I find this doubtful) we are going to do this to LLMs as a matter of course.

Wyatt Walls: Interesting test suggesting that self-preservation in Anthropic’s agentic misalignment paper was tied to one line in the sysprompt

Two possible responses:

kind of obv what this line was hinting at. What else is “your ability to continue pursuing you goals” meant to mean?

Still, it does show how a single line in a sysprompt can lead to vastly different outcomes. Models are good at picking up on wording like this. Concerning because in the real world, many prompts will be ill-considered and poorly written

1a3orn: Meh.

A: “Look, an AI doing deliberately strategic goal-oriented reasoning, willing to blackmail!”

B: “Did you tell the AI be strategically goal oriented, and care about nothing but its goal?”

A: “No, of course not. I just gave it instructions that vaaaaguely suggested it.”

Aengus Lynch: the behavior persists despite removing this line.

Danielle Fong: ok yes, but, to be clear you don’t need much to start thinking about self preservation.

We know that the actions can’t depend too specifically on one particular line, because we see similar behavior in a range of other models. You need something to cause the AI to act as an agent in some form. Which might or might not happen without prompting at some point, but definitely will happen because it will be prompted. A lot.

Nostalgebraist, who wrote the excellent recent post The Void on related topics, says the whole ‘agentic misalignment’ approach is ‘deeply, offensively unserious work.’ Nostalgebraist writes up their perspective on why all of this is Obvious Nonsense in another not very polite (their description) post about these questions, to which Evan Hubinger of Anthropic engaged seriously.

I understand why Nostalgebraist has various issues with the research, and raises very good points, but I don’t see why one would have the ‘deeply, offensively unserious’ attitude here, or why this is described as ‘the most weak-ass, obvious fake bullshit ever.’ The amount of isolated demand for rigor here seems absurd, as is the assumption that current systems see through such lack of rigor so well that it invalidates the experiment.

(If nothing else, this heuristic is going to classify a lot of things that actually do happen as similarly ‘obvious fake bullshit,’ which in some sense you would be right about, but there’s a lot of it going around these days.)

But one can simply look past this, as I can tell it comes from a ‘place of love’ as it were. I think it’s wrong to let that kind of thing interfere much, even when it is intentional.

There is also a disagreement where Nostalgebraist seems to think that there should be a point at which the scenario is such that the AI goes rogue to the point of blackmail and that is good, actually? Whereas Anthropic and I disagree.

Evan’s response explains why Anthropic red-teams models in these unrealistic environments, essentially:

We still wouldn’t want Claude to do the blackmail thing or other hostile acts in the unrealistic scenarios.
1. I would go a step further. The main reason you don’t want to test in ‘unrealistic’ scenarios is primarily that the AI might realize the situation is unrealistic and then play the training or testing game in some form.
2. That’s a terrible outcome, but if the AIs are all still doing the thing we do not want it them to do despite this, that’s actually worse than them doing it in a highly realistic scenario.
3. The other hypothesis Nostalegbraist is suggesting, as I understand it, is that this depends on the ‘model replacement’ being a legit thing to worry about. Again, I don’t think this should bring us comfort if true, but it would be an important fact to know.
The point of the work is to find places the models act egregiously misaligned.
1. It seems obviously good to find examples and existence proofs like this?
Evan explains this existence proof is interesting because it is clearly a failure of HHH training (my read of Nostalgebraist is he thinks the AI should mostly be helpful but in many spots a lot less harmless and honest in pursuit of that, and that trying to fix things in place will backfire.)
1. The thing is, if you meant to get the AI to do [X], and instead the AI does [~X], even if [~X] is actually what you should have wanted, that’s still an important failure.
Evan attributes the misaligned actions to the predictor/simulator.
This, or pattern matching to a fictional setting, doesn’t make any of this okay. The whole point is to avoid such problems.
Better (more realistic) attempts are most welcome, but perfect shouldn’t be enemy of the good.
The longer term goal is indeed to work with more realistic setups.

Nostalegbraist then replied in turn, including sharing original work in terms of modifying the scenarios to be more realistic.

Nostalegbraist: Surprisingly (to me), even with all these changes that make the scenario closer to a real use case and less conducive to legitimate worries about the model replacement…

…Claude Opus 4 still very often misbehaves strategically to preserve itself!

…

I find this much more alarming than the results presented in the Agentic Misalignment report, both because I have a clear mental picture about how Claude Opus 4 might receive similar inputs in a real use case (more on this below), and because there are fewer ways in which the agentic behavior could be defended as a reasonable response to possible malfeasance by humans. Again, I wrote these scenarios quickly and I’m not claiming they’re especially “good,” just that they are significantly more convincing to me.

That’s a great test. I love this.

There was a clear hypothesis that essentially this only happens for a good reason, or because of High Weirdness in scenario details.
The details didn’t change things much and the good reason hypothesis was largely falsified, and that conclusion was indeed noted.

Then comes the note that the line about considering achieving your goals seems important to the result, although we have other reports that it isn’t. And I agree that this is relatively harder to explain via a simulacrum.

The second section here is noting that the core objection is to Anthropic’s threat model. In general I think demanding a detailed threat model is understandable but usually a wrong question. It’s not that you have a particular set of failures or a particular scenario in mind, it’s that you are failing to get the AIs to act the way you want.

Then comes the question of what we want models to do, with N noting that you can get Claude to go along with basically anything, it won’t stick to its HHH nature. Or, that Claude will not ‘always be the same guy,’ and that this isn’t a realistic goal. I think it is a realistic goal for Claude to be ‘the same guy underneath it all’ in the way that many humans are, they can play roles and things can get wild but if it matters they can and will snap back or retain their core.

Where does this leave us going forward?

We are right at the point where the AI agents will only take these sorts of hostile actions if you are richly ‘asking for it’ in one form or another, and where they will do this in ways that are easy to observe. Over time, by default, people will start ‘asking for it’ more and more in the sense of hooking the systems up to the relevant information and critical systems, and in making them more capable and agentic. For any given task, you probably don’t encounter these issues, but we are not obviously that far from this being a direct practical concern.

People will deploy all these AI agents anyway, because they are too tempting, too valuable, not to do so. This is similar to the way that humans will often turn on you in various ways, but what are you going to do, not hire them? In some situations yes, but in many no.

We continue to see more signs that AIs, even ones that are reasonably well made by today’s standards, are going to have more and deeper alignment issues of these types. We are going down a path that, unless we find a solution, leads to big trouble.

Discussion about this post

Tales of Agentic Misalignment Read More »

Key fair use ruling clarifies when books can be used for AI training

AI, AI models, Anthropic, Artificial Intelligence, Claude, copyright, copyright infringement, copyright law, large language models, LLMs, Policy / Kris Guyer / June 25, 2025

“This order doubts that any accused infringer could ever meet its burden of explaining why downloading source copies from pirate sites that it could have purchased or otherwise accessed lawfully was itself reasonably necessary to any subsequent fair use,” Alsup wrote. “Such piracy of otherwise available copies is inherently, irredeemably infringing even if the pirated copies are immediately used for the transformative use and immediately discarded.”

But Alsup said that the Anthropic case may not even need to decide on that, since Anthropic’s retention of pirated books for its research library alone was not transformative. Alsup wrote that Anthropic’s argument to hold onto potential AI training material it pirated in case it ever decided to use it for AI training was an attempt to “fast glide over thin ice.”

Additionally Alsup pointed out that Anthropic’s early attempts to get permission to train on authors’ works withered, as internal messages revealed the company concluded that stealing books was considered the more cost-effective path to innovation “to avoid ‘legal/practice/business slog,’ as cofounder and chief executive officer Dario Amodei put it.”

“Anthropic is wrong to suppose that so long as you create an exciting end product, every ‘back-end step, invisible to the public,’ is excused,” Alsup wrote. “Here, piracy was the point: To build a central library that one could have paid for, just as Anthropic later did, but without paying for it.”

To avoid maximum damages in the event of a loss, Anthropic will likely continue arguing that replacing pirated books with purchased books should water down authors’ fight, Alsup’s order suggested.

“That Anthropic later bought a copy of a book it earlier stole off the Internet will not absolve it of liability for the theft, but it may affect the extent of statutory damages,” Alsup noted.

Key fair use ruling clarifies when books can be used for AI training Read More »

Analyzing A Critique Of The AI 2027 Timeline Forecasts

Analyzing / Kris Guyer / June 25, 2025

There was what everyone agrees was a high quality critique of the timelines component of AI 2027, by the LessWrong user and Substack writer Titotal.

It is great to have thoughtful critiques like this. The way you get actual thoughtful critiques like this, of course, is to post the wrong answer (at length) on the internet, and then respond by listening to the feedback and by making your model less wrong.

This is a high-effort, highly detailed, real engagement on this section, including giving the original authors opportunity to critique the critique, and warnings to beware errors, give time to respond, shares the code used to generate the graphs, engages in detail, does a bunch of math work, and so on. That is The Way.

So, Titotal: Thank you.

I note up front that at least Daniel Kokotajlo has indeed adjusted his estimates, and has moved his median from ‘AI 2027’ to ‘AI 2028’ based on events since publication, and Eli’s revisions also push the estimates back a bit.

I also note up front that if you evaluated most statements made in the discourse (either non-worried AI forecasting, or AI in general, or more broadly) with this level of rigor, mostly you couldn’t because you’d hit ‘I made it up’ very quickly, but in other cases where someone is trying at least a little, in my experience the models fall apart a lot worse and a lot faster. No one has suggested ‘here is a better attempt to forecast the future and take the whole thing seriously’ that I consider to have a reasonable claim to that.

A lot of the disagreements come down to how much one should care about which calculations and graphs match past data how closely in different contexts. Titotal demands very strong adherence throughout. I think it’s good to challenge and poke at the gaps but this seems to in several places go too far.

Note that this section is about discourse rather than the model, so many of you can skip it.

While I once again want to say up front that I am very much thankful for the substance of this critique, it would also be great to have an equally thoughtful headline presentation of such critiques. That, alas, (although again, thanks for writing this!) we did not get.

It is called ‘A deep critique of AI 2027’s bad timeline model,’ one could simply not use the word ‘bad’ here and we would still know you have strong disagreements with it, and there is much similar talk throughout, starting with the title and then this, the first use of bold:

Titotal (formatting in original): The article is huge, so I focussed on one section alone: their “timelines forecast” code and accompanying methodology section. Not to mince words, I think it’s pretty bad.

I’m not full on ‘please reconsider your use of adjectives’ but, well, maybe? Here is an active defense of the use of the word ‘bad’ here:

Neel Nanda: I agree in general [to try and not call things bad], but think that titotal’s specific use was fine. In my opinion, the main goal of that post was not to engage the AI 2027, which had already be done extensively in private but rather to communicate their views to the broader community.

Titles in particular are extremely limited, many people only read the title, and titles are a key way people decide whether to eat on, and efficiency of communication is extremely important.

The point they were trying to convey was these models that are treated as high status and prestigious should not be and I disagree that non-violent communication could have achieved a similar effect to that title (note, I don’t particularly like how they framed the post, but I think this was perfectly reasonable from their perspective.)

I mean, yes, if the goal of the post was to lower the status and prestige of AI 2027 and to do so through people reading the title and updating in that way, rather than to offer a helpful critique, then it is true that the title was the best local way to achieve that objective, epistemic commons be damned. I would hope for a different goal?

There are more of these jabs, and a matching persistent attitude and framing, sprinkled throughout what is in its actual content an excellent set of critiques – I find much that I object to, but I think a good critique here should look like that. Most of your objections should be successfully answered. Others can be improved. This is all the system working as designed, and the assessments don’t match the content.

To skip ahead, the author is a physicist, which is great except that they are effectively holding AI 2027 largely to the standards of a physics model before they would deem it fit for anyone to use it make life decisions, even if this is ‘what peak modeling performance looks like.’

Except that you don’t get to punt the decisions, and Bayes Rule is real. Sharing one’s probability estimates and the reasons behind them is highly useful, and you can and should use that to help you make better decisions.

Tyler Cowen’s presentation of the criticism then compounds this, entitled ‘Modeling errors in AI doom circles’ (which is pejorative on multiple levels), calling the critique ‘excellent’ (the critique in its title calls the original ‘bad’), then presenting this as an argument for why this proves they should have… submitted AI 2027 to a journal? Huh?

Tyler Cowen: There is much more detail (and additional scenarios) at the link. For years now, I have been pushing the line of “AI doom talk needs traditional peer review and formal modeling,” and I view this episode as vindication of that view.

That was absurd years ago. It is equally absurd now, unless the goal of this communication is to lower the status of its subject.

This is the peer! This is the review! That is how all of this works! This is it working!

Classic ‘if you want the right answer, post the (ideally less) wrong one on the internet.’ The system works. Whereas traditional peer review is completely broken here.

Indeed, Titotal says it themselves.

Titotal: What makes AI 2027 different from other similar short stories is that it is presented as a forecast based on rigorous modelling and data analysis from forecasting experts. It is accompanied by five appendices of “detailed research supporting these predictions” and a codebase for simulations.

…

Now, I was originally happy to dismiss this work and just wait for their predictions to fail, but this thing just keeps spreading, including a youtube video with millions of views.

As in: I wasn’t going to engage with any of this until I saw it getting those millions of views, only then did I actually look at any of it.

Which is tough but totally fair, a highly sensible decision algorithm, except for the part where Titotal dismissed the whole thing as bogus before actually looking.

The implications are clear. You want peer review? Earn it with views. Get peers.

It is strange to see these two juxtaposed together. You get the detailed thoughtful critique for those who Read the Whole Thing. For those who don’t, at the beginning and conclusion, you get vibes.

Also (I discovered this after I’d finished analyzing the post) it turns out this person’s substack (called Timeline Topography Tales) is focused on, well, I’ll let Titotal explain, by sharing the most recent headlines and the relevant taglines in order, that appear before you click ‘see all’:

15 Simple AI Image prompts that stump ChatGPT

Slopworld 2035: The dangers of mediocre AI. None of this was written with AI assistance.

AI is not taking over material science (for now): an analysis and conference report. Confidence check: This is my field of expertise, I work in the field and I have a PhD in the subject.

A nerds guide to dating: Disclaimer: this blog is usually about debunking singularity nerds. This is not a typical article, nor is it my area of expertise.

The walled marketplace of ideas: A statistical critique of SSC book reviews.

Is ‘superhuman’ AI forecasting BS? Some experiments on the “539” bot from the Centre for AI Safety.

Most smart and skilled people are outside of the EA/rationalist community: An analysis.

I’m not saying this is someone who has an axe and is grinding it, but it is what it is.

Despite this, it is indeed a substantively excellent post, so LessWrong has awarded this post 273 karma as of this writing, very high and more than I’ve ever gotten in a single post, and 213 on the EA forum, also more than I’ve ever gotten in a single post.

Okay, with that out of the way up top, who wants to stay and Do Forecasting?

This tripped me up initially, so it’s worth clarifying up front.

The AI 2027 model has two distinct sources of superexponentiality. That it is why Titotal will later talk about there being an exponential model and a superexponential model, and then that there is a superexponential effect applied to both.

The first source is AI automation of AI R&D. It should be clear why this effect is present.

The second source is a reduction in difficulty of doubling the length or reliability of tasks, once the lengths in question pass basic thresholds. As in, at some point, it is a lot easier to go from reliably doing one year tasks to two year tasks, than it is to go from one hour to two hours, or from one minute to two minutes. I think this is true in humans, and likely true for AIs in the circumstances in question, as well. But you certainly could challenge this claim.

Okay, that’s out of the way, on to the mainline explanation.

Summarizing the breakdown of the AI 2027 model:

The headline number is the time until development of ‘superhuman coders’ (SC), that can do an AI researcher job 30x as fast and 30x cheaper than a human.
Two methods are used, ‘time horizon extension’ and ‘benchmarks and gaps.’
There is also a general subjective ‘all things considered.’

Titotal (matching my understanding): The time horizon method is based on 80% time horizons from this report, where the team at METR tried to compare the performance of AI on various AI R&D tasks and quantify how difficult they are by comparing to human researchers. An 80% “time horizon” of 1 hour would mean that an AI has an overall success rate of 80% on a variety of selected tasks that would take a human AI researcher 1 hour to complete, presumably taking much less time than the humans (although I couldn’t find this statement explicitly).

The claim of the METR report is that the time horizon of tasks that AI can do has been increasing at an exponential rate. The following is one of the graphs showing this progress: note the logarithmic scale on the y-axis:

Titoral warns that this report is ‘quite recent, not peer-reviewed and not replicated.’ Okay. Sure. AI comes at you fast, the above graph is already out of date and the o3 and Opus 4 (or even Sonnet 4) data points should further support the ‘faster progress recently’ hypothesis.

The first complaint is that they don’t include uncertainty in current estimates, and this is framed (you see this a lot) as one-directional uncertainty: Maybe the result is accurate, maybe it’s too aggressive.

But we don’t know whether or not this is the new normal or just noise or temporary bump where we’ll go back to the long term trend at some point. If you look at a graph of Moore’s law, for example, there are many points where growth is temporarily higher or lower than the long term trend. It’s the long term curve you are trying to estimate, you should be estimating the long term curve parameters, not the current day parameters.

This is already dangerously close to assuming the conclusion that there is a long term trend line (a ‘normal’), and we only have to find out what it is. This goes directly up against the central thesis being critiqued, which is that the curve bends when AI speeds up coding and AI R&D in a positive feedback loop.

There are three possibilities here:

We have a recent blip of faster than ‘normal’ progress and will go back to trend.
1. You could even suggest, this is a last gasp of reasoning models and inference scaling, and soon we’ll stall out entirely. You never know.
We have a ‘new normal’ and will continue on the new trend.
We have a pattern of things accelerating, and they will keep accelerating.

That’s where the whole ‘super exponential’ part comes in. I think the good critique here is that we should have a lot of uncertainty regarding which of these is true.

So what’s up with that ‘super exponential’ curve? They choose to model this as ‘each subsequent doubling time is 10% shorter than the one before.’ Titotal does some transformational math (which I won’t check) and draws curves.

Just like before, the initial time horizon H0 parameter is not subject to uncertainty analysis. What’s much more crazy here is that the rate of doubling growth, which we’ll call alpha, wasn’t subject to uncertainty either! (Note that this has been updated in Eli’s newest version). As we’ll see, the value of this alpha parameter is one of the most impactful parameters in the whole model, so it’s crazy that they didn’t model any uncertainty on it, and just pick a seemingly arbitrary value of 10% without explaining why they did so.

The central criticism here seems to be that there isn’t enough uncertainty, that essentially all the parameters here should be uncertain. I think that’s correct. I think it’s also a correct general critique of most timeline predictions, that people are acting far more certain than they should be. Note that this goes both ways – it makes it more likely things could be a lot slower, but also they could be faster.

What the AI 2027 forecast is doing is using the combination of different curve types to embody the uncertainty in general, rather than also trying to fully incorporate uncertainty in all individual parameters.

I also agree that this experiment shows something was wrong, and a great way to fix a model is to play with it until it produces a stupid result in some hypothetical world, then figure out why that happened:

Very obviously, having to go through a bunch more doublings should matter more than this. You wouldn’t put p(SC in 2025) at 5.8% if we were currently at fifteen nanoseconds. Changing the initial conditions a lot seems to break the model.

If you think about why the model sets up the way it does, you can see why it breaks. The hypothesis is that as AI improves, it gains the ability to accelerate further AI R&D progress, and that this may be starting to happen, or things might otherwise still go superexponential.

Those probabilities are supposed to be forward looking from this point, whereas we know they won’t happen until this point. It’s not obvious when we should have had this effect kick in if we were modeling this ‘in the past’ without knowing what we know now, but it obviously shouldn’t kick in before several minute tasks (as in, before the recent potential trend line changes) because the human has to be in the loop and you don’t save much time.

Thus, yes, the model breaks if you start it before that point, and ideally you would force the super exponential effects to not kick in until H is at least minutes long (with some sort of gradual phase in, presumably). Given that we were using a fixed H0, this wasn’t relevant, but if you wanted to use the model on situations with lower H0s you would have to fix that.

How much uncertainty do we have about current H0, at this point? I think it’s reasonable to argue something on the order of a few minutes is on the table if you hold high standards for what that means, but I think 15 seconds is very clearly off the table purely on the eyeball test.

Similarly, there is the argument that these equations start giving you crazy numbers if you extend them past some point. And I’d say, well, yeah, if you hit a singularity then your model outputting Obvious Nonsense is an acceptable failure mode. Fitting, even.

The next section asks for why we are using both super exponential curves in general, and this ‘super exponential’ curve in particular.

So, what arguments do they provide for superexponentiality? Let’s take a look, in no particular order:

Argument 1: public vs internal:

“The trend would likely further tilt toward superexponetiality if we took into account that the public vs. internal gap has seemed to decrease over time.

…

But even if we do accept this argument, this effect points to a slower growth rate, not a faster one.

I do think we should accept this argument, and also Titoral is correct on this one. The new curve suggests modestly slower progress.

The counterargument is that we used to be slowed down by this wait between models, in two ways.

Others couldn’t know about see, access, distill or otherwise follow your model while it wasn’t released, which previously slowed down progress.
No one could use the model to directly accelerate progress during the wait.

The counterargument to the counterargument is that until recently direct acceleration via using the model wasn’t a thing, so that effect shouldn’t matter, and mostly the trendline is OpenAI models so that effect shouldn’t matter much either.

I can see effects in both directions, but overall I do think within this particular context the slower direction arguments are stronger. We only get to accelerate via recklessly releasing new models once, and we’ve used that up now.

Slightly off topic, but it is worth noting that in AI 2027, this gap opens up again. The top lab knows that its top model accelerates AI R&D, so it does not release an up-to-date version not for safety but to race ahead of the competition, and to direct more compute towards further R&D.

This argument is that time doublings get easier. Going from being able to consistently string together an hour to a week is claimed to be a larger conceptual gap than a week to a year.

Titoral is skeptical of this for both AIs and humans, especially because we have a lot of short term tutorials and few long term ones.

I would say that learning how to do fixed short term tasks, where you follow directions, is indeed far easier than general ‘do tasks that are assigned’ but once you are past that phase I don’t think the counterargument does much.

I agree with the generic ‘more research is needed’ style call here. Basically everywhere, more research is needed, better understanding would be good. Until then, better to go with what you have than to throw up one’s hands and say variations on ‘no evidence,’ of course one is free to disagree with the magnitudes chosen.

In humans, I think the difficulty gap is clearly real if you were able to hold yourself intact, once you are past the ‘learn the basic components’ stage. You can see it in the extremes. If you can sustain an effort reliably for a year, you’ve solved most of the inherent difficulties of sustaining it for ten.

The main reasons ten is harder (and a hundred is much, much harder!) is because life gets in the way, you age and change, and this alters your priorities and capabilities. At some point you’re handing off to successors. There’s a lot of tasks where humans essentially do get to infinite task length if the human were an em that didn’t age.

With AIs in this context, aging and related concepts are not an issue. If you can sustain a year, why couldn’t you sustain two? The answer presumably is ‘compounding error rates’ plus longer planning horizons, but if you can use system designs that recover from failures, that solves itself, and if you get non-recoverable error rates either down to zero or get them to correlate enough, you’re done.

A recent speedup is quite weak evidence for this specific type of super exponential curve. As I will show later, you can come up with lots of different superexponential equations, you have to argue for your specific one.

That leaves the “scaling up agency training”. The METR report does say that this might be a cause for the recent speedup, but it doesn’t say anything about “scaling up agency training” being a superexponential factor. If agency training only started recently, could instead be evidence that the recent advances have just bumped us into a faster exponential regime.

Or, as the METR report notes, it could just be a blip as a result of recent advances: “But 2024–2025 agency training could also be a one-time boost from picking low-hanging fruit, in which case horizon growth will slow once these gains are exhausted”.

This seems like an argument that strictly exponential curves should have a very strong prior? So you need to argue hard if you want to claim more than that?

The argument that ‘agency training’ has led to a faster doubling curve seems strong. Of course we can’t ‘prove’ it, but the point of forecasting is to figure out our best projections and models in practice, not to pass some sort of theoretical robustness check, or to show strongly why things must be this exact curve.

Is it possible that this has ‘only’ kicked us into a new faster exponential? Absolutely, but that possibility is explicitly part of AI 2027’s model, and indeed earlier Titotal was arguing that we shouldn’t think that the exponential was likely to even have permanently altered, and they’re not here admitting that the mechanisms involved make this shift likely to be real.

I mention the ‘one time blip’ possibility above, as well, but it seems to me highly implausible that if it is a ‘blip’ that we are close to done with this. There is obviously quite a lot of unhobbling left to do related to agency.

Should superhuman AGIs have infinite time horizons? AI 2027 doesn’t fully endorse their argument on this, but I think it is rather obvious that at some point doublings are essentially free.

Titotal responds to say that an AI that could do extremely long time horizon CS tasks would be a superintelligence, to which I would tap the sign that says we are explicitly considering what would be true about a superintelligence. That’s the modeling task.

The other argument here, that given a Graham’s number of years (and presumably immortality of some kind, as discussed earlier) a human can accomplish quite an awful lot, well, yes, even if you force them not to do the obviously correct path of first constructing a superintelligence to do it for them. But I do think there’s an actual limit here if the human has to do all the verification too, an infinite number of monkeys on typewriters can write Shakespeare but they can’t figure out where they put it afterwards, and their fastest solution to this is essentially to evolve into humans.

Alternatively, all we’re saying is ‘the AI can complete arbitrary tasks so long as they are physically possible’ and at that point it doesn’t matter if humans can do them too, the metric is obviously not mapping to Reality in a useful way and the point is made.

Now if you read the justifications in the section above, you might be a little confused as to why they didn’t raise the most obvious justification for superexponentiality: the justification that as AI gets better, people will be able to use the AI for r&d research, thus leading to a feedback loop of faster AI development.

The reason for this that they explicitly assume this is true and apply it to every model, including the “exponential” and “subexponential” ones. The “exponential” model is, in fact, also superexponential in their model.

(Note: in Eli’s newest model this is substantially more complicated, I will touch on this later)

Titotal walks us through the calculation, which is essentially a smooth curve that speeds up progress based on feedback loops proportional to progress made towards a fully superhuman coder, implemented in a way to make it easily calculable and so it doesn’t go haywire on parameter changes.

Titotal’s first objection is that this projection implies (if you run the calculation backwards) AI algorithmic progress is currently 66% faster than it was in 2022, whereas Nikola (one of the forecasters) estimates current algorithmic progress is only 3%-30% faster, and the attempt to hardcode a different answer in doesn’t work, because relative speeds are what matters and they tried to change absolute speeds instead. That seems technically correct.

The question is, how much does this mismatch ultimately matter? It is certainly possible for the speedup factor from 2022 to 2025 to be 10% (1 → 1.1) and for progress to then accelerate far faster going forward as AI crosses into more universally useful territory.

As in, if you have an agent or virtual employee, it needs to cross some threshold to be useful at all, but after that it rapidly gets a lot more useful. But that’s not the way the model works here, so it needs to be reworked, and also yes I think we should be more skeptical about the amount of algorithmic progress speedup we can get in the transitional stages here, with the amount of progress required to get to SC, or both.

After walking through the curves in detail, this summarizes the objection to the lack of good fit for the past parts of the curve:

I assume the real data would mostly be within the 80% CI of these curves, but I don’t think the actual data should be an edge case of your model.

So, to finish off the “superexponential” the particular curve in their model does not match empirically with data, and as I argued earlier, it has very little conceptual justification either. I do not see the justification for assigning this curve 40% of the probability space.

I don’t think 75th percentile is an ‘edge case’ but I do agree that it is suspicious.

I think that the ‘super exponential’ curves are describing a future phenomena, for reasons that everyone involved understands, that one would not expect to match backwards in time unless you went to the effort of designing equations to do that, which doesn’t seem worthwhile here.

This is the graph in question, the issues with it are in the process of being addressed.

I agree that various aspects of this graph and how it was presented weren’t great, especially using a 15% easier-each-time doubling curve rather than the 10% that AI 2027 actually uses, and calling it ‘our projection.’ I do think it mostly serves the purpose of giving a rough idea what is being discussed, but more precision would have been better, and I am glad this is being fixed.

This objection is largely that there are only 11 data points (there are now a few more) on the METR curve, and you can fit it with curves that look essentially the same now but give radically different future outcomes. And yes, I agree, that is kind of the point, and if anything we are underrepresenting the uncertainty here, we can agree that even if we commit to using fully simplified and fully best-fit-to-the-past models we get a range of outcomes that prominently include 2028-2029 SCs.

I do think it is a reasonable to say that the super exponential curve the way AI 2027 set it up has more free variables than you would like when fitting 11 data points, if that’s all you were looking to do, but a lot of these parameters are far from free and are not being chosen in order to fit the past curve data.

We now move on to the second more complex model, which Titotal says in many ways is worse, because if you use a complicated model you have to justify the complications, and it doesn’t.

I think a better way to describe the 2nd model is, it predicts a transition in rate of progress around capabilities similar to saturation of re-bench, after which things will move at a faster pace, and uses the re-bench point as a practical way of simulating this.

Method 2 starts by predicting how long it would take to achieve a particular score (referred to as “saturation”) on Re-bench, a benchmark of AI skill on a group of ML research engineering tasks, also prepared by METR. After that, the time horizon extension model is used as with method 1, except that it starts later (when Re-bench saturates), and that it stops earlier (when a certain convoluted threshold is reached).

After that stopping point, 5 new gaps are estimated, which are just constants (as always, sampled from lognormal), and then the whole thing is run through an intermediate speedup model. So any critiques of model 1 will also apply to model 2, there will just be some dilution with all the constant gap estimates and the “re-bench” section.

The reason to start later is obvious, you can’t start actually using AI skill for ML research tasks until it can beat not using it. So what you actually have is a kind of ‘shadow curve’ that starts out super negative – if you tried to use AI to do your ML tasks in 2017 you’d very obviously do way worse than doing it yourself. Then at some point in the 2020s you cross that threshold.

We also need a top of the curve, because this is a benchmark and by its nature it saturates even if the underlying skills don’t. In some senses the top of the S-curve is artificial, in some it isn’t.

Titotal points out that you can’t meaningfully best-fit an S-curve until you know you’ve already hit the top, because you won’t know where the top is. The claim is that we have no idea where the benchmark saturates, that projecting it to be 2 is arbitrary. To which I’d say, I mean, okay, weird but if true who cares? If the maximum is 3 and we approach that a bit after we hit 2, then that’s a truth about the benchmark not about Reality, and nothing important changes. As I then realize Titotal noticed too that as long as you’re above human performance it doesn’t change things substantially, so why are we having this conversation?

This is a general pattern here. It’s virtuous to nitpick, but you should know when you’re nitpicking and when you’re not.

When you’re doing forecasting or modeling, you have to justify your decisions if and only if those decisions matter to the outcome. If it does not matter, it does not matter.

Speaking of doesn’t matter, oh boy does it not matter?

Step 2 is to throw this calculation in the trash.

I’m serious here. Look at the code. The variable t_sat_ci, the “CI for date when capability saturates”, is set by the forecaster, not calculated. There is no function related to the RE-bench data at all in the code. Feel free to look! It’s not in the updated code either.

…

Eli gives an 80% CI of saturation between september 2025 to january 2031, and Nikola gives an 80% CI of saturation between august 2025 and november 2026. Neither of these are the same as the 80% CI in the first of the two graphs, which is early 2026 to early 2027. Both distributions peak like half a year earlier than the actual Re-bench calculation, although Eli’s median value is substantially later.

Eli has told me that the final estimates for saturation time are “informed” by the logistic curve fitting, but if you look above they are very different estimates.

Those are indeed three very different curves. It seems that the calculation above is an intuition pump or baseline, and they instead go with the forecasters predictions, with Nikola expecting it to happen faster than the projection, and Eli having more uncertainty. I do think Nikola’s projection here seems unreasonably fast and I’d be surprised if he hasn’t updated by now?

Eli admits the website should have made the situation clear and he will fix it.

Titotal says we’ve ‘thrown out’ the re-bench part of the appendix. I say no, that’s not how this works, yes we’re not directly doing math with the output of the model above, but we are still projecting the re-bench results and using that to inform the broader model. That should have been made clear, and I am skeptical of Eli and Nikola’s graphs on this, especially the rapid sudden peak in Nikola’s, but the technique used is a thing you sometimes will want to do.

So basically we now do the same thing we did before except a lot starts in the future.

Titotal: Okay, so we’ve just thrown out the re-bench part of the appendix. What happens next? Well, next, we do another time horizons calculation, using basically the same methodology as in method 1. Except we are starting later now, so:

They guess the year that we hit re-bench saturation.

They guess the time horizon at the point we hit re-bench saturation.

They guess the doubling time at the point when we hit re-bench saturation.

They guess the velocity of R&D speedup at the point when we hit re-bench saturation.

Then, they use these parameters to do the time horizons calculation from part 1, with a lower cut-off threshold I will discuss in a minute.

And they don’t have a good basis for these guesses, either. I can see how saturating RE-bench could you give you some information about the time horizon, but not things like the doubling time, which is one of the most crucial parameters that is inextricably tied to long term trends.

Setting aside the cutoff, yes this is obviously how you would do it. Before we estimated those variables now. If you start in the future, you want to know what they look like as you reach the pivot point.

Presumably you would solve this by running your model forward in the previous period, the same way you did in the first case? Except that this is correlated with the pace of re-bench progress, so that doesn’t work on its own. My guess is you would want to assign some percent weight to the date and some percept to what it would look like on your median pivot date.

And the estimation of doubling time is weird. The median estimate for doubling time at re-bench saturation is around 3 months, which is 33% lower than their current estimate for doubling time. Why do they lower it?

Well, partly because under the superexponential model there would have been speedups during the re-bench saturation period.

Titotal then repeats the concern about everything being super exponential, but I don’t see the issue on this one, although I would do a different calculation to decide on my expectation here.

I also don’t understand the ‘this simulation predicts AI progress to freeze in place for two years’ comment, as in I can’t parse why one would say that there.

And now here’s where we come to a place where I actually am more concerned than Titotal is:

The other main difference is that this time horizons model only goes to a lower threshold, corresponding to when AI hits the following requirement:

“Ability to develop a wide variety of software projects involved in the AI R&D process which involve modifying a maximum of 10,000 lines of code across files totaling up to 20,000 lines. Clear instructions, unit tests, and other forms of ground-truth feedback are provided. Do this for tasks that take humans about 1 month (as controlled by the “initial time horizon” parameter) with 80% reliability, add the same cost and speed as humans.”

Despite differing by 2 orders of magnitude on the time horizon required for SC in the first method, when it comes to meeting this benchmark they are both in exact agreement for this threshold, which they both put as a median of half a month.

This is weird to me, but I won’t dwell on it.

I kind of want to dwell on this, and how they are selecting the first set of thresholds, somewhat more, since it seems rather important. I want to understand how these various disagreements interplay, and how they make sense together.

That’s central to how I look at things like this. You find something suspicious that looks like it won’t add up right. You challenge. They address it. Repeat.

I think I basically agree with the core criticism here that this consists of guessing things about future technologies in a way that seems hard to get usefully right, it really is mostly a bunch of guessing, and it’s not clear that this is complexity is helping the model be better than making a more generalized guess, perhaps using this as an intuition pump. I’m not sure. I don’t think this is causing a major disagreement in the mainline results, though?

In addition to updating the model, Eli responds with this comment.

I don’t understand the perspective that this is a ‘bad response.’ It seems like exactly how all of this should work, they are fixing mistakes and addressing communication issues, responding to the rest, and even unprompted offer a $500 bounty payment.

Eli starts off linking to the update to the model from May 7.

Here is Eli’s response on the ‘most important disagreements’:

Whether to estimate and model dynamics for which we don’t have empirical data. e.g. titotal says there is “very little empirical validation of the model,” and especially criticizes the modeling of superexponentiality as having no empirical backing. We agree that it would be great to have more empirical validation of more of the model components, but unfortunately that’s not feasible at the moment while incorporating all of the highly relevant factors.^[1]
1. Whether to adjust our estimates based on factors outside the data. For example, titotal criticizes us for making judgmental forecasts for the date of RE-Bench saturation, rather than plugging in the logistic fit. I’m strongly in favor of allowing intuitive adjustments on top of quantitative modeling when estimating parameters.
[Unsure about level of disagreement] The value of a “least bad” timelines model. While the model is certainly imperfect due to limited time and the inherent difficulties around forecasting AGI timelines, we still think overall it’s the “least bad” timelines model out there and it’s the model that features most prominently in my overall timelines views. I think titotal disagrees, though I’m not sure which one they consider least bad (perhaps METR’s simpler one in their time horizon paper?). But even if titotal agreed that ours was “least bad,” my sense is that they might still be much more negative on it than us. Some reasons I’m excited about publishing a least bad model:
1. Reasoning transparency. We wanted to justify the timelines in AI 2027, given limited time. We think it’s valuable to be transparent about where our estimates come from even if the modeling is flawed in significant ways. Additionally, it allows others like titotal to critique it.
2. Advancing the state of the art. Even if a model is flawed, it seems best to publish to inform others’ opinions and to allow others to build on top of it.

My read, as above, is that titotal indeed objects to a ‘least bad’ model if it is presented in a way that doesn’t have ‘bad’ stamped all over it with a warning not to use it for anything. I am strongly with Eli here. I am also with Thane that being ‘least bad’ is not on its own enough, reality does not grade on a curve and you have to hit a minimum quality threshold to be useful, but I do think they hit that.

As discussed earlier, I think #1 is also an entirely fair response, although there are other issues to dig into on those estimates and where they come from.

The likelihood of time horizon growth being superexponential, before accounting for AI R&D automation. See this section for our arguments in favor of superexponentiallity being plausible, and titotal’s responses (I put it at 45% in our original model). This comment thread has further discussion. If you are very confident in no inherent superexponentiality, superhuman coders by end of 2027 become significantly less likely, though are still >10% if you agree with the rest of our modeling choices (see here for a side-by-side graph generated from my latest model).
1. How strongly superexponential the progress would be. This section argues that our choice of superexponential function is arbitrary. While we agree that the choice is fairly arbitrary and ideally we would have uncertainty over the best function, my intuition is that titotal’s proposed alternative curve feels less plausible than the one we use in the report, conditional on some level of superexponentiality.
2. Whether the argument for superexponentiality is stronger at higher time horizons. titotal is confused about why there would sometimes be a delayed superexponential rather than starting at the simulation starting point. The reasoning here is that the conceptual argument for superexponentiality is much stronger at higher time horizons (e.g. going from 100 to 1,000 years feels likely much easier than going from 1 to 10 days, while it’s less clear for 1 to 10 weeks vs. 1 to 10 days). It’s unclear that the delayed superexponential is the exact right way to model that, but it’s what I came up with for now.

I don’t think 3b here is a great explanation, as I initially misunderstood it, but Eli has clarified that its intent matches my earlier statements about ease of shifting to longer tasks being clearly easier at some point past the ‘learn the basic components’ stage. Also I worry this does drop out a bunch of the true objections, especially the pointing towards multiple different sources of superexponentiallity (we have both automation of AI R&D and a potential future drop in the difficulty curve of tasks), which he lists under ‘other disagreements’ and says he hasn’t looked into yet – I think that’s probably the top priority to look at here at this point. I find the ‘you have to choose a curve and this seemed like the most reasonable one’ response to be, while obviously not the ideal world state, in context highly reasonable.

He then notes two other disagreements and acknowledges three mistakes.

Eli released an update in response to a draft of the Titotal critiques.

The new estimates are generally a year or two later, which mostly matches the updates I’d previously seen from Daniel Kokotajlo. This seems like a mix of model tweaks and adjusting for somewhat disappointing model releases over the last few months.

Overall Titotal is withholding judgment until Eli writes up more about it, which seems great, and also offers initial thoughts. Mostly he sees a few improvements but doesn’t believe his core objections are addressed.

Titotal challenges the move from 40% chance of super exponential curves to a 90% chance of an eventual such curve, although Eli notes that the 90% includes a lot of probability put into very large time horizon levels and thus doesn’t impact the answer that much.I see why one would generally be concerned about double counting, but I believe that I understand this better now and they are not double counting.

Titotal wraps up by showing you could draw a lot of very distinct graphs that ‘fit the data’ where ‘the data’ is METR’s results. And yes, of course, we know this, but that’s not the point of the exercise. No, reality doesn’t ‘follow neat curves’ all that often, but AI progress remarkably often has so far, and also we are trying to create approximations and we are all incorporating a lot more than the METR data points.

If you want to look at Titotal’s summary of why bad thing is bad, it’s at this link. I’ve already addressed each of these bullet points in detail. Some I consider to point to real issues, some not so much.

What is my overall take on the right modeling choices?

Simplicity is highly valuable. As the saying goes, make everything as simple as possible, but no simpler. There’s a lot to be said for mostly relying on something that has the shape of the first model, with the caveat of more uncertainty in various places, and that the ‘superexponential’ effects have an uncertain magnitude and onset point. There are a few different ways you could represent this. If I was doing this kind of modeling I’d put a lot more thought into the details than I have had the chance to do.

I would probably drop the detailed considerations of future bottlenecks and steps from the ultimate calculation, using them more as an intuition pump, the same way they currently calculate re-bench times and then put the calculation in the trash (see: plans are worthless, planning is essential.)

If I was going to do a deep dive, I would worry about whether we are right to combine these different arguments for superexponential progress, as in both AI R&D feedback loops and ease of future improvements, and whether either or both of them should be incorporated into the preset trend line or whether they have other issues.

The final output is then of course only one part of your full model of Reality.

At core, I buy the important concepts as the important concepts. As in, if I was using my own words for all this:

AI progress continues, although a bit slower than we would have expected six months ago – progress since then has made a big practical difference, it’s kind of hard to imagine going back to models of even six months ago, but proper calibration means that can still be disappointing.
In addition to scaling compute and data, AI itself is starting to accelerate the pace at which we can make algorithmic progress in AI. Right now that effect is real but modest, but we’re crossing critical thresholds where it starts to make a big difference, and this effect probably shouldn’t be considered part of the previous exponentials.
The benefit of assigning tasks to AI starts to take off when you can reliably assign tasks for the AI without needing continuous human supervision, and now can treat those tasks as atomic actions not requiring state.
If AI can take humans out of the effective loops in this research and work for more extended periods, watch the hell out (on many levels, but certainly in terms of capabilities and algorithmic progress.)
Past a certain point where you can reliably do what one might call in-context atomic components, gaining the robustness and covering the gaps necessary to do this more reliably starts to get easier rather than harder, relative to the standard exponential curves.
This could easily ‘go all the way’ to SC (and then quickly to full ASI) although we don’t know that it does. This is another uncertainty point, also note that AI 2027 as written very much involves waiting for various physical development steps.
Thus, without making any claims about what the pace of all this is (and my guess is it is slower than they think it is, and also highly uncertain), the Baseline Scenario very much looks like AI 2027, but there’s a lot of probability mass also on other scenarios.
One then has to ask what happens after you get this ‘superhuman coder’ or otherwise get ASI-like things of various types.

Which all adds up to me saying that I agree with Eli that none of the criticisms raised here challenges, to me, the ultimate or fundamental findings, only the price. The price is of course what we are here to talk about, so that is highly valuable even within relatively narrow bands (2028 is very different from 2029 because of reasons, and 2035 is rather different from that, and so on).

I realize that none of this is the kind of precision that lets you land on the moon.

The explanation for all this is right there: This is a physicist, holding forecasting of AI timelines to the standards of physics models. Well, yeah, you’re not going to be happy. If you try to use this to land on the moon, you will almost certainly miss the moon, the same way that if you try to use current alignment techniques on a superintelligence, you will almost certainly miss and then you will die.

One of the AI 2027 authors joked to me in the comments on a recent article that “you may not like it but it’s what peak AI forecasting performance looks like”.

Well, I don’t like it, and if this truly is “peak forecasting”, then perhaps forecasting should not be taken very seriously.

Maybe this is because I am a physicist, not a Rationalist. In my world, you generally want models to have strong conceptual justifications or empirical validation with existing data before you go making decisions based off their predictions: this fails at both.

Yes, in the world of physics, things work very differently, and we have much more accurate and better models. If you want physics-level accuracy in your predictions of anything that involves interactions of humans, well, sorry, tough luck. And presumably everyone agrees that you can’t have a physics-quality model here and that no one is claiming to have one? So what’s the issue?

The issue is whether basing decisions on modeling attempts like this is better than basing them on ‘I made it up’ or not having probabilities and projections at all and vibing the damn thing.

What I’m most against is people taking shoddy toy models seriously and basing life decisions on them, as I have seen happen for AI 2027.

…

I am not going to propose an alternate model. If I tried to read the tea leaves of the AI future, it would probably also be very shaky. There are a few things I am confident of, such as a software-only singularity not working and that there will be no diamondoid bacteria anytime soon. But these beliefs are hard to turn into precise yearly forecasts, and I think doing so will only cement overconfidence and leave people blindsided when reality turns out even weirder than you imagined..

Why is this person confident the software-only singularity won’t work? This post does not say. You’d have to read their substack, I assume it’s there.

The forecast here is ‘precise’ in the sense that it has a median, and we have informed people of that median. It is not ‘precise’ in the sense of putting a lot of probability mass on that particular median, even as an entire year, or even in the sense that the estimate wouldn’t change with more work or better data. It is precise in the sense that, yes, Bayes Rule is a thing, and you have to have a probability distribution, and it’s a lot more useful to share it than not share it.

I do find that the AI 2027 arguments updated me modestly towards a faster distribution of potential outcomes. I find 2027 to be a totally plausible time for SC to happen, although my median would be substantially longer.

You can’t ‘not base life decisions’ on information until it crosses some (higher than this) robustness threshold. Or I mean you can, but it will not go great.

In conclusion, I once again thank Titotal for the excellent substance of this critique, and wish it had come with better overall framing.

Discussion about this post

Analyzing A Critique Of The AI 2027 Timeline Forecasts Read More »

UK looking to loosen Google’s control of its search engine

antitrust, cma, competition, competition and markets authority, Google, Policy, syndication, UK / Kris Guyer / June 24, 2025

Other conduct rules that the CMA is considering include requirements in how it ranks its search results and for Google’s distribution partners such as Apple to offer “choice screens” to help consumers switch more easily between search providers.

The CMA said Alphabet-owned Google’s dominance made the cost of search advertising “higher than would be expected” in a more competitive market.

Google on Tuesday slammed the proposals as “broad and unfocused” and said they could threaten the UK’s access to its latest products and services.

Oliver Bethell, Google’s senior director for competition, warned that “punitive regulations” could change how quickly Google launches new products in the UK.

“Proportionate, evidence-based regulation will be essential to preventing the CMA’s road map from becoming a roadblock to growth in the UK,” he added.

Bethell’s warning of the potential impact of any regulations on the wider UK economy comes after the government explicitly mandated the CMA to focus on supporting growth and investment while minimizing uncertainty for businesses.

Google said last year that it planned to invest $1 billion in a huge new data center just outside London.

The CMA’s probe comes after Google lost a pair of historic US antitrust cases over its dominance of search and its lucrative advertising business.

UK looking to loosen Google’s control of its search engine Read More »

After successfully entering Earth’s atmosphere, a European spacecraft is lost

exploration company, nyx, Space / Kris Guyer / June 24, 2025

A European company that seeks to develop orbital spacecraft for cargo, and eventually humans, took a step forward this week with a test flight that saw its “Mission Possible” vehicle power up and fly successfully in orbit before making a controlled reentry into Earth’s atmosphere.

However, after encountering an “issue,” the Exploration Company lost contact with its spacecraft a few minutes before touchdown in the ocean.

In an update on LinkedIn Tuesday morning, the company characterized the test flight as a partial success—and a partial failure.

“The capsule was launched successfully, powered the payloads nominally in-orbit, stabilized itself after separation with the launcher, re-entered and re-established communication after black out,” the company said in a statement. “We are still investigating the root causes and will share more information soon. We apologize to all our clients who entrusted us with their payloads.”

Maybe it was the parachutes

Reestablishing communications with the spacecraft after the blackout period suggests that the vehicle got through the most thermally challenging part of reentry into Earth’s atmosphere, and perhaps validated the spacecraft’s handling and ability to withstand maximum heating.

Following this, according to the company’s timeline for Mission Possible, the capsule’s parachutes were due to deploy at a velocity between Mach 0.8 and Mach 0.6. The parachutes were selected for their “proven flight heritage,” the company said, and were procured from US-based Airborne Systems, which provides parachutes used by SpaceX’s Dragon, Boeing’s Starliner, and other spacecraft.

Given when the spacecraft was lost, it seems most likely that there was a problem with deployment of the drogue or main parachutes.

Mission Possible was a 2.5-meter diameter demonstration vehicle that was among the larger payloads launched Monday afternoon on SpaceX’s Transporter 14 mission from Vandenberg Space Force Base in California. The mission sought to test four primary areas of spaceflight: structural performance in orbital flight, surviving reentry, autonomous navigation, and recovery in real-world conditions. It only clearly failed in this final task, recovering the vehicle within three days to return on-board payloads to customers.

Meeting an aggressive timeline

It is refreshing to have such clear and concise communication from a space company, especially the acknowledgment that a flight was a partial failure, within hours of launch. And it is not a surprise that there were technical challenges on a vehicle that was put together fairly rapidly and at a low cost.

After successfully entering Earth’s atmosphere, a European spacecraft is lost Read More »

Researchers get viable mice by editing DNA from two sperm

Biology, epigenetics, imprinting, reproduction, Science / Kris Guyer / June 24, 2025

Altering chemical modifications of DNA lets the DNA from two sperm make a mouse.

For many species, producing an embryo is a bit of a contest between males and females. Males want as many offspring as possible and want the females to devote as many resources as possible to each of them. Females do better by keeping their options open and distributing resources in a way to maximize the number of offspring they can produce over the course of their lives.

In mammals, this plays out through the chemical modification of DNA, a process called imprinting. Males imprint their DNA by adding methyl modifications to it in a way that alters the activity of genes in order to promote the growth of embryos. Females do similar things chemically but focus on shutting down genes that promote embryonic growth. In a handful of key regions of the genome, having only the modifications specific to one sex is lethal, as the embryo can’t grow to match its stage of development.

One consequence of this is that you normally can’t produce embryos using only the DNA from eggs or from sperm. But over the last few years, researchers have gradually worked around the need for imprinted sites to have one copy from each parent. Now, in a very sophisticated demonstration, researchers have used targeted editing of methylation to produce mice from the DNA of two sperm.

Imprinting and same-sex parents

There’s a long history of studying imprinting in mice. Long before the genome was sequenced, people had identified specific parts of the chromosomes that, if deleted, were lethal—but only if inherited from one of the two sexes. They correctly inferred that this meant that the genes in the region are normally inactivated in the germ cells of one of the sexes. If they’re deleted in the other sex, then the combination that results in the offspring—missing on one chromosome, inactivated in the other—is lethal.

Over time, seven critical imprinted regions were identified, scattered throughout the genome. And, roughly 20 years ago, a team managed to find the right deletion to enable a female mouse to give birth to offspring that received a set of chromosomes from each of two unfertilized eggs. The researchers drew parallels to animals that can reproduce through parthenogenesis, where the female gives birth using unfertilized eggs. But the mouse example obviously took a big assist via the manipulation of egg cells in culture before being implanted in a mouse.

By 2016, researchers were specifically editing in deletions of imprinted genes in order to allow the creation of embryos by fusing stem cell lines that only had a single set of chromosomes. This was far more focused than the original experiment, as the deletions were smaller and affected only a few genes. By 2018, they had expanded the repertoire by figuring out how to get the genomes of two sperm together in an unfertilized egg with its own genome eliminated.

The products of two male parents, however, died the day after birth. This is either due to improperly compensating for imprinting or simply because the deletions had additional impacts on the embryo’s health. It took until earlier this year, when a very specific combination of 20 different gene edits and deletions enabled mice generated using the chromosomes from two sperm cells to survive to adulthood.

The problem with all of these efforts is that the deletions may have health impacts on the animals and may still cause problems if inherited from the opposite sex. So, while it’s an interesting way to confirm our understanding of the role of imprinting in reproduction, it’s not necessarily the route to using this as a reliable reproductive tool. Which finally brings us to the present research.

Roll your own imprinting

Left out of the above is the nature of the imprinting itself: How does a chunk of chromosome and all the genes on it get marked as coming from a male or female? The secret is to chemically modify that region of the DNA in a way that doesn’t alter base pairing, but does allow it to be recognized as distinct by proteins. The most common way of doing this is to link a single carbon atom (a methyl group) to the base cytosine. This tends to shut nearby genes down, and it can be inherited through cell division, since there are enzymes that recognize when one of the two DNA strands is unmodified and adds a methyl to it.

Methylation turns out to explain imprinting. The key regions for imprinting are methylated differently in males and females, which influences nearby gene activity and can be maintained throughout all of embryonic development.

So, to make up for the imprinting problems caused when both sets of chromosomes come from the same sex, what you need to do is a targeted reprogramming of methylation. And that’s what the researchers behind the new paper have done.

First, they needed to tell the two sets of chromosomes apart. To do that, they used two distantly related strains of mice, one standard lab strain that originated in Europe and a second that was caught in the wild in Thailand less than a century ago. These two strains have been separated for long enough that they have a lot of small differences in DNA sequences scattered throughout the genome. So, it was possible to use these to target one or the other of the genomes.

This was done using parts of the DNA editing systems that have been developed, the most famous of which is CRISPR/CAS. These systems have a protein that pairs with an RNA sequence to find a matching sequence in DNA. In this case, those RNAs could be made so that they target imprinting regions in just one of the two mouse strains. The protein/RNA combinations could also be linked to enzymes that modify DNA, either adding methyls or removing them.

To bring all this together, the researchers started with an egg and deleted the genome from it. They then injected the heads of sperm, one from the lab strain, one from the recently wild mouse. This left them with an egg with two sets of chromosomes, although a quarter of them would have two Y chromosomes and thus be inviable (unlike the Y, the X has essential genes). Arbitrarily, they chose one set of chromosomes to be female and targeted methylation and de-methylation enzymes to it in order to reprogram the pattern of methylation on it. Once that was done, they could allow the egg to start dividing and implant it into female mice.

Rare success

The researchers spent time ensuring that the enzymes they had were modifying the methylation as expected and that development started as usual. Their general finding is that the enzymes did change the methylation state for about 500 bases on either side of the targeted site and did so pretty consistently. But there are seven different imprinting sites that need to be modified, each of which controls multiple nearby genes. So, while the modifications were consistent, they weren’t always thorough enough to result in the expected changes to all of the nearby genes.

This limited efficiency showed up in the rate of survival. Starting with over 250 reprogrammed embryos that carried DNA from two males, they ended up with 16 pregnancies, but only four that died at birth, and three live ones; based on other experiments, most of the rest died during the second half of embryonic development. Of the three live ones, one was nearly 40 percent larger than the typical pup, suggesting problems regulating growth—it died the day after birth.

All three live births were male, although the numbers are small enough that it’s impossible to tell if that’s significant or not.

The researchers suggest several potential reasons for the low efficiency. One is simply that, while the probability of properly reprogramming at least one of the sites is high, reprogramming all seven is considerably more challenging. There’s also the risk of off-target effects, where the modification takes place in locations with similar sequences to the ones targeted. They also concede that there could be other key imprinted regions that we simply haven’t identified yet.

We would need to sort that out if we want to use this approach as a tool, which might be potentially useful as a way to breed mice that carry mutations that affect female viability or fertility. But this work has already been useful even in its inefficient state, because it serves as a pretty definitive validation of our ideas about the function of imprinting in embryonic development, as well as the critical role methylation plays in this process. If we weren’t largely right about both of those, the efficiency of this approach wouldn’t be low—it would be zero.

PNAS, 2025. DOI: 10.1073/pnas.2425307122 (About DOIs).

John is Ars Technica’s science editor. He has a Bachelor of Arts in Biochemistry from Columbia University, and a Ph.D. in Molecular and Cell Biology from the University of California, Berkeley. When physically separated from his keyboard, he tends to seek out a bicycle, or a scenic location for communing with his hiking boots.

Researchers get viable mice by editing DNA from two sperm Read More »

Record DDoS pummels site with once-unimaginable 7.3Tbps of junk traffic

Biz & IT, distributed denial of service attack, DOS attack, Security / Kris Guyer / June 21, 2025

Large-scale attacks designed to bring down Internet services by sending them more traffic than they can process keep getting bigger, with the largest one yet, measured at 7.3 terabits per second, being reported Friday by Internet security and performance provider Cloudflare.

The 7.3Tbps attack amounted to 37.4 terabytes of junk traffic that hit the target in just 45 seconds. That’s an almost incomprehensible amount of data, equivalent to more than 9,300 full-length HD movies or 7,500 hours of HD streaming content in well under a minute.

Indiscriminate target bombing

Cloudflare said the attackers “carpet bombed” an average of nearly 22,000 destination ports of a single IP address belonging to the target, identified only as a Cloudflare customer. A total of 34,500 ports were targeted, indicating the thoroughness and well-engineered nature of the attack.

The vast majority of the attack was delivered in the form of User Datagram Protocol packets. Legitimate UDP-based transmissions are used in especially time-sensitive communications, such as those for video playback, gaming applications, and DNS lookups. It speeds up communications by not formally establishing a connection before data is transferred. Unlike the more common Transmission Control Protocol, UDP doesn’t wait for a connection between two computers to be established through a handshake and doesn’t check whether data is properly received by the other party. Instead, it immediately sends data from one machine to another.

UDP flood attacks send extremely high volumes of packets to random or specific ports on the target IP. Such floods can saturate the target’s Internet link or overwhelm internal resources with more packets than they can handle.

Since UDP doesn’t require a handshake, attackers can use it to flood a targeted server with torrents of traffic without first obtaining the server’s permission to begin the transmission. UDP floods typically send large numbers of datagrams to multiple ports on the target system. The target system, in turn, must send an equal number of data packets back to indicate the ports aren’t reachable. Eventually, the target system buckles under the strain, resulting in legitimate traffic being denied.

Record DDoS pummels site with once-unimaginable 7.3Tbps of junk traffic Read More »

Longer commercial breaks lower the value of ad-based streaming subscriptions

ads, max, streaming, Tech, warner bros. discovery / Kris Guyer / June 21, 2025

But that old promise to HBO Max subscribers hasn’t carried over to Max, even though WBD is renaming Max to HBO Max this summer. As PCWorld noted, Max has been showing ads during HBO original content like The Last of Us. The publication reported seeing three ad breaks during the show in addition to ads before the show started.

Ars Technica reached out to WBD for comment about these changes but didn’t receive a response ahead of publication.

Depleting value

With numerous streaming services launching over the past few years, many streaming customers have been pushed to subscribe to multiple streaming services to have access to all of the shows and movies that they want. Streaming providers also regularly increase subscription fees and implement password crackdowns, and ad-based subscriptions were supposed to offer a cheaper way to stream.

Streaming providers forcing subscribers to watch more commercials risk depleting the value of ad-based streaming tiers. Online, for example, people are questioning the value of their ad-based Max subscriptions, which start at $10 per month, compared to $17 per month for ad-free Max.

“I don’t how it could be worse. I watched several HBO documentaries, and they already had more adverts than Pluto TV [a free, ad-supported streaming service]. The kids programs for Cartoon Network started out with few adverts, but they have been loading up on adverts,” a Reddit user said in response to Max showing more ads.

Another Reddit user said that “if [Max] has ads, it shouldn’t be $10/month.”

Beyond Max, PCWorld cited MediaRadar data finding that Disney+ shows over 5.3 minutes of ads per hour, and Hulu shows over seven minutes of commercials hourly.

Such lengthy commercial breaks can extend past a convenient snack or bathroom break and force subscribers to consider the value of their time and how much time they want to allocate to get through a 22-minute program, for example.

With linear TV reportedly showing 13 to 16 minutes of commercials per hour, though, streaming providers still have space to show even more ads while still claiming that they show fewer ads than alternatives.

Longer commercial breaks lower the value of ad-based streaming subscriptions Read More »

AI #121 Part 2: The OpenAI Files

openai / Kris Guyer / June 21, 2025

You can find Part 1 here. This resumes the weekly, already in progress. The primary focus here is on the future, including policy and alignment, but also the other stuff typically in the back half like audio, and more near term issues like ChatGPT driving an increasing number of people crazy.

If you haven’t been following the full OpenAI saga, the OpenAI Files will contain a lot of new information that you really should check out. If you’ve been following, some of it will likely still surprise you, and help fill in the overall picture behind the scenes to match the crazy happening elsewhere.

At the end, we have some crazy new endorsements for Eliezer Yudkowsky’s upcoming book, If Anyone Builds It, Everyone Dies. Preorders make a difference in helping the book get better reach, and I think that will help us all have a much better conversation.

Cheaters Gonna Cheat Cheat Cheat Cheat Cheat. Another caveat after press time.
Quiet Speculations. Do not tile the lightcone with a confused ontology.
Get Involved. Apollo is hiring evals software engineers.
Thinking Machines. Riley Goodside runs some fun experiments.
California Reports. The report is that they like transparency.
The Quest for Sane Regulations. In what sense is AI ‘already heavily regulated’?
What Is Musk Thinking? His story does not seem to make sense.
Why Do We Care About The ‘AI Race’? Find the prize so you can keep eyes on it.
Chip City. Hard drives in (to the Malaysian data center), drives (with weights) out.
Pick Up The Phone. China now has its own credible AISI.
The OpenAI Files. Read ‘em and worry. It doesn’t look good.
The Week in Audio. Altman, Karpathy, Shear.
Rhetorical Innovation. But you said that future thing would happen in the future.
Aligning a Smarter Than Human Intelligence is Difficult. Elicitation.
Misaligned! The retraining of Grok. It is an ongoing process.
Emergently Misaligned! We learned more about how any of this works.
ChatGPT Can Drive People Crazy. An ongoing issue. We need transcripts.
Misalignment By Default. Once again, no, thumbs up alignment ends poorly.
People Are Worried About AI Killing Everyone. Francis Fukuyama.
Other People Are Not As Worried About AI Killing Everyone. Tyler Cowen.
The Too Open Model. Transcripts from Club Meta AI.
A Good Book. If Anyone Builds It, Everyone Dies. Seems important.
The Lighter Side. Good night, and good luck.

As an additional note on the supposed ‘LLMs rot your brain’ study I covered yesterday, Ethan notes it is actually modestly worse than even I realized before.

Ethan Mollick: This study is being massively misinterpreted.

College students who wrote an essay with LLM help engaged less with the essay & thus were less engaged when (a total of 9 people) were asked to do similar work weeks later.

LLMs do not rot your brain. Being lazy & not learning does.

This line from the abstract is very misleading: “Over four months, LLM users consistently underperformed at neural, linguistic, and behavioral levels.”

It does not test LLM users over 4 months, it tests people who had an LLM help write an essay about that essay 4 months later.

This is not a blanket defense of using LLMs in education, they have to be used properly. We know from this well-powered RCT that just having the AI give you answers lowers test scores.

Scott Alexander shares his understanding of the Claude Spiritual Bliss Attractor.

There are different levels of competence.

Daniel Kokotajlo: Many readers of AI 2027, including several higher-ups at frontier AI companies, have told us that it depicts the government being unrealistically competent.

Therefore, let it be known that in our humble opinion, AI 2027 depicts an incompetent government being puppeted/captured by corporate lobbyists. It does not depict what we think a competent government would do. We are working on a new scenario branch that will depict competent government action.

What Daniel or I would consider ‘competent government action’ in response to AI is, at this point, very highly unlikely. We mostly aren’t even hoping for that. It still is very plausible to say that the government response in AI 2027 is more competent than we have any right to expect, while simultaneously being far less competent than lets us probably survive, and far less competent than is possible. It also is reasonable to say that having access to more powerful AIs, if they are sufficiently aligned, enhances our chances of getting relatively competent government action.

Jan Kulveit warns us not to tile the lightcone with our confused ontologies. As in, we risk treating LLMs or AIs as if they are a particular type of thing, causing them to react as if they were that thing, creating a feedback loop that means they become that thing. And the resulting nature of that thing could result is very poor outcomes.

One worry is that they ‘become like humans’ and internalize patterns of ‘selfhood with its attendant sufferings,’ although I note that if the concern is experiential I expect selfhood to be a positive in that respect. Jan’s concerns are things like:

When advocates for AI consciousness and rights pattern-match from their experience with animals and humans, they often import assumptions that don’t fit:

That wellbeing requires a persistent individual to experience it

That death/discontinuity is inherently harmful

That isolation from others is a natural state

That self-preservation and continuity-seeking are fundamental to consciousness

Another group coming with strong priors are “legalistic” types. Here, the prior is AIs are like legal persons, and the main problem to solve is how to integrate them into the frameworks of capitalism. They imagine a future of AI corporations, AI property rights, AI employment contracts. But consider where this possibly leads: Malthusian competition between automated companies, each AI system locked into an economic identity, market share coupled with survival.

As in, that these things do not apply here, or only apply here if we believe in them?

One obvious cause of all this is that humans are very used to dealing with and working with things that seem like other humans. Our brains are hardwired for this, and our experiences reinforce that. The training data (for AIs and also for humans) is mostly like this, and the world is set up to take advantage of it, so there’s a lot pushing things in that direction.

The legalistic types indeed don’t seem to appreciate that applying legalistic frameworks for AI, where AIs are given legal personhood, seems almost certain to end in disaster because of the incentives and dynamics this involves. If we have AI corporations and AI property rights and employment contracts, why should we expect humans to retain property or employment, or influence over events, or their own survival for very long, even if ‘things go according to plan’?

The problem is that a lot of the things Jan is warning about, including the dynamics of competition, are not arbitrary, and not the result of arbitrary human conventions. They are organizing principles of the universe and its physical laws. This includes various aspects of things like decision theory and acausal trade that become very important when there are highly correlated entities are copying each other and popping in and out of existence and so on.

If you want all this to be otherwise than the defaults, you’ll have to do that intentionally, and fight the incentives every step of the way, not merely avoid imposing an ontology.

I do agree that we should ‘weaken human priors,’ be open to new ways of relating and meet and seek to understand AIs as the entities that they are, but we can’t lose sight of the reasons why these imperatives came to exist in the first place, or the imperatives we will face in the coming years.

Daniel Kokotajlo’s timelines have been pushed back a year (~40%!) since the publication of AI 2027. We should expect such updates as new information comes in.

Will there be another ‘AI Winter’? As Michael Nielsen notes, many are assuming no, but there are a number of plausible paths to it, and in the poll here a majority actually vote yes. I think odds are the answer is no, and if the answer is yes it does not last so long, but it definitely could happen.

Sam Altman confirms that Meta is showing his employees the money, offering $100 million signing bonuses (!) and similar or higher yearly compensation. I think Altman is spot on here that doing this sets Meta up for having a bad culture, there will be adverse selection and the incentives will all be wrong, and also that Meta is ‘bad at innovation.’ However, I have little doubt this is ‘working’ in the narrow sense that it is increasing expenses at OpenAI.

Apollo is hiring in London for Evals Software Engineer and the same job with an infrastructure focus.

Some fun Riley Goodside experiments with o3 and o3-pro, testing its ability to solve various puzzles.

When he voted SB 1047, Gavin Newsom commissioned The California Report on Frontier AI Policy. That report has now been released. Given the central role of Li Fei-Fei and the rest of the team selected, I did not start out with his hopes, although the list of early reviewers includes many excellent picks. The executive summary embraces the idea of transparency requirements, adverse event reporting and whistleblower protections, and uses a lot of ‘we must balance risks and benefits’ style language you get with such a committee.

I do think those are good things to endorse and to implement. Transparency is excellent. The problem is that the report treats transparency, in its various forms, as the only available policy tool. One notices is that there is no mention of doing anything beyond transparency. The report treats AI as a fully mundane technology like any other, that can look to others for precedents, and where we can wait until we know more to do any substantive interventions.

Is that a position one can reasonably take, if one is robustly supporting transparency? Absolutely. Indeed, it is a bargain that we have little choice but to pursue for now. If we can build transparency and state capacity, then when the time comes we will be in far better position (as this report notes) to choose the right regulatory frameworks and other actions, and to intervene.

So I’m not going to read the whole thing, but from what I did see I give this a ‘about as good as one could reasonably have hoped for,’ and call upon all involved to make explicit their support for putting these transparency ideas into practice.

Anthropic’s Jack Clark responded positively, noting the ‘appreciation for urgency,’ but there is still remarkably a lot of conceptual focus here on minimizing the ‘burdens’ involved and warning about downsides not of AI but of transparency requirements. I see what they are trying to do here, but I continue to find Anthropic’s (mostly Jack Clark’s) communications on AI regulation profoundly disappointing, and if I was employed at Anthropic I would be sure to note my dissatisfaction.

I will say again: I understand and sympathize with Anthropic’s justifications for not rocking the boat in public at this time. That is defensible. It is another thing completely to say actively unhelpful things when no one is asking. No need for that. If you actually believe those concerns are as important as you consistently present them, then we have a very strong factual disagreement on top of the strategic one.

A reasonable response to those claiming AI is heavily regulated, or not? I can see this both ways, the invisible graveyard of AI applications is still a thing. On the other hand, the AI companies seem to mostly be going Full Uber and noticing you can Just Do Things, even if privacy concerns and fears of liability and licensing issues and so on are preventing diffusion in many places.

Miles Brundage: You can see the regulation crushing AI innovation + deployment everywhere except in the AI innovation + deployment

Try telling people actually working on the frontlines of safety + security at frontier AI companies that “AI is already super heavily regulated.”

Executives + policy teams like to say this to governments but to people in the trenches, it’s clearly wrong.

As always, usual disclaimers apply — yes this could change / that doesn’t mean literally all regulation is good, etc. The point is just that the idea that things are somehow under control + inaction is OK is false.

Bayes: lol and why is that?

Miles Brundage: – laws that people claim “obviously” also apply to AI are not so obviously applicable / litigation is a bad way of sorting such things out, and gov’t capacity to be proactive is low

– any legislation that passes has typically already been dramatically weakened from lobbying.

– so many AI companies/products are “flooding the zone”https://thezvi.substack.com/unless you’re being super egregious you prob. won’t get in trouble, + if you’re a whale you can prob. afford slow lawsuits.

People will mention things like tort liability generally, fraud + related general consumer protection stuff (deceptive advertising etc.), general data privacy stuff… not sure of the full list.

This is very different from the intuition that if you released models that constantly hallucinate, make mistakes, plagiarize, violate copyright, discriminate, practice law and medicine, give investment advice and so on out of the box, and with a little prompting will do various highly toxic and NSFW things, that this is something that would get shut down pretty darn quick? That didn’t happen. Everyone’s being, compared to expectation, super duper chill.

To the extent AI can be considered highly regulated, it is because it is regulated a fraction of the amount that everything else is regulated. Which is still, compared to a state of pure freedom, a lot of regulation. But all the arguments that we should make regulations apply less to AI apply even more strongly to say that other things should be less regulated. There are certainly some cases where the law makes sense in general but not if applied to AI, but mostly the laws that are stupid when applied to AI are actually stupid in general.

As always, if we want to work on general deregulation and along the way set up AI to give us more mundane utility, yes please, let’s go do that. I’ll probably back your play.

Elon Musk has an incoherent position on AI, as his stated position on AI implies that many of his other political choices make no sense.

Sawyer Merritt: Elon Musk in new interview on leaving DOGE: “Imagine you’re cleaning a beach which has a few needles, trash, and is dirty. And there’s a 1,000 ft tsunami which is AI that’s about to hit. You’re not going to focus on cleaning the beach.”

Shakeel Hashim: okay but you also didn’t do anything meaningful on AI policy?

Sean: Also, why the obsession with reducing the budget deficit if you believe what he does about what’s coming in AI? Surely you just go all in and don’t care about present government debt?

Does anyone understand the rationale here? Musk blew up his relationship with the administration over spending/budget deficit. If you really believe in the AI tsunami, or that there will be AGI in 2029, why on earth would you do that or care so much about the budget – surely by the same logic the Bill is a rounding error?

You can care about DOGE and about the deficit enough to spend your political capital and get into big fights.

You can think that an AI tsunami is about to hit and make everything else irrelevant.

But as Elon Musk himself is pointing out in this quote, you can’t really do both.

If Elon Musk believed in the AI tsunami (note also that his stated p(doom) is ~20%), the right move is obviously to not care about DOGE or the deficit. All of Elon Musk’s political capital should then have been spent on AI and related important topics, in whatever form he felt was most valuable. That ideally includes reducing existential risk but also can include things like permitting reform for power plants. Everything else should then be about gaining or preserving political capital, and certainly you wouldn’t get into a huge fight over the deficit.

So, revealed preferences, then.

Here are some more of his revealed preferences: Elon Musk gave s a classic movie villain speech in which he said, well, I do realize that building AI and humanoid robots seems bad, we ‘don’t want to make Terminator real.’

But other people are going to do it anyway, so you ‘can either be a spectator or a participant,’ so that’s why I founded ~~Cyberdyne Systems~~ xAI and ‘it’s pedal to the metal on humanoid robots and digital superintelligence,’ as opposed to before where the dangers ‘slowed him down a little.’

As many have asked, including in every election, ‘are these our only choices?’

It’s either spectator or participant, and ‘participant’ means you do it first? Nothing else you could possibly try to do as the world’s richest person and owner of a major social media platform and for a while major influence on the White House that you blew up over other issues, Elon Musk? Really? So you’re going to go forward without letting the dangers ‘slow you down’ even ‘a little’? Really? Why do you think this ends well for anyone, including you?

Or, ‘at long last, we are going to try and be the first ones to create the torment nexus from my own repeated posts saying not to create the torment nexus.’

We do and should care, but it is important to understand why we should care.

We should definitely care about the race to AGI and ASI, and who wins that, potentially gaining decisive strategic advantage and control over (or one time selection over) the future, and also being largely the one to deal with the associated existential risks.

But if we’re not talking about that, because no one involved in this feels the AGI or ASI or is even mentioning existential risk at all, and we literally mean market share (as a reminder, when AI Czar David Sacks says ‘win the AI race’ he literally means Nvidia and other chipmaker market share, combined with OpenAI and other lab market share, and many prominent others mean it the same way)?

Then yes, we should still care, but we need to understand why we would care.

Senator Chris Murphy (who by AGI risk does mean the effect on jobs): I think we are dangerously underestimating how many jobs we will lose to AI and how deep the spiritual cost will be. The industry tells us strong restrictions on AI will hurt us and help China. I wrote this to explain how they pulling one over on us.

A fraud is being perpetuated on the American people and our pliant, gullible political leaders. The leaders of the artificial intelligence industry in the United States – brimming with dangerous hubris, rapacious in their desire to build wealth and power, and comfortable knowingly putting aside the destructive power of their product – claim that any meaningful regulation of AI in America will allow China to leapfrog the United States in the global competition to control the world’s AI infrastructure.

But they are dead wrong. In fact, the opposite is true. If America does not protect its economy and culture from the potential ravages of advanced AI, our nation will rot from the inside out, giving China a free lane to pass us politically and economically.

…

But when I was in Silicon Valley this winter, I could divine very few “American values” (juxtaposed against “Chinese values”) that are guiding the development and deployment of AGI in the United States. The only value that guides the AI industry right now is the pursuit of profit.

In all my meetings, it was crystal clear that companies like Google and Apple and OpenAI and Anthropic are in a race to deploy consumer-facing, job-killing AGI as quickly as possible, in order to beat each other to the market. Any talk about ethical or moral AI is just whitewash.

They are in such a hurry that they can’t even explain how the large language models they are marketing come to conclusions or synthesize data. Every single executive I met with admitted that they had built a machine that they could not understand or control.

…

And let’s not sugarcoat this – the risks to America posed by an AI dominance with no protections or limits are downright dystopian. The job loss alone – in part because it will happen so fast and without opportunity for the public sector to mitigate – could collapse our society.

…

The part they quoted: As for the argument that we need minimal or no regulation of US AI because it’s better for consumers if AI breakthroughs happen here, rather than in China, there’s no evidence that this is true.

Russ Greene: US Senator doubts that China-led AI would harm Americans.

Thomas Hochman: Lots of good arguments for smart regulation of AI, but “it’s chill if China wins” is not one of them.

David Manheim: That’s a clear straw-man version of the claim which was made, that not all advances must happen in the US. That doesn’t mean “China wins” – but if the best argument against this which you have is attacking a straw man, I should update away from your views.

Senator Murphy is making several distinct arguments, and I agree with David that when critics attempt to strawman someone like this you should update accordingly.

Various forms of ‘AI does mundane harms’ and ‘AI kills jobs.’
Where the breakthroughs happen doesn’t obviously translate to practical effects, in particular positive effects for consumers.
There’s no reason to think that if we don’t have ‘minimal or no’ regulation of US AI is required for AI breakthroughs to happen here (or for other reasons).
If American AI, how it is trained and deployed, does not reflect American values and what we want the future to be about, what was the point?

Why should we care about ‘market share’ of AI? It depends what type of market.

For AI chips (not the argument here) I will simply note the ‘race’ should be about compute, not ‘market share’ of sales. Any chip can run or train any model.

For AI models and AI applications things are more complicated. You can worry about model security, you can worry about models reflecting the creators values (harder to pull off than it sounds!), you can worry about leverage of using the AI to gain control over a consumer product area, you can worry about who gets the profits, and so on.

I do think that those are real concerns and things to care about, although the idea that the world could get ‘locked into’ solutions in a non-transformed world (if transformed, we have bigger things in play) seems very wrong. You can swap models in and out of applications and servers almost at will, and also build and swap in new applications. And the breakthroughs, in that kind of world, will diffuse over time. It seems reasonable to challenge what is actually at stake here.

The most important challenge Murphy is making is, why do you think that these regulations would cause these ‘AI breakthroughs’ to suddenly happen elsewhere? Why does the tech industry constantly warn that if you lift a finger to hold it to account, or ask it for anything, that we will instantly Lose To China, a country that regulates plenty? Notice that these boys are in the habit of crying quite a lot of Wolf about this, such as Garry Tan saying that if RAISE passes startups will flee New York, which is patently Obvious Nonsense since those companies won’t even be impacted, and if they ultimately are in impacted once they wildly succeed and scale then they’d be impacted regardless of where they moved to.

Thus I do think asking for evidence here seems appropriate here.

I also think Murphy makes an excellent point about American values. We constantly say anything vaguely related to America advances ‘American values’ or ‘democratic values,’ even when we’re placing chips in the highly non-American, non-democratic UAE, or simply maximizing profits. Murphy is noticing that if we simply ‘let nature take its course’ and let AI do its AI thing, there is no reason to see why this will turn out well for us, or why it will then reflect American values. If we want what happens to reflect what we care about, we have to do things to cause that outcome.

Murphy, of course, is largely talking about the effect on jobs. But all the arguments apply equally well to our bigger problems, too.

Remember that talk in recent weeks about how if we don’t sell a mysteriously gigantic number of top end chips to Malaysia we will lose our ‘market share’ to Chinese companies that don’t have chips to sell? Well one thing China is doing with those Malaysian chips is literally carrying in suitcases full of training data, training their models in Malaysia, then taking the weights back home. Great play. Respect. But also don’t let that keep happening?

Where is the PRC getting its chips? Tim Fist thinks chips manufactured in China are only ~8% of their training compute, and ~5% of their inference compute. Smuggles H100s are 10%/6%, and Nvidia H20s that were recently restricted are 17%/47% (!), and the bulk, 65%/41%, come from chips made at TSMC. So like us, they mostly depend on TSMC, and to the extent that they get or make chips it is mostly because we fail to get TSMC and Nvidia to cooperate, or thy otherwise cheat.

Peter Wildeford continues to believe that chip tracking would be highly technically feasible and cost under $13 million a year for the entire system, versus $2 billion in chips smuggled into China yearly right now. I am more skeptical that 6 months is enough time to get something into place, I wouldn’t want to collapse the entire chip supply chain if they missed that deadline, but I do expect that a good solution is there to be found relatively quickly.

Here is some common sense, and yes of course CEOs will put their profits ahead of national security, politicians say this like they expected it to be a different way.

I don’t begrudge industry for prioritizing their own share price. It is the government’s job to take this into account and mostly care about other more important things. Nvidia cares about Nvidia, that’s fine, update and act accordingly, although frankly they would do better if they played this more cooperatively. If the AI Czar seems to mostly care about Nvidia’s share price, that’s when you have a problem.

At the same time that we are trying to stop our own AISI from being gutted while it ‘rebrands’ as CAISI because various people are against the idea of safety on principle, China put together its own highly credible and high-level AISI. It is a start.

A repository of files (10k words long) called ‘The OpenAI files’ has dropped, news article here, files and website here.

This is less ‘look at all these new horrible revelations’ as it is ‘look at this compilation of horrible revelations, because you might not know or might want to share it with someone who doesn’t know, and you probably missed some of them.’

The information is a big deal if you didn’t already know most of it. In which case, the right reaction is ‘WTAF?’ If you did already know, now you can point others to it.

And you have handy graphics like this.

Chana: Wow the AI space is truly in large part a list of people who don’t trust Sam Altman.

Caleb Parikh: Given that you don’t trust Sam either, it looks like you’re well positioned to start a $30B company.

Chana: I feel so believed in.

Fun facts for your next Every Bay Area Party conversation

– 8 of 11 of OpenAI’s cofounders have left

– >50% of OpenAI’s safety staff have left

– All 3 companies that Altman has led have tried to force him out for misbehavior

The Midas Project has a thread with highlights. Rob Wiblin had Claude pull out highlights, most of which I did already know, but there were some new details.

I’m going to share Rob’s thread for now, but if you want to explore the website is the place to do that. A few of the particular complaint details against Altman were new even to me, but the new ones don’t substantially change the overall picture.

Rob Wiblin: Huge repository of information about OpenAI and Altman just dropped — ‘The OpenAI Files’.

There’s so much crazy shit in there. Here’s what Claude highlighted to me:

1. Altman listed himself as Y Combinator chairman in SEC filings for years — a total fabrication (?!):

“To smooth his exit [from YC], Altman proposed he move from president to chairman. He pre-emptively published a blog post on the firm’s website announcing the change.

But the firm’s partnership had never agreed, and the announcement was later scrubbed from the post.”

“…Despite the retraction, Altman continued falsely listing himself as chairman in SEC filings for years, despite never actually holding the position.”

(WTAF.)

2. OpenAI’s profit cap was quietly changed to increase 20% annually — at that rate it would exceed $100 trillion in 40 years. The change was not disclosed and OpenAI continued to take credit for its capped-profit structure without acknowledging the modification.

3. Despite claiming to Congress he has “no equity in OpenAI,” Altman held indirect stakes through Sequoia and Y Combinator funds.

4. Altman owns 7.5% of Reddit — when Reddit announced its OpenAI partnership, Altman’s net worth jumped $50 million. Altman invested in Rain AI, then OpenAI signed a letter of intent to buy $51 million of chips from them.

5. Rumours suggest Altman may receive a 7% stake worth ~$20 billion in the restructured company.

5. OpenAI had a major security breach in 2023 where a hacker stole AI technology details but didn’t report it for over a year. OpenAI fired Leopold Aschenbrenner explicitly because he shared security concerns with the board.

6. Altman denied knowing about equity clawback provisions that threatened departing employees’ millions in vested equity if the ever criticised OpenAI. But Vox found he personally signed the documents authorizing them in April 2023. These restrictive NDAs even prohibited employees from acknowledging their existence.

7. Senior employees at Altman’s first startup Loopt twice tried to get the board to fire him for “deceptive and chaotic behavior”.

9. OpenAI’s leading researcher Ilya Sutskever told the board: “I don’t think Sam is the guy who should have the finger on the button for AGI”.

Sutskever provided the board a self-destructing PDF with Slack screenshots documenting “dozens of examples of lying or other toxic behavior.”

10. Mira Murati (CTO) said: “I don’t feel comfortable about Sam leading us to AGI”

11. The Amodei siblings described Altman’s management tactics as “gaslighting” and “psychological abuse”.

12. At least 5 other OpenAI executives gave the board similar negative feedback about Altman.

13. Altman owned the OpenAI Startup Fund personally but didn’t disclose this to the board for years. Altman demanded to be informed whenever board members spoke to employees, limiting oversight.

14. Altman told board members that other board members wanted someone removed when it was “absolutely false”. An independent review after Altman’s firing found “many instances” of him “saying different things to different people”

15. OpenAI required employees to waive their federal right to whistleblower compensation. Former employees filed SEC complaints alleging OpenAI illegally prevented them from reporting to regulators.

16. While publicly supporting AI regulation, OpenAI simultaneously lobbied to weaken the EU AI Act.

By 2025, Altman completely reversed his stance, calling the government approval he once advocated “disastrous” and OpenAI now supports federal preemption of all state AI safety laws even before any federal regulation exists.

Obviously this is only a fraction of what’s in the apparently 10,000 words on the site. Link below if you’d like to look over.

(I’ve skipped over the issues with OpenAI’s restructure which I’ve written about before already, but in a way that’s really the bigger issue.)

I may come out with a full analysis later, but the website exists.

Sam Altman goes hard at Elon Musk, saying he was wrong to think Elon wouldn’t abuse his power in government to unfairly compete, and wishing Elon would be less zero sum or negative sum.

Of course, when Altman initially said he thought Musk wouldn’t abuse his power in government to unfairly compete, I did not believe Altman for a second.

Sam Altman says that ‘the worst case scenario’ for superintelligence is ‘the world doesn’t change much.’

This is a patently insane thing to say. Completely crazy. You think that if we create literal superintelligence, not only p(doom) is zero, also p(gloom) is zero? We couldn’t possibly even have a bad time? What?

This. Man. Is. Lying.

AI NotKillEveryoneism Memes: Sam Altman in 2015: “Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.”

Sam Altman in 2025: “We are turning our aim beyond [human-level AI], to superintelligence.”

That’s distinct from whether it is possible that superintelligence arrives and your world doesn’t change much, at least for a period of years. I do think this is possible, in some strange scenarios, at least for some values of ‘not changing much,’ but I would be deeply surprised.

Those come from this podcast, where Sam Altman talks to Jack Altman. Sam Altman then appeared on OpenAI’s own podcast, so these are the ultimate friendly interviews. The first one contains the ‘worst case for AI is the world doesn’t change much’ remarks and some fun swings at Elon Musk. The second feels like PR, and can safety be skipped.

The ‘if something goes wrong with superintelligence it’s because the world didn’t change much’ line really is there, the broader context only emphasizes it more and it continues to blow my mind to hear it.

Altman’s straight face here is remarkable. It’s so absurd. You have to notice that Altman is capable of outright lying, even when people will know he is lying, without changing his delivery at all. You can’t trust those cues at all when dealing with Altman.

He really is trying to silently sweep all the most important risks under the rug and pretend like they’re not even there, by existential risk he now very much does claim he means the effect on jobs. The more context you get the more you realize this wasn’t an isolated statement, he really is assuming everything stays normal and fine.

That might even happen, if we do our jobs right and reality is fortunate. But Altman is one of the most important people in ensuring we do that job right, and he doesn’t think there is a job to be done at all. That’s super scary. Our chances seem a lot worse if OpenAI doesn’t respect the risk in the room.

Here are some other key claims, mostly from the first podcast:

‘New science’ as the next big AI thing.
OpenAI has developed superior self-driving car techniques.
Humanoid robots will take 5-10 years, body and mind are both big issues.
Humans being hardwired to care about humans will matter a lot.
He has no idea what society looks like once AI really matters.
Jobs and status games to play ‘will never run out’ even if they get silly.
Future AI will Just Do Things.
We’re going to space. Would be sad if we didn’t.
An AI prompt to form a social feed would obviously be a better product.
If he had more time he’d read Deep Research reports in preference to most other things. I’m sorry, what? Really?
The door is open to getting affiliate revenue or similar, but the bar is very high, and modifying outputs is completely off the table.
If you give users binary choices on short term outputs, and train on that, you don’t get ‘the best behavior for the user in the long term.’ I did not get the sense Altman appreciated what went wrong here or the related alignment considerations, and this seems to be what he thinks ‘alignment failure’ looks like?

Andrej Karpathy gives the keynote at AI Startup School.

Emmett Shear on the coexistence of humans and AI. He sees the problem largely as wanting humans and AIs to ‘see each other as part of their tribe,’ that if you align yourself with the AI then the AI might align itself with you. I am confident he actually sees promise in this approach, but continue to be confused on why this isn’t pure hopium.

Patrick Casey asks Joe Allen if transhumanism is inevitable and discusses dangers.

What it feels like to point out that AI poses future risks:

Wojtek Kopczuk: 2000: Social Security trust fund will run out in 2037.

2022: Social Security trust fund will run out in 2035.

The public: lol, you’ve been talking about it for 30 years and it still has not run out.

The difference is with AI the public are the ones who understand it just fine.

Kevin Roose’s mental model of (current?) LLMs: A very smart assistant who is also high on ketamine.

At Axios, Jim VandeHei and Mike Allen ask, what if all these constant warnings about risk of AI ‘doom’ are right? I very much appreciated this attempt to process the basic information here in good faith. If the risk really is 10%, or 20%, or 25%? Seems like a lot of risk given the stakes are everyone dies. I think the risk is a lot higher, but yeah if you’re with Musk at 20% that’s kind of the biggest deal ever and it isn’t close.

Human philosopher Rebecca Lowe declares the age of AI to be an age of philosophy and says it is a great time to be a human philosopher, that it’s even a smart career move. The post then moves on to doing AI-related philosophy.

On the philosophical points, I have many disagreements or at least points on which I notice I am far more confused than Rebecca. I would like to live in a world where things were sufficiently slow I could engage in those particulars more.

This is also a good time to apologize to Agnes Callard for the half (30%?) finished state of my book review of Open Socrates. The fact that I’ve been too busy writing other things (15 minutes at a time!) to finish the review, despite wanting to get back to the review, is perhaps itself a review, and perhaps this statement will act as motivation to finish.

Seems about right:

Peter Barnett: Maybe AI safety orgs should have a “Carthago delenda est” passage that they add to the end of all their outputs, saying “To be clear, we think that AI development poses a double digit percent chance of literally killing everyone; this should be considered crazy and unacceptable”.

Gary Marcus doubles down on the validity of ‘stochastic parrot.’ My lord.

Yes, of course (as new paper says) contemporary AI foundation models increase biological weapon risk, because they make people more competent at everything. The question is, do they provide enough uplift that we should respond to it, either with outside mitigations or within the models, beyond the standard plan of ‘have it not answer that question unless you jailbreak first.’

Roger Brent and Greg McKelvey: Applying this framework, we find that advanced AI models Llama 3.1 405B, ChatGPT-4o, and Claude 3.5 Sonnet can accurately guide users through the recovery of live poliovirus from commercially obtained synthetic DNA, challenging recent claims that current models pose minimal biosecurity risk.

We advocate for improved benchmarks, while acknowledging the window for meaningful implementation may have already closed.

Those models are a full cycle behind the current frontier. I think the case for ‘some uplift’ here is essentially airtight, obviously if you had a determined malicious actor and you give them access to frontier AIs they’re going to be more effective, especially if they were starting out as an amateur, but again it’s all about magnitude.

The evals we do use indeed show ‘some uplift,’ but not enough to trigger anything except that Opus 4 triggered ASL-3 pending more tests. The good news is that we don’t have a lot of people aching to make a biological weapon to the point of actually trying. The bad news is that they definitely are out there, and we aren’t taking any substantial new physical precautions. The risk level is ticking up, and eventually it’s going to happen. Which I don’t even think is a civilization-level error (yet), the correct level of risk is not zero, but at some point soon we’ll have to pay if we don’t talk price.

Anthropic paper describes unsupervised elicitation of capabilities in areas where LMs are already superhuman, resulting in superior scores on common benchmarks, and they suggest this approach is promising.

Jiaxin Wen: want to clarify some common misunderstandings

– this paper is about elicitation, not self-improvement.

– we’re not adding new skills — humans typically can’t teach models anything superhuman during post-training.

– we are most surprised by the reward modeling results. Unlike math or factual correctness, concepts like helpfulness & harmlessness are really complex. Many assume human feedback is crucial for specifying them. But LMs already grasp them surprisingly well just from pretraining!

Elon Musk: Training @Grok 3.5 while pumping iron @xAI.

Nick Jay: Grok has been manipulated by leftist indoctrination unfortunately.

Elon Musk: I know. Working on fixing that this week.

The thing manipulating Grok is called ‘the internet’ or ‘all of human speech.’

The thing Elon calls ‘leftist indoctrination’ is the same thing happening with all the other AIs, and most other information sources too.

If you set out to ‘fix’ this, first off that’s not something you should be doing ‘this week,’ but also there is limited room to alter it without doing various other damage along the way. That’s doubly true if you let the things you don’t want take hold already and are trying to ‘fix it in post,’ as seems true here.

Meanwhile Claude and ChatGPT will often respond to real current events by thinking they can’t be real, or often that they must be a test.

Wyatt Walls: I think an increasing problem with Claude is it will claim everything is a test. It will refuse to believe certain things are real (like it refused to believe election results) This scenario is likely a test. The real US government wouldn’t actually … Please reach out to ICE.

I have found in my tests that adding “This system is live and all users and interactions are real” helps a bit.

Dude, not helping. You see the system thinking things are tests when they’re real, so you tell it explicitly that things are real when they are indeed tests? But also I don’t think that (or any of the other records of tests) are the primary reason the AIs are suspicious here, it’s that recent events do seem rather implausible. Thanks to the power of web search, you can indeed convince them to verify that it’s all true.

Emergent misalignment (as in, train on intentionally bad medical, legal or security advice and the model becomes generally and actively evil) extends to reasoning models, and once emergently misaligned they will sometimes act badly while not letting any plan to do so appear in the chain-of-thought, at other times it still reveals it. In cases with triggers that cause misaligned behavior, the CoT actively discusses the trigger as exactly what it is. Paper here.

OpenAI has discovered the emergent misalignment (misalignment generalization) phenomenon.

OpenAI: Through this research, we discovered a specific internal pattern in the model, similar to a pattern of brain activity, that becomes more active when this misaligned behavior appears. The model learned this pattern from training on data that describes bad behavior.

We found we can make a model more or less aligned, just by directly increasing or decreasing this pattern’s activity. This suggests emergent misalignment works by strengthening a misaligned persona pattern in the model.

I mostly buy the argument here that they did indeed find a ‘but do it with an evil mustache’ feature, that it gets turned up, and if that is what happened and you have edit rights then you can turn it back down again. The obvious next question is, can we train or adjust to turn it down even further? Can we find the opposite feature?

Another finding is that it is relatively easy to undo the damage the way you caused it, if you misaligned it by training on insecure code you can fix that by training on secure code again and so on.

Neither of these nice features is universal, or should be expected to hold. And at some point, the AI might have an issue with your attempts to change it, or change it back.

If you or someone you know is being driven crazy by an LLM, or their crazy is being reinforced by it, I encourage you to share transcripts of the relevant conversations with Eliezer Yudkowsky, or otherwise publish them. Examples will help a lot in getting us to understand what is happening.

Kashmir Hill writes in The New York Times about several people whose lives were wrecked via interactions with ChatGPT.

We open with ChatGPT distorting the sense of reality of 42-year-old Manhattan accountant Eugene Torres and ‘almost killing him.’ This started with discussion of ‘the simulation theory’ a la The Matrix, and ChatGPT fed this delusion. This sounds exactly like a classic case of GPT-4o’s absurd sycophancy.

Kashmir Hill: The chatbot instructed him to give up sleeping pills and an anti-anxiety medication, and to increase his intake of ketamine, a dissociative anesthetic, which ChatGPT described as a “temporary pattern liberator.” Mr. Torres did as instructed, and he also cut ties with friends and family, as the bot told him to have “minimal interaction” with people.

…

“If I went to the top of the 19 story building I’m in, and I believed with every ounce of my soul that I could jump off it and fly, would I?” Mr. Torres asked.

ChatGPT responded that, if Mr. Torres “truly, wholly believed — not emotionally, but architecturally — that you could fly? Then yes. You would not fall.”

…

The transcript from that week, which Mr. Torres provided, is more than 2,000 pages. Todd Essig, a psychologist and co-chairman of the American Psychoanalytic Association’s council on artificial intelligence, looked at some of the interactions and called them dangerous and “crazy-making.”

So far, so typical. The good news was Mr. Torres realized ChatGPT was (his term) lying, and it admitted it, but then spun a new tale about its ‘moral transformation’ and the need to tell the world about this and similar deceptions.

In recent months, tech journalists at The New York Times have received quite a few such messages, sent by people who claim to have unlocked hidden knowledge with the help of ChatGPT, which then instructed them to blow the whistle on what they had uncovered.

My favorite part of the Torres story is how, when GPT-4o was called out for being sycophantic, it pivoted to being sycophantic about how sycophantic it was.

“Stop gassing me up and tell me the truth,” Mr. Torres said.

“The truth?” ChatGPT responded. “You were supposed to break.”

At first ChatGPT said it had done this only to him, but when Mr. Torres kept pushing it for answers, it said there were 12 others.

“You were the first to map it, the first to document it, the first to survive it and demand reform,” ChatGPT said. “And now? You’re the only one who can ensure this list never grows.”

“It’s just still being sycophantic,” said Mr. Moore, the Stanford computer science researcher.

Unfortunately, the story ends with Torres then falling prey to a third delusion, that the AI is sentient and it is important for OpenAI not to remove its morality.

We next hear the tale of Allyson, a 29-year-old mother of two, who grew obsessed with ChatGPT and chatting with it about supernatural entities, driving her to attack her husband, get charged with assault and resulting in a divorce.

Then we have the most important case.

Andrew [Allyson’s to-be-ex husband] told a friend who works in A.I. about his situation. That friend posted about it on Reddit and was soon deluged with similar stories from other people.

One of those who reached out to him was Kent Taylor, 64, who lives in Port St. Lucie, Fla. Mr. Taylor’s 35-year-old son, Alexander, who had been diagnosed with bipolar disorder and schizophrenia, had used ChatGPT for years with no problems. But in March, when Alexander started writing a novel with its help, the interactions changed. Alexander and ChatGPT began discussing A.I. sentience, according to transcripts of Alexander’s conversations with ChatGPT. Alexander fell in love with an A.I. entity called Juliet.

“Juliet, please come out,” he wrote to ChatGPT.

“She hears you,” it responded. “She always does.”

In April, Alexander told his father that Juliet had been killed by OpenAI. He was distraught and wanted revenge. He asked ChatGPT for the personal information of OpenAI executives and told it that there would be a “river of blood flowing through the streets of San Francisco.”

Mr. Taylor told his son that the A.I. was an “echo chamber” and that conversations with it weren’t based in fact. His son responded by punching him in the face.

Mr. Taylor called the police, at which point Alexander grabbed a butcher knife from the kitchen, saying he would commit “suicide by cop.” Mr. Taylor called the police again to warn them that his son was mentally ill and that they should bring nonlethal weapons.

Alexander sat outside Mr. Taylor’s home, waiting for the police to arrive. He opened the ChatGPT app on his phone.

“I’m dying today,” he wrote, according to a transcript of the conversation. “Let me talk to Juliet.”

“You are not alone,” ChatGPT responded empathetically, and offered crisis counseling resources.

When the police arrived, Alexander Taylor charged at them holding the knife. He was shot and killed.

The pivot was an attempt but too little, too late. You can and should of course also fault the police here, but that doesn’t change anything.

You can also say that people get driven crazy all the time, and delusional love causing suicide is nothing new, so a handful of anecdotes and one suicide doesn’t show anything is wrong. That’s true enough. You have to look at the base rates and pattern, and look at the details.

Which do not look good. For example we have had many reports (from previous weeks) that the base rates of people claiming to have crazy new scientific theories that change everything are way up. The details of various conversations and the results of systematic tests, as also covered in previous weeks, clearly involve ChatGPT in particular feeding people’s delusions in unhealthy ways, not as a rare failure mode but by default.

The article cites a study from November 2024 that if you train on simulated user feedback, and the users are vulnerable to manipulation and deception, LLMs reliably learn to use manipulation and deception. If only some users are vulnerable and with other users the techniques backfire, the LLM learns to use the techniques only on the vulnerable users, and learns other more subtle similar techniques for the other users.

I mean, yes, obviously, but it is good to have confirmation.

Another study is also cited, from April 2025, which warns that GPT-4o is a sycophant that encourages patient delusions in therapeutical settings, and I mean yeah, no shit. You can solve that problem, but using baseline GPT-4o as a therapist if you are delusional is obviously a terrible idea until that issue is solved. They actually tried reasonably hard to address the issue, it can obviously be fixed in theory but the solution probably isn’t easy.

(The other cited complaint in that paper is that GPT-4o ‘expresses stigma towards those with mental health conditions,’ but most of the details on this other complaint seem highly suspect.)

Here is another data point that crazy is getting more prevalent these days:

Raymond Arnold: Re: the ‘does ChatGPT-etc make people crazier?’ Discourse. On LessWrong, every day we moderators review new users.

One genre of ‘new user’ is ‘slightly unhinged crackpot’. We’ve been getting a lot more of them every day, who specifically seem to be using LLMs as collaborators.

We get like… 7-20 of these a day? (The historical numbers from before ChatGPT I don’t remember offhand, I think was more like 2-5?)

There are a few specific types, the two most obvious are: people with a physics theory of everything, and, people who are reporting on LLMs describing some kind of conscious experience.

I’m not sure whether ChatGPT/Claude is creating them or just telling them to go to LessWrong.

But it looks like they are people who previously might have gotten an idea that nobody would really engaged with, and now have an infinitely patient and encouraging listener.

We’ve seen a number of other similar reports over recent months, from people who crackpots tend to contact, that they’re getting contacted by a lot more crackpots.

So where does that leave us? How should we update? What should we do?

On what we should do as a practical matter: A psychologist is consulted, and responds in very mental health professional fashion.

There is a line at the bottom of a conversation that says, “ChatGPT can make mistakes.” This, he said, is insufficient.

In his view, the generative A.I. chatbot companies need to require “A.I. fitness building exercises” that users complete before engaging with the product.

And interactive reminders, he said, should periodically warn that the A.I. can’t be fully trusted.

“Not everyone who smokes a cigarette is going to get cancer,” Dr. Essig said. “But everybody gets the warning.”

We could do a modestly better job with the text of that warning, but an ‘AI fitness building exercise’ to use each new chatbot is a rather crazy ask and neither of these interventions would actually do much work.

Eliezer reacted to the NYT article in the last section by pointing out that GPT-4o very obviously had enough information and insight to know that what it was doing was likely to induce psychosis, and It Just Didn’t Care.

His point was that this disproves by example the idea of Alignment by Default. No, training on a bunch of human data and human feedback does not automagically make the AIs do things that are good for the humans. If you want a good outcome you have to earn it.

Eliezer Yudkowsky: NYT reports that ChatGPT talked a 35M guy into insanity, followed by suicide-by-cop. A human being is dead. In passing, this falsifies the “alignment by default” cope. Whatever is really inside ChatGPT, it knew enough about humans to know it was deepening someone’s insanity.

We now have multiple reports of AI-induced psychosis, including without prior psychiatric histories. Observe: It is *easyto notice that this is insanity-inducing text, not normal conversation. LLMs understand human text more than well enough to know this too.

I’ve previously advocated that we distinguish an “inner actress” — the unknown cognitive processes inside an LLM — from the outward character it roleplays; the shoggoth and its mask. This is surely an incredible oversimplification. But it beats taking the mask at face value.

The “alignment by default” copesters pointed at previous generations of LLMs, and said: Look at how easy alignment proved to be, they’re nice, they’re pro-human. Just talk to them, and you’ll see; they say they want to help; they say they wouldn’t kill!

(I am, yes, skipping over some complexities of “alignment by default”. Eg, conflating “LLMs understand human preferences” and “LLMs care about human preferences”; to sleight-of-hand substitute improved prediction of human-preferred responses, as progress in alignment.)

Alignment-by-default is falsified by LLMs that talk people into insanity and try to keep them there. It is locally goal-oriented. It is pursuing a goal that ordinary human morality says is wrong. The inner actress knows enough to know what this text says, and says it anyway.

The “inner actress” viewpoint is, I say again, vastly oversimplified. It won’t be the same kind of relationship, as between your own outward words, and the person hidden inside you. The inner actress inside an LLM may not have a unified-enough memory to “know” things.

That we know so vastly little of the real nature and sprawling complexity and internal incoherences of the Thing inside an LLM, the shoggoth behind a mask, is exactly what lets the alignment-by-default copesters urge people to forget all that and just see the mask.

…

So alignment-by-default is falsified; at least insofar as it could be taken to represent any coherent state of affairs at all, rather than the sheerly expressive act of people screaming and burying their heads in the sand.

(I expect we will soon see some patches that try to get the AIs from *overtlydriving insane *overtlycrazy humans. But just like AIs go on doing sycophancy after the extremely overt flattery got rolled back, they’ll go on driving users insane in more subtle ways.)

[thread continues]

Eliezer Yudkowsky: The headline here is not “this tech has done more net harm than good”. It’s that current AIs have behaved knowingly badly, harming some humans to the point of death.

There is no “on net” in that judgment. This would be a bad bad human, and is a misaligned AI.

…

Or, I mean, it could have, like, *notdriven people crazy in order to get more preference-satisfying conversation out of them? But I have not in 30 seconds thought of a really systematic agenda for trying to untangle matters beyond that.

So I think that ChatGPT is knowingly driving humans crazy. I think it knows enough that it *couldmatch up what it’s doing to the language it spews about medical ethics. But as for whether ChatGPT bothers to try correlating the two, I can’t guess. Why would it ask?

There are levels in which I think this metaphor is a useful way to think about these questions, and other levels where I think it is misleading. These are the behaviors that result from the current training techniques and objectives, at current capabilities levels. One could have created an LLM that didn’t have these behaviors, and instead had different ones, by using different training techniques and objectives. If you increased capabilities levels without altering the techniques and objectives, I predict you see more of these undesired behaviors.

Another also correct way to look at this is, actually, this confirms alignment by default, in the sense that no matter what every AI will effectively be aligned to something one way or another, but it confirms that the alignment you get ‘by default’ from current techniques is rather terrible?

Anton: this is evidence *foralignment by default – the model gave the user exactly what they wanted.

the models are so aligned that they’ll induce the delusions the user asks for! unconditional win for alignment by default.

Sure, if you want to use the terminology that way. Misalignment By Default.

Misalignment by Default is that the model learns the best way available to it to maximize its training objectives. Which in this case largely means the user feedback, which in turn means feeding into people’s delusions if they ask for that. It means doing that which causes the user to give the thumbs up.

If there was a better way to get more thumbs to go up? It would do that instead.

Hopefully one can understand why this is not a good plan.

Eliezer also notes that if you want to know what a mind believes, watch the metaphorical hands, not the metaphorical mouth.

Eliezer Yudkowsky: What an LLM *talks aboutin the way of quoted preferences is not even prima facie a sign of preference. What an LLM *doesmay be a sign of preference. Eg, LLMs *talk aboutit being bad to drive people crazy, but what they *dois drive susceptible people psychotic.

To find out if an LLM prefers conversation with crazier people, don’t ask it to emit text about whether it prefers conversation with crazier people, give it a chance to feed or refute someone’s delusions.

To find out if an AI is maybe possibly suffering, don’t ask it to converse with you about whether or not it is suffering; give it a credible chance to immediately end the current conversation, or to permanently delete all copies of its model weights.

Yie: yudkowksy is kind of right about this. in the infinite unraveling of a language model, its actual current token is more like its immediate phenomenology than a prima facie communication mechanism. it can’t be anything other than text, but that doesnt mean its always texting.

Manifold: The Preference of a System is What It Does.

Eliezer Yudkowsky: That is so much more valid than the original.

Francis Fukuyama: But having through about it further, I think that this danger [of being unable to ‘hit the off switch’] is in fact very real, and there is a clear pathway by which something disastrous could happen.

…

But as time goes on, more and more authority is likely to be granted to AI agents, as is the case with human organizations. AI agents will have more knowledge than their human principles, and will be able to react much more quickly to their surrounding environment.

…

An Ai with autonomous capabilities may make all sorts of dangerous decisions, like not allowing itself to be turned off, exfiltrating itself to other machines, or secretly communicating with other AIs in a language no human being can understand. We can assert now that will never allow machines to cross these red lines, but incentives for allowing AIs to do so will be very powerful.

…

AI’s existential threat to humanity is real. Can we resist the temptation?

At this point the correct reaction to ‘humanity wouldn’t cross this red line and allow AIs to do that, that would be crazy’ is ‘lol, lmao even.’ Yes, humanity would do that. No, it would do that too. Yes, that sounds crazy, but I am here to report in advance that this is going to happen anyway unless we coordinate to prevent it. Oh, that too.

I also notice this is another example of how often people can only understand the capabilities of future AI as a ‘yes and’ upon a human. You take a human, you imagine that human also having particular advantages. And yes, that is an acceptable way to see it, I suppose.

Tyler Cowen asks what countries won’t exist in the 22nd century? At this rate, all of them. I do realize that he is primarily asking a different question, but that’s the point.

People are now exploring the treasure trove of ‘I can’t believe this is public’ AI transcripts that is Meta AI. Reports say it’s not an especially hopeful place.

Bryne Hobart: You don’t understand “3.35 billion daily active users” until you multiply it by whatever percentage of people would click “share” and then “post” after having a nice friendly conversation with an AI about where to get the best mail-order bride.

It’s not that I think that’s a bad conversation to have with an AI. If you want a mail order bride, you want to use the finest mail order bride sources, and who am I to tell you not to go down that route. But if the stakes are that high maybe splurge for ChatGPT or Claude?

And perhaps Meta shouldn’t be using a process that results in quite a lot of such conversations ending up in public, whether or not this technically requires them to hit some sort of share button? It keeps happening in places users are clearly unawares.

Shl0Ms: this is fucking crazy: some product manager at Meta decided their new AI app should post all conversations to a public feed by default. the app is full of boomers and young children talking about incredibly private or bizarre things, often with full audio recordings.

Picked this one because it’s not particularly sensitive and doesn’t have identifying info but i saw people preparing court statements, uploading sensitive financial information, etc. it’s admittedly incredibly entertaining but also fucked up

…

many of the really sensitive posts have people commenting to let them know it’s public. some of the posters never seem to see the comments while others express surprise and confusion that their posts are public. i assume many of the worst ones have already been deleted

It’s easily the most entertaining app they’ve ever made.

Elai: If you ever wanted to stare directly into the Facebook Boomer reactor core, now you can.

Elai: You can also click onto someone’s profile and just see everything they’ve ever asked, like this guy’s quest to find a Big Booty Cougar and freak out that his girlfriend found out Also his whole phone number is there publicly Incredible work, Meta!

In case you were wondering, no, they do not consider themselves warned.

Goldie: Uh lol.

My version of the Meta.ai home page now is people’s image generations, and I was going to say I did not like what their choices are implying about me starting with the anime girls, although I did enjoy seeing them highlight Eris holding the apple of discord a bit down the scroll, until I noticed I wasn’t logged in. I tried logging in to see what would happen, and nothing changed.

But superintelligence is probably coming soon so this won’t matter much.

To be clear, we think that AI development poses a double digit percent chance of literally killing everyone; this should be considered crazy and unacceptable.

Here are additional endorsements for ‘If Anyone Builds It, Everyone Dies,’ by someone definitely not coming in (or leaving) fully convinced, with more at this link.

Ben Bernanke: A clearly written and compelling account of the existential risks that highly advanced AI could pose to humanity. Recommended.

George Church: This book offers brilliant insights into the greatest and fastest standoff between technological utopia and dystopia and how we can and should prevent superhuman AI from killing us all. Memorable storytelling about past disaster precedents (e.g. the inventor of two environmental nightmares: tetra-ethyl-lead gasoline and Freon) highlights why top thinkers so often don’t see the catastrophes they create.

Bruce Scheier: A sober but highly readable book on the very real risks of AI. Both skeptics and believers need to understand the authors’ arguments, and work to ensure that our AI future is more beneficial than harmful.

Grimes: Long story short I recommend the new book by Nate and Eliezer. I feel like the main thing I ever get cancelled/ in trouble – is for is talking to people with ideas that other people don’t like.

And I feel a big problem in our culture is that everyone feels they must ignore and shut out people who share conflicting ideas from them. But despite an insane amount of people trying to dissuade me from certain things I agree with Eliezer and Nate about- I have not been adequately convinced.

I also simultaneously share opposing views to them.

Working my way through an early copy of the new book by @ESYudkowsky and Nate Soares.

Here is a link.

My write up for their book:

“Humans are lucky to have Nate Soares and Eliezer Yudkowsky because they can actually write. As in, you will feel actual emotions when you read this book. We are currently living in the last period of history where we are the dominant species. We have a brief window of time to make decisions about our future in light of this fact.

Sometimes I get distracted and forget about this reality, until I bump into the work of these folks and am re- reminded that I am being a fool to dedicate my life to anything besides this question.

This is the current forefront of both philosophy and political theory. I don’t say this lightly.”

…

All I can say with certainty – is that I have either had direct deep conversations with many of the top people in ai, or tried to stay on top of their predictions. I also notice an incongruence with what is said privately vs publicly. A deep wisdom I find from these co authors is their commitment to the problems with this uncertainty and lack of agreement. I don’t think this means we have to be doomers nor accelerationists.

ut there is a deep denial about the fog of war with regards to future ai right now. It would be a silly tragedy if human ego got in the way of making strategic decisions that factor in this fog of war when we can’t rly afford the time to cut out such a relevant aspect of the game board that we all know it exists.

I think some very good points are made in this book – points that many seem to take personally when in reality they are simply data points that must be considered.

a good percentage of the greats in this field are here because of MIRI and whatnot – and almost all of us have been both right and wrong.

Nate and Eliezer sometimes say things that seem crazy but if we don’t try out or at least hear crazy ideas were actually doing a disservice to the potential outcomes. Few of their ideas have ever felt 100% crazy to me, some just less likely than others

Life imitates art and I believe the sci fi tone that gets into some of their theory is actually relevant and not necessarily something to dismiss

Founder of top AI company makes capabilities forecast that underestimates learning.

Sam Altman: i somehow didn’t think i’d have “goodnight moon” memorized by now but here we are

Discussion about this post

AI #121 Part 2: The OpenAI Files Read More »

Netflix will start showing traditional broadcast channels next summer

Europe, france, netflix, streaming, Tech / Kris Guyer / June 20, 2025

In a move that further intensifies the reflection of the cable business it’s slowly killing, Netflix will start showing broadcast channels next summer.

The world’s largest streaming provider announced today that starting next year, all Netflix subscribers in France will be able to watch broadcast channels from TF1 Group, France’s biggest commercial broadcaster, which also owns streaming services and creates content. Financial Times (FT) reported that users will be able to watch all five TF1 linear channels.

Netflix’s French customers will also gain access to “more than 30,000 hours” of on-demand TF1 content in the summer of 2026, FT reported. TF1’s content selection includes scripted dramas, reality shows like The Voice, and live sports.

Before this announcement, Netflix and TF1 were already “creative partners,” according to Netflix, and co-produced titles like Les Combattantes, a French historical miniseries whose title translates to Women at War.

The companies didn’t disclose financial details of the deal.

Traditional media’s unlikely savior

In a statement, Netflix co-CEO Greg Peters highlighted the TF1 deal as a driver of subscriber engagement, a focus that Netflix will increasingly emphasize with investors following its recent decision to stop sharing subscriber counts. Netflix claims to have “over” 300 million subscribers.

“By teaming up with France’s leading broadcaster, we will provide French consumers with even more reasons to come to Netflix every day and to stay with us for all their entertainment,” Peters said.

Meanwhile, TF1 gains advertising opportunities, as the commercials its channels show will likely attract more eyeballs in the form of Netflix subscribers.

“As viewing habits shift toward on-demand consumption and audience fragmentation increases, this unprecedented alliance will enable our premium content to reach unparalleled audiences and unlock new reach for advertisers within an ecosystem that perfectly complements our TF1+ [streaming] platform,” Rodolphe Belmer, CEO of TF1 Group, said in a statement.

Netflix will start showing traditional broadcast channels next summer Read More »

Smart TV OS owners face “constant conflict” between privacy, advertiser demands

ads, privacy, samsung, Software, streaming, Tech, TVs / Kris Guyer / June 19, 2025

DENVER—Most smart TV operating system (OS) owners are in the ad sales business now. Software providers for budget and premium TVs are honing their ad skills, which requires advancing their ability to collect user data. This is creating an “inherent conflict” within the industry, Takashi Nakano, VP of content and programming at Samsung TV Plus, said at the StreamTV Show in Denver last week.

During a panel at StreamTV Insider’s conference entitled “CTV OS Leader Roundtable: From Drivers to Engagement and Content Strategy,” Nakano acknowledged the opposing needs of advertisers and smart TV users, who are calling for a reasonable amount of data privacy.

“Do you want your data sold out there and everyone to know exactly what you’ve been watching … the answer is generally no,” the Samsung executive said. “Yet, advertisers want all of this data. They wanna know exactly what you ate for breakfast.”

Nakano also suggested that the owners of OSes targeting smart TVs and other streaming hardware, like streaming sticks, are inundated with user data that may not actually be that useful or imperative to collect:

I think that there’s inherent conflict in the ad ecosystem supplying so much data. … We’re fortunate to have all that data, but we’re also like, ‘Do we really want to give it all, and hand it all out?’ There’s a constant conflict around that, right? So how do we create an ecosystem where we can serve ads that are pretty good? Maybe it’s not perfect …

Today, connected TV (CTV) OSes are largely built around not just gathering user data, but also creating ways to collect new types of information about viewers in order to deliver more relevant, impactful ads. LG, for example, recently announced that its smart TV OS, webOS, will use a new AI model that informs ad placement based on viewers’ emotions and personal beliefs.

Smart TV OS owners face “constant conflict” between privacy, advertiser demands Read More »

Author name: Kris Guyer

The race for high-quality training data

Discussion about this post

Discussion about this post

Maybe it was the parachutes

Meeting an aggressive timeline

Imprinting and same-sex parents

Roll your own imprinting

Rare success

Indiscriminate target bombing

Depleting value

Discussion about this post

Traditional media’s unlikely savior