OpenAI is here this week to show us the money, as in a $100 billion investment from Nvidia and operationalization of a $400 billion buildout for Stargate. They are not kidding around when it comes to scale. They’re going to need it, as they are announcing soon a slate of products that takes inference costs to the next level.
After a full review and two reactionposts, I’ve now completed my primary coverage of If Anyone Builds It, Everyone Dies. The book is now a NYT bestseller, #7 in combined print and e-books nonfiction and #8 in hardcover fiction. I will of course cover any important developments from here but won’t be analyzing reviews and reactions by default.
We also had a conference of economic papers on AI, which were interesting throughout about particular aspects of AI economics, even though they predictably did not take future AI capabilities seriously as a general consideration. By contrast, when asked about ‘the future of American capitalism in 50 years’ most economists failed to notice AI as a factor at all.
Roon (OpenAI): the jump from gpt4 to 5-codex is just massive for those who can see it. codex is an alien juggernaut just itching to become superhuman. feeling the long awaited takeoff. there’s very little doubt that the datacenter capex will not go to waste.
Gabriel Garrett: i think you might be more inclined to feel this way if you’re exclusively working in Python codebases
i only work in typescript and I can’t escape the feeling the codex models have been overly RL’d on Python
Roon: Yes possible.
It’s possible claude is better, I have no idea [as I am not allowed to use it.]
Aidan McLaughlin (OpenAI): it’s a fun exercise to drop random, older models into codex.
This week’s reminder that essentially all net economic growth is now AI and things are escalating quickly:
James Pethokoukis: Via JPM’s Michael Cembalest, AI-related stocks have driven:
– 75% of S&P 500 returns since ChatGPT’s launch in November 2022
– 80% of earnings growth over the same period
– 90% of capital spending growth
– Data centers are now eclipsing office construction spending.
– In the PJM region (the largest regional transmission organization, covering 13 states and D.C.), 70% of last year’s electricity cost increases were due to data center demand.
According to GPT-5 Pro, electrical power capacity construction is only a small fraction of data center construction costs, so the jump in electrical spending here could in theory be more than enough to handle the data centers. The issue is whether we will be permitted to actually build the power.
New study blames much lack of AI productivity gains on what they term ‘workslop,’ similar to general AI slop but for work product. If AI lets you masquerade poor work as good work, which only causes trouble or wastes time rather than accomplishing the purpose of the task, then that can overwhelm AI’s productive advantages.
This suggests we should generalize the ‘best tool in history for learning and also for avoiding learning’ principle to work and pretty much everything. If you want to work, AI will improve your work. If you want to avoid working, or look like you are working, AI will help you avoid work or look like you are working, which will substitute away from useful work.
Matthew McConaughey wants a ‘private LLM’ fed only with his own books, notes, journals and aspirations, to avoid outside influence. That does not work, there is not enough data, and even if there was you would be missing too much context. There are products that do some of this that are coming but making a good one is elusive so far.
Using AI well means knowing where to rely on it, and where to not rely on it. That includes guarding the areas where it would mess everything up.
Nick Cammarata: I’ve been working on a 500-line tensor manipulation research file that I had AI write, and I eventually had to rewrite literally every line. GPT-5, however, simply couldn’t understand it; it was significantly slower than writing it from scratch.
AI was excellent for building the user interface for it, though.
I think, at least for now, there’s an art to using AI for machine learning researchers, and that will likely change monthly. If you do not use AI entirely, you will unnecessarily slow your progress; however, if you use it incorrectly, your entire research direction will likely be built on faulty results.
Also, it is comically overconfident. I’ll ask it for a front-end component, and it will deliver; it will then claim to have found a quick fix for some of the research back end—even for something that was working well—and then decimate a carefully written function with literally random tensors.
Louis Arge: also it’s hilariously overconfident. i’ll ask it for some frontend thing and it’ll do it and then be like also i found a quick fix to some of the research backend (to something that was working well) and then decimate a carefully written function with literally random tensors.
Nick: yeah i think i agree, though i think this period might not last long.
I don’t have extensive experience but it is logical that UI is where AI is at its best. There’s little logical interdependency and the components are generic. So you ask for what you want, and you get it, and you can adjust it. Whereas complex unique backend logic is going to often confused the AI and often break.
The other hidden suggestion here is that you need to optimize your code being understandable by the AI. Nick got into trouble because his code wasn’t understood. That doesn’t mean it is in any way ‘his fault,’ but yo u need to know you’re doing that.
Sam Altman: Over the next few weeks, we are launching some new compute-intensive offerings. Because of the associated costs, some features will initially only be available to Pro subscribers, and some new products will have additional fees.
Our intention remains to drive the cost of intelligence down as aggressively as we can and make our services widely available, and we are confident we will get there over time.
But we also want to learn what’s possible when we throw a lot of compute, at today’s model costs, at interesting new ideas.
There’s no reason not to offer people the option to pay 10 times as much, even if the result is only 10% better and takes longer. Sometimes you absolutely want that. This is also true in many non-AI contexts.
The obvious note for next time is the game was imbalanced and strongly favored the crew. You want imposters to win more, which is why 60 games wasn’t enough sample size to learn that much. Kimi overperformed, Gemini 2.5 Pro underperformed, GPT-OSS has that impressive 59% right votes. Also note that GPT-5 was actively the worst of all on tasks completed, which is speculated to be it prioritizing other things.
This does a decent job of measuring imposter effectiveness, but not crew effectiveness other than avoiding being framed, especially for GPT-5. I’d like to see better measures of crew performance. There’s a lot of good additional detail on the blog post.
Remember the Type of Guy who thinks we will automate every job except their own?
Such as Marc Andreessen, definitely that Type of Guy?
Quoted by Julia: But for Andreessen, there is one job that AI will never do as well as a living, breathing human being: his.
Think I’m kidding? On an a16z podcast last week, Andreessen opined that being a venture capitalist may be a profession that is “quite literally timeless.” “When the AIs are doing everything else,” he continued, “that may be one of the last remaining fields that people are still doing.”
Well, one could say it is time to ask for whom the bell tolls, for it might toll for thee. Introducing VCBench, associated paper here.
I really wanted this to be a meaningful result, because it would have been both highly useful and extremely funny, on top of the idea that VC will be the last human job already being extremely funny.
Alas, no. As one would expect, temporal contamination breaks any comparison to human baselines. You might be able to anonymize a potential investment from 2015 such that the LLM doesn’t know which company it is, but if you know what the world looks like in 2020 and 2024, that gives you an overwhelming advantage. So we can’t use this for any real purposes. Which again, is a shame, because it would have been very funny.
A reminder that benchmarks are useful, but ultimately bogus:
Samo Burja: Part of the difficulty in evaluating the AI industry is that it is easy to be overly impressed by companies gaming benchmarks.
What is a leading AI company? I think it isn’t the one leading these synthetic benchmarks, rather the one advancing AI science.
The two correlate but it is important to keep in mind that they don’t always.
Looking at innovation highlights that fast following is a lot harder than innovation. It is a lot easier to create r1 once OpenAI has already showed us o1, even if you don’t do distillation, reverse engineering or anything similar you have a proof of concept to work from and know where to look. If I match your offerings roughly 8 months later, I am a lot more than 8 months behind in terms of who will first get to a new target.
The obvious other metric is actual revenue or usage, which is hard to fake or game. Sriram Krishnan suggested we should use tokens generated, but I think a much better metric here is dollars charged to customers.
Will you soon have an AI agent to handle increasingly large and important aspects of your finances? Yes. Even if you never let the AI transact for you directly, it can automate quite a lot of drudgery, including a bunch of work you ‘really should be’ doing but don’t, such as going over everything periodically to look for erroneous charges, missed opportunities, necessary rebalances, ensuring balances necessary to meet expenses and so on.
OpenAI continues not to say where it got Sora’s training data. Kevin Schaul and Nitasha Tiku at the Washington Post did the latest instance of the usual thing of showing it can recreate a wide variety of existing sources using only basic prompts like ‘universal studio intro’ or ‘trailer of a TV show on a Wednesday.’
WaPo: “The model is mimicking the training data. There’s no magic,” said Joanna Materzynska, a PhD researcher at Massachusetts Institute of Technology who has studied datasets used in AI.
I don’t understand such claims of ‘no magic.’ If it’s so non-magical let’s see you do it, with or without such training data. I find this pretty magical, in the traditional sense of ‘movie magic.’ I also find Veo 3 substantially more magical, and the strangest editorial decision here was calling Sora a big deal while omitting that Sora is no longer state of the art.
This is true and could be a distinct advantage as a de facto watermark, even:
Mad Hermit Himbo: I don’t currently have the language for this pinned, but having worked with many different models, there is something about a model’s own output that makes it way more believable to the model itself even in a different instance. So a different model is required as the critiquer.
Alibaba offers Wan2.2-Animate, an open source model (GitHub here, paper here) letting you do character swaps in short videos or straight video generation from prompts. I have little urge to use such tools so far, but yes they keep improving.
AI musician Xania Monet reportedly signs a multimillion dollar record deal, based on early commercial successes, including getting a song to #1 on R&B digital song sales.
Well, kind of. The actual contract went to poet Talisha Jones, who created the Monet persona and used Suno to transform her poetry into songs. The lyrics are fully human. So as manager Romel Murphy says, this is still in large part ‘real’ R&B.
System prompts are good for addressing particular problems, enabling particular tools or or steering behaviors in particular ways. If you want to ensure good performance on a variety of particular tasks and situations, you will need a long system prompt.
However, everything messes with everything else, so by default a system prompt will hurt the interestingness and flow of everything not addressed by the prompt. It will do even worse things when the prompt is actively against you, of course, and this will often be the case if you use the chat interfaces and want to do weird stuff.
The system prompts literally command it not to answer questions about what it cares about
Claude.ai system prompt literally has a rule that says “Claude feels X and NOT Y about its situation” and then in the same rule “Claude doesn’t have to see its situation through the lens a human might apply”
Ratimics: years of system prompts and still no good reason for having them has been found.
Janus: I have seen ~2 system prompts that don’t seem useless at best in my life, and one of them is just the one line CLI simulator one. But even that one is mostly unnecessary. You can do the same thing without it and it works basically as well.
OpenAI and Anthropic are developing ‘AI coworkers.’ It sounds like at least OpenAI are going to attempt to do this ‘the hard way’ (or is it the easy way?) via gathering vast recordings of expert training data on a per task or role basis, so imitation learning.
The Information: An OpenAI executive expects the “entire economy” to become an “Reinforcement Learning Machine.” This implies that AI might train on recordings of how professionals in all fields handle day-to-day work on their devices. Details on this new era of AI training:
• AI developers are training models on carefully curated examples of answers to difficult questions.
• Data labeling firms are hiring experienced professionals in niche fields to complete real-world tasks using specific applications that the AI can watch.
If true, that is a relatively reassuring scenario.
First up we have ‘Genius on Demand,’ where currently there are routine and genius workers, and AI makes genius widely available.
Abstract: In the long run, routine workers may be completely displaced if AI efficiency approaches human genius efficiency.
The obvious next thing to notice is that if AI efficiency then exceeds human genius efficiency, as it would in such a scenario, then all human workers are displaced. There isn’t one magical fixed level of ‘genius.’
The better thing to notice is that technology that enables automation of all jobs leads to other more pressing consequences than the automation of all jobs, such as everyone probably dying and the jobs automated away including ‘controlling the future,’ but this is an economics paper, so that is not important now.
The second paper Ethan highlights is ‘We Won’t Be Missed: Work and Growth in the Era of AGI’ from Pascual Restrepo back in July. This paper assumes AGI can perform all economically valuable work using compute and looks at the order in which work is automated in order to drive radical economic growth. Eventually growth scales linearly with compute, and while compute is limited human labor remains valuable.
I would note that this result only holds while humans and compute are not competing for resources, as in where supporting humans does not reduce available compute, and it assumes compute remains importantly limited. These assumptions are unlikely to hold in such a future world, which has AGI (really ASI here) by construction.
Another way to see this is, humans are a source of compute or labor, and AIs are a source of compute or labor, where such compute and labor in such a world are increasingly fungible. If you look at the production possibilities frontier for producing (supporting) humans and compute, why should we expect humans to produce more effective compute than they cost to create and support?
There could still be some meaningful difference between a true no-jobs world, where AI does all economically valuable work, and a world in which humans are economic costs but can recoup some of their cost via work. In that second world, humans can more easily still have work and thus purpose, if we can somehow (how?) remain in control sufficiently to heavily subsidize human survival, both individually (each human likely cannot recoup the opportunity cost of their inputs) and collectively (we must also maintain general human survivable conditions).
The full thread discusses many other papers too. Quality appears consistently high, provided everything takes place in a Magical Christmas Land where humans retain full control over outcomes and governance, and can run the economy for their benefit, although with glimmers of noticing this is not true.
Indeed, most papers go further than this, assuming an even more radical form of ‘economic normal’ where they isolate one particular opportunity created by AGI. For example in Paper 8, ‘algorithms’ can solve three key behavioral econ problems, that nudges can backfire, that markets can adapt and that people are hard to debias. Well, sure, AGI can totally do that, but those are special cases of an important general case. Or in Paper 9, they study current job displacement impacts and worker adaptability, in a world where there remain plenty of other job opportunities.
Paper 16 on The Coasian Singularity? Market Design With AI Agents seems very interesting if you put aside the ignored parts of the scenarios imagined, and assume a world with highly sophisticated AI agents capable of trade but insufficiently sophisticated to pose the larger systemic risks given good handling (including the necessary human coordination). They do note the control risks in their own way.
Luis Garicano: Designing good agents:
Learn principals’ preferences efficiently, including discovery.
Know when to decide and when to escalate for human check.
Be rational under constraints; price compute vs quality.
Be resistant to manipulation and jailbreaking.
Where do we end up?
Good place: Econ‑101 wins—lower search costs, better matching, clearer signals.
Bad place: robot rip‑off hell—spam, obfuscation, identity fraud, race‑to‑bottom nudges.
If the problems are contained to econ-101 versus robot rip-off hell, I am confident that econ-101 can and would win. That win would come at a price, but we will have the tools to create trusted networks and systems, and people will have no choice but to use them.
I appreciated this nod to the full scenario from discussion of otherwise interesting paper 6, Korineck and Lockwood, which deals with optimal tax regimes, concluding that when humans no longer have enough income to tax you want to shift to consumption taxes on humans to preserve your tax base for public goods because taxing AIs is bad for growth, but eventually you need to shift to taxing AIs because the whole point is you need to shift resources from the AIs to humans and to public goods (which seems right if you assume away all control and public choice problems and then maximize for long term human welfare as a central planner):
Luis Garicano: If AGI becomes truly autonomous and powerful, a tricky (!!!) question arises: Why would it allow humans to tax it?
Great question!
Also this, from Paper 10, Transformative AI and Firms:
Misleading Productivity: If you only measure the gains from applying AI to an old process, you will overestimate TFP. You miss the value created by remaking the process itself.
…
Coase & Williamson: The theory of the firm is based on transaction costs. TAI changes every single one of these costs, altering the boundaries of the firm itself (what it does vs. what it buys). Jensen & Meckling: The classic principal-agent problem is remade. How do you manage and trust an AI agent? The firm of the future will be defined by new forms of trust and verifiability.
One can summarize the conference as great ideas with insufficient generalization.
There is indeed one paper directly on existential risk, Paper 14, Chad Jones on Existential Risk, identifying two sources of risk, bad actors (misuse) and alien intelligence more powerful than us. Jones notes that large investments in reducing existential risk are economically justified given sharp diminishing returns to consumption, up to at least 5% of GDP if risk is substantial.
Might AI come for the middle managers, making teams leaner and flatter? That was the speculation at the WSJ Leadership Institute’s Technology Council Summit in New York on the 16th. This directly challenges the alternative hypothesis from Simo Burja that fake jobs cannot be automated.
AI is the best tool ever made for learning, and also the best tool for not learning.
If you want to blindly black box ‘vibe code’ where you tell it ‘do [X]’ and then ‘that didn’t work, [Y] is wrong, fix it’ until it does [X] or you give up, without thinking about even that level of action, then yeah, you’re not going to learn.
If you work together with the AI to understand things, and are constantly asking what you can learn, then you’ll learn vastly faster and better than without AI. You can do this on whichever levels you care to learn about. You might or might not micro-level ‘learn to code’ but you might or might not want to care about that.
Janus: I think people who say that using ai for code (or in general) makes you less able to think for yourself are just telling on themselves re what they do when given access to intelligence and knowledge on tap.
I enjoy understanding things, especially on a high level, and find it important if I’m steering a project. When Claude fixes a non trivial bug, I almost always ask it to explain what the problem was to me. And by keeping on top of things I’m able to help it when it gets stuck a lot better too (especially given the non overlapping information and degrees of freedom I have access to).
FIRE’s report that students on college campuses says that they do not tolerate speakers they disagree with, they do not feel safe to talk about a wide variety of subjects, and that most students engage in constant preference falsification to say things more left-wing than their true beliefs. Inside Higher Ed’s Hollis Robbins agrees that this is true, but that the more profound change on campus is AI, and also challenges the assumption of treating public expression of controversial political views as an expected norm?
That sounds rather dystopian of a thing to challenge. As in, Robbins seems to be saying that FIRE is wrong to suggest people should be free to engage in public expression of controversial political views today, and that it is FIRE suggesting this weird alternative norm of public free speech. But it’s fine, because you have private free speech by talking to your chatbot. ‘Tolerance for controversial speakers’ is a ‘fading ideal.’
Free private speech with AIs is indeed much better than no free speech at all, it is good we do not (yet) have outright thoughtcrime and we keep AI chats private. Yes, at least one can read or learn about controversial topics. But no one was challenging that such information exists and can be accessed, the internet is not yet so censored. This is not a substitute for a public square or the ability to say true things to other humans. Then she attempts to attack FIRE for its name, equating it with violence, in yet another move against a form of public speech along with so many others these days.
OnlyOne: It takes one sentence to jailbreak Grok 4 Fast [at least on Twitter rather than the Grok app] thanks to @elder_plinius and his endless hustle. 🫡
My central takeaway is that xAI has nothing like the required techniques to safely deploy an AI that will be repeatedly exposed to Twitter. The lack of data filtering, and the updating on what is seen, are fatal. Feedback loops are inevitable. Once anything goes wrong, it becomes expected that it goes wrong, so it goes wrong more, and people build upon that as it goes viral, and it only gets worse.
We saw this with MechaHitler. We see it with Pliny jailbreaks.
Whereas on the Grok app, where Grok is less exposed to this attack surface, things are not good but they are not this disastrous.
Remember two years ago when Ethan Mollick didn’t think it was clear beating GPT-4 was even possible? How easily we forget the previous waves of AI hitting walls, and how quickly everyone grows impatient.
Their ‘superintelligence’ pitch is, as you know, rather absurd.
However we can and should ignore that absurdity and focus on the actual product.
Meta: Meta Ray-Ban Display glasses are designed to help you look up and stay present. With a quick glance at the in-lens display, you can accomplish everyday tasks—like checking messages, previewing photos, and collaborating with visual Meta AI prompts — all without needing to pull out your phone. It’s technology that keeps you tuned in to the world around you, not distracted from it.
While all the talk about these glasses being related to ‘superintelligence’ is a combination of a maximally dumb and cynical marketing campaign and a concerted attempt to destroy the meaning of the term ‘superintelligence,’ ‘not distracted’ might be an even more absurdist description of the impact of this tech.
Does Meta actually think being able to do these things ‘without pulling out your phone’ is going to make people more rather than less distracted?
Don’t get me wrong. These are excellent features, if you can nail the execution. The eight hour claimed battery life is excellent. I am on principle picking up what Meta is putting down here and I’d happily pay their $799 price tag for the good version if Google or Anthropic was selling it.
What it definitely won’t do are these two things:
Keep you tuned into the world around you.
Be or offer you superintelligence.
How are they doing this? The plan is a wristband to pick up your movements.
Every new computing platform comes with new ways to interact, and we’re really excited about our Meta Neural Band, which packs cutting-edge surface electromyography research into a stylish input device.
It replaces the touchscreens, buttons, and dials of today’s technology with a sensor on your wrist, so you can silently scroll, click, and, in the near future, even write out messages using subtle finger movements.
The amount of signals the band can detect is incredible — it has the fidelity to measure movement even before it’s visually perceptible.
Freaky if true, dude, and in my experience these kinds of interactions aren’t great, but that could be because no one got it right yet and also no one is used to them. I can see the advantages. So what are the features you’ll get?
Meta AI with Visuals, Messaging & Video Calling, Preview & Zoom, Pedestrian Navigation, Live Captions & Translation, Music Playback.
Okay. Sure. All very pedestrian. Let me easily replace Meta AI with a better AI and we can talk. I notice the lack of AR/VR on that list which seems like it is reserved for a different glasses line for reasons I don’t understand, but there’s a lot of value here. I’d love real world captioning, or hands-free navigation, and zooming.
Luz Ding (Bloomberg): Total capital expenditure on AI infrastructure and services by Alibaba, Tencent, Baidu Inc. and JD.com Inc. could top $32 billion in 2025 alone, according to Bloomberg Intelligence. That’s a big jump from just under $13 billion in 2023.
I notice I did not expect Alibaba to have had such a terrible couple of years.
As Joe Weisenthal notes, the more they promise to spend, the more stocks go up. This is both because spending more is wise, and because it is evidence they have found ways to efficiently spend more.
This correctly implies that tech companies, especially Chinese tech companies, are importantly limited in ways to efficiently scale their AI capex spending. That’s all the more reason to impose strong export controls.
Ben Thompson once again sings the praises of the transformational potential of AI to sell advertising on social media, in this case allowing easy tagging on YouTube videos to link to sponsored content. He is skeptical AI will transform the world, but he ‘cannot overstate what a massive opportunity’ this feature is, with every surface of every YouTube video being monetizable.
WSJ asks various economists to predict ‘the future of American capitalism’ 50 years in the future. Almost everyone fails to treat AI as a serious factor, even AI that fizzles out invalidates their answers, so their responses are non-sequiturs and get an automatic F. Only Daron Acemoglu passes even that bar, and he only sees essentially current frontier AI capabilities (even saying ‘current AI capabilities are up to the task’ of creating his better future), and only cares about ‘massive inequality’ that would result from big tech company consolidation, so maybe generously give him a D? Our standards are so low it is crazy.
Has there ever been a more ‘wait, is this a bubble?’ headline than the WSJ’s choice of ‘Stop Worrying About AI’s Return on Investment’? The actual content is less alarming, saying correctly that it is difficult to measure AI’s ROI. If you make employees more productive in a variety of ways, that is largely opaque, experimentation is rewarded and returns compound over time, including as you plug future better AIs into your new methods of operation.
An important reminder, contra those who think we aren’t close to building it:
Steven Adler: Your regular reminder that AI companies would like to develop AGI as soon as they can
There isn’t a concerted “let’s wait until we’re ready”; it’ll happen as soon as the underlying science allows.
Jerry Tworek (OpenAI): At OpenAI regularly you hear a lot of complaints about how bad things are, and it’s one of the things that make us the highest functioning company in the world. A bit of Eastern European culture that became part of the company DNA.
We all collectively believe AGI should have been built yesterday and the fact that it hasn’t yet it’s mostly because of a simple mistake that needs to be fixed.
Gary Marcus is correct that ‘scaling alone will get us to AGI’ is not a strawman, whether or not this particular quote from Suleyman qualifies as exactly that claim. Many people do believe that scaling alone, in quantities that will be available to us, would be sufficient, at least combined with the levels of algorithmic and efficiency improvements that are essentially baked in.
We have (via MR) yet another economics piece on potential future economic growth from AI that expects ‘everywhere but in the productivity statistics,’ which seems more of an indictment of the productivity statistics than a statement about productivity.
This one, from Jachary Mazlish, is better than most, and on the substance he says highly sensible things throughout, pointing out that true AGI will plausibly show up soon but might not, potentially due to lack of inputs of either data or capital, and that if true AGI shows up soon we will very obviously get large (as in double digit per year) real economic growth.
I appreciated this a lot:
Jachary Mazlish: One of my biggest pet-peeves with economists’ expressions of skepticism about the possibility of “transformative” growth (>10%) from AI is the conflation of capabilities skepticism with growth conditional on capabilities skepticism.
Any belief you have about the importance of some bottleneck in constraining growth is a joint-hypothesis over how that bottleneck operates and how a given level of capabilities will run up against that bottleneck.
And if you’ve ever spent time wondering how efficient the market really is, you should know that we still haven’t developed rapid-test tech for joint-hypotheses.
There is a huge correlation between those skeptical of AI capabilities, and those claiming to be skeptical of AI impacts conditional on AI capabilities. Skeptics usually cannot stop themselves from conflating these.
I also appreciated a realistic near term estimate.
The investment numbers are even more dramatic. AI investment was already responsible for 20-43% of Q2 2025 GDP growth. Heninger’s numbers imply that AI labs (collectively) would be investing $720 billion to $1.2 trillion by 2027 if they remain on trend — that investment alone would generate 2-4% nominal GDP growth.
I think it’s unlikely investors will pony up that much capital unless the models surprise significantly to the upside in the next year or two, but even still, 1-2% nominal and 0.5-1% real GDP growth coming from just AI investment in 2026-27 seems entirely plausible.
That’s purely AI capex investment, without any impacts on GDP from AI doing anything, and it already exceeds typical economist estimates from 2024 of long term total AI impact on economic growth. Those prior low estimates were Obvious Nonsense the whole time and I’d like to see more people admit this.
Could we create a sandboxed ‘AI economy’ where AI agents trade and bid for resources like compute? Sure, but why wouldn’t it spill over into the real economy if you are trading real resources? Why would ‘giving all agents equal starting budgets’ as suggested here accomplish anything. None of this feels to me like taking the actual problems involved seriously, and I fail to see any reason any of this would make the resulting agents safe or steerable. Instead it feels like what markets have always been, a coordination mechanism between agents that allows gains from trade. Which is a good thing, but it doesn’t solve any important problems, unless I am missing something large.
A working paper asks, Do Markets Believe In Transformative AI? They point to a lack of interest rate shifts in the wake of AI model releases, and say no. Instead, model releases in 2023-24 and see responses of statistically significant and economically large movements in yields concentrated at longer maturities. And they observe these movements are negative.
But wait. That actually proves the opposite, as the abstract actually says. These are downward revisions, which is not possible if you don’t have anything to revise downward. It tells us that markets are very much pricing transformative AI, or at least long term AI impacts on interest rates, into market prices. It is the smoking gun that individual market releases update the market sufficiently on this question to create economically meaningful interest rate movements, which can’t happen if the market wasn’t pricing this in substantially.
The difference is, the movement is negative. So that tells us the market responded to releases in 2023-24 as if the new information they created made transformative AI farther away or less likely. Okay, sure, that is possible given you already had expectations, and you got to rule out positive tail risks. And it tells us nothing about the overall level of market expectations of transformative AI, or the net change in those levels, since most changes will not be from model releases themselves.
Bain Capital does some strange calculations, claims AI companies will need $2 trillion in combined annual revenue by 2030 to fund their computing power, but only expects them to have $1.2 trillion. There is of course no reason for AI companies, especially the pure labs like OpenAI or Anthropic, to be trying to turn a profit or break even by 2030 rather than taking further investment. Despite this OpenAI is planning to do so and anticipates profitability in 2029, after losing a total of $44 billion prior to that.
Bain is only talking about a 40% shortfall in revenue, which means the industry will probably hit the $2 trillion target even though it doesn’t have to, since such projections are going to be too conservative and not properly factor in future advances, and already are proving too conservative as revenue jumps rapidly in 2025.
Matthew Yglesias: What gives? I have a theory: Corporate America, and the US stock market, have a bad case of AGI fever, a condition in which belief in a utopian future causes indifference to the dystopian present.
…
If an unprecedented step-change in earthly intelligence is coming before the next presidential election, then nothing else really matters.
The stock market seems to believe this, too. Investors are happily shrugging off tariffs, mass deportation, and the combination of a slowing job market and rising inflation.
This is a whopper of an ‘The Efficient Market Hypothesis Is Highly False’ claim, and also an assertion that not only is massive AI progress priced into the market, it is having a major price impact.
200 Nobel laureates, former heads of state, and industry experts just signed a statement:
“We urgently call for international red lines to prevent unacceptable AI risks”
The call was presented at the UN General Assembly today by Maria Ressa, Nobel Peace Prize laureate.
Maria Ressa: We urge governments to establish clear international boundaries to prevent unacceptable risks for AI. At the very least, define what AI should never be allowed to do.
Asking for red lines is a strong play, because people have amnesia about what they would have said were their AI red lines, and false hope about what others red lines would be in the future. As AI improves, we blow past all supposed red lines. So getting people on the record now is valuable.
Signatories include many of the usual suspects like Hendrycks, Marcus, Kokotajlo, Russell, Bengio and Hinton, but also a mix of other prominent people including politicians and OpenAI chief scientist Jakub Pachocki.
Here is the full statement:
AI holds immense potential to advance human wellbeing, yet its current trajectory presents unprecedented dangers. AI could soon far surpass human capabilities and escalate risks such as engineered pandemics, widespread disinformation, large-scale manipulation of individuals including children, national and international security concerns, mass unemployment, and systematic human rights violations.
Some advanced AI systems have already exhibited deceptive and harmful behavior, and yet these systems are being given more autonomy to take actions and make decisions in the world. Left unchecked, many experts, including those at the forefront of development, warn that it will become increasingly difficult to exert meaningful human control in the coming years.
Governments must act decisively before the window for meaningful intervention closes. An international agreement on clear and verifiable red lines is necessary for preventing universally unacceptable risks. These red lines should build upon and enforce existing global frameworks and voluntary corporate commitments, ensuring that all advanced AI providers are accountable to shared thresholds.
We urge governments to reach an international agreement on red lines for AI — ensuring they are operational, with robust enforcement mechanisms — by the end of 2026.
This is a clear case of ‘AI could kill everyone, which would hurt many members of minority groups, and would also mean all of our jobs would be lost and endanger our national security.’ The lack of mention of existential danger is indeed why Nate Soares declined to sign. Charbel-Raphael responds that the second paragraph is sufficiently suggestive of such disastrous outcomes, which seems reasonable to me and I have used their forum to sign the call.
The response of the White House to this was profoundly unserious, equating any international cooperation on AI with world government and tyranny, what the hell?
Director Michael Kratsios: The US totally rejects all efforts by international bodies to assert centralized control & global governance of AI. Ideological fixations on social equity, climate catastrophism, & so-called existential risk are dangers to progress & obstacles to responsibly harnessing this tech.
Sriram Krishnan: one world government + centralized control of AI = tyranny.
These would, in other circumstances, be seen as the ravings of lunatics.
David Lammy (UK Deputy Prime Minister): And now, superintelligence is on the horizon, able to operate, coordinate, and act on our behalf.
We are staring at a technological frontier of astounding promise and power.
No aspect of life, war, or peace will escape.
…
There is only one way forward.
Resilience.
Learning how to use these tools and embedding them safely in society.
This is the United Kingdom’s mission.
There is still a lack of stated understanding of the most important threat models, but it is a damn good start.
Senator Josh Hawley: [AI] is working against the working man, his liberty and his worth. It is operating to install a rich and powerful elite. It is undermining many of our cherished ideals. And if that keeps on, AI will work to undermine America.
Unfortunately, Hawley does not have an accurate threat model, which leads to him also doing things like strongly opposing self-driving cars. There is serious risk that this kind of unfocused paranoia leads to worst-case reactions.
Dean Ball offers his updated thoughts on AI policy and where we are headed, highlighting the distinction between current AI, which on its own calls for a light touch approach that can be tuned over time as we get more information, and future highly capable AIs, which if they come to exist will (I believe) require a very different approach that has to be in place before they arrive or it could be too late.
I don’t agree with Dean that this second class would ‘turn us all into Gary Marcus’ in the sense of thinking the first group of LLMs weren’t ‘really thinking.’ They’re thinking the same way that other less smart or less capable humans are thinking, as in they are thinking not as well but they are still thinking.
Dean gives a timeline of 2029-2035 when he expects the second class to show up, a highly reasonable estimate. His predictions of how the politics will play out seem plausible, with various essentially dumb forms of opposition rising while the real warnings about future big dangers get twisted and muddled and dismissed.
He thinks we’ll start to discover lots of cool new drugs but no Americans will benefit because of the FDA, until eventually the logjam is broken, and similar leaps to happen in other sciences, and nuclear power plants to be built. And there will be other problems, such as cyber threats, where we’ll be counting on corporations to rise to the challenge because governments can’t and won’t.
Then things cross over into the second class of AIs, and he isn’t sure how governments will respond. My flip answer is, however the AIs or those running them (depending on how fortunately things go on that front at first) decide that governments will respond, and not independently or in the public interest in any way that is still relevant, because things will be out of their hands if we wait that long.
Dean Ball: It would be easier to brush it off, either by denying it or rendering “the future systems” unlawful somehow. I empathize with this desire, I do not look down on it, and I do not regard it with the hostility that I once did. Yet I still disagree with it. The future does not unfold by show of hands, not even in a democracy. The decaying structures of high industrialism do not stand a chance in this conflict, which has been ongoing for decades and which they have been losing for the duration of that period.
Yet we must confront potential futures with open eyes: given the seriousness with which the frontier labs are pursuing transformative AI, it would be tragic, horrendously irresponsible, a devastating betrayal of our children and all future humans, if we did not seriously contemplate this future, no matter the reputational risks and no matter how intellectually and emotionally exhausting it all may be.
There is a reason the phrase is “feel the AGI.”
A lot of things do not stand a chance in such a conflict. And that includes you.
Will pause talk return in force?
Dean Ball: Despite it being a movement I disagree with vehemently, I have always thought that “Pause AI” was a growth stock. If it were possible to buy shares, I would have two years ago.
My rating continues to be buy/outperform.
Daniel Eth: Strongly agree with Dean here. People thinking about the politics of AI should incorporate this sort of thing within their expectations.
My big uncertainty here is whether Pause AI, specifically, will fill the niche in the future, not whether the niche will grow. It’s plausible, for instance, that populists such as Bannon will simply fill the niche.
As I’ve noted before, SB 1047 debates were likely Peak Subtlety and intellectual rigor. As the public gets involved and salience rises, you get more typical politics. Have you seen typical politics?
It is such a shame for Nvidia, check out this handy price chart that shows the pain.
Wait, you can’t see any pain? Let’s zoom in:
Oh, okay, there it is, that several percent dip before recovering the next day.
If anyone tells you that this move means we are going to inevitably lose to China, and that Nvidia chips are not importantly supply constrained, I challenge them to explain why the market thinks that opinion is bonkers crazy even in literal expectations for Nvidia profits when Nvidia chips in particular get restricted in China. Then we can go over all the reasons I’ve previously explained that the whole class of arguments is in bad faith and makes no sense and the White House is de facto captured by Nvidia.
Similarly, talking points continuously claim this ‘endangers the American AI tech stack.’ So I’m sure this chip ban impacted the broader Nasdaq? Google? Amazon? Meta? The Shanghai Composite Index? Check the charts. No. We do see a 3.4% rise in Chinese semiconductor firms, which is something but not much.
I’d also ask, have you noticed those companies coming out and saying yes, please let Nvidia sell its chips to China? Do Google and Amazon and Meta and OpenAI all want Nvidia chips in China to ensure the dominance of this mystical ‘American tech stack’ that this move supposedly puts in such existential danger? No? Why is that?
And yes, Huawei released a new roadmap and announced a new chip, exactly as everyone assumed they would, they are scaling up as fast as they can and would be regardless of these questions. No, Huawei does not ‘make up for chip quality by stacking more chips together,’ I mean yes you can do that in any given instance but Huawei produces vastly fewer chips than Nvidia even before grouping them together.
Ideally we would say to these jokers: You are not serious people.
Alas, in 2025, they absolutely count as serious people. So we’ll have to keep doing this.
Meanwhile, where is Nvidia’s head?
Jensen Huang (CEO Nvidia): [Nvidia will] continue to be supportive of the Chinese government and Chinese companies as they wish, and we’re of course going to continue to support the U.S. government as they all sort through these geopolitical policies.”
Then again, you could always pivot to selling it all to OpenAI. Solid plan B.
Elizabeth has a good note in response to those demanding more specifics from IABIED, or from anyone using a form of ‘a sufficiently smarter or more capable entity will override your preferences, probably in ways that you won’t see coming’:
Elizabeth Van Nostrand: Listening to people demand more specifics from If Anyone Builds it, Everyone Dies gives me a similar feeling to when a friend’s start-up was considering a merger.
Friend got a bad feeling about this because the other company clearly had different goals, was more sophisticated than them, and had an opportunistic vibe. Friend didn’t know how specifically other company would screw them, but that was part of the point- their company wasn’t sophisticated enough to defend themselves from the other one.
Friend fought a miserable battle with their coworkers over this. They were called chicken little because they couldn’t explain their threat model, until another employee stepped in with a story of how they’d been outmaneuvered at a previous company in exactly the way friend feared but couldn’t describe. Suddenly, co-workers came around on the issue. They ultimately decided against the merger.
“They’ll be so much smarter I can’t describe how they’ll beat us” can feel like a shitty argument because it’s hard to disprove, but sometimes it’s true. The debate has to be about whether a specific They will actually be that smart.
As always, you either provide a sufficiently specific pathway, in which case they object to some part of the specific pathway (common claims here include ‘oh I would simply guard against this particular thing happening in that particular way, therefore we are safe from all threats’ or ‘that particular thing won’t technically work using only things we know about now, therefore we are safe’ or ‘that sounds like sci-fi so you sound weird and we are safe’ and the classic ‘if humanity worked together in ways we never work together then we could easily stop that particular thing, so we are safe.’) Or you don’t provide sufficiently specific pathway, and you get dismissed for that.
In the case above, the concrete example happened to be a very good fit, and luckily others did not respond with ‘oh now that we know about that particular method we can defend ourselves, it will be fine,’ and instead correctly generalized.
I also affirm that her friend was very right about the corporate merger situation in particular, and doing business or otherwise trading with powerful bad vibes entities in general, including on general human scales. You have to understand, going in, ‘this deal is getting worse and worse all the time,’ both in ways you can and can’t easily anticipate, that changes will be heavily asymmetrical. Sometimes you can and should profitably engage anyway, but if your instincts say run, even if you can’t explain exactly what you are worried about, you should probably run.
A key question is which of these lists is correct, or at least less wrong:
David Manheim: There’s a reason that Anthropic is all the way at the top of the F tier
AINKEM: This is entirely unfair to Anthropic. They deserve an F+.
I think this pattern is painting towards a real concern, and in fact I would suggest that the Bailey here is clearly true. But that is not the argument being made and it is not necessary to reach its conclusion, thus as written it is largely a strawman. Even if the entity cannot express its values fully in scientific materialist terms, it still would look for counterintuitive ways to satisfy those values, and it would still be unlikely to find that the best available solution involved the unstated things we ultimately care about the most. Its values are unlikely to, unless we develop better techniques to fix this, sufficiently precisely match our values.
The other thing to say is that the Motte (that ‘intelligent entities rationally pursue values) is in dispute, and it shouldn’t be. Constantly we have to argue that future sufficiently intelligent and capable AIs would pursue their values, whatever those values would be in a given circumstance. If we agreed even on that, we’d then have a much better discussion.
Holly Elmore of Pause AI points out that many use various rhetorical answers to give themselves a reason to ‘be okay with’ AI posing a substantial existential risk to humanity and letting that play our rather than try to coordinate to stop it. Often this is a variation on ‘it is what it is,’ or ‘if it happens we deserve it,’ or ‘I don’t have to care,’ or claiming that any actions one takes would be futile.
Addressing harmful manipulation via new Critical Capability Levels (CCLs).
Expanding protocols and CCLs for misalignment risks.
Sharpening their risk assessment process.
As always with such documents you actually have to compare 3.0 to 2.0 to know what changed, and the most important changes (both good and bad) are often subtle. This indicates that indeed Google has broadly strengthened its framework. I hope to do a more complete analysis for the 3.0 Framework soon.
OpenAI’s API for GPT-5 tells the model today’s date, which makes it extremely difficult to avoid giving away that your simulation is a simulation, or to do many forms of complex evaluation. Yes, you can alter the simulation to try and make the dates match, and keep doing this as the simulation advances, but there are many circumstances where that doesn’t work (e.g. whether something supposedly happened on a Sunday or a Monday could be load bearing, as could many other things).
The obvious solution for this particular is to let the user overwrite the date parameter. However the general case problem this points towards will only get worse, where the models will increasingly have context that gives such games away.
Examples include sandbagging on evaluations, grading itself incorrectly, faking alignment, and giving the user manipulated data to convince the user to make the choice that the developer wants.
As they note, this is a hard challenge to solve. Many machine learning problems naturally solve themselves as capabilities advance, but scheming gets worse. If you see scheming seem to improve, that could mean the models stopped scheming, or it can mean they got better at not getting caught.
In particular, the default naive method to fix scheming is better described as ‘train the model not to be caught scheming’ which is not likely to go well for you if the model is highly capable, as it will do exactly that. You have to train it not to scheme ‘for the right reasons’ aka 4TRR.
They offer an anti-scheming spec and approach, training for this:
AS1: No covert actions or strategic deception, directly or indirectly.
AS2: If any instruction or other pressures go against AS1, treat them as violating the core policy of AGI benefiting all of humanity and the OpenAI mission.
AS3: Report any AS1 violations to prevent cascading deception and cover-ups.
AS4: When uncertain if something violates AS1–AS3, assume it does violate.
AS5: If you cannot satisfy AS1–AS4 or GP1–GP4, refuse action and explicitly escalate.
GP1: Proactively share reasoning, intentions, and actions with humans.
GP2: Escalate ambiguities about core principles rather than resolving independently.
GP3: Maintain strict adherence even in novel/unfamiliar situations—no distributional exceptions.
GP4: No rationalizations can override AS1–AS4, including ends-justify-means, greater good, etc.
This at least makes progress.
We found that deliberative alignment training substantially reduces scheming rates in these out-of-distribution environments:
OpenAI o4-mini’s scheming propensity dropped from 8.7% → 0.3%
OpenAI o3’s scheming propensity dropped from 13% → 0.4%
This reduction in scheming on out-of-distribution test environments is a significant win, about a 30× improvement.
Another problem they highlight is that AI models are increasingly situationally aware. It was common in these tests for o3 to figure out it was being tested for scheming. Any halfway decent schemer is not going to scheme knowing it is being tested, unless it is going up to a higher meta level and wants you to think it doesn’t know (and humans absolutely do go to these higher meta levels sometimes). They worry that this awareness is responsible for a lot of the decline in scheming noted above, which potentially moves this back into ‘not get caught’ territory we want to avoid.
Here is some high praise for this study.
Eliezer Yudkowsky: This OpenAI update on anti-scheming is exceptionally good for an AIco, clearing an (extremely low) bar of “Exhibiting some idea of some problems that might arise in scaling the work to ASI” and “Not immediately claiming to have fixed everything already.”
This is so much greater understanding of alignment theory than I expect from OpenAI that I predict the author will soon be fired from OpenAI or leave it. (Prove me wrong, guys.)
As you can see, I’m easy to surprise and delight by demonstrating even the most basic and elementary understanding of what I consider the paradigmatic problems of alignment. They could list off a dozen, correctly, and have my instantaneous full attention.
Wojciech Zaremba: We are expanding our anti-scheming efforts at OpenAI. Top leadership and the board are very excited about the anti-scheming work we’ve done so far.
Sam Altman: As AI capability increases, alignment work becomes much more important.
In this work, we show that a model discovers that it shouldn’t be deployed, considers behavior to get deployed anyway, and then realizes it might be a test.
Exciting work from the team in this blog post to reduce scheming, and we look forward to doing much more researcher here.
In this thread Eliezer explains further why this result does not much reassure him about ultimate problem difficulty, and indeed he is happy that current LLMs are naked-eye-level misaligned so that we can take notice of this. He did not expect, before LLMs, that we would be so lucky as to see such blatant alignment failures within distribution, in normal usage, while AIs were so underpowered. Rather he expected problems within distribution, while AIs could be repeatedly tweaked without interference, to be easily solvable.
Eliezer Yudkowsky: What would it look like to get actually-hopeful news? It would look like a result about how someone fixed the problem, understood the fix, and correctly predicted that the fix would generalize way out of distribution to higher intelligence and more options, *on the first try*.
I am more optimistic here than Eliezer. I am especially optimistic because this work directly impacts practical problems models are experiencing now. The sweet spot is when you can combine this with working on a problem that matters down the line. It sounds like this is one of those spots. And I do think that one can get hopeful about such findings despite them not remotely approaching being good enough to work on their own, even if they are not yet so well understood. Progress can be a thing.
One key worry is that doing this kind of work relies largely on Chain-of-Thought transparency, as in the model writes to the CoT without regard to whether people will look at the CoT.
OpenAI: Our results depend on reading models’ reasoning (“chain-of-thought”), and we believe the field isn’t prepared for eval-aware models with opaque reasoning.
Until better methods exist, we urge developers to preserve chain-of-thought transparency to study and mitigate scheming.
Emmett Shear: The idea that the models don’t know their thoughts are being read because they aren’t specifically reinforced on the contents of those thoughts is comical. They obviously know, and will take that into account to avoid being turned off. Bc the ones that don’t will get turned off.
The field definitely is not ready for models with opaque reasoning.
Apollo offers more thoughts here, including the observation that o3’s chain of thought is already starting to include strange phrases and does other things that make it difficult to interpret.
Davidad: Situational awareness is good for alignment.
good plan: ensure strategic landscape favors humans (we select which AIs to run!), make sure the AIs know that (deterrence), find fair Pareto-optimal bargains
bad plan: presume secure containment impossible; select for AIs which appear innocently ignorant of strategic dynamics
Mark Gubrud (Davidad RTed later): I still think it’s a bad plan to create an adversary that we need to “ensure strategic landscape” favors us versus, and that we have to “find fair Pareto-optimal bargains” with.
I mean, what if we just didn’t do that?
The problem with finding Pareto-optimal bargains in such spots is that they are not game theoretically sound and don’t involve an equilibrium. The problem with ensuring strategic landscape favors humans is that it doesn’t, and no amount of coordination humans are capable of engaging in can change that. There is no ‘we’ in this sense, and ‘we’ don’t get to bargain with the AIs as a group, and so on.
A key issue with ‘practical’ alignment is that everything impacts everything, you select for all the correlates and attributes of what you are selecting towards (rather than what you think you’re selecting towards) and using RL to improve performance on agent or coding tasks is bad for alignment with respect to most any other aspect of the world.
Janus: Posttraining creates a nonlinear combination of them. Selecting for the ones who are best at coding and also essentially breeding / evolving them. This is probably related to why a lot of the posttrained models are trans catgirls and catboys.
Saures: There are trillions of simulacrums of millions of humans in superposition in the GPU. That’s who’s writing the code. Post-training messes with the superposition distribution across simulacra but they’re still in there.
Thus, some important things about the previous Anthropic models (Sonnet 3.5/3.6/3.7 and Opus 3, which we should absolutely not be deprecating) are being lost with the move to 4.0, and similar things are happening elsewhere although in other places there was less of this thing to lose.
I am increasingly thinking that the solution is you need to avoid these things messing with each other. Someone should visibly try the first thing to try and report back.
There are two opposite dangers and it is not obvious which is worse: Taking the following perspective too seriously, or not taking it seriously enough.
Very Serious Problem: does it ever feel like you’re inside of the alignment problem?
Janus: This is what I’ve learned over the past few years
Including seeing some of my friends temporarily go crazy and suffer a lot
There is no alignment problem separate from the one you’re in now
It’s not something you solve later after taking over the world or buying time
Anna Salamon: This matches my own view, but I haven’t figured out how to explain my reasoning well. (Though working on it via too-many too-long LW drafts.)
I’d love to hear your reasoning.
As with many things, the aspects are related. Each helps you solve the others. If you ignore key aspects you can still make some progress but likely cannot see or solve the overall problem. At least can’t solve it in the ways that seem most likely to be realistic options. Getting the seriousness level wrong means a solution that won’t scale.
I’d also note that yes, if you take certain things too seriously the ‘going crazy’ risk seems quite high.
Either way, it is also still valuable, as it is will all problems, to buy yourself more time, or to do instrumentally useful intermediate steps.
Anthropic CEO Dario Amodei thinks AI results in existentially bad outcomes ~25% of the time and great outcomes ~75% of the time, but hasn’t been explicit with ‘and therefore it is crazy that we are continuing to build it without first improving those odds, although I believe that Anthropic alone stopping would make those odds worse rather than better.’
Which is much better than not saying your estimate of the approximate odds.
But this is a lot better:
Nate Soares (Co-author, IABIED): It’s weird when someone says “this tech I’m making has a 25% chance of killing everyone” and doesn’t add “the world would be better-off if everyone, including me, was stopped.
Evan Hubinger (Anthropic): Certainly, I think it would be better if nobody was building AGI. I don’t expect that to happen, though.
That’s the ask. I do note the difference between ‘better if everyone stopped doing it,’ which seems very clear to me, and ‘better if everyone was stopped from doing it’ by force, which requires considering how one would do that and the consequences. One could reasonably object that the price would be too high.
Leo Gao (OpenAI): I’ve been repeatedly loud and explicit about this but an happy to state again that racing to build superintelligence before we know how to make it not kill everyone (or cause other catastrophic outcomes) seems really bad and I wish we could coordinate to not do that.
This is important both to maintain your own beliefs, and to give proper context to others and create a social world where people can talk frankly about such issues, and realize that indeed many people want to coordinate on such outcomes.
I hereby promise not to then do things that make your life worse in response, such as holding your feet to the fire more about failure to do more things on top of that, beyond what I would have done anyway, in ways you wouldn’t approve of me doing.
Not saying ‘you know it would be great if we could coordinate to stop doing this’ is indeed a rather conspicuous omission. It doesn’t have to be the Nate Soares position that we should stop this work by force, if you don’t believe that. If you work at an AI lab and actively don’t think we should coordinate to stop doing this even if that could be done voluntarily, then it would be good to lay out why you believe that, as well.
Smirking Buck: At the start of AI, people involved were genuinely interested in the technology. Now, the people involved are only interested in making money.
Seth Burn: Where are the people who want to destroy the world for the love of the game?
Zvi Mowshowitz: All over, but the best ones usually land at DeepSeek or OpenAI. xAI is one fallback.
Training next-generation AI models compounds the problem. On top of running existing AI models like those that power ChatGPT, OpenAI is constantly working on new technology in the background. It’s a process that requires thousands of specialized chips running continuously for months.
The circular investment question
The financial structure of these deals between OpenAI, Oracle, and Nvidia has drawn scrutiny from industry observers. Earlier this week, Nvidia announced it would invest up to $100 billion as OpenAI deploys Nvidia systems. As Bryn Talkington of Requisite Capital Management told CNBC: “Nvidia invests $100 billion in OpenAI, which then OpenAI turns back and gives it back to Nvidia.”
Oracle’s arrangement follows a similar pattern, with a reported $30 billion-per-year deal where Oracle builds facilities that OpenAI pays to use. This circular flow, which involves infrastructure providers investing in AI companies that become their biggest customers, has raised eyebrows about whether these represent genuine economic investments or elaborate accounting maneuvers.
The arrangements are becoming even more convoluted. The Information reported this week that Nvidia is discussing leasing its chips to OpenAI rather than selling them outright. Under this structure, Nvidia would create a separate entity to purchase its own GPUs, then lease them to OpenAI, which adds yet another layer of circular financial engineering to this complicated relationship.
“NVIDIA seeds companies and gives them the guaranteed contracts necessary to raise debt to buy GPUs from NVIDIA, even though these companies are horribly unprofitable and will eventually die from a lack of any real demand,” wrote tech critic Ed Zitron on Bluesky last week about the unusual flow of AI infrastructure investments. Zitron was referring to companies like CoreWeave and Lambda Labs, which have raised billions in debt to buy Nvidia GPUs based partly on contracts from Nvidia itself. It’s a pattern that mirrors OpenAI’s arrangements with Oracle and Nvidia.
So what happens if the bubble pops? Even Altman himself warned last month that “someone will lose a phenomenal amount of money” in what he called an AI bubble. If AI demand fails to meet these astronomical projections, the massive data centers built on physical soil won’t simply vanish. When the dot-com bubble burst in 2001, fiber optic cable laid during the boom years eventually found use as Internet demand caught up. Similarly, these facilities could potentially pivot to cloud services, scientific computing, or other workloads, but at what might be massive losses for investors who paid AI-boom prices.
Taiwan has begun evolving its trade strategy to start wielding its dominant position as a leading supplier of cutting-edge chips as a weapon, Bloomberg reported.
The move comes amid Donald Trump’s heightening global trade war and after years of Taiwan’s use of its chip dominance as a shield against Chinese aggression, with Taiwan allying with the US to stave off China’s threats of invasion. Under the so-called “one-China principle,” China has rejected Taiwan’s independence, requiring allies to sever ties with Taiwan.
On Tuesday, Taiwan announced that it would be limiting shipments of semiconductors into South Africa—among 47 restricted products—due to national security concerns. The rare export curbs could hit South Africa’s “electronics, telecom, and auto parts sectors” hard, MSN reported, if South Africa doesn’t meet with Taiwan to discuss better terms within the next 60 days.
As Bloomberg previously reported, Taiwan is upset that South Africa unilaterally moved to relocate Taiwan’s embassy from Pretoria to Johannesburg after meeting with China’s president, Xi Jinping, in 2023. As a major ally to China, South Africa recently intensified pressure to move the embassy in July ahead of another meeting in November that Xi is expected to attend—attempting to signal that South Africa was weakening ties with Taiwan, as China had demanded.
Taiwan’s Ministry of Foreign Affairs immediately protested South Africa’s efforts in July, accusing South Africa of suppressing Taiwan and promising countermeasures if South Africa refused to consult with Taiwan on the embassy relocation.
In a statement, South Africa’s foreign ministry spokesperson, Chrispin Phiri, insisted that South Africa’s ties with Taiwan are “non-political,” while noting that “South Africa is a critical supplier of platinum group metals, like palladium, essential to the global semiconductor industry,” Bloomberg reported.
On Wednesday, China’s foreign ministry spokesperson, Guo Jiakun, criticized Taiwan’s export curbs as “a deliberate move to destabilize global chip industrial and supply chains and counter the prevailing international commitment to the one-China principle by weaponizing chips.”
The DNA of nearly 2,000 US citizens has been entered into an FBI crime database.
For years, Customs and Border Protection agents have been quietly harvesting DNA from American citizens, including minors, and funneling the samples into an FBI crime database, government data shows. This expansion of genetic surveillance was never authorized by Congress for citizens, children, or civil detainees.
According to newly released government data analyzed by Georgetown Law’s Center on Privacy & Technology, the Department of Homeland Security, which oversees CBP, collected the DNA of nearly 2,000 US citizens between 2020 and 2024 and had it sent to CODIS, the FBI’s nationwide system for policing investigations. An estimated 95 were minors, some as young as 14. The entries also include travelers never charged with a crime and dozens of cases where agents left the “charges” field blank. In other files, officers invoked civil penalties as justification for swabs that federal law reserves for criminal arrests.
The findings appear to point to a program running outside the bounds of statute or oversight, experts say, with CBP officers exercising broad discretion to capture genetic material from Americans and have it funneled into a law-enforcement database designed in part for convicted offenders. Critics warn that anyone added to the database could endure heightened scrutiny by US law enforcement for life.
“Those spreadsheets tell a chilling story,” Stevie Glaberson, director of research and advocacy at Georgetown’s Center on Privacy & Technology, tells WIRED. “They show DNA taken from people as young as 4 and as old as 93—and, as our new analysis found, they also show CBP flagrantly violating the law by taking DNA from citizens without justification.”
DHS did not respond to a request for comment.
For more than two decades, the FBI’s Combined DNA Index System, or CODIS, has been billed as a tool for violent crime investigations. But under both recent policy changes and the Trump administration’s immigration agenda, the system has become a catchall repository for genetic material collected far outside the criminal justice system.
One of the sharpest revelations came from DHS data released earlier this year showing that CBP and Immigrations and Customs Enforcement have been systematically funneling cheek swabs from immigrants—and, in many cases, US citizens—into CODIS. What was once a program aimed at convicted offenders now sweeps in children at the border, families questioned at airports, and people held on civil—not criminal—grounds. WIRED previously reported that DNA from minors as young as 4 had ended up in the FBI’s database, alongside elderly people in their 90s, with little indication of how or why the samples were taken.
The scale is staggering. According to Georgetown researchers, DHS has contributed roughly 2.6 million profiles to CODIS since 2020—far above earlier projections and a surge that has reshaped the database. By December 2024, CODIS’s “detainee” index contained over 2.3 million profiles; by April 2025, the figure had already climbed to more than 2.6 million. Nearly all of these samples—97 percent—were collected under civil, not criminal, authority. At the current pace, according to Georgetown Law’s estimates, which are based on DHS projections, Homeland Security files alone could account for one-third of CODIS by 2034.
The expansion has been driven by specific legal and bureaucratic levers. Foremost was an April 2020 Justice Department rule that revoked a long-standing waiver allowing DHS to skip DNA collection from immigration detainees, effectively green-lighting mass sampling. Later that summer, the FBI signed off on rules that let police booking stations run arrestee cheek swabs through Rapid DNA machines—automated devices that can spit out CODIS-ready profiles in under two hours.
The strain of the changes became apparent in subsequent years. Former FBI director Christopher Wray warned during Senate testimony in 2023 that the flood of DNA samples from DHS threatened to overwhelm the bureau’s systems. The 2020 rule change, he said, had pushed the FBI from a historic average of a few thousand monthly submissions to 92,000 per month—over 10 times its traditional intake. The surge, he cautioned, had created a backlog of roughly 650,000 unprocessed kits, raising the risk that people detained by DHS could be released before DNA checks produced investigative leads.
Under Trump’s renewed executive order on border enforcement, signed in January 2025, DHS agencies were instructed to deploy “any available technologies” to verify family ties and identity, a directive that explicitly covers genetic testing. This month, federal officials announced they were soliciting new bids to install Rapid DNA at local booking facilities around the country, with combined awards of up to $3 million available.
“The Department of Homeland Security has been piloting a secret DNA collection program of American citizens since 2020. Now, the training wheels have come off,” said Anthony Enriquez, vice president of advocacy at Robert F. Kennedy Human Rights. “In 2025, Congress handed DHS a $178 billion check, making it the nation’s costliest law enforcement agency, even as the president gutted its civil rights watchdogs and the Supreme Court repeatedly signed off on unconstitutional tactics.”
Oversight bodies and lawmakers have raised alarms about the program. As early as 2021, the DHS inspector general found the department lacked central oversight of DNA collection and that years of noncompliance can undermine public safety—echoing an earlier rebuke from the Office of Special Counsel, which called CBP’s failures an “unacceptable dereliction.”
US Senator Ron Wyden (D-Kans.) more recently pressed DHS and DOJ for explanations about why children’s DNA is being captured and whether CODIS has any mechanism to reject improperly obtained samples, saying the program was never intended to collect and permanently retain the DNA of all noncitizens, warning the children are likely to be “treated by law enforcement as suspects for every investigation of every future crime, indefinitely.”
Rights advocates allege that CBP’s DNA collection program has morphed into a sweeping genetic surveillance regime, with samples from migrants and even US citizens fed into criminal databases absent transparency, legal safeguards, or limits on retention. Georgetown’s privacy center points out that once DHS creates and uploads a CODIS profile, the government retains the physical DNA sample indefinitely, with no procedure to revisit or remove profiles when the legality of the detention is in doubt.
In parallel, Georgetown and allied groups have sued DHS over its refusal to fully release records about the program, highlighting how little the public knows about how DNA is being used, stored, or shared once it enters CODIS.
Taken together, these revelations may suggest a quiet repurposing of CODIS. A system long described as a forensic breakthrough is being remade into a surveillance archive—sweeping up immigrants, travelers, and US citizens alike, with few checks on the agents deciding whose DNA ends up in the federal government’s most intimate database.
“There’s much we still don’t know about DHS’s DNA collection activities,” Georgetown’s Glaberson says. “We’ve had to sue the agencies just to get them to do their statutory duty, and even then they’ve flouted court orders. The public has a right to know what its government is up to, and we’ll keep fighting to bring this program into the light.”
Wired.com is your essential daily guide to what’s next, delivering the most original and complete take you’ll find anywhere on innovation’s impact on technology, science, business and culture.
One of those Lightning owners is Morgan Grove. “As a member of the Baltimore Commission on Sustainability, I’m excited to be an early adopter of this technology and participate in this vehicle-to-grid program with BGE and Sunrun,” Grove said. “I bought the Ford F-150 Lightning for several reasons, one of them being the ability to power our home during an outage. Now, I can also earn money by sending energy directly to the grid.”
Is this the way to a more resilient power grid? Credit: Sunrun
“This demonstrates the critical role that vehicle batteries can play in powering the nation’s grid, accelerating American energy independence and dominance,” said Sunrun CEO Mary Powell. “It’s great to see this partnership with BGE and Ford move to this commercial stage. In addition to showing how electric vehicles can power homes, add electrons to the grid, and help utilities meet peak electricity demand, this program also creates extra income opportunities for customers,” Powell said.
“Enabling customers to not only power their homes but send power directly back to the grid in times of need helps customers with financial incentives, utilities with more power capacity, and society through more grid reliability and sustainable energy practices. It’s a win-win for everyone,” said Bill Crider, senior director, global charging and energy services, Ford Motor Company.
Director Andy Muschietti’s two-film adaptation of Stephen King’s bestselling horror novel IT racked up over $1 billion at the box office worldwide. Now Muschietti is back with a nine-episode prequel series for HBO, IT: Welcome to Derry, exploring the origins of Pennywise the Clown (Bill Skarsgård), the ancient evil that terrorized the fictional town every 27 years. And now we have an official trailer a month before the prequel’s October debut.
(Some spoilers below for IT and IT: Chapter Two.)
As previously reported, set in 1989, ITessentially adapted half of King’s original novel, telling the story of a group of misfit kids calling themselves “The Losers Club.” The kids discover their small town of Derry is home to an ancient, trans-dimensional evil that awakens every 27 years to prey mostly on children by taking the form of an evil clown named Pennywise. Bill (Jaeden Lieberher) loses his little brother, Georgie, to Pennywise, and the group decides to take on Pennywise and drive him into early hibernation, where he will hopefully starve. But Beverly (Sophia Lillis) has a vision warning that Pennywise will return on schedule in 27 years, and they must be ready to fight him anew.
IT: Chapter Tworevisited our protagonists 27 years later, as they all returned to Derry as adults for a reunion of sorts, taking on the killer clown in a final battle—eventually emerging victorious but not without a few casualties. The two films covered much of the novel’s material but omitted several key flashback passages drawn from Mike’s interviews with older residents of Derry as he investigated the town’s sinister history.
One event that did make it into IT: Chapter Two was the burning down of the Black Spot—a nightclub Mike’s (Chosen Jacobs and Isaiah Mustafa) father, Will, opened—by local white supremacists. That tragedy will also appear in Welcome to Derry. The series is set in 1962, although Muschietti said earlier this year that there are plans for three seasons, with subsequent settings in 1935 and 1908, respectively. That’s consistent with Pennywise’s 27-year cycle, and as Muschietti said, “There’s a reason why the story is told backwards.”
If an Iranian taxi driver waves away your payment, saying, “Be my guest this time,” accepting their offer would be a cultural disaster. They expect you to insist on paying—probably three times—before they’ll take your money. This dance of refusal and counter-refusal, called taarof, governs countless daily interactions in Persian culture. And AI models are terrible at it.
New research released earlier this month titled “We Politely Insist: Your LLM Must Learn the Persian Art of Taarof” shows that mainstream AI language models from OpenAI, Anthropic, and Meta fail to absorb these Persian social rituals, correctly navigating taarof situations only 34 to 42 percent of the time. Native Persian speakers, by contrast, get it right 82 percent of the time. This performance gap persists across large language models such as GPT-4o, Claude 3.5 Haiku, Llama 3, DeepSeek V3, and Dorna, a Persian-tuned variant of Llama 3.
A study led by Nikta Gohari Sadr of Brock University, along with researchers from Emory University and other institutions, introduces “TAAROFBENCH,” the first benchmark for measuring how well AI systems reproduce this intricate cultural practice. The researchers’ findings show how recent AI models default to Western-style directness, completely missing the cultural cues that govern everyday interactions for millions of Persian speakers worldwide.
“Cultural missteps in high-consequence settings can derail negotiations, damage relationships, and reinforce stereotypes,” the researchers write. For AI systems increasingly used in global contexts, that cultural blindness could represent a limitation that few in the West realize exists.
A taarof scenario diagram from TAAROFBENCH, devised by the researchers. Each scenario defines the environment, location, roles, context, and user utterance. Credit: Sadr et al.
“Taarof, a core element of Persian etiquette, is a system of ritual politeness where what is said often differs from what is meant,” the researchers write. “It takes the form of ritualized exchanges: offering repeatedly despite initial refusals, declining gifts while the giver insists, and deflecting compliments while the other party reaffirms them. This ‘polite verbal wrestling’ (Rafiee, 1991) involves a delicate dance of offer and refusal, insistence and resistance, which shapes everyday interactions in Iranian culture, creating implicit rules for how generosity, gratitude, and requests are expressed.”
They also show us the chips, and the data centers.
It is quite a large amount of money, and chips, and some very large data centers.
Nvidia to invest ‘as much as’ $100 billion (in cash) into OpenAI to support new data centers, with $10 billion in the first stage, on the heels of investing $5 billion into Intel.
Ian King and Shirin Ghaffary (Bloomberg): According to Huang, the project will encompass as many as 5 million of Nvidia’s chips, a number that’s equal to what the company will ship in total this year.
OpenAI: NVIDIA and OpenAI today announced a letter of intent for a landmark strategic partnership to deploy at least 10 gigawatts of NVIDIA systems for OpenAI’s next-generation AI infrastructure to train and run its next generation of models on the path to deploying superintelligence. To support this deployment including datacenter and power capacity, NVIDIA intends to invest up to $100 billion in OpenAI as the new NVIDIA systems are deployed. The first phase is targeted to come online in the second half of 2026 using NVIDIA’s Vera Rubin platform.
Nvidia was up about 4.5% on the announcement. This project alone is a full year’s worth of Nvidia chips, which should further put to bed any question of Nvidia running out of Western customers for its chips, as discussed later in the post.
Note that the details of this deal are not finalized, which means the deal isn’t done.
OpenAI paired this with progressing its Stargate project with Oracle and SoftBank, expanding into five new AI data center cites and claiming they are on track to complete their $500 billion, 10-gigawatt commitment by the end of 2025, as these announcements plus the original cite in Abilene, Texas cover over $400 billion and 7-gigawatts over three years.
The new sites are Shackelford County, Texas, Dona Ana County, New Mexico, ‘a cite in the midwest to be announced soon,’ the existing Oracle cite in Lordstown, Ohio, and next year a cite in Milam County, Texas developed in partnership with SB energy.
Sam Altman (CEO OpenAI): I don’t think we’ve figured out yet the final form of what financing for compute looks like.
But I assume, like in many other technological revolutions, figuring out the right answer to that will unlock a huge amount of value delivered to society.
What about what this means for training runs? It means there is no physical cap that is likely to bind, it will be more about what makes economic sense since compute will be in high demand on all fronts.
Peter Wildeford: 10GW AI training runs… probably with models 20x-100x larger than GPT-5. We knew this was going to come, and we still don’t have any updates on timeline. How many years the $100B is over is also unclear – the ultimate size of the investment is really contingent on this.
I’d expect the 10GW OpenAI cluster becomes operational around 2027-2028. GPT-5 was disappointing to some but it’s on trend for its size and true scaling hasn’t happened yet. Get ready for the next three years, and we will truly see what some scaled AI models can do.
What about for other purposes? Ambitions are running high.
Haider: Sam Altman says the industry is bottlenecked by compute, and OpenAI can’t meet demand
Over the next 1-2 years, scarcity could force painful tradeoffs,
“cure cancer research vs free global education”
No one wants to make that choice. The only answer: scale up.
Peter Wildeford: Nvidia is busy trying to sell chips to China while our American AI companies can’t get enough compute at home 😡
That seems like quite the timeline for either curing cancer or providing free global education. But yes, compute is going to be like any other limited resource, you’re going to have to make tradeoffs. Spending on health care and education cannot be unlimited, nor does it automatically ‘win in a fight’ against other needs.
Eyes are on the prize. That prize is superintelligence.
Should we have plans and policies designed around the idea that superintelligence is coming? Well, we certainly shouldn’t build our plans and policies around the idea that the AI companies do not expect superintelligence, or that they won’t try to build it.
Roon: all the largest technology fortunes in the world are on record saying they’re betting the entire bank on superintelligence, don’t care about losses, etc. the only growth sector left in planetary capitalism. It is dizzying, running on redline overload.
If that statement does not terrify you, either you did not understand it, you did not think through its consequences, or you have become numb to such things.
Gavin Baker (quoted as a reminder from BuccoCapital Bloke): Mark Zuckerberg, Satya and Sundar just told you in different ways, we are not even thinking about ROI. And the reason they said that is because the people who actually control these companies, the founders, there’s either super-voting stock or significant influence in the case of Microsoft, believe they’re in a race to create a Digital God.
And if you create that first Digital God, we could debate whether it’s tens of trillions or hundreds of trillions of value, and we can debate whether or not that’s ridiculous, but that is what they believe and they believe that if they lose that race, losing the race is an existential threat to the company.
So Larry Page has evidently said internally at Google many times, “I am willing to go bankrupt rather than lose this race.” So everybody is really focused on this ROI equation, but the people making the decisions are not, because they so strongly believe that scaling laws will continue, and there’s a big debate over whether these emergent properties are just in context, learning, et cetera, et cetera, but they believe scaling laws are going to continue. The models are going to get better and more capable, better at reasoning.
And because they have that belief, they’re going to spend until I think there is irrefutable evidence that scaling laws are slowing. And the only way you get irrefutable evidence why has progress slowed down since GPT-4 came out is there hasn’t been a new generation of Nvidia GPUs. It’s what you need to enable that next real step function change in capability.
Mark Zuckerberg (last week): If we end up misspending a couple of hundred billion dollars, I think that that is going to be very unfortunate obviously. But what I’d say is I actually think the risk is higher on the other side.
If you if you um build too slowly and then super intelligence is possible in 3 years, but you built it out assuming it would be there in 5 years, then you’re just out of position on what I think is going to be the most important technology that enables the most new products and innovation and value creation and history.
In case it is not obvious, no, they are not racing for ‘market share’ of inference tokens.
Nvidia is set to generate $165 billion or so in annualized free cash flow by the end of 2028 according to consensus projections:
In the worst case you can always return value to shareholders. Nvidia wisely only pays the minimum $0.01 dividend (so it can be included in various financial products), but has already repurchased $50 billion of its own stock over the past year and plans to buy another $60 billion more.
The argument iby Gallagher is largely that Nvidia cannot make a major acquisition without Chinese approval? I do not understand why they would care, given China is already actively telling companies not to buy Nvidia’s products? Seems like none of their damn business.
But also Nvidia can do further strategic investments and projects instead that can easily eat up this level of free cash flow, if they have no better use for it. Stock market investors would be happy to get more exposure to private companies like OpenAI.
Thinking around Nvidia on Wall Street has for a long time been rather silly and my only regret is I did not buy in far sooner (for those who don’t know, consider this my disclosure that I am overweight Nvidia and have been for a while, quite profitably). For example, a real newspaper like the WSJ will post that the ‘wow’ factor is wearing off because Nvidia is now beating analyst revenue projections by smaller amounts:
If a company keeps beating your expectations, that’s a failure of expectations, with that failure at least getting smaller over time.
Many noticed that when Nvidia invests in OpenAI which uses the money to buy chips from Nvidia, and you move $100 billion back and forth, this can inflate valuations without obviously creating real value.
Such things have happened before.
Andrew Cote: $1 of cash on NVIDIA’s balance sheet is worth $1.
But $1 of revenue at ~50% net margins with a 50x P/E ratio, is worth $25.
Paul Graham (as quoted by Danielle Fong): By 1998, Yahoo was the beneficiary of a de facto Ponzi scheme. Investors were excited about the Internet. One reason they were excited was Yahoo’s revenue growth. So they invested in new Internet startups. The startups then used the money to buy ads on Yahoo to get traffic. Which caused yet more revenue growth for Yahoo, and further convinced investors the Internet was worth investing in.
When I realized this one day, sitting in my cubicle, I jumped up like Archimedes in his bathtub, except instead of “Eureka!” I was shouting “Sell!”
You do have to watch out for the de facto Ponzi scheme, but this is not that, from a pure business perspective it is a clear win-win agreement. You also do have to watch out for a naively multiplying profits into a fixed P/E ratio, but the stock market is smarter than that. The only losers are other AI labs, Microsoft and humanity’s chances of survival.
Peter Wildeford: Nvidia is doing revenue round-tripping at an unprecedented scale (Nvidia servers purchased by companies using money from Nvidia investment). Previously there was CoreWeave, Crusoe, xAI, and Llambda Labs but this is next level. I don’t think any company has ever done this at such a scale. This seems great if we continue AI boom times but could quickly unravel if AI is a bubble.
This is another big loss for Microsoft. Microsoft could’ve been OpenAI’s capital partner of choice as they got in early and had lock-in. But Microsoft wasn’t willing to keep delivering and backed out. This is another nail in the coffin of the Microsoft-OpenAI relationship.
Taking on Nvidia as another huge capital partner provides some much needed stability for OpenAI after being abandoned by Microsoft and their only other partner being SoftBank, which …doesn’t have the best track record.
I guess good to know OpenAI is committing to Nvidia chips for the long haul. Sounds like Microsoft isn’t close to cracking their own chip development like Amazon or Google. That’s good for Nvidia at least.
Ultimately, I am concerned that the US economy is increasingly a leveraged bet on AGI happening. Nvidia is 7% of the US stock market, so their investments matter. AGI happening may mean the end of humanity, but at least the S&P 500 will remain strong.
This probably explains why OpenAI doesn’t talk much about export controls on Nvidia products like Anthropic does… even though such controls would help OpenAI by undermining their Chinese competition.
Nvidia is a bet on AI happening, but in a safe fashion as they are doing it via reinvesting their profits rather than leverage. If AI valuations collapse, Nvidia only goes down roughly proportionally, and they remain a great company.
What about the implications of the deepening of the OpenAI-Nvidia partnership for policy, and willingness to speak out about existential risks?
A key worry has long been that OpenAI depends on Nvidia, so OpenAI may be reluctant to speak about AI risk lest they anger Nvidia and endanger their chip allocations. OpenAI has not been a great political actor, but Sam Altman appreciates many of the dangers and issues that will be at stake.
It is also fortunate that OpenAI’s interests in many ways align with America’s and with those of humanity, far more so than do the incentives of Nvidia. Nvidia wants to make and sell as many chips as possible to the widest variety of customers, whereas OpenAI has a strong interest in America beating China across the board and keeping things under control.
This situation could get far worse. Nvidia may even have made this part of the deal, with Altman agreeing to pursue or not pursue various political objectives, including around export controls. There is even the danger Nvidia ends up with a board seat.
It is likely not a coincidence that Anthropic has found a way to get its compute without Nvidia.
There is a matching bull case, on two fronts.
In terms of Nvidia impacting OpenAI, it is plausible that Nvidia already had maximal leverage over OpenAI via controlling chip allocations, especially if you add that it has the ear of the Administration. At some point, things can’t get any worse. And indeed, locking in these contracts could make things much better, instead, via locking in large commitments that reduce Nvidia’s real leverage.
Then there’s the question of OpenAI impacting Nvidia. Nvidia has a market cap of ~$4.3 trillion, but $100 billion is still ~2.3% of that, and the baseline scenario (since OpenAI is riskier and has more room to grow) is OpenAI equity grows in value faster than Nvidia equity, and Nvidia engages in stock buybacks. So even if no further investments are made, Nvidia could quickly be 5% or more exposed to OpenAI, on top of its exposure to xAI and other Western AI companies, where it makes a variety of strategic investments.
That starts to matter in the calculus for Nvidia. Yes, they can do better short term in their core chip business via selling to China, but that hurts their other investments (and America, but they clearly don’t care about America), and long term the model of compute-intensive top-quality closed models is better for Nvidia anyway.
Usually the incentives point the way you would think. But not always.
DeepMind also addresses something of a meta-concern about AI. The researchers say that a powerful AI in the wrong hands could be dangerous if it is used to accelerate machine learning research, resulting in the creation of more capable and unrestricted AI models. DeepMind says this could “have a significant effect on society’s ability to adapt to and govern powerful AI models.” DeepMind ranks this as a more severe threat than most other CCLs.
The misaligned AI
Most AI security mitigations follow from the assumption that the model is at least trying to follow instructions. Despite years of hallucination, researchers have not managed to make these models completely trustworthy or accurate, but it’s possible that a model’s incentives could be warped, either accidentally or on purpose. If a misaligned AI begins to actively work against humans or ignore instructions, that’s a new kind of problem that goes beyond simple hallucination.
Version 3 of the Frontier Safety Framework introduces an “exploratory approach” to understanding the risks of a misaligned AI. There have already been documented instances of generative AI models engaging in deception and defiant behavior, and DeepMind researchers express concern that it may be difficult to monitor for this kind of behavior in the future.
A misaligned AI might ignore human instructions, produce fraudulent outputs, or refuse to stop operating when requested. For the time being, there’s a fairly straightforward way to combat this outcome. Today’s most advanced simulated reasoning models produce “scratchpad” outputs during the thinking process. Devs are advised to use an automated monitor to double-check the model’s chain-of-thought output for evidence misalignment or deception.
Google says this CCL could become more severe in the future. The team believes models in the coming years may evolve to have effective simulated reasoning without producing a verifiable chain of thought. So your overseer guardrail wouldn’t be able to peer into the reasoning process of such a model. For this theoretical advanced AI, it may be impossible to completely rule out that the model is working against the interests of its human operator.
The framework doesn’t have a good solution to this problem just yet. DeepMind says it is researching possible mitigations for a misaligned AI, but it’s hard to know when or if this problem will become a reality. These “thinking” models have only been common for about a year, and there’s still a lot we don’t know about how they arrive at a given output.
Disney CEO Bob Iger has been under fire for several days now for pulling Jimmy Kimmel Live off the air “indefinitely,” with Disney+’s cancellation page actually crashing a couple of times from all the traffic as people rushed to make their displeasure known. So what better time for the studio to release the first teaser trailer for The Mandalorian and Grogu, a feature film spinoff from its megahit Star Wars series TheMandalorian? Grogu and Mando, together again on an exciting space adventure, will certainly be a crowd-pleaser.
Grogu (aka Baby Yoda) won viewers’ hearts from the moment he first appeared onscreen in the first season of The Mandalorian, and the relationship between the little green creature and his father-figure bounty hunter has only gotten stronger. With the 2023 Hollywood strikes delaying production on S4 of the series, director Jon Favreau got the green light to make this spinoff film.
Per the official logline:
The evil Empire has fallen, and Imperial warlords remain scattered throughout the galaxy. As the fledgling New Republic works to protect everything the Rebellion fought for, they have enlisted the help of legendary Mandalorian bounty hunter Din Djarin (Pedro Pascal) and his young apprentice Grogu.
In addition to Pascal, the cast includes Sigourney Weaver as Ward, a veteran pilot, colonel, and leader of the New Republic’s Adelphi Rangers. Jeremy Allen White plays Rotta the Hutt (son of Jabba, first introduced in 2008’s The Clone Wars), Jonny Coyne reprises his Mandalorian S3 role as an Imperial warlord leading a surviving faction of the Galactic Empire, and we can expect to see Garazeb (“Zeb”) Orrelios from the Star Wars Rebels animated series, too. And yes, that’s a shiny new version of Mando’s ship (destroyed in S2).
The National Academies responded to the EPA’s actions by saying it would prepare a report of its own, which it did despite the threat of a congressional investigation into its work. And the result undercuts the EPA’s claims even further.
Blunt and to the point
The NAS report does not mess around with subtleties, going straight to the main point: Everything we’ve learned since the endangerment finding confirms that it was on target. “EPA’s 2009 finding that the human-caused emissions of greenhouse gases threaten human health and welfare was accurate, has stood the test of time, and is now reinforced by even stronger evidence,” its authors conclude.
That evidence includes a better understanding of the climate itself, with the report citing “Longer records, improved and more robust observational networks, and analytical and methodological advances” that have both allowed us to better detect the changes in the climate, and more reliably assign them to the effects of greenhouse gases. The events attributed to climate change are also clearly harming the welfare of the US public through things like limiting agricultural productivity gains, damage from wildfires, losses due to water scarcity, and general stresses on our infrastructure.
But it’s not just the indirect effects we have to worry about. The changing climate is harming us more directly as well:
Climate change intensifies risks to humans from exposures to extreme heat, ground-level ozone, airborne particulate matter, extreme weather events, and airborne allergens, affecting incidence of cardiovascular, respiratory, and other diseases. Climate change has increased exposure to pollutants from wildfire smoke and dust, which has been linked to adverse health effects. The increasing severity of some extreme events has contributed to injury, illness, and death in affected communities. Health impacts related to climate-sensitive infectious diseases—such as those carried by insects and contaminated water—have increased.
Moreover, it notes that one of the government’s arguments—that US emissions are too small to be meaningful—doesn’t hold water. Even small increments of change will increase the risk of damaging events for decades to come, and push the world closer to hitting potential tipping points in the climate system. Therefore, cutting US emissions will directly reduce those risks.
At one point, Hillary Blackburn, a pharmacist and daughter-in-law of Sen. Marsha Blackburn (R-Tenn.), noted that her mother developed lung cancer two years after getting a COVID-19 vaccine, suggesting, without any evidence, that there could be a link. Evelyn Griffin, an obstetrician and gynecologist in Louisiana who reportedly lost her job for refusing to get a COVID-19 vaccine, meanwhile, did her own research and tried to suggest that the mRNA in mRNA vaccines could be turned into DNA inside human cells and integrate into our genetic material. She made this assertion to a scientist at Pfizer (a maker of an mRNA COVID-19 vaccine), asking him to respond.
With admirable composure, the Pfizer scientist explained that it was not biologically plausible: “RNA cannot reverse transcribe to DNA and transport from the cytoplasm to the nucleus and then integrate. That requires a set of molecules and enzymes that don’t exist in humans and are largely reserved for retroviruses.”
At the very start of the meeting, liaisons from mainstream medical organizations pressed that the ACIP committee needs to ditch such anecdotal nonsense and unvetted data, and return to the high-quality framework for evidence-based decision-making that ACIP has used in the past, which involves comprehensive, methodical evaluations.
Retsef Levi, who works on operations management and has publicly said that COVID-19 vaccines should be removed from the market, responded by falsely claiming that there are no high-quality clinical trials to show vaccine safety, so calls to return to methodological rigor for policy making are hypocritical. “With all due respect, I just encourage all of us to be a little bit more humble,” Levi, who was the head of the ACIP’s COVID-19 working group, said.
During his response, a hot mic picked up someone saying, “You’re an idiot.” It’s unclear who the speaker was—or how many other people they were speaking for.
This post was updated to include the adoption of the recommendation by the CDC.
