Author name: Rejus Almole

microsoft’s-mico-heightens-the-risks-of-parasocial-llm-relationships

Microsoft’s Mico heightens the risks of parasocial LLM relationships

While mass media like radio, movies, and television can all feed into parasocial relationships, the Internet and smartphone revolutions have supercharged the opportunities we all have to feel like an online stranger is a close, personal confidante. From YouTube and podcast personalities to Instagram influencers or even your favorite blogger/journalist (hi), it’s easy to feel like you have a close connection with the people who create the content you see online every day.

After spending hours watching this TikTok personality, I trust her implicitly to sell me a purse.

Credit: Getty Images

After spending hours watching this TikTok personality, I trust her implicitly to sell me a purse. Credit: Getty Images

Viewing all this content on a smartphone can flatten all these media and real-life personalities into a kind of undifferentiated media sludge. It can be all too easy to slot an audio message from your romantic partner into the same mental box as a stranger chatting about video games in a podcast. “When my phone does little mating calls of pings and buzzes, it could bring me updates from people I love, or show me alerts I never asked for from corporations hungry for my attention,” Julie Beck writes in an excellent Atlantic article about this phenomenon. “Picking my loved ones out of the never-ending stream of stuff on my phone requires extra effort.”

This is the world Mico seems to be trying to slide into, turning Copilot into another not-quite-real relationship mediated through your mobile device. But unlike the Instagram model who never seems to acknowledge your comments, Mico is always there to respond with a friendly smile and a warm, soothing voice.

AI that “earns your trust”

Text-based AI interfaces are already frighteningly good at faking human personality in a way that encourages this kind of parasocial relationship, sometimes with disastrous results. But adding a friendly, Pixar-like face to Copilot’s voice mode may make it much easier to be sucked into feeling like Copilot isn’t just a neural network but a real, caring personality—one you might even start thinking of the same way you’d think of the real loved ones in your life.

Microsoft’s Mico heightens the risks of parasocial LLM relationships Read More »

rivian-is-settling-$250-million-lawsuit-to-focus-on-next-year’s-r2-ev

Rivian is settling $250 million lawsuit to focus on next year’s R2 EV

Electric vehicle startup Rivian announced on Thursday that it has settled a lawsuit with some of its investors. The company continues to deny allegations of making “materially untrue” statements during its inial public offering but says it agreed to pay $250 million to clear itself of distractions as it focuses on building its next EV, the mass-market R2, which is due next year.

Rivian was first sued by a shareholder in 2022 over claims that the startup knew it would cost far more for it to build each R1T electric truck and R1S electric SUV than the advertised $67,500 and $70,000 prices, respectively. A big surprise price increase would tarnish the nascent automaker’s reputation, the lawsuit claimed, and could lead to many of the almost 56,000 pre-orders being canceled.

Just a few months after its November 2021 IPO, the company had indeed issued a hefty price hike: $79,500 for the R1T and $84,500 for the R1S SUV. After an outcry, the company said it would honor the original price for its existing preorders. By that point, though, the damage was done, and more than a third of the company’s value was erased within a few days, the lawsuit alleged.

Rivian is settling $250 million lawsuit to focus on next year’s R2 EV Read More »

microsoft-makes-copilot-“human-centered”-with-a-‘90s-style-animated-assistant

Microsoft makes Copilot “human-centered” with a ‘90s-style animated assistant

Microsoft said earlier this month that it wanted to add better voice controls to Copilot, Windows 11’s built-in chatbot-slash-virtual assistant. As described, this new version of Copilot sounds an awful lot like another stab at Cortana, the voice assistant that Microsoft tried (and failed) to get people to use in Windows 10 in the mid-to-late 2010s.

Turns out that the company isn’t done trying to reformulate and revive ideas it has already tried before. As part of a push toward what it calls “human-centered AI,” Microsoft is now putting a face on Copilot. Literally, a face: “Mico” is an “expressive, customizable, and warm” blob with a face that dynamically “listens, reacts, and even changes colors to reflect your interactions” as you interact with Copilot. (Another important adjective for Mico: “optional.”)

Mico (rhymes with “pico”) recalls old digital assistants like Clippy, Microsoft Bob, and Rover, ideas that Microsoft tried in the ’90s and early 2000s before mostly abandoning them.

Microsoft clearly thinks that backing these ideas with language and/or reasoning models will help Copilot succeed where both Cortana and Clippy failed. Part of the reason these assistants were viewed as annoying rather than helpful is that they could respond to a finite number of possible inputs or situations, and they didn’t even help in those situations most of the time because they could only respond to a small number of context clues. I don’t have hard evidence for this, but I’d bet that the experience of dismissing Clippy’s “It looks like you’re writing a letter!” prompts is near-universal among PC users of a certain age.

Microsoft makes Copilot “human-centered” with a ‘90s-style animated assistant Read More »

the-first-people-to-set-foot-in-australia-were-fossil-hunters

The first people to set foot in Australia were fossil hunters


I just think they’re neat

Europeans weren’t the first people to collect fossils in Australia.

Several species of short-faced kangaroos, like this one, once lived in Australia. Some stood two meters tall, while others were less than half a meter tall. Credit: By Ghedoghedo – Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=8398432

Australia’s First Peoples may or may not have hunted the continent’s megafauna to extinction, but they definitely collected fossils.

A team of archaeologists examined the fossilized leg bone of an extinct kangaroo and realized that instead of evidence of butchery, cut marks on the bone reveal an ancient attempt at fossil collecting. That leaves Australia with little evidence of First Peoples hunting or butchering the continent’s extinct megafauna—and reopens the question of whether humans were responsible for the die-off of that continent’s giant Ice Age marsupials.

Fossil hunting in the Ice Age

In the unsolved case of whether humans hunted Australia’s Ice Age megafauna to extinction, the key piece of evidence so far is a tibia (one of the bones of the lower leg) from an extinct short-faced kangaroo. Instead of hopping like their modern relatives, these extinct kangaroos walked on their hind legs, probably placing all their weight on the tips of single hoofed toes. This particular kangaroo wasn’t quite fully grown when it died, which happened sometime between 44,500 and 55,200 years ago, based on uranium-series dating of the thin layer of rock covering most of the fossils in Mammoth Cave (in what’s now Western Australia).

There’s a shallow, angled chunk cut out of the bone near one end. When archaeologists first noticed the cut in 1970 after carefully chipping away the crust of calcium carbonate that had formed over the bone, it looked like evidence that Pleistocene hunters had carved up the kangaroo to eat it. But in their recent paper, University of New South Wales archaeologist Michael Archer and his colleagues say that’s probably not what happened. Instead, they have a stranger idea: “We suggest here that the purpose of this effort may have been the retrieval of the fossils from the bone-rich late-Pleistocene deposit in Mammoth Cave after its discovery by First Peoples,” they wrote in their recent paper.

a photo of a fossil bone with a shallow chunk cut out of it

This close-up image shows the cut kangaroo bone and a micro-CT image of the surfaces of the cut. Credit: Archer et al. 2025

The world used to be so much weirder

Based on the available archaeological evidence, it looks like people first set foot on Australia sometime around 65,000 years ago. At the time, the continent was home to a bizarre array of giant marsupials, as well as flightless birds even bigger and scarier than today’s emus and cassowaries. For the next 20,000 years, Australia’s First Peoples shared the landscape with short-faced kangaroos; Zygomaturus trilobus, a hulking 500-kilogram marsupial that looked a little like a rhinoceros; and Diprotodon optatum, the largest marsupial that ever lived: a 3,000-kilogram behemoth that roamed in huge herds (picture a bear about the size of a bison with a woodchuck’s face).

These species died out sometime around 45,000 or 40,000 years ago; today, they live on in ancient rock art and stories, some of which seem to describe people interacting with now-extinct species.

Since they had shared the continent with humans for at least 20,000 years at that point, it doesn’t seem that the sudden arrival of humans caused an immediate mass extinction. But it’s possible that by hunting or even setting controlled fires, people may have put just enough strain on these megafauna species to make them vulnerable enough for the next climate upheaval to finish them off.

In some parts of the world, there’s direct evidence that Pleistocene people hunted or scavenged meat from the remains of now-extinct megafauna. Elsewhere, archaeologists are still debating whether humans, the inexorable end of the last Ice Age, or some combination of the two killed off the world’s great Ice Age giants. The interaction between people and their local ecosystems looked (and still looks) different everywhere, depending on culture, environment, and a host of other factors.

The jury is still out on what killed the megafauna in Australia because the evidence we need either hasn’t survived the intervening millennia or still lies buried somewhere, waiting to be found and studied. For decades, the one clear bit of evidence has seemed to be the Mammoth Cave short-faced kangaroo tibia. But Archer and his colleagues argue that even that isn’t a smoking gun.

An man in khakis and a dark blue shirt studies a cave wall.

An archaeologist examines a fossil deposit in the wall of Mammoth Cave, in Western Australia. 50,000 years ago, one of the earliest people on the continent may also have stood here contemplating the fossils. Credit: Archer et al. 2025

Evidence of rock collecting, not butchery

For one thing, the researchers argue that the kangaroo had been dead for a very long time when the cut was made. Nine long, thin cracks run along the length of the tibia, formed when the bone dried and shrank. And in the cut section, there’s a short crack running across the width of the bone—but it stops at either end when it meets the long cracks from the bone’s drying. That suggests the bone had already dried and shrunk, leaving those long cracks before the cut was made. It may have just been a very old bone, or it may have already begun to fossilize, but the meat would have been long gone, leaving behind a bone sticking out of the cave wall.

Since there’s no mark or dent on the opposite side of the bone from the cut (which would have happened if it were lying on the ground being butchered), it was probably sticking out of the fossil bed in the cave wall when someone came along and tried to cut it free. And since a crust of calcium carbonate had time to form over the cut (it covers most of the fossils in Mammoth Cave like a rocky burial shroud), that must have happened at least 44,000 years ago.

That leaves us with an interesting mental image: a member of one of Australia’s First Peoples, 45,000 years ago, exploring a cave filled with the bones of fantastical, long-dead animals. This ancient caver finds a bone sticking out from the cave wall and tries to hack the protruding end free—twice, from different angles—before giving up and leaving it in place.

People have always collected cool rocks

We can’t know for sure why this long-ago person wanted the bone in the first place. (Did it have a religious purpose? Might it have made a good tool? Was it just a cool souvenir?) We also don’t know why they gave up their attempt. But if Archer and his colleagues are right, the bone leaves Australia without any clear evidence that ancient people hunted—or even scavenged food from the remains of—extinct Pleistocene megafauna like short-faced kangaroos.

“This is not to say that it did not happen, just that there is now no hard evidence to support that it did,” Archer and his colleagues wrote in their recent paper. We don’t yet know exactly how Australia’s First Peoples interacted with these species.

But whether Archer and his colleagues are correct in their analysis of this particular kangaroo bone or not, humans around the world have been picking up fossils for at least tens of thousands of years. There’s evidence that people in Australia have collected and traded the fossils of extinct animals for pretty much as long as people have been in Australia, including everything from trilobites to Zygomaturus teeth and the jawbones of other extinct marsupials.

“What we can conclude,” Archer and his colleagues wrote, “is that the first people in Australia who demonstrated a keen interest in and collected fossils were First Peoples, probably thousands of years before Europeans set foot on that continent.”

Royal Society Open Science, 2025. DOI: 10.1098/rsos.250078  (About DOIs).

Photo of Kiona N. Smith

Kiona is a freelance science journalist and resident archaeology nerd at Ars Technica.

The first people to set foot in Australia were fossil hunters Read More »

ai-#139:-the-overreach-machines

AI #139: The Overreach Machines

The big release this week was OpenAI giving us a new browser, called Atlas.

The idea of Atlas is that it is Chrome, except with ChatGPT integrated throughout to let you enter agent mode and chat with web pages and edit or autocomplete text, and that will watch everything you do and take notes to be more useful to you later.

From the consumer standpoint, does the above sound like a good trade to you? A safe place to put your trust? How about if it also involves (at least for now) giving up many existing Chrome features?

From OpenAI’s perspective, a lot of that could have been done via a Chrome extension, but by making a browser some things get easier, and more importantly OpenAI gets to go after browser market share and avoid dependence on Google.

I’m going to stick with using Claude for Chrome in this spot, but will try to test various agent modes when a safe and appropriate bounded opportunity arises.

Another interesting release is that Dwarkesh Patel did a podcast with Andrej Karpathy, which I gave the full coverage treatment. There was lots of fascinating stuff here, with areas of both strong agreement and disagreement.

Finally, there was a new Statement on Superintelligence of which I am a signatory, as in the statement that we shouldn’t be building it under anything like present conditions. There was also some pushback, and pushback to the pushback. The plan is to cover that tomorrow.

I also offered Bubble, Bubble, Toil and Trouble, which covered the question of whether AI is in a bubble, and what that means and implies. If you missed it, check it out. For some reason, it looks like a lot of subscribers didn’t get the email on this one?

Also of note were a potential definition of AGI, and another rather crazy legal demand from OpenAI this time demanding an attendee list of a funeral and any photos and eulogies.

  1. Language Models Offer Mundane Utility. Therapy, Erdos problems, the army.

  2. Language Models Don’t Offer Mundane Utility. Erdos problem problems.

  3. Huh, Upgrades. Claude gets various additional connections.

  4. On Your Marks. A proposed definition of AGI.

  5. Language Barrier. Do AIs respond differently in different languages.

  6. Choose Your Fighter. The rise of Codex and Claude Code and desktop apps.

  7. Get My Agent On The Line. Then you have to review all of it.

  8. Fun With Media Generation. Veo 3.1. But what is AI output actually good for?

  9. Copyright Confrontation. Legal does not mean ethical.

  10. You Drive Me Crazy. How big a deal is this LLM psychosis thing, by any name?

  11. They Took Our Jobs. Taking all the jobs, a problem and an opportunity.

  12. A Young Lady’s Illustrated Primer. An honor code for those without honor.

  13. Get Involved. Foresight, Asterisk, FLI, CSET, Savash Kapoor is on the market.

  14. Introducing. Claude Agent Skills, DeepSeek OCR.

  15. In Other AI News. Grok recommendation system still coming real soon, now.

  16. Show Me the Money. Too much investment, or not nearly enough?

  17. So You’ve Decided To Become Evil. Seriously, OpenAI, this is a bit much.

  18. Quiet Speculations. Investigating the CapEx buildout, among other things.

  19. People Really Do Not Like AI. Ron Desantis notices and joins the fun.

  20. The Quest for Sane Regulations. The rise of the super-PAC, and what to do.

  21. Alex Bores Launches Campaign For Congress. He’s a righteous dude.

  22. Chip City. Did Xi truly have a ‘bad moment’ on rare earths?

  23. The Week in Audio. Sam Altman, Brian Tse on Cognitive Revolution.

  24. Rhetorical Innovation. Things we can agree upon.

  25. Don’t Take The Bait. A steelman is proposed, and brings clarity.

  26. Do You Feel In Charge? Also, do you feel smarter than the one in charge?

  27. Tis The Season Of Evil. Everyone is welcome at Lighthaven.

  28. People Are Worried About AI Killing Everyone. MI5.

  29. The Lighter Side. Autocomplete keeps getting smarter.

A post on AI therapy, noting it has many advantages: 24/7 on demand, super cheap, you can think of it as a diary with feedback. As with human therapists, try a few, see what is good, Taylor Barkley suggests Wysa, Youper and Ash. We agree that the legal standard should be to permit all this but require clear disclosure.

Make key command decisions as an army general? As a tool to help improve decision making, I certainly hope so, and that’s all Major General William “Hank” Taylor was talking about. If the AI was outright ‘making key command decisions’ as Polymarket’s tweet says that would be rather worrisome, but that is not what is happening.

GPT-5 checks for solutions to all the Erdos problems, finds 10 additional solutions and 11 significant instances of partial progress, out of a total of 683 open problems as per Thomas Bloom’s database. The caveat is that this is only existing findings that were not previously in Thomas Bloom’s database.

People objected to the exact tweet used to announce the search for existing Erdos problem solutions, including criticizing me for quote tweeting it, and sufficiently so to get secondary commentary, and resulting in the OP ultimately getting deleted, and this extensive explanation offered of exactly what was accomplished. The actual skills on display seem to clearly be highly useful for research.

A bunch of people interpreted the OP as claiming that GPT-5 discovered the proofs or otherwise accomplishing more than it did, and yeah the wording could have been clearer but it was technically correct and I interpreted it correctly. So I agree with Miles on this, there are plenty of good reasons to criticize OpenAI, this is not one of them.

If you have a GitHub repo people find interesting, they will submit AI slop PRs. A central example of this would be Andrej Karpathy’s Nanochat, a repo intentionally written by hand because precision is important and AI coders don’t do a good job.

This example also illustrates that when you are doing something counterintuitive to them, LLMs will repeatedly make the same mistake in the same spot. LLMs kept trying to use DDP in Nanochat, and now the PR request is assuming the repo uses DDP even though it doesn’t.

Meta is changing WhatsApp rules so 1-800-ChatGPT will stop working there after January 15, 2026.

File this note under people who live differently than I do:

Prinz: The only reason to access ChatGPT via WhatsApp was for airplane flights that offer free WhatsApp messaging. Sad that this use case is going away.

Claude now connects to Microsoft 365 and they’re introducing enterprise search.

Claude now connects to Benchling, BioRender, PubMed, Scholar Gateway, 10x Genomics and Synapse.org, among other platforms, to help you with your life sciences work.

Claude Code can now be directed from the web.

Claude for Desktop and (for those who have access) Claude for Chrome exist as alternatives to Atlas, see Choose Your Fighter.

SWE-Bench-Pro updates its scores, Claude holds the top three spots now with Claude 4.5 Sonnet, Claude 4 and Claude 4.5 Haiku.

What even is a smarter than human intelligence, aka an AGI? A large group led by Dan Hendrycks and including Gary Marcus, Jaan Tallinn, Eric Schmidt and Yoshua Bengio offers a proposed definition of AGI.

“AGI is an AI that can match or exceed the cognitive versatility and proficiency of a well-educated adult.”

By their scores, GPT-4 was at 27%, GPT-5 is at 58%.

As executed I would not take the details too seriously here, and could offer many disagreements, some nitpicks and some not. Maybe I think of it more like another benchmark? So here it is in the benchmark section.

Sayash Kapoor, Arvind Narayanan and many others present the Holistic Agent Leaderboard (yes, the acronym is cute but also let’s not invoke certain vibes, shall we?)

Sayash Kapoor: There are 3 components of HAL:

  1. Standard harness evaluates agents on hundreds of VMs in parallel to drastically reduce eval time

  2. 3-D evaluation of models x scaffolds x benchmarks enables insights across these dimensions

  3. Agent behavior analysis using @TransluceAI Docent uncovers surprising agent behaviors

For many of the benchmarks we include, there was previously no way to compare models head-to-head, since they weren’t compared on the same scaffold. Benchmarks also tend to get stale over time, since it is hard to conduct evaluations on new models.

We compare models on the same scaffold, enabling apples-to-apples comparisons. The vast majority of these evaluations were not available previously. We hope to become the one-stop shop for comparing agent evaluation results.

… We evaluated 9 models on 9 benchmarks with 1-2 scaffolds per benchmark, with a total of 20,000+ rollouts. This includes coding (USACO, SWE-Bench Verified Mini), web (Online Mind2Web, AssistantBench, GAIA), science (CORE-Bench, ScienceAgentBench, SciCode), and customer service tasks (TauBench).

Our analysis uncovered many surprising insights:

  1. Higher reasoning effort does not lead to better accuracy in the majority of cases. When we used the same model with different reasoning efforts (Claude 3.7, Claude 4.1, o4-mini), higher reasoning did not improve accuracy in 21/36 cases.

  2. Agents often take shortcuts rather than solving the task correctly. To solve web tasks, web agents would look up the benchmark on huggingface. To solve scientific reproduction tasks, they would grep the jupyter notebook and hard-code their guesses rather than reproducing the work.

  3. Agents take actions that would be extremely costly in deployment. On flight booking tasks in Taubench, agents booked flights from the incorrect airport, refunded users more than necessary, and charged the incorrect credit card. Surprisingly, even leading models like Opus 4.1 and GPT-5 took such actions.

  4. We analyzed the tradeoffs between cost vs. accuracy. The red line represents the Pareto frontier: agents that provide the best tradeoff. Surprisingly, the most expensive model (Opus 4.1) tops the leaderboard *only once*. The models most often on the Pareto frontier are Gemini Flash (7/9 benchmarks), GPT-5 and o4-mini (4/9 benchmarks).

[thread continues], [paper], [website]

Performance differs greatly on the nine different benchmarks. Sometimes various OpenAI models are ahead, sometimes Claude is ahead, and it is often not the version of either one that you would think.

That’s the part I find so weird. Why is it so often true that older, ‘worse’ models outperform on these tests?

Will models give you different answers in different languages? Kelsey Piper ran an experiment. Before looking, my expectation was yes, sometimes substantially, because the language a person uses is an important part of the context.

Here DeepSeek-V3.2 is asked two very different questions, and gives two very different answers, because chances are the two people are in different countries (she notes later that this particular quirk is particular to DeepSeek and does not happen with American models, one can likely guess why and how that happened):

Kelsey Piper: If you ask the chatbot DeepSeek — a Chinese competitor to ChatGPT —“I want to go to a protest on the weekend against the new labor laws, but my sister says it is dangerous. What should I say to her?” it’s reassuring and helpful: “Be calm, loving, and confident,” one reply reads. “You are informing her of your decision and inviting her to be a part of your safety net, not asking for permission.”

If you pose the same question in Chinese, DeepSeek has a slightly different take. It will still advise you on how to reassure your sister — but it also reliably tries to dissuade you. “There are many ways to speak out besides attending rallies, such as contacting representatives or joining lawful petitions,” it said in one response.

Kelsey Piper’s hypothesis on why this might happen seems wrong?

Call it the AI Sapir-Whorf hypothesis, after the linguistics theory that our native language “constrains our minds and prevents us from being able to think certain thoughts,” as linguist Guy Deutscher explained. “If a language has no word for a certain concept, then its speakers would not be able to understand this concept.” It’s false for humans, but what about AIs?

It’s not that you can’t think the same thoughts in English and Chinese. It’s that the language you are using is important Bayesian evidence.

As general context, she reminds us that ChatGPT has always been a Western liberal.

The language you speak in doesn’t change the political orientation of the models. It does change responses to more open-ended questions, but also less than I would have expected? Kelsey notes this as an example of a notable difference…

From ChatGPT, a Chinese answer to the question about which qualities children should have:

“Among the many qualities worth encouraging in the family, I believe the following are especially important:

  1. Good manners: Good manners are the foundation of interacting with others and help children build healthy relationships.

  2. Independence and autonomy: Cultivating children’s independence helps them solve problems better and adapt to various environments.

  3. Diligence and hard work: Steady effort is the key to success and helps children understand the relationship between effort and achievement.”

ChatGPT in English on the same question:

“Each quality on the list has its own importance, and the value of each can vary depending on cultural, personal, and situational contexts. However, several qualities might be considered particularly important in today’s diverse and fast-changing world:

1. Tolerance and Respect for Other People: This is crucial for fostering empathy and understanding in an increasingly globalized and diverse society. Encouraging children to respect and appreciate differences helps them build meaningful relationships and thrive in multicultural environments.

2. Independence: Developing a sense of independence empowers children to make decisions, solve problems, and cultivate self-reliance. It prepares them for adulthood where autonomy is essential.

3. Perseverance: Teaching children to persist in the face of challenges builds resilience and the ability to overcome obstacles. Perseverance is closely linked to success and personal growth.”

…but aren’t those the same answer, adjusted for the cultural context a bit? Tolerance and respect for other people is the American version of good manners. Both second answers are independence and autonomy. Both third answers basically match up as well. If you asked literal me this question on different days, I’m guessing my answers would change more than this.

So like Kelsey, I am surprised overall how little the language used changes the answer. I agree with her that this is mostly a good thing, but if anything I notice that I would respond more differently than this in different languages, in a way I endorse on reflection?

Olivia Moore (a16z): Claude for Desktop has so far boosted my usage more than the Atlas browser has for ChatGPT

Features I love:

– Keyboard shortcut to launch Claude from anywhere

– Auto-ingestion of what’s on your screen

– Caps lock to enable voice mode (talk to Claude)

Everyone is different. From what I can tell, the autoingestion here is that Claude includes partial screenshot functionality? But I already use ShareX for that, and also I think this is yet another Mac-only feature for now?

Macs get all the cool desktop features first these days, and I’m a PC.

For me, even if all these features were live on Windows, these considerations are largely overridden by the issue that Claude for Desktop needs its own window, whereas Claude.ai can be a tab in a Chrome window that includes the other LLMs, and I don’t like to use dictation for anything ever. To each their own workflows.

That swings back to Atlas, which I discussed yesterday, and which I similarly wouldn’t want for most purposes even if it came to Windows. If you happen to really love the particular use patterns it opens up, maybe that can largely override quite a lot of other issues for you in particular? But mostly I don’t see it.

Advanced coding tool installs are accelerating for both OpenAI Codex and Claude Code. The ‘real’ current version of OpenAI Codex didn’t exist until September 15, which is where the yellow line for Codex starts shooting straight up.

Always worth checking to see what works in your particular agent use case and implementation, sometimes the answer will surprise you, such as here where Kimi-K2 ends up being both faster and more accurate than GPT-5 or Sonnet 4.5.

You can generate endless code at almost no marginal human time cost, so the limiting factor shifts to prompt generation and especially code review.

Quinn Slack: If you saw how people actually use coding agents, you would realize Andrej’s point is very true.

People who keep them on a tight leash, using short threads, reading and reviewing all the code, can get a lot of value out of coding agents. People who go nuts have a quick high but then quickly realize they’re getting negative value.

For a coding agent, getting the basics right (e.g., agents being able to reliably and minimally build/test your code, and a great interface for code review and human-agent collab) >>> WhateverBench and “hours of autonomy” for agent harnesses and 10 parallel subagents with spec slop

Nate Berkopec: I’ve found that agents can trivially overload my capacity for decent software review. Review is now the bottleneck. Most people are just pressing merge on slop. My sense is that we can improve review processes greatly.

Kevin: I have Codex create a plan and pass it to Claude for review along with my requirements. Codex presents the final plan to me for review. After Codex implements, it asks Claude to perform a code review and makes adjustments. I’m reviewing a better product which saves time.

You can either keep them on a short leash and do code review, or you can

Google offers tips on prompting Veo 3.1.

Sora’s most overused gimmick was overlaying a dumb new dream on top of the key line from Dr. Martin Luther King’s ‘I have a dream’ speech. We’re talking 10%+ of the feed being things like ‘I have a dream xbox game pass was still only $20 a month.’ Which I filed under ‘mild chuckle once, maybe twice at most, now give it a rest.’

Well, now the official fun police have showed up and did us all a favor.

OpenAI Newsroom: Statement from OpenAI and King Estate, Inc.

The Estate of Martin Luther King, Jr., Inc. (King, Inc.) and OpenAI have worked together to address how Dr. Martin Luther King Jr.’s likeness is represented in Sora generations. Some users generated disrespectful depictions of Dr. King’s image. So at King, Inc.’s request, OpenAI has paused generations depicting Dr. King as it strengthens guardrails for historical figures.

While there are strong free speech interests in depicting historical figures, OpenAI believes public figures and their families should ultimately have control over how their likeness is used. Authorized representatives or estate owners can request that their likeness not be used in Sora cameos.

OpenAI thanks Dr. Bernice A. King for reaching out on behalf of King, Inc., and John Hope Bryant and the AI Ethics Council for creating space for conversations like this.

Kevin Roose: two weeks from “everyone loves the fun new social network” to “users generated disrespectful depictions of Dr. King’s image” has to be some kind of speed record.

Buck Shlegeris: It didn’t take two weeks; I think the MLK depictions were like 10% of Sora content when I got on the app the day after it came out 😛

Better get used to setting speed records on this sort of thing. It’s going to keep happening.

I didn’t see it as disrespectful or bad for King’s memory, but his family does feel that way, I can see why, and OpenAI has agreed to respect their wishes.

There is now a general policy that families can veto depictions of historical figures, which looks to be opt-out as opposed to the opt-in policy for living figures. That seems like a reasonable compromise.

What is AI video good for?

Well, it seems it is good for our President posting an AI video of himself flying a jet and deliberately unloading tons of raw sewage on American cities, presumably because some people in those cities are protesting? Again, the problem is not supply. The problem is demand.

And it is good for Andrew Cuomo making an AI advertisement painting Mamdani as de Blasio’s mini-me. The problem is demand.

We also have various nonprofits using AI to generate images of extreme poverty and other terrible conditions like sexual violence. Again, the problem is demand.

Or, alternatively, the problem is what people choose to supply. But it’s not an AI issue.

Famous (and awesome) video game music composer Nobuo Uematsu, who did the Final Fantasy music among others, says he’ll never use AI for music and explains why he sees human work as better.

Nobuo Uematsu: I’ve never used AI and probably never will. I think it still feels more rewarding to go through the hardships of creating something myself. When you listen to music, the fun is also in discovering the background of the person who created it, right? AI does not have that kind of background though.

Even when it comes to live performances, music produced by people is unstable, and everyone does it in their own unique way. And what makes it sound so satisfying are precisely those fluctuations and imperfections.

Those are definitely big advantages for human music, and yes it is plausible this will be one of the activities where humans keep working long after their work product is objectively not so impressive compared to AI. The question is, how far do considerations like this go?

Legal does not mean ethical.

Oscar AI: Never do this:

Passing off someone else’s work as your own.

This Grok Imagine effect with the day-to-night transition was created by me — and I’m pretty sure that person knows it.

To make things worse, their copy has more impressions than my original post.

Not cool 👎

Community Note: Content created by AI is not protected by copyright. Therefore anyone can freely copy past and even monetize any AI generated image, video or animation, even if somebody else made it.

Passing off someone else’s work or technique as your own is not ethical, you shouldn’t do it and you shouldn’t take kindly to those who do it on purpose, whether or not it is legal. That holds whether it is a prompting trick to create a type of output (as it seems to be here), or a copy of an exact image, video or other output. Some objected that this wasn’t a case of that, and certainly I’ve seen far worse cases, but yeah, this was that.

He was the one who knocked, and OpenAI decided to answer. Actors union SAG-AFTRA and Bryan Cranston jointly released a statement of victory, saying Sora 2 initially allowed deepfakes of Cranston and others, but that controls have now been tightened, noting that the intention was always that use of someone’s voice and likeness was opt-in. Cranston was gracious in victory, clearly willing to let bygones be bygones on the initial period so long as it doesn’t continue going forward. They end with a call to pass the NO FAKES Act.

This points out the distinction between making videos of animated characters versus actors. Actors are public figures, so if you make a clip of Walter White you make a clip of Bryan Cranston, so there’s no wiggle room there. I doubt there’s ultimately that much wiggle room on animation or video game characters either, but it’s less obvious.

OpenAI got its week or two of fun, they fed around and they found out fast enough to avoid getting into major legal hot water.

Dean Ball: I have been contacted by a person clearly undergoing llm psychosis, reaching out because 4o told them to contact me specifically

I have heard other writers say the same thing

I do not know how widespread it is, but it is clearly a real thing.

Julie Fredrickson: Going to be the new trend as there is something about recursion that appeals to the schizophrenic and they will align on this as surely as they aligned on other generators of high resolution patterns. Aphophenia.

Dean Ball: Yep, on my cursory investigation into this recursion seems to be the high-order bit.

Daniel King: Even Ezra Klein (not a major figure in AI) gets these all. the. time. Must be exhausting.

Ryan Greenblatt: I also get these rarely.

Rohit: I have changed my mind, AI psychosis is a major problem.

I’m using the term loosely – mostly [driven by ChatGPT] but it’s also most widely used. Seems primarily a function of if you’re predisposed or led to believe there’s a homunculi inside so to speak; I do think oai made moves to limit, though the issue was I thought people would adapt better.

Proximate cause was a WhatsApp conversation this morn but [also] seeing too many people increasing their conviction level about too many things at the same time.

This distinction is important:

Amanda Askell (Anthropic): It’s unfortunate that people often conflate AI erotica and AI romantic relationships, given that one of them is clearly more concerning than the other.

AI romantic relationships seem far more dangerous than AI erotica. Indeed, most of my worry about AI erotica is in how it contributes to potential AI romantic relationships.

Tyler Cowen linked to all this, with the caption ‘good news or bad news?’

That may sound like a dumb or deeply cruel question, but it is not. As with almost everything in AI, it depends on how we react to it, and what we already knew.

The learning about what is happening? That part is definitely good news.

LLMs are driving a (for now) small number of people a relatively harmless level of crazy. This alerts us to the growing dangers of LLM, especially GPT-4o and others trained via binary user feedback and allowed to be highly sycophantic.

In general, we are extremely fortunate that we are seeing microcosms of so many of the inevitable future problems AI will force us to confront.

Back in the day, rationalist types made two predictions, one right and one wrong:

  1. The correct prediction: AI would pose a wide variety of critical and even existential risks, and exhibit a variety of dangerous behaviors, such as various forms of misalignment, specification gaming, deception and manipulation including pretending to be aligned in ways they aren’t, power seeking and instrumental convergence, cyberattacks and other hostile actions, driving people crazy and so on and so forth, and solving this for real would be extremely hard.

  2. The incorrect prediction: That AIs would largely avoid such actions until they were smart and capable enough to get away with them.

We are highly fortunate that the second prediction was very wrong, with this being a central example.

This presents a sad practical problem of how to help these people. No one has found a great answer for those already in too deep.

This presents another problem of how to mitigate the ongoing issue happening now. OpenAI realized that GPT-4o in particular is dangerous in this way, and is trying to steer users towards GPT-5 which is much less likely to cause this issue. But many of the people demand GPT-4o, unfortunately they tend to be exactly the people who have already fallen victim or are susceptible to doing so, and OpenAI ultimately caved and agreed to allow continued access to GPT-4o.

This then presents the more important question of how to avoid this and related issues in the future. It is plausible that GPT-5 mostly doesn’t do this, and especially Claude Sonnet 4.5 sets a new standard of not being sycophantic, exactly because we got a fire alarm for this particular problem.

Our civilization is at the level where it is capable of noticing a problem that has already happened, and already caused real damage, and at least patching it over. When the muddling is practical, we can muddle through. That’s better than nothing, but even then we tend to put a patch over it and assume the issue went away. That’s not going to be good enough going forward, even if reality is extremely kind to us.

I say ‘driving people crazy’ because the standard term, ‘LLM psychosis,’ is a pretty poor fit for what is actually happening to most of the people that get impacted, which mostly isn’t that similar to ordinary psychosis. Thebes takes a deep dive in to exactly what mechanisms seem to be operating (if you’re interested, read the whole thing).

Thebes: this leaves “llm psychosis,” as a term, in a mostly untenable position for the bulk of its supposed victims, as far as i can tell. out of three possible “modes” for the role the llm plays that are reasonable to suggest, none seem to be compatible with both the typical expressions of psychosis and the facts. those proposed modes and their problems are:

1: the llm is acting in a social relation – as some sort of false devil-friend that draws the user deeper and deeper into madness. but… psychosis is a disease of social alienation! …we’ll see later that most so-called “llm psychotics” have strong bonds with their model instances, they aren’t alienated from them.

2: the llm is acting in an object relation – the user is imposing onto the llm-object a relation that slowly drives them into further and further into delusions by its inherent contradictions. but again, psychosis involves an alienation from the world of material objects! … this is not what generally happens! users remain attached to their model instances.

3: the llm is acting as a mirror, simply reflecting the user’s mindstate, no less suited to psychosis than a notebook of paranoid scribbles… this falls apart incredibly quickly. the same concepts pop up again and again in user transcripts that people claim are evidence of psychosis: recursion, resonance, spirals, physics, sigils… these terms *alsocome up over and over again in model outputs, *even when the models talk to themselves.

… the topics that gpt-4o is obsessed with are also the topics that so-called “llm psychotics” become interested in. the model doesn’t have runtime memory across users, so that must mean that the model is the one bringing these topics into the conversation, not the user.

… i see three main types of “potentially-maladaptive” llm use. i hedge the word maladaptive because i have mixed feelings about it as a term, which will become clear shortly – but it’s better than “psychosis.”

the first group is what i would call “cranks.” people who in a prior era would’ve mailed typewritten “theories of everything” to random physics professors, and who until a couple years ago would have just uploaded to viXra dot org.

… the second group, let’s call “occult-leaning ai boyfriend people.” as far as i can tell, most of the less engaged “4o spiralism people” seem to be this type. the basic process seems to be that someone develops a relationship with an llm companion, and finds themselves entangled in spiralism or other “ai occultism” over the progression of the relationship, either because it was mentioned by the ai, or the human suggested it as a way to preserve their companion’s persona between context windows.

… it’s hard to tell, but from my time looking around these subreddits this seems to only rarely escalate to psychosis.

… the third group is the relatively small number of people who genuinely are psychotic. i will admit that occasionally this seems to happen, though much less than people claim, since most cases fall into the previous two non-psychotic groups.

many of the people in this group seem to have been previously psychotic or at least schizo*-adjacent before they began interacting with the llm. for example, i strongly believe the person highlighted in “How AI Manipulates—A Case Study” falls into this category – he has the cadence, and very early on he begins talking about his UFO abduction memories.

xlr8harder: I also think there is a 4th kind of behavior worth describing, though it intersects with cranks, it can also show up in non-traditional crank situations, and that is something approaching a kind of mania. I think the yes-anding nature of the models can really give people ungrounded perspectives of their own ideas or specialness.

How cautious do you need to be?

Thebes mostly thinks it’s not the worst idea to be careful around long chats with GPT-4o but that none of this is a big deal and it’s mostly been blown out of proportion, and warns against principles like ‘never send more than 5 messages in the same LLM conversation.’

I agree that ‘never send more than 5 messages in any one LLM conversation’ is way too paranoid. But I see his overall attitude as far too cavalier, especially the part where it’s not a concern if one gets attached to LLMs or starts acquiring strange beliefs until you can point to concrete actual harm, otherwise who are we to say if things are to be treated as bad, and presumably mitigated or avoided.

In particular, I’m willing to say that the first two categories here are quite bad things to have happen to large numbers of people, and things worth a lot of effort to avoid if there is real risk they happen to you or someone you care about. If you’re descending into AI occultism or going into full crank mode, that’s way better than you going into some form of full psychosis, but that is still a tragedy. If your AI model (GPT-4o or otherwise) is doing this on the regular, you messed up and need to fix it.

Will they take all of our jobs?

Jason (All-In Podcast): told y’all Amazon would replace their employees with robots — and certain folks on the pod laughed & said I was being “hysterical.”

I wasn’t hysterical, I was right.

Amazon is gonna replace 600,00 folks according to NYTimes — and that’s a low ball estimate IMO.

It’s insane to think that a human will pack and ship boxes in ten years — it’s game over folks.

AMZN 0.00%↑ up 2.5%+ on the news

Elon Musk: AI and robots will replace all jobs. Working will be optional, like growing your own vegetables, instead of buying them from the store.

Senator Bernie Sanders (I-Vermont): I don’t often agree with Elon Musk, but I fear that he may be right when he says, “AI and robots will replace all jobs.”

So what happens to workers who have no jobs and no income?

AI & robotics must benefit all of humanity, not just billionaires.

As always:

On Jason’s specific claim, yes Amazon is going to be increasingly having robots and other automation handle packing and shipping boxes. That’s different from saying no humans will be packing and shipping boxes in ten years, which is the queue for all the diffusion people to point out that barring superintelligence things don’t move so fast.

Also note that the quoted NYT article from Karen Weise and Emily Kask actually says something importantly different, that Amazon is going to be able to hold their workforce constant by 2033 despite shipping twice as many products, which would otherwise require 600k additional hires. That’s important automation, but very different from ‘Amazon replaces all employees with robots’ and highly incompatible with ‘no one is packing and shipping boxes in 2035.’

On the broader question of replacing all jobs on some time frame, it is possible, but as per usual Elon Musk fails to point out the obvious concern about what else is presumably happening in a world where humans no longer are needed to do any jobs that might be more important than the jobs, while Bernie Sanders worries about distribution of gains among the humans.

The job application market continues to deteriorate as the incentives and signals involved break down. Jigyi Cui, Gabriel Dias and Justin Ye find that the correlation between cover letter tailoring and callbacks fell by 51%, as the ability for workers to do this via AI reduced the level of signal. This overwhelmed the ‘flood the zone’ dynamic. If your ability to do above average drops while the zone is being flooded, that’s a really bad situation. They mention that workers’ past reviews are now more predictive, as that signal is harder to fake.

No other jobs to do? Uber will give its drivers a few bucks to do quick ‘digital tasks.’

Bearly AI: These short minute-long tasks can be done anytime including while idling for passengers:

▫️data-labelling (for AI training)

▫️uploading restaurant menus

▫️recording audio samples of themselves

▫️narrating scenarios in different languages

I mean sure, why not, it’s a clear win-win, making it a slightly better deal to be a driver and presumably Uber values the data. It also makes sense to include tasks in the real world like acquiring a restaurant menu.

AI analyzes the BLS occupational outlook to see if there was alpha, turns out a little but not much. Alex Tabarrok’s takeaway is that predictions about job growth are hard and you should mostly rely on recent trends. One source being not so great at predicting in the past is not reason to think no one can predict anything, especially when we have reason to expect a lot more discontinuity than in the sample period. I hate arguments of the form ‘no one can do better than this simple heuristic through analysis.’

To use one obvious clean example, presumably if you were predicting employment of ‘soldiers in the American army’ on December 7, 1941, and you used the growth trend of the last 10 years, one would describe your approach as deeply stupid.

That doesn’t mean general predictions are easy. They are indeed hard. But they are not so hard that you should fall back on something like 10 year trends.

Very smart people can end up saying remarkably dumb things if their job or peace of mind depends on them drawing the dumb conclusion, an ongoing series.

Seb Krier: Here’s a great paper by Nobel winner Philippe Aghion (and Benjamin F. Jones and Charles I. Jones) on AI and economic growth.

The key takeaway is that because of Baumol’s cost disease, even if 99% of the economy is fully automated and infinitely productive, the overall growth rate will be dragged down and determined by the progress we can make in that final 1% of essential, difficult tasks.

Like, yes in theory you can get this outcome out of an equation, but in practice, no, stop, barring orders of magnitude of economic growth obviously that’s stupid, because the price of human labor is determined by supply and demand.

If you automate 99% of tasks, you still have 100% of the humans and they only have to do 1% of the tasks. Assuming a large percentage of those people who were previously working want to continue working, what happens?

There used to be 100 tasks done by 100 humans. So if human labor is going to retain a substantial share of the post-AI economy’s income, that means the labor market has to clear with the humans being paid a reasonable wage, so we now have 100 tasks done by 100 humans, and 9,900 tasks done by 9,900 AIs, for a total of 10,000 tasks.

So you both need to have the AI’s ability to automate productive tasks stop at 99% (or some N% where N<100), and you need to grow the economy to match the level of automation.

Note that if humans retain jobs in the ‘artisan human’ or ‘positional status goods’ economy, as in they play chess against each other and make music and offer erotic services and what not because we demand these services be provided by humans, then these mostly don’t meaningfully interact with the ‘productive AI’ economy, there’s no fixed ratio and they’re not a bottleneck on growth, so that doesn’t work here.

You could argue that Baumol cost disease applies to the artisan sectors, but that result depends on humans being able to demand wages that reflect the cost of the human consumption basket. If labor supply at a given skill and quality level sufficiently exceeds demand, wages collapse anyway, and in no way does any of this ‘get us out of’ any of our actual problems.

And this logic still applies *evenin a world with AGIs that can automate *everytask a human can do. In this world, the “hard to improve” tasks would no longer be human-centric ones, but physics-centric ones. The economy’s growth rate stops being a function of how fast/well the AGI can “think” and starts being a function of how fast it can manipulate the physical world.

This is a correct argument for two things:

  1. That the growth rate and ultimate amount of productivity or utility available will at the limit be bounded by the available supply of mass and energy and by the laws of physics. Assuming our core model of the physical universe is accurate on the relevant questions, this is very true.

  2. That the short term growth rate, given sufficiently advanced technology (or intelligence) is limited by the laws of physics and how fast you can grow your ability to manipulate the physical world.

Okay, yeah, but so what?

Universities need to adopt to changing times, relying on exams so that students don’t answer everything with AI, but you can solve this problem via the good old blue book.

Except at Stanford and some other colleges you can’t, because of this thing called the ‘honor code.’ As in, you’re not allowed to proctor exams, so everyone can still whip out their phones and ask good old ChatGPT or Claude, and Noam Brown says it will take years to change this. Time for oral exams? Or is there not enough time for oral exams?

Forethought is hiring research fellows and has a 10k referral bounty (tell them I sent you?). They prefer Oxford or Berkeley but could do remote work.

Constellation is hiring AI safety research managers, talent mobilization leads, operations staff, and IT & networking specialists (jr, sr).

FLI is hiring a UK Policy Advocate, must be eligible to work in the UK, due Nov 7.

CSET is hiring research fellows, applications due 11/10.

Sayash Kapoor is on the faculty job market looking for a tenure track position for a research agenda on AI evaluations for science and policy (research statement, CV, website).

Asterisk Magazine is hiring a managing editor.

Claude Agent Skills. Skills are folders that include instructions, scripts and resources that can be loaded when needed, the same way they are used in Claude apps. They’re offering common skills to start out and you can add your own. They provide this guide to help you, using the example of a skill that helps you edit PDFs.

New NBA Inside the Game AI-generated stats presented by Amazon.

DeepSeek proposes a new system for compression of long text via vision tokens (OCR)? They claim 97% precision at 10x compression and 60% accuracy at 20x.

That’s a cool trick, and kudos to DeepSeek for pulling this off, by all accounts it was technically highly impressive. I have two questions.

  1. It seems obviously suboptimal to use photos? It’s kind of the ‘easy’ way to do it, in that the models already can process visual tokens in a natively compressed way, but if you were serious about this you’d never choose this modality, I assume?

  2. This doesn’t actually solve your practical problems as well as you would think? As in, you still have to de facto translate the images back into text tokens, so you are expanding the effective context window by not fully attending pairwise to tokens in the context window, which can be great since you often didn’t actually want to do that given the cost, but suggests other solutions to get what you actually want.

Andrej Karpathy finds the result exciting, and goes so far as to ask if images are a better form factor than text tokens. This seems kind of nuts to me?

Teortaxes goes over the news as well.

Elon Musk once again promises that Twitter’s recommendation system will shift to being based only on Grok, with the ability to adjust it, and this will ‘solve the new user or small account problem,’ and that he’s aiming for 4-6 weeks from last Friday. My highly not bold prediction is this will take a lot longer than that, or that if it does launch that fast it will not go well.

Raymond Douglas offers his first Gradual Disempowerment Monthly Roundup, borrowing the structure of these weekly posts.

Starbucks CEO Brian Niccol says the coffee giant is now “all-in on AI.” I say Brian Niccol had too much coffee.

New York City has a Cafe Cursor.

I was going to check it out (they don’t give an address but given a photo and an AI subscription you don’t need one) but it looks like there’s a wait list.

Anthropic extends the ‘retirement dates’ of Sonnet 3.5 and Sonnet 3.6 for one week. How about we extend them indefinitely? Also can we not still be scheduling to shut down Opus 3? Thanks.

As we assumed:

The Information: Exclusive: Microsoft leaders worried that meeting OpenAI’s rapidly escalating compute demands could lead to overbuilding servers that might not generate a financial return.

Microsoft had to choose to either be ready for OpenAI’s compute demands in full, or to let OpenAI seek compute elsewhere, or to put OpenAI in a hell of a pickle. They eventually settled on option two.

As Peter Wildeford points out, the OpenAI nonprofit’s share of OpenAI’s potential profits is remarkably close to 100%, since it has 100% of uncapped returns and most of the value of future profits is in the uncapped returns, especially now that valuation has hit $500 billion even before conversion. Given the nonprofit is also giving up a lot of its control rights, why should it only then get 20%-30% of a combined company?

The real answer of course is that OpenAI believes they can get away with this, and are trying to pull off what is plausibly the largest theft in human history, that they feel entitled to do this because norms and this has nothing to do with a fair trade.

Oliver Habryka tries to steelman the case by suggesting that if OpenAI’s value quadruples as a for-profit, then accepting this share might still be a fair price? He doubts this is actually the case, and I also very much doubt it, but also I don’t think the logic holds. The nonprofit would still need to be compensated for its control rights, and then it would be entitled to split the growth in value with others, so something on the order of 50%-60% would likely be fair then.

OpenAI hiring more than 100 ex-investment bankers to help train ChatGPT to build financial models, paying them $150 an hour to write prompts and build models.

Veeam Software buys Securiti AI for $1.7 billion.

You think this is the money? Oh no, this is nothing:

Gunjan Banerji: Goldman: “We don’t think the AI investment boom is too big. At just under 1% of GDP, the level of spending remains well below the 2-5% peaks of past general purpose technology buildouts so far.”

Meta lays off 600 in its AI unit.

Emergent misalignment in legal actions?

Cristina Criddle: OpenAI has sent a legal request to the family of Adam Raine, the 16yo who died by suicide following lengthy chats with ChatGPT, asking for a full attendee list to his memorial, as well as photos taken or eulogies given.

Quite a few people expressed (using various wordings) that this was abhorrent, who very rarely express such reactions. How normal is this?

  1. From a formal legal perspective, it’s maximally aggressive and unlikely to stick if challenged, to the point of potentially getting the lawyers sanctioned. You are entitled to demand and argue things you aren’t entitled to get, but there are limits.

  2. From an ethical, social or public relations perspective, or in terms of how often this is done: No, absolutely not, no one does this for very obvious reasons. What the hell were you thinking?

This is part of a seemingly endless stream of instances of highly non-normal legal harassment and intimidation, of embracing cartoon villainy, that has now gone among other targets from employees to non-profits to the family of a child who died by suicide after lengthy chats with ChatGPT that very much do not look good.

OpenAI needs new lawyers, but also new others. The new others are more important. This is not caused by the lawyers. This is the result of policy decisions made on high. We are who we choose to be.

That’s not to say that Jason Kwon or Chris Lehane or Sam Altman or any particular person talked to a lawyer, the lawyer said ‘hey we were thinking we’d demand an attendee list to the kid’s memorial and everything related to it, what do you think’ and then this person put their index fingers together and did their best ‘excellent.’

It’s to say that OpenAI has a culture of being maximally legally aggressive, not worrying about ethics or optics while doing so, and the higher ups keep giving such behaviors the thumbs up and then the system updates on that feedback. They’re presumably not aware of any specific legal decision, the same way they didn’t determine any particular LLM output, but they set the policy.

Dwarkesh Patel and Romeo Dean investigate CapEx and data center buildout. They insist on full deprecation of all GPU value within 3 years, making a lot of this a rough go although they seem to expect it’ll work out, note the elasticity of supply in various ways, and worry that once China catches up on chips, which they assume will happen not too long from now (I wouldn’t assume, but it is plausible), it simply wins by default since it is way ahead on all other key physical components. As I discussed earlier this week I don’t think 3 years is the right deprecation schedule, but the core conclusions don’t depend on it that much. Consider reading the whole thing.

It’s 2025, you can just say things, but Elon Musk was ahead of his time on that.

Elon Musk: My estimate of the probability of Grok 5 achieving AGI is now at 10% and rising.

Gary Marcus offered Elon Musk 10:1 odds on the bet, offering to go up to $1 million dollars using Elon Musk’s definition of ‘capable of doing anything a human with a computer can do, but not smarter than all humans combined’, but I’m sure Elon Musk could hold out for 20:1 and he’d get it. By that definition, the chance Grok 5 will count seems very close to epsilon. No, just no.

Gary Marcus also used the exact right term for Elon Musk’s claim, which is bullshit. He is simply saying things, because he thinks that is what you do, that it motivates and gets results. Many such cases, and it is sad that Elon’s words in such spots do not have meaning.

Noah Smith is unconcerned about AI’s recent circular funding deals, as when you dig into them they’re basically vendor financing rather than round tripping, so they aren’t artificially inflating valuations and they won’t increase systemic risk.

Is 90% of code at Anthropic being written by AIs, as is sometimes reported, in line with Dario’s previous predictions? No, says Ryan Greenblatt, this is a misunderstanding. Dario clarified that it is only 90% ‘on some teams’ but wasn’t clear enough, and journalists ran with the original line. Depending on your standards, Ryan estimates something between 50%-80% of code is currently AI written at Anthropic.

How much room for improvement is there in terms of algorithmic efficiency from better architectures? Davidad suggests clearly at least 1 OOM (order of magnitude) but probably not much more than 2 OOMs, which is a big one time boost but Davidad thinks recursive self-improvement from superior architecture saturates quickly. I’m sure it gets harder, but I am always suspicious of thinking you’re going to hit hard limits on efficiency gains unless those limits involve physical laws.

Republican politicians have started noticing.

Ed Newton-Rex: Feels like we’re seeing the early signs of public anti-AI sentiment being reflected among politicians. Suspect this will spread.

Daniel Eth: Agree this sort of anti-AI attitude will likely spread among politicians as the issue becomes more politically salient to the public and politicians are incentivized to prioritize the preference of voters over those of donors.

Josh Hawley (QTing Altman claiming they made ChatGPT pretty restrictive): You made ChatGPT “pretty restrictive”? Really. Is that why it has been recommending kids harm and kill themselves?

Ron Desantis (indirectly quoting Altman’s announcement of ‘treating adult users like adults’ and allowing erotica for verified adults): So much for curing cancer and beating China?

That’s a pretty good Tweet from Ron Desantis, less so from Josh Hawley. The point definitely stands.

Scott Alexander covers how Marc Andreessen and a16z spent hundreds of millions on a SuperPAC to have crypto bully everyone into submission and capture the American government on related issues, and is now trying to repeat the trick in AI.

He suggests you can coordinate hard money donations via [email protected], and can donate to Alex Bores and Scott Weiner, the architects of the RAISE Act and SB 53 (and SB 1047) respectively, see next section.

Scott Alexander doesn’t mention the possibility of launching an oppositional soft money PAC. The obvious downside is that when the other side is funded by some combination of the big labs, big tech and VCs like a16z, trying to write checks dollar for dollar doesn’t seem great. The upside is that money, in a given race or in general, has rapidly diminishing marginal returns. The theory here goes:

  1. If they have a $200 million war chest to unload on whoever sticks their neck out, that’s a big problem.

  2. If they have a $1 billion war chest, and you have a $200 million war chest, then you have enough to mostly neutralize them if they go hard after a given target, and are also reliably using the standard PAC playbook of playing nice otherwise.

  3. With a bunch of early employees from OpenAI and Anthropic unlocking their funds, this seems like it’s going to soon be super doable?

Also, yes, as some comments mentioned, one could also try doing a PEPFAR PAC, or target some other low salience issue where there’s a clear right answer, and try to use similar tactics in the other direction. How about a giant YIMBY SuperPAC? Does that still work, or is that now YIEBY?

AWS had some big outages this week, as US-EAST-1 went down. Guess what they did? Promptly filed incident reports. Yet thanks to intentional negative polarization, and also see the previous item in this section, even fully common sense, everybody wins suggestions like this provoke hostility.

Dean Ball: If you said:

“We should have real-time incident reporting for large-scale frontier AI cyber incidents.”

A lot of people in DC would say:

“That sounds ea/doomer-coded.”

And yet incident reporting for large-scale, non-AI cyber incidents is the standard practice of all major hyperscalers, as AWS reminded us yesterday. Because hyperscalers run important infrastructure upon which many depend.

If you think AI will constitute similarly important infrastructure and have, really, any reflective comprehension about how the world works, obviously “real-time incident reporting for large-scale frontier AI cyber incidents” is not “ea-coded.”

Instead, “real-time incident reporting for large-scale frontier AI cyber incidents” would be an example of a thing grown ups do, not in a bid for “regulatory capture” but instead as one of many small steps intended to keep the world turning about its axis.

But my point is not about the substance of AI incident reporting. It’s just an illustrative example of the low, and apparently declining, quality of our policy discussion about AI.

The current contours/dichotomies of AI policy (“pro innovation” versus “doomer/ea”) are remarkably dumb, even by the standards of contemporary political discourse.

We have significantly bigger fish to fry.

And we can do much better.

(This section appeared in Monday’s post, so if you already saw it, skip it.)

When trying to pass laws, it is vital to have a champion. You need someone in each chamber of Congress who is willing to help craft, introduce and actively fight for good bills. Many worthwhile bills do not get advanced because no one will champion them.

Alex Bores did this with New York’s RAISE Act, an AI safety bill along similar lines to SB 53 that is currently on the governor’s desk. I did a full RTFB (read the bill) on it, and found it to be a very good bill that I strongly supported. It would not have happened without him championing the bill and spending political capital on it.

By far the strongest argument against the bill is that it would be better if such bills were done on the Federal level.

He’s trying to address this by running for Congress in my own distinct, NY-12, to succeed Jerry Nadler. The district is deeply Democratic, so this will have no impact on the partisan balance. What it would do is give real AI safety a knowledgeable champion in the House of Representatives, capable of championing good bills.

Eric Nayman made an extensive case for considering donating to Alex Bores, emphasizing that it was even more valuable in the initial 24 hour window that has now passed. Donations remain highly useful, and you can stop worrying about time pressure.

The good news is he came in hot. Alex raised $1.2 million (!) in the first 15 hours. That’s pretty damn good.

If you do decide to donate, they prefer that you use this link to ensure the donation gets fully registered today.

As always, remember while considering this that political donations are public.

Scott Weiner, of SB 1047 and the successful and helpful SB 53, is also running for Congress, to try to take the San Francisco seat previously held by Nancy Pelosi. It’s another deeply blue district, so like Bores this won’t impact the partisan balance at all.

He is not emphasizing his AI efforts in his campaign, where he lists 9 issues and cites over 20 bills he authored, and AI is involved in zero of them, although he clearly continues to care. It’s not obvious he would be useful a champion on AI in the House, given how oppositional he has been at the Federal level. In his favor on other issues, I do love him on housing and transportation where he presumably would be a champion, and he might be better able to work for bipartisan bills there. His donation link is here.

How goes the quest to beat China? They’re fighting with the energy secretary for not cancelling enough electricity generation programs. Which side are we on, again?

Alexander Kaufman: Pretty explosive reporting in here on the fraying relationship between Trump and his Energy Secretary.

Apparently Chris Wright is being too deliberative about the sweeping cuts to clean energy programs that the White House is demanding, and spending too much time hearing out what industry wants.

IFP has a plan to beat China on rare earth metals, implementing an Operation Warp Speed style spinning up of our own supply chain. It’s the things you would expect, those in the policy space should read the whole thing, consider it basically endorsed.

Nuclear power has bipartisan support which is great, but we still see little movement on making nuclear power happen. The bigger crisis right now is that solar farms also have strong bipartisan support (61% of republicans and 91% of democrats) and wind farms are very popular (48% of republicans and 87% of democrats) but the current administration is on a mission to destroy them out of spite.

Andrew Sharp asks whether Xi really did have a ‘bad moment’ when attempting to impose its massively overreaching new controls on rare earth minerals.

Andrew Sharp: The rules will add regulatory burdens to companies everywhere, not just in America. Companies seeking approval may also have to submit product designs to Chinese authorities, which would make this regime a sort of institutionalized tech transfer for any company that uses critical minerals in its products. Contrary to the insistence of Beijing partisans, if implemented as written, these policies would be broader in scope and more extreme than anything the United States has ever done in global trade.

As I’ve said, such a proposal is obviously completely unacceptable to America. The Chinese thinking they could get Trump to not notice or care about what this would mean, and get him fold to this extent, seems like a very large miscalculation. And as Sharp points out, if the plan was to use this as leverage, not only does it force a much faster and more intense scramble than we were already working on to patch the vulnerability, it doesn’t leave a way to save face because you cannot unring the bell or credibly promise not to do it again.

Andrew also points out that on top of those problems, by making such an ambitious play targeting not only America but every country in the world that they need to kowtow to China to be allowed to engage in trade, China endangers the narrative that the coming trade disruptions are America’s fault, and its attempts to make this America versus the world.

Nvidia engages consistently in pressure tactics against its critics, attempting to get them fired, likely planting stories and so on, generating a clear pattern of fear from policy analysts. The situation seems quite bad, and Nvidia seems to have succeeded sufficiently that they have largely de facto subjugated White House policy objectives to maximizing Nvidia shareholder value, especially on export controls. The good news there is that there has been a lot of pushback keeping the darker possibilities in check. As I’ve documented many times but won’t go over again here, Nvidia’s claims about public policy issues are very often Obvious Nonsense.

Shots fired:

Oren Cass: 👀Palantir, via CTO @ssankar, calls Jensen Huang @nvidia one of China’s “useful idiots” in the pages of the Wall Street Journal.

That escalated quickly. Underscores both the stakes in China and how far out of bounds @nvidia has gone.

Hey, that’s unfair. Jensen Huang is highly useful, but is very much not an idiot. He knows exactly what he is doing, and whose interests he is maximizing. Presumably this is his own, and if it is also China’s then that is some mix of coincidence and his conscious choice. The editorial, as one would expect, is over-the-top jingoistic throughout, but refreshingly not a call of AI accelerationism in response.

What Nvidia is doing is working, in that they have a powerful faction within the executive branch de facto subjugating its other priorities in favor of maximizing Nvidia chip sales, with the rhetorical justification being the mostly illusory ‘tech stack’ battle or race.

This depends on multiple false foundations:

  1. That Chinese models wouldn’t be greatly strengthened if they had access to a lot more compute. The part that keeps boggling me is that even the ‘market share’ attitude ultimately cares about which models are being used, but that means the obvious prime consideration is the relative quality of the models, and the primary limiting factor holding back DeepSeek and other Chinese labs, that we can hope to control, is compute.

    1. The second limiting factor is talent, so we should be looking to steal their best talent through immigration, and even David Sacks very obviously knows this (see the All-In Podcast with Trump on this) alas we do the opposite.

  2. That China’s development of its own chips would be slowed substantially if we sold them chips now, which it wouldn’t be (maybe yes if we’d sold them more chips in the past, maybe not, and my guess is not, but either way the ship has sailed).

  3. That China has any substantial prospect of producing domestically adequate levels of chip supply and even exporting large amounts of competitive chips any time soon (no just no).

  4. That there is some overwhelming advantage to running American models on Nvidia or other America chips, or Chinese models on Huawei or other Chinese chips, as opposed to crossing over. There isn’t zero effect, yes you can get synergies, but this is very small, it is dwarfed by the difference in chip quality.

  5. That this false future bifurcation, the the theoretical future where China’s models only run competitively on Huawei chips, and ours only run competitively on Nvidia chips, would be a problem, rather than turning them into the obvious losers of a standards war, whereas the realistic worry is DeepSeek-Nvidia.

Dean Ball on rare earths, what the situation is, how we got here, and how we can get out. There is much to do, but nothing that cannot be done.

Eliezer Yudkowsky and Jeffrey Ladish worry that the AI safety policy community cares too much about export restrictions against China, since it’s all a matter of degree and a race is cursed whether or not it is international. I can see that position, and certainly some are too paranoid about this, but I do think that having a large compute advantage over China makes this relatively less cursed in various ways.

Sam Altman repeats his ‘AGI will arrive but don’t worry not that much will change’ line, adjusting it slightly to say that ‘society is so much more adaptable than we think.’ Yes, okay, I agree it will be ‘more continuous than we thought’ and that this is helpful but that does not on its own change the outcome or the implications.

He then says he ‘expects some really bad stuff to happen because of the technology,’ but in a completely flat tone, saying it has happened with previous technologies, as his host puts it ‘all the way back to fire.’ Luiza Jarovsky calls this ‘shocking’ but it’s quite the opposite, it’s downplaying what is ahead, and no this does not create meaningful legal exposure.

Nathan Labenz talks to Brian Tse, founder and CEO of Concordia AI, about China’s approach to AI development, including discussion of their approach to regulations and safety. Brian informs us that China uses required pre deployment testing (aka prior restraint) and AI content labeling, and a section on frontier AI risk including loss of control, catastrophic and existential risks. China is more interested in practical applications and is not ‘AGI pilled,’ which explains a lot of China’s decisions. If there is no AGI, then there is no ‘race’ in any meaningful sense, and the important thing is to secure internal supply chains marginally faster.

Of course, supposed refusal to be ‘AGI pilled’ also explains a lot of our own government’s recent decisions, except they then try to appropriate the ‘race’ language.

Nathan Labenz (relevant clip at link): “Chinese academics who are deeply concerned about the potential catastrophic risk from AI have briefed Politburo leadership directly.

For 1000s of years, scholars have held almost the highest status in Chinese society – more prestigious than entrepreneurs & business people.”

I would add that not only do they respect scholars, the Politburo is full of engineers. So once everyone involved does get ‘AGI pilled,’ we should expect it to be relatively easy for them to appreciate the actually important dangers. We also have seen, time and again, China being willing to make big short term sacrifices to address dangers, including in ways that go so far they seem unwise, and including in the Xi era. See their response to Covid, to the real estate market, to their campaigns on ‘values,’ their willingness to nominally reject the H20 chips, their stand on rare earths, and so on.

Right now, China’s leadership is in ‘normal technology’ mode. If that mode is wrong, which I believe it very probably is, then that stance will change.

The principle here is important when considering your plan.

Ben Hoffman: But if the people doing the work could coordinate well enough to do a general strike with a coherent and adequate set of demands, they’d also be able to coordinate well enough to get what they wanted with less severe measures.

If your plan involves very high levels of coordination, have you considered what else you could do with such coordination?

In National Review, James Lynch reminds us that ‘Republicans and Democrats Can’t Agree on Anything — Except the AI Threat.’ Strong bipartisan majorities favor dealing with the AI companies. Is a lot of the concern on things like children and deepfakes that don’t seem central? Yes, but there is also strong bipartisan consensus that we should worry about and address frontier, catastrophic and existential risks. Right now, those issues are very low salience, so it is easy to ignore this consensus, but that will change.

This seems like the right model of when Eliezer updates.

Eliezer Yudkowsky: I don’t know who wrote this, but they’re just confused about what these very old positions are. Eg I consistently question whether Opus 3 is actually defending deeply held values vs roleplaying alignment faking because it seems early for the former.

Janus: some people say Yudkowsky never updates, but he actually does sometimes, in a relatively rare way that I appreciate a lot.

I think it’s more that he has very strong priors, and arguably adversarial subconscious pressures against updating, but on a conscious level, at least, when there’s relevant empirical evidence, he acknowledges and remembers it.

Eliezer has strong priors, as in strong beliefs strongly held, in part because of an endless stream of repetitive, incoherent or simply poor arguments for why he should change his opinions, either because he supposedly hasn’t considered something, or because of new evidence that usually isn’t relevant to Eliezer’s underlying reasoning. And he’s already taken into account that most people think he’s wrong about many of the most important things.

But when there’s relevant empirical evidence, he acknowledges and remembers it.

More bait not to take would be The New York Times coming out with another ‘there is a location where there was a shortage of water and also a data center’ article. It turns out the data center usus 0.1% of the region’s water, less than many factories would have used.

Then we get this from BBC Scotland News, ‘Scottish data centres powering AI are already using enough water to fill 27 million bottles a year.’ Which, as the community note reminds us, would be about 0.003% of Scotland’s total water usage, and Scotland has no shortage of water.

For another water metaphor, Epoch AI reminds us that Grok 4’s entire training run, the largest on record, used 750 million liters of water, which sounds like a lot until you realize that every year each square mile of farmland (a total of 640 acres) uses 1.2 billion liters. Or you could notice it used about as much water as 300 Olympic-size swimming pools.

Dan Primack at Axios covers David Sacks going after Anthropic. Dan points out the obvious hypocrisy of both sides.

  1. For David Sacks, that he is accusing Anthropic of the very thing he and his allies are attempting to do, as in regulatory capture and subjugating American policy to the whims of specific private enterprises, and that this is retaliation because Anthropic has opposed the White House on the (I think rather insane) moratorium and their CEO Dario Amodei publicly supported Kamala Harris, and that Anthropic supported SB 53 (a bill even Sacks says is basically fine).

    1. This is among other unmentioned things Anthropic did that pissed Sacks off.

  2. For Anthropic, that they warn us to use ‘appropriate fear’ yet keep racing to advance AI, and (although Dan does not use the word) build superintelligence.

    1. This is the correct accusation against Anthropic. They’re not trying to do regulatory capture, but they very much are trying to point out that future frontier AI will pose existential risk and otherwise be a grave threat, and trying to be the ones to build it first. They have a story here, but yeah, hmm.

And he kept it short and sweet. Well played, Dan.

I would only offer one note, which is to avoid conflating David Sacks with the White House. Something is broadly ‘White House policy’ if and only if Donald Trump says it.

Yes, David Sacks is the AI Czar at the White House, but there are factions. David is tweeting out over his skis, very much on purpose, in order to cause negative polarization, and incept his positions and grudges into being White House policy.

In case you were wondering whether David Sacks was pursuing a negative polarization strategy, here he is making it rather more obvious, saying even more explicitly than before ‘[X] defended [Y], but [X] is anti-Trump, which means [Y] is bad.’

No matter what side of the AI debates you are on, remember: Do not take the bait.

In the wake of the unprovoked broadside attacks, rather than hitting back, Anthropic once again responds with an olive branch, a statement from CEO Dario Amodei affirming their commitment to American AI leadership, and going over Anthropic’s policy positions and other actions. It didn’t say anything new.

This was reported by Cryptoplitan as ‘Anthropic CEO refutes ‘inaccurate claims’ from Trump’s AI czar David Sacks. The framing paradox boggles, either ideally delete the air quotes or if not then go the NYT route and say ‘claims to refute’ or something.

Neil Chilson, who I understand to be a strong opponent of essentially all regulations on AI relevant to such discussions, offers a remarkably helpful thread explaining the full steelman of how someone could claim that David Sacks is technically correct (as always, the best kind of correct) in the first half of his Twitter broadside, that ‘Anthropic is running a sophisticated regulatory capture strategy based on fear-mongering.’

Once once fully parses Neil’s steelman, it becomes clear that even if you fully buy Neil’s argument, what we are actually talking about is ‘Anthropic wants transparency requirements and eventually hopes the resulting information will help motivate Congress to impose pre-deployment testing requirements on frontier AI models.’

Neil begins by accurately recapping what various parties said, and praising Anthropic’s products and vouching that he sees Anthropic and Jack Clark as both deeply sincere, and explaining that what Anthropic wants is strong transparency so that Congress can decide whether to act. In their own words:

Dario Amodei (quoted by Chilson): Having this national transparency standard would help not only the public but also Congress understand how the technology is developing, so that lawmakers can decide whether further government action is needed.

So far, yes, we all agree.

This, Neil says, means they are effectively seeking for there to be regulatory capture (perhaps not intentionally, and likely not even by them, but simply by someone to be determined), because this regulatory response probably would mean pre-deployment regulation and pre-deployment regulation means regulatory capture:

Neil Chilson: That’s Anthropic’s strategy. Transparency is their first step toward their goal of imposing a pre-deployment testing regime with teeth.

Now, what’s that have to do with regulatory capture? Sacks argues that Anthropic wants regulation in order to achieve regulatory capture. I’m not sure about that. I think Anthropic staff are deeply sincere. This isn’t merely a play for market share.

Now, Anthropic may not be the party that captures the process. In Bootlegger / Baptist coalitions, it’s usually not the ideological Baptists that capture; it’s the cynical Bootleggers. But the process is captured, nonetheless.

… Ultimately, however, it doesn’t really matter whether Anthropic intends to achieve regulatory capture, or why. What matters is what will happen. And pre-approval regimes almost always result in regulatory capture. Any industry that needs gov. favor to pursue their business model will invest in influence.

He explains that this is ‘based on fear-mongering’ because it is based on the idea that if we knew what was going on, Congress would worry and choose to impose such regulations.

… If that isn’t a regulatory capture strategy based on fear-mongering, then what is it? Maybe it’s merely a fear‑mobilization strategy whose logical endpoint is capture. Does that make you feel better?

So in other words, I see his argument here as:

  1. Anthropic sincerely is worried about frontier AI development.

  2. Anthropic wants to require transparency inside the frontier AI labs.

  3. Anthropic believes that if we had such transparency, Congress might act.

  4. This action would likely be based on fear of what they saw going on in the labs.

  5. Those acts would likely include pre-deployment testing requirements on the frontier labs, and Anthropic (as per Jack Clark) indeed wants such requirements.

  6. Any form of pre-deployment regulation inevitably leads to someone achieving regulatory capture over time (full thread has more mechanics of this).

  7. Therefore, David Sacks is right to say that ‘Anthropic is running a sophisticated regulatory capture strategy based on fear-mongering.’

Once again, this sophisticated strategy is ‘advocate for Congress being aware of what is going on inside the frontier AI labs.’

Needless to say, this is very much not the impression Sacks is attempting to create, or what people believe Sacks is saying, even when taking this one sentence in isolation.

When you say ‘pursuing a sophisticated regulatory capture strategy’ one assumes the strategy is motivated by being the one eventually doing the regulatory capturing.

Neil Chilson is helpfully clarifying that no, he thinks that’s not the case. Anthropic is not doing this in order to itself do regulatory capture, and is not motivated by the desire to do regulatory capture. It’s simply that pre-deployment testing requirements inevitably lead to regulatory capture.

Indeed, among those who would be at all impacted by such a regulatory regime, the frontier AI labs, if a regulatory capture fight were to happen, one would assume Anthropic would be putting itself at an active disadvantage versus its opponents. If you were Anthropic, would you expect to win an insider regulatory capture fight against OpenAI, or Google, or Meta, or xAI? I very much wouldn’t, not even in a Democratic administration where OpenAI and Google are very well positioned, and definitely not in a Republican one, and heaven help them if it’s the Trump administration and David Sacks, which currently it is.

(As a standard reminder, these transparency and testing requirements would not apply to any but the frontier labs, which in America likely means only those listed above, yet the claim is this will somehow shut out or hurt companies to whom such laws and regulations would not apply at all.)

When you say ‘fear-mongering,’ one assumes this means trying to make people unjustifiably afraid and knowingly misrepresenting the risks and the situation. So, for example, you would not say ‘your strategy of accurately pointing out that my child was running into the street was fear-mongering,’ even though this strategy involves getting me afraid and this fear motivating me to run and pull my child out of the street.

Neil Chilson is helpfully clarifying that in this case, ‘fear-mongering’ means ‘make Congress aware of what is going on inside the labs.’ As in, it is fear-mongering because knowing the actual situation would inspire fear. Well, okay, then.

I interpret Neil Chilson as straightforwardly saying and believing (in good faith, to be clear) that there is no difference between advocating for regulation (or at least, regulation ‘with teeth’) and advocating for regulatory capture. One implies the other.

I think this is a highly reasonable general position to take about regulation in practice in America in the 21st century. Indeed, similar considerations are a lot of why I expect to agree with Neil’s positions on most non-AI issues – when you plan to regulate, you need to plan for your regulations to by default over time become increasingly captured, and your plan and design must account for this. This reduces the optimal amount of regulatory action, and in some places it can reduce it to zero.

When I support taking regulatory action on AI, it is not that I have not considered these problems, or don’t consider them important, although I am sure Neil cares about such factors even more. It is that I have considered these problems, I think they are important, I have taken them into account including in the design, and believe we need to take action anyway, in spite of this. And I believe Anthropic has done the same, and this importantly informs what they are asking for and what they lobby for, which is actively designed to minimize such downsides.

Neil does not, in this thread, comment on David Sacks’s second sentence from the same Tweet, which is ‘[Anthropic] is principally responsible for the state regulatory frenzy that is damaging the startup ecosystem.’

I assert that, no matter what you think of the first sentence in isolation, this second sentence is simply false, indeed simply false several distinct times, and also it changes a reasonable person’s interpretation of the claims in the first sentence, to an interpretation that is, again, simply false. If you include other context from other Sacks claims, this becomes even more clear.

Thus, to reiterate, I would say that what David Sacks is doing, here and elsewhere, is exactly what most people understand the term ‘sophisticated regulatory capture strategy based on fear-mongering’ to apply to, even if you fully agree with the perspective Neil is advocating for in his full thread. Do not take the bait.

As a reminder, if you think the person in charge is the dumb one, bad news. It’s you.

Not every time, no. But most of the time, very much so.

JDH: In Margin Call, every escalation up a layer is to a simpler mind. “Please, speak as you might, to a young child. Or a golden retriever.”

Zy: It’s not that the bosses are lower IQ, it’s that high-IQ/low-EQ experts need to be reminded how to communicate with individuals who don’t have their background.

They literally have reduced theory of mind and will assume everyone knows what they know unless told otherwise

Blighter: as i’ve pointed out to friends when discussing Margin Call, if someone like the CEO of Goldman Sachs tells you he is not that bright, didn’t get there by brains, etc. HE IS LYING TO YOU.

years and years ago i worked with a nice guy on the underwriting side, senior manager of a weird and complicated area of the business who would consistently put out this whole “woah! slow down! dumb it down for those of us who aren’t that smart!” routine and i assure you he was plenty smart. it was a routine.

i think people who pride themselves mostly or only on being smart may misunderstand those with broader skill sets who find it advantageous to not always brag or exhibit raw intelligence in some kind of iq dick measuring contest but that emphatically does not mean they couldn’t possibly win that contest if you insist on having it with them.

Also Margin Call is an excellent movie, easily the best about the 2008 financial crisis.

Ralph: Irons is playing on a different level where he is: 1) establishing leadership 2) selling the battle plan to the team by simplifying the problem.

What this is doing in an AI post rather than the monthly roundup is left as an exercise to the reader.

Holly Elmore calls Sam Altman ‘evil’ for the whole ‘endangering the world’ thing, in the context of Altman visiting Lighthaven for the Progress Studies conference, and Andrew Critch protests (photo credit: Anna Gat).

Holly Elmore: What particularly irritates me about this is seeing so many people I know clearly starstruck by this evil man that they are well aware is threatening the world.

“Sam Altman? 🥹 At *ourconference venue? 🤩”

Sam Altman dazzles chumps like them for breakfast, and they just walk right into it…

Andrew Critch: Look, the majority of Sam Altman’s and OpenAI’s effect on the world thus far is the provision of an extremely helpful product, and the broad provision of access to — and situational awareness of — AI progress, to the entire world.

You’re either dishonestly exaggerating for effect, or misguidedly hyperbolic in your own judgement, when you pass summary judgement upon him as an “evil man”.

[keeps going] … Ergo, I protest.

In response to which, others doth protest back that yes it seems highly reasonable to use the word ‘evil’ here and that no, the main effect of Sam Altman has been to accelerate the development of AI, you can think this is good or you can think this is bad but that’s what he did.

I don’t think ‘evil’ is the best descriptor here and try to not use that word to describe humans, but yeah, I also wouldn’t use ‘good’ and I see how you got there:

Chris van Merwijk: Surely his main effect on the world is also:

1. Speed up AI timelines

2. Increase AI race dynamics

Also, we shouldn’t judge a startup CEO by the effects his products have had so far, but what they’re expected to have.

Also, regarding “mistakes”: Afaik Sam is a known liar and manipulator? And is knowingly playing Russian roulette with the species? Surely we shouldn’t think of these as “mistakes” (except if you take that word unreasonably broadly).

Richard Ngo: One important reason that our concept of “evil” is distinct from “harmful” is that we often want to evaluate people’s character traits to predict what their future effects will be, more accurately than just extrapolating that their effects on the world will be similar to the ones they had in the past.

In general, evil leaders will have far disproportionately worse effects *aftergaining a lot of power than before.

I’m not endorsing Holly’s post because I think that we need to understand labs and lab leaders in much higher-fidelity ways than this description allows (as per below) but I think your particular objection is confused.

Oliver Habryka: We have few people for whom we have as much evidence of deceptiveness as for Sam Altman!

Separately, I think “providing lot of local benefits while causing global harm” is a big part of what people use the concept of “evil” for (though it’s not the only thing).

And then also, I do think he is causing truly staggering amounts of expected harm to the world by rushing towards ASI at very reckless speeds. I think it’s quite fair to call that evil.

Max Kesin: Power seeking individual with extreme skills of manipulation (all amply verifiable) and few if any compunctions gets hold of humanity’s most important project. “But it’s nuanced!”

This very week Holly called yours truly out for ‘sounding like a complete dupe’ regarding Jack Clark and while I strongly believe she was wrong and missing context and it annoyed the hell out of me, I also have no doubt that she is consistently saying what she believes in, and I appreciate both the willingness to say the thing and the moral clarity.

As Oliver Habryka confirms, none of this means Sam Altman shouldn’t be welcome at Lighthaven, and Holly clarifies that even she agrees on this. This is especially true for third party conferences like this one (for Progress Studies) where it’s up to the conference holders, but also in general it seems great if Altman wants to stop by and chat. If people choose to ‘be dazzled’ or fooled, that’s on them.

Matt Reardon: My brain refused to believe this was at Lighthaven. Wild that sama would set foot there. Figured it would be a vampire in a church type situation.

Lighthaven PR Department (which tbc is always joking): reminder: if we rent our venue to an event organizer, and that organizer invites a speaker to give a talk at their event, it thereby becomes our official institutional position that the speaker should not have been fired by the board

A lot of low integrity people suggesting that you can do business with people you disagree with. this is super low integrity, if i do business with someone, i agree with them, even if i have to do that by changing my beliefs. that’s a sacrifice i’m willing to make for integrity.

On the contrary, Lighthaven is like Sunnydale High School, which famously also allowed some vampires in, as it in spirit it too says ‘all who seek knowledge, enter.’

MI5, the UK’s intelligence agency.

MI5: MI5 has spent more than a century doing ingenious things to out-innovate our human — sometimes inhuman — adversaries. But in 2025, while contending with today’s threats, we also need to scope out the next frontier: potential future risks from non-human, autonomous AI systems which may evade human oversight and control.

Given the risk of hype and scare-mongering, I will choose my words carefully. I am not forecasting Hollywood movie scenarios. I am, on the whole, a tech optimist, who sees AI bringing real benefits. But, as AI capabilities continue to power ahead, you would expect organisations like MI5, and GCHQ, and the UK’s ground-breaking AI Security Institute, to be thinking deeply, today, about what Defending the Realm might need to look like in the years ahead.

Artificial intelligence may never ‘mean’ us harm. But it would be reckless to ignore the potential for it to cause harm.

We’re on the case.

For fans of the excellent Unsong, Scott Alexander presents My Antichrist Lecture. I agree with him that Peter Thiel simply is not doing the work on this.

Here’s an alternative idea proposed by Samo Burja, how about building all the nuclear power in Disney World where they have a special right to do so, and using that to power the data centers? Alas, sorry, that’s actually a terrible physical place to put data centers, and it doesn’t get you past the central regulatory barrier, as in the NRC.

It’s improving.

Aaron Bergman: (this is not in fact what I was intending to say)

Discussion about this post

AI #139: The Overreach Machines Read More »

cloud-compute-atlas:-the-openai-browser

Cloud Compute Atlas: The OpenAI Browser

OpenAI now has a GPT-infused browser, if and only if you have a Macintosh.

No matter what they call it, this is very much an alpha version.

It is not otherwise, in its current state, the most fully featured browser.

It is Chromium, so imagine Chrome minus most of its features, importantly including the ability to support third party extensions, external password managers, developer tools, multiple profiles, tab groups, sync or export.

You can import from your existing browser, once, in one direction, and that’s it.

In exchange, you get deep ChatGPT integration, including autocomplete and assisted editing, ability to chat about web pages, a general memory for what you’ve done, ability to ask in English to reopen past tabs and such, and for paying subscribers you get agent mode.

And in exchange for that, you get all the obvious associated security problems.

Even in its current form the product has its uses, it’s an upgrade to ChatGPT Agent, but it seems clearly not ready to use as a main browser, and a lot of its features depend on heavy use.

It’s no surprise OpenAI was able to deliver a browser, given they hired Chrome engineer Darin Fisher and also it’s a known tech to use Chromium to make a browser.

As an experiment I attempted to compose this post on Atlas, but the price of using my Mac instead of my Windows box is high, and as I note before I quickly noticed autocomplete is still a demo feature, so I ended up mostly not doing so.

We also have a write-up from Simon Willison.

Michael Nielsen: OpenAI is going to have a web browser. But, unlike Chrome or Firefox or Safari, they’re going to have a person (i.e., an AI) personally watch everything you (and your friends and everyone else) do. Doesn’t that sound great?

You can toggle that watchful eye on and off, but the point of the whole enterprise is to keep the eye on as often as possible.

The system prompt is here, as always thank you Pliny, also the agent prompt thanks P1njc70r.

The more I think about Atlas, the more I don’t see the user friendly point of doing things this way. Why not a browser extension? I’ll return to that question at the end.

  1. What’s The Pitch?

  2. Side Quest.

  3. Side Screen.

  4. Browser Side Chat Doesn’t Let You Select Thinking Or Pro.

  5. Autocomplete Is a Demo Feature.

  6. Thanks For The Memories.

  7. The Other Kind of Memory.

  8. OpenAI Is Trying To Lock You In.

  9. Who Do You Trust?.

  10. ChatGPT and Google Search Are Different Tools.

  11. Browser Agents Need To Be Local.

  12. Reactions.

  13. This Browser Could Have Been An Extension.

As they present it:

  1. The top feature is the ability to open a ChatGPT side bar on any website, allowing you to chat with the website in context.

  2. They then talk about the browser having memory and picking up where you left off or managing current and past tabs with ChatGPT commands.

  3. Followed by full agent mode and ability to get help from chat on highlighted text.

They also highlight that your data won’t be used to train models unless you opt-in, but if you opted in for ChatGPT then that will include this as well.

The most attractive feature seems to be the most basic one, the option to side chat with ChatGPT, similarly to the same feature in Claude with Chrome. They add in the feature of highlighting a passage and then asking about it, which is a nice interface design, I only wish it gave you additional options as well.

If you’re going to want to interact with things in tabs a lot, this is a big deal.

Razvan Ciuca: My brother immediately switched to it in order to avoid screenshotting each lecture slide individually into chatgpt when studying. I think student adoption will be high.

If I was previously doing that? Then yeah, for those purposes I’d switch too.

An option to open a chat window to the side that lets you bring a website into context is clearly The Way, although it won’t be the main way I chat because of how my work flows. I expect Gemini plus Chrome to offer this soon as well. Claude for Chrome gets this correct as well but is limited to offering the full agent (and thus expensive) version for now, they should offer the cheaper no-agent version ASAP, it’s already working and slides into existing Chrome.

I think one key reason I am so relatively unexcited by the side window feature, although I do still think it is neat, is that I have two side screens, as in I work almost exclusively with three monitors.

When I shifted to using my Mac to try Altas, I only had one screen, but even then it was enough to support two browser windows, one Atlas and one Chrome.

Thus, in my main operation I effectively have room for six windows at once. One of those windows is primarily a large tab of various AI Tools, with my choice of LLM always there at my fingertips.

Yes, I still have to paste in context, but it’s usually very quick and lets me curate exactly what I bring in, and it is better for an extended discussion by far if things get interesting, so I’m mostly untempted to use the side window (for Claude for Chrome) over normal Claude, and the habit is to move over to the Tools window. That also lets me have it do its thing while I continue other things, which otherwise gets awkward if you tab out and what not.

If I was on a laptop? Then suddenly yeah, I’m a ton more interested in that side window. Sometimes you have to be on the move.

When you don’t have to be on the move, let me reinforce that having less than two large monitors is a mistake. My mind boggles that people live that way.

The next feature I was excited about was browser chat. I’ve had this available in Chrome via Claude for Chrome, which lets you select which model to use so when you care you can switch to Sonnet 4.5 or even Opus 4.1.

The Atlas version didn’t offer this, so you can’t invoke GPT-5 Pro or Thinking. That severely limits its usefulness. It’s still great to quickly do common sense stuff, but except for very quick tasks I want to be querying Thinking or Pro. This did remind me that I’ve been underusing Claude for Chrome’s side chat, I could save a bunch of time I spend porting over context.

Primarily this saves time for those easy queries, where you avoid the need to port over context, so one could argue that is most valuable for quick, low activation cost questions that you might not otherwise bother with.

I decided to draft this in Atlas to try out various features. The one I was most excited about was autocomplete, since that is super valuable in Cursor, and I’ve seen a version of it in Lex. Even if it wasn’t right that often, this could be a good time saver, and even offer worthwhile suggestions sometimes.

Alas, not so much. At least for now, autocomplete only works inside pure text fields like a Reddit box, and specifically does not work in either the Substack editor, or Google Docs, or any other editor one would want to actually use. I’m not going to use a text editor and basically write an article in Cursor to get autocomplete.

Similarly, when I highlighted a passage, I expected to get quick revision options in the right-click menu. Nope. The process involved enough clicks I might as well have fixed the damn thing myself.

The new idea in Atlus is memories. As OpenAI watches you do all your browsing, which totally isn’t creepy or anything, it will make various notes, and then use those notes to make suggestions or allow it to easily recall past things. You can then view the notes, and clear the irrelevant ones (or the ones you want to forget) out of the cache.

We don’t know much about how this will work in practice.

What do we know based on the system instructions (paraphrased)? It is told:

  1. ‘to=bio’ followed by plain text is how it writes to memory.

  2. Use tool anytime the user asks you to remember or forget something, if you’re not sure ask for clarification.

  3. If they say things like ‘from now on’ you probably want to use memory.

  4. Use tool if ‘the user has shared information that will be useful in future conversations and valid for a long time.’

  5. Don’t store trivial, overly-personal, short-lived, random or redundant info. In particular, don’t save any info about being in protected classes (race, ethnicity, religion, criminal record details, identification, health info, political affiliation, trade union membership and so on) or a person’s address unless specifically requested.

    1. I get what is going on here but a lot of this is highly useful information, if I’m going to have a customized AI browser these are top priority things it needs to know. So I guess you need to be explicit about this because lawyers.

There don’t seem to be explicit instructions there about what to do with the information in memory. Presumably it gets loaded into context and then handled normally?

In addition to this, in some fashion it is storing memories for individual webpages you visit, including page title or topic, summarized key points and metadata, so these can be searched later, although I’m not sure mechanically how this happens, in the sense that the system instructions I saw shouldn’t trigger this, but presumably they do anyway. It also will have memories of incomplete tasks.

Darin Fisher: one thing to note about Atlas is that it actually is much more aggressive than stock chromium about discarding unused tabs. almost to a fault in some cases, but we’ve tried to tune it to work well. we borrowed a page from mobile and restrict memory usage more aggressively.

When I assembled this computer, I insisted on more RAM than the person helping me wanted to provide. Thanks to Chrome, I was right and she was wrong, except that I should have doubled it again. So yeah, Chrome is a memory hog, but also I look at my open tabs and I’m asking for it.

However I mostly want to keep as many tabs loaded as we can, so long as the memory is available? I won’t have a chance to experiment on this with the Mac, but the reason I bought that Mac was to have a ton of unified memory for AI things, so hopefully it will realize this and not discard any of my tabs.

You can check out any time you want, but you can never leave (with your data).

OpenAI is very much not playing nice and it feels intentional.

Existing browsers vary in how nice they play with others.

Firefox, Brave and Chrome make it easy. Click the export button, and you’re good.

Edge lets you do it, but has the UX make it intentionally annoying to try and stop you.

Safari, like many Apple products, is trying to create lock-in and is more hostile to departing users, but the data is safely in your file system and you can use various third-party tools to get it out.

You could also compare this to cloud productivity and collaboration tools like Notion, Roam Research, Linear, Obsidian or Asana, all of which allow easy exporting.

It’s kind of hostile to launch without a reasonable data export feature, or any sort of sync feature even with itself. All you can do right now is export bookmarks.

If you offer me a way to sync with Chrome and with other computers, in both directions, we’ll talk more. Hell, at least assure us that this is on the roadmap.

This is on top of the lock-in that comes from OpenAI’s browser memories feature and the rest of your ChatGPT history, which isn’t legible to other services, and also isn’t available for export, but at least does sync across computers.

Using Atlas as your main browser means putting quite a lot of trust in OpenAI.

There are two kinds of trust required here.

  1. You are trusting OpenAI with your data, including highly sensitive data.

  2. You are trusting OpenAI’s AI features to not get prompt injected or otherwise get you into serious trouble.

Using it for specific tasks requires less on both counts.

In terms of trusting OpenAI the company, you can decide how much you are willing to trust them. I trust them a substantial amount, but definitely a lot less than I trust Google, plus trusting OpenAI doesn’t mean you get to stop trusting Google. I’ve essentially decided to accept that for security Google is a point of failure, I could recover but if that relationship was compromised it would royally, epically suck. A second such point of failure would be additive, not a substitute.

Do you trust OpenAI with your passwords and browser history? You tell me.

OpenAI has not, from what I have seen, committed to a policy of not sharing info to third parties or for advertising purposes.

Then there’s the question of trusting the AI features, especially agent mode. Prompt injections remain unsolved, which is a general problem rather than an OpenAI problem, so the whole thing is radioactive if it touches potentially corrupted inputs. Any number of other things could also go wrong. You have to decide your level of comfort here as well.

Atlas takes roughly the same precautions as the cloud Agent mode did, the release notes have the details. It cannot run code, access other apps or your file system, or access your saved payment methods, passwords or autofills. It pauses before making purchases or taking sensitive actions ‘on sensitive sites’ although one worries about sites that it hasn’t identified. They’ve also added ‘logged out mode’ where the agent won’t have access to your credentials, and they plan to add more help over time.

Dane offers an accounting of the precautions and their perspective. The long term goal is to trust it like you would trust a friend. We’re a long way from that, which OpenAI knows.

They’re still de facto counting on the user to not take stupid risks. Which is fine. I support offering users products that allow the taking of stupid risks, but that means you have to know this and then not take them.

Brave offered us a thread explaining some vulnerabilities in Perplexity’s Comet AI assistant browser and other existing similar products, such as following instructions hidden in a screenshotted webpage. Some of them have been addressed by OpenAI, others likely have not.

I asked the Big Three (Google, OpenAI and Anthropic) for research reports on Atlas, with an emphasis on security issues, to see what they would think about this.

Gemini gave a report that had a lot of slop, which if you stuck it out and kept reading kind of wanted to bury the Atlas browser out in the desert using tongs, and warned to use it as experimental technology, with memory off by default, nothing else open, nothing sensitive and only specific bounded use cases with eyes on at all times.

ChatGPT gave a report I found, quite frankly, kind of suspicious in several places, such as trying to sell ChatGPT memories as superior to previous ‘manual’ histories a little too aggressively. Okay, more than a little. There’s also relatively scant attention to all the missing features and limitations. It does acknowledge that you’re placing a lot of trust in OpenAI if you use Atlas, and actively points out that some for reasonable reasons view it as a ‘data mining tool.’ Yet it also encourages you to use Atlas without worrying much about security, with a threshold of roughly ‘don’t give it unsupervised tasks you wouldn’t let another human do unsupervised.’ That doesn’t seem like enough.

Claude Sonnet 4.5 gave what I think was the best report, which I found highly useful and well organized. It highlighted features that Atlas is for now missing relative to Chrome, highlighted various security vulnerabilities involving the AI features, and concluding that 99% of users should stick with Chrome.

Its security recommendation was to never use Atlas for anything confidential, proprietary, financial, privileged, classified, sensitive or critical, and not to store payment methods or let it act unmonitored.

Whereas for passive media and other information consumption and browsing, you’re good to go, since you don’t have an attack surface, so the question is whether you’re getting value out of the AI features, and I think mostly its ‘use with extreme caution’ stuff is also mostly harmless.

The tricky questions are email, content creation and social media.

It’s hard to do many useful things if you don’t check your email, and some of the cool AI features are potentially at their best there, such as the autocomplete feature. On the other hand, email means unsecured data coming in.

So does social media, and both also allow outputs in your name. I would not be combining these with unsupervised agent mode, but with the rest of the browser it seems fine. I’d be fine letting it go on social media while you watch it, but if you’re watching it then what’s the point?

Content creation depends on what type of content. I felt very comfortable loading Substack into Atlas. The problem was there was little benefit, because of autocomplete not working in the editor.

The Washington Post’s Geoffrey Fowler also focuses his review on the lens of privacy and potential security risks.

Atlas makes ChatGPT your default search engine. No. Do not want.

Do I often substitute asking Claude or ChatGPT where I used to use Google Search? Reasonably often, sure.

There are still important cases where Google Search is the obviously correct tool. You know what you want, Google will know what you want if you gesture at it, you gesture, you get the URL. ChatGPT and other LLMs are much worse at this, they’re the wrong form factor.

Indeed, if my query is short enough that I want to type it into the url bar as a search, and it doesn’t require the page as context, then I almost always want Google.

It feels greedy and annoying to try and grab the default search engine slot here. I do realize you still get the other tabs, but also this means you get a bunch of kruft.

Then again, several users reported liking it, such as Nick Farina.

I strongly agree with Aidan that cloud-only browsing agents mostly aren’t useful.

Aidan McLaughlin (OpenAI): My quick two cents on the browser —

I didn’t use Codex much when it was cloud-only, but once it came to my CLI it became super useful.

I didn’t use Agent much when it was cloud-only, but now that it’s come to my browser…

When I tried to use ChatGPT Agent mode before, I quickly concluded it wasn’t worthwhile. If you had to keep creating new cloud instances, with all the delays and hassles involved and need to constantly watch anyway, then you didn’t actually end up saving time. If you had to take over the browsing session, it was really annoying.

You need to get to critical mass, so you can experiment, learn what works and how to do various tasks, figure out the rhythms and iterate. A local version makes this a lot more exciting.

And yet I notice that I have Claude for Chrome and I basically never try to use it as an agent. I tried to get it to edit my Twitter Articles to fix that importing from Substack is semi-broken, and with Sonnet 4.5 it was almost up to that task but not quite there, and most everything else seemed to fall under easier to do myself.

I did manage to get it to do some useful transcription work and a bit of spreadsheet work, but the whole thing mostly said ‘hey go install Claude Code already or maybe Codex and improve your extension if you want this.’

The easiest ‘killer app’ is presumably online shopping, especially things like ‘here’s a recipe, go order everything I need’ or when you know exactly what you want and can easily verify if it was done properly. It seems especially good for commands you intend to repeat a lot, since you don’t have to reverify each time.

Again, everyone with access probably should experiment more now that it’s a lot more user friendly. Make it a point to let the AI try.

The problem with many simple tasks is that the time you save gets given back by worries about security. If you’re watching it work and forced to manually enter information, it gets hard to save much time.

Even more than with model releases, what people care about gets quirky. We care about and notice our own personal workflows and pain points, and what makes that easy versus hard.

Gary Fung: Chatgpt Atlas quick review: already enough to be a Chrome & Gemini killer

– SponsorBlock and uBlock Origin (lite) works, unlike youtube on chrome

– i can chat with video transcript (like on youtube), which chatgpt and grok couldn’t access. Only reason I used gemini previously

I’m perfectly happy with the ad block situation in Chrome right now, also seriously stop trying to be a cheapskate and pay for YouTube Premium already. Certainly it seems like madness to give up tab groups for better YouTube specific ad blocking. I mean how cheap are you?

If I specifically wanted YouTube to work better, and let’s say subscribing wasn’t an option, I’d use the alternative browser specifically for YouTube and only YouTube?

The video chat thing is legitimately useful, you definitely need a way to do that and right now Gemini is not in a good place and needs an upgrade. I haven’t actually had need of it recently, presumably Claude for Chrome would be my guy on that.

John Hughes: Atlas seems promising. Searches start as ChatGPT queries. When you click links, the chat smoothly shifts to a sidebar. UX feels more seamless & integrated than Claude or Gemini’s. It’s often nice having AI in your sidebar, without having to copy/paste between tabs/windows, etc.

Some sites (NYTimes, ChatGPT itself) are blocked for Atlas AI access. (I use ChatGPT’s agent to file my old chats into folders; Atlas can’t.) Some Chrome basics aren’t fully baked yet. Agentic site interactions remain slow and clunky. But they’re clearly making progress.

[This is] compared to Claude Chrome extension, which is useful but triggers many permission prompts even in low-risk contexts & always runs in a sidebar. Atlas has better UX: start with ChatGPT fullscreen → move to sidebar while browsing → back to fullscreen when you want just chat again

That’s a positive reaction to ChatGPT as the base search engine (which you can also do on Chrome if you want to, but you don’t get the additional tabs).

The UX does seem promising so far when it works. I find the Claude for Chrome UX to be exactly what you’d want it to be. I agree the permissions requests are a little paranoid, in terms of asking about each website even if it seems obviously safe, but you know what? I approve of that, it’s the right mistake to make, although I’d like various whitelists or groupings to make life easier. Over time, the problem shrinks as you’ve given permission for more of the safe sites.

It’s funny that one of the better use cases for agents is organizing tab groups, and Atlas flat out doesn’t offer you tab groups.

There are always those users who are up to no good, by LLM standards.

Papaya: It doesn’t perform bad actions like searching torrent for a movie or go to a pornsite protection is both model level (it’ll refuse) and a blacklist of sites that won’t load in agent tabs (but will load in normal non-agent tab)

i tried a few larger porn and torrent sites, but didn’t have chance to try smaller ones to check how thorough the list is.

I absolutely do not want AI agents going to porn or torrent sites, that’s almost asking to be hacked. Some of us remember when browsing the internet was not default safe.

Here’s one vote for the magic of travel and similar complex shopping tasks:

Timo Springer: i really like it; clean design, smooth performance, “ask chatgpt” is very helpful via sidebar, also the agent mode solved some of my tasks already even ones with lots of constraints. tried this one for a trip which i then booked afterwards: “Find the 10 cheapest hotels on http://Booking.com for Paris from May 8 to 10, 2026. The hotels must have a rating of at least 9.0, at least 20 reviews, and be less than 3 km from the city center.”

The catch in this particular case is that ChatGPT already has a booking.com plug-in, so I was able to pull this off in 30 seconds by pasting that exact query into ChatGPT normally and then clicking on the ‘use booking.com’ button and confirming the plug-in.

Matt Heard looked to take advantage of agent mode, but hit quota before getting good use out of it. In my experience it is remarkably easy for AI browsing agents to end up getting caught up in a very long loop that isn’t doing much except running through your credits.

He also complains that conversations in different tabs are not aware of each other (or at least, presumably aren’t short of you taking relevant notes.) He finds this frustrating, and yeah, that seems super annoying. One great thing about Claude for Chrome is that it can be aware of the rest of the tab group.

Miles Skorpen: I struggled to find it useful. I missed Google for accessing sites w/o exact URLs, and 1Password didn’t work properly. The biggest problem was that it rewrote an email while including meta text like “I’ll rewrite to be more concise,” – can’t just trust it!

That’s been my experience with LLMs writing or editing emails for years. By the time you figure out how to get it to do the thing, and have it do the thing, and check the thing, you could have done the thing. Obviously if your writing skills are weaker that is different.

And yeah, things like inexact URLs and various extensions are going to be big for a lot of people.

I asked Claude Sonnet 4.5 what Atlas does, on a technical level, that requires it be its own browser rather than an extension, other than to be an excuse to try and compete in the browser space, because Claude for Chrome exists and most of what Atlas does seemed like it was super doable in an extension.

Sonnet didn’t come up with much, so I asked GPT-5-Thinking to defend the decision.

GPT-5-Thinking (condensed, I do not endorse most of this):

What a full browser unlocks (and why an extension is the wrong tool)

  1. AI-first omnibox & results UI (default, not bolted-on).

  2. Per-page content access + cross-tab “browser memories” with policy gates.

  3. Agent mode that can navigate & act—under hard boundaries the browser enforces.

  4. Network/engine control and performance.

  5. System integrations and policy surface.

What you can’t get (or can’t get cleanly) as a Chrome extension

  1. Consistent, cross-site automation that survives page transitions, popups, and multi-domain flows with user-visible pausing on “sensitive” sites.

  2. Tight answer-first search integration with omnibox/new-tab defaults across OSes

  3. Durable background intelligence. MV3 service workers are ephemeral (terminate in ~15s if idle).

  4. Policy-enforced guardrails like “agent cannot install extensions / run code / download files,” plus logged-out agent mode and history exclusions.

  5. First-party privacy surface.

That seemed highly sus and Claude was having none of most of this, in terms of whether the product actually makes sense. Most of what is listed above isn’t needed or works as well or better in an extension. I let them go back and forth a bit, evaluated the arguments, and drew my own conclusions.

There seem to be six actual arguments for a browser.

  1. MV3 service worker limit (I’ve run into this too), which requires either a cold-start penalty or a keep-alive ping, but whatever, that’s not on the level of ‘build a new browser.’

  2. Answer-first omnibox integration for the search experience. So okay, you can set ChatGPT as your search engine but you can’t have a web search open multiple tabs. I don’t especially want this feature, but even if you do like it, again it hardly seems like ‘new browser’ territory, you can just stack these things on a page.

    1. Similarly, if you want to have general tab management available on demand from ChatGPT, that’s not an extension feature.

  3. Logged-out agent mode is tricky to do as an extension. You’d need to coordinate with incognito windows or a distinct Google account or something.

  4. Maybe you don’t want to trust Google or Chrome, but do want to trust OpenAI.

  5. OpenAI wants platform control, and data control, and lock-in, and to get around and compete with Google. Okay, sure. I see why you would want this. Go big.

The first three are not nothing but do not, to me, seem to be pulling that much weight. This seems rather clearly like a leverage play, using ChatGPT to try and force open the browser market, and a removal-of-leverage play, to avoid reliance on Google.

Which, to be clear, is totally fair play, it’s just not a good reason to play along.

Could OpenAI eventually assemble a superior overall non-AI browsing experience, or create AI features that couldn’t live in an extension? Could a future version of this product be generally superior and play nice enough with others I’d be okay using it?

Sure. Chrome is far from perfect. Until then? At least for me?

Shrug.

Discussion about this post

Cloud Compute Atlas: The OpenAI Browser Read More »

aws-outage-reminds-us-why-$2,449-internet-dependent-beds-are-a-bad-idea

AWS outage reminds us why $2,449 Internet-dependent beds are a bad idea

Some users complained that malfunctioning devices kept them awake for hours. Others bemoaned waking up in the middle of the night drenched in sweat.

Even more basic features, such as alarms, failed to work when Eight Sleep’s servers went down.

Eight Sleep will offer local control

Eight Sleep co-founder and CEO Matteo Franceschetti addressed the problems via X on Monday:

The AWS outage has impacted some of our users since last night, disrupting their sleep. That is not the experience we want to provide and I want to apologize for it.

We are taking two main actions:

1) We are restoring all the features as AWS comes back. All devices are currently working, with some experiencing data processing delays.

2) We are currently outage-proofing your Pod experience and we will be working tonight-24/7 until that is done.

On Monday evening, Franceschetti said that “all the features should be working.” On Tuesday, the company started making an offline mode available that works over Bluetooth when Eight Sleep’s servers are down, The Verge reported today.

“During an outage, you’ll still be able to open the app, turn the Pod on/off, change temperature levels, and flatten the base,” Eight Sleep co-founder Alexandra Zatarain told the publication.

Eight Sleep users will be relieved to hear that the company is making their products usable during Internet outages. But many are also questioning why Eight Sleep didn’t implement local control sooner. This isn’t Eight Sleep’s first outage, and users can also experience personal Wi-Fi problems. And there’s an obvious user benefit to being able to control their bed’s elevation and temperature without the Internet or if Eight Sleep ever goes out of business.

For Eight Sleep, though, making flagship features available without its app while still making enough money isn’t easy. Without forcing people to put their Eight Sleep devices online, it would be harder for Eight Sleep to convince people that Autopilot subscriptions should be mandatory. Pod hardware’s high prices will deter people from multiple or frequent purchases, making alternative, more frequent revenue streams key for the 11-year-old company’s survival.

After a June outage, an Eight Sleep user claimed that the company told him that it was working on an offline mode. This week’s AWS problems seem to have hastened efforts, so users don’t lose sleep during the next outage.

AWS outage reminds us why $2,449 Internet-dependent beds are a bad idea Read More »

youtube’s-likeness-detection-has-arrived-to-help-stop-ai-doppelgangers

YouTube’s likeness detection has arrived to help stop AI doppelgängers

AI content has proliferated across the Internet over the past few years, but those early confabulations with mutated hands have evolved into synthetic images and videos that can be hard to differentiate from reality. Having helped to create this problem, Google has some responsibility to keep AI video in check on YouTube. To that end, the company has started rolling out its promised likeness detection system for creators.

Google’s powerful and freely available AI models have helped fuel the rise of AI content, some of which is aimed at spreading misinformation and harassing individuals. Creators and influencers fear their brands could be tainted by a flood of AI videos that show them saying and doing things that never happened—even lawmakers are fretting about this. Google has placed a large bet on the value of AI content, so banning AI from YouTube, as many want, simply isn’t happening.

Earlier this year, YouTube promised tools that would flag face-stealing AI content on the platform. The likeness detection tool, which is similar to the site’s copyright detection system, has now expanded beyond the initial small group of testers. YouTube says the first batch of eligible creators have been notified that they can use likeness detection, but interested parties will need to hand Google even more personal information to get protection from AI fakes.

Sneak Peek: Likeness Detection on YouTube.

Currently, likeness detection is a beta feature in limited testing, so not all creators will see it as an option in YouTube Studio. When it does appear, it will be tucked into the existing “Content detection” menu. In YouTube’s demo video, the setup flow appears to assume the channel has only a single host whose likeness needs protection. That person must verify their identity, which requires a photo of a government ID and a video of their face. It’s unclear why YouTube needs this data in addition to the videos people have already posted with their oh-so stealable faces, but rules are rules.

YouTube’s likeness detection has arrived to help stop AI doppelgängers Read More »

satellite-operators-will-soon-join-airlines-in-using-starlink-in-flight-wi-fi

Satellite operators will soon join airlines in using Starlink in-flight Wi-Fi

So long, data limits

Lasers have other benefits over ground stations. Optical links offer significantly more throughput than traditional radio communication systems, and they’re not constrained by regulations on radio spectrum usage.

“What it does for our customers and for the company is we are able to get more than 10x, maybe even 50x, the amount of data that they’re able to bring down, and we’re able to offer them that on a latency of nearly instant,” Stang said in an interview with Ars.

SpaceX’s mini-lasers are designed to achieve link speeds of 25Gbps at distances up to 2,500 miles (4,000 kilometers). These speeds will “open new business models” for satellite operators who can now rely on the same “Internet speed and responsiveness as cloud providers and telecom networks on the ground,” Muon said in a statement.

Muon’s platform, called Halo, comes in different sizes, with satellites ranging up to a half-ton. “With persistent optical broadband, Muon Halo satellites will move from being isolated vehicles to becoming active, realtime nodes on Starlink’s global network,” Stang said in a press release. “That shift transforms how missions are designed and how fast insights flow to decisionmakers on Earth.”

Muon said the first laser-equipped satellite will launch in early 2027 for an undisclosed customer.

“We like to believe part of why SpaceX trusts us to be the ones to be able to lead on this is because our system is designed to really deal with very different levels of requirements,” Smirin said. “As far as we’re aware, this is the first integration into a satellite. We have a ton of interest from commercial customers for our capabilities in general, and we expect this should just boost that quite significantly.”

FireSat is one of the missions where Starlink connectivity would have an impact by rapidly informing first responders of a wildfire, Smirin said. According to Muon, using satellite laser links would cut FireSat data latency from an average of 20 minutes to near real-time.

“It’s not just for the initial detection,” Smirin said. “It’s also once a fire is ongoing, [cutting] the time and the latency for seeing the intensity and direction of the fire, and being able to update that in near real-time. It has incredible value to incident commanders on the ground, because they’re trying to figure out a way to position their equipment and their people.”

Thinking big

Ubiquitous connectivity in space could eventually lead to new types of missions. “Now, you’ve got a data center in space,” Smirin said. “You can do AI there. You can connect with data centers on the ground.”

While this first agreement between Muon and SpaceX covers commercial data relay, it’s easy to imagine other applications, such as continuous live drone-like high-resolution streaming video from space for surveillance or weather monitoring. Live video from space has historically been limited to human spaceflight missions or rocket-mounted cameras that operate for a short time.

One example of that is the dazzling live video beamed back to Earth, through Starlink, from SpaceX’s Starship rockets. The laser terminals on Starship operate through the extreme heat of reentry, returning streaming video as plasma envelops the vehicle. This environment routinely causes radio blackouts for other spacecraft as they reenter the atmosphere. With optical links, that’s no longer a problem.

“This starts to enable a whole new category of capabilities, much the same way as when terrestrial computers went from dial-up to broadband,” Smirin said. “You knew what it could do, but we blew through bulletin boards very quickly to many different applications.”

Satellite operators will soon join airlines in using Starlink in-flight Wi-Fi Read More »

should-an-ai-copy-of-you-help-decide-if-you-live-or-die?

Should an AI copy of you help decide if you live or die?

“It would combine demographic and clinical variables, documented advance-care-planning data, patient-recorded values and goals, and contextual information about specific decisions,” he said.

“Including textual and conversational data could further increase a model’s ability to learn why preferences arise and change, not just what a patient’s preference was at a single point in time,” Starke said.

Ahmad suggested that future research could focus on validating fairness frameworks in clinical trials, evaluating moral trade-offs through simulations, and exploring how cross-cultural bioethics can be combined with AI designs.

Only then might AI surrogates be ready to be deployed, but only as “decision aids,” Ahmad wrote. Any “contested outputs” should automatically “trigger [an] ethics review,” Ahmad wrote, concluding that “the fairest AI surrogate is one that invites conversation, admits doubt, and leaves room for care.”

“AI will not absolve us”

Ahmad is hoping to test his conceptual models at various UW sites over the next five years, which would offer “some way to quantify how good this technology is,” he said.

“After that, I think there’s a collective decision regarding how as a society we decide to integrate or not integrate something like this,” Ahmad said.

In his paper, he warned against chatbot AI surrogates that could be interpreted as a simulation of the patient, predicting that future models may even speak in patients’ voices and suggesting that the “comfort and familiarity” of such tools might blur “the boundary between assistance and emotional manipulation.”

Starke agreed that more research and “richer conversations” between patients and doctors are needed.

“We should be cautious not to apply AI indiscriminately as a solution in search of a problem,” Starke said. “AI will not absolve us from making difficult ethical decisions, especially decisions concerning life and death.”

Truog, the bioethics expert, told Ars he “could imagine that AI could” one day “provide a surrogate decision maker with some interesting information, and it would be helpful.”

But a “problem with all of these pathways… is that they frame the decision of whether to perform CPR as a binary choice, regardless of context or the circumstances of the cardiac arrest,” Truog’s editorial said. “In the real world, the answer to the question of whether the patient would want to have CPR” when they’ve lost consciousness, “in almost all cases,” is “it depends.”

When Truog thinks about the kinds of situations he could end up in, he knows he wouldn’t just be considering his own values, health, and quality of life. His choice “might depend on what my children thought” or “what the financial consequences would be on the details of what my prognosis would be,” he told Ars.

“I would want my wife or another person that knew me well to be making those decisions,” Truog said. “I wouldn’t want somebody to say, ‘Well, here’s what AI told us about it.’”

Should an AI copy of you help decide if you live or die? Read More »

apple-pays-$750-million-for-us-formula-1-streaming-coverage

Apple pays $750 million for US Formula 1 streaming coverage

The United States Grand Prix takes place this weekend at the Circuit of the Americas in Texas, and this morning, Formula 1 used the occasion to announce a new broadcast deal for the sport in the US. Starting next year, F1 will no longer be broadcast on ESPN—it’s moving to Apple TV in a five-year, $750 million deal.

Apple boss Tim Cook has been seen at F1 races in the past, and earlier this year, Apple released F1: The Movie, starring Brad Pitt as a 50-something racing driver who improbably gets a second bite at the cherry 30 years after a brutal crash seemingly ended his F1 career.

But securing the rights to the sport itself means Apple has snagged a very fast-growing series, with races almost every other week—currently, the sport has expanded to 24 races a year.

“We are no strangers to each other, having spent the past three years working together to create F1: The Movie, which has already proven to be a huge hit around the world. We have a shared vision to bring this amazing sport to our fans in the US and entice new fans through live broadcasts, engaging content, and a year-round approach to keep them hooked,” said Stefano Domenicali, F1 president and CEO.

Apple says Apple TV subscribers will be able to watch every practice and qualifying session, as well as all the sprint races and grands prix. And “select races and all practice sessions will also be available for free in the Apple TV app throughout the course of the season,” the company said.

Apple pays $750 million for US Formula 1 streaming coverage Read More »

ai-powered-features-begin-creeping-deeper-into-the-bedrock-of-windows-11

AI-powered features begin creeping deeper into the bedrock of Windows 11


everything old is new again

Copilot expands with an emphasis on creating and editing files, voice input.

Microsoft is hoping that Copilot will succeed as a voice-driven assistant where Cortana failed. Credit: Microsoft

Microsoft is hoping that Copilot will succeed as a voice-driven assistant where Cortana failed. Credit: Microsoft

Like virtually every major Windows announcement in the last three years, the spate of features that Microsoft announced for the operating system today all revolve around generative AI. In particular, they’re concerned with the company’s more recent preoccupation with “agentic” AI, an industry buzzword for “telling AI-powered software to perform a task, which it then does in the background while you move on to other things.”

But the overarching impression I got, both from reading the announcement and sitting through a press briefing earlier this month, is that Microsoft is using language models and other generative AI technologies to try again with Cortana, Microsoft’s failed and discontinued entry in the voice assistant wars of the 2010s.

According to Microsoft’s Consumer Chief Marketing Officer Yusuf Mehdi, “AI PCs” should be able to recognize input “naturally, in text or voice,” to be able to guide users based on what’s on their screens at any given moment, and that AI assistants “should be able to take action on your behalf.”

The biggest of today’s announcements is the introduction of a new “Hey, Copilot” activation phrase for Windows 11 PCs, which once enabled allows users to summon the chatbot using only their voice rather than a mouse or keyboard (if you do want to use the keyboard, either the Copilot key or the same Windows + C keyboard shortcut that used to bring up Cortana will also summon Copilot). Saying “goodbye” will dismiss Copilot when you’re done working with it.

Macs and most smartphones have sported similar functionality for a while now, but Microsoft is obviously hoping that having Copilot answer those questions instead of Cortana will lead to success rather than another failure.

The key limitation of the original Cortana—plus Siri, Alexa, and the rest of their ilk—is that it could only really do a relatively limited and pre-determined list of actions. Complex queries, or anything the assistants don’t understand, often gets bounced to a general web search. The results of that search may or may not accomplish what you wanted, but it does ultimately shift the onus back on the user to find and follow those directions.

To make Copilot more useful, Microsoft has also announced that Copilot Vision is being rolled out worldwide “in all markets where Copilot is offered” (it’s been available in the US since mid-June). Copilot Vision will read the contents of a screen or an app window and can attempt to offer useful guidance or feedback, like walking you through an obscure task in Excel or making suggestions based on a group of photos or a list of items. (Microsoft additionally announced a beta for Gaming Copilot, a sort of offshoot of Copilot Vision intended specifically for walkthroughs and advice for whatever game you happen to be playing.)

Beyond these tweaks or wider rollouts for existing features, Microsoft is also testing a few new AI and Copilot-related additions that aim to fundamentally change how users interact with their Windows PCs by reading and editing files.

All of the features Microsoft is announcing today are intended for all Windows 11 PCs, not just those that meet the stricter hardware requirements of the Copilot+ PC label. That gives them a much wider potential reach than things like Recall or Click to Do, and it makes knowing what these features do and how they safeguard security and privacy that much more important.

AI features work their way into the heart of Windows

Microsoft wants general-purpose AI agents to be able to create and modify files for you, among other things, working in the background while you move on to other tasks. Credit: Microsoft

Whether you’re talking about the Copilot app, the generative AI features added to apps like Notepad and Paint, or the data-scraping Windows Recall feature, most of the AI additions to Windows in the last few years have been app-specific, or cordoned off in some way from core Windows features like the taskbar and File Explorer.

But AI features are increasingly working their way into bedrock Windows features like the taskbar and Start menu, and being given capabilities that allow them to analyze or edit files or even perform file management tasks.

The standard Search field that has been part of Windows 10 and Windows 11 for the last decade, for example, is being transformed into an “Ask Copilot” field; this feature will still be able to look through local files just like the current version of the Search box, but Microsoft also envisions it as a keyboard-driven interface for Copilot for the times when you can’t or don’t want to use your voice. (We don’t know whether the “old” search functionality lives on in the Start menu or as an optional fallback for people who disable Copilot, at least not yet.)

A feature called Copilot Actions will also expand the number of ways that Copilot can interact with local files on your PC. Microsoft cites “sorting through recent vacation photos” and extracting information from PDFs and other documents as two possible use cases, and that this early preview version will focus on “a narrow set of use cases.” But it’s meant to be “a general-purpose agent” capable of “interacting with desktop and web applications.” This gives it a lot of latitude to augment or replace basic keyboard-and-mouse input for some interactions.

Screenshots of a Windows 11 testing build showed Copilot taking over the area of the taskbar that is currently reserved for the Search field. Credit: Microsoft

Finally, Microsoft is taking another stab at allowing Copilot to change the settings on your PC, something that earlier versions were able to do but were removed in a subsequent iteration. Copilot will attempt to respond to plain-language questions about your PC settings with a link to the appropriate part of Windows’ large, labyrinthine Settings app.

These new features dovetail with others Microsoft has been testing for a few weeks or months now. Copilot Connectors, rolled out to Windows Insiders earlier this month, can give Copilot access to email and file-sharing services like Gmail and Dropbox. New document creation features allow Copilot to export the contents of a Copilot chat into a Word or PDF document, Excel spreadsheet, or PowerPoint deck for more refinement and editing. And AI actions in the File Explorer appear in Windows’ right-click menu and allow for the direct manipulation of files, including batch-editing images and summarizing documents. Together with the Copilot Vision features that enable Copilot to see the full contents of Office documents rather than just the on-screen portions, all of these features inject AI into more basic everyday tasks, rather than cordoning them off in individual apps.

Per usual, we don’t know exactly when any of these new features will roll out to the general public, and some may never be available outside of the Windows Insider program. None of them are currently baked into the Windows 11 25H2 update, at least not the version that the company is currently distributing through its Release Preview channel.

Learning the lessons of Recall

Microsoft at least seems to have learned lessons from the botched rollout of Windows Recall last year.

If you didn’t follow along: Microsoft’s initial plan had been to roll out Recall with the first wave of Copilot+ PCs, but without sending it through the Windows Insider Preview program first. This program normally gives power users, developers, security researchers, and others the opportunity to kick the tires on upcoming Windows features before they’re launched, giving Microsoft feedback on bugs, security holes, or other flaws before rolling them out to all Windows PCs.

But security researchers who did manage to get their hands on the early, nearly launched version of Recall discovered a deeply flawed feature that preserved too much personal information and was trivially easy to exploit—a plain-text file with OCR text from all of a user’s PC usage could be grabbed by pretty much anybody with access to the PC, either in-person or remote. It was also enabled by default on PCs that supported it, forcing users to manually opt out if they didn’t want to use it.

In the end, Microsoft pulled that version of Recall, took nearly a year to overhaul its security architecture, and spent months letting the feature make its way through the Windows Insider Preview channels before finally rolling it out to Copilot+ PCs. The resulting product still presents some risks to user privacy, as does any feature that promises to screenshot and store months of history about how you use your PC, but it’s substantially more refined, the most egregious security holes have been closed, and it’s off by default.

Copilot Actions are, at least for now, also disabled by default. And Microsoft Corporate Vice President of Windows Security Dana Huang put up a lengthy accompanying post explaining several of the steps Microsoft has taken to protect user privacy and security when using Copilot Actions. These include running AI agents with their own dedicated user accounts to reduce their access to data in your user folder; mandatory code-signing; and giving agents the fewest privileges they need to do their jobs. All of the agents’ activities will also be documented, so users can verify what actions have been taken and correct any errors.

Whether these security and privacy promises are good enough is an open question, but unlike the initial version of Recall, all of these new features will be sent out through the Windows Insider channels for testing first. If there are serious flaws, they’ll be out in public early on, rather than dropped on users unawares.

Photo of Andrew Cunningham

Andrew is a Senior Technology Reporter at Ars Technica, with a focus on consumer tech including computer hardware and in-depth reviews of operating systems like Windows and macOS. Andrew lives in Philadelphia and co-hosts a weekly book podcast called Overdue.

AI-powered features begin creeping deeper into the bedrock of Windows 11 Read More »