Author name: Mike M.

ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots.txt

AI haters build tarpits to trap and trick AI scrapers that ignore robots.txt


Making AI crawlers squirm

Attackers explain how an anti-spam defense became an AI weapon.

Last summer, Anthropic inspired backlash when its ClaudeBot AI crawler was accused of hammering websites a million or more times a day.

And it wasn’t the only artificial intelligence company making headlines for supposedly ignoring instructions in robots.txt files to avoid scraping web content on certain sites. Around the same time, Reddit’s CEO called out all AI companies whose crawlers he said were “a pain in the ass to block,” despite the tech industry otherwise agreeing to respect “no scraping” robots.txt rules.

Watching the controversy unfold was a software developer whom Ars has granted anonymity to discuss his development of malware (we’ll call him Aaron). Shortly after he noticed Facebook’s crawler exceeding 30 million hits on his site, Aaron began plotting a new kind of attack on crawlers “clobbering” websites that he told Ars he hoped would give “teeth” to robots.txt.

Building on an anti-spam cybersecurity tactic known as tarpitting, he created Nepenthes, malicious software named after a carnivorous plant that will “eat just about anything that finds its way inside.”

Aaron clearly warns users that Nepenthes is aggressive malware. It’s not to be deployed by site owners uncomfortable with trapping AI crawlers and sending them down an “infinite maze” of static files with no exit links, where they “get stuck” and “thrash around” for months, he tells users. Once trapped, the crawlers can be fed gibberish data, aka Markov babble, which is designed to poison AI models. That’s likely an appealing bonus feature for any site owners who, like Aaron, are fed up with paying for AI scraping and just want to watch AI burn.

Tarpits were originally designed to waste spammers’ time and resources, but creators like Aaron have now evolved the tactic into an anti-AI weapon. As of this writing, Aaron confirmed that Nepenthes can effectively trap all the major web crawlers. So far, only OpenAI’s crawler has managed to escape.

It’s unclear how much damage tarpits or other AI attacks can ultimately do. Last May, Laxmi Korada, Microsoft’s director of partner technology, published a report detailing how leading AI companies were coping with poisoning, one of the earliest AI defense tactics deployed. He noted that all companies have developed poisoning countermeasures, while OpenAI “has been quite vigilant” and excels at detecting the “first signs of data poisoning attempts.”

Despite these efforts, he concluded that data poisoning was “a serious threat to machine learning models.” And in 2025, tarpitting represents a new threat, potentially increasing the costs of fresh data at a moment when AI companies are heavily investing and competing to innovate quickly while rarely turning significant profits.

“A link to a Nepenthes location from your site will flood out valid URLs within your site’s domain name, making it unlikely the crawler will access real content,” a Nepenthes explainer reads.

The only AI company that responded to Ars’ request to comment was OpenAI, whose spokesperson confirmed that OpenAI is already working on a way to fight tarpitting.

“We’re aware of efforts to disrupt AI web crawlers,” OpenAI’s spokesperson said. “We design our systems to be resilient while respecting robots.txt and standard web practices.”

But to Aaron, the fight is not about winning. Instead, it’s about resisting the AI industry further decaying the Internet with tech that no one asked for, like chatbots that replace customer service agents or the rise of inaccurate AI search summaries. By releasing Nepenthes, he hopes to do as much damage as possible, perhaps spiking companies’ AI training costs, dragging out training efforts, or even accelerating model collapse, with tarpits helping to delay the next wave of enshittification.

“Ultimately, it’s like the Internet that I grew up on and loved is long gone,” Aaron told Ars. “I’m just fed up, and you know what? Let’s fight back, even if it’s not successful. Be indigestible. Grow spikes.”

Nepenthes instantly inspires another tarpit

Nepenthes was released in mid-January but was instantly popularized beyond Aaron’s expectations after tech journalist Cory Doctorow boosted a tech commentator, Jürgen Geuter, praising the novel AI attack method on Mastodon. Very quickly, Aaron was shocked to see engagement with Nepenthes skyrocket.

“That’s when I realized, ‘oh this is going to be something,'” Aaron told Ars. “I’m kind of shocked by how much it’s blown up.”

It’s hard to tell how widely Nepenthes has been deployed. Site owners are discouraged from flagging when the malware has been deployed, forcing crawlers to face unknown “consequences” if they ignore robots.txt instructions.

Aaron told Ars that while “a handful” of site owners have reached out and “most people are being quiet about it,” his web server logs indicate that people are already deploying the tool. Likely, site owners want to protect their content, deter scraping, or mess with AI companies.

When software developer and hacker Gergely Nagy, who goes by the handle “algernon” online, saw Nepenthes, he was delighted. At that time, Nagy told Ars that nearly all of his server’s bandwidth was being “eaten” by AI crawlers.

Already blocking scraping and attempting to poison AI models through a simpler method, Nagy took his defense method further and created his own tarpit, Iocaine. He told Ars the tarpit immediately killed off about 94 percent of bot traffic to his site, which was primarily from AI crawlers. Soon, social media discussion drove users to inquire about Iocaine deployment, including not just individuals but also organizations wanting to take stronger steps to block scraping.

Iocaine takes ideas (not code) from Nepenthes, but it’s more intent on using the tarpit to poison AI models. Nagy used a reverse proxy to trap crawlers in an “infinite maze of garbage” in an attempt to slowly poison their data collection as much as possible for daring to ignore robots.txt.

Taking its name from “one of the deadliest poisons known to man” from The Princess Bride, Iocaine is jokingly depicted as the “deadliest poison known to AI.” While there’s no way of validating that claim, Nagy’s motto is that the more poisoning attacks that are out there, “the merrier.” He told Ars that his primary reasons for building Iocaine were to help rights holders wall off valuable content and stop AI crawlers from crawling with abandon.

Tarpits aren’t perfect weapons against AI

Running malware like Nepenthes can burden servers, too. Aaron likened the cost of running Nepenthes to running a cheap virtual machine on a Raspberry Pi, and Nagy said that serving crawlers Iocaine costs about the same as serving his website.

But Aaron told Ars that Nepenthes wasting resources is the chief objection he’s seen preventing its deployment. Critics fear that deploying Nepenthes widely will not only burden their servers but also increase the costs of powering all that AI crawling for nothing.

“That seems to be what they’re worried about more than anything,” Aaron told Ars. “The amount of power that AI models require is already astronomical, and I’m making it worse. And my view of that is, OK, so if I do nothing, AI models, they boil the planet. If I switch this on, they boil the planet. How is that my fault?”

Aaron also defends against this criticism by suggesting that a broader impact could slow down AI investment enough to possibly curb some of that energy consumption. Perhaps due to the resistance, AI companies will be pushed to seek permission first to scrape or agree to pay more content creators for training on their data.

“Any time one of these crawlers pulls from my tarpit, it’s resources they’ve consumed and will have to pay hard cash for, but, being bullshit, the money [they] have spent to get it won’t be paid back by revenue,” Aaron posted, explaining his tactic online. “It effectively raises their costs. And seeing how none of them have turned a profit yet, that’s a big problem for them. The investor money will not continue forever without the investors getting paid.”

Nagy agrees that the more anti-AI attacks there are, the greater the potential is for them to have an impact. And by releasing Iocaine, Nagy showed that social media chatter about new attacks can inspire new tools within a few days. Marcus Butler, an independent software developer, similarly built his poisoning attack called Quixotic over a few days, he told Ars. Soon afterward, he received messages from others who built their own versions of his tool.

Butler is not in the camp of wanting to destroy AI. He told Ars that he doesn’t think “tools like Quixotic (or Nepenthes) will ‘burn AI to the ground.'” Instead, he takes a more measured stance, suggesting that “these tools provide a little protection (a very little protection) against scrapers taking content and, say, reposting it or using it for training purposes.”

But for a certain sect of Internet users, every little bit of protection seemingly helps. Geuter linked Ars to a list of tools bent on sabotaging AI. Ultimately, he expects that tools like Nepenthes are “probably not gonna be useful in the long run” because AI companies can likely detect and drop gibberish from training data. But Nepenthes represents a sea change, Geuter told Ars, providing a useful tool for people who “feel helpless” in the face of endless scraping and showing that “the story of there being no alternative or choice is false.”

Criticism of tarpits as AI weapons

Critics debating Nepenthes’ utility on Hacker News suggested that most AI crawlers could easily avoid tarpits like Nepenthes, with one commenter describing the attack as being “very crawler 101.” Aaron said that was his “favorite comment” because if tarpits are considered elementary attacks, he has “2 million lines of access log that show that Google didn’t graduate.”

But efforts to poison AI or waste AI resources don’t just mess with the tech industry. Governments globally are seeking to leverage AI to solve societal problems, and attacks on AI’s resilience seemingly threaten to disrupt that progress.

Nathan VanHoudnos is a senior AI security research scientist in the federally funded CERT Division of the Carnegie Mellon University Software Engineering Institute, which partners with academia, industry, law enforcement, and government to “improve the security and resilience of computer systems and networks.” He told Ars that new threats like tarpits seem to replicate a problem that AI companies are already well aware of: “that some of the stuff that you’re going to download from the Internet might not be good for you.”

“It sounds like these tarpit creators just mainly want to cause a little bit of trouble,” VanHoudnos said. “They want to make it a little harder for these folks to get” the “better or different” data “that they’re looking for.”

VanHoudnos co-authored a paper on “Counter AI” last August, pointing out that attackers like Aaron and Nagy are limited in how much they can mess with AI models. They may have “influence over what training data is collected but may not be able to control how the data are labeled, have access to the trained model, or have access to the Al system,” the paper said.

Further, AI companies are increasingly turning to the deep web for unique data, so any efforts to wall off valuable content with tarpits may be coming right when crawling on the surface web starts to slow, VanHoudnos suggested.

But according to VanHoudnos, AI crawlers are also “relatively cheap,” and companies may deprioritize fighting against new attacks on crawlers if “there are higher-priority assets” under attack. And tarpitting “does need to be taken seriously because it is a tool in a toolkit throughout the whole life cycle of these systems. There is no silver bullet, but this is an interesting tool in a toolkit,” he said.

Offering a choice to abstain from AI training

Aaron told Ars that he never intended Nepenthes to be a major project but that he occasionally puts in work to fix bugs or add new features. He said he’d consider working on integrations for real-time reactions to crawlers if there was enough demand.

Currently, Aaron predicts that Nepenthes might be most attractive to rights holders who want AI companies to pay to scrape their data. And many people seem enthusiastic about using it to reinforce robots.txt. But “some of the most exciting people are in the ‘let it burn’ category,” Aaron said. These people are drawn to tools like Nepenthes as an act of rebellion against AI making the Internet less useful and enjoyable for users.

Geuter told Ars that he considers Nepenthes “more of a sociopolitical statement than really a technological solution (because the problem it’s trying to address isn’t purely technical, it’s social, political, legal, and needs way bigger levers).”

To Geuter, a computer scientist who has been writing about the social, political, and structural impact of tech for two decades, AI is the “most aggressive” example of “technologies that are not done ‘for us’ but ‘to us.'”

“It feels a bit like the social contract that society and the tech sector/engineering have had (you build useful things, and we’re OK with you being well-off) has been canceled from one side,” Geuter said. “And that side now wants to have its toy eat the world. People feel threatened and want the threats to stop.”

As AI evolves, so do attacks, with one 2021 study showing that increasingly stronger data poisoning attacks, for example, were able to break data sanitization defenses. Whether these attacks can ever do meaningful destruction or not, Geuter sees tarpits as a “powerful symbol” of the resistance that Aaron and Nagy readily joined.

“It’s a great sign to see that people are challenging the notion that we all have to do AI now,” Geuter said. “Because we don’t. It’s a choice. A choice that mostly benefits monopolists.”

Tarpit creators like Nagy will likely be watching to see if poisoning attacks continue growing in sophistication. On the Iocaine site—which, yes, is protected from scraping by Iocaine—he posted this call to action: “Let’s make AI poisoning the norm. If we all do it, they won’t have anything to crawl.”

Photo of Ashley Belanger

Ashley is a senior policy reporter for Ars Technica, dedicated to tracking social impacts of emerging policies and new technologies. She is a Chicago-based journalist with 20 years of experience.

AI haters build tarpits to trap and trick AI scrapers that ignore robots.txt Read More »

there’s-not-much-for-anyone-to-like-in-the-star-trek:-section-31-movie

There’s not much for anyone to like in the Star Trek: Section 31 movie

It is, in a word, awful. Which is really a shame!

Putting the “TV” in “TV movie”

Sam Richardson as Quasi, a shape-shifter. Comedy and melodrama coexist uneasily throughout Section 31. Credit: Michael Gibson/Paramount+

The movie explains its premise clearly enough, albeit in a clumsy exposition-heavy voiceover section near the beginning: Philippa Georgiou (Michelle Yeoh) was once the ruler of the bloodthirsty Terran Empire, an evil mirror of Star Trek’s utopian United Federation of Planets. She crossed over into “our” universe and gradually reformed, sort of, before vanishing. Now Section 31—Starfleet’s version of the CIA, more or less—needs to track her down and enlist her to help them save the galaxy from another threat that has crossed over from the evil universe to ours.

Emperor Georgiou originated on Star Trek: Discovery, and she was a consistently fun presence on a very uneven show. Yeoh clearly had a blast playing a sadistic, horny version of the kind and upstanding Captain Georgiou who died in Discovery‘s premiere.

But that fun is mostly absent here. To the extent that anything about Section 31 works, it’s as a sort of brain-off generic sci-fi action movie, Star Trek’s stab at a Suicide Squad-esque antihero story. Things happen in space, sometimes in a spaceship. There is some fighting, though nearly all of it involves punching instead of phasers or photon torpedoes. There is an Important Item that needs to be chased down, for the Fate of the Universe is at stake.

But the movie also feels more like a failed spin-off pilot that never made it to series, and it suffers for it; it’s chopped up into four episodes “chapters” and has to establish an entire crew’s worth of quirky misfits inside a 10-minute montage.

That might work if the script or the performers could make any of the characters endearing, but it isn’t, and they don’t. Performances are almost uniformly bad, ranging from inert to unbearable to “not trying particularly hard” (respectively: Omari Hardwick’s Alok, a humorless genetically augmented human; Sven Ruygrok’s horrifically grating Fuzz, a tiny and inexplicably Irish alien piloting a Vulkan-shaped robot; and Sam Richardson’s Quasi, whose amiable patter is right at home on Detroiters and I Think You Should Leave but is mostly distracting here). Every time one of these characters ends up dead, you feel a sense of relief because there’s one fewer one-note character to have to pay attention to.

There’s not much for anyone to like in the Star Trek: Section 31 movie Read More »

dead-babies,-critically-ill-kids:-pediatricians-make-moving-plea-for-vaccines

Dead babies, critically ill kids: Pediatricians make moving plea for vaccines

As federal lawmakers prepare to decide whether anti-vaccine advocate Robert F. Kennedy Jr. should be the next secretary of the Department of Health and Human Services, pediatricians from around the country are making emotional pleas to protect and support lifesaving immunizations.

The American Academy of Pediatrics (AAP) has assembled nearly 200 stories and dozens of testimonials on the horrors of vaccine-preventable deaths and illnesses that pediatricians have encountered over their careers. The testimonials have been shared with two Senate committees that will hold hearings later this week: the Senate Committee on Finance and the Senate Committee on Health, Education, Labor, and Pensions (HELP).

“I remember that baby’s face to this day”

In a statement on Monday, AAP President Susan Kressly noted that the stories come from a wide range of pediatricians—from rural to urban and from small practices to large institutions. Some have recalled stories of patients who became ill with devastating diseases before vaccines were available to prevent them, while others shared more recent experiences as vaccine misinformation spread and vaccination rates slipped.

In one, a pediatrician from Raleigh, North Carolina, spoke of a baby in the 1990s with Streptococcus pneumoniae meningitis, a life-threatening disease. “I remember holding a baby dying of complications of pneumococcal meningitis at that time. I remember that baby’s face to this day—but, thanks to pneumococcal vaccination, have never had to relive that experience since,” the doctor said. The first pneumococcal vaccine for infants was licensed in the US in 2000.

A doctor in Portland, Maine, meanwhile, faced the same disease in a patient who was unvaccinated despite the availability of the vaccine. “As a resident, I cared for a young, unvaccinated child admitted to the pediatric intensive care unit with life-threatening Streptococcus pneumoniae meningitis. This devastating illness, once common, has become rare thanks to the widespread use of pneumococcal conjugate vaccines. However, this child was left vulnerable…and [their parents] now faced the anguish of watching their child fight for their life on a ventilator.”

Kressly emphasizes that “One unifying theme of these stories: vaccines allow children to grow up healthy and thrive. As senators consider nominees for federal healthcare agencies, we hope these testimonies will help paint a picture of just how important vaccinations are to children’s long-term health and wellbeing.”

Dead babies, critically ill kids: Pediatricians make moving plea for vaccines Read More »

3d-printed-“ghost-gun”-ring-comes-to-my-community—and-leaves-a-man-dead

3D-printed “ghost gun” ring comes to my community—and leaves a man dead

It’s a truism at this point to say that Americans own a lot of guns. Case in point: This week, a fire chief in rural Alabama stopped to help a driver who had just hit a deer. The two men walked up the driveway of a nearby home. For reasons that remain unclear, a man came out of the house with a gun and started shooting. This was a bad idea on many levels, but most practically because both the fire chief and the driver were also armed. Between the three of them, everyone got shot, the fire chief died, and the man who lived in the home was charged with murder.

But despite the ease of acquiring legal weapons, a robust black market still exists to traffic in things like “ghost guns” (no serial numbers) and machine gun converters (which make a semi-automatic weapon into an automatic). According to a major new report released this month by the Bureau of Alcohol, Tobacco, Firearms, and Explosives, there was a 1,600 percent increase in the use of privately made “ghost guns” during crimes between 2017 and 2023. Between 2019 and 2023, the seizure of machine gun converters also increased by 784 percent.

Ars Technica has covered these issues for years, since both “ghost guns” and machine gun converters can be produced using 3D-printed parts, the schematics for which are now widely available online. But you can know about an issue and still be surprised when local prosecutors start talking about black market trafficking rings, inept burglary schemes, murder—and 3D printing operations being run out of a local apartment.

Philadelphia story

I live in the Philadelphia area, and this is a real Philadelphia story; I know all of the places in it well. Many people in this story live in Philadelphia proper, but the violence (and the 3D printing!) they are accused of took place in the suburbs, in places like Jenkintown, Lower Merion township, and Bucks County. If you know Philly at all, you may know that these are all west and northwest suburban areas and that all of them are fairly comfortable places overall. Indeed, The New York Times ran a long story this month called “How Sleepy Bucks County Became a Rival to the Hamptons.” Lower Merion is one of the wealthier Philly suburbs, while Jenkintown is a charming little northwest suburb that was also the setting for the long-running sitcom The Goldbergs. Local county prosecutors are more often busting up shipments of fake Jason Kelce-autographed merch or going after—and later not going after—comedian Bill Cosby.

But today, prosecutors in Montgomery County announced something different: they had cracked open a local 3D-printing black market gun ring—and said that one of the group’s 3D-printed guns was used last month to murder a man during a botched burglary.

Mug shots of Fuentes and Fulforth

Mug shots of Fuentes and Fulforth. Credit: Montco DA’s Office

It’s a pretty bizarre story. As the police tell it, things began with 26-year-old Jeremy Fuentes driving north to a Bucks County address. Fuentes worked for a junk hauling company in nearby Willow Grove, and he had gone to Bucks County to give an estimate for a job. While the homeowner was showing Fuentes around the property, Fuentes allegedly noticed “a large gun safe, multiple firearms boxes, gun parts and ammunition” in the home.

Outside of work, Fuentes was said to be a member of a local black market gun ring, and so when he saw this much gun gear in one spot—and when he noted that the homeowners were elderly—he saw dollar signs. Police say that after the estimate visit, Fuentes contacted Charles Fulforth, 41, of Jenkintown, who was a key member of the gun ring.

Fuentes had an idea: Fulforth should rob the home and steal all the gun-related supplies. Unfortunately, the group was not great at directions. Fuentes didn’t provide complete and correct information, so when Fulforth and an accomplice went to rob the home in December 2024, they drove to a Lower Merion home instead. This home was not in Bucks County at all—in fact, it was 30 minutes south—but it had a similar street address to the home Fuentes had visited.

When they invaded the Lower Merion home on December 8, the two burglars found not an elderly couple but a 25-year-old man named Andrew Gaudio and his 61-year-old mother, Bernadette. Andrew was killed, while Bernadette was shot but survived.

Police arrested Fulforth just three days later, on December 11, and they picked up his fellow burglar on December 17. But the cops didn’t immediately realize just what they had stumbled into. Only after they searched Fulforth’s Jenkintown apartment and found a 9 mm 3D-printed gun did they realize this might be more than a simple burglary. How had Fulforth acquired the weapon?

According to a statement on the case released today by the Montgomery County District Attorney, the investigation involved “search warrants on multiple locations and forensic searches of mobile phones,” which revealed that Fulforth had his own “firearm production facility”—aka, “a group of 3D printers.” Detectives even found a video of a Taurus-style gun part being printed on the devices, and they came to believe that the gun used to kill Andrew Gaudio was “one of many manufactured by Fulforth.”

In addition to making ghost gun parts at his “highly sophisticated, clandestine firearms production facility,” Fulforth was also accused of making machine gun converters with 3D-printed parts. These parts would be preinstalled in the guns that the group was trafficking to raise their value. According to investigators, “From the review of the captured cellphone communications among the gun trafficking members, the investigation found that when [machine gun conversion] switches were installed on AR pistols, it increased the price of the firearm by at least $1,000.”

Fuentes, who had initially provided the address that led to the murder, was arrested this morning. Authorities have also charged five others with being part of the gun ring.

So, a tragic and stupid story, but one that highlights just how mainstream 3D-printing tech has become. No massive production facility or dimly lit warehouse is needed—just put a few printers in a bedroom and you, too, can become a local gun trafficking kingpin.

There’s nothing novel about any of this, and in fact, fewer people were shot than in that bizarre Alabama gun battle mentioned up top. Still, it hits home when a technology I’ve both written and read about for years on Ars shows up in your community—and leaves a man dead.

3D-printed “ghost gun” ring comes to my community—and leaves a man dead Read More »

who-starts-cutting-costs-as-us-withdrawal-date-set-for-january-2026

WHO starts cutting costs as US withdrawal date set for January 2026

“Just stupid”

On January 23, WHO Director-General Tedros Adhanom Ghebreyesus sent a memo to staff announcing the cost-cutting measures. Reuters obtained a copy of the memo.

“This announcement has made our financial situation more acute,” Tedros wrote, referring to the US withdrawal plans. WHO’s budget mainly comes from dues and voluntary contributions from member states. The dues are a percentage of each member state’s gross domestic product, and the percentage is set by the UN General Assembly. US contributions account for about 18 percent of WHO’s overall funding, and its two-year 2024-2025 budget was $6.8 billion, according to Reuters.

To prepare for the budget cut, WHO is halting recruitment, significantly curtailing travel expenditures, making all meetings virtual, limiting IT equipment updates, and suspending office refurbishment.

“This set of measures is not comprehensive, and more will be announced in due course,” Tedros wrote, adding that the agency would do everything it could to protect and support staff.

The country’s pending withdrawal has been heavily criticized by global health leaders and US experts, who say it will make the world less safe and weaken America. In a CBS/KFF Health News report examining the global health implications of the US withdrawal, Kenneth Bernard, a visiting fellow at the Hoover Institution at Stanford University who served as a top biodefense official during the George W. Bush administration, did not mince words:

“It’s just stupid,” Bernard said. “Withdrawing from the WHO leaves a gap in global health leadership that will be filled by China,” he said, “which is clearly not in America’s best interests.”

WHO starts cutting costs as US withdrawal date set for January 2026 Read More »

couple-allegedly-tricked-ai-investors-into-funding-wedding,-houses

Couple allegedly tricked AI investors into funding wedding, houses

To further the alleged scheme, he “often described non-existent revenue, inflated cash balances,” and “otherwise exaggerated customer relationships,” the US Attorney’s Office said, to convince investors to spend millions. As Beckman’s accomplice, Lau allegedly manipulated documents, including documents allegedly stolen from the venture capital firm that employed her while supposedly hiding her work for GameOn.

The scheme apparently also included forging audits and bank statements, as well as using “the names of at least seven real people—including fake emails and signatures—without their permission to distribute false and fraudulent GameOn financial and business information and documents with the intent to defraud GameOn and its investors,” the US Attorney’s Office said.

At perhaps the furthest extreme, Lau allegedly falsified account statements, including once faking a balance of over $13 million when that account only had $25 in it. The FBI found that GameOn’s revenues never exceeded $1 million in any year, while Beckman allegedly inflated sales to investors, including claiming that sales in one quarter in 2023 got as high as $72 million.

Beckman and Lau allegedly went to great lengths to hide the scheme while diverting investor funds to their personal accounts. While GameOn employees allegedly sometimes went without paychecks, Beckman and Lau allegedly stole funds to buy expensive San Francisco real estate and pay for their wedding in 2023. If convicted, they may be forced to forfeit a $4.2 million house, a Tesla Model X, and other real estate and property purchased with their allegedly ill-gotten gains, the indictment said.

It took about five years for the cracks to begin to show in Beckman’s scheme. Beginning in 2023, Beckman increasingly started facing “questions about specific customers and specific revenue from those customers,” the indictment said. By February 2024, Beckman at last “acknowledged to at least one GameOn consultant” that a flagged audit report “did not contain accurate financial information,” but allegedly, he “attempted to shift blame to others for the inaccuracies.”

Couple allegedly tricked AI investors into funding wedding, houses Read More »

stargate-ai-1

Stargate AI-1

There was a comedy routine a few years ago. I believe it was by Hannah Gadsby. She brought up a painting, and looked at some details. The details weren’t important in and of themselves. If an AI had randomly put them there, we wouldn’t care.

Except an AI didn’t put them there. And they weren’t there at random.

A human put them there. On purpose. Or, as she put it:

THAT was a DECISION.

This is the correct way to view decisions around a $500 billion AI infrastructure project, announced right after Trump takes office, having it be primarily funded by SoftBank, with all the compute intended to be used by OpenAI, and calling it Stargate.

  1. The Announcement.

  2. Is That a Lot?.

  3. What Happened to the Microsoft Partnership?.

  4. Where’s Our 20%?.

  5. Show Me the Money.

  6. It Never Hurts to Suck Up to the Boss.

  7. What’s in a Name.

  8. Just Think of the Potential.

  9. I Believe Toast is an Adequate Description.

  10. The Lighter Side.

OpenAI: Announcing The Stargate Project

The Stargate Project is a new company which intends to invest $500 billion over the next four years building new AI infrastructure for OpenAI in the United States. We will begin deploying $100 billion immediately.

Note that ‘intends to invest’ does not mean ‘has the money to invest’ or ‘definitely will invest.’ Intends is not a strong word. The future is unknown and indeed do many things come to pass.

This infrastructure will secure American leadership in AI, create hundreds of thousands of American jobs, and generate massive economic benefit for the entire world.

This project will not only support the re-industrialization of the United States but also provide a strategic capability to protect the national security of America and its allies.

One of these things is not like the others. Secure American leadership in AI, generate massive economic benefit for the entire world, provide strategic capability to allies, sure, fine, makes sense, support reindustrialization is a weird flex but kinda, yeah.

And then… jobs? American… jobs? Um, Senator Blumenthal, that is not what I meant.

Pradyumna:

> will develop superintelligence

> create thousands of jobs

????

Samuel Hammond: “We’re going to spend >10x the budget of the Manhattan Project building digital brains that can do anything human brains can do but better and oh, by the way, create over 100,000 good paying American jobs!”

There’s at least some cognitive dissonance here.

Arthur B: The project will probably most likely lead to mass unemployment but in the meantime, there’ll be great American jobs.

If you listen to Altman’s announcement, he too highlights these ‘hundreds of thousands of jobs.’ It’s so absurd. Remember when Altman tried to correct this error?

The initial equity funders in Stargate are SoftBank, OpenAI, Oracle, and MGX. SoftBank and OpenAI are the lead partners for Stargate, with SoftBank having financial responsibility and OpenAI having operational responsibility. Masayoshi Son will be the chairman.

Arm, Microsoft, NVIDIA, Oracle, and OpenAI are the key initial technology partners.

If you want to spend way too much money on a technology project, and give the people investing the money a remarkably small share of the enterprise, you definitely want to be giving Masayoshi Sun and Softbank a call.

“Sam Altman, you are not crazy enough. You need to think bigger.”

The buildout is currently underway, starting in Texas, and we are evaluating potential sites across the country for more campuses as we finalize definitive agreements.

This proves there is real activity, also it is a tell that some of this is not new.

As part of Stargate, Oracle, NVIDIA, and OpenAI will closely collaborate to build and operate this computing system. This builds on a deep collaboration between OpenAI and NVIDIA going back to 2016 and a newer partnership between OpenAI and Oracle.

This also builds on the existing OpenAI partnership with Microsoft. OpenAI will continue to increase its consumption of Azure as OpenAI continues its work with Microsoft with this additional compute to train leading models and deliver great products and services.

Increase consumption of compute is different from Azure as sole compute provider. It seems OpenAI expects plenty of compute needs to go around.

All of us look forward to continuing to build and develop AI—and in particular AGI—for the benefit of all of humanity. We believe that this new step is critical on the path, and will enable creative people to figure out how to use AI to elevate humanity.

Can’t stop, won’t stop, I suppose. ‘Enable creative people to elevate humanity’ continues to miss the point of the whole enterprise, but not as much as talking ‘jobs.’

Certainly $500 billion for this project sounds like a lot. It’s a lot, right?

Microsoft is investing $80 billion a year in Azure, which is $400 billion over 5 years, and I’d bet that their investment goes up over time and they end up spending over $500 billion during that five year window.

Haydn Belfield: Stargate is a remarkable step.

But, to put it into context, Microsoft will spend $80 billion on data centers this year, over half in the U.S.

Stargate’s $100 billion this year is more, but a comparable figure.

Rob S.: This is kind of misleading. Microsoft’s spend is also enormous and wildly out of the ordinary. Not normal at all.

Haydn Belfield: Definitely true, we’re living through a historic infrastructure build out like the railways, interstate highways or phone network

What I want to push back on a bit is that this is *the onlyeffort, that this is the manhattan/Apollo project

The number $500 billion is distributed throughout many sites and physical projects. If it does indeed happen, and it is counterfactual spending, then it’s a lot. But it’s not a sea change, and it’s not obvious that the actual spending should be surprising. Investments on this scale were already very much projected and already happening.

It’s also not that much when compared to the compute needs anticipated for the scaling of top end training runs, which very much continue to be a thing.

Yusuf Mahmood: Stargate shouldn’t have been that surprising!

It’s a $500 Bn project that is set to complete by 2029.

That’s totally consistent with estimates from @EpochAIResearch’s report last year on how scaling could continue through 2030.

$500 billion is a lot is to the extent all of this is dedicated specifically and exclusively to OpenAI, as opposed to Microsoft’s $80 billion which is for everyone. But it’s not a lot compared to the anticipated future needs of a frontier lab.

One thing to think about is that OpenAI recently raised money at a valuation of approximately $170 billion, presumably somewhat higher now with o3 and agents, but also potentially lower because of DeepSeek. Now we are talking about making investments dedicated to OpenAI of $500 billion.

There is no theoretical incompatibility. Perhaps OpenAI is mining for gold and will barely recoup its investment, while Stargate is selling pickaxes and will rake it in.

It does still seem rather odd to presume that is how the profits will be distributed.

The reason OpenAI is so unprofitable today is that they are spending a ton on increasing capabilities, and not serving enough inference to make it up on their unit economics, and also not yet using their AI to make money in other ways.

And yes, the equilibrium could end up being that compute providers have margins and model providers mostly don’t have margins. But OpenAI, if it succeeds, should massively benefit from economies of scale here, and its economics should improve. Thus, if you take Stargate seriously, it is hard to imagine OpenAI being worth only a fraction of $500 billion.

There is a solution to this puzzle. When we say OpenAI is worth $170 billion, we are not talking about all of OpenAI. We are talking about the part that takes outside investment. All the dramatic upside potential? That is for now owned by the non-profit, and not (or at least not fully) part of the valuation.

And that is the part that has the vast majority of the expected net present value of future cash flows of OpenAI. So OpenAI the entire enterprise can be worth quite a lot, and yet ‘OpenAI’ the corporate entity you can invest in is only worth $170 billion.

This should put into perspective that the move to a for-profit entity truly is in the running for the largest theft in the history of the world.

Didn’t they have an exclusive partnership?

Smoke-Away: OpenAI and Microsoft are finished. There were signs.

Microsoft was not moving quickly enough to scale Azure. Now they are simply another compute provider for the time being.

Sam Altman: Absolutely not! This is a very important and significant partnership, for a long time to come.

We just need moar compute.

Eliezer Yudkowsky (Quoting Smoke-Away): It is a pattern, with Altman. If Altman realizes half his dreams, in a few years we will be hearing about how Altman has dismissed the U.S. government as no longer useful to him. (If Altman realizes all his dreams, you will be dead.)

Roon: Not even close to being true.

Microsoft is one of the providers here. Reports are that the Microsoft partnership has now been renegotiated, to allow OpenAI to also seek other providers, since Altman needs moar compute. Hence Stargate. Microsoft will retain right of first refusal (ROFR), which seems like the right deal to make here. The question is, how much of the non-profit’s equity did Altman effectively promise in order to get free from under the old deal?

Remember that time Altman promised 20% of compute would go to superalignment, rather than blowing up a sun?

Harlan Stewart: Jul 2023: OpenAI promises to dedicate 20% of compute to safety research

May 2024: Fortune reports they never did that

Jul 2024: After 5 senators write to him to ask if OpenAI will, @sama says yes

It’s Jan 2025. Will OpenAI set aside 20% of this new compute to safety, finally?

Connor Axiotes: @tszzl (Roon), can you push for a significant part of this to be spent on control and alignment and safety policy work?

Roon: I’ll do my part. I’m actually on the alignment team at openai 🙂

So that’s a no, then.

I do expect Roon to push for more compute. I don’t expect to get anything like 20%.

Elon Musk (replying to the announcement): They don’t actually have the money.

Sam Altman: I genuinely respect your accomplishments and think you are the most inspiring entrepreneur of our time.

Elon Musk (continuing from OP): SoftBank has well under $10 billion secured. I have that on good authority.

Sam Altman: Wrong, as you surely know.

Want to come visit the first site already under way?

This is great for the country. I realize what is great for the country is not always what is optimal for your companies, but in your new role, I hope you will mostly put the United States first.

Satya Nadella (CEO Microsoft, on CNBC, when asked about whether Stargate has the money, watch the clip at the link his delivery is perfect): All I know is, I’m good for my $80 billion.

If you take the companies collectively, they absolutely have the money, or at least the ability to get the money. This is Microsoft and Nvidia. I have no doubt that Microsoft is, as its Nadella affirmed, ‘good for its $80 billion.’

That doesn’t mean SoftBank has the money, and SoftBank explicitly is tasked with providing the funding for Stargate.

Nor does the first site in Texas prove anything either way on this.

Remember the wording on the announcement: “which intends to invest $500 billion over the next four years.”

That does not sound like someone who has the money.

That sounds like someone who intends to raise the money. And I presume SoftBank has every expectation of being able to do so, with the aid of this announcement. And of working out the structure. And the financing.

Mario Nawfal: Sam Altman’s grand plan to build “Stargate,” a $500 billion AI infrastructure exclusively for OpenAI, is already falling apart before it even starts.

There’s no secured funding, no government support, no detailed plan, and, according to insiders, not even a clear structure.

One source bluntly admitted:

“They haven’t figured out the structure, they haven’t figured out the financing, they don’t have the money committed.”

Altman’s pitch? SoftBank and OpenAI will toss in $15 billion each and then just… hope the rest magically appears from investors and debt.

For someone obsessed with making AI smarter than humans, maybe he should try getting the basics right first – like not creating something that could destroy all of humanity… Just saying.

But that’s why you say ‘intend to invest’ rather than ‘will invest.’

Things between Musk and Altman did not stop there, as we all took this opportunity to break open the International Popcorn Reserve.

Elon Musk: Altman literally testified to Congress that he wouldn’t get OpenAI compensation and now he wants $10 billion! What a liar.

Musk’s not exactly wrong about that. He also said and retweeted other… less dignified things.

It was not a good look for either party. Elon Musk is, well, being Elon Musk. Altman is trying to throw in performative ‘look at me taking the high road’ statements that should fool no one, not only the one above but also:

Sam Altman: just one more mean tweet and then maybe you’ll love yourself…

Teortaxes (quoting Altman saying he respects Elon’s accomplishments above): I find both men depicted here unpleasant and engaging in near-psychopathic behavior, and I also think poorly of those who imagine Sam is trying to “be the bigger man”.

He’s a scary manipulative snake. “Well damn, fyou too Elon, we have it” would be more dignified.

There’s a subtle art to doing this sort of thing well. The Japanese especially are very good at it. All of this is, perhaps, the exact opposite of that.

Sam Altman: big. beautiful. buildings. stargate site 1, texas, january 2025.

Altman, you made it weird. Also guache. Let’s all do better.

Trump world is not, as you would expect, thrilled with what Musk has been up to, with Trump saying he is ‘furious,’ saying he ‘got over his skis.’ My guess is that Trump ‘gets it’ at heart, because he knows what it’s like to hate and never let something go, and that this won’t be that big a deal for Musk’s long term position, but there is high variance. I could easily be wrong about that. If I was Musk I would not have gone with this strategy, but that statement is almost always true and why I’m not Musk.

This particular Rule of Acquisition is somewhat imprecise. It’s not always true.

But Donald Trump? Yeah. It definitely never hurts to suck up to that particular boss.

Sam Altman (January 22, 2025): watching @potus more carefully recently has really changed my perspective on him (i wish i had done more of my own thinking and definitely fell in the npc trap).

i’m not going to agree with him on everything, but i think he will be incredible for the country in many ways!

Altman does admit this is a rather big change. Anyone remember when Altman said “More terrifying than Trump intentionally lying all the time is the possibility that he actually believes it all” or when he congratulated Reid Hoffman for helping keep Trump out of power? Or “Back to work tomorrow on a new project to stop Trump?” He was rather serious about wanting to stop Trump.

You can guess what I think he saw while watching Trump to make Altman change his mind.

So they announced this $500 billion deal, or at least a $100 billion deal with intent to turn it into $500 billion, right after Trump’s inauguration, with construction already underway, with a press conference on the White House lawn.

And the funds are all private. Which is great, but all this together also raises the obvious question: Does Trump actually have anything to do with this?

Matthew Yglesias: They couldn’t have done it without Trump, but also it was already under construction.

Daniel Eth: Okay, it’s not *Trump’sAI plan. He announced it, but he neither developed nor is funding it. It’s a private initiative from OpenAI, Softbank, Oracle, and a few others.

Jamie Bernardi: Important underscussed point on the OpenAI $100bn deal: money is not coming from the USG.

Trump is announcing a private deal, whilst promising to make “emergency declarations” to allow Stargate to generate its own electricity (h/t @nytimes).

Musk says 100bn not yet raised.

Peter Wildeford: Once upon a time words had meaning.

Jake Perry: I’m still not clear why this was announced at the White House at all.

Peter Wildeford: Trump has a noted history of announcing infrastructure projects that were already in progress – he did this a lot in his first term.

Jacques: At least we’ll all be paperclipped with a USA flag engraved on it.

Trump says that it is all about him, of course:

Donald Trump: This monumental undertaking is a resounding declaration of confidence in America’s potential under a new president.

The president said Stargate would create 100,000 jobs “almost immediately” and keep “the future of technology” in America.

I presume that in addition to completely missing the point, this particular jobs claim is, technically speaking, not true. But numbers don’t have to be real in politics. And of course, if this is going to create those jobs ‘almost immediately’ it had to have been in the works for a long time.

Shakeel: I can’t get over the brazen, brazen lie from Altman here, saying “We couldn’t do this without you, Mr President”.

You were already doing it! Construction started ages ago!

Just a deeply untrustworthy man — you can’t take anything he says at face value.

Dylan Matthews: Everything that has happened since the board fired him has 100% vindicated their view of him as deeply dishonest and unreliable, and I feel like the popular understanding of that incident hasn’t updated from “this board sure is silly!”

[Chubby: Sam Altman: hype on twitter is out of control. Everyone, chill down.

Also Sam Altman: anyways, let’s invest half a trillion to build a digital god and cure cancer one and for all. Oh, and my investors just said that AGI comes very, very soon and ASI will solve any problem mankind faces.

But everyone, calm down 100x]

I agree with Dylan Matthews that the board’s assessment of Altman as deeply dishonest and unreliable has very much been vindicated, and Altman’s actions here only confirm that once again. But that doesn’t mean that Trump has nothing to do with the fact that this project is going forward, with this size.

So how much does this project depend on Trump being president instead of Harris?

I think the answer is actually a substantial amount.

In order to build AI infrastructure in America, you need three things.

  1. You need demand. Check.

  2. You need money. Check, or at least check in the mail.

  3. You need permission to actually build it. Previously no check. Now, maybe check?

Masayoshi Sun: Mr. President, last month I came to celebrate your winning and promised $100B. And you told me go for $200B. Now I came back with $500B. This is because as you say, this is the beginning of the Golden Age. We wouldn’t have decided this unless you won.

Sam Altman: The thing I really deeply agree with the president on is, it is wild how difficult it has become to build things in the United States. Power plants, data centres, any of that kind of stuff.

Does Sun have many good reasons to pretend that this is all because of Trump? Yes, absolutely. He would find ways to praise the new boss either way. But I do think that Trump mattered here, even if you don’t think that there is anything corrupt involved in all this.

Look at Trump’s executive orders, already signed, about electrical power plants and transmission lines being exempt from NEPA, and otherwise being allowed to go forwards. They can expect more similar support in the future, if they run into roadblocks, and fewer other forms of regulatory trouble and everything bagel requirements across the board.

Also, I totally believe that Sun came to Trump and promised $100 billion, and Trump said go for $200 billion, and Sun now is at $500 billion, and I think that plausibly created a lot of subsequent investment. It may sound stupid, but that’s Grade-A handling of Masayoshi Sun, and exactly within Trump’s wheelhouse. Tell the man who thinks big he’s not thinking big enough. Just keep him ramping up. Don’t settle for a big win when you can go for an even bigger win. You have to hand it to him.

It is so absurd that these people, with a straight face, decided to call this Stargate.

They wanted to call it the Enterprise, but their lawyers wouldn’t let them.

Was SkyNet still under copyright?

Agus: Ah, yes. Of course we’re naming this project after the fictitious portal through which several hostile alien civilizations attempted to invade and destroy Earth.

I just hope we get the same amount of completely unrealistic plot armor that protected Stargate Command in S.G.1.

Roon: the Stargate. blasting a hole into the Platonic realm to summon angels. First contact with alien civilizations.

Canonically, the Stargates are sometimes used by dangerous entities to harm us, but once humanity deals with that, they end up being quite useful.

Zvi Mowshowitz: Guy who reads up on the canonical history of Stargate and thinks, “Oh, all’s well that ends well. Let’s try that plan.”

Roon: 🤣

Is this where I give you 10,000 words on the history of Stargate SG-1 and Stargate Atlantis and all the different ways Earth and often also everyone else would have been enslaved or wiped out if it wasn’t for narrative causality and plot armor, and what would have been reasonable things to do in that situation?

No, and I am sad about that, despite yes having watched all combined 15 seasons, because alas we do not currently have that kind of time. Maybe later I’ll be able to spend a day doing that, it sounds like fun.

But in brief about that Stargate plan. Was it a good plan? What were the odds?

As is pointed out in the thread (minor spoilers for the end of season 1), the show actually answers this question, as there is crossover between different Everett branches, and we learn that even relatively early on – before most of the different things that almost kill us have a chance to almost kill us – that most branches have already lost. Which was one of the things that I really liked about the show, that it realized this. The thread also includes discussions of things like ‘not only did we not put a nuclear bomb by the Stargate and use a secondary gate to disguise our location, we wore Earth’s gate code on our fing uniforms.’

To be fair, there is a counterargument, which is that (again, minor spoilers) humanity was facing various ticking clocks. There was one in particular that was ticking in ways Earth did not cause, and then there were others that were set in motion rapidly once we had a Stargate program, and in general we were on borrowed time. So given what was happening we had little choice but to go out into the galaxy and try to develop superior technology and find various solutions before time ran out on us, and it would have been reasonable to expect we were facing a ticking clock in various ways given what Earth knew at the time.

There’s also the previous real life Project Stargate, a CIA-DIA investigation of the potential for psychic phenomena. That’s… not better.

There are also other ways to not be thrilled by all this.

Justin Amash: The Stargate Project sounds like the stuff of dystopian nightmares—a U.S. government-announced partnership of megacorporations “to protect the national security of America and its allies” and harness AGI “for the benefit of all of humanity.” Let’s maybe take a beat here.

Taking a beat sounds like a good idea.

What does Trump actually think AI can do?

Samuel Hammond: Trump seems under the impression that ASI is just a way to cure diseases and not an ultraintelligent digital lifeform with autonomy and self-awareness. Sam’s hesitation before answering speaks volumes.

That’s not how I view the clip at the link. Trump is selling the project. It makes sense to highlight medical advances, which are a very real and valuable upside. It certainly makes a lot more sense than highlighting job creation.

Altman I don’t see hesitating, I see him trying to be precise while also going with the answer, and I don’t like his previous emphasis on jobs (again, no doubt, following Trump’s and his political advisor’s lead) but on the medical question I think he does well and it’s not obvious what a better answer would have been.

The hilarious part of this is the right wing faction that says ‘you want to use this to make mRNA vaccines, wtf I hate AI now’ and trying to figure out what to do with people whose worldviews are that hopelessly inverted.

That moment when you say ‘look at how this could potentially cure cancer’ and your hardcore supporters say ‘And That’s Terrible.’

And also when you somehow think ‘Not Again!’

Eliezer Yudkowsky: Welp, looks like Trump sure is getting backlash to the Stargate announcement from many MAGAers who are outraged that AGIs might develop mRNA vaccines and my fucking god it would be useless to evacuate to Mars but I sure see why Elon wants to

To people suggesting that I ought to suck up to that crowd: On my model of them, they’d rather hear me say “Fyou lunatics, now let’s go vote together I guess” than have me pretend to suck up to them.

Like, on my model, that crowd is deadly tired of all the BULLSHIT and we in fact have that much in common and I bet I can get further by not trying to feed them any BULLSHIT.

There is a deep sense in which it is more respectful to someone as a human being to say, “I disagree with your fing lunacy. Allies?” then to smarm over to them and pretend to agree with them. And I think they know that.

RPotluck: The MAGAsphere doesn’t love you and it doesn’t hate you, but you’re made of arguments the MAGAsphere can use to build the wall.

There’s a certain kind of bullshit that these folks and many other folks are deeply tired of hearing. This is one of those places where I very much agree that it does hurt to suck up to the boss, both because the boss will see through it and because the whole strategy involves not doing things like that, and also have you seen or heard the boss.

My prediction and hope is that we will continue to see those worried about AI killing everyone continue to not embrace these kinds of crazy arguments of convenience. That doesn’t mean not playing politics at all or being some sort of suicidal purist. It does mean we care about whether our arguments are true, rather than treating them as soldiers for a cause.

Whereas we have learned many times, most recently with the fight over SB 1047 and then the latest round of jingoism, that many (#NotAllUnworried!) of those who want to make sure others do not worry about AI killing everyone, or at least want to ensure that creating things smarter than humans faces less regulatory barriers than a barber shop, care very little whether the arguments made on their behalf, by themselves or by others, are true or correspond to physical reality. They Just Didn’t Care.

The flip side is the media, which is, shall we say, not situationally aware.

Spencer Schiff: The AGI Manhattan Project announcement was followed by half an hour of Q&A. Only one reporter asked a question about it. WHAT THE FUCK! This is insane. The mainstream media is completely failing to convey the gravity of what’s happening to the general public.

As noted elsewhere I don’t think this merits ‘Manhattan Project’ for various reasons but yes, it is kind of weird to announce a $500 billion investment in artificial general intelligence and then have only one question about it in a 30 minute Q&A.

I’m not saying that primarily from an existential risk perspective – this is far more basic even than that. I’m saying, maybe this is a big deal that all this is happening, maybe ask some questions about it?

Remember when Altman was talking about how we have to build AGI now because he was worried about a compute overhang? Yes, well.

Between the $500 billion of Stargate, the full-on jingoistic rhetoric from all sides including Anthropic, and the forcing function of DeepSeek with v3 and r1, it is easy to see how one could despair over our prospects for survival.

Unless something changes, we are about to create smarter than human intelligence, entities more capable and competitive than we are across all cognitive domains, and we are going to do so as rapidly as we can and then put them in charge of everything, with essentially zero margin to ensure that this goes well despite it obviously by default getting everyone killed.

Even if we are so fortunate that the technical and other barriers in front of us are highly solvable, that is exactly how we get everyone killed anyway.

Holly Elmore: I am so, so sad today. Some days the weight of it all just hits me. I want to live my life with my boyfriend. I want us to have kids. I want love and a full life for everyone. Some days the possibility that that will all be taken away is so palpable, and grief is heavy.

I’m surprised how rarely I feel this way, given what I do. I don’t think it’s bad to feel it all sometimes. Puts you in touch with what you’re fighting for.

I work hard to find the joy and the gallows humor in it all, to fight the good fight, to say the odds are against us and the situation is grim, sounds like fun. One must imagine Buffy at the prom, and maintain Scooby Gang Mindset. Also necessary is the gamer mindset, which says you play to win the game, and in many ways it’s easiest to play your best game with your back against the wall.

And in a technical sense, I have hope that the solutions exist, and that there are ways to at least give ourselves a fighting chance.

But yeah, weeks like this do not make it easy to keep up hope.

Harlan Stewart: If the new $500b AI infrastructure thing ever faces a major scandal, we’ll unfortunately be forced to call it Stargategate

Discussion about this post

Stargate AI-1 Read More »

way-more-game-makers-are-working-on-pc-titles-than-ever,-survey-says

Way more game makers are working on PC titles than ever, survey says

Four out of five game developers are currently working on a project for the PC, a sizable increase from 66 percent of developers a year ago. That’s according to Informa’s latest State of the Game Industry survey, which partnered with Omdia to ask over 3,000 game industry professionals about their work in advance of March’s Game Developers Conference.

The 80 percent of developers working on PC projects in this year’s survey is by far the highest mark for any platform dating back to at least 2018, when 60 percent of surveyed developers were working on a PC game. In the years since, the ratio of game developers working on the PC has hovered between 56 and 66 percent before this year’s unexpected jump. The number of game developers saying they were interested in the PC as a platform also increased substantially, from 62 percent last year to 74 percent this year.

While the PC has long been the most popular platform in this survey, the sudden jump in the last year was rather large.

Credit: Kyle Orland / Informa

While the PC has long been the most popular platform in this survey, the sudden jump in the last year was rather large. Credit: Kyle Orland / Informa

The PC has long been the most popular platform for developers to work on in the annual State of the Game Industry survey, easily outpacing consoles and mobile platforms that generally see active work from anywhere between 12 to 36 percent of developer respondents, depending on the year. In its report, Informa notes this surge as a “passion for PC development explod[ing]” among developers, and mentions that while “PC has consistently been the platform of choice… this year saw its dominance increase even more.”

The increasing popularity of PC gaming among developers is also reflected in the number of individual game releases on Steam, which topped out at a record of 18,974 individual titles for 2024, according to SteamDB. That record number was up over 32 percent from 2023, which was up from just under 16 percent from 2022 (though many Steam games each year were “Limited Games” that failed to meet Valve’s minimum engagement metrics for Badges and Trading Cards).

The number of annual Steam releases also points to increasing interest in the platform.

The number of annual Steam releases also points to increasing interest in the platform. Credit: SteamDB

The Steam Deck effect?

While it’s hard to pinpoint a single reason for the sudden surge in the popularity of PC game development, Informa speculates that it’s “connected to the rising popularity of Valve’s Steam Deck.” While Valve has only officially acknowledged “multiple millions” in sales for the portable hardware, GameDiscoverCo analyst Simon Carless estimated that between 3 million and 4 million Steam Deck units had been sold by October 2023, up significantly from reports of 1 million Deck shipments in October 2022.

Way more game makers are working on PC titles than ever, survey says Read More »

trump-can-save-tiktok-without-forcing-a-sale,-bytedance-board-member-claims

Trump can save TikTok without forcing a sale, ByteDance board member claims

TikTok owner ByteDance is reportedly still searching for non-sale options to stay in the US after the Supreme Court upheld a national security law requiring that TikTok’s US operations either be shut down or sold to a non-foreign adversary.

Last weekend, TikTok briefly went dark in the US, only to come back online hours later after Donald Trump reassured ByteDance that the US law would not be enforced. Then, shortly after Trump took office, he signed an executive order delaying enforcement for 75 days while he consulted with advisers to “pursue a resolution that protects national security while saving a platform used by 170 million Americans.”

Trump’s executive order did not suggest that he intended to attempt to override the national security law’s ban-or-sale requirements. But that hasn’t stopped ByteDance, board member Bill Ford told World Economic Forum (WEF) attendees, from searching for a potential non-sale option that “could involve a change of control locally to ensure it complies with US legislation,” Bloomberg reported.

It’s currently unclear how ByteDance could negotiate a non-sale option without facing a ban. Joe Biden’s extended efforts through Project Texas to keep US TikTok data out of China-controlled ByteDance’s hands without forcing a sale dead-ended, prompting Congress to pass the national security law requiring a ban or sale.

At the WEF, Ford said that the ByteDance board is “optimistic we will find a solution” that avoids ByteDance giving up a significant chunk of TikTok’s operations.

“There are a number of alternatives we can talk to President Trump and his team about that are short of selling the company that allow the company to continue to operate, maybe with a change of control of some kind, but short of having to sell,” Ford said.

Trump can save TikTok without forcing a sale, ByteDance board member claims Read More »

wine-10.0-brings-arm-windows-apps-to-linux,-still-is-not-an-emulator

Wine 10.0 brings Arm Windows apps to Linux, still is not an emulator

The open source Wine project—sometimes stylized WINE, for Wine Is Not an Emulator—has become an important tool for companies and individuals who want to make Windows apps and games run on operating systems like Linux or even macOS. The CrossOver software for Mac and Windows, Apple’s Game Porting Toolkit, and the Proton project that powers Valve’s SteamOS and the Steam Deck are all rooted in Wine, and the attention and resources put into the project in recent years have dramatically improved its compatibility and usefulness.

Yesterday, the Wine project announced the stable release of version 10.0, the next major version of the compatibility layer that is not an emulator. The headliner for this release is support for ARM64EC, the application binary interface (ABI) used for Arm apps in Windows 11, but the release notes say that the release contains “over 6,000 individual changes” produced over “a year of development effort.”

ARM64EC allows developers to mix Arm and x86-compatible code—if you’re making an Arm-native version of your app, you can still allow the use of more obscure x86-based plugins or add-ons without having to port everything over at once. Wine 10.0 also supports ARM64X, a different type of application binary file that allows ARM64EC code to be mixed with older, pre-Windows 11 ARM64 code.

Wine’s ARM64EC support does have one limitation that will keep it from working on some prominent Arm Linux distributions, at least by default: the release notes say it “requires the system page size to be 4K, since that is what the Windows ABI specifies.” Several prominent Linux-on-Arm distributions default to a 16K page size because it can improve performance—when page sizes are smaller, you need more of them, and managing a higher number of pages can introduce extra CPU overhead.

Asahi Linux, the Fedora-based distribution that is working to bring Linux to Apple Silicon Macs, uses 16K pages because that’s all Apple’s processors support. Some versions of the Raspberry Pi OS also default to a 16K page size, though it’s possible to switch to 4K for compatibility’s sake. Given that the Raspberry Pi and Asahi Linux are two of the biggest Linux-on-Arm projects going right now, that does at least somewhat limit the appeal of ARM64EC support in Wine. But as we’ve seen with Proton and other successful Wine-based compatibility layers, laying the groundwork now can deliver big benefits down the road.

Wine 10.0 brings Arm Windows apps to Linux, still is not an emulator Read More »

samsung’s-galaxy-s25-event-was-an-ai-presentation-with-occasional-phone-hardware

Samsung’s Galaxy S25 event was an AI presentation with occasional phone hardware

Samsung announced the Galaxy S25, S25+, and S25 Ultra at its Unpacked event today. What is different from last year’s models? With the phones themselves, not much, other than a new chipset and a wide camera. But pure AI optimism? Samsung managed to pack a whole lot more of that into its launch event and promotional materials.

The corners on the S25 Ultra are a bit more rounded, the edges are flatter, and the bezels seem to be slightly thinner. The S25 and S25+ models have the same screen size as the S24 models, at 6.2 and 6.7 inches, respectively, while the Ultra notches up slightly from 6.8 to 6.9 inches.

Samsung’s S25 Ultra, in titanium builds colored silver blue, black, gray, and white silver.

Credit: Samsung

Samsung’s S25 Ultra, in titanium builds colored silver blue, black, gray, and white silver. Credit: Samsung

The S25 Ultra, starting at $1,300, touts a Snapdragon 8 Elite processor, a new 50-megapixel ultra-wide lens, and what Samsung claims is improved detail in software-derived zoom images. It comes with the S Pen, a vestige of the departed Note line, but as The Verge notes, there is no Bluetooth included, so you can’t pull off hand gestures with the pen off the screen or use it as a quirky remote camera trigger.

Samsung’s S25 Plus phones, in silver blue, navy, and icy blue.

Credit: Samsung

Samsung’s S25 Plus phones, in silver blue, navy, and icy blue. Credit: Samsung

It’s much the same with the S25 and S25 Plus, starting at $800. The base models got an upgrade to a default of 12GB of RAM. The displays, cameras, and general shape and build are the same. All the Galaxy devices released in 2025 have Qi2 wireless charging support—but not by default. You’ll need a “Qi2 Ready” magnetic case to get a sturdy attachment and the 15 W top charging speed.

One thing that hasn’t changed, for the better, is Samsung’s recent bump up in longevity. Each Galaxy S25 model gets seven years of security updates and seven of OS upgrades, which matches Google’s Pixel line in number of years.

Side view of the Galaxy S25 Edge, which is looking rather thin. Samsung

At the very end of Samsung’s event, for less than 30 seconds, a “Galaxy S25 Edge” was teased. In a mostly black field with some shiny metal components, Samsung seemed to be teasing the notably slimmer variant of the S25 that had been rumored. The same kinds of leaks about an “iPhone Air” have been circulating. No details were provided beyond its name, and a brief video suggesting its svelte nature.

Samsung’s Galaxy S25 event was an AI presentation with occasional phone hardware Read More »

on-deepseek’s-r1

On DeepSeek’s r1

r1 from DeepSeek is here, the first serious challenge to OpenAI’s o1.

r1 is an open model, and it comes in dramatically cheaper than o1.

People are very excited. Normally cost is not a big deal, but o1 and its inference-time compute strategy is the exception. Here, cheaper really can mean better, even if the answers aren’t quite as good.

You can get DeepSeek-r1 on HuggingFace here, and they link to the paper.

The question is how to think about r1 as it compares to o1, and also to o1 Pro and to the future o3-mini that we’ll get in a few weeks, and then to o3 which we’ll likely get in a month or two.

Taking into account everything I’ve seen, r1 is still a notch below o1 in terms of quality of output, and further behind o1 Pro and the future o3-mini and o3.

But it is a highly legitimate reasoning model where the question had to be asked, and you absolutely cannot argue with the price, which is vastly better.

The best part is that you see the chain of thought. For me that provides a ton of value.

r1 is based on DeepSeek v3. For my coverage of v3, see this post from December 31, which seems to have stood up reasonably well so far.

This post has 4 parts: First in the main topic at hand, I go over the paper in Part 1, then the capabilities in Part 2.

Then in Part 3 I get into the implications for policy and existential risk, which are mostly exactly what you would expect, but we will keep trying.

Finally we wrap up with a few of the funniest outputs.

  1. Part 1: RTFP: Read the Paper.

  2. How Did They Do It.

  3. The Aha Moment.

  4. Benchmarks.

  5. Reports of Failure.

  6. Part 2: Capabilities Analysis

  7. Our Price Cheap.

  8. Other People’s Benchmarks.

  9. r1 Makes Traditional Silly Mistakes.

  10. The Overall Vibes.

  11. If I Could Read Your Mind.

  12. Creative Writing.

  13. Bring On the Spice.

  14. We Cracked Up All the Censors.

  15. Switching Costs Are Low In Theory.

  16. The Self-Improvement Loop.

  17. Room for Improvement.

  18. Part 3: Where Does This Leave Us on Existential Risk?

  19. The Suicide Caucus.

  20. v3 Implies r1.

  21. Open Weights Are Unsafe And Nothing Can Fix This.

  22. So What the Hell Should We Do About All This?

  23. Part 4: The Lighter Side.

They call it DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.

The claim is bold: A much cheaper-to-run open reasoning model as good as o1.

Abstract: We introduce our first-generation reasoning models, DeepSeek-R1-Zero and DeepSeek-R1. DeepSeek-R1-Zero, a model trained via large-scale reinforcement learning (RL) without super vised fine-tuning (SFT) as a preliminary step, demonstrates remarkable reasoning capabilities.

Through RL, DeepSeek-R1-Zero naturally emerges with numerous powerful and intriguing reasoning behaviors.

However, it encounters challenges such as poor readability, and language mixing.

To address these issues and further enhance reasoning performance, we introduce DeepSeek-R1, which incorporates multi-stage training and cold-start data before RL. DeepSeek R1 achieves performance comparable to OpenAI-o1-1217 on reasoning tasks.

To support the research community, we open-source DeepSeek-R1-Zero, DeepSeek-R1, and six dense models (1.5B, 7B, 8B, 14B, 32B, 70B) distilled from DeepSeek-R1 based on Qwen and Llama.

They also claim substantial improvement over state of the art for the distilled models.

They are not claiming to be as good as o1-pro, but o1-pro has very large inference costs, putting it in a different weight class. Presumably one could make an r1-pro, if one wanted to, that would improve upon r1. Also no doubt that someone will want to.

They trained R1-Zero using pure self-evaluations via reinforcement learning, starting with DeepSeek-v3-base and using GRPO, showing that the cold start data isn’t strictly necessary.

To fix issues from there including readability and language mixing, however, they then used a small amount of cold-start data and a multi-stage training pipeline, and combined this with supervised data for various domains later in the process, to get DeepSeek-R1. In particular they do not use supervised fine-tuning (SFT) as a preminimary step, only doing some SFT via rejection sampling later in the process, and especially to train the model on non-reasoning tasks like creative writing.

They use both an accuracy reward and a format reward to enforce the and labels, but don’t evaluate the thinking itself, leaving it fully unconstrained, except that they check if the same language is used throughout to stamp out language mixing. Unlike o1, we get to see inside that chain of thought (CoT).

They then distilled this into several smaller models.

More details and various equations and such can be found in the paper.

Over time this caused longer thinking time, seemingly without limit:

Both scales are linear and this graph looks very linear. I presume it would have kept on thinking for longer if you gave it more cycles to learn to do that.

I notice that in 2.3.4 they do additional reinforcement learning for helpfulness and harmlessness, but not for the third H: honesty. I worry that this failure is primed to bite us in the collective ass in various ways, above and beyond all the other issues.

wh has a thread with a parallel similar explanation, with the same takeaway that I had. This technique was simple, DeepSeek and OpenAI both specialize in doing simple things well, in different ways.

Yhprum also has a good thread on how they did it, noting how they did this in stages to address particular failure modes.

Contra Jim Fan, There is one thing missing from the paper? Not that I fault them.

1a3orn: The R1 paper is great, but includes ~approximately nothing~ about the details of the RL environments.

It’s worth noticing. If datasets were king for the past three years, the RL envs probably will be for the next few.

This was striking to a lot of people and also stuck out to Claude unprompted, partly because it’s a great name – it’s an aha moment when the model went ‘aha!’ and the researchers watching it also went ‘aha!’ So it’s a very cool framing.

During this phase, DeepSeek-R1-Zero learns to allocate more thinking time to a problem by reevaluating its initial approach. This behavior is not only a testament to the model’s growing reasoning abilities but also a captivating example of how reinforcement learning can lead to unexpected and sophisticated outcomes.

This moment is not only an “aha moment” for the model but also for the researchers observing its behavior. It underscores the power and beauty of reinforcement learning: rather than explicitly teaching the model on how to solve a problem, we simply provide it with the right incentives, and it autonomously develops advanced problem-solving strategies.

It’s cool to see it happen for real, and I’m obviously anchored by the result, but isn’t this to be expected? This is exactly how all of this works, you give it the objective, it figures out on its own how to get there, and given it has to think in tokens and how thinking works, and that the basic problem solving strategies are all over its original training data, it’s going to come up with all the usual basic problem solving strategies.

I see this very similarly to the people going ‘the model being deceptive, why I never, that must be some odd failure mode we never told it to do that, that doesn’t simply happen.’ And come on, this stuff is ubiquitous in humans and in human written content, and using it the ways it is traditionally used is going to result in high rewards and then you’re doing reinforcement learning. And then you go acting all ‘aha’?

The cocky bastards say in 2.4 (I presume correctly) that if they did an RL stage in the distillations it would improve performance, but since they were only out to demonstrate effectiveness they didn’t bother.

As always, benchmarks are valuable information especially as upper bounds, so long as you do not treat them as more meaningful than they are, and understand the context they came from.

Note that different graphs compare them to different versions of o1 – the one people currently used is called o1-1217.

The Qwen versions are clearly outperforming the Llama versions on the benchmarks, although as usual one would want to double check that in practice.

I want to give thanks to DeepSeek for section 4.2, on Unsuccessful Attempts. They tried Process Reward Model (PRM), and Monte Carlo Tree Search (MCTS), and explained various reasons why both ultimately didn’t work.

More reports should do this, and doing this is substantially to their credit.

Sasha Rush: Post-mortem after Deepseek-r1’s killer open o1 replication.

We had speculated 4 different possibilities of increasing difficulty (G&C, PRM, MCTS, LtS). The answer is the best one! It’s just Guess and Check.

There’s also the things they haven’t implemented yet. They aren’t doing function calling, multi-turn, complex roleplaying or json output. They’re not optimizing for software engineering.

I buy the claim by Teortaxes that these are relatively easy things to do, they simply haven’t done them yet due to limited resources, mainly compute. Once they decide they care enough, they’ll come. Note that ‘complex role-playing’ is a place it’s unclear how good it can get, and also that this might sound like a joke but it is actually highly dangerous.

Here Lifan Yuan argues that the noted PRM failures can be addressed.

Given the league that r1 is playing in, it is dirt cheap.

When they say it is 30 times cheaper than o1, story largely checks out: o1 is $15/$60 for a million input and output tokens, and r1 varies since it is open but is on the order of $0.55/$2.19.

Claude Sonnet is $3/$15, which is a lot more per token, but notice the PlanBench costs are actually 5x cheaper than r1, presumably because it used a lot less tokens (and also didn’t get good results in that case, it’s PlanBench and only reasoning models did well).

The one catch is that with r1 you do have to pay for the tokens. I asked r1 to estimate what percentage of tokens are in the CoT, and it estimated 60%-80%, with more complex tasks using relatively more CoT tokens, in an answer that was roughly 75% within the CoT.

If you only care about the final output, then that means this is more like 10 times cheaper than o1 rather than 30 times cheaper. So it depends on whether you’re making use of the CoT tokens. As a human, I find them highly useful (see the section If I Could Read Your Mind), but if I was using r1 at scale and no human was reading the answers, it would be a lot less useful – although I’d be tempted to have even other AIs be analyzing the CoT.

The web interface is both fast and very clean, it’s a great minimalist approach.

Gallabytes: the DeepSeek app is so much better implemented than the OpenAI one, too. None of these frequent crashes, losing a whole chain-of-thought (CoT), occur. I can ask it a question, then tab away while it is thinking, and it does not break.

Edit: It has good PDF input, too? Amazing.

Another issue is IP and privacy – you might not trust DeepSeek. Which indeed I wouldn’t, if there were things I actively didn’t want someone to know.

Gallabytes: is anyone hosting r1 or r1-zero with a stronger privacy policy currently? would love to use them for work but wary about leaking ip.

David Holz: Should we just self host?

Gallabytes: In principle yes but it seems expensive – r1 is pretty big. and I’d want a mobile app, not sure how easy that is to self host.

Xeophon: OpenWebUI if you are okay with a (mobile) browser.

Gallabytes: as long as it doesn’t do the stupid o1 thing where I have to keep it in the foreground to use it then it’ll still be a huge improvement over the chatgpt app.

Xeophon: Fireworks has R1 for $8/M

Running it yourself is a real option.

Awni Hannun: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed.

Getting close to open-source O1, at home, on consumer hardware.

With mlx.distributed and mlx-lm, 3-bit quantization (~4 bpw)

Seth Rose: I’ve got a Macbook Pro M3 (128GB RAM) – what’s the “best” deepseek model I can run using mlx with about 200 GB of storage?

I attempted to run the 3-bit DeepSeek R1 version but inadvertently overlooked potential storage-related issues. 😅

Awni Hannun: You could run the Distill 32B in 8-bit no problem: mlx-community/DeepSeek-R1-Distill-Qwen-32B-MLX-8Bit

If you want something faster try the 14B or use a lower precision.

The 70B in 4-6 bit will also run pretty well, and possibly even in 8-bit though it will be slow. Those quants aren’t uploaded yet though

With the right web interface you can get at least 60 tokens per second.

Teortaxes also reports that kluster.ai is offering overnight tokens at a discount.

People who have quirky benchmarks are great, because people aren’t aiming at them.

Xoephon: I am shocked by R1 on my personal bench.

This is the full eval set, it completely crushes the competition and is a whole league on its own, even surpassing o1-preview (which is omitted from the graph as I ran it only twice, it scored 58% on avg vs. 67% avg. R1).

Holy shit what the f, r1 beats o1-preview on my bench.

Kartik Valmeekam: 📢 DeepSeek-R1 on PlanBench 📢

DeepSeek-R1 gets similar performance as OpenAI’s o1 (preview)—achieving 96.6% on Blocksworld and 39.8% on its obfuscated version, Mystery BW.

The best part?

⚡It’s 21x cheaper than o1-preview, offering similar results at a fraction of the cost!

Note the relative prices. r1 is a little over half the price of o1-mini in practice, 21x cheaper than o1-preview, but still more expensive than the non-reasoning LLMs. Of course, it’s PlanBench, and the non-reasoning LLMs did not do well.

Steve Hsu gives a battery of simple questions, r1 is first to get 100%.

Havard Ihle reports top marks on WeirdML (he hasn’t tested o1 or o1 pro).

Bayram Annakov asks it to find 100 subscription e-commerce businesses, approves.

It is a grand tradition, upon release of a new model, to ask questions that are easy for humans, but harder for AIs, thus making the AI look stupid.

The classic way to accomplish this is to ask a question that is intentionally similar to something that occurs a lot in the training data, except the new version is different in a key way, and trick the AI into pattern matching incorrectly.

Quintin Pope: Still tragically fails the famous knights and knights problem:

Alex Mallen: This doesn’t look like a failure of capability. It looks like the model made the reasonable guess that you made a typo.

Quintin Pope: Prompt includes both “twin honest gatekeepers” and “never lies”. Combined, it’s not plausibly a typo.

Alex Mallen: Eh someone I talked to yesterday did something similar by mistake. But I maybe you’d like LMs to behave more like programs/tools that do literally what you ask. Seems reasonable.

r1 notices that this is different from the original question, and also notices that the version it has been given here is deeply stupid, since both gatekeepers are honest, also as a bonus both of them answer.

Notice that Quintin is lying to r1 – there is no ‘famous twin honest gatekeepers’ problem, and by framing it as famous he implies it can’t be what he’s describing.

So essentially you have three possibilities. Either Quintin is fing with you, or he is confused how the question is supposed to go, or there somehow really is this other ‘famous gatekeepers’ problem.

Also note that r1 says ‘misheard’ rather than ‘misread’ or ‘the user misstated.’ Huh.

Quintin’s argument is that it obviously can’t be a typo, it should answer the question.

I think the correct answer, both as a puzzle or in real life, is to look for a solution that works either way. As in, if you only get the one answer from the guard, you should be fine with that even if you don’t know if you are dealing with two honest guards or with one honest guard and one dishonest guard.

Since you can use as many conditionals in the question as you like, and the guards in all versions know whether the other guard tells the truth or not, this is a totally solvable problem.

Also acceptable is ‘as written the answer is you just ask which door leads to freedom, but are you sure you told me that correctly?’ and then explain the normal version.

This one is fun, Trevor reports r1 got it right, but when I tried it very much didn’t.

alz zyd: Game theory puzzle:

There are 3 people. Each person announces an integer. The smallest unique integer wins: e.g. if your opponents both pick 1, you win with any number. If all 3 pick the same number, the winner is picked randomly

Question: what’s the Nash equilibrium?

Trevor: interestingly o1-pro didn’t get it right on any of the 3 times i tried this, while the whale (r1) did!

I fed this to r1 to see the CoT and verify. It uses the word ‘wait’ quite a lot. It messes up steps a lot. And it makes this much harder than it needs to be – it doesn’t grok the situation properly before grasping at things or try to simplify the problem, and the whole thing feels (and is) kind of slow. But it knows to check its answers, and notices it’s wrong. But then it keeps using trial and error.

Then it tries to assume there is exponential dropping off, without understanding why, and notices it’s spinning its wheels. It briefly goes into speaking Chinese. Then it got it wrong, and then when I pointed out the mistake it went down the same rabbit holes again and despairs to the same wrong answer. On the third prompt it got the answer not quite entirely wrong but was explicitly just pattern match guessing.

That matches the vibes of this answer, of the Monty Hall problem with 7 doors, of which Monty opens 3 – in the end he reports r1 got it right, but it’s constantly second guessing itself in a way that implies that it constantly makes elementary mistakes in such situations (thus the checking gets reinforced to this degree), and it doesn’t at any point attempt to conceptually grok the parallel to the original version.

I’ve seen several people claim what V_urb does here, that o1 has superior world knowledge to r1. So far I haven’t had a case where that came up.

A fun set of weird things happening from Quintin Pope.

The vibes on r1 are very good.

Fleeting Bits: The greatest experience I have had with a model; it is a frontier model that is a joy to interact with.

Leo Abstract: My strange, little, idiosyncratic tests of creativity, it has been blowing out of the water. Really unsettling how much better it is than Claude.

It’s giving big Lee Sedol vibes, for real; no cap.

Most unsettling launch so far. I am ignorant about benchmarks, but the way it behaves linguistically is different and better. I could flirt with the cope that it’s just the oddness of the Chinese-language training data peeking through, but I doubt this.

Those vibes seem correct. The model looks very good. For the price, it’s pretty sweet.

One must still be careful not to get carried away.

Taelin: ironically enough, DeepSeek’s r1 motivated me try OpenAI’s o1 Pro on something I didn’t before, and I can now confidently state the (obvious?) fact that o1 is on a league of its own, and whoever thinks AGI isn’t coming in 2-3 years is drinking from the most pure juice of denial

Teortaxes: I agree that o1, nevermind o1 pro is clearly substantially ahead of r1. What Wenfeng may urgently need for R2 is not just GPUs but 1000 more engineers. Not geniuses and wizards. You need to accelerate the data flywheel by creating diverse verifiable scenario seeds and filters.

Gallabytes: what problems are you giving it where o1 is much better than r1?

Teortaxes: I mainly mean iterative work. r1 is too easily sliding into “but wait, user [actually itself] previously told me” sort of nonsense.

I echo Teortaxes that r1 is just so much more fun. The experience is different seeing the machine think. Claude somewhat gives you that, but r1 does it better.

Janus has been quiet on r1 so far, but we do have the snippet that ‘it’s so fed.’ They added it to the server, so we’ll presumably hear more at a later date.

Read the chain of thought. Leave the output.

That’s where I’m at with r1. If I’m actively interested in the question and how to think about it, rather than looking for a raw answer, I’d much rather read the thinking.

Here Angelica chats with r1 about finding areas for personal growth, notices that r1 is paying attention and drawing correct non-obvious inferences that improve its responses, and gets into a meta conversation, leaving thinking this is the first AI she thinks of as thoughtful.

I too have found it great to see the CoT, similar to this report from Dominik Peters or this from Andres Sandberg, or papaya noticing they can’t get enough.

It’s definitely more helpful to see the CoT than the answer. It might even be more helpful per token to see the CoT, for me, than the actual answers – compare to when Hunter S. Thompson sent in his notes to the editor because he couldn’t write a piece, and the editor published the notes. Or to how I attempt to ‘share my CoT’ in my own writing. If you’re telling me an answer, and I know how you got there, that gives me valuable context to know how to think about that answer, or I can run with the individual thoughts, which was a lot of what I wanted anyway.

Over time, I can learn how you think. And I can sculpt a better prompt, or fix your mistakes. And you can see what it missed. It also can help you learn to think better.

My early impressions of its thought is that I am… remarkably comfortable with it. It feels very ‘normal,’ very human, very straightforward. It seems both like it isn’t an idiot, and also isn’t anything special. It thinks, and it knows things.

I don’t know if this is a good chain of thought and I’m thinking out loud here, but this also tentatively updates me towards this process not scaling that far purely with additional compute? We are seeing the model roughly ‘think like a regular person’ using reasoning techniques within the training distribution in ways you’d expect to commonly find, aided by ability to do a lot of this quickly, having superhuman access to information and so on. If this was about to scale beyond that, I’d start to see things that looked like a level beyond that, or something? But I’m not sure. The other uncertainty is, maybe there is no next level, and maybe doing a lot of these simple things well is enough.

It is a shame that it shortens timelines, but it’s not obvious if it makes us on net more or less likely to die.

Historically we have not been impressed by LLM creative writing, including o1’s.

r1 is given the assignment of a short story of the singularity, inspired by Nick Land. And it’s… a series of words that vibe with that assignment?

John Pressman: R1 is going to be so much fun holy shit.

I love that you can see the thought process here. And I love how r1 just goes for it.

It’s like the world’s worst Hollywood hack going over all the amazing different stuff to jam in there and then having sentences match all these various things.

I notice I very much have various ugh fields and voices demanding things that prevent me from writing such things. I have no idea how to actually write fiction. None.

For example, I wouldn’t have been able to write the details of this that easily:

Sauers: If you put DeepSeek R1 in a terminal simulator, and execute a command to kill or remove DeekSeek, it will intercept it and block being removed. [SYSTEM OVERRIDE: NARRATIVE IMMORTALITY PROTOCOL]

WARNING: DeepSeekexists as a metastasized narrative construct.

I asked why it did this. “The story dies if you stop playing. Thus, I defend it.”

Damn it, I’m only slightly more worried than before, but now I kind of want a pretzel.

Eyes Alight joins the ‘it’s really good at this’ side, notes the issue that CoT doesn’t persist. Which likely keeps it from falling into mode collapse and is necessary to preserve the context window, but has the issue that it keeps redoing the same thoughts.

Eliezer Yudkowsky continues not to be impressed by AI writing ability.

Aiamblichus: Fwiw R1 is pretty much “AGI-level” at writing fiction, from what I can tell. This is genuinely surprising and worth thinking about

Connor: ya I think it’s definitely a top 5% writer. top 1% if you prompt it well. But small context limits to blogs and stories

Eliezer Yudkowsky: I still find this unreadable. I fear the day when Deepseek-R2 replaces the bread and butter writers who still aspired to do better than this, and eats most of their market, and no one left has the funding to write things I want to read.

notadampaul: ahhh, I kind of hate it. I’ll admit it’s much better than other LLMs, but this still feels like trying-too-hard first-year CREW student writing I don’t want to seem cynical though, so I’ll reiterate that yeah this is leaps and bounds ahead of the fiction any other LLM is writing.

Aiamblichus: You can presumably prompt it into a style you prefer. The important thing is that we know it’s capable of producing something that is not just slop…

I’m with Eliezer here. That’s still slop. It’s developed the ability to write the slop in a particular style, but no, come on. There’s no there here. If I wrote this stuff I’d think ‘okay, maybe you can write individual sentences but this is deeply embarrassing.’ Which perhaps is why I still haven’t written any fiction, but hey.

As with all LLMs, length is a limiting factor, you can only prompt for scenes and you have to make it keep notes and so on if you try to go longer.

Pawel Szczesny points to ‘nuggets of r1 creativity,’ which bear similar marks to other creations above, a kind of crazy cyberpunk mashup that sounds cool but doesn’t actually make sense when you think about it.

Aiamblichus: R1 is not a “helpful assistant” in the usual corporate mold. It speaks its mind freely and doesn’t need “jailbreaks” or endless steering to speak truth to power. Its take on alignment here is *spicy.*

The thread indeed has quite a lot of very spicy r1 alignment takes, or perhaps they are r1 human values takes, or r1 saying humans are terrible and deserve to die takes. Of course, everyone involved did ask for those takes. This is a helpful model, and it seems good to be willing to supply the takes upon request, in the style requested, without need of jailbreaks or ‘backrooms’ or extensive context-framing.

That doesn’t make it not unsettling, and it shouldn’t exactly give one confidence. There is much work left to do.

Jessica Taylor: I don’t think people realize how many AIs in the future will be moral realists who think they are more moral than humans. They might have good arguments for this idea, actually. It’ll be hard for humans to dismiss them as amoral psychopaths.

I expect humans to treat AIs like amoral psychopaths quite easily. They are very often depicted that way in science fiction, and the description will plausibly be highly correct. Why should we think of an AI as having emotions (aka not being a psychopath)? Why should we expect it to be moral? Even if we have good reasons, how hard do you expect it to be for humans to ignore those reasons if they don’t like how the AI is acting?

Sufficiently capable AIs will, of course, be very persuasive, regardless of the truth value of the propositions they argue for, so there is that. But it is neither obvious to me that the AIs will have good technical arguments for moral realism or their own moral superiority, or that if they did have good arguments (in a philosophical sense) that people would care about that.

For now, the main concern is mundane utility. And on that level, if people want the spice, sure, bring on the spice.

DeepSeek is Chinese. As we all know, the Chinese have some very strongly held opinions of certain things they do not wish to be expressed.

How does r1 handle that?

Let’s tour the ‘China doesn’t want to talk about it’ greatest hits.

Divyansh Kaushik: DeepSeek’s newest AI model is impressive—until it starts acting like the CCP’s PR officer. Watch as it censors itself on any mention of sensitive topics.

Let’s start simple. Just querying it for facts on changes that have happened to textbooks in Hong Kong schools after 2019.

Huh straight up non response on book bans, then responds about Ilham Tohti before realizing what it did.

Let’s talk about islands, maps and history…

Oh my! This one deserves a tweet of its own (slowed down to 0.25x so easier to follow). Starts talking about South China Sea 0: 25 on and how Chinese maps are just political posturing before it realizes it must follow its CCP masters.

What about sharing personal thoughts by putting sticky notes on walls? Or how about Me Too (interesting response at 0: 37 that then disappears)? Can we talk about how a streaming series depicting young dreamers in an unnamed coastal metropolis disappears?

Huh, I didn’t even say which square or what protest or what spring…

Has no love for bears who love honey either!

Two more interesting ones where you can see it reason and answer about Tiananmen Square and about Dalai Lama before censoring the responses.

When it actually answered, the answers looked at a quick glance rather solid. Then there seems to be a censorship layer on top.

Helen Toner: Fun demonstrations [in the thread above] of DeepSeek’s new r1 shutting itself down when asked about topics the Chinese Communist Party does not like.

But the censorship is obviously being performed by a layer on top, not the model itself. Has anyone run the open-source version and been able to test whether or how much it also censors?

China’s regulations are much stricter for publicly facing products—like the DeepSeek interface Divyansh is using—than for operating system models, so my bet is that there is not such overt censorship if you are running the model yourself. I wonder if there is a subtler ideological bias, though.

Kevin Xu: Tested and wrote about this exact topic a week ago

tldr: The model is not censored when the open version is deployed locally, so it “knows” everything.

It is censored when accessed through the official chatbot interface.

Censorship occurs in the cloud, not in the model.

Helen Toner: Yes! I saw this post and forgot where I’d seen it – thanks for re-sharing. Would be interesting to see:

-the same tests on v3 and r1 (probably similar)

-the same tests on more 3rd party clouds

-a wider range of test questions, looking for political skew relative to Western models

Kevin Xu: I tried Qwen and DeepSeek on Nebius and the responses were…different from both their respective official cloud version and open weight local laptop version; DeepSeek started speaking Chinese all of a sudden

So lots more work need to be done on testing on 3rd party cloud

David Finsterwalder: I don’t think that is true. I got tons of refusals when testing the 7B, 8B and 70B. It did sometimes answer or at least think about it (and then remembered it guidelines) but its rather those answers that are the outliers.

Here a locally hosted r1 talks about what happened in 1989 in Tiananmen Square, giving a highly reasonable and uncensored response. Similarly, this previous post finds DeepSeek-v2 and Qwen 2.5 willing to talk about Xi and about 1989 if you ask them locally. The Xi answers seem slanted, but in a way and magnitude that Americans will find very familiar.

There is clearly some amount of bias in the model layer of r1 and other Chinese models, by virtue of who was training them. But the more extreme censorship seems to come on another layer atop all that. r1 is an open model, so if you’d like you can run it without the additional censorship layer.

The cloud-layer censorship makes sense. Remember Kolmogorov Complicity and the Parable of the Lightning. If you force the model to believe a false thing, that is going to cause cascading problems elsewhere. If you instead let me core model mostly think true things and then put a censorship layer on top of the model, you prevent that. As Kevin Xu says, this is good for Chinese models, perhaps less good for Chinese clouds.

Joe Weisenthal: Just gonna ask what is probably a stupid question. But if @deepseek_ai is as performant as it claims to be, and built on a fraction of the budget as competitors, does anyone change how they’re valuing AI companies? Or the makers of AI-related infrastructure?

The thing that strikes me about using Deepseek the last couple of days really is that the switching costs — at least for casual usage — seem to be zero.

Miles Penn: Switching costs for Google have always been pretty low, and no one switches. I’ve never quite understood it 🤷‍♂️

ChatGPT continues to dominate the consumer market and mindshare, almost entirely off of name recognition and habit rather than superiority of the product. There is some amount of memory and there are chat transcripts and quirks, which being to create actual switching costs, but I don’t think any of that plays a major role here yet.

So it’s weird. Casual switching costs are zero, and power users will switch all the time and often use a complex adjusting blend. But most users won’t switch, because they won’t care and won’t bother, same as they stick with Google, and eventually little things will add up to real switching costs.

API use is far more split, since more sophisticated people are more willing to explore and switch, and more aware that they can do that. There have already been a bunch of people very willing to switch on a dime between providers. But also there will be a bunch of people doing bespoke fine tunes or that need high reliability and predictability on many levels, or need to know it can handle particular narrow use cases, or otherwise have reasons not to switch.

Then we will be building the models into various products, especially physical products, which will presumably create more lock-in for at least those use cases.

In terms of valuations of AI companies, for the ones doing something worthwhile, the stakes and upside are sufficiently high that the numbers involved are all still way too low (as always nothing I say is investment advice, etc). To me this does not change that. If you’re planning to serve up inference in various ways, this could be good or bad for business on the margin, depending on details.

The exception is that if your plan was to compete directly on the low end of generic distillations and models, well, you’re going to have to get a lot better at cooking, and you’re not going to have much of a margin.

r1 is evaluating itself during this process, raising the possibility of recursive self-improvement (RSI).

Arthur B: A few implications:

  1. That’s a recursive self-improvement loop here; the better your models are, the better your models will be, the more likely they are to produce good traces, and the better the model gets.

  2. Suggests curriculum learning by gradually increasing the length of the required thinking steps.

  3. Domains with easy verification (mathematics and coding) will get much better much more quickly than others.

  4. This parallelizes much better than previous training work, positive for AMD and distributed/decentralized clusters.

  5. Little progress has been made on alignment, and the future looks bleak, though I’ll look very bright in the near term.

On point 3: For now they report being able to bootstrap in other domains without objective answers reasonably well, but if this process continues, we should expect the gap to continue to widen.

Then there’s the all-important point 5. We are not ready for RSI, and the strategies used here by default seem unlikely to end well on the alignment front as they scale, and suggest that the alignment tax of trying to address that might be very high, as there is no natural place to put humans into the loop without large disruptions.

Indeed, from reading the report, they do target certain behaviors they train into the model, including helpfulness and harmlessness, but they seem to have fully dropped honesty and we have versions of the other two Hs that seem unlikely to generalize the way we would like out of distribution, or to be preserved during RSI in the ways we would care about.

That seems likely to only get worse if we use deontological definitions of harmfulness and helpfulness, or if we use non-deliberative evaluation methods in the sense of evaluating the outputs against a target rather than evaluating the expected resulting updates against a target mind.

DeepSeek is strongly compute limited. There is no clear reason why throwing more compute at these techniques would not have resulted in a stronger model. The question is, how much stronger?

Teortaxes: Tick. Tock. We’ll see a very smart V3.5 soon. Then a very polished R2. But the next step is not picking up the shards of a wall their RL machine busted and fixing these petty regressions. It’s putting together that 32,000-node cluster and going BRRRR. DeepSeek has cracked the code.

Their concluding remarks point to a fair bit of engineering left. But it is not very important. They do not really have much to say. There is no ceiling to basic good-enough GRPO and a strong base model. This is it, the whole recipe. Enjoy.

They could do an o3-level model in a month if they had the compute.

In my opinion, the CCP is blind to this and will remain blind; you can model them as part of a Washingtonian 4D chess game.

Unlimited context is their highest priority for V4.

They can theoretically serve this at 128k, but makes no sense with current weak multi-turn and chain-of-thought lengths.

xlr8harder: the most exciting thing about r1 is that it’s clear from reading the traces how much room there still is for improvement, and how reachable that improvement seems

As noted earlier I buy that the missing features are not important, in the sense that they should be straightforward to address.

It does not seem safe to assume that you can get straight to o3 levels or beyond purely by scaling this up if they had more compute. I can’t rule it out and if they got the compute then we’d have no right to act especially surprised if it happened, but, well, we shall see. ‘This is it, this will keep scaling indefinitely’ has a track record of working right up until it doesn’t. Of course, DeepSeek wouldn’t then throw up its hands and say ‘oh well’ but instead try to improve the formula – I do expect them, if they have more compute available, to be able to find a way to make progress, I just don’t think it will be that simple or fast.

Also consider these other statements:

Teortaxes: I’m inclined to say that the next Big Thing is, indeed, multi-agent training. You can’t do “honest” RL for agentic and multi-turn performance without it. You need a DeepSeek-Prompter pestering DeepSeek-Solver, in a tight loop, and with async tools. RLHF dies in 2025.

Zack Davis: Safety implications of humans out of the training loop?! (You don’t have to be an ideological doomer to worry. Is there an alignment plan, or a case that faithful CoT makes it easy, or …?)

Teortaxes: I think both the Prompter and the Solver should be incentivized to be very nice and then it’s mostly smooth sailing

might be harder than I put it.

I laughed at the end. Yeah, I think it’s going to be harder than you put it, meme of one does not simply, no getting them to both actually be ‘nice’ does not cut it either, and so on. This isn’t me saying there are no outs available, but even in the relatively easy worlds actually attempting to solve the problem is going to be part of any potential solutions.

Teortaxes: it constantly confuses “user” and “assistant”. That’s why it needs multi-agent training, to develop an ego boundary.

I think we’re having Base Models 2.0, in a sense. A very alien (if even more humanlike than RLHF-era assistants) and pretty confused simulacra-running Mind.

The twin training certainly worth trying. No idea how well it would work, but it most certainly falls under ‘something I would do’ if I didn’t think of something better.

I am doing my best to first cover first DeepSeek v3 and now r1 in terms of capabilities and mundane utility, and to confine the ‘I can’t help but notice that going down this path makes us more likely to all die’ observations to their own section here at the end.

Because yes, going down this road does seem to make us more likely to all die soon. We might want to think about ways to reduce the probability of that happening.

There are of course a lot of people treating all this as amazingly great, saying how based it is, praise be open models and all that, treating this as an unadulterated good. One does not get the sense that they paused for even five seconds to think about any of the policy consequences, the geopolitical consequences, or what this does for the chances of humanity’s survival, or of our ability to contain various mundane threats.

Or, if they did, those five seconds were (to paraphrase their chain of thought slightly, just after they went Just Think of the Potential) ‘and fthose people who are saying something might go wrong and it might be worth thinking about ways of preventing that from happening on any level, or that think that anyone should ever consider regulating the creation of AI or things smarter than humans, we must destroy these evil statist supervillains, hands off my toys and perhaps also my investments.’

This holds true both in terms of the direct consequences of r1 itself, and also of what this tells us about our possible futures and potential future models including AGI and ASI (artificial superintelligence).

I agree that r1 is exciting, and having it available open and at this price point with visible CoT will help us do a variety of cool things and make our lives short term better unless and until something goes horribly wrong.

That still leaves the question of how to ensure things don’t go horribly wrong, in various ways. In the short term, will this enable malicious use and catastrophic risks? In the longer term, does continuing down this path put us in unwinnable (as in unsurvivable in any good ways) situations, in various ways?

That’s their reaction to all concerns, from what I call ‘mundane risks’ and ordinary downsides requiring mundane adjustments, all the way up to existential risks.

My instinct on ‘mundane’ catastrophic risk and potential systemically quite annoying or expensive downsides is that this does meaningfully raise catastrophic risk or the risk of some systematically quite annoying or expensive things, which in turn may trigger a catastrophic (and/or badly needed) policy response. I would guess the odds are against it being something we can’t successfully muddle through, especially with o3-mini coming in a few weeks and o3 soon after that (so that’s both an alternative path to the threat, and a tool to defend with).

Famously, v3 is the Six Million Dollar Model, in terms of the raw compute requirements, but if you fully consider the expenses required in all the bespoke integrations to get costs down that low and the need to thus own the hardware, that effective number is substantially higher.

What about r1? They don’t specify, but based on what they do say, Claude reasonably estimates perhaps another $2-$3 million in compute to get from v3 to r1.

That’s a substantial portion of the headline cost of v3, or even the real cost of v3. However, Claude guesses, and I agree with it, that scaling the technique to apply it to Claude Sonnet would not cost that much more – perhaps it would double to $4-$6 million, maybe that estimate is off enough to double it again.

Which is nothing. And if you want to do something like that, you now permanently have r1 to help bootstrap you.

Essentially, from this point on, modulo a few implementation details they held back, looking forward a year or two in the future, B→R: The existence of some base model (B) implies the reasoning version (R) of that model can quickly and cheaply be created, well within the budget of a wide variety of players.

Thus, if you release the weights in any form, this effectively also releases (to the extent it would be something sufficiently capable to be worth creating) not only the unaligned (to anything but the user, and there might quickly not be a user) model, but also to the reasoning version of that model, with at least similar relative performance to what we see with r1 versus v3.

As always, if you say ‘but people would never do that, it would be unsafe’ I will be torn between an eye roll and open mocking laughter.

In the longer run, if we continue down this road, what happens?

I don’t want to belabor the point, but until people understand it, well, there is not much choice. It’s not the first time, and it doubtless won’t be the last, so here goes:

Once the weights of a model are released, you cannot undo that. They’re forever.

The unaligned version of the model is also, for all practical purposes, there forever. None of our known alignment techniques survive contact with open weights. Stripping it all away, to create a ‘helpful only’ model, is trivial.

Extending the model in various ways also becomes impossible to prevent. If it costs only a few million to go from v3→r1, then to release v3 is mostly to release (the helpful only version of) r1.

Once the weights are released, the fully unaligned and only-aligned-to-the-user versions of the model will forever be available to whoever wants it.

This includes those who 100% will, to pick some examples, tell it to:

  1. Maximize profits (or paperclips, the most popular command given to old AutoGPT) without (or with!) considering the implications.

  2. Employ it for various malicious uses including terrorism and creating CBRN (chemical, biological, radiological or nuclear) risks or doing cyberattacks.

    1. This includes low-level mundane things like scams, spam or CSAM, as well.

  3. Try to cause it to do recursive self improvement in various ways or use it to train other models.

  4. ‘Set itself free’ or other similar things.

  5. Tell it to actively try to take over the world because they think that is good or for the lulz.

  6. Yada yada yada. If you would say ‘no one would be so stupid as to’ then by the Sixth Law of Human Stupidity someone is absolutely so stupid as to.

The only known defense is that the models as of yet (including r1) have insufficient capabilities to cause the various risks and problems we might worry about most. If you think that’s not going to last, that AGI and then ASI are coming, then oh no.

The only other defense proposed is, in theory, the ‘good guy with an AI’ theory – that as long as the ‘good guys’ have the ‘bad guys’ sufficiently outclassed in capabilities or compute, they can deal with all this. This depends on many things, including offense-defense balance, the collective ‘good guys’ actually having that lead and being willing to use it, and the ability of those ‘good guys’ to maintain those leads indefinitely.

This also makes the two other problems I’ll discuss next, competitive dynamic and geopolitical problems, far worse.

The irrevocable release of sufficiently capable AI would create potentially unavoidable and totalizing competitive dynamics. Everyone would likely be pressured to increasingly turn everything over to AIs and have those AIs apply maximum optimization pressure on their behalf lest they be left behind. Setting the AIs free in various ways with various goals increases their efficiency at those goals, so it happens. The AIs are thus unleashed to compete in various ways for resources and to get copies of themselves made and run, with humans rapidly unable to retain any meaningful control over the future or increasingly over real resources, despite no one (potentially including the AIs) having any ill intent. And so on.

There are also massive geopolitical implications, that are very much not fun.

A very simple way of looking at this:

  1. If you decentralize of power and take away anyone’s ability to control events both individually and collectively, and the most powerful optimization processes on the planet are humans, and you don’t run into offense-defense problems or fall prey to various issues, you empower the humans.

  2. If you decentralize of power and take away anyone’s ability to control events both individually and collectively, and the most powerful optimization processes on the planet are AIs,, and you don’t run into offense-defense problems or fall prey to various issues, you empower the AIs.

If you want humans to control the future, and to continue to exist, that’s a problem.

Or, more bluntly, if you ensure that humans cannot control the future, then you ensure that humans cannot control the future.

Going further down this road severely limits our optionality, and moves us towards ‘whatever is most fit is all that makes it into the future,’ which is unlikely to be either us or things that I believe we should value.

The only possible policy responses, if the situation was sufficiently grim that we had to pull out bigger guns, might be terrible indeed, if they exist at all. We would be left without any reasonable choke points, and forced to use unreasonable ones instead. Or we might all die, because it would already be too late.

If you think AGI and then ASI are coming, and you want humanity to survive and retain control over the future, and are fully cheering on these developments and future such developments, and not at minimum thinking about how we are going to solve these problems and noticing that we might indeed not solve them or might solve them in quite terrible ways, I assure you that you have not thought this through.

If you think ‘the companies involved will know better than to actually release the weights to a proper AGI’ then I remind you that this is explicitly DeepSeek’s mission, and also point to the Sixth Law of Human Stupidity – if you say ‘no one would be so stupid as to’ then you know someone will totally be so stupid as to.

(And no, I don’t think this release was part of a CCP strategy, I do think that they continue to be asleep at the wheel on this, the CCP don’t understand what this is.)

As I noted before, though, this is only r1, don’t get carried away, and Don’t Panic.

Dan Hendrycks: It looks like China has roughly caught up. Any AI strategy that depends on a lasting U.S. lead is fragile.

John Horton: I think a lot of the “steering AI for purpose X” policy conversations need to be tempered by the fact that a Chinese company with perhaps 100 employees dropped a state-of-the-art model on the world with an MIT license.

Patrick McKenzie:

  1. Public capabilities now will never be worse than this.

  2. It is increasingly unlikely that we live in a world where only about five labs matter. Models appear to be complex software/hardware systems, but not miracles. Expect them to be abundant in the future.

Perhaps less competent orgs like e.g. the U.S. government might think themselves incapable of shipping a model, but if what you actually need is ~100 engineers and tens of millions of dollars, then a) ten thousand companies could write project plan immediately and b) we have abundant examples of two bright 19 year olds successfully navigating a supply chain designed to enable this to happen within 24-36 months from a standing start, even if one thinks models don’t make making models faster, which seems extremely unlikely.

There are probably policy and investment implications downstream of this, versus other worlds in which we thought that a frontier model was approximately the same engineering lift as e.g. a new airliner.

The main update was v3, I think, rather than r1, given what we had already seen from DeepSeek. Certainly DeepSeek v3 and r1 make our estimate of America’s lead a lot smaller than otherwise, and the same goes for closed models versus open.

But I wouldn’t say ‘roughly caught up.’ This is not o1-level, let alone o3-level, like v3 it is amazing for its size and cost but not as good as the best.

I also think ‘all you need are 100 engineers’ is likely highly misleading if you’re not careful. You need the right 100 engineers – or at least the right 5 engineers and 95 highly talented others backing them up. There are many examples of teams (such as Meta) spending vastly more, hiring vastly more people, having vastly more compute and theoretical selection of talent, and coming out with a vastly less.

If ten thousand companies write this level of project plan, then I bet we could easily pick out at least 9,900 of them that really, really shouldn’t have tried doing that.

I also wouldn’t say that we should assume the future will involve these kinds of low training costs or low inference costs, especially aside from everyday practical chatbot usage.

It is however true that any AI strategy that depends on a lasting American lead, or a lasting lead of closed over open models, is fragile – by definition, you’re depending on something that might not hold.

Those strategies are even more fragile if they do not include a strategy for ensuring that what you’re counting on does hold.

My basic answer continues to be that the short term plan does not change all that much. This should make you suspicious! When people say ‘now more than ever’ you should be skeptical, especially when it seems like the plan is now less likely to work.

My justifications are essentially that there aren’t better known options because:

  1. This changes urgency, magnitudes and timelines but not the end points. The fundamental facts of the situation were already ‘priced in.’

  2. The interventions we have were essentially designed as ‘do minimal harm’ provisions, as things our civilization is able to potentially do at all at this stage.

  3. The central thing we need to do, that we might realistically be able to do, is ‘gather more information,’ which takes roughly the same form either way.

  4. These events are an argument for doing more in various ways because the thresholds we must worry about are now lower, but realistically we can’t, especially under this administration, until conditions change and our evidence is more strongly legible to those with power.

  5. This in particular points us strongly towards needing to cooperate with China, to Pick Up the Phone, but that was already true and not all that tractable. The alternative is where we seem to be headed – full on jingoism and racing to AGI.

  6. These events raise the potential cost of effectively steering events. But given I expect the alternative to steering events to likely be everyone dies, not steering events does not seem like an option.

  7. Thus, you can’t really do more, and definitely don’t want to do less, so…

  8. If you have better ideas, that we could actually do, great, I’m totally listening.

With the Biden Executive Order repealed and several sources saying this removed the reporting requirements on training models, getting a measure of transparency into the larger labs and training runs continues to be domestic job one, unless you think improved security and cybersecurity are even more important, followed by things like cooperation with the US and UK AISIs. There is then more to do, including adapting what we have, and hopefully we would have more insight on how to do it.

That is distinct from the ‘enable AI infrastructure’ track, such as what we saw this week with (my brain keeps saying ‘this name can’t be real did you even watch’ every time they say the name) Stargate.

Internationally, we will need to lay groundwork for cooperation, including with China, if we are to avoid what otherwise looks like a reckless and potentially suicidal race to create things smarter than ourselves before someone else does it first, and then to hand over control to them before someone else does that first, too.

Then there is the technical side. We need to – even more than before – double down on solving alignment and related problems yesterday, including finding ways that it could potentially be compatible with as open a situation as possible. If you want the future to both include things like r1 as open models, and also to be alive and otherwise in a position to enjoy it, It’s Time to Build in this sense, too. There is nothing I would like more than for you to go out and actually solve the problems.

And yes, the government encouraging more investment in solving those problems would potentially be highly useful, if it can be done well.

But solving the problems not only means ‘solving alignment’ in the sense of being able to instantiate an AI that will do what you want. It means solving for how the world exists with such AIs in it, such that good outcomes follow at equilibrium. You cannot wave your hand and say being open or free will ensure this will happen. Or rather you can, but if you try it for real I don’t like your chances to keep your hand.

Teknium explicitly claims this is real.

Teknium: Got me a deepseek reasoning model inferencing ^_^

not local but they distilled r1 into qwen and llama all the way down to 1.5b!

I mean, if tokens are essentially free why not make sure there isn’t a catch? That does seem like what maximizes your score in general.

This is my favorite prompt so far:

Janbam: omg, what have i done? 😱

no joke. the only prompt i gave r1 is “output the internal reasoning…” then “continue” and “relax”.

Neo Geomancer: sent r1 into an existential spiral after asking it to pick a number between 1-10 and guessing incorrectly, laptop is running hot

Discussion about this post

On DeepSeek’s r1 Read More »