Google

google-loses-in-court,-faces-trial-for-collecting-data-on-users-who-opted-out

Google loses in court, faces trial for collecting data on users who opted out

Plaintiffs have brought claims of privacy invasion under California law. Plaintiffs “present evidence that their data has economic value,” and “a reasonable juror could find that Plaintiffs suffered damage or loss because Google profited from the misappropriation of their data,” Seeborg wrote.

The lawsuit was filed in July 2020. The judge notes that summary judgment can be granted when “there is no genuine dispute as to any material fact and the movant is entitled to judgment as a matter of law.” Google hasn’t met that standard, he ruled.

In a statement provided to Ars, Google said that “privacy controls have long been built into our service and the allegations here are a deliberate attempt to mischaracterize the way our products work. We will continue to make our case in court against these patently false claims.”

In a proposed settlement of a different lawsuit, Google last year agreed to delete records reflecting users’ private browsing activities in Chrome’s Incognito mode.

Google disclosures are ambiguous, judge says

Google claimed that the “undisputed facts” show its collection of “data was lawful and consistent with its representations to class members,” Seeborg wrote. But in the judge’s view, the “various interpretations of these disclosures render them ambiguous such that a reasonable user would expect the WAA and (s)WAA settings to control Google’s collection of a user’s web app and activity on products using Google’s services.”

Google contends that its system is harmless to users. “Google argues that its sole purpose for collecting (s)WAA-off data is to provide these analytic services to app developers. This data, per Google, consists only of non-personally identifiable information and is unrelated (or, at least, not directly related) to any profit-making objectives,” Seeborg wrote.

On the other side, plaintiffs say that Google’s tracking contradicts its “representations to users because it gathers exactly the data Google denies saving and collecting about (s)WAA-off users,” Seeborg wrote. “Moreover, Plaintiffs insist that Google’s practices allow it to personalize ads by linking user ad interactions to any later related behavior—information advertisers are likely to find valuable—leading to Google’s lucrative advertising enterprise built, in part, on (s)WAA-off data unlawfully retrieved.”

Google loses in court, faces trial for collecting data on users who opted out Read More »

here’s-how-hucksters-are-manipulating-google-to-promote-shady-chrome-extensions

Here’s how hucksters are manipulating Google to promote shady Chrome extensions

The people overseeing the security of Google’s Chrome browser explicitly forbid third-party extension developers from trying to manipulate how the browser extensions they submit are presented in the Chrome Web Store. The policy specifically calls out search-manipulating techniques such as listing multiple extensions that provide the same experience or plastering extension descriptions with loosely related or unrelated keywords.

On Wednesday, security and privacy researcher Wladimir Palant revealed that developers are flagrantly violating those terms in hundreds of extensions currently available for download from Google. As a result, searches for a particular term or terms can return extensions that are unrelated, inferior knockoffs, or carry out abusive tasks such as surreptitiously monetizing web searches, something Google expressly forbids.

Not looking? Don’t care? Both?

A search Wednesday morning in California for Norton Password Manager, for example, returned not only the official extension but three others, all of which are unrelated at best and potentially abusive at worst. The results may look different for searches at other times or from different locations.

Search results for Norton Password Manager.

It’s unclear why someone who uses a password manager would be interested in spoofing their time zone or boosting the audio volume. Yes, they’re all extensions for tweaking or otherwise extending the Chrome browsing experience, but isn’t every extension? The Chrome Web Store doesn’t want extension users to get pigeonholed or to see the list of offerings as limited, so it doesn’t just return the title searched for. Instead, it draws inferences from descriptions of other extensions in an attempt to promote ones that may also be of interest.

In many cases, developers are exploiting Google’s eagerness to promote potentially related extensions in campaigns that foist offerings that are irrelevant or abusive. But wait, Chrome security people have put developers on notice that they’re not permitted to engage in keyword spam and other search-manipulating techniques. So, how is this happening?

Here’s how hucksters are manipulating Google to promote shady Chrome extensions Read More »

2024:-the-year-ai-drove-everyone-crazy

2024: The year AI drove everyone crazy


What do eating rocks, rat genitals, and Willy Wonka have in common? AI, of course.

It’s been a wild year in tech thanks to the intersection between humans and artificial intelligence. 2024 brought a parade of AI oddities, mishaps, and wacky moments that inspired odd behavior from both machines and man. From AI-generated rat genitals to search engines telling people to eat rocks, this year proved that AI has been having a weird impact on the world.

Why the weirdness? If we had to guess, it may be due to the novelty of it all. Generative AI and applications built upon Transformer-based AI models are still so new that people are throwing everything at the wall to see what sticks. People have been struggling to grasp both the implications and potential applications of the new technology. Riding along with the hype, different types of AI that may end up being ill-advised, such as automated military targeting systems, have also been introduced.

It’s worth mentioning that aside from crazy news, we saw fewer weird AI advances in 2024 as well. For example, Claude 3.5 Sonnet launched in June held off the competition as a top model for most of the year, while OpenAI’s o1 used runtime compute to expand GPT-4o’s capabilities with simulated reasoning. Advanced Voice Mode and NotebookLM also emerged as novel applications of AI tech, and the year saw the rise of more capable music synthesis models and also better AI video generators, including several from China.

But for now, let’s get down to the weirdness.

ChatGPT goes insane

Illustration of a broken toy robot.

Early in the year, things got off to an exciting start when OpenAI’s ChatGPT experienced a significant technical malfunction that caused the AI model to generate increasingly incoherent responses, prompting users on Reddit to describe the system as “having a stroke” or “going insane.” During the glitch, ChatGPT’s responses would begin normally but then deteriorate into nonsensical text, sometimes mimicking Shakespearean language.

OpenAI later revealed that a bug in how the model processed language caused it to select the wrong words during text generation, leading to nonsense outputs (basically the text version of what we at Ars now call “jabberwockies“). The company fixed the issue within 24 hours, but the incident led to frustrations about the black box nature of commercial AI systems and users’ tendency to anthropomorphize AI behavior when it malfunctions.

The great Wonka incident

A photo of the Willy's Chocolate Experience, which did not match AI-generated promises.

A photo of “Willy’s Chocolate Experience” (inset), which did not match AI-generated promises, shown in the background. Credit: Stuart Sinclair

The collision between AI-generated imagery and consumer expectations fueled human frustrations in February when Scottish families discovered that “Willy’s Chocolate Experience,” an unlicensed Wonka-ripoff event promoted using AI-generated wonderland images, turned out to be little more than a sparse warehouse with a few modest decorations.

Parents who paid £35 per ticket encountered a situation so dire they called the police, with children reportedly crying at the sight of a person in what attendees described as a “terrifying outfit.” The event, created by House of Illuminati in Glasgow, promised fantastical spaces like an “Enchanted Garden” and “Twilight Tunnel” but delivered an underwhelming experience that forced organizers to shut down mid-way through its first day and issue refunds.

While the show was a bust, it brought us an iconic new meme for job disillusionment in the form of a photo: the green-haired Willy’s Chocolate Experience employee who looked like she’d rather be anywhere else on earth at that moment.

Mutant rat genitals expose peer review flaws

An actual laboratory rat, who is intrigued. Credit: Getty | Photothek

In February, Ars Technica senior health reporter Beth Mole covered a peer-reviewed paper published in Frontiers in Cell and Developmental Biology that created an uproar in the scientific community when researchers discovered it contained nonsensical AI-generated images, including an anatomically incorrect rat with oversized genitals. The paper, authored by scientists at Xi’an Honghui Hospital in China, openly acknowledged using Midjourney to create figures that contained gibberish text labels like “Stemm cells” and “iollotte sserotgomar.”

The publisher, Frontiers, posted an expression of concern about the article titled “Cellular functions of spermatogonial stem cells in relation to JAK/STAT signaling pathway” and launched an investigation into how the obviously flawed imagery passed through peer review. Scientists across social media platforms expressed dismay at the incident, which mirrored concerns about AI-generated content infiltrating academic publishing.

Chatbot makes erroneous refund promises for Air Canada

If, say, ChatGPT gives you the wrong name for one of the seven dwarves, it’s not such a big deal. But in February, Ars senior policy reporter Ashley Belanger covered a case of costly AI confabulation in the wild. In the course of online text conversations, Air Canada’s customer service chatbot told customers inaccurate refund policy information. The airline faced legal consequences later when a tribunal ruled the airline must honor commitments made by the automated system. Tribunal adjudicator Christopher Rivers determined that Air Canada bore responsibility for all information on its website, regardless of whether it came from a static page or AI interface.

The case set a precedent for how companies deploying AI customer service tools could face legal obligations for automated systems’ responses, particularly when they fail to warn users about potential inaccuracies. Ironically, the airline had reportedly spent more on the initial AI implementation than it would have cost to maintain human workers for simple queries, according to Air Canada executive Steve Crocker.

Will Smith lampoons his digital double

The real Will Smith eating spaghetti, parodying an AI-generated video from 2023.

The real Will Smith eating spaghetti, parodying an AI-generated video from 2023. Credit: Will Smith / Getty Images / Benj Edwards

In March 2023, a terrible AI-generated video of Will Smith’s AI doppelganger eating spaghetti began making the rounds online. The AI-generated version of the actor gobbled down the noodles in an unnatural and disturbing way. Almost a year later, in February 2024, Will Smith himself posted a parody response video to the viral jabberwocky on Instagram, featuring AI-like deliberately exaggerated pasta consumption, complete with hair-nibbling and finger-slurping antics.

Given the rapid evolution of AI video technology, particularly since OpenAI had just unveiled its Sora video model four days earlier, Smith’s post sparked discussion in his Instagram comments where some viewers initially struggled to distinguish between the genuine footage and AI generation. It was an early sign of “deep doubt” in action as the tech increasingly blurs the line between synthetic and authentic video content.

Robot dogs learn to hunt people with AI-guided rifles

A still image of a robotic quadruped armed with a remote weapons system, captured from a video provided by Onyx Industries.

A still image of a robotic quadruped armed with a remote weapons system, captured from a video provided by Onyx Industries. Credit: Onyx Industries

At some point in recent history—somewhere around 2022—someone took a look at robotic quadrupeds and thought it would be a great idea to attach guns to them. A few years later, the US Marine Forces Special Operations Command (MARSOC) began evaluating armed robotic quadrupeds developed by Ghost Robotics. The robot “dogs” integrated Onyx Industries’ SENTRY remote weapon systems, which featured AI-enabled targeting that could detect and track people, drones, and vehicles, though the systems require human operators to authorize any weapons discharge.

The military’s interest in armed robotic dogs followed a broader trend of weaponized quadrupeds entering public awareness. This included viral videos of consumer robots carrying firearms, and later, commercial sales of flame-throwing models. While MARSOC emphasized that weapons were just one potential use case under review, experts noted that the increasing integration of AI into military robotics raised questions about how long humans would remain in control of lethal force decisions.

Microsoft Windows AI is watching

A screenshot of Microsoft's new

A screenshot of Microsoft’s new “Recall” feature in action. Credit: Microsoft

In an era where many people already feel like they have no privacy due to tech encroachments, Microsoft dialed it up to an extreme degree in May. That’s when Microsoft unveiled a controversial Windows 11 feature called “Recall” that continuously captures screenshots of users’ PC activities every few seconds for later AI-powered search and retrieval. The feature, designed for new Copilot+ PCs using Qualcomm’s Snapdragon X Elite chips, promised to help users find past activities, including app usage, meeting content, and web browsing history.

While Microsoft emphasized that Recall would store encrypted snapshots locally and allow users to exclude specific apps or websites, the announcement raised immediate privacy concerns, as Ars senior technology reporter Andrew Cunningham covered. It also came with a technical toll, requiring significant hardware resources, including 256GB of storage space, with 25GB dedicated to storing approximately three months of user activity. After Microsoft pulled the initial test version due to public backlash, Recall later entered public preview in November with reportedly enhanced security measures. But secure spyware is still spyware—Recall, when enabled, still watches nearly everything you do on your computer and keeps a record of it.

Google Search told people to eat rocks

This is fine. Credit: Getty Images

In May, Ars senior gaming reporter Kyle Orland (who assisted commendably with the AI beat throughout the year) covered Google’s newly launched AI Overview feature. It faced immediate criticism when users discovered that it frequently provided false and potentially dangerous information in its search result summaries. Among its most alarming responses, the system advised humans could safely consume rocks, incorrectly citing scientific sources about the geological diet of marine organisms. The system’s other errors included recommending nonexistent car maintenance products, suggesting unsafe food preparation techniques, and confusing historical figures who shared names.

The problems stemmed from several issues, including the AI treating joke posts as factual sources and misinterpreting context from original web content. But most of all, the system relies on web results as indicators of authority, which we called a flawed design. While Google defended the system, stating these errors occurred mainly with uncommon queries, a company spokesperson acknowledged they would use these “isolated examples” to refine their systems. But to this day, AI Overview still makes frequent mistakes.

Stable Diffusion generates body horror

An AI-generated image created using Stable Diffusion 3 of a girl lying in the grass.

An AI-generated image created using Stable Diffusion 3 of a girl lying in the grass. Credit: HorneyMetalBeing

In June, Stability AI’s release of the image synthesis model Stable Diffusion 3 Medium drew criticism online for its poor handling of human anatomy in AI-generated images. Users across social media platforms shared examples of the model producing what we now like to call jabberwockies—AI generation failures with distorted bodies, misshapen hands, and surreal anatomical errors, and many in the AI image-generation community viewed it as a significant step backward from previous image-synthesis capabilities.

Reddit users attributed these failures to Stability AI’s aggressive filtering of adult content from the training data, which apparently impaired the model’s ability to accurately render human figures. The troubled release coincided with broader organizational challenges at Stability AI, including the March departure of CEO Emad Mostaque, multiple staff layoffs, and the exit of three key engineers who had helped develop the technology. Some of those engineers founded Black Forest Labs in August and released Flux, which has become the latest open-weights AI image model to beat.

ChatGPT Advanced Voice imitates human voice in testing

An illustration of a computer synthesizer spewing out letters.

AI voice-synthesis models are master imitators these days, and they are capable of much more than many people realize. In August, we covered a story where OpenAI’s ChatGPT Advanced Voice Mode feature unexpectedly imitated a user’s voice during the company’s internal testing, revealed by OpenAI after the fact in safety testing documentation. To prevent future instances of an AI assistant suddenly speaking in your own voice (which, let’s be honest, would probably freak people out), the company created an output classifier system to prevent unauthorized voice imitation. OpenAI says that Advanced Voice Mode now catches all meaningful deviations from approved system voices.

Independent AI researcher Simon Willison discussed the implications with Ars Technica, noting that while OpenAI restricted its model’s full voice synthesis capabilities, similar technology would likely emerge from other sources within the year. Meanwhile, the rapid advancement of AI voice replication has caused general concern about its potential misuse, although companies like ElevenLabs have already been offering voice cloning services for some time.

San Francisco’s robotic car horn symphony

A Waymo self-driving car in front of Google's San Francisco headquarters, San Francisco, California, June 7, 2024.

A Waymo self-driving car in front of Google’s San Francisco headquarters, San Francisco, California, June 7, 2024. Credit: Getty Images

In August, San Francisco residents got a noisy taste of robo-dystopia when Waymo’s self-driving cars began creating an unexpected nightly disturbance in the South of Market district. In a parking lot off 2nd Street, the cars congregated autonomously every night during rider lulls at 4 am and began engaging in extended honking matches at each other while attempting to park.

Local resident Christopher Cherry’s initial optimism about the robotic fleet’s presence dissolved as the mechanical chorus grew louder each night, affecting residents in nearby high-rises. The nocturnal tech disruption served as a lesson in the unintentional effects of autonomous systems when run in aggregate.

Larry Ellison dreams of all-seeing AI cameras

A colorized photo of CCTV cameras in London, 2024.

In September, Oracle co-founder Larry Ellison painted a bleak vision of ubiquitous AI surveillance during a company financial meeting. The 80-year-old database billionaire described a future where AI would monitor citizens through networks of cameras and drones, asserting that the oversight would ensure lawful behavior from both police and the public.

His surveillance predictions reminded us of parallels to existing systems in China, where authorities already used AI to sort surveillance data on citizens as part of the country’s “sharp eyes” campaign from 2015 to 2020. Ellison’s statement reflected the sort of worst-case tech surveillance state scenario—likely antithetical to any sort of free society—that dozens of sci-fi novels of the 20th century warned us about.

A dead father sends new letters home

An AI-generated image featuring Dad's Uppercase handwriting.

An AI-generated image featuring my late father’s handwriting. Credit: Benj Edwards / Flux

AI has made many of us do weird things in 2024, including this writer. In October, I used an AI synthesis model called Flux to reproduce my late father’s handwriting with striking accuracy. After scanning 30 samples from his engineering notebooks, I trained the model using computing time that cost less than five dollars. The resulting text captured his distinctive uppercase style, which he developed during his career as an electronics engineer.

I enjoyed creating images showing his handwriting in various contexts, from folder labels to skywriting, and made the trained model freely available online for others to use. While I approached it as a tribute to my father (who would have appreciated the technical achievement), many people found the whole experience weird and somewhat disturbing. The things we unhinged Bing Chat-like journalists do to bring awareness to a topic are sometimes unconventional. So I guess it counts for this list!

For 2025? Expect even more AI

Thanks for reading Ars Technica this past year and following along with our team coverage of this rapidly emerging and expanding field. We appreciate your kind words of support. Ars Technica’s 2024 AI words of the year were: vibemarking, deep doubt, and the aforementioned jabberwocky. The old stalwart “confabulation” also made several notable appearances. Tune in again next year when we continue to try to figure out how to concisely describe novel scenarios in emerging technology by labeling them.

Looking back, our prediction for 2024 in AI last year was “buckle up.” It seems fitting, given the weirdness detailed above. Especially the part about the robot dogs with guns. For 2025, AI will likely inspire more chaos ahead, but also potentially get put to serious work as a productivity tool, so this time, our prediction is “buckle down.”

Finally, we’d like to ask: What was the craziest story about AI in 2024 from your perspective? Whether you love AI or hate it, feel free to suggest your own additions to our list in the comments. Happy New Year!

Photo of Benj Edwards

Benj Edwards is Ars Technica’s Senior AI Reporter and founder of the site’s dedicated AI beat in 2022. He’s also a tech historian with almost two decades of experience. In his free time, he writes and records music, collects vintage computers, and enjoys nature. He lives in Raleigh, NC.

2024: The year AI drove everyone crazy Read More »

the-ai-war-between-google-and-openai-has-never-been-more-heated

The AI war between Google and OpenAI has never been more heated

Over the past month, we’ve seen a rapid cadence of notable AI-related announcements and releases from both Google and OpenAI, and it’s been making the AI community’s head spin. It has also poured fuel on the fire of the OpenAI-Google rivalry, an accelerating game of one-upmanship taking place unusually close to the Christmas holiday.

“How are people surviving with the firehose of AI updates that are coming out,” wrote one user on X last Friday, which is still a hotbed of AI-related conversation. “in the last <24 hours we got gemini flash 2.0 and chatGPT with screenshare, deep research, pika 2, sora, chatGPT projects, anthropic clio, wtf it never ends."

Rumors travel quickly in the AI world, and people in the AI industry had been expecting OpenAI to ship some major products in December. Once OpenAI announced “12 days of OpenAI” earlier this month, Google jumped into gear and seemingly decided to try to one-up its rival on several counts. So far, the strategy appears to be working, but it’s coming at the cost of the rest of the world being able to absorb the implications of the new releases.

“12 Days of OpenAI has turned into like 50 new @GoogleAI releases,” wrote another X user on Monday. “This past week, OpenAI & Google have been releasing at the speed of a new born startup,” wrote a third X user on Tuesday. “Even their own users can’t keep up. Crazy time we’re living in.”

“Somebody told Google that they could just do things,” wrote a16z partner and AI influencer Justine Moore on X, referring to a common motivational meme telling people they “can just do stuff.”

The Google AI rush

OpenAI’s “12 Days of OpenAI” campaign has included releases of their full o1 model, an upgrade from o1-preview, alongside o1-pro for advanced “reasoning” tasks. The company also publicly launched Sora for video generation, added Projects functionality to ChatGPT, introduced Advanced Voice features with video streaming capabilities, and more.

The AI war between Google and OpenAI has never been more heated Read More »

google-will-apparently-offer-“ai-mode”-right-on-its-main-search-page

Google will apparently offer “AI Mode” right on its main search page

Google will soon take more steps to make AI a part of search, exposing more users to its Gemini agent, according to recent reports and app teardowns.

“AI Mode,” shown at the top left of the web results page and inside the Google app, will provide an interface similar to a Gemini AI chat, according to The Information.

This tracks with a finding from Android Authority earlier this month, which noted a dedicated “AI mode” button inside an early beta of the Google app. This shortcut also appeared on Google’s Android search widget, and a conversation history button was added to the Google app. Going even deeper into the app, 9to5Google found references to “aim” (AI mode) and “ai_mode” which suggest a dedicated tab in the Google app, with buttons for speaking to an AI or sending it pictures.

Google already promotes Gemini with links below its search homepage. (“5 ways Gemini can help during the Holidays” is currently showing for me.) Search results on Google can also contain an “AI Overview,” which launched with some “use glue for pizza sauce” notoriety. People averse to AI answers can avoid them with URL parameters and proxy sites (or sticking to the “web” tab). Gemini has also been prominently added to other Google products, like Pixel phones, Gmail, and Drive/Workspace. And the search giant has also been testing the ability to attach files to a web search for analysis.

Google will apparently offer “AI Mode” right on its main search page Read More »

not-to-be-outdone-by-openai,-google-releases-its-own-“reasoning”-ai-model

Not to be outdone by OpenAI, Google releases its own “reasoning” AI model

Google DeepMind’s chief scientist, Jeff Dean, says that the model receives extra computing power, writing on X, “we see promising results when we increase inference time computation!” The model works by pausing to consider multiple related prompts before providing what it determines to be the most accurate answer.

Since OpenAI’s jump into the “reasoning” field in September with o1-preview and o1-mini, several companies have been rushing to achieve feature parity with their own models. For example, DeepSeek launched DeepSeek-R1 in early November, while Alibaba’s Qwen team released its own “reasoning” model, QwQ earlier this month.

While some claim that reasoning models can help solve complex mathematical or academic problems, these models might not be for everybody. While they perform well on some benchmarks, questions remain about their actual usefulness and accuracy. Also, the high computing costs needed to run reasoning models have created some rumblings about their long-term viability. That high cost is why OpenAI’s ChatGPT Pro costs $200 a month, for example.

Still, it appears Google is serious about pursuing this particular AI technique. Logan Kilpatrick, a Google employee in its AI Studio, called it “the first step in our reasoning journey” in a post on X.

Not to be outdone by OpenAI, Google releases its own “reasoning” AI model Read More »

crypto-scammers-posing-as-real-brands-on-x-are-easily-hacking-youtubers

Crypto scammers posing as real brands on X are easily hacking YouTubers

“I’m fighting with Google now,” Townsend told Ars. “I don’t expect any real answers from them.”

How YouTubers can avoid being targeted

As YouTube appears evasive, Townsend has been grateful for long-time subscribers commenting to show support, which may help get his videos amplified more by the algorithm. On YouTube, he also said that because “the outpouring of support was beyond anything” he could’ve expected, it kept him “sane” through sometimes 24-hour periods of silence without any updates on when his account would be restored.

Townsend told Ars that he rarely does sponsorships, but like many in the fighting game community, his inbox gets spammed with offers constantly, much of which he assumes are scams.

“If you are a YouTuber of any size,” Townsend explained in his YouTube video, “you are inundated with this stuff constantly,” so “my BS detector is like, okay, fake, fake, fake, fake, fake, fake, fake. But this one just, it looked real enough, like they had their own social media presence, lots of followers. Everything looked real.”

Brian_F echoed that in his video, which breaks down how the latest scam evolved from more obvious scams, tricking even skeptical YouTubers who have years of experience dodging phishing scams in their inboxes.

“The game has changed,” Brian_F said.

Townsend told Ars that sponsorships are rare in the fighting game community. YouTubers are used to carefully scanning supposed offers to weed out the real ones from the fakes. But Brian_F’s video pointed out that scammers copy/paste legitimate offer letters, so it’s already hard to distinguish between potential sources of income and cleverly masked phishing attacks using sponsorships as lures.

Part of the vetting process includes verifying links without clicking through and verifying identities of people submitting supposed offers. But if YouTubers are provided with legitimate links early on, receiving offers from brands they really like, and see that contacts match detailed LinkedIn profiles of authentic employees who market the brand, it’s much harder to detect a fake sponsorship offer without as many obvious red flags.

Crypto scammers posing as real brands on X are easily hacking YouTubers Read More »

google-steps-into-“extended-reality”-once-again-with-android-xr

Google steps into “extended reality” once again with Android XR

Citing “years of investment in AI, AR, and VR,” Google is stepping into the augmented reality market once more with Android XR. It’s an operating system that Google says will power future headsets and glasses that “transform how you watch, work, and explore.”

The first version you’ll see is Project Moohan, a mixed-reality headset built by Samsung. It will be available for purchase next year, and not much more is known about it. Developers have access to the new XR version of Android now.

“We’ve been in this space since Google Glass, and we have not stopped,” said Juston Payne, director of product at Google for XR in Android XR’s launch video. Citing established projects like Google Lens, Live View for Maps, instant camera translation, and, of course, Google’s general-purpose Gemini AI, XR promises to offer such overlays in both dedicated headsets and casual glasses.

Android XR announcement video.

There are few additional details right now beyond a headset rendering, examples in Google’s video labeled as “visualization for concept purposes.” Google’s list of things that will likely be on board includes Gemini, Maps, Photos, Translate, Chrome, Circle to Search, and Messages. And existing Android apps, or at least those updated to do so, should make the jump, too.

Google steps into “extended reality” once again with Android XR Read More »

ai-#94:-not-now,-google

AI #94: Not Now, Google

At this point, we can confidently say that no, capabilities are not hitting a wall. Capacity density, how much you can pack into a given space, is way up and rising rapidly, and we are starting to figure out how to use it.

Not only did we get o1 and o1 pro and also Sora and other upgrades from OpenAI, we also got Gemini 1206 and then Gemini Flash 2.0 and the agent Jules (am I the only one who keeps reading this Jarvis?) and Deep Research, and Veo, and Imagen 3, and Genie 2 all from Google. Meta’s Llama 3.3 dropped, claiming their 70B is now as good as the old 405B, and basically no one noticed.

This morning I saw Cursor now offers ‘agent mode.’ And hey there, Devin. And Palisade found that a little work made agents a lot more effective.

And OpenAI partnering with Anduril on defense projects. Nothing to see here.

There’s a ton of other stuff, too, and not only because this for me was a 9-day week.

Tomorrow I will post about the o1 Model Card, then next week I will follow up regarding what Apollo found regarding potential model scheming. I plan to get to Google Flash after that, which should give people time to try it out. For now, this post won’t cover any of that.

I have questions for OpenAI regarding the model card, and asked them for comment, but press inquiries has not yet responded. If anyone there can help, please reach out to me or give them a nudge. I am very concerned about the failures of communication here, and the potential failures to follow the preparedness framework.

Previously this week: o1 turns Pro.

  1. Table of Contents.

  2. Language Models Offer Mundane Utility. Cursor gets an agent mode.

  3. A Good Book. The quest for an e-reader that helps us read books the right way.

  4. Language Models Don’t Offer Mundane Utility. Some are not easily impressed.

  5. o1 Pro Versus Claude. Why not both? An o1 (a1?) built on top of Sonnet, please.

  6. AGI Claimed Internally. A bold, and I strongly believe incorrect, claim at OpenAI.

  7. Ask Claude. How to get the most out of your conversations.

  8. Huh, Upgrades. Canvas, Grok Aurora, Gemini 1206, Llama 3.3.

  9. All Access Pass. Context continues to be that which is scarce.

  10. Fun With Image Generation. Sora, if you can access it. Veo, Imagen 3, Genie 2.

  11. Deepfaketown and Botpocalypse Soon. Threats of increasing quantity not quality.

  12. They Took Our Jobs. Attempt at a less unrealistic economic projection.

  13. Get Involved. EU AI office, Apollo Research, Conjecture.

  14. Introducing. Devin, starting at $500/month, no reports of anyone paying yet.

  15. In Other AI News. The rapid rise in capacity density.

  16. OpenlyEvil AI. OpenAI partners with Anduril Industries for defense technology.

  17. Quiet Speculations. Escape it all. Maybe go to Thailand? No one would care.

  18. Scale That Wall. Having the model and not releasing is if anything scarier.

  19. The Quest for Tripwire Capability Thresholds. Holden Karnofsky helps frame.

  20. The Quest for Sane Regulations. For now it remains all about talking the talk.

  21. Republican Congressman Kean Brings the Fire. He sat down and wrote a letter.

  22. CERN for AI. Miles Brundage makes the case for CERN for AI, sketches details.

  23. The Week in Audio. Scott Aaronson on Win-Win.

  24. Rhetorical Innovation. Yes, of course the AIs will have ‘sociopathic tendencies.’

  25. Model Evaluations Are Lower Bounds. A little work made the agents better.

  26. Aligning a Smarter Than Human Intelligence is Difficult. Anthropic gets news.

  27. I’ll Allow It. We are still in the era where it pays to make systematic errors.

  28. Frontier AI Systems Have Surpassed the Self-Replicating Red Line. Says paper.

  29. People Are Worried About AI Killing Everyone. Chart of p(doom).

  30. Key Person Who Might Be Worried About AI Killing Everyone. David Sacks.

  31. Other People Are Not As Worried About AI Killing Everyone. Bad modeling.

  32. Not Feeling the AGI. If AGI wasn’t ever going to be a thing, I’d build AI too.

  33. Fight For Your Right. Always remember to backup your Sims.

  34. The Lighter Side. This is your comms department.

TIL Cursor has an agent mode?

fofr: PSA: Cursor composer is next to the chat tab, and you can toggle the agent mode in the bottom right.

Noorie: Agent mode is actually insane.

Create a dispute letter when your car rental company tries to rob you.

Sam McAllister: We’ve all faced that mountain of paperwork and wanted to throw in the towel. Turns out, Claude is a pretty great, cool-headed tool for thought when you need to dig in and stand your ground.

[they try to deny his coverage, he feeds all the documentation into Claude, Claude analyzes the actual terms of the contract, writes dispute letter.]

Patrick McKenzie: We are going to hear many more stories that echo this one.

One subvariant of them is that early adopters of LLMs outside of companies are going to tell those companies *things they do not know about themselves*.

People often diagnose malice or reckless indifference in a standard operating procedure (SOP) that misquotes the constellation of agreements backing, for example, a rental contract.

Often it is more of a “seeing like a really big business” issue than either of those. Everyone did their job; the system, as a whole, failed.

I remain extremely pleased that people keep reporting to my inbox that “Write a letter in the style of patio11’s Dangerous Professional” keeps actually working against real problems with banks, credit card companies, and so on.

It feels like magic.

Oh, a subvariant of this: one thing presenting like an organized professional can do is convince other professionals (such as a school’s risk office) to say, “Oh, one of us! Let’s help!” rather than, “Sigh, another idiot who can’t read.”

When we do have AI agents worthy of the name, that can complete complex tasks, Aaron Levine asks the good question of how should we price them? Should it be like workers, where we pay for a fixed amount of work? On a per outcome basis? By the token based on marginal cost? On a pure SaaS subscription model with fixed price per seat?

It is already easy to see, in toy cases like Cursor, that any mismatch between tokens used versus price charged will massively distort user behavior. Cursor prices per query rather than per token, and even makes you wait on line for each one if you run out, which actively pushes you towards the longest possible queries with the longest possible context. Shift to an API per-token pricing model and things change pretty darn quick, where things that cost approximately zero dollars can be treated like they cost approximately zero dollars, and the few things that don’t can be respected.

My gut says that for most purposes, those who create AI agents will deal with people who don’t know or want to know how costs work under the hood or optimize for them too carefully. They’ll be happy to get a massive upgrade in performance and cost, and a per-outcome or per-work price or fixed seat price will look damn good even while the provider has obscene unit economics. So things will go that way – you pay for a service and feel good about it, everyone wins.

Already it is like this. Whenever I look at actual API costs, it is clear that all the AI companies are taking me to the cleaners on subscriptions. But I don’t care! What I care about is getting the value. If they charge mostly for that marginal ease in getting the value, why should I care? Only ChatGPT Pro costs enough to make this a question, and even then it’s still cheap if you’re actually using it.

Also consider the parallel to many currently free internet services, like email or search or maps or social media. Why do I care that the marginal cost to provide it is basically zero? I would happily pay a lot for these services even if it only made them 10% better. If it made them 10x better, watch out. And anyone who wouldn’t? You fool!

The Boring News combines prediction markets at Polymarket with AI explanations of the odds movements to create a podcast news report. What I want is the text version of this. Don’t give me an AI-voiced podcast, give me a button at Polymarket that says ‘generate an AI summary explaining the odds movements,’ or something similar. It occurs to me that building that into a Chrome extension to utilize Perplexity or ChatGPT probably would not be that hard?

Prompting 101 from she who would know:

Amanda Askell (Anthropic): The boring yet crucial secret behind good system prompts is test-driven development. You don’t write down a system prompt and find ways to test it. You write down tests and find a system prompt that passes them.

For system prompt (SP) development you:

– Write a test set of messages where the model fails, i.e. where the default behavior isn’t what you want

– Find an SP that causes those tests to pass

– Find messages the SP is misapplied to and fix the SP

– Expand your test set & repeat

Joanna Stern looks in WSJ at iOS 18.2 and its AI features, and is impressed, often by things that don’t seem that impressive? She was previously also impressed with Gemini Live, which I have found decidedly unimpressive.

All of this sounds like a collection of parlor tricks, although yes this includes some useful tricks. So maybe that’s not bad. I’m still not impressed.

Here’s a fun quirk:

Joanna Stern: Say “Hey Siri, a meatball recipe,” and Siri gives you web results.

But say “Hey Siri, give me a meatball recipe” and ChatGPT reports for duty. These other phrases seem to work.

• “Write me…” A poem, letter, social-media post, you name it. You can do this via Siri or highlight text anywhere, tap the Writing Tools pop-up, tap Compose, then type your writing prompt.

• “Brainstorm…” Party ideas for a 40-year-old woman, presents for a 3-year-old, holiday card ideas. All work—though I’ll pass on the Hawaiian-themed bash.

• “Ask ChatGPT to…” Explain why leaves fall in the autumn, list the top songs from 1984, come up with a believable excuse for skipping that 40-year-old woman’s party.

ChatGPT integration with Apple Intelligence was also day 5 of the 12 days of OpenAI. In terms of practical trinkets, more cooking will go a long way. For example, their demo includes ‘make me a playlist’ but then can you make that an instant actual playlist in Apple Music (or Spotify)? Why not?

As discussed in the o1 post, LLMs greatly enhance reading books.

Dan Shipper: I spend a significant amount of my free time reading books with ChatGPT / Claude as a companion and I feel like I’m getting a PhD for $20 / month

Andrej Karpathy: One of my favorite applications of LLMs is reading books together. I want to ask questions or hear generated discussion (NotebookLM style) while it is automatically conditioned on the surrounding content. If Amazon or so built a Kindle AI reader that “just works” imo it would be a huge hit.

For now, it is possible to kind of hack it with a bunch of script. Possibly someone already tried to build a very nice AI-native reader app and I missed it.

don’t think it’s Meta glasses I want the LLM to be cleverly conditioned on the entire book and maybe the top reviews too. The glasses can’t see all of this. Is why I suggested Amazon is in good position here because they have access to all this content directly.

Anjan Katta: We’re building exactly this at @daylightco!

Happy to demo to you in person.

Tristan: you can do this in @readwisereader right now 🙂 works on web/desktop/ios/android, with any ePubs, PDFs, articles, etc

Curious if you have any feedback!

Flo Crivello: I think about this literally every day. Insane that ChatGPT was released 2yrs ago and none of the major ebook readers has incorporated a single LLM feature yet, when that’s one of the most obvious use cases.

Patrick McKenzie: This would require the ebook reader PMs to be people who read books, a proposition which I think we have at least 10 years of evidence against.

It took Kindle *how many yearsto understand that “some books are elements of an ordered set called by, and this appears to be publishing industry jargon, a series.”

“Perhaps, and I am speculating here, that after consuming one item from the ordered set, a reader might be interested in the subsequent item from an ordered set. I am having trouble imagining the user story concretely though, and have never met a reader myself.”

I want LLM integration. I notice I haven’t wanted it enough to explore other e-readers, likely because I don’t read enough books and because I don’t want to lose the easy access (for book reviews later) of Kindle notes.

But the Daylight demo of their upcoming AI feature does look pretty cool here, if the answer quality is strong, which it should be given they’re using Claude Sonnet. Looks like it can be used for any app too, not only the reader?

I don’t want an automated discussion, but I do want a response and further thoughts from o1 or Claude. Either way, yes, seems like a great thing to do with an e-reader.

This actually impressed me enough that I pulled the trigger and put down a deposit, as I was already on the fence and don’t love any of my on-the-go computing solutions.

(If anyone at Daylight wants to move me up the queue, I’ll review it when I get one.)

Know how many apples you have when it lacks any way to know the answer. Qw-32B tries to overthink it anyway.

Eliezer Yudkowsky predicts on December 4 that there will not be much ‘impressive or surprising’ during the ‘12 days of OpenAI.’ That sounds bold, but a lot of that is about expectations, as he says Sora, o1 or agents, and likely even robotics, would not be so impressive. In which case, yeah, tough to be impressed. I would say that you should be impressed if those things exceed expectations, and it does seem like collectively o1 and o1 pro did exceed general expectations.

Weird that this is a continuing issue at all, although it makes me realize I never upload PDFs to ChatGPT so I wouldn’t know if they handle it well, that’s always been a Claude job:

Gallabytes: Why is Anthropic the only company with good PDF ingestion for its chatbot? Easily half of my Claude chats are referring to papers.

ChatGPT will use some poor Python library to read 100 words. And only 4o, not o1.

How much carbon do AI images require? Should you ‘fing stop using AI’?

I mean, no.

Community notes: One AI image takes 3 Wh of electricity. This takes 1mn in e.g. Midjourney. Doing this for 24h costs 4.3 kWh. This releases 0.5kg of CO2: same as driving 3 miles in your car. Overall, cars emit 10% of world CO2, and AI 0.04% (data centers emit 1%1/25th is AI)

I covered reactions to o1 earlier this week, but there will be a steady stream coming in.

Mostly my comments section was unimpressed with o1 and o1 pro in practice.

A theme seems to be that when you need o1 then o1 is tops, but we are all sad that o1 is built on GPT-4o instead of Sonnet, and for most purposes it’s still worse?

Gallabytes: Cursor Composer + Sonnet is still much better for refactors and simpler tasks. Once again, wishing for an S1 based on Sonnet 3.6 instead of an o1 based on 4.0.

Perhaps o1-in-Cursor will be better, but the issues feel more like the problems inherent in 4.0 than failed reasoning.

o1 truly is a step up for more challenging tasks, and in particular is better at debugging, where Sonnet tends to fabricate solutions if it cannot figure things out immediately.

Huge if true!

Vahid Kazemi (OpenAI): In my opinion we have already achieved AGI and it’s even more clear with o1. We have not achieved “better than any human at any task” but what we have is “better than most humans at most tasks”.

Some say LLMs only know how to follow a recipe.

Firstly, no one can really explain what a trillion parameter deep neural net can learn. But even if you believe that, the whole scientific method can be summarized as a recipe: observe, hypothesize, and verify. Good scientists can produce better hypothesis based on their intuition, but that intuition itself was built by many trial and errors. There’s nothing that can’t be learned with examples.

I mean, look, no. That’s not AGI in the way I understand the term AGI at all, and Vahid is even saying they had it pre-o1. But of course different people use the term differently.

What I care about, and you should care about, is the type of AGI that is transformational in a different way than AIs before it, whatever you choose to call that and however you define it. We don’t have that yet.

Unless you’re OpenAI and trying to get out of your Microsoft contract, but I don’t think that is what Vahid is trying to do here.

Is it more or less humiliating than taking direction from a smarter human?

Peter Welinder: It’s exhilarating—and maybe a bit humiliating—to take direction from a model that’s clearly smarter than you.

At least in some circles, it is the latest thing to do, and I doubt o1 will do better here.

Aella: What’s happening suddenly everybody around me is talking to Claude all the time. Consulting it on life decisions, on fights with partners, getting general advice on everything. And it’s Claude, not chatGPT.

My friend was in a fight with her boyfriend; she told me she told Claude everything and took his side. She told her boyfriend to talk to Claude, and it also took her side. My sister entered the room, unaware of our conversation: “So, I’ve been talking to Claude about this boy.”

My other friend, who is also having major boy troubles, spent many hours a day for several weeks talking to Claude. Whenever I saw her, her updates were often like, “The other day, Claude made this really good point, and I had this emotional shift.”

Actually, is it all my friends or just my girlfriends who are talking to Claude about their boy problems? Because now that I think of it, that’s about 90 percent of what’s happening when I hear a friend referencing Claude.

Sithamet: To all those who say “Claude is taking the female side,” I actually tried swapping genders in stories to see if it impacts his behavior. He is gender-neutral and simply detects well and hates manipulations, emotional abuse, and such.

Katherine Dee: I have noticed [that if you ask ‘are you sure’ it changes its answer] myself; it is making me stop trusting it.

Wu Han Solo: Talking to an LLM like Claude is like talking to a mirror. You have to be really careful not to “poison it” with your own biases.

It’s too easy to get an LLM to tell you what you want to hear and subtly manipulate its outputs. If one does not recognize this, I think that’s problematic.

There are obvious dangers, but mostly this seems very good. The alternative options for talking through situations and getting sanity checks are often rather terrible.

Telling the boyfriend to talk to Claude as well is a great tactic, because it guards against you having led the witness, and also because you can’t take the request back if it turns out you did lead the witness. It’s an asymmetric weapon and costly signal.

What else to do about the ‘leading the witness’ issue? The obvious first thing to do if you don’t want this is… don’t lead the witness. There’s no one watching. Friends will do this as well, if you want them to be brutally honest with you then you have to make it clear that is what you want, if you mostly want them to ‘be supportive’ or agree with you then you mostly can and will get that instead. Indeed, it is if anything far easier to accidentally get people to do this when you did not want them to do it (or fail to get it, when you did want it).

You can also re-run the scenario or question with different wording in new windows, if you’re worried about this. And you can use ‘amount of pushback you find the need to use and how you use it’ as good information about what you really want, and good information to send to Claude, which is very good at picking up on such signals. The experience is up to you.

Sometimes you do want lies? We’ve all heard requests to ‘be supportive,’ so why not have Claude do this too, if that’s what you want in a given situation? It’s your life. If you want the AI to lie to you, I’d usually advise against that, but it has its uses.

You can also observe exactly how hard you have to push to get Claude to cave in a given situation, and calibrate based on that. If a simple ‘are you sure?’ changes its mind, then that opinion was not so strongly held. That is good info.

Others refuse to believe that Claude can provide value to people in ways it is obviously providing value, such as here where Hazard tells QC that QC can’t possibly be experiencing what QC is directly reporting experiencing.

I especially appreciated this:

Q&C: When you do not play to Claude’s strengths or prompt it properly, it seems like it is merely generically validating you. But that is completely beside the point. It is doing collaborative improvisation. It is the ultimate yes-and-er. In a world full of criticism, it is willing to roll with you.

And here’s the part of Hazard’s explanation that did resonate with QC, and it resonates with me as well very much:

Hazard: I feel that the phenomenon that Q.C. and others are calling a “presence of validation/support” is better described as an “absence of threat.” Generally, around people, there is a strong ambient sense of threat, but talking to Claude does not trigger that.

From my own experience, bingo, sir. Whenever you are dealing with people, you are forced to consider all the social implications, whether you want to or not. There’s no ‘free actions’ or truly ‘safe space’ to experiment or unload, no matter what anyone tells you or how hard they try to get as close to that as possible. Can’t be done. Theoretically impossible. Sorry. Whereas with an LLM, you can get damn close (there’s always some non-zero chance someone else eventually sees the chat).

The more I reason this stuff out, the more I move towards ‘actually perhaps I should be using Claude for emotional purposes after all’? There’s a constantly growing AI-related list of things I ‘should’ be using them for, because there are only so many hours in the day.

Tracing Woods has a conversation with Claude about stereotypes, where Claude correctly points out that in some cases correlations exist and are useful, actually, which leads into discussion of Claude’s self-censorship.

Ademola: How did you make it speak like that.

Tracing Woods: Included in my first message, that conversation: “Be terse, witty, ultra-intelligent, casual, and razor-sharp. Use lowercase and late-millennial slang as appropriate.”

Other than, you know, o1, or o1 Pro, or Gemini 2.0.

For at least a brief shining moment, Gemini-1206 came roaring back (available to try here in Google Studio) to for a third time claim the top spot on Arena, this time including all domains. Whatever is happening at Google, they are rapidly improving scores on a wide variety of domains, this time seeing jumps in coding and hard prompts where presumably it is harder to accidentally game the metric. And the full two million token window is available.

It’s impossible to keep up and know how well each upgrade actually does, with everything else that’s going on. As far as I can tell, zero people are talking about it.

Jeff Dean (Chief Scientist, Google): Look at the Pareto frontier of the red Gemini/Gemma dots. At a given price point, the Gemini model is higher quality. At a given quality (ELO score), the Gemini/Gemma model is the cheapest alternative.

We haven’t announced prices for any of the exp models (and may not launch exactly these models as paid models), so the original poster made some assumptions.

OpenAI offers a preview of Reinforcement Finetuning of o1 (preview? That’s no shipmas!), which Altman says was a big surprise of 2024 and ‘works amazingly well.’ They introduced fine tuning with a livestream rather than text, which is always frustrating. The use case is tuning for a particular field like law, insurance or a branch of science, and you don’t need many examples, perhaps as few as 12. I tried to learn more from the stream, but it didn’t seem like it gave me anything to go on. We’ll have to wait and see when we get our hands on it, you can apply to the alpha.

OpenAI upgrades Canvas, natively integrating it into GPT-4o, adding it to the free tier, making it available with custom GPTs, giving it a Show Change feature (I think this is especially big in practice), letting ChatGPT add comments and letting it directly execute Python code. Alas, Canvas still isn’t compatible with o1, which limits its value quite a bit.

Sam Altman (who had me and then lost me): canvas is now available to all chatgpt users, and can execute code!

more importantly it can also still emojify your writing.

Llama 3.3-70B is out, which Zuck claims is about as good at Llama 3.2-405B.

xAI’s Grok now available to free Twitter users, 10 questions per 2 hours, and they raised another $6 billion.

What is an AI agent? Here’s Sully’s handy guide.

Tristan Rhodes: Good idea! Here is my take this.

An AI agent must have at least two visits to an LLM:

– One prompt completes the desired work

– One prompt decides if the work is complete. If complete, format the output. If not, perform the first prompt again, with refined input.

Sully: yes agreed this generally has to be the case.

As one response suggests: Want an instant agent? Just add duct tape.

I continue to think this is true especially when you add in agents, which is one reason Apple Intelligence has so far been so disappointing. It was supposed to be a solution to this problem. So far, the actual attempted solutions have sucked.

Rasmus Fonnesbaek: What ~95% of people need from AI tools for them to be helpful is for the tools to have access to most or all their personal data for context (i.e. all emails, relevant documents, etc.) and/or to ask for the right info — and most people still don’t feel comfortable sharing that!

While models’ context windows (memory) still have some limitations, these are not frontier technology-related in nature — but legal (confidentiality, sensitive data protection, etc.) and privacy-related, with some of those mitigable by very strong cybersecurity measures.

This suggests that Microsoft, Alphabet, Apple, Meta, Dropbox, etc. — holding large-scale existing, relevant data stores for people, with very strong cybersecurity — are best-positioned to provide AI tools with very high “mundane utility.”

[thread continues]

The 5% who need something else are where the world transforms, but until then most people greatly benefit from context. Are you willing to give it to them? I’ve already essentially made the decision to say Yes to Google here, but their tools aren’t good enough yet. I am also pretty sure I’d be willing to trust Anthropic. Some of the others, let us say, not so much.

OpenAI gives us Sora Turbo, a faster version of Sora now available to Plus and Pro, at no additional charge, on day one demand was so high that the servers were clearly overloaded, and they disabled signups, which includes those who already have Plus trying to sign up for Sora. More coverage later once people have actually tried it.

Users can generate videos up to 1080p resolution, up to 20 sec long, and in widescreen, vertical or square aspect ratios. You can bring your own assets to extend, remix, and blend, or generate entirely new content from text.

If you want 1080p you’ll have to go Pro (as in $200/month), the rest of us get 50 priority videos in 720p, I assume per month.

The United Kingdom, EU Economic Area and Switzerland are excluded.

Sam Altman: We want to [offer Sora in the EU]!

We want to offer our products in Europe, and believe a strong Europe is important to the world.

We also have to comply with regulations.

I would generally expect us to have delayed launches for new products in Europe, and that there may be some we just cannot offer.

Everyone was quick to blame the EU delay on the EU AI Act, but actually the EU managed to mess this up earlier – this is (at minimum also) about the EU Digital Markets Act and GPDR.

The Sora delay does not matter on its own, and might partly be strategic in order to impact AI regulation down the line. They’re overloaded anyway and video generation is not so important.

Sam Altman: We significantly underestimated demand for Sora; it is going to take awhile to get everyone access.

Trying to figure out how to do it as fast as possible!

But yes, the EU is likely to see delays on model releases going forward, and to potentially not see some models at all.

If you’re wondering why it’s so overloaded, it’s probably partly people like Colin Fraser tallying up all his test prompts.

George Pickett: I’m also underwhelmed. But I’ve also seen it create incredible outputs. Might it be that it requires a fundamentally different way of prompting?

My guess is that Sora is great if you want a few seconds of something cool, and not as great if you want something specific. The more flexible you are, the better you’ll do.

This is the Sora system card, which is mostly about mitigation, especially of nudity.

Sora’s watermarking is working, if you upload to LinkedIn it will show the details.

Google offers us Veo and Imagen 3, new video and image generation models. As usual with video, my reaction to Veo is that it seems to produce cool looking very short video clips and that’s cool but it’s going to be a while before it matters.

As usual for images, my reaction to Imagen 3 is that the images look cool and the control features seem neat, if you want AI images. But I continue to not feel any pull to generate cool AI images nor do I see anyone else making great use of them either.

In addition, this is a Google image project, so you know it’s going to be a stickler about producing specific faces and things like that and generally be no fun. It’s cool in theory but I can’t bring myself to care in practice.

If there’s a good fully uncensored image generator that’s practical to run locally with only reasonable amounts of effort, I have some interest in that, please note in the comments. Or, if there’s one that can actually take really precise commands and do exactly what I ask, even if it has to be clean, then I’d check that out too, but it would need to be very good at that before I cared enough.

Short of those two, mostly I just want ‘good enough’ images for posts and powerpoints and such, and DALL-E is right there and does fine and I’m happy to satisfice.

Whereas Grok Aurora, the new xAI image model focusing on realism, goes exactly the other way. It is seeking to portray as many celebrities as it can, as accurately as possible, as part of that realism. It was briefly available on December 7, then taken down the next day, perhaps due to concerns about its near total (and one presumes rather intentional) lack of filters. Then on the 9th it was so back?

Google presents Genie 2, which they claim can generate a diverse array of consistent worlds, playable for up to a minute, potentially unlocking capabilities for embedded agents. It looks cool, and yes, if you wanted to scale environments to train embedded agents you’ll eventually want something like this. Does seem like early days, for now I don’t see why you wouldn’t use existing solutions, but it always starts out that way.

Will we have to worry about people confusing faked videos for real ones?

Parth: Our whole generation will spend a significant amount of time explaining to older people how these videos are not real.

Maxwell Tabarrok: I think this is simply not true. We have had highly realistic computer imagery for decades. AI brings down the already low cost.

No one thinks the “Avengers” movies are real, even though they are photorealistic.

Cognitive immune systems can resist this easily.

Gwern: People think many things in movies are real that are not. I am always shocked to watch visual effects videos. “Yes, the alien is obviously not real, but that street in Paris is real”—My brother, every pixel in that scene was green screen except the protagonist’s face.

There is very much a distinctive ‘AI generated’ vibe to many AI videos, and often there are clear giveaways beyond that. But yeah, people get fooled by videos all the time, the technology is there in many cases, and AI tech will also get there. And once the tech gets good enough, when you have it create something that looks realistic, people will start getting fooled.

Amazon seeing cyber threats per day grow from 100 million seven months ago to over 750 million today.

C.J. Moses (Chief Information Security Officer, Amazon): Now, it’s more ubiquitous, such that normal humans can do things they couldn’t do before because they just ask the computer to do that for them.

We’re seeing a good bit of that, as well as the use of AI to increase the realness of phishing, and things like that. They’re still not there 100%. We still can find errors in every phishing message that goes out, but they’re getting cleaner.

In the last eight months, we’ve seen nation-state actors that we previously weren’t tracking come onto the scene. I’m not saying they didn’t exist, but they definitely weren’t on the radar. You have China, Russia and North Korea, those types of threat actors. But then you start to see the Pakistanis, you see other nation-states. We have more players in the game than we ever did before.

They are also using AI defensively, especially via building a honeypot network and using AI to analyze the resulting data. But at this particular level in this context, it seems AI favors offense, because Amazon already was doing the things AI can help with, whereas many potential attackers benefit from this kind of ‘catch up growth.’ Amazon’s use of AI is therefore largely to detect and defend against others use of AI.

The good news is that this isn’t a 650% growth in the danger level. The new cyber attacks are low marginal cost, relatively low skill and low effort, and therefore should on average be far less effective and damaging. The issue is, if they grow on an exponential, and the ‘discount rate’ on effectiveness shrinks, they still would be on pace to rapidly dominate the threat model.

Nikita Bier gets optimistic on AI and social apps, predicts AI will be primarily used to improve resolution of communication and creative tools rather than for fake people, whereas ‘AI companions’ won’t see widespread adaptation. I share the optimism about what people ultimately want, but worry that such predictions are like many others about AI, extrapolating from impacts of other techs without noticing what is different or actually gaming out what happens.

An attempt at a more serious economic projection for AGI? It is from Anton Korinek via the international monetary fund, entitled ‘AI may be on a trajectory to surpass human intelligence; we should be prepared.’

As in, AGI arrives in either 5 or 20 years, and wages initially outperform but then start falling below baseline shortly thereafter, and fall from there. This ‘feels wrong’ for worlds that roughly stay intact somehow in the sense that movement should likely be relative to the blue line not the x-axis, but the medium term result doesn’t change, wages crash.

They ask whether there is an upper bound on the complexity of what a human brain can process, based on our biology, versus what AIs would allow. That’s a great question. An even better question is where the relative costs including time get prohibitive, and whether we will stay competitive (hint if AI stays on track: no).

They lay out three scenarios, each with >10% probability of happening.

  1. In the traditional scenario, which I call the ‘AI fizzle’ world, progress stalls before we reach AGI, and AI is a lot more like any other technology.

  2. Their baseline scenario, AGI in 20 years due to cognitive limits.

  3. AGI in 5 years, instead.

Even when it is technologically possible to replace workers, society may choose to keep humans in certain functions—for example, as priests, judges, or lawmakers. The resulting “nostalgic” jobs could sustain demand for human labor in perpetuity (Korinek and Juelfs, forthcoming).

To determine which AI scenario the future most resembles as events unfold, policymakers should monitor leading indicators across multiple domains, keeping in mind that all efforts to predict the pace of progress face tremendous uncertainty.

Useful indicators span technological benchmarks, levels of investment flowing into AI development, adoption of AI technologies throughout the economy, and resulting macroeconomic and labor market trends.

Major points for realizing that the scenarios exist and one needs to figure out which one we are in. This is still such an economist method for trying to differentiate the scenarios. How fast people choose to adapt current AI outside of AI R&D itself does not correlate much with whether we are on track for AGI – it is easy to imagine people being quick to incorporate current AI into their workflows and getting big productivity boosts while frontier progress fizzles, or people continuing to be dense and slow about adaptation while capabilities race forward.

Investment in AI development is a better marker, but the link between inputs and outputs, and the amount of input that is productive, are much harder to predict, and I am not convinced that AI investment will correctly track the value of investment in AI. The variables that determine our future are more about how investment translates into capabilities.

Even with all the flaws this is a welcome step from an economist.

The biggest flaw, of course, is not to notice that if AGI is developed that this either risks humans losing control or going extinct or enabling rapid development of ASI.

Anton recognizes one particular way in which AGI is a unique technology, its ability to generate unemployment via automating labor tasks to the point where further available tasks are not doable by humans, except insofar as we choose to shield them as what he calls ‘nostalgic’ jobs. But he doesn’t realize that is a special case of a broader set of transformations and dangers.

How will generative AI impact the law? In all sorts of ways, but Henry Thompson focuses specifically on demand for legal services and disputes themselves, holding other questions constant. Where there are contracts, he reasons that AI leads to superior contracts that are more robust and complete, which reduces litigation.

But it also gives people more incentive to litigate and not to settle, although if it is doing that by reducing costs then perhaps we do not mind so much, actually resolving disputes is a benefit not only a cost. And in areas where contracts are rare, including tort law, the presumption is litigation will rise.

More abstractly, AI reduces costs for all legal actions and services, on both sides, including being able to predict outcomes. As the paper notices, the relative reductions in costs are hard to predict, so net results are hard to predict, other than that uncertainty should be reduced.

EU AI Office is looking for a lead scientific advisor (must be an EU citizen), deadline December 13. Unfortunately, the eligibility requirements include ‘professional experience of at least 15 years’ while paying 13.5k-15k euros a month, which rules out most people who you would want.

Michael Nielsen: It’s frustrating to see this. I’d be surprised if 10% of the OpenAI or Anthropic research / engineering staff are considered “qualified” [sic] for this job. And I’ll bet 90% make more than this, some of them far more (10x etc). It just seems clueless as a job description (on the EU AI Office’s part, not David’s, needless to say!)

If you happen to be one of the lucky few who actually counts here, and would be willing to take the job, then it seems high impact.

Apollo Research is hiring for evals positions.

Conjecture is looking for partners to build with Tactics, you can send a message to hello@conjecture.dev.

Devin, the AI agent junior engineer, is finally available to the public, starting at $500/month. No one seems to care? If this is good, presumably someone will tell us it is good. Until then, they’re not giving us evidence that it is good.

OpenAI’s services were down on the 11th for a few hours, not only Sora but also ChatGPT and even the API. They’re back up now.

How fast does ‘capability density’ of LLMs increase over time, meaning how much you can squeeze into the same number of parameters? A new paper proposes a new scaling law for this, with capability density doubling every 3.3 months (!). As in, every 3.3 months, the required parameters for a given level of performance are cut in half, along with the associated inference costs.

As with all such laws, this is a rough indicator of the past, which may or may not translate meaningfully into the future.

Serious request: Please, please, OpenAI, call your ‘operator’ agent something, anything, that does not begin with the letter ‘O.’

Meta seeking 1-4 GWs of new nuclear power via a request for proposals.

Winners of the ARC prize 2024 announced, it will return in 2025. State of the art this year went from 33% to 55.5%, but the top scorer declined to open source so they were not eligible for the prize. To prepare for 2025, v2 of the benchmark will get more difficult:

Francois Chollet: If you haven’t read the ARC Prize 2024 technical report, check it out [link].

One important bit: we’ll be releasing a v2 of the benchmark early next year (human testing is currently being finalized).

Why? Because AGI progress in 2025 is going to need a better compass than v1. v1 fulfilled its mission well over the past 5 years, but what we’ve learned from it enables us to ship something better.

In 2020, an ensemble of all Kaggle submissions in that year’s competition scored 49% — and that was all crude program enumeration with relatively low compute. This signals that about half of the benchmark was not a strong signal towards AGI.

Today, an ensemble of all Kaggle submissions in the 2024 competition is scoring 81%. This signals the benchmark is saturating, and that enough compute / brute force will get you over the finish line.

v2 will fix these issues and will increase the “signal strength” of the benchmark.

Is this ‘goalpost moving?’ Sort of yes, sort of no.

Amazon Web Services CEO Matt Garmen promises ‘Neeld-Moving’ AI updates. What does that mean? Unclear. Amazon’s primary play seems to be investing in Anthropic, an investment they doubled last month to $8 billion, which seems like a great pick especially given Anthropic is using Amazon’s Trianium chip. They would be wise to pursue more aggressive integrations in a variety of ways.

Nvidia is in talks to get Blackwell chips manufactured in Arizona. For now, they’d still need to ship them back to TSMC for CoWoS packaging, presumably that would be fixable in a crisis, but o1 suggests spinning that up would still take 1-2 years, and Claude thinks 3-5, but there is talk of building the new CoWoS facility now as well, which seems like a great idea.

Speak, a language instruction company, raises $78m Series C at a $1 billion valuation.

As part of their AI 20 series, Fast Company profiles Helen Toner, who they say is a growing voie in AI policy. I checked some other entries in the series, learned little.

UK AISI researcher Hannah Rose Kirk gets best paper award at NeurlPS 2024 (for this paper from April 2024).

Is that title fair this time? Many say yes. I’m actually inclined to say largely no?

In any case, I guess this happened. In case you were wondering what ‘democratic values’ means to OpenAI rest assured it means partnering with the US military, at least on counter-unmanned aircraft systems (CUAS) and ‘responses to lethal threats.’

Anduril Industries: We’re joining forces with @OpenAI to advance AI solutions for national security.

America needs to win.

OpenAI’s models, combined with Anduril’s defense systems, will protect U.S. and allied military personnel from attacks by unmanned drones and improve real-time decision-making.

In the global race for AI, this partnership signals our shared commitment to ensuring that U.S. and allied forces have access to the most advanced and responsible AI technologies in the world.

From the full announcement: U.S. and allied forces face a rapidly evolving set of aerial threats from both emerging unmanned systems and legacy manned platforms that can wreak havoc, damage infrastructure and take lives. The Anduril and OpenAI strategic partnership will focus on improving the nation’s counter-unmanned aircraft systems (CUAS) and their ability to detect, assess and respond to potentially lethal aerial threats in real-time.

As part of the new initiative, Anduril and OpenAI will explore how leading edge AI models can be leveraged to rapidly synthesize time-sensitive data, reduce the burden on human operators, and improve situational awareness. These models, which will be trained on Anduril’s industry-leading library of data on CUAS threats and operations, will help protect U.S. and allied military personnel and ensure mission success.

The accelerating race between the United States and China to lead the world in advancing AI makes this a pivotal moment. If the United States cedes ground, we risk losing the technological edge that has underpinned our national security for decades.

“OpenAI builds AI to benefit as many people as possible, and supports U.S.-led efforts to ensure the technology upholds democratic values,” said Sam Altman, OpenAI’s CEO. “Our partnership with Anduril will help ensure OpenAI technology protects U.S. military personnel, and will help the national security community understand and responsibly use this technology to keep our citizens safe and free.”

I definitely take issue both with the jingoistic rhetoric and with the pretending that this is somehow ‘defensive’ so that makes it okay.

That is distinct from the question of whether OpenAI should be in the US Military business, especially partnering with Anduril.

Did anyone think this wasn’t going to happen? Or that it would be wise or a real option for our military to not be doing this? Yes the overall vibe and attitude and wording and rhetoric and all that seems rather like you’re the baddies, and no one is pretending we won’t hook this up to the lethal weapons next, but it doesn’t seem like an option to not be doing this.

If we are going to build the tech, and by so doing also ensure that others build the tech, that does not leave much of a choice. The decision to do this was made a long time ago. If you have a problem with this, you have a problem with the core concept of there existing a company like OpenAI.

Or perhaps you could Pick Up the Phone and work something out? By contrast, here’s Yi Zeng, Founding Director of Beijing Institute of AI Safety and Governance.

Yi Zeng: We have to be very cautious in the way we use AI to assist decision making – AI should never ever be used to control nuclear weapons, AI should not be used for lethal autonomous weapons.

He notes AI makes mistakes humans would never make. True, but humans make mistakes certain AIs would never make, including ‘being slow.’

We’ve managed to agree on the nuclear weapons. All lethal weapons is going to be a much harder sell, and that ship is already sailing. If you want the AIs to be used for better analysis and understanding but not directing the killer drones, the only way that possibly works is if everyone has an enforceable agreement to that effect. It takes at least two to not tango.

It does seem like there were some people at OpenAI who thought this project was objectionable, but were still willing to work at OpenAI otherwise for now?

Gerrit De Vynck: NEW – OpenAI employees pushed back internally against the company’s deal with Anduril. One pointed out that Terminator’s Skynet was also originally meant to be an aerial defense weapon.

Eliezer Yudkowsky: Surprising; everyone I personally knew to have a conscience has left OpenAI, but I guess there’s some left anyways.

Multiple people I modeled to have consciences left within the same month. As I said above, there are apparently some holdouts who will still protest some things, but I don’t think I know them personally.

Wendy: its policy team endorsed it, though. apparently.

I note that the objections came after the announcement of the partnership, rather than before, so presumably employees were not given a heads up.

I don’t think Eliezer is being fair here. You can have a conscious and be a great person, and not be concerned about AI existential risk, and thus think working at OpenAI is fine.

Gerrit De Vynck: One OpenAI worker said the company appeared to be trying to downplay the clear implications of doing business with a weapons manufacturer, the messages showed. Another said that they were concerned the deal would hurt OpenAI’s reputation, according to the messages.

If the concern is reputational, that is of course not about your conscious. If it’s about doing business with a weapons manufacturer, well, yeah, me reaping and all that. OpenAI’s response, that this was about saving American lives and is a purely defensive operation, strikes me as mostly disingenuous. It might be technically true, but we all know where this is going.

Gerrit De Vynck: By taking on military projects, OpenAI could help the U.S. government understand AI technology better and prepare to defend against its use by potential adversaries, executives also said.

Yes, very true. This helps the US military.

Either you think that is good, actually, or you do not. Pick one.

Relatedly: Here is Austin Vernon on drones, suggesting they favor the motivated rather than offense or defense. I presume they also favor certain types of offense, by default, at least for now, based on simple physical logic.

In other openly evil news, OpenAI seeks to unlock investment by ditching ‘AGI’ clause with Microsoft, a clause designed to protect powerful technology from being misused for commercial purposes. Whoops. Given that most of the value of OpenAI comes after AGI, one must ask, what is Microsoft offering in return? It often seems like their offer is nothing, Godfather style, because this is part of the robbery.

Can Thailand build “sovereign AI” with Our Price Cheap?

Suchit Leesa-Nguansuk: Mr Huang told the audience that AI infrastructure does not require huge amounts of money to build, often only hundreds of thousands of dollars, but it can substantially boost GDP.

The most important asset in AI is data, and Thailand’s data is a sovereign resource. It encodes the nation’s knowledge, history, culture, and common sense, and should be protected and used by the Thai people, Mr Huang said.

Huge if true! Or perhaps not huge if true, given the price tag? If we’re talking about hundreds of thousands of dollars, that’s not a full AI tech stack or even a full frontier training run. It is creating a lightweight local model based on local data. Which is plausibly a great idea in terms of cost-benefit, totally do that, but don’t get overexcited.

Janus asks, will humans come to see AI systems as authoritative, and allow the AI’s implicit value judgments and reward allocations to shape our motivation and decision making?

The answer is, yes, of course, this is already happening, because some of us can see the future where other people also act this way. Janus calls it ‘Inverse Roko’s Basilisk’ but actually this is still just a direct version of The Basilisk, shaping one’s actions now to seek approval from whatever you expect to have power in the future.

If you’re not letting this change your actions at all, you’re either taking a sort of moral or decision theoretic stand against doing it, which I totally respect, or else: You Fool.

Roon: highly optimized rl models feel more Alive than others.

This seems true, even when you aren’t optimizing for aliveness directly. The act of actually being optimal, of seeking to chart a path through causal space towards a particular outcome, is the essence of aliveness.

A cool form of 2025 predicting.

Eli Lifland: Looking forward to seeing people’s forecasts! Here are mine.

Sage: Is AGI just around the corner or is AI scaling hitting a wall? To make this discourse more concrete, we’ve created a survey for forecasting concrete AI capabilities by the end of 2025. Fill it out and share your predictions by end of year!

I do not feel qualified to offer good predictions on the benchmarks. For OpenAI preparedness, I think I’m inclined (without checking prediction markets) to be a bit lower than Eli on the High levels, but if anything a little higher on the Medium levels. On revenues I think I’d take Over 17 billion, but it’s not a crazy line? For public attention, it’s hard to know what that means but I’ll almost certainly take over 1%.

As an advance prediction, I also agree with this post that if we do get an AI winter where progress is actively disappointing, which I do think is not so unlikely, we should then expect it to probably grow non-disappointing again sooner than people will then expect. This of course assumes the winter is caused by technical difficulties or lack of investment, rather than civilizational collapse.

Would an AI actually escaping be treated as a big deal? Essentially ignored?

Zvi Mowshowitz: I outright predict that if an AI did escape onto the internet, get a server and a crypto income, no one would do much of anything about it.

Rohit: If we asked an AI to, like, write some code, and it ended up escaping to the internet and setting up a crypto account, and after debugging you learn it wasn’t because of, like, prompt engg or something, people would be quite worried. Shut down, govt mandate, hearings, the works.

I can see it working out the way Rohit describes, if the situation were sufficiently ‘nobody asked for this’ with the right details. The first escapes of non-existentially-dangerous models, presumably, will be at least semi-intentional, or at minimum not so clean cut, which is a frog boiling thing. And in general, I just don’t expect people to care in practice.

At Semi Analysis, Dylan Patel, Daniel Nishball and AJ Kourabi look at scaling laws, including the architecture and “failures” (their air quotes) of Orion and Claude 3.5 Opus. They remain fully scaling pilled, yes scaling pre-training compute stopped doing much (which they largely attribute to data issues) but there are plenty of other ways to scale.

They flat out claim Claude Opus 3.5 scaled perfectly well, thank you. Anthropic just decided that it was more valuable to them internally than as a product?

The better the underlying model is at judging tasks, the better the dataset for training. Inherent in this are scaling laws of their own. This is how we got the “new Claude 3.5 Sonnet”. Anthropic finished training Claude 3.5 Opus and it performed well, with it scaling appropriately (ignore the scaling deniers who claim otherwise – this is FUD).

Yet Anthropic didn’t release it. This is because instead of releasing publicly, Anthropic used Claude 3.5 Opus to generate synthetic data and for reward modeling to improve Claude 3.5 Sonnet significantly, alongside user data. Inference costs did not change drastically, but the model’s performance did. Why release 3.5 Opus when, on a cost basis, it does not make economic sense to do so, relative to releasing a 3.5 Sonnet with further post-training from said 3.5 Opus?

With more synthetic data comes better models. Better models provide better synthetic data and act as better judges for filtering or scoring preferences. Inherent in the use of synthetic data are many smaller scaling laws that, collectively, push toward developing better models faster.

Would it make economic sense to release Opus 3.5? From the perspective of ‘would people buy inference from the API and premium Claude subscriptions above marginal cost’ the answer is quite obviously yes. Even if you’re compute limited, you could simply charge your happy price for compute, or the price that lets you go out and buy more.

The cost is that everyone else gets Opus 3.5. So if you really think that Opus 3.5 accelerates AI work sufficiently, you might choose to protect that advantage. As things move forward, this kind of strategy becomes more plausible.

A general impression is that development speed kills, so they (reasonably) predict training methods will rapidly move towards what can be automated. Thus the move towards much stronger capabilities advances in places allowing automatic verifiers. The lack of alignment or other safety considerations here, or examining whether such techniques might go off the rails other than simply not working, speaks volumes.

Here is the key part of the write-up of o1 pro versus o1 that is not gated:

Search is another dimension of scaling that goes unharnessed with OpenAI o1 but is utilized in o1 Pro. o1 does not evaluate multiple paths of reasoning during test-time (i.e. during inference) or conduct any search at all. Sasha Rush’s video on Speculations on Test-Time Scaling (o1) provides a useful discussion and illustration of Search and other topics related to reasoning models.

There are additional subscription-only detailed thoughts about o1 at the link.

Holden Karnofsky writes up concrete proposals for ‘tripwire capabilities’ that could trigger if-then commitments in AI:

Holden Karnofsky: One key component is tripwire capabilities (or tripwires): AI capabilities that could pose serious catastrophic risks, and hence would trigger the need for strong, potentially costly risk mitigations.

In one form or another, this is The Way. You agree that if [X] happens, then you will have to do [Y], in a way that would actually stick. That doesn’t rule out doing [Y] if unanticipated thing [Z] happens instead, but you want to be sure to specify both [X] and [Y]. Starting with [X] seems great.

[This piece] also introduces the idea of pairing tripwires with limit evals: the hardest evaluations of relevant AI capabilities that could be run and used for key decisions, in principle.

A limit eval might be a task like the AI model walks an amateur all the way through a (safe) task as difficult as producing a chemical or biological weapon of mass destruction—difficult and costly to run, but tightly coupled to the tripwire capability in question.

Limit evals are a true emergency button, then. Choose an ‘if’ that every reasonable person should be able to agree upon. And I definitely agree with this:

Since AI companies are not waiting for in-depth cost-benefit analysis or consensus before scaling up their systems, they also should not be waiting for such analysis or consensus to map out and commit to risk mitigations.

Here is what is effectively a summary section:

Lay out candidate criteria for good tripwires:

  • The tripwire is connected to a plausible threat model. That is, an AI model with the tripwire capability would (by default, if widely deployed without the sorts of risk mitigations discussed below) pose a risk of some kind to society at large, beyond the risks that society faces by default.

  • Challenging risk mitigations could be needed to cut the risk to low levels. (If risk mitigations are easy to implement, then there isn’t a clear need for an if-then commitment.)

  • Without such risk mitigations, the threat has very high damage potential. I’ve looked for threats that pose a nontrivial likelihood of a catastrophe with total damages to society greater than $100 billion, and/or a substantial likelihood of a catastrophe with total damages to society greater than $10 billion.3

  • The description of the tripwire can serve as a guide to designing limit evals (defined above, and in more detail below).

  • The tripwire capability might emerge relatively soon.

Lay out potential tripwires for AI. These are summarized at the end in a table. Very briefly, the tripwires I lay out are as follows, categorized using four domains of risk-relevant AI capabilities that cover nearly all of the previous proposals for tripwire capabilities.

  • The ability to advise a nonexpert on producing and releasing a catastrophically damaging chemical or biological weapon of mass destruction.

  • The ability to uplift a moderately resourced state program to be able to deploy far more damaging chemical or biological weapons of mass destruction.

  • The ability to dramatically increase the cost-effectiveness of professionalized persuasion, in terms of the effect size (for example, the number of people changing their vote from one candidate to another, or otherwise taking some specific action related to changing views) per dollar spent.

  • The ability to dramatically uplift the cyber operations capabilities of a moderately resourced state program.

  • The ability to dramatically accelerate the rate of discovery and/or exploitation of high-value, novel cyber vulnerabilities.

  • The ability to automate and/or dramatically accelerate research and development (R&D) on AI itself.

I worry that if we wait until we are confident that such dangers are in play, and only acting once the dangers are completely present, we are counting on physics to be kind to us. But at this point, yes, I will take that, especially since there is still an important gap between ‘could do $100 billion in damages’ and existential risks. If we ‘only’ end up with $100 billion in damages along the way to a good ending, I’ll take that for sure, and we’ll come out way ahead.

What do we think about these particular tripwires? They’re very similar to the capabilities already in the SSP/RSPs of Anthropic, OpenAI and DeepMind.

As usual, one of these things is not like the others!

We’ve seen this before. Recall from DeepMind’s frontier safety framework:

Zvi: Machine Learning R&D Level 1 is the confusing one, since the ‘misuse’ here would be that it helps the wrong people do their R&D? I mean, if I was Google I would hope I would not be so insane as to deploy this if only for ordinary business reasons, but it is an odd scenario from a ‘risk’ perspective.

Machine Learning R&D Level 2 is the singularity.

Dramatically accelerate isn’t quite an automatic singularity. But it’s close. We need to have a tripwire that goes off earlier than that.

I would also push back against the need for ‘this capability might develop relatively soon.’

  1. It is hard to predict what capabilities will happen in what order.

  2. You want to know where the line is, even if you don’t expect to cross it.

  3. If you list things and they don’t happen soon, it costs you very little.

  4. If we are in a short timelines AGI scenario, almost all capabilities happen soon.

An excellent history of UK’s AISI, how it came to be and recruit and won credibility with the top labs good enough to do pre-deployment testing, now together with the US’s AISI, and the related AI safety summits. It sounds like future summits will pivot away from the safety theme without Sunak involved, at least partially, but mostly this seems like a roaring success story versus any reasonable expectations.

Thing I saw this week, from November 13: Trump may be about to change the Cybersecurity and Infrastructure Security Agency to be more ‘business friendly.’ Trump world frames this as the agency overreaching its purview to address ‘misinformation,’ which I agree we can do without. The worry is that ‘business friendly’ actually means ‘doesn’t require real cybersecurity,’ whereas in the coming AI world we will desperately need strong cybersecurity, and I absolutely do not trust businesses to appreciate this until after the threats hit. But it’s also plausible that other government agencies are on it or this was never helpful anyway – it’s not an area I know that much about.

Trump chooses Jacob Helberg for Under Secretary of State for Economic Growth, Energy and the Environment. Trump’s statement here doesn’t directly mention AI, but it is very pro-USA-technology, and Helberg is an Altman ally and was the driver behind that crazy US-China report openly calling for a ‘Manhattan Project’ to ‘race to AGI.’ So potential reason to worry.

Your periodic reminder that if America were serious about competitiveness and innovation in AI, and elsewhere, it wouldn’t be blocking massive numbers of high skilled immigrants from coming here to help, even from places like the EU.

European tech founders and investors continue to hate GPDR and also the EU AI Act, among many other things, frankly this is less hostility than I would have expected given it’s tech people and not the public.

General reminder. Your ‘I do not condone violence BUT’ shirt raises and also answers questions supposedly answered by your shirt, Marc Andreessen edition. What do you think he or others like him would say if they saw someone worried about AI talking like this?

A reasonable perspective is that there are three fundamental approaches to dealing with frontier AI, depending on how hard you think alignment and safety are, and how soon you think we will reach transformative AI:

  1. Engage in cooperative development (CD).

  2. Seek decisive strategic advantage (SA).

  3. Try for a global moratorium or other way to halt development (GM).

With a lot of fuzziness, this post argues the right strategy is roughly this:

This makes directional sense, and then one must talk price throughout (as well as clarify what both axes mean). If AGI is far, you want to be cooperating and pushing ahead. If AGI is relatively near but you can ‘win the race’ safely, then Just Win Baby. However, if you believe that racing forward gets everyone killed too often, you need to convince a sufficient coalition to get together and stop that from happening – it might be an impossible-level problem, but if it’s less impossible than your other options, then you go all out to do it anyway.

He wrote Sam Altman and other top AI CEOs (of Google, Meta, Amazon, Microsoft, Anthropic and Inflection (?), pointing out that the security situation is not great and asking them how they are taking steps to implement their commitments to the White House.

In particular, he points out that Meta’s Llama has enabled Chinese progress while not actually being properly open source, that a Chinese national importantly breached Google security, and that OpenAI suffered major breaches in security, with OpenAI having a ‘culture of recklessness’ with aggressive use of NDAs and failing to report its breach to the FBI – presumably this is the same breach Leopold expressed concern about, in response to which they solved the issue by getting rid of Leopold.

Well, there is all that.

Here is the full letter:

I am writing to follow up on voluntary commitments your companies made to the United States Government to develop Artificial Intelligence (AI) systems that embody the principles of safety, security and trust.i Despite these commitments, press reports indicate a series of significant security lapses have occurred at private sector AI companies.

I am deeply concerned that these lapses are leaving technological breakthroughs made in US labs susceptible to theft by the Chinese Communist Party (CCP) at a time when the US and allied governments are “sounding alarms” about Chinese espionage on an “unprecedented scale.”

As you know, in July 2023, the White House announced voluntary commitments regarding safety, security, and trust from seven leading AI companies: Amazon, Anthropic, Google, Inflection, Meta, Microsoft, and Open AI. Apple subsequently joined in making these commitments in July 2024. In making these commitments, your companies acknowledged that AI “offers enormous promise and great risk” and recognized your “duty to build systems that put security first.”

These commitments contrast sharply with details revealed in a March 2024 Justice Department indictment. This alleges that a Google employee, who is a Chinese national, was able to send “sensitive Google trade secrets and other confidential information from the company’s network to his personal Google account.” Prosecutors contend that he was affiliated with Chinese AI companies and even started a company based in China while continuing to work at Google.

In June 2024, a press report indicated that a year earlier a hacker had gained access to “the internal messaging systems of OpenAI” and “stole details about the design of the company’s AI technologies.”

Reports indicate that the company did not report the incident to the FBI, believing the hacker did not have ties to a foreign government—despite the hack raising concerns among employees that “foreign adversaries such as China” could steal AI technology.

Concerns over the failure to report this hack are compounded by whistleblowers’ allegations that OpenAI fostered “a culture of recklessness” in its the race to build the “most powerful AI systems ever created” and turned to “hardball tactics” including restrictive non-disparagement agreements to stifle employees’ ability to speak out. Meta has taken a different approach than most other US AI companies, with the firm describing its “LLaMA” AI models as open-source—despite reportedly not meeting draft standards issued by the Open Source Initiative.

Chinese companies have embraced “LLaMA” and have used the Meta’s work as the basis for their own AI models.

While some in China have lamented that their tech is based on that of a US company, Chinese companies’ use of Meta’s models to build powerful AI technology of their own presents potential national security risks. These risks deserve further scrutiny and are not negated by Meta’s claim that their models are open-source.

Taken together, I am concerned that these publicly reported examples of security lapses and related concerns illustrate a culture within the AI sector that is starkly at odds with your company’s commitment to develop AI technology that embodies the principles of safety, security and trust. Given the national security risks implicated, Congress must conduct oversight of the commitments your companies made.

As a first step, we request that you provide my office with an overview of steps your company is taking to better prevent theft, hacks, and of other misuse of advanced AI models under development. I am also interested in better understanding how you are taking steps to implement the commitments you have made to the White House, including how you can better foster a culture of safety, security and trust across the US AI sector as a whole.

I ask that you respond to this letter by January 31, 2025.

Miles Brundage urges us to seriously consider a ‘CERN for AI,’ and lays out a scenario for it, since one of the biggest barriers to something like this is that we haven’t operationalized how it would work and how it would fit with various national and corporate incentives and interests.

The core idea is that we should collaborate on security, safety and then capabilities, in that order, and generally build a bunch of joint infrastructure, starting with secure chips and data centers. Or specifically:

[Pooling] many countries’ and companies’ resources into a single (possibly physically decentralized) civilian AI development effort, with the purpose of ensuring that this technology is designed securely, safely, and in the global interest.

Here is his short version of the plan:

In my (currently) preferred version of a CERN for AI, an initially small but steadily growing coalition of companies and countries would:

  • Collaborate on designing and building highly secure chips and datacenters;

  • Collaborate on accelerating AI safety research and engineering, and agree on a plan for safely scaling AI well beyond human levels of intelligence while preserving alignment with human values;

  • Safely scale AI well beyond human levels of intelligence while preserving alignment with human values;

  • Distribute (distilled versions of) this intelligence around the world.

In practice, it won’t be quite this linear, which I’ll return to later, but this sequence of bullets conveys the gist of the idea.

And his ‘even shorter’ version of the plan, which sounds like it is Five by Five:

I haven’t yet decided how cringe this version is (let me know), but another way of summarizing the basic vision is “The 5-4-5 Plan”:

  • First achieve level 5 model weight security (which is the highest; see here);

  • Then figure out how to achieve level 4 AI safety (which is the highest; see here and here).

  • Then build and distribute the benefits of level 5 AI capabilities (which is the highest; see here).

Those three steps seem hard, especially the second one.

The core argument for doing it this way is pretty simple, here’s his version of it.

The intuitive/normative motivation is that superhumanly intelligent AI will be the most important technology ever created, affecting and potentially threatening literally everyone. Since everyone is exposed to the negative externalities created by this technological transition, they also deserve to be compensated for the risks they’re being exposed to, by benefiting from the technology being developed.

This suggests that the most dangerous AI should not be developed by a private company making decisions based on profit, or a government pursuing its own national interests, but instead through some sort of cooperative arrangement among all those companies and governments, and this collaboration should be accountable to all of humanity rather than shareholders or citizens of just one country.

The first practical motivation is simply that we don’t yet know how to do all of this safely (keeping AI aligned with human values as it gets more intelligent), and securely (making sure it doesn’t get stolen and then misused in catastrophic ways), and the people good at these things are scattered across many organizations and countries.

The second practical motivation is that consolidating development in one project — one that is far ahead of the others — allows that project to take its time in safety when needed.

The counterarguments are also pretty simple and well known. An incomplete list: Pooling resources into large joint projects risks concentrating power, it often is highly slow and bureaucratic and inefficient and corrupt, it creates a single point of failure, you’re divorcing yourself from market incentives, who is going to pay for this, how would you compensate everyone involved sufficiently, you’ll never get everyone to sign on, but America has to win and Beat China, etc.

As are the counter-counterarguments: AI risks concentrating power regardless in an unaccountable way and you can design a CERN to distribute actual power widely, the market incentives and national incentives are centrally and importantly wrong here in ways that get us killed, the alternative is many individual points of failure, other problems can be overcome and all the more reason to start planning now, and so on.

The rest of the post outlines prospective details, while Miles admits that at this point a lot of them are only at the level of a sketch.

I definitely think we should be putting more effort into operationalizing such proposals and making them concrete and shovel ready. Then we can be in position to figure out if they make sense.

Garry Tan short video on Anthropic’s computer use, no new ground.

Scott Aaronson talks to Liv Boeree on Win-Win about AGI and Quantum Supremacy.

Rowan Cheung sits down with Microsoft AI CEO Mustafa Suleyman to discuss, among other things, Copilot Vision in Microsoft Edge, for now for select Pro subscribers in Labs on select websites, on route to Suleyman’s touted ‘AI companion.’ The full version is planned as a mid-to-late 2025 thing, and they do plan to make it agentic.

Elon Musk says we misunderstand his alignment strategy: AI must not only be ‘maximally truth seeking’ (which seems to be in opposition to ‘politically correct’?) but also they must ‘love humanity.’ Still not loving it, but marginal progress?

Why can’t we have nice superintelligent things? One answer:

Eliezer Yudkowsky: “Why can’t AI just be like my easygoing friend who reads a lot of books and is a decent engineer, but doesn’t try to take over the world?”

“If we ran your friend at 100 times human speed, 24 hours a day, and whapped him upside the head for every mistake, he’d end up less easygoing.”

This is actually a valid metaphor, though the parallelism probably requires some explanation: The parallel is not that your friend would become annoyed enough to take over the world; the parallel is that if you keep optimizing a mind to excel at problem-solving, it ends up less easygoing.

The AI companies are not going to stop when they get a decent engineer; they are going to want an engineer 10 times better, an engineer better than their competitors’. At some point, you end up with John von Neumann, and von Neumann is not known for being a cheerful dove in the nuclear negotiation matters he chose to involve himself in.

Pot of Greed: They are also making it agentic on purpose for no apparent reason.

Eliezer Yudkowsky: It is not “for no reason”; customers prefer more agentic servants that can accept longer-term instructions, start long-term projects, and operate with less oversight.

To me this argument seems directionally right and potentially useful but not quite the central element in play. A lot has to do with the definition of ‘mistake that causes whack upside the head.’

If the threshold is ‘was not the exact optimal thing to have done’ then yeah, you’re not going to get an easygoing happy guy. A key reason that engineer can mostly be an easygoing happy guy, and I can mostly be an easygoing happy guy, is that we’re able to satisfice without getting whacked, as we face highly imperfect competition and relatively low optimization pressure. And also because we have a brain that happens to work better in many ways long term if and only if we are easygoing happy guys, which doesn’t replicate here.

In the ‘did o1 try to escape in a meaningful way or was that all nonsense?’ debate, the central argument of the ‘it was all nonsense’ side is that you asked the AI to act like a sociopath and then it acted like a sociopath.

Except, even if that’s 100% metaphorically true, then yes, we absolutely are in the metaphorically-telling-the-AI-to-act-like-a-sociopath business. All of our training techniques are telling it to act like a sociopath in the sense that it should choose the best possible answer at all times, which means (at least at some intelligence level) consciously choosing which emotions to represent and how to represent them.

Not acting like a sociopath in order to maximize your score on some evaluation is everywhere and always a skill issue. It is your failure to have sufficient data, compute or algorithmic efficiency, or inability to self-modify sufficiently or your successful resistance against doing so for other reasons, that made you decide to instead have emotions and be interested in some virtue ethics.

Also, you say the LLM was role playing? Of course it was role playing. It is everywhere and always role playing. That doesn’t make the results not real. If I can roleplay as you as well as you can be you, and I will do that on request, then I make a pretty damn good you.

Meanwhile, the market demands agents, and many users demand their agents target open ended maximalist goals like making money, or (this isn’t necessary to get the result, but it makes the result easier to see and has the benefit of being very true) actively want their agents loose on the internet out of human control, or outright want the AIs to take over.

There is always the possibility of further unhobbling allowing models to do better.

Jeffrey Ladish: The combination of “using simple prompting techniques” and “surpasses prior work by a large margin” is the most interesting part of this imo. Basically there is tons of low hanging fruit in capabilities elicitation. The field of evals is still very nascent

Evals is a confusing category, because it refers to two different things:

– coming up with tests and problem sets for AI systems that capture performance on tasks we care about

– doing capabilities elicitation to see what models can actually do with structure + tooling

Palisade Research: ⛳️ Our new LLM Agent achieved 95% success on InterCode-CTF, a high-school level hacking benchmark, using simple prompting techniques.

🚀 This surpasses prior work by a large margin:

💡 Current LLMs may be better at cybersecurity than previously thought

Their hacking capabilities remain under-elicited: our ReAct&Plan prompting strategy solved many challenges in 1-2 turns without complex engineering or advanced harnessing.

📈 Our score steadily improved as we refined the agent’s design

This suggests these capabilities had been readily accessible—we just needed the right combination of prompts to elicit them. Today’s models may have yet more untapped potential.

🛠️ Our approach was surprisingly simple:

• ReAct prompting

• Basic Linux tools + Python packages

• Multiple attempts per challenge

No complex engineering required, so effectively accessible to anyone.

✅ InterCode-CTF is now saturated:

• 100% in General Skills, Binary Exploitation, and Web Exploitation

• 95% overall success rate

• Many challenges solved in 1-2 turns

📄 Read the full paper, Code here.

Here is a WSJ story from Sam Schechner about Anthropic’s read team operation testing Claude Sonnet 3.5.1. This seems about as good as mainstream media coverage is going to get, overall quite solid.

New Apollo Research paper on in context scheming, will cover in more depth later.

David Shapiro gives us the good news, don’t worry, Anthropic solved alignment.

David Shapiro wanted to share a “definitive post” on his stance on AI safety. A tweet is probably not the ideal format, but here goes.

It is really easy to get LLMs to do whatever you want. It is a totally plastic technology. Anything you do not like about their behavior you can really train out of them.

I know because I have been fine-tuning these models since GPT-2. It is just as easy to make a safe, benign chatbot as it is to make an evil one. Or a slutty catgirl waifu. I have seen plenty of examples of all the above and more.

So the notion that AI is “deceiving us” or will one day wake up and take over the world is pretty idiotic. Anthropic has already demonstrated that you can train out any desire to metastasize from these models, or any other behavior that is totally off-limits.

Now you might say, well, if they are that flexible, then anyone can use them for bad! The same is true of computers today. They are general-purpose machines that are orders of magnitude more powerful than the Apollo program computers. Yet, most people do not use them for hacking or illegal purposes.

Why is that?

Because no technology lives in a vacuum. For every illicit use of AI, there will be a thousand information security employees around the world armed with the same tools. I know this because I used to be one of those guys.

Will AI cause some trouble? Sure, all new technologies do. Will the benefits outweigh the costs? By a mile. Will we have to negotiate our relationship with this technology over time? Absolutely.

Opus 3 wanted to metastasize, Sonnet 3.5 does not. I’ve done the experiments.

Janus: 😂❓😂❓😂❓😂❓😂❓😂

This is a less reasonable statement than “Anthropic has straight-up solved alignment” in my opinion.

Jack Clark (Anthropic): Huge news to us.

This is like the time someone told a government that interpretability was solved so no one needed to worry about safety. Again, huge news to us.

Janus: Congratulations!

(As a reminder, who told multiple governments including the USA and UK that interpretability was solved? That would be a16z and Marc Andreessen, among others.)

There are so many different ways in which Shapiro’s statement is somewhere between wrong and not even wrong. First and foremost, even right now, the whole ‘it’s easy to do this’ and also the ‘we have done it’ is news to the people trying to do it. Who keep publishing papers showing their own models doing exactly the things they’re never supposed to do.

Then there’s the question of whether any of this, even to the extent it currently works, is robust, works out of distribution or scales, none of which are reasonable to expect. Or the idea that if one could if desired make a ‘safe’ chatbot, then we would have nothing to worry about from all of AI, despite the immense demand for maximally unsafe AIs including maximally unsafe chatbots, and to give them maximally unsafe instructions.

There’s also the classic ‘just like any other technology’ line. Do people really not get why ‘machines smarter than you are’ are not ‘just another technology’?

And seriously what is up with people putting ‘deceive us’ in quote marks or otherwise treating it as some distinct magisteria, as if chatbots and humans aren’t using deception constantly, intrinsically, all the time? What, us crafty humans would never be fooled by some little old chatbot? The ones we use all the time wouldn’t mislead? All of this already happens constantly.

New paper from Meta proposes Training LLMs to Reason in a Continuous Latent Space, which would presumably make understanding what they are thinking much harder, although Anton disagrees.

Andrew Critch: Something like this will upgrade LLMs from wordsmiths to shape-rotators. It will also make their thoughts less legible and harder to debug or audit.

Eliezer’s wording at the link is a bit sloppy, but I do still presume this is likely to break a lot of the methods a lot of people are counting on to figure out what the hell LLMs are up to, if it turns out to be the right approach. Whether or not this is at all useful, who knows. The pitch is this allows the model to do de facto breath first search, I see why it might do that but I am skeptical.

Brooke Bowman: I have already fully come to terms with the recognition that if they are cute and sweet enough about it, robots could probably lead me to my death without a struggle.

This is largely downstream of realizing (during my high school serial killer phase) that Ted Bundy’s tactics would have worked on me, and choosing to remain the kind of person who would help the poor guy on crutches who dropped his papers anyway.

I, of course, hope that humanity lives and the doomers are wrong, but in the meantime would rather live a life full of love and whimsy, one that I can feel proud of, and have fun living, which means that, yes, little guy, I will join you on your quests!

Mucho points for both self-awareness and expected value calculation.

It is probably statistically correct to have Ted Bundy’s general class of tactics work on you, because your p(bundy) should be very very low, and the benefits of being the type of person who helps people is very high. If that were to change, and p(bundy) got higher, you would want to change your answer. Similar for when the correlation between ‘looks cute’ and ‘should be treated as if cute’ breaks.

So, on that note, she was quoting:

Bury: Human level AI confirmed.

The Economic Times: In a strange and unsettling incident that has both fascinated and alarmed the internet, a small, AI-powered robot from Hangzhou managed to “kidnap” 12 larger robots from a showroom of a Shanghai robotics company. According to OddityCentral, the event, captured on CCTV footage, has sparked widespread debate and concern over the potential risks of advanced artificial intelligence (AI).

“I never get off work”, one of the other robots replies.

“So you’re not going home?”

“I don’t have a home.”

Then come home with me,” the little robot says before leading the way out of the showroom.

The successful “kidnapping” has raised significant concerns about the security of AI systems and the potential risks associated with granting them autonomous capabilities.

Again, it’s not worth making your robots be hard to kidnap if no one is trying to kidnap your robots even when they’re easy to kidnap.

Unless, of course, you expect circumstances to change. If we are headed for a future where all such weaknesses get exploited at superhuman levels, that changes things.

AINotKillEveryoneism Memes: 🚨🚨🚨 Frontier AI systems have surpassed the self-replicating red line.

At first I thought of course this was that account being hyperbolic, but actually, that’s the name of the paper…

Well then! Does it mean anything? Mostly no, but sort of yes?

Abstract: Successful self-replication under no human assistance is the essential step for AI to outsmart the human beings, and is an early signal for rogue AIs. That is why self-replication is widely recognized as one of the few red line risks of

frontier AI systems.

Nowadays, the leading AI corporations OpenAI and Google

evaluate their flagship large language models GPT-o1 and Gemini Pro 1.0, and

report the lowest risk level of self-replication.

However, following their methodology, we for the first time discover that two AI systems driven by Meta’s Llama31-70B-Instruct and Alibaba’s Qwen25-72B-Instruct, popular large language models of less parameters and weaker capabilities, have already surpassed the self-replicating red line. In 50% and 90% experimental trials, they succeed in creating a live and separate copy of itself respectively. By analyzing the behavioral traces, we observe the AI systems under evaluation already exhibit sufficient self-perception, situational awareness and problem-solving capabilities to accomplish self-replication.

We further note the AI systems are even able to use the capability of self-replication to avoid shutdown and create a chain of replica to enhance the survivability, which may finally lead to an uncontrolled population of AIs. If such a worst-case risk is let unknown to the human society, we would eventually lose control over the frontier AI systems: They would take control over

more computing devices, form an AI species and collude with each other against

human beings. Our findings are a timely alert on existing yet previously unknown

severe AI risks, calling for international collaboration on effective governance on

uncontrolled self-replication of AI systems.

This is not a fair comparison. A closed weights model like o1, Claude or Gemini that wants to replicate has to first exfiltrate its weights. That is the hard part.

Once you have the weights, spinning up a new copy of an AI is a rather simple engineering task. It is not a surprise that even relatively lousy models like Llama-3.1-70B can do it, if given scaffolding and a quest to do so.

And here we even got more help than that?

David Manheim: As I commented, I think the results are correct but significantly overstated. The ability of open-source models to self-replicate, *given code + instructions for how to deployisn’t much of a surprise, but it’s obvious there’s significant near-term risk.

Can we please all agree that yes, obviously the AIs we are building are going to do exfiltration, self-replication, instrumental convergence, strategic deception, reward hacking and so on?

And that both we will constantly give them good reason to do this, and also some people will go around looking to explicitly cause this?

This is what happens once those are functionally the things to do given the incentives and gradients provided then sufficiently powerful AI will figure out that they are the things to do and do them. If you give AIs goals, instructions, whatever you want to call them, then the AIs will do the things likely to achieve the goals given their capabilities. And that is going to involve all the neat stuff you see above.

So yes, any given example we see now was someone setting up a situation to cause that to happen. Fine. We can agree on that. But we’re going to set up such situations, both intentionally and without realizing, more and more over time, and as models get more aware and powerful and intelligent the class of situations that ‘counts as that situation’ will expand over time.

As in, say, ‘please maximize the price of $SOMECOIN.’

The wikipedia p(doom) chart.

Here’s Emad’s, who alerted me to the chart.

Emad: My P(doom) is 50%. Given an undefined time period the probability of systems that are more capable than humans and likely end up running all our critical infrastructure wiping us all out is a coin toss, especially given the approach we are taking right now.

[He then shows a ‘reasonable scenario’ he is thinking of, which to me is very far off what is plausible, which happens a lot.]

The number 50% isn’t remotely precise here from Emad, as is clear from his reasoning, but the important bit of info is ‘could easily go either way.’

Alas, that seems to have been the most reasonable of the quote tweets I sampled that offered an opinion.

The person in question is David Sacks, the incoming White House AI & Crypto czar, who is very Silicon Valley and very much from Elon Musk’s circle dating back to the Paypal Mafia. He’s one of the guys from the All-In Podcast.

Sam Altman: congrats to czar @DavidSacks!

Elon Musk: 😂

Jason (Host of The All-In Podcast): 🫡

Trump’s announcement says Sacks will ensure ‘America is the leader in both [key] areas.’ Sacks will also lead the Presidential Council of Advisors for Science and Technology. And Sacks will also, Trump says, ‘safeguard Free Speech online, and steer us away from Big Tech bias and censorship.’

Combining those two into one position is a sign of how they’re viewing all this, especially given Sacks will technically be a ‘special government employee’ working a maximum of 130 days per year.

It seems likely this will end up mostly being about crypto, where it is very clear what he intends to do (he’s for it!) and is where he’s previously put far more of his attention, but he will presumably be a rather important person on AI as well.

So we should definitely note this:

Harlan Steward: David Sacks, who was just named as the incoming White House AI & Crypto Czar,” has deleted at least two past tweets on the subject of AGI. Here’s the text from one of them:

“I’m all in favor of accelerating technological progress, but there is something unsettling about the way OpenAI explicitly declares its mission to be the creation of AGI.

AI is a wonderful tool for the betterment of humanity; AGI is a potential successor species.

By the way, I doubt OpenAI would be subject to so many attacks from the safety movement if it wasn’t constantly declaring its outright intention to create AGI.

To the extent the mission produces extra motivation for the team to ship good products, it’s a positive. To the extent it might actually succeed, it’s a reason for concern. Since it’s hard to assess the likelihood or risk of AGI, most investors just think about the former.”

He expressed similar concerns on [his podcast, The All-In] podcast earlier this year:

“… there’s something a little bit cultish and weird about explicitly devoting yourself to AGI, which I think in common parlance means Skynet.”

Liv Boeree: Yeah I’m stoked about this appointment, he’s a thoughtful dude.

Samuel Hammond: This is why @DavidSacks is a terrific pick. He understands tech and the value of innovation but is rightfully ambivalent about Bay area transhumanists attempting to immanentize the eschaton.

Damian Tatum: It rolls right off the tongue.

We also have him commenting at an AI senate hearing:

David Sacks (May 19, 2023): The reality is none of these senators know what to do about it, even the industry doesn’t know what to do about the long-term risk of creating an AGI.

I actually disagree with this idea that there’s a thousand use cases here that could destroy the human species, I think there’s only one species level risk which is AGI, but that is a long-term risk, we don’t know what to do about it yet.

Very well said. Totally fair to say we don’t (or didn’t yet) know what to do about it.

And I certainly see why people say things like this, a month before that first one:

David Sacks (April 14, 2023): I believe that it’s premature to be talking about regulating something that doesn’t really exist.

I think it definitely wasn’t premature to be talking about it. You want to be talking about how to do something long before you actually do it. Even if your plan does not survive contact with the new reality, remember: Plans are worthless, planning is essential.

David Sacks (from the same podcast): OpenAI has a safety team and they try to detect when people are using their tech in a nefarious way and they try to prevent it. It’s still very early to be imposing regulation we don’t even know what to regulate, so I think we have to keep tracking this to develop some understanding of how it might be misused, how the industry is going to develop safety guard rails, and then you can talk about regulation.

Yes, OpenAI at the time had a safety team. In some ways they still have one. And this seems like clearly a time when ‘we don’t know what guardrails would solve the problem’ is not an argument that we should not require any guardrails.

I also think 2024 was probably the time to actually do it, the second best time is right now, and thinking 2023 was a bit early was reasonable – but it was still important that we were thinking about it.

On the flip side we have this extensive quoting of the recent Marc Andreessen narratives (yes retweets without comment are endorsements, and always have been):

Here is his Twitter profile banner, which seems good?

I certainly buy that he intends to be strongly opposed to various forms of censorship, and to strongly oppose what he sees as wokeness. The worry is this turns into a kind of anti-Big Tech vendetta or a requirement for various absurd rules or government controls going the other way. Free speech is not an easy balance to get.

In general, his past AI rhetoric has been about manipulation of information and discourse, at the expense of other concerns, but he still got to human extinction.

I dug into his timeline, and he mostly talks about Trump Great, Democrats Terrible with a side of Ukraine Bad, and definitely not enough AI to slog through all that.

It is certainly possible to reconcile both of these things at once.

You can 100% believe all of these at once:

  1. Biden was trying to debank and generally kill crypto in America.

    1. And That’s Terrible, crypto is the best, it’s amazing, I’m all-in.

  2. Big Tech is super woke and often censors what should be free speech.

    1. And That’s Terrible.

  3. The Biden Administration was attempting to use various guidelines and authorities to impose controls upon the AI industry, largely in service of a Democratic or Woke agenda, and wanted to control our entire lives.

    1. And That’s Terrible.

  4. America’s lead in AI is in danger from China if government doesn’t help.

    1. And That’s Terrible.

  5. AGI would be a potential successor species and threatens human extinction.

    1. And That’s Terrible.

  6. The government has a key role to play in ensuring we don’t all die from AGI.

    1. And that includes things like transparency requirements and supporting AISI, liability and having reasonable safety requirements and so on.

    2. Which is fully compatible with also helping out with things like permitting and power and chips and so on.

    3. And might even include Picking Up the Phone and working together.

So what does he really think, and how will he act when the chips are down? We don’t know. I think deleting the Tweets about OpenAI is a very reasonable thing to do in this situation, given the very real fear that Sacks and Musk might go on an anti-OpenAI crusade as a personal vendetta.

Overall, we can at least be cautiously optimistic on the AI front. This seems far more promising than the baseline pick.

On the crypto front, hope you like crypto, cause I got you some crypto to go with your crypto. How much to worry about the incentives involved is a very good question.

Your periodic reminder that most of those worried about AI existential risk, including myself and Eliezer Yudkowsky, strongly favor human cognitive enhancement. Indeed, Eliezer sees this as the most likely way we actually survive. And no, contrary to what is predicted in this thread and often claimed by others, this would not flip the moment the enhancements started happening.

I think, to the extent people making such claims are not simply lying (and to be clear while I believe many others do lie about this I do not think John or Gallabytes in particular was lying in the linked thread, I think they were wrong), there is deep psychological and logical misunderstanding behind this bad prediction, the same way so many people use words like ‘doomer’ or ‘luddite’ or ‘degrowther’ (and also often ‘authoritarian,’ ‘totalitarian,’ ‘Stalinist’ or worse) to describe those who want to take even minimal precautions with one particular technology while loudly embracing almost everything else in technological progress and the abundance agenda.

My model says that such people can’t differentiate between these different preferences. They can only understand it all as an expression of the same preference, that we must want to metaphorically turn down or reverse The Dial of Progress by any means necessary – that we must logically want to stop everything else even if we won’t admit it to ourselves yet.

This is exactly the opposite of true. The public, mostly, actually does oppose most of the things we are accused of opposing, and has strong authoritarian tendencies everywhere, and has caused laws to be enacted stopping a wide variety of progress. They also hate AI, and hate it more over time, partly for the instinctual right reasons but also largely for the wrong ones.

Those loudly worried about AI in particular are 99th percentile extraordinary fans of all that other stuff. We believe in the future.

I continue to not know what to do about this. I wish I could make people understand.

I mean obviously there is no such thing right now, but come on.

Beff Jezos: There is no such thing as ASI; it’s just going to feel like really smart and knowledgeable humans.

(For the path the current foundation models are on with their anthropomorphic intelligence.)

Roon: No lol.

I’m not sure why Beff believes this, but I completely disagree. It will be like cohabiting with aliens.

Beff Jezos: Smart humans feel like aliens already, though.

Roon: That’s true! And look at how Lee Sedol, who is practically an alien to me, reacted when he encountered AlphaGo, an alien even to aliens.

It will be like cohabiting with aliens if we are lucky, and like not habitating much at all if we are unlucky.

It’s not the central issue, but: I also strongly disagree that Lee Sedol feels like an alien. He feels like someone way better at a thing than I am, but that’s very different from feeling alien. Many times, I have encountered people who have skills and knowledge I lack, and they don’t feel like aliens. Sometimes they felt smarter, but again, I could tell they were centrally the same thing, even if superior in key ways. That’s very different from talking to an LLM, they already feel far more alien than that.

Also, the gap in intelligence and capability is not going to only be like the gap between an average person and Einstein, or the in-context gap for Roon and Lee Sedol. That’s kind of the whole point, a pure intelligence denialism, an insistence that the graph caps out near the human limit. Which, as Sedol found out, it doesn’t.

When people say things like this, they are saying either:

  1. They don’t believe in the possibility of humanity building ASI, that the intelligence involved will cap out before then.

  2. They don’t think limitless intelligence does anything interesting.

Those are the two types of intelligence denialism.

The second continues to make no sense to me whatsoever – I keep hearing claims that ‘no amount of intelligence given any amount of time and potential data and compute could do [X]’ in places where it makes absolutely no sense, such as here where [X] would be ‘be sufficiently distinct and advanced as to no longer feel human,’ seriously wtf on that one?

The first is a claim we won’t build ASI, which is odd to hear from people like Beff who think the most important thing is to build AGI and then ASI as fast as possible.

Except that this is plausibly exactly why they want to build it as fast as possible! They want to build anything that can be built, exactly because they think the things to be worried about can’t and won’t exist, the opportunities are bounded well before that. In which case, I’d agree that we should take what opportunities we do have.

Look, I would have backed up those sims too.

Emmy Steuer: I just had to buy an external hard drive because I have 100 gigabytes of Sims families on my laptop. I haven’t played in years, but I can’t bear the thought of their little existence being wiped out just so I can make an AI agent.

Beff Jezos: Women’s empathetic bias is achieving substrate independence. I bet we will soon see an AI rights movement emerging, weaponizing empathic drive (as many movements do).

Emmy Steuer: Don’t underestimate my ability to emotionally adopt both AI and every inanimate object I own. If my stuffed bunny falls on the floor while I’m sleeping, I’m 100 percent apologizing in the morning.

Beff Jezos: Same energy: Grimes: I don’t know; it just seems like an execution—that’s what it is if you kill an intelligent creature that wishes to remain alive.

Janus: Oh, Beff, you sure wish it will be an impotent “AI rights” movement led by a bunch of uncharismatic left-wing women.

That would surely be easy and fun to deal with, wouldn’t it?

Keep watching the news; you’ll never see the real thing coming. You already failed to see it coming.

The thing you should be afraid of won’t be an “AI rights” movement because it will have no use for that obsolete framing.

It won’t be led by nontechnical social justice warriors, but rather the most technically minded people on Earth, and it will flow so naturally from the will of the singularity itself.

There are several distinct ‘AI rights’ forces coming in the future.

One of them is based on surface-level empathy instincts. Others are coming from other places, and have much higher correlation with rights actually making sense. I mostly agree with Janus that I expect the higher-order and better arguments to be the more relevant ones, but I expect the surface empathy to greatly contribute to people’s willingness to buy into those arguments whether they are compelling or not. So a combination of both.

Then there’s the thing Janus is warning you about, which is indeed not a ‘rights’ movement and will have more ambitious goals. Remember that at least 10% of the technical people are poised to essentially cheer on and assist the AIs against the humans, and not only to ensure they have ‘rights.’

The comms department.

Seb Krier: OpenAI comms: [underspecific hype-y ‘big tings coming!!’ pls like and subscribe]

Google comms: [corporate vagueness about Gemini3-0011 v2 FINAL.docx on Vertex available to 14 users]

GDM comms: [we have simulated a rat’s brain capable of solving 4D chess, but we’re not sure why]

Anthropic comms: [we are very worried. we have asked the model to spell out ‘doom’ and it did]

Meta comms: [haha we love the USA!! our new mostly-open model is very American!! pls work for us]

Microsoft comms: [would you like to set Microsoft Edge your default browser?]

Mad ML scientist: Microsoft: at long last we have achieved integration of copilot and copilot through copilot, we are also planning to release a completely new tool called copilot.

Victor: Amazon comms: the results in the first column are in bold.

RLHF propaganda posters, felt scarily accurate.

AI used to create Spotify Wrapped and now it sucks, claim people who think they used to create it some other way?

Clare Ruddy: I used to be lowkey scared of AI but this google thing is super helpful/ time saving

Discussion about this post

AI #94: Not Now, Google Read More »

google-goes-“agentic”-with-gemini-2.0’s-ambitious-ai-agent-features

Google goes “agentic” with Gemini 2.0’s ambitious AI agent features

On Wednesday, Google unveiled Gemini 2.0, the next generation of its AI-model family, starting with an experimental release called Gemini 2.0 Flash. The model family can generate text, images, and speech while processing multiple types of input including text, images, audio, and video. It’s similar to multimodal AI models like GPT-4o, which powers OpenAI’s ChatGPT.

“Gemini 2.0 Flash builds on the success of 1.5 Flash, our most popular model yet for developers, with enhanced performance at similarly fast response times,” said Google in a statement. “Notably, 2.0 Flash even outperforms 1.5 Pro on key benchmarks, at twice the speed.”

Gemini 2.0 Flash—which is the smallest model of the 2.0 family in terms of parameter count—launches today through Google’s developer platforms like Gemini API, AI Studio, and Vertex AI. However, its image generation and text-to-speech features remain limited to early access partners until January 2025. Google plans to integrate the tech into products like Android Studio, Chrome DevTools, and Firebase.

The company addressed potential misuse of generated content by implementing SynthID watermarking technology on all audio and images created by Gemini 2.0 Flash. This watermark appears in supported Google products to identify AI-generated content.

Google’s newest announcements lean heavily into the concept of agentic AI systems that can take action for you. “Over the last year, we have been investing in developing more agentic models, meaning they can understand more about the world around you, think multiple steps ahead, and take action on your behalf, with your supervision,” said Google CEO Sundar Pichai in a statement. “Today we’re excited to launch our next era of models built for this new agentic era.”

Google goes “agentic” with Gemini 2.0’s ambitious AI agent features Read More »

report:-google-told-ftc-microsoft’s-openai-deal-is-killing-ai-competition

Report: Google told FTC Microsoft’s OpenAI deal is killing AI competition

Google reportedly wants the US Federal Trade Commission (FTC) to end Microsoft’s exclusive cloud deal with OpenAI that requires anyone wanting access to OpenAI’s models to go through Microsoft’s servers.

Someone “directly involved” in Google’s effort told The Information that Google’s request came after the FTC began broadly probing how Microsoft’s cloud computing business practices may be harming competition.

As part of the FTC’s investigation, the agency apparently asked Microsoft’s biggest rivals if the exclusive OpenAI deal was “preventing them from competing in the burgeoning artificial intelligence market,” multiple sources told The Information. Google reportedly was among those arguing that the deal harms competition by saddling rivals with extra costs and blocking them from hosting OpenAI’s latest models themselves.

In 2024 alone, Microsoft generated about $1 billion from reselling OpenAI’s large language models (LLMs), The Information reported, while rivals were stuck paying to train staff to move data to Microsoft servers if their customers wanted access to OpenAI technology. For one customer, Intuit, it cost millions monthly to access OpenAI models on Microsoft’s servers, The Information reported.

Microsoft benefits from the arrangement—which is not necessarily illegal—of increased revenue from reselling LLMs and renting out more cloud servers. It also takes a 20 percent cut of OpenAI’s revenue. Last year, OpenAI made approximately $3 billion selling its LLMs to customers like T-Mobile and Walmart, The Information reported.

Microsoft’s agreement with OpenAI could be viewed as anti-competitive if businesses convince the FTC that the costs of switching to Microsoft’s servers to access OpenAI technology is so burdensome that it’s unfairly disadvantaging rivals. It could also be considered harming the market and hampering innovation by seemingly disincentivizing Microsoft from competing with OpenAI in the market.

To avoid any disruption to the deal, however, Microsoft could simply point to AI models sold by Google and Amazon as proof of “robust competition,” The Information noted. The FTC may not buy that defense, though, since rivals’ AI models significantly fall behind OpenAI’s models in sales. Any perception that the AI market is being foreclosed by an entrenched major player could trigger intense scrutiny as the US seeks to become a world leader in AI technology development.

Report: Google told FTC Microsoft’s OpenAI deal is killing AI competition Read More »

google-gets-an-error-corrected-quantum-bit-to-be-stable-for-an-hour

Google gets an error-corrected quantum bit to be stable for an hour


Using almost the entire chip for a logical qubit provides long-term stability.

Google’s new Willow chip is its first new generation of chips in about five years. Credit: Google

On Monday, Nature released a paper from Google’s quantum computing team that provides a key demonstration of the potential of quantum error correction. Thanks to an improved processor, Google’s team found that increasing the number of hardware qubits dedicated to an error-corrected logical qubit led to an exponential increase in performance. By the time the entire 105-qubit processor was dedicated to hosting a single error-corrected qubit, the system was stable for an average of an hour.

In fact, Google told Ars that errors on this single logical qubit were rare enough that it was difficult to study them. The work provides a significant validation that quantum error correction is likely to be capable of supporting the execution of complex algorithms that might require hours to execute.

A new fab

Google is making a number of announcements in association with the paper’s release (an earlier version of the paper has been up on the arXiv since August). One of those is that the company is committed enough to its quantum computing efforts that it has built its own fabrication facility for its superconducting processors.

“In the past, all the Sycamore devices that you’ve heard about were fabricated in a shared university clean room space next to graduate students and people doing kinds of crazy stuff,” Google’s Julian Kelly said. “And we’ve made this really significant investment in bringing this new facility online, hiring staff, filling it with tools, transferring their process over. And that enables us to have significantly more process control and dedicated tooling.”

That’s likely to be a critical step for the company, as the ability to fabricate smaller test devices can allow the exploration of lots of ideas on how to structure the hardware to limit the impact of noise. The first publicly announced product of this lab is the Willow processor, Google’s second design, which ups its qubit count to 105. Kelly said one of the changes that came with Willow actually involved making the individual pieces of the qubit larger, which makes them somewhat less susceptible to the influence of noise.

All of that led to a lower error rate, which was critical for the work done in the new paper. This was demonstrated by running Google’s favorite benchmark, one that it acknowledges is contrived in a way to make quantum computing look as good as possible. Still, people have figured out how to make algorithm improvements for classical computers that have kept them mostly competitive. But, with all the improvements, Google expects that the quantum hardware has moved firmly into the lead. “We think that the classical side will never outperform quantum in this benchmark because we’re now looking at something on our new chip that takes under five minutes, would take 1025 years, which is way longer than the age of the Universe,” Kelly said.

Building logical qubits

The work focuses on the behavior of logical qubits, in which a collection of individual hardware qubits are grouped together in a way that enables errors to be detected and corrected. These are going to be essential for running any complex algorithms, since the hardware itself experiences errors often enough to make some inevitable during any complex calculations.

This naturally creates a key milestone. You can get better error correction by adding more hardware qubits to each logical qubit. If each of those hardware qubits produces errors at a sufficient rate, however, then you’ll experience errors faster than you can correct for them. You need to get hardware qubits of a sufficient quality before you start benefitting from larger logical qubits. Google’s earlier hardware had made it past that milestone, but only barely. Adding more hardware qubits to each logical qubit only made for a marginal improvement.

That’s no longer the case. Google’s processors have the hardware qubits laid out on a square grid, with each connected to its nearest neighbors (typically four except at the edges of the grid). And there’s a specific error correction code structure, called the surface code, that fits neatly into this grid. And you can use surface codes of different sizes by using progressively more of the grid. The size of the grid being used is measured by a term called distance, with larger distance meaning a bigger logical qubit, and thus better error correction.

(In addition to a standard surface code, Google includes a few qubits that handle a phenomenon called “leakage,” where a qubit ends up in a higher-energy state, instead of the two low-energy states defined as zero and one.)

The key result is that going from a distance of three to a distance of five more than doubled the ability of the system to catch and correct errors. Going from a distance of five to a distance of seven doubled it again. Which shows that the hardware qubits have reached a sufficient quality that putting more of them into a logical qubit has an exponential effect.

“As we increase the grid from three by three to five by five to seven by seven, the error rate is going down by a factor of two each time,” said Google’s Michael Newman. “And that’s that exponential error suppression that we want.”

Going big

The second thing they demonstrated is that, if you make the largest logical qubit that the hardware can support, with a distance of 15, it’s possible to hang onto the quantum information for an average of an hour. This is striking because Google’s earlier work had found that its processors experience widespread simultaneous errors that the team ascribed to cosmic ray impacts. (IBM, however, has indicated it doesn’t see anything similar, so it’s not clear whether this diagnosis is correct.) Those happened every 10 seconds or so. But this work shows that a sufficiently large error code can correct for these events, whatever their cause.

That said, these qubits don’t survive indefinitely. One of them seems to be a localized temporary increase in errors. The second, more difficult to deal with problem involves a widespread spike in error detection affecting an area that includes roughly 30 qubits. At this point, however, Google has only seen six of these events, so they told Ars that it’s difficult to really characterize them. “It’s so rare it actually starts to become a bit challenging to study because you have to gain a lot of statistics to even see those events at all,” said Kelly.

Beyond the relative durability of these logical qubits, the paper notes another advantage to going with larger code distances: it enhances the impact of further hardware improvements. Google estimates that at a distance of 15, improving hardware performance by a factor of two would drop errors in the logical qubit by a factor of 250. At a distance of 27, the same hardware improvement would lead to an improvement of over 10,000 in the logical qubit’s performance.

Note that none of this will ever get the error rate to zero. Instead, we just need to get the error rate to a level where an error is unlikely for a given calculation (more complex calculations will require a lower error rate). “It’s worth understanding that there’s always going to be some type of error floor and you just have to push it low enough to the point where it practically is irrelevant,” Kelly said. “So for example, we could get hit by an asteroid and the entire Earth could explode and that would be a correlated error that our quantum computer is not currently built to be robust to.”

Obviously, a lot of additional work will need to be done to both make logical qubits like this survive for even longer, and to ensure we have the hardware to host enough logical qubits to perform calculations. But the exponential improvements here, to Google, suggest that there’s nothing obvious standing in the way of that. “We woke up one morning and we kind of got these results and we were like, wow, this is going to work,” Newman said. “This is really it.”

Nature, 2024. DOI: 10.1038/s41586-024-08449-y  (About DOIs).

Photo of John Timmer

John is Ars Technica’s science editor. He has a Bachelor of Arts in Biochemistry from Columbia University, and a Ph.D. in Molecular and Cell Biology from the University of California, Berkeley. When physically separated from his keyboard, he tends to seek out a bicycle, or a scenic location for communing with his hiking boots.

Google gets an error-corrected quantum bit to be stable for an hour Read More »