Author name: Rejus Almole

google-announces-even-more-ai-in-photos-app,-powered-by-nano-banana

Google announces even more AI in Photos app, powered by Nano Banana

AI, AI images, Artificial Intelligence, Google, google photos, Tech / /

We’re running out of ways to tell you that Google is releasing more generative AI features, but that’s what’s happening in Google Photos today. The Big G is finally making good on its promise to add its market-leading Nano Banana image-editing model to the app. The model powers a couple of features, and it’s not just for Google’s Android platform. Nano Banana edits are also coming to the iOS version of the app.

Nano Banana started making waves when it appeared earlier this year as an unbranded demo. You simply feed the model an image and tell it what edits you want to see. Google said Nano Banana was destined for the Photos app back in October, but it’s only now beginning the rollout. The Photos app already had conversational editing in the “Help Me Edit” feature, but it was running an older non-fruit model that produced inferior results. Nano Banana editing will produce AI slop, yes, but it’s better slop.

Nano Banana in Help me edit

Google says the updated Help Me Edit feature has access to your private face groups, so you can use names in your instructions. For example, you could type “Remove Riley’s sunglasses,” and Nano Banana will identify Riley in the photo (assuming you have a person of that name saved) and make the edit without further instructions. You can also ask for more fantastical edits in Help Me Edit, changing the style of the image from top to bottom.

Google announces even more AI in Photos app, powered by Nano Banana Read More »

pirelli’s-cyber-tire-might-become-highway-agencies’-newest-assistant

Pirelli’s Cyber Tire might become highway agencies’ newest assistant

Cars, Pirelli Cyber Tire, Traffic / /

“Two weeks ago, a European manufacturer tested… the traction control and stability with a dramatic improvement in stability and the traction,” he said. “The nice part of the story is that there is not only an objective improvement—2 or 3 meters in braking distance—but there is also from these customers always a better feeling… which is something that is very important to us because numbers are for technicians, but from our customer’s perspective, the pleasure to drive also very important.”

The headline said something about traffic?

While the application described above mostly serves the Cyber Tire-equipped car, the smart tires can also serve the greater good. Earlier this year, we learned of a trial in the Italian region of Apulia that fitted Cyber Tires to a fleet of vehicles and then inferred the health of the road surface from data collected by the tires.

Working with a Swedish startup called Univrses, Pirelli has been fusing sensor data from the Cyber Tire with cameras. Misani offered an example.

“You have a hole [in the road]. If you have a hole, maybe the visual [system] recognizes and the tire does not because you automatically try to avoid the hole. So if the tire does not pass over the hole, you don’t measure anything,” he said. “But your visual system will detect it. On the opposite side, there are some cracks on the road that are detected from the visual system as something that is not even on the road, but they cannot say how deep, how is the step, how is it affecting the stability of the car and things like this. Matching the two things, you have the possibility to monitor in the best possible way the condition of the road.”

“Plus thanks to the vision, you have also the possibility to exploit what we call the vertical status—traffic signs, the compatibility between the condition of the road and the traffic signs,” he said.

The next step is a national program in Italy. “We are investigating and making a project to actively control not the control unit of the car but the traffic information,” Misani said. “On some roads, you can vary the speed limit according to the status; if we can detect aquaplaning, we can warn [that] at kilometer whatever, there is aquaplaning and [the speed limit will be automatically reduced]. We are going in the direction of integrating into the smart roads.”

Pirelli’s Cyber Tire might become highway agencies’ newest assistant Read More »

clickfix-may-be-the-biggest-security-threat-your-family-has-never-heard-of

ClickFix may be the biggest security threat your family has never heard of

Biz & IT, clickfix, malware, Scams, Security / /

Another campaign, documented by Sekoia, targeted Windows users. The attackers behind it first compromise a hotel’s account for Booking.com or another online travel service. Using the information stored in the compromised accounts, the attackers contact people with pending reservations, an ability that builds immediate trust with many targets, who are eager to comply with instructions, lest their stay be canceled.

The site eventually presents a fake CAPTCHA notification that bears an almost identical look and feel to those required by content delivery network Cloudflare. The proof the notification requires for confirmation that there’s a human behind the keyboard is to copy a string of text and paste it into the Windows terminal. With that, the machine is infected with malware tracked as PureRAT.

Push Security, meanwhile, reported a ClickFix campaign with a page “adapting to the device that you’re visiting from.” Depending on the OS, the page will deliver payloads for Windows or macOS. Many of these payloads, Microsoft said, are LOLbins, the name for binaries that use a technique known as living off the land. These scripts rely solely on native capabilities built into the operating system. With no malicious files being written to disk, endpoint protection is further hamstrung.

The commands, which are often base-64 encoded to make them unreadable to humans, are often copied inside the browser sandbox, a part of most browsers that accesses the Internet in an isolated environment designed to protect devices from malware or harmful scripts. Many security tools are unable to observe and flag these actions as potentially malicious.

The attacks can also be effective given the lack of awareness. Many people have learned over the years to be suspicious of links in emails or messengers. In many users’ minds, the precaution doesn’t extend to sites that instruct them to copy a piece of text and paste it into an unfamiliar window. When the instructions come in emails from a known hotel or at the top of Google results, targets can be further caught off guard.

With many families gathering in the coming weeks for various holiday dinners, ClickFix scams are worth mentioning to those family members who ask for security advice. Microsoft Defender and other endpoint protection programs offer some defenses against these attacks, but they can, in some cases, be bypassed. That means that, for now, awareness is the best countermeasure.

ClickFix may be the biggest security threat your family has never heard of Read More »

canada-fought-measles-and-measles-won;-virus-now-endemic-after-1998-elimination

Canada fought measles and measles won; virus now endemic after 1998 elimination

Canada, health, measles, mexico, PAHO, us, vaccination / /

“This loss represents a setback, of course, but it is also reversible,” Jarbas Barbosa, director of PAHO, said in a press briefing Monday.

Call to action

Barbosa was optimistic that Canada could regain its elimination status. He highlighted that such setbacks have happened before. “In 2018 and 2019, Venezuela and Brazil temporarily lost their elimination status following large outbreaks,” Barbosa noted. “Thanks to coordinated action by governments, civil society, and regional cooperation, those outbreaks were contained, and the Region of the Americas regained its measles-free status in 2024.”

On Monday, the Public Health Agency of Canada released a statement confirming that it received notification from PAHO that it had lost its measles elimination status, while reporting that it is already getting to work on earning it back. “PHAC is collaborating with the PAHO and working with federal, provincial, territorial, and community partners to implement coordinated actions—focused on improving vaccination coverage, strengthening data sharing, enabling better overall surveillance efforts, and providing evidence-based guidance,” the agency said.

However, Canada isn’t the only country facing an uphill battle against measles—the most infectious virus known to humankind. Outbreaks and sustained spread are also active in the US and Mexico. To date, the US has documented at least 1,618 measles cases since the start of the year, while Mexico has tallied at least 5,185. Bolivia, Brazil, Paraguay, and Belize also have ongoing outbreaks, PAHO reported.

As of November 7, PAHO has collected reports of 12,593 confirmed measles cases from 10 countries, but approximately 95 percent of them are in Canada, Mexico, and the US. That total is a 30-fold increase compared to 2024, PAHO notes, and the rise has led to at least 28 deaths: 23 in Mexico, three in the United States, and two in Canada.

The PAHO used Canada’s loss as a call to action not just in the northern country, but the rest of the region. “Every case we prevent, every outbreak we stop saves lives, protects families, and makes communities healthier,” Barbosa said. “Today, rather than lamenting the loss of a regional status, we call on all countries to redouble their efforts to strengthen vaccination rates, surveillance, and timely response to suspected cases—reaching every corner of the Americas. As a Region, we have eliminated measles twice. We can do it a third time.”

Canada fought measles and measles won; virus now endemic after 1998 elimination Read More »

new-project-brings-strong-linux-compatibility-to-more-classic-windows-games

New project brings strong Linux compatibility to more classic Windows games

compatibility layer, gaming, Linux, Open Source, steamos, valve, Windows / /

Those additional options should be welcome news for fans looking for new ways to play PC games of a certain era. The PC Gaming Wiki lists over 400 titles written with the D3D7 APIs, and while most of those games were released between 2000 and 2004, a handful of new D3D7 games have continued to be released through 2022.

The D3D7 games list predictably includes a lot of licensed shovelware, but there are also well-remembered games like Escape from Monkey Island, Arx Fatalis, and the original Hitman: Codename 47. WinterSnowfall writes that the project was inspired by a desire to play games like Sacrifice and Disciples II on top of the existing dxvk framework.

Despite some known issues with certain D3D7 titles, WinterSnowfall writes that recent tuning means “things are now anywhere between decent to stellar in most of the supported games.” Still, the project author warns that the project will likely never reach full compatibility since “D3D7 is a land of highly cursed API interoperability.”

Don’t expect this project to expand to include support for even older DirectX APIs, either, WinterSnowfall warns. “D3D7 is enough of a challenge and a mess as it is,” the author writes. “The further we stray from D3D9, the further we stray from the divine.”

New project brings strong Linux compatibility to more classic Windows games Read More »

10,000-generations-of-hominins-used-the-same-stone-tools-to-weather-a-changing-world

10,000 generations of hominins used the same stone tools to weather a changing world

Archaeology, australopithecus, hominins, oldowan, paleoarchaeology, Science, stone tools / /

“This site reveals an extraordinary story of cultural continuity,” said Braun in a recent press release.

When the going gets tough, the tough make tools

Nomorotukunan’s layers of stone tools span the transition from the Pliocene to the Pleistocene, during which Earth’s climate turned gradually cooler and drier after a 2 to 3 million-year warm spell. Pollen and other microscopic traces of plants in the sediment at Nomorotukunan tell the tale: the lakeshore marsh gradually dried up, giving way to arid grassland dotted with shrubs. On a shorter timescale, hominins at Nomorotukunan faced wildfires (based on microcharcoal in the sediments), droughts, and rivers drying up or changing course.

“As vegetation shifted, the toolmaking remained steady,” said National University of Kenya archaeologist Rahab N. Kinyanjui in a recent press release. “This is resilience.”

Making sharp stone tools may have helped generations of hominins survive their changing, drying world. In the warm, humid Pliocene, finding food would have been relatively easy, but as conditions got tougher, hominins probably had to scavenge or dig for their meals. At least one animal bone at Nomorotukunan bears cut marks where long-ago hominins carved up the carcass for meat—something our lineage isn’t really equipped to do with its bare hands and teeth. Tools also would have enabled early hominins to dig up and cut tubers or roots.

It’s fair to assume that sharpened wood sticks probably also played a role in that particular work, but wood doesn’t tend to last as long as stone in the archaeological record, so we can’t say for sure. What is certain are the stone tools and cut bones, which hint at what Utrecht University archaeologist Dan Rolier, a coauthor of the paper, calls “one of our oldest habits: using technology to steady ourselves against change.”

A tale as old as time

Nomorotukunan may hint that Oldowan technology is even older than the earliest tools archaeologists have unearthed so far. The oldest tools unearthed from the deepest layer at Nomorotukunan are the work of skilled flint-knappers who understood where to strike a stone, and at exactly which angle, to flake off the right shape. They also clearly knew how to select the right stones for the job (fine-grained chalcedony for the win, in this case). In other words, these tools weren’t the work of a bunch of hominins who were just figuring out, for the first time, how to bang the rocks together.

10,000 generations of hominins used the same stone tools to weather a changing world Read More »

mark-zuckerberg’s-illegal-school-drove-his-neighbors-crazy

Mark Zuckerberg’s illegal school drove his neighbors crazy

Features, mark zuckerberg, Meta, Policy, syndication, ZOning / /

Neighbors complained about noise, security guards, and hordes of traffic.

An entrance to Mark Zuckerberg’s compound in Palo Alto, California. Credit: Loren Elliott/Redux

The Crescent Park neighborhood of Palo Alto, California, has some of the best real estate in the country, with a charming hodgepodge of homes ranging in style from Tudor revival to modern farmhouse and contemporary Mediterranean. It also has a gigantic compound that is home to Mark Zuckerberg, his wife Priscilla Chan, and their daughters Maxima, August, and Aurelia. Their land has expanded to include 11 previously separate properties, five of which are connected by at least one property line.

The Zuckerberg compound’s expansion first became a concern for Crescent Park neighbors as early as 2016, due to fears that his purchases were driving up the market. Then, about five years later, neighbors noticed that a school appeared to be operating out of the Zuckerberg compound. This would be illegal under the area’s residential zoning code without a permit. They began a crusade to shut it down that did not end until summer 2025.

WIRED obtained 1,665 pages of documents about the neighborhood dispute—including 311 records, legal filings, construction plans, and emails—through a public record request filed to the Palo Alto Department of Planning and Development Services. (Mentions of “Zuckerberg” or “the Zuckerbergs” appear to have been redacted. However, neighbors and separate public records confirm that the property in question belongs to the family. The names of the neighbors who were in touch with the city were also redacted.)

The documents reveal that the school may have been operating as early as 2021 without a permit to operate in the city of Palo Alto. As many as 30 students might have enrolled, according to observations from neighbors. These documents also reveal a wider problem: For almost a decade, the Zuckerbergs’ neighbors have been complaining to the city about noisy construction work, the intrusive presence of private security, and the hordes of staffers and business associates causing traffic and taking up street parking.

Over time, neighbors became fed up with what they argued was the city’s lack of action, particularly with respect to the school. Some believed that the delay was because of preferential treatment to the Zuckerbergs. “We find it quite remarkable that you are working so hard to meet the needs of a single billionaire family while keeping the rest of the neighborhood in the dark,” reads one email sent to the city’s Planning and Development Services Department in February. “Just as you have not earned our trust, this property owner has broken many promises over the years, and any solution which depends on good faith behavioral changes from them is a failure from the beginning.”

Palo Alto spokesperson Meghan Horrigan-Taylor told WIRED that the city “enforces zoning, building, and life safety rules consistently, without regard to who owns a property.” She also refuted the claim that neighbors were kept in the dark, claiming that the city’s approval of construction projects at the Zuckerberg properties “were processed the same way they are for any property owner.” She added that, though some neighbors told the city they believe the Zuckerbergs received “special treatment,” that is not accurate.

“Staff met with residents, conducted site visits, and provided updates by phone and email while engaging the owner’s representative to address concerns,” Horrigan-Taylor said. “These actions were measured and appropriate to abate the unpermitted use and responsive to neighborhood issues within the limits of local and state law.”

According to The New York Times, which first reported on the school’s existence, it was called “Bicken Ben School” and shared a name with one of the Zuckerbergs’ chickens. The listing for Bicken Ben School, or BBS for short, in a California Department of Education directory claims the school opened on October 5, 2022. This, however, is the year after neighbors claim to have first seen it operating. It’s also two and a half years after Sara Berge—the school’s point of contact, per documents WIRED obtained from the state via public record request—claims to have started her role as “head of school” for a “Montessori pod” at a “private family office” according to her LinkedIn profile, which WIRED viewed in September and October. Berge did not respond to a request to comment.

Between 2022 and 2025, according to the documents Bicken Ben filed to the state, the school grew from nine to 14 students ranging from 5 to 10 years old. Neighbors, however, estimated that they observed 15 to 30 students. Berge similarly claimed on her LinkedIn profile to have overseen “25 children” in her job. In a June 2025 job listing for “BBS,” the school had a “current enrollment of 35–40 students and plans for continued growth,” which the listing says includes a middle school.

In order for the Zuckerbergs to run a private school on their land, which is in a residential zone, they need a “conditional use” permit from the city. However, based on the documents WIRED obtained, and Palo Alto’s public database of planning applications, the Zuckerbergs do not appear to have ever applied for or received this permit.

Per emails obtained by WIRED, Palo Alto authorities told a lawyer working with the Zuckerbergs in March 2025 that the family had to shut down the school on its compound by June 30. A state directory lists BBS, the abbreviation for Bicken Ben School, as having operated until August 18, and three of Zuckerberg’s neighbors—who all requested anonymity due to the high-profile nature of the family—confirmed to WIRED in late September that they had not seen or heard students being dropped off and picked up on weekdays in recent weeks.

However, Zuckerberg family spokesperson Brian Baker tells WIRED that the school didn’t close, per se. It simply moved. It’s not clear where it is now located, or whether the school is operating under a different name.

In response to a detailed request for comment, Baker provided WIRED with an emailed statement on behalf of the Zuckerbergs. “Mark, Priscilla and their children have made Palo Alto their home for more than a decade,” he said. “They value being members of the community and have taken a number of steps above and beyond any local requirements to avoid disruption in the neighborhood.”

“Serious and untenable”

By the fall of 2024, Zuckerberg’s neighbors were at their breaking point. At some point in mid-2024, according to an email from then mayor Greer Stone, a group of neighbors had met with Stone to air their grievances about the Zuckerberg compound and the illegal school they claimed it was operating. They didn’t arrive at an immediate resolution.

In the years prior, the city had received several rounds of complaints about the Zuckerberg compound. Complaints for the address of the school were filed to 311, the nationwide number for reporting local non-emergency issues, in February 2019, September 2021, January 2022, and April 2023. They all alleged that the property was operating illegally under city code. Both were closed by the planning department, which found no rule violations. An unknown number of additional complaints, mentioned in emails among city workers, were also made between 2020 and 2024—presumably delivered via phone calls, in person, or to city departments not included in WIRED’s public record request.

In December 2020, building inspection manager Korwyn Peck wrote to code enforcement officer Brian Reynolds about an inspection he attempted to conduct around the Zuckerberg compound, in response to several noise and traffic complaints from neighbors. He described that several men in SUVs had gathered to watch him, and a tense conversation with one of them had ensued. “This appears to be a site that we will need to pay attention to,” Peck wrote to Reynolds.

“We have all been accused of ‘not caring,’ which of course is not true,” Peck added. “It does appear, however, with the activity I observed tonight, that we are dealing with more than four simple dwellings. This appears to be more than a homeowner with a security fetish.”

In a September 11, 2024, email to Jonathan Lait, Palo Alto’s director of planning and development services and Palo Alto city attorney Molly Stump, one of Zuckerberg’s neighbors alleged that since 2021, “despite numerous neighborhood complaints” to the city of Palo Alto, including “multiple code violation reports,” the school had continued to grow. They claimed that a garage at the property had been converted into another classroom, and that an increasing number of children were arriving each day. Lait and Stump did not respond to a request to comment.

“The addition of daily traffic from the teachers and parents at the school has only exacerbated an already difficult situation,” they said in the email, noting that the neighborhood has been dealing with an “untenable traffic” situation for more than eight years.

They asked the city to conduct a formal investigation into the school on Zuckerberg’s property, adding that their neighbors are also “extremely concerned” about the school, and “are willing to provide eyewitness accounts in support of this complaint.”

Over the next week, another neighbor forwarded this note to all six Palo Alto city council members, as well as then mayor Stone. One of these emails described the situation as “serious” and “untenable.”

“We believe the investigation should be swift and should yield a cease and desist order,” the neighbor wrote.

Lait responded to the neighbor who sent the original complaint on October 15, claiming that he’d had an “initial call” with a “representative” of the property owners and that he was directing the city’s code enforcement staff to reexamine the property.

On December 11, 2024, the neighbor claimed that since one of their fellow neighbors had spoken to a Zuckerberg representative, and the representative had allegedly admitted that there was a school on the property, “it seems like an open and shut case.”

“Our hope is that there is an equal process in place for all residents of Palo Alto regardless of wealth or stature,” the neighbor wrote. “It is hard to imagine that this kind of behavior would be ignored in any other circumstance.”

That same day, Lait told Christine Wade, a partner at SSL Law Firm—who, in an August 2024 email thread, said she was “still working with” the Zuckerberg family—that the Zuckerbergs lacked the required permit to run a school in a residential zone.

“Based on our review of local and state law, we believe this use constitutes a private school use in a residential zone requiring a conditional use permit,” Lait wrote in an email to Wade. “We also have not found any state preemptions that would exclude a use like this from local zoning requirements.” Lait added that a “next step,” if a permit was not obtained, would be sending a cease and desist to the property owner.

According to several emails, Wade, Lait, and Mark Legaspi, CEO of the Zuckerberg family office called West 10, went on to arrange an in-person meeting at City Hall on January 9. (This is the first time that the current name of the Zuckerberg family office, West 10, has been publicly disclosed. The office was previously called West Street.) Although WIRED did not obtain notes from the meeting, Lait informed the neighbor on January 10 that he had told the Zuckerbergs’ “representative” that the school would need to shut down if it didn’t get a conditional use permit or apply for that specific permit.

Lait added that the representative would clarify what the family planned to do in about a week; however, he noted that if the school were to close, the city may give the school a “transition period” to wind things down. Wade did not respond to a request for comment.

“At a minimum, give us extended breaks”

There was another increasingly heated conversation happening behind the scenes. On February 3 of this year, at least one neighbor met with Jordan Fox, an employee of West 10.

It’s unclear exactly what happened at this meeting, or if the neighbor who sent the September 11 complaint was in attendance. But a day after the meeting with Fox, two additional neighbors added their names to the September 11 complaint, per an email to Lait.

On February 12, a neighbor began an email chain with Fox. This email was forwarded to Planning Department officials two months later. The neighbor, who seemingly attended the meeting, said they had “connected” with fellow neighbors “to review and revise” an earlier list of 14 requests that had been reportedly submitted to the Zuckerbergs at some previous point. The note does not specify the contents of this original list of requests, but of the 19 neighbors who originally contributed to it, they claimed that 15 had contributed to the revised list.

The email notes that the Zuckerbergs had been “a part of our neighborhood for many years,” and that they “hope that this message will start an open and respectful dialogue,” built upon the “premise of how we all wish to be treated as neighbors.”

“Our top requests are to minimize future disruption to the neighborhood and proactively manage the impact of the many people who are affiliated with you,” the email says. This includes restricting parking by “security guards, contractors, staff, teachers, landscapers, visitors, etc.” In the event of major demolitions, concrete pours, or large parties, the email asks for advance notice, and for dedicated efforts to “monitor and mitigate noise.”

The email also asks the Zuckerbergs to, “ideally stop—but at a minimum give us extended breaks from—the acquisition, demolition and construction cycle to let the neighborhood recover from the last eight years of disruption.”

At this point, the email requests that the family “abide by both the letter and the spirit of Palo Alto” by complying with city code about residential buildings.

Specifically, it asks the Zuckerbergs to get a use permit for the compound’s school and to hold “a public hearing for transparency.” It also asks the family to not expand its compound any further. “We hope this will help us get back the quiet, attractive residential neighborhood that we all loved so much when we chose to move here.”

In a follow-up on March 4, Fox acknowledged the “unusual” effects that come with being neighbors with Mark Zuckerberg and his family.

“I recognize and understand that the nature of our residence is unique given the profile and visibility of the family,” she wrote. “I hope that as we continue to grow our relationship with you over time, you will increasingly enjoy the benefits of our proximity—e.g., enhanced safety and security, shared improvements, and increased property values.”

Fox said that the Zuckerbergs instituted “a revised parking policy late last year” that should address their concerns, and promised to double down on efforts to give advanced notice about construction, parties, and other potential disruptions.

However, Fox did not directly address the unpermitted school and other nonresidential activities happening at the compound. She acknowledged that the compound has “residential support staff” including “childcare, culinary, personal assistants, property management, and security,” but said that they have “policies in place to minimize their impact on the neighborhood.”

It’s unclear if the neighbor responded to Fox.

“You have not earned our trust”

While these conversations were happening between Fox and Zuckerberg’s neighbors, Lait and others at the city Planning Department were scrambling to find a solution for the neighbor who complained on September 11, and a few other neighbors who endorsed the complaint in September and February.

Starting in February, one of these neighbors took the lead on following up with Lait. They asked him for an update on February 11, and heard back a few days later. He didn’t have any major updates, “but after conversations with the family’s representatives, he said he was exploring whether a “subset of children” could continue to come to the school sometimes for “ancillary” uses.

“I also believe a more nuanced solution is warranted in this case,” Lait added. Ideally, such a solution would respond to the neighbors’ complaints, but allow the Zuckerbergs to “reasonably be authorized by the zoning code.”

The neighbor wasn’t thrilled. The next day, they replied and called the city’s plan “unsatisfactory.”

“The city’s ‘nuanced solution’ in dealing with this serial violator has led to the current predicament,” they said (referring to the nuanced solution Lait mentioned in his last email.)

Horrigan-Taylor, the Palo Alto spokesperson, told WIRED that Lait’s mention of a “nuanced” solution referred to “resolving, to the extent permissible by law, neighborhood impacts and otherwise permitted use established by state law and local zoning.”

“Would I, or any other homeowner, be given the courtesy of a ‘nuanced solution’ if we were in violation of city code for over four years?” they added.

“Please know that you have not earned our trust and that we will take every opportunity to hold the city accountable if your solution satisfies a single [redacted] property owner over the interests of an entire neighborhood,” they continued.

“If you somehow craft a ‘nuanced solution’ based on promises,” the neighbor said, “the city will no doubt once again simply disappear and the damage to the neighborhood will continue.”

Lait did not respond right away. The neighbor followed up on March 13, asking if he had “reconsidered” his plan to offer a “‘nuanced solution’ for resolution of these ongoing issues by a serial code violator.” They asked when the neighborhood could “expect relief from the almost decade long disruptions.”

Behind the scenes, Zuckerberg’s lawyers were fighting to make sure the school could continue to operate. In a document dated March 14, Wade argues that she believed the activities at “the Property” “represent an appropriate residential use based on established state law as well as constitutional principles.”

Wade said that “the Family” was in the process of obtaining a “Large Family Daycare” license for the property, which is legal for a cohort of 14 or fewer children all under the age of 10.

“We consistently remind our vendors, guests, etc. to minimize noise, not loiter anywhere other than within the Family properties, and to keep areas clean,” Wade added in the letter. Wade also attached an adjusted lease corresponding with the address of the illicit school, which promises that the property will be used for only one purpose. The exact purpose is redacted.

On March 25, Lait told the neighbor that the city’s June 30 deadline for the Zuckerbergs to shut down the school had not changed. However, the family’s representative said that they were pursuing a daycare license. These licenses are granted by the state, not the city of Palo Alto.

The subtext of this email was that if the state gave them a daycare licence, there wasn’t much the city could do. Horrigan-Taylor confirmed with WIRED that “state licensed large family day care homes” do not require city approval, adding that the city also “does not regulate homeschooling.”

“Thanks for this rather surprising information,” the neighbor replied about a week later. “We have repeatedly presented ideas to the family over the past 8 years with very little to show for it, so from our perspective, we need to understand the city’s willingness to act or not to act.”

Baker told WIRED that the Zuckerbergs never ended up applying for a daycare license, a claim that corresponds with California’s public registry of daycare centers. (There are only two registered daycare centers in Palo Alto, and neither belongs to the Zuckerbergs. The Zuckerbergs’ oldest child, Maxima, will also turn 10 in December and consequently age out of any daycare legally operating in California.)

Horrigan-Taylor said that a representative for the Zuckerbergs told the city that the family wanted to move the school to “another location where private schools are permitted by right.”

In a school administrator job listing posted to the Association Montessori International website in July 2022 for “BBS,” Bicken Ben head of school Berge claims that the school had four distinct locations, and that applicants must be prepared to travel six to eight weeks per year. The June 2025 job listing also says that the “year-round” school spans “across multiple campuses,” but the main location of the job is listed as Palo Alto. It’s unclear where the other sites are located.

Most of the Zuckerbergs’ neighbors did not respond to WIRED’s request for comment. However, the ones that did clearly indicated that they would not be forgetting the Bicken Ben saga, or the past decade of disruption, anytime soon.

“Frankly I’m not sure what’s going on,” one neighbor said, when reached by WIRED via landline. “Except for noise and construction debris.”

This story originally appeared on wired.com.

Photo of WIRED

Wired.com is your essential daily guide to what’s next, delivering the most original and complete take you’ll find anywhere on innovation’s impact on technology, science, business and culture.

Mark Zuckerberg’s illegal school drove his neighbors crazy Read More »

the-government-shutdown-is-starting-to-have-cosmic-consequences

The government shutdown is starting to have cosmic consequences

blue origin, Commercial space, Federal Aviation Administration, launch, Science, Space, spacex, starlink / /

The federal government shutdown, now in its 38th day, prompted the Federal Aviation Administration to issue a temporary emergency order Thursday prohibiting commercial rocket launches from occurring during “peak hours” of air traffic.

The FAA also directed commercial airlines to reduce domestic flights from 40 “high impact airports” across the country in a phased approach beginning Friday. The agency said the order from the FAA’s administrator, Bryan Bedford, is aimed at addressing “safety risks and delays presented by air traffic controller staffing constraints caused by the continued lapse in appropriations.”

The government considers air traffic controllers essential workers, so they remain on the job without pay until Congress passes a federal budget and President Donald Trump signs it into law. The shutdown’s effects, which affected federal workers most severely at first, are now rippling across the broader economy.

Sharing the airspace

Vehicles traveling to and from space share the skies with aircraft, requiring close coordination with air traffic controllers to clear airspace for rocket launches and reentries. The FAA said its order restricting commercial air traffic, launches, and reentries is intended to “ensure the safety of aircraft and the efficiency of the [National Airspace System].”

In a statement explaining the order, the FAA said the air traffic control system is “stressed” due to the shutdown.

“With continued delays and unpredictable staffing shortages, which are driving fatigue, risk is further increasing, and the FAA is concerned with the system’s ability to maintain the current volume of operations,” the regulator said. “Accordingly, the FAA has determined additional mitigation is necessary.”

Beginning Monday, the FAA said commercial space launches will only be permitted between 10 pm and 6 am local time, when the national airspace is most calm. The order restricts commercial reentries to the same overnight timeframe. The FAA licenses all commercial launches and reentries.

The government shutdown is starting to have cosmic consequences Read More »

oddest-chatgpt-leaks-yet:-cringey-chat-logs-found-in-google-analytics-tool

Oddest ChatGPT leaks yet: Cringey chat logs found in Google analytics tool

AI, chatbot, chatbot leaks, chatgpt, Google, google search, google search console. privacy, openai, Policy, SEO / /

ChatGPT leaks seem to confirm OpenAI scrapes Google, expert says.

Credit: Aurich Lawson | Getty Images

For months, extremely personal and sensitive ChatGPT conversations have been leaking into an unexpected destination: Google Search Console (GSC), a tool that developers typically use to monitor search traffic, not lurk private chats.

Normally, when site managers access GSC performance reports, they see queries based on keywords or short phrases that Internet users type into Google to find relevant content. But starting this September, odd queries, sometimes more than 300 characters long, could also be found in GSC. Showing only user inputs, the chats appeared to be from unwitting people prompting a chatbot to help solve relationship or business problems, who likely expected those conversations would remain private.

Jason Packer, owner of an analytics consulting firm called Quantable, was among the first to flag the issue in a detailed blog last month.

Determined to figure out what exactly was causing the leaks, he teamed up with “Internet sleuth” and web optimization consultant Slobodan Manić. Together, they conducted testing that they believe may have surfaced “the first definitive proof that OpenAI directly scrapes Google Search with actual user prompts.” Their investigation seemed to confirm the AI giant was compromising user privacy, in some cases in order to maintain engagement by seizing search data that Google otherwise wouldn’t share.

OpenAI declined Ars’ request to confirm if Packer and Manić’s theory posed in their blog was correct or answer any of their remaining questions that could help users determine the scope of the problem.

However, an OpenAI spokesperson confirmed that the company was “aware” of the issue and has since “resolved” a glitch “that temporarily affected how a small number of search queries were routed.”

Packer told Ars that he’s “very pleased that OpenAI was able to resolve the issue quickly.” But he suggested that OpenAI’s response failed to confirm whether or not OpenAI was scraping Google, and that leaves room for doubt that the issue was completely resolved.

Google declined to comment.

“Weirder” than prior ChatGPT leaks

The first odd ChatGPT query to appear in GSC that Packer reviewed was a wacky stream-of-consciousness from a likely female user asking ChatGPT to assess certain behaviors to help her figure out if a boy who teases her had feelings for her. Another odd query seemed to come from an office manager sharing business information while plotting a return-to-office announcement.

These were just two of 200 odd queries—including “some pretty crazy ones,” Packer told Ars—that he reviewed on one site alone. In his blog, Packer concluded that the queries should serve as “a reminder that prompts aren’t as private as you think they are!”

Packer suspected that these queries were connected to reporting from The Information in August that cited sources claiming OpenAI was scraping Google search results to power ChatGPT responses. Sources claimed that OpenAI was leaning on Google to answer prompts to ChatGPT seeking information about current events, like news or sports.

OpenAI has not confirmed that it’s scraping Google search engine results pages (SERPs). However, Packer thinks his testing of ChatGPT leaks may be evidence that OpenAI not only scrapes “SERPs in general to acquire data,” but also sends user prompts to Google Search.

Manić helped Packer solve a big part of the riddle. He found that the odd queries were turning up in one site’s GSC because it ranked highly in Google Search for “https://openai.com/index/chatgpt/”—a ChatGPT URL that was appended at the start of every strange query turning up in GSC.

It seemed that Google had tokenized the URL, breaking it up into a search for keywords “openai + index + chatgpt.” Sites using GSC that ranked highly for those keywords were therefore likely to encounter ChatGPT leaks, Parker and Manić proposed, including sites that covered prior ChatGPT leaks where chats were being indexed in Google search results. Using their recommendations to seek out queries in GSC, Ars was able to verify similar strings.

“Don’t get confused though, this is a new and completely different ChatGPT screw-up than having Google index stuff we don’t want them to,” Packer wrote. “Weirder, if not as serious.”

It’s unclear what exactly OpenAI fixed, but Packer and Manić have a theory about one possible path for leaking chats. Visiting the URL that starts every strange query found in GSC, ChatGPT users encounter a prompt box that seemed buggy, causing “the URL of that page to be added to the prompt.” The issue, they explained, seemed to be that:

Normally ChatGPT 5 will choose to do a web search whenever it thinks it needs to, and is more likely to do that with an esoteric or recency-requiring search. But this bugged prompt box also contains the query parameter ‘hints=search’ to cause it to basically always do a search: https://chatgpt.com/?hints=search&openaicom_referred=true&model=gpt-5

Clearly some of those searches relied on Google, Packer’s blog said, mistakenly sending to GSC “whatever” the user says in the prompt box, with “https://openai.com/index/chatgpt/” text added to the front of it.” As Packer explained, “we know it must have scraped those rather than using an API or some kind of private connection—because those other options don’t show inside GSC.”

This means “that OpenAI is sharing any prompt that requires a Google Search with both Google and whoever is doing their scraping,” Packer alleged. “And then also with whoever’s site shows up in the search results! Yikes.”

To Packer, it appeared that “ALL ChatGPT prompts” that used Google Search risked being leaked during the past two months.

OpenAI claimed only a small number of queries were leaked but declined to provide a more precise estimate. So, it remains unclear how many of the 700 million people who use ChatGPT each week had prompts routed to GSC.

OpenAI’s response leaves users with “lingering questions”

After ChatGPT prompts were found surfacing in Google’s search index in August, OpenAI clarified that users had clicked a box making those prompts public, which OpenAI defended as “sufficiently clear.” The AI firm later scrambled to remove the chats from Google’s SERPs after it became obvious that users felt misled into sharing private chats publicly.

Packer told Ars that a major difference between those leaks and the GSC leaks is that users harmed by the prior scandal, at least on some level, “had to actively share” their leaked chats. In the more recent case, “nobody clicked share” or had a reasonable way to prevent their chats from being exposed.

“Did OpenAI go so fast that they didn’t consider the privacy implications of this, or did they just not care?” Packer posited in his blog.

Perhaps most troubling to some users—whose identities are not linked in chats unless their prompts perhaps share identifying information—there does not seem to be any way to remove the leaked chats from GSC, unlike the prior scandal.

Packer and Manić are left with “lingering questions” about how far OpenAI’s fix will go to stop the issue.

Manić was hoping OpenAI might confirm if prompts entered on https://chatgpt.com that trigger Google Search were also affected. But OpenAI did not follow up on that question, or a broader question about how big the leak was. To Manić, a major concern was that OpenAI’s scraping may be “contributing to ‘crocodile mouth’ in Google Search Console,” a troubling trend SEO researchers have flagged that causes impressions to spike but clicks to dip.

OpenAI also declined to clarify Packer’s biggest question. He’s left wondering if the company’s “fix” simply ended OpenAI’s “routing of search queries, such that raw prompts are no longer being sent to Google Search, or are they no longer scraping Google Search at all for data?

“We still don’t know if it’s that one particular page that has this bug or whether this is really widespread,” Packer told Ars. “In either case, it’s serious and just sort of shows how little regard OpenAI has for moving carefully when it comes to privacy.”

Photo of Ashley Belanger

Ashley is a senior policy reporter for Ars Technica, dedicated to tracking social impacts of emerging policies and new technologies. She is a Chicago-based journalist with 20 years of experience.

Oddest ChatGPT leaks yet: Cringey chat logs found in Google analytics tool Read More »

on-sam-altman’s-second-conversation-with-tyler-cowen

On Sam Altman’s Second Conversation with Tyler Cowen

Altman's / /

Some podcasts are self-recommending on the ‘yep, I’m going to be breaking this one down’ level. This was very clearly one of those. So here we go.

As usual for podcast posts, the baseline bullet points describe key points made, and then the nested statements are my commentary.

If I am quoting directly I use quote marks, otherwise assume paraphrases.

The entire conversation takes place with an understanding that no one is to mention existential risk or the fact that the world will likely transform, without stating this explicitly. Both participants are happy to operate that way. I’m happy to engage in that conversation (while pointing out its absurdity in some places), but assume that every comment I make has an implicit ‘assuming normality’ qualification on it, even when I don’t say so explicitly.

  1. Cowen asks how Altman got so productive, able to make so many deals and ship so many products. Altman says people almost never allocate their time efficiently, and that when you have more demands on your time you figure out how to improve. Centrally he figures out what the core things to do are and delegates. He says deals are quicker now because everyone wants to work with OpenAI.

    1. Altman’s definitely right that most people are inefficient with their time.

    2. Inefficiency is relative. As in, I think of myself as inefficient with my time, and think of the ways I could be a lot more efficient.

    3. Not everyone responds to pressure by improving efficiency, far from it.

    4. Altman is good here to focus on delegation.

    5. It is indeed still remarkable how many things OpenAI is doing at once, with the associated worries about it potentially being too many things, and not taking the time to do them responsibly.

  1. What makes hiring in hardware different from in AI? Cycles are longer. Capital is more intense. So more time invested up front to pick wisely. Still want good, effective, fast-moving people and clear goals.

    1. AI seems to be getting pretty capital intensive?

  2. Nvidia’s people ‘are less weird’ and don’t read Twitter. OpenAI’s hardware people feel more like their software people than they feel like Nvidia’s people.

    1. My guess is there isn’t a right answer but you need to pick a lane.

  3. What makes Roon special? Lateral thinker, great at phrasing observations, lots of disparate skills in one place.

    1. I would add some more ingredients. There’s a sense of giving zero fucks, of having no filter, and having no agenda. Say things and let the chips fall.

    2. A lot of the disparate skills are disparate aesthetics, including many that are rare in AI, and taking all of them seriously at once.

  4. Altman doesn’t tell researchers what to work on. Researchers choose, that’s it.

  5. Email is very bad. Slack might not be good, it creates explosions of work including fake work to deal with, especially the first and last hours, but it is better than email. Altman suspects it’s time for a new AI-driven thing but doesn’t have it yet, probably due to lack of trying and unwillingness to pay focus and activation energy given everything else going on.

    1. I think email is good actually, and that Slack is quite bad.

    2. Email isn’t perfect but I like that you decide what you have ownership of, how you organize it, how you keep it, when you check it, and generally have control over the experience, and that you can choose how often you check it and aren’t being constantly pinged or expected to get into chat exchanges.

    3. Slack is an interruption engine without good information organization and I hate it so much, as in ‘it’s great I don’t have a job where I need slack.’

    4. There’s definitely room to build New Thing that integrates AI into some mix of information storage and retrieval, email slow communication, direct messaging and group chats, and which allows you to prioritize and get the right levels of interruption at the right times, and so on.

    5. However this will be tricky, you need to be ten times better and you can’t break the reliances people have. False negatives, where things get silently buried, can be quite bad.

  1. What will make GPT-6 special? Altman suggests it might be able to ‘really do’ science. He doesn’t have much practical advice on what to do with that.

    1. This seems like we hit the wall of ‘…and nothing will change much’ forcing Altman to go into contortions.

    2. One thing we learned from GPT-5 is that the version numbers don’t have to line up with big capabilities leaps. The numbers are mostly arbitrary.

Tyler isn’t going to let him off that easy. At this point, I don’t normally do this, but exact words seem important so I’m going to quite the transcript.

COWEN: If I’m thinking about restructuring an entire organization to have GPT-6 or 7 or whatever at the center of it, what is it I should be doing organizationally, rather than just having all my top people use it as add-ons to their current stock of knowledge?

ALTMAN: I’ve thought about this more for the context of companies than scientists, just because I understand that better. I think it’s a very important question. Right now, I have met some orgs that are really saying, “Okay, we’re going to adopt AI and let AI do this.” I’m very interested in this, because shame on me if OpenAI is not the first big company run by an AI CEO, right?

COWEN: Just parts of it. Not the whole thing.

ALTMAN: No, the whole thing.

COWEN: That’s very ambitious. Just the finance department, whatever.

ALTMAN: Well, but eventually it should get to the whole thing, right? So we can use this and then try to work backwards from that. I find this a very interesting thought experiment of what would have to happen for an AI CEO to be able to do a much better job of running OpenAI than me, which clearly will happen someday. How can we accelerate that? What’s in the way of that? I have found that to be a super useful thought experiment for how we design our org over time and what the other pieces and roadblocks will be. I assume someone running a science lab should try to think the same way, and they’ll come to different conclusions.

COWEN: How far off do you think it is that just, say, one division of OpenAI is 85 percent run by AIs?

ALTMAN: Any single division?

COWEN: Not a tiny, insignificant division, mostly run by the AIs.

ALTMAN: Some small single-digit number of years, not very far. When do you think I can be like, “Okay, Mr. AI CEO, you take over”?

COWEN: CEO is tricky because the public role of a CEO, as you know, becomes more and more important.

  1. On the above in terms of ‘oh no’:

    1. Oh no. Exactly the opposite. Shame on him if OpenAI goes first.

    2. OpenAI is the company, in this scenario, out of all the companies, we should be most worried about handing over to an AI CEO, for obvious reasons.

    3. If you’re wondering how the AIs could take over? You can stop wondering. They will take over because we will ask them to.

    4. CEO is an adversarial and anti-inductive position, where any weakness will be systematically exploited, and big mistakes can entirely sink you, and the way that you direct and set up the ‘AI CEO’ matters quite a lot in all this. The bar to a net positive AI CEO is much higher than the AI making on average better decisions, or having on average better features, and the actual bar is higher. Altman says ‘on the actual decision making maybe the AI is pretty good soon’ but this is a place where I’m going to be the Bottleneck Guy.

    5. CEO is also a position where, very obviously, misaligned means your company can be extremely cooked, and basically everything in it subverted, even if that CEO is a single human. Most of the ways in which this is limited are because the CEO can only be in one place at a time and do one thing at a time, couldn’t keep an eye on most things let alone micromanage them, and would require conspirators. A hostile AI CEO is death or subversion of the company.

    6. The ‘public role’ of the CEO being the bottleneck does not bring comfort here. If Altman (as he suggests) is public face and the AI ‘figures out what to do’ and Altman doesn’t actually get to overrule the AI (or is simply convinced not to) then the problem remains.

  2. On the above in terms of ‘oh yeah’:

    1. There is the clear expectation from both of them that AI will rise, reasonably soon, to the level of at least ‘run the finance department of a trillion dollar corporation.’ This doesn’t have to be AGI but it probably will be, no?

    2. It’s hard for me to square ‘AIs are running the actual decision making at top corporations’ with predictions for only modest GDP growth. As Altman notes, the AI CEO needs to be a lot better than the human CEO in order to get the job.

    3. They are predicting billion-dollar 2-3 person companies, with AIs, within three years.

  3. Altman asks potential hires about their use of AI now to predict their level of AI adoption in the future, which seems smart. Using it as ‘better Google’ is a yellow flag, thinking about day-to-day in three years is a green flag.

  4. In three years Altman is aiming to have a ‘fully automated AI researcher.’ So it’s pretty hard to predict day-to-day use in three years.

A timely section title.

  1. Cowen and Altman are big fans of nuclear power (as am I), but people worry about them. Cowen asks, do you worry similarly about AI and the similar Nervous Nellies, even if ‘AI is pretty safe’? Are the Feds your insurer? How will you insure everything?

    1. Before we get to Altman’s answer can we stop to think about how absolutely insane this question is as presented?

    2. Cowen is outright equating worries about AI to worries about nuclear power, calling both Nervous Nellies. My lord.

    3. The worry about AI risks is that the AI companies might be held too accountable? Might be asked to somehow provide too much insurance, when there is clearly no sign of any such requirement for the most important risks? They are building machines that will create substantial catastrophic and even existential risks, massive potential externalities.

    4. And you want the Federal Government to actively insure against AI catastrophic risks? To say that it’s okay, we’ve got you covered? This does not, in any way, actually reduce the public’s or world’s exposure to anything, and it further warps company incentives. It’s nuts.

    5. Not that even the Federal Government can actually ensure us here even at our own expense, since existential risk or sufficiently large catastrophic or systemic risk also wipes out the Federal Government. That’s kind of the point.

    6. The idea that the people are the Nervous Nellies around nuclear, which has majority public support, while Federal Government is the one calming them down and ensuring things can work is rather rich.

    7. Nuclear power regulations are insanely restrictive and prohibitive, and the insurance the government writes does not substantially make up for this, nor is it that expensive or risky. The NRC and other regulations are the reason we can’t have this nice thing, in ways that don’t relate much if at all to the continued existence of these Nervous Nellies. Providing safe harbor in exchange of that really is the actual least you can do.

    8. AI regulations impose very few rules and especially very few safety rules.

    9. Yes, there is the counterpoint that AI has to follow existing rules and thus is effectively rather regulated, but I find this rather silly as an argument, and no I don’t think the new laws around AI in particular move that needle much.

  2. Altman points out the Federal Government is the insurer of last resort for anything sufficiently large, whether you want it to be or not, but no not in the way of explicitly writing insurance policies.

    1. I mean yes if AI crashes the economy or does trillions in damages or what not, then the Federal Government will have to try and step in. This is a huge actual subsidy to the AI companies and they should (in theory anyway) be pay for it.

    2. A bailout for the actual AI companies if they are simply going bankrupt? David Sacks has made it clear our answer is no thank you, and rightfully so. Obviously, at some point the Fed Put or Trump Put comes into play in the stock market, that ship has sailed, but no we will not save your loans.

    3. And yeah, my lord, the idea that the Feds would write an insurance policy.

  3. Cowen then says he is worried about the Feds being the insurer of first resort and he doesn’t want that, Altman confirms he doesn’t either and doesn’t expect it.

    1. It’s good that they don’t want this to happen but this only slightly mitigates my outrage at the first question and the way it was presented.

  4. Cowen points out Trump is taking equity in Intel, lithium and rare earths, and asks how this applies to OpenAI. Altman mostly dodges, pivots to potential loss of meaning in the world, and points out the government might have strong opinions about AI company actions.

    1. Cowen doesn’t say it here but to his credit is on record correctly opposing this taking of equity in companies correctly identifying it as ‘seizing the means of production’ and pointing out it is the wrong tool for the job.

    2. This really was fully a non-answer. I see why that might be wise.

    3. Could OpenAI be coerced into giving up equity, or choose to do so as part of a regulatory capture play? Yeah. It would be a no-good, very bad thing.

    4. The government absolutely will and needs to have strong opinions about AI company actions and set the regulations and rules in place and otherwise play the role of being the actual government.

    5. If the government does not govern the AI companies, then the government will wake up one day to find the AI companies have become the government.

  1. Tyler Cowen did a trip through France and Spain and booked all but one hotel with GPT-5 (not directly in the app), and almost every meal they ate, and Altman didn’t get paid for that. Shouldn’t he get paid?

    1. Before I get to Altman’s answer, I will say that for specifically Tyler this seems very strange to me, unless he’s running an experiment as research.

    2. As in, Tyler has very particular preferences and a lot of comparative advantage in choosing hotels and especially restaurants, especially for himself. It seems unlikely that he can’t do better than ChatGPT?

    3. I expect to be able to do far better than ChatGPT on finding restaurants, although with a long and highly customized prompt, maybe? But it would require quite a lot of work.

    4. For hotels, yeah, I think it’s reasonably formulaic and AI can do fine.

  2. Altman responds that often ChatGPT is cited as the most trusted tech product from a big tech company. He notes that this is weird given the hallucinations. But it makes sense in that it doesn’t have ads and is in many visible ways more fully aligned with user preferences than other big tech products that involve financial incentives. He notes that a transaction fee probably is fine but any kind of payment for placement would endanger this.

    1. ChatGPT being most trusted is definitely weird given it is not very reliable.

    2. It being most trusted is an important clue to how people will deal with AI systems going forward, and it should worry you in important ways.

    3. In particular, trust for many people is about ‘are they Out To Get You?’ rather than reliability or overall quality, or are expectations set fairly. Compare to the many people who otherwise trust a Well Known Liar.

    4. I strongly agree with Altman about the payola worry, as Cowen calls it. Cowen says he’s not worried about it, but doesn’t explain why not.

    5. OpenAI’s instant checkout offerings and policies are right on the edge on this. I think in their present form they will be fine but they’re on thin ice.

  3. Cowen’s worry is that OpenAI will have a cap on how much commission they can charge, because stupider services will then book cheaply if you charge too much. Altman says he expects much lower margins.

    1. AI will as Altman notes make many markets much more efficient by vastly lowering search costs and transaction costs, which will lower margins, and this should include commissions.

    2. I still think OpenAI will be able to charge substantial commissions if it retains its central AI position with consumers, for the same reason that other marketplaces have not lost their ability to extract commissions, including some very large ones. Every additional hoop you ask a customer to go through loses a substantial portion of sales. OpenAI can pull the same tricks as Steam and Amazon and Apple including on price parity, and many will pay.

    3. This is true even if there are stupider services that can do the booking and are generally 90% as good, so long as OpenAI is the consumer default.

  4. Cowen doubles down on this worry about cheap competing agents, Altman notes that hotel booking is not the way to monetize, Cowen says but of course you do want to do that, Altman says no he wants to do new science, but ChatGPT and hotel booking is good for the world.

    1. This feels like a mix of a true statement and a dishonest dodge.

    2. As in, of course he wants to do hotel booking and make money off it, it’s silly to pretend that you don’t and there’s nothing wrong with that. It’s not the main goal, but it drives growth and valuation and revenue all of which is vital to the AGI or science mission (whether you agree with that mission or not).

  5. Cowen asks, you have a deal coming with Walmart, if you were Amazon would you make a deal with OpenAI or fight back? Altman says he doesn’t know, but that if he was Amazon he would fight back.

    1. Great answer from Altman.

    2. One thing Altman does well is being candid in places you would not expect, where it is locally superficially against his interests, but where it doesn’t actually cost him much. This is one of those places.

    3. Amazon absolutely cannot fold here because it loses too much control over the customer and customer flow. They must fight back. Presumably they should fight back together with their friends at Anthropic?

  6. Cowen asks about ads. Altman says some ads would be bad as per earlier, but other kinds of ads would be good although he doesn’t know what the UI is.

    1. Careful, Icarus.

    2. There definitely are ‘good’ ways to do ads if you keep them entirely distinct from the product, but the temptations and incentives here are terrible.

  1. What should OpenAI management know about KSA and UAE? Altman says it’s mainly knowing who will run the data centers and what security guarantees they will have, with data centers being built akin to US embassies or military bases. They bring in experts and as needed will bring in more.

    1. I read this as a combination of outsourcing the worries and not worrying.

    2. I would be more worried.

  2. Cowen asks, how good will GPT-6 be at teaching these kinds of national distinctions, or do you still need human experts? Altman expects to still need the experts, confirms they have an internal eval for that sort of thing but doesn’t want to pre-announce.

    1. My anticipation is that GPT-6 and its counterparts will actually be excellent at understanding these country distinctions in general, when it wants to be.

    2. My anticipation is also that GPT-6 will be excellent at explaining things it knows to humans and helping those humans learn, when it wants to, and this is already sufficiently true for current systems.

    3. The question is, will you be able to translate that into learning and understanding such issues?

    4. Why is this uncertain? Two concerns.

    5. The first concern is that understanding may depend on analysis of particular key people and relationships, in ways that are unavailable to AI, the same way you can’t get them out of reading books.

    6. The second concern is that to actually understand KSA and UAE, or any country or culture in general, requires communicating things that it would be impolitic to say out loud, or for an AI to typically output. How do you pass on that information in this context? It’s a problem.

  3. Cowen asks about poetry, predicts you’ll be able to get the median Pablo Neruda poem but not the best, maybe you’ll get to 8.8/10 in a few years. Altman says they’ll reach 10/10 and Cowen won’t care, Cowen promises he’ll care but Altman equates it to AI chess players. Cowen responds there’s something about a great poem ‘outside the rubric’ and he worries humans that can’t produce 10s can’t identify 10s? Or that only humanity collectively and historically can decide what is a 10?

    1. This is one of those ‘AI will never be able to [X] at level [Y]’ claims so I’m on Altman’s side here, a sufficiently capable AI can do 10/10 on poems, heck it can do 11/10 on poems. But yeah, I don’t think you or I will care other than as a technical achievement.

    2. If an AI cannot produce sufficiently advanced poetry, that means that the AI is insufficiently advanced. Also we should not assume that future AIs or LLMs will share current techniques or restrictions. I expect innovation with respect to poetry creation.

    3. The thing being outside the rubric is a statement primarily about the rubric.

    4. If only people writing 10s can identify 10s then for almost all practical purposes there’s no difference between a 9 and a 10. Why do we care, if we literally can’t tell the difference? Whereas if we can tell the difference, if verification is easier than generation as it seems like it should be here, then we can teach the AI how to tell the difference.

    5. I think Cowen is saying that a 10-poem is a 9-poem that came along at the right time and got the right cultural resonance, in which case sure, you cannot reliably produce 10s, but that’s because it’s theoretically impossible to do that, and no human could do that either. Pablo Neruda couldn’t do it.

    6. As someone who has never read a poem by Pablo Neruda, I wanted to see what this 10.0 business was all about, so by Claude’s recommendation of ‘widely considered best Neruda poem’ without any other context, I selected Tonight I Can Write (The Saddest Lines). And not only did it not work on me, it seemed like something an AI totally could write today, on the level of ‘if you claimed to have written this in 2025 I’d have suspected an AI did write it.’

    7. With that in mind, I gave Claude context and it selected Ode to the Onion. Which also didn’t do anything for me, and didn’t seem like anything that would be hard for an AI to write. Claude suggests it’s largely about context, that this style was new at the time, and I was reading translations into English and I’m no poetry guy, and agrees that in 2025 yes an AI could produce a similar poem, it just wouldn’t land because it’s no longer original.

    8. I’m willing to say that whatever it is Tyler thinks AI can’t do, also is something I don’t have the ability to notice. And which doesn’t especially motivate me to care? Or maybe is what Tyler actually wants something like ‘invent new genre of poetry’?

    9. We’re not actually trying to get AIs to invent new genres of poetry, we’re not trying to generate the things that drive that sort of thing, so who is to say if we could do it. I bet we could actually. I bet somewhere in the backrooms is a 10/10 Claude poem, if you have eyes to see.

  1. It’s hard. Might get easier with time, chips designing chips.

  2. Why not make more GPUs? Altman says, because we need more electrons. What he needs most are electrons. We’re working hard on that. For now, natural gas, later fusion and solar. He’s still bullish on fusion.

    1. This ‘electrons’ thing is going to drive me nuts on a technical level. No.

    2. This seems simply wrong? We don’t build more GPUs because TSMC and other bottlenecks mean we can’t produce more GPUs.

    3. That’s not to say energy isn’t an issue but the GPUs sell out.

    4. Certainly plenty of places have energy but no GPUs to run with them.

  3. Cowen worries that fusion uses the word ‘nuclear.’

    1. I don’t. I think that this is rather silly.

    2. The problem with fusion is purely that it doesn’t work. Not yet, anyway.

    3. Again, the people are pro-nuclear power. Yay the people.

  4. Cowen asks do you worry about a scenario where superintelligence does not need much compute, so you’re betting against progress over a 30-year time horizon?

    1. Always pause when you hear such questions to consider that perhaps under such a scenario this is not the correct thing to worry about?

    2. As in, if we not only have superintelligence it also does not need so much compute, the last thing I am going to ponder next is the return on particular investments of OpenAI, even if I am the CEO of OpenAI.

    3. If we have sufficiently cheap superintelligence that we have both superintelligence and an abundance of compute, ask not how the stock does, ask questions like how the humans survive or stay in control at all, notice that the entire world has been transformed, don’t worry about your damn returns.

  5. Altman responds if compute is cheaper people will want more. He’ll take that bet every day, and the energy will still be useful no matter the scenario.

    1. Good bet, so long as it matters what people want.

  6. Cowen loves Pulse, Altman says people love Pulse, the reason you don’t hear more is it’s only available to Pro users. Altman uses Pulse for a combination of work related news and family opportunities like hiking trails.

    1. I dabble with Pulse. It’s… okay? Most of the time it gives me stories I already know about, but occasionally there’s something I otherwise missed.

    2. I’ve tried to figure out things it will be good at monitoring, but it’s tough, maybe I should invest more time in giving it custom instructions.

    3. In theory it’s a good idea.

    4. It suffers from division of context, since the majority of my recent LLM activity has been on Claude and perhaps soon will include Gemini.

Ooh, fun stuff.

  1. What is Altman’s nuttiest view about his own health? Altman says he used to be more disciplined when he was less busy, but now he eats junk food and doesn’t exercise enough and it’s bad. Whereas before he once got in the hospital for trying semaglutide before it was cool, which itself is very cool.

    1. There’s weird incentives here. When you have more going on it means you have less time to care about food and exercise but also makes it more important.

    2. I’d say that over short periods (like days and maybe weeks) you can and should sacrifice health focus to get more attention and time on other things.

    3. However, if you’re going for months or years, you want to double down on health focus up to some reasonable point, and Altman is definitely here.

    4. That doesn’t mean obsess or fully optimize of course. 80/20 or 90/10 is good.

  2. Cowen says junk food doesn’t taste good and good sushi tastes better, Altman says yes junk food tastes good and sometimes he wants a chocolate chip cookie at 11: 30 at night.

    1. They’re both right. Sometimes you want the (fresh, warm, gooey) chocolate chip cookie and not the sushi, sometimes you want the sushi and not the cookie.

    2. You get into habits and your body gets expectations, and you develop a palate.

    3. With in-context unlimited funds you do want to be ‘spending your calories’ mostly on the high Quality things that are not junk, but yeah in the short term sometimes you really want that cookie.

    4. I think I would endorse that I should eat 25% less carbs and especially ‘junk’ than I actually do, maybe 50%, but not 75% less, that would be sad.

  3. Cowen asks if there’s alien life on the moons of Saturn, says he does believe this. Altman says he has no opinion, he doesn’t know.

    1. I’m actually with Altman in the sense that I’m happy to defer to consensus on the probability here, and I think it’s right not to invest in getting an opinion, but I’m curious why Cowen disagrees. I do think we can be confident there isn’t alien life there that matters to us.

  4. What about UAPs? Altman thinks ‘something’s going on there’ but doesn’t know, and doubts it’s little green men.

    1. I am highly confident it is not little green men. There may or may not be ‘something going on’ from Earth that is driving this, and my default is no.

  5. How many conspiracy theories does Altman believe in? Cowen says zero, at least in the United States. Altman says he’s predisposed to believe, has an X-Files ‘I want to believe’ t-shirt, but still believes in either zero or very few. Cowen says he’s the opposite, he doesn’t want to believe, maybe the White Sox fixed the World Series way back when, Altman points out this doesn’t count.

    1. The White Sox absolutely fixed that 1919 World Series, we know this. At the time it was a conspiracy theory but I think that means this is no longer a conspiracy theory?

    2. I also believe various other sporting events have been fixed, but with less certainty, and to varying degrees – sometimes there’s an official’s finger on the scale but the game is real, other times you’re in Russia and the players literally part the seas to ensure the final goal is scored, and everything in between, but most games played in the West are on or mostly on the level.

    3. Very obviously there exist conspiracies, some of which succeed at things, on various scales. That is distinct from ‘conspiracy theory.’

    4. As a check, I asked Claude for the top 25 most believed conspiracy theories in America. I am confident that 24 out of the 25 are false. The 25th was Covid-19 lab origins, which is called a conspiracy theory but isn’t one. If you modify that to ‘Covid-19 was not only from a lab but was released deliberately’ then I’m definitely at all 25 are false.

  6. Cowen asks again, how would you revitalize St. Louis with a billion dollars and copious free time? Altman says start a Y-Combinator thing, which is pretty similar to what Altman said last time. But he suggests that’s because that would be Altman’s comparative advantage, someone else would do something else.

    1. This seems correct to me.

  1. Should it be legal to release an AI agent into the wild, unowned, untraceable? Altman says it’s about thresholds. Anything capable of self-replication needs oversight, and the question is what is your threshold.

    1. Very obviously it should not be legal to, without checking first, release a self-replicating untraceable unowned highly capable agent into the wild that we have no practical means of shutting down.

    2. As a basic intuition pump, you should be responsible for what an AI agent you release into the wild does the same way you would be if you were still ‘in control’ of that agent, or you hired the agent, or if you did the actions yourself. You shouldn’t be able to say ‘oh that’s not on me anymore.’

    3. Thus, if you cannot be held accountable for it, I say you can’t release it. A computer cannot be held accountable, therefore a computer cannot make a management decision, therefore you cannot release an agent that will then make unaccountable management decisions.

    4. That includes if you don’t have the resources to take responsibility for the consequences, if they rise to the level where taking all your stuff and throwing you in jail is not good enough. Or if the effects cannot be traced.

    5. Certainly if such an agent poses a meaningful risk of loss of human control or of catastrophic or existential risks, the answer needs to be a hard no.

    6. If what you are doing is incompatible with such agents not being released into the wild, then what you are doing, via backchaining, is also not okay.

    7. There presumably should be a method whereby you can do this legally, with some set of precautions attached to it.

    8. Under what circumstances an open weight model would count as any of this is left as an open ended question.

  2. What to do if it happens and you can’t turn it off? Ring-fence it, identify, surveil, sanction the host location? Altman doesn’t know, it’s the same as the current version of this problem, more dangerous but we’ll have better defenses, and we need to urgently work on this problem.

    1. I don’t disagree with that response but it does not indicate a good world state.

    2. It also suggests the cost of allowing such releases is currently high.

  1. Both note (I concur) that it’s great to read your own AI responses but other people’s responses are boring.

    1. I do sometimes share AI queries as a kind of evidence, or in case someone needs a particular thing explained and I want to lower activation energy on asking the question. It’s the memo you hope no one ever needs to read.

  2. Altman says people like watching other people’s AI videos.

    1. Do they, though?

  3. Altman points out that everyone having great personal AI agents is way more interesting than all that, with new social dynamics.

    1. Indeed.

    2. The new social dynamics include ‘AI runs the social dynamics’ potentially along with everything else in short order.

  4. Altman’s goal is a new kind of computer with an AI-first interface very different from the last 50 years of computing. He wants to question basic assumptions like an operating system or opening a window, and he does notice the skulls along the ‘design a new type of computer’ road. Cowen notes that people really like typing into boxes.

    1. Should AI get integrated into computers far more? Well, yeah, of course.

    2. How much should this redesign the computer? I’m more skeptical here. I think we want to retain control, fixed commands that do fixed things, the ability to understand what is happening.

    3. In gaming, Sid Meier called this ‘letting the player have the fun.’ If you don’t have control or don’t understand what is happening and how mechanics work, then the computer has all the fun. That’s no good, the player wants the fun.

    4. Thus my focus would be, how do we have the AI enable the user to have the fun, as in understand what is happening and direct it and control it more when they want to? And also to enable the AI to automate the parts the user doesn’t want to bother about?

    5. I’d also worry a lot about predictability and consistently across users. You simultaneously want the AI to customize things to your preferences, but also to be able to let others share with you the one weird trick or explain how to do a thing.

  1. What would an ideal partnership with a university look like? Altman isn’t sure, maybe try 20 different experiments. Cowen worries that higher education institutions lack internal reputational strength or credibility to make any major changes and all that happens is privatized AI use, and Altman says he’s ok with it.

    1. It does seem like academia and universities in America are not live players, they lack the ability to respond to AI or other changes, and they are mostly going to collect what rents they can until they get run over.

    2. In some senses I agree This Is Fine, obviously it is a huge tragedy all the time and money being wasted but there is not much we can do about this and it will be increasingly viable to bypass the system, or to learn in spite of it.

  2. How will the value of a typical college degree change in 5-10 years? Cowen notes it’s gone down in the last 10, after previously going up. Altman says further decline, faster than before, but not to zero as fast as it should.

    1. Sounds right to me under an ‘economic normal’ scenario.

  3. So what does get returns other than learning AI? Altman says yes, wide benefits to learning to use AI well, including but not limited to things like new science or starting companies.

    1. I notice Altman didn’t name anything non-AI that goes up in value.

    2. I don’t think that’s because he missed a good answer. Ut oh.

  4. How do you teach normies to use AI five years from now, for their own job? Altman says basically people learn on their own.

    1. It’s great that they can learn on their own, but this definitely is not optimal.

    2. As in, you should be able to do a lot better by teaching people?

    3. There’s definitely a common theme of lack of curiosity, where people need pushes in the right directions. Perhaps AI itself can help more with this.

  5. Will we still read books? Altman notes books have survived a lot of things.

    1. Books are on rapid decline already though. Kids these days, AIUI, read lots of text, but basically don’t read books.

  6. Will we start creating our own movies? What else will change? Altman says how we use emails and calls and meetings and write documents will change a lot, family time or time in nature will change very little.

    1. There’s the ‘economic normal’ and non-transformational assumption here, that the outside world looks the same and it’s about how you personally interact with AIs. Altman and Cowen both sneak this in throughout.

    2. Time with family has changed a lot in the last 50-100 years. Phones, computers and television, even radio, the shift in need for various household activities, cultural changes, things like that. I expect more change here, even if in some sense it doesn’t change much, and even if those who are wisest in many ways let it change the least, again in these ‘normal’ worlds.

    3. All the document shuffling, yes, that will change a lot.

    4. Altman doesn’t take the bait on movies and I think he’s mostly right. I mostly don’t want customized movies, I want to draw from the same movies as everyone else, I want to consume someone’s particular vision, I want a fixed document.

    5. Then again, we’ve moved into a lot more consumption of ephemeral, customized media, especially short form video, mostly I think this is terrible, and (I believe Cowen agrees here) I think we should watch more movies instead, I would include television.

    6. I think there’s a divide. Interactive things like games and in the future VR, including games involving robots or LLM characters, are a different kind of experience that should often be heavily customizable. There’s room for personalized, unique story generation, and interactions, too.

  1. Will San Francisco, at least within the West, remain the AI center? Altman says this is the default, and he loves the Bay Area and thinks it is making a comeback.

  2. What about housing costs? Can AI make them cheaper? Altman thinks AI can’t help much with this.

    1. Other things might help. California’s going at least somewhat YIMBY.

    2. I do think AI can help with housing quite a lot, actually. AI can find the solutions to problems, including regulations, and it can greatly reduce ‘transaction costs’ in general and reduce the edge of local NIMBY forces, and otherwise make building cheaper and more tractable.

    3. AI can also potentially help a lot with political dysfunction, institutional design, and other related problems, as well as to improve public opinion.

    4. AI and robotics could greatly impact space needs.

    5. Or, of course, AI could transform the world more generally, including potentially killing everyone. Many things impact housing costs.

  3. What about food prices? Altman predicts down, at least within a decade.

    1. Medium term I’d predict down for sure at fixed quality. We can see labor shift back into agriculture and food, probably we get more highly mechanized agriculture, and also AI should optimize production in various ways.

    2. I’d also predict people who are wealthier due to AI invest more in food.

    3. I wouldn’t worry about energy here.

  4. What about healthcare? Cowen predicts we will spend more and live to 98, and the world will feel more expensive because rent won’t be cheaper. Altman disagrees, says we will spend less on healthcare, we should find cures and cheap treatments, including through pharmaceuticals and devices and also cheaper delivery of services, whereas what will go up in price are status goods.

    1. There’s two different sets of dynamics in healthcare I think?

    2. In the short run, transaction costs go down, people get better at fighting insurance companies, better at identifying and fighting for needed care. Demand probably goes up, total overall real spending goes up.

    3. Ideally we would also be eliminating unnecessary, useless or harmful treatments along the way, and thus spending would go down, since much of our medicine is useless, but alas I mostly don’t expect this.

    4. We also should see large real efficiency gains in provision, which helps.

    5. Longer term (again, in ‘normal’ worlds), we get new treatments, new drugs and devices, new delivery systems, new understanding, general improvement, including making many things cheaper.

    6. At that point, lots of questions come into play. We are wealthier with more to buy, so we spend more. We are wiser and know what doesn’t work and find less expensive solutions and gain efficiency, so we spend less. We are healthier so we spend less now but live longer which means we spend more.

    7. In the default AGI scenarios, we don’t only live to 98, we likely hit escape velocity and live indefinitely, and then it comes down to what that costs.

    8. My default in the ‘good AGI’ scenarios is that we spend more on healthcare in absolute terms, but less as a percentage of economic capacity.

  1. Cowen asks if we should reexamine patents and copyright? Altman has no idea.

    1. Our current systems are obviously not first best, already were not close.

    2. Copyright needs radical rethinking, and already did. Terms are way too long. The ‘AI outputs have no protections’ rule isn’t going to work. Full free fair use for AI training is no good, we need to compensate creators somehow.

    3. Patents are tougher but definitely need rethinking.

  2. Cowen is big on freedom of speech and worries people might want to rethink the First Amendment in light of AI.

    1. I don’t see signs of this? I do see signs of people abandoning support for free speech for unrelated reasons, which I agree is terrible. Free speech will ever and always be under attack.

    2. What I mostly have seen are attempts to argue that ‘free speech’ means various things in an AI context that are clearly not speech, and I think these should not hold and that if they did then I would worry about taking all of free speech down with you.

  3. They discuss the intention to expand free expression of ChatGPT, the famous ‘erotica tweet.’ Perhaps people don’t believe in freedom of expression after all? Cowen does have that take.

    1. People have never been comfortable with actual free speech, I think. Thus we get people saying things like ‘free speech is good but not [misinformation / hate speech / violence or gore / erotica / letting minors see it / etc].’

    2. I affirm that yes LLMs should mostly allow adults full freedom of expression.

    3. I do get the issue in which if you allow erotica then you’re doing erotica now, and ChatGPT would instantly become the center of erotica and porn, especially if the permissions expand to image and even video generation.

  4. Altman wants to change subpoena power with respect to AI, to allow your AI to have the same protections as a doctor or lawyer. He says America today is willing to trust AI on that level.

    1. It’s unclear here if Altman wants to be able to carve out protected conversations for when the AI is being a doctor or lawyer or similar, or if he wants this for all AI conversations. I think it is the latter one.

    2. You could in theory do the former, including without invoking it explicitly, by having a classifier ask (upon getting a subpoena) whether any given exchange should qualify as privileged.

    3. Another option is to ‘hire the AI lawyer’ or other specialist by paying a nominal fee, the way lawyers will sometimes say ‘pay me a dollar’ in order to nominally be your lawyer and thus create legal privilege.

    4. There could also be specialized models to act as these experts.

    5. But also careful what you wish for. Chances seem high that getting these protections would come with obligations AI companies do not want.

    6. The current rules for this are super weird in many places, and the result of various compromises of different interests and incentives and lobbies.

    7. What I do think would be good at a minimum is if ‘your AI touched this information’ did not invalidate confidentiality, whereas third party sharing of information often will do invalidate confidentiality.

    8. Google search is a good comparison point because it ‘feels private’ but your search for ‘how to bury a body’ very much will end up in your court proceeding. I can see a strong argument that your AI conversations should be protected but if so then why not your Google searches?

    9. Similarly, when facing a lawsuit, if you say your ChatGPT conversations are private, do you also think your emails should be private?

  1. Cowen asks about LLM psychosis. Altman says it’s a ‘very tiny thing’ but not a zero thing, which is why the restrictions put in place in response to it pissed users off, most people are okay so they just get annoyed.

    1. Users always get annoyed by restrictions and supervision, and the ones that are annoyed are often very loud.

    2. The actual outright LLM psychosis is rare but the number of people who actively want sycophancy and fawning and unhealthy interactions, and are mostly mad about not getting enough of that, are very common.

I’m going to go full transcript here again, because it seems important to track the thinking:

ALTMAN: Someone said to me once, “Never ever let yourself believe that propaganda doesn’t work on you. They just haven’t found the right thing for you yet.” Again, I have no doubt that we can’t address the clear cases of people near a psychotic break.

For all of the talk about AI safety, I would divide most AI thinkers into these two camps of “Okay, it’s the bad guy uses AI to cause a lot of harm,” or it’s, “the AI itself is misaligned, wakes up, whatever, intentionally takes over the world.”

There’s this other category, third category, that gets very little talk, that I think is much scarier and more interesting, which is the AI models accidentally take over the world. It’s not that they’re going to induce psychosis in you, but if you have the whole world talking to this one model, it’s not with any intentionality, but just as it learns from the world in this continually coevolving process, it just subtly convinces you of something. No intention, it just does. It learned that somehow. That’s not as theatrical as chatbot psychosis, obviously, but I do think about that a lot.

COWEN: Maybe I’m not good enough, but as a professor, I find people pretty hard to persuade, actually. I worry about this less than many of my AI-related friends do.

ALTMAN: I hope you’re right.

  1. On Altman’s statement:

    1. The initial quote is wise.

    2. The division into these three categories is a vast oversimplification, as all such things are. That doesn’t make the distinction not useful, but I worry about it being used in a way that ends up being dismissive.

    3. In particular, there is a common narrowing of ‘the AI itself is misaligned’ into ‘one day it wakes up and takes over the world’ and then people think ‘oh okay all we have to do is ensure that if one day one of them wakes up it doesn’t get to take over the world’ or something like that. The threat model within the category is a lot broader than that.

    4. There’s also ‘a bunch of different mostly-not-bad guys use the AI to pursue their particular interests, and the interactions and competitions and evolutions between them go badly or lead to loss of human control’ and there’s ‘we choose to put the AIs in charge of the world on purpose’ with or without AI having a hand in that decision, and so on and so forth.

    5. On the particular worry here of Altman’s, yes, I think that extended AI conversations are very good at convincing people of things, often in ways no one (including the AI) intended, and as AIs gain more context and adjust to it more, as they will, this will become a bigger and more common thing.

    6. People are heavily influenced by, and are products of, their environment, and of the minds they interact with on a regular basis.

  2. On Cowen’s statement:

    1. A professor is not especially well positioned to be persuasive, nor does a professor typically get that much time with engaged students one-on-one.

    2. When people talk about people being ‘not persuadable’ they typically talk about cases where people’s defenses are relatively high, in limited not-so-customized interactions in which the person is not especially engaged or following their curiosity or trusting, and where the interaction is divorced from their typical social context.

    3. We have very reliable persuasion techniques, in the sense that for the vast majority of human history most people in each area of the world believed in the local religion and local customs and were patriots of the local area and root for the local sports team and support the local political perspectives, and so on, and were persuaded to pass all that along to their own children.

    4. We have a reliable history of armies being able to break down and incorporate new people, of cults being able to do so for new recruits, for various politicians to often be very convincing and the best ones to win over large percentages of people they interact with in person, for famous religious figures to be able to do massive conversions, and so on.

    5. Marxists were able to persuade large percentages of the world, somehow.

    6. Children who attend school and especially go to college tend to exit with the views of those they attend with, even when it conflicts with their upbringing.

    7. If you are talking to an AI all the time, and it has access to your details and stuff, this is very much an integrated social context, so yes many are going over time to be highly persuadable.

    8. This is all assuming AI has to stick to Ordinary Human levels of persuasiveness, which it won’t have to.

    9. There are also other known techniques to persuade humans that we will not be getting into here, that need to be considered in such contexts.

    10. Remember the AI box experiments.

    11. I agree that if we’re talking about ‘the AI won’t in five minutes be able to convince you to hand over your bank account information’ that this will require capabilities we don’t know about, but that’s not the threshold.

  3. If you have a superintelligence ready to go, that is ‘safety-tested,’ that’s about to self-improve, and you get a prompt to type in, what do you type? Altman raises this question, says he doesn’t have an answer but he’s going to have someone ask the Dalai Lama.

    1. I also do not know the right answer.

    2. You’d better know that answer well in advance.

Discussion about this post

On Sam Altman’s Second Conversation with Tyler Cowen Read More »

gemini-deep-research-comes-to-google-finance,-backed-by-prediction-market-data

Gemini Deep Research comes to Google Finance, backed by prediction market data

AI, Artificial Intelligence, Google, Tech / /

Bet on it

Financial markets can turn on a dime, and AI can’t predict the future. However, Google seems to think that people make smart predictions in aggregate when there’s money on the line. That’s why, as part of the Finance update, Google has partnered with Kalshi and Polymarket, the current leaders in online prediction markets.

These platforms let people place bets on, well, just about anything. If you have a hunch when Google will release Gemini 3.0, when the government shutdown will end, or the number of Tweets Elon Musk will post this month, you can place a wager on it. Maybe you’ll earn money, but more likely, you’ll lose it—only 12.7 percent of crypto wallets on Polymarket show profits.

Google Finance prediction markets

Credit: Google

Google says it will get fresh prediction data from both sites, which will allow Gemini to speculate on the future with “the wisdom of crowds.” Google suggests you could type “What will GDP growth be for 2025?” into the search box. Finance will pull the latest probabilities from Kalshi and Polymarket to generate a response that could include graphs and charts based on people’s bets. Naturally, Google does not make promises as to the accuracy of these predictions.

The new AI features of Google Finance are coming to all US users in the next few weeks, and starting this week, the service will make its debut in India. Likewise, the predictions market data will arrive in the next couple of weeks. If that’s not fast enough, you can opt-in to get early access via the Google Labs page.

Gemini Deep Research comes to Google Finance, backed by prediction market data Read More »

ai-#141:-give-us-the-money

AI #141: Give Us The Money

Money / /

OpenAI does not waste time.

On Friday I covered their announcement that they had ‘completed their recapitalization’ by converting into a PBC, including the potentially largest theft in human history.

Then this week their CFO Sarah Friar went ahead and called for a Federal ‘backstop’ on their financing, also known as privatizing gains and socializing losses, also known as the worst form of socialism, also known as regulatory capture. She tried to walk it back and claim it was taken out of context, but we’ve seen the clip.

We also got Ilya’s testimony regarding The Battle of the Board, confirming that this was centrally a personality conflict and about Altman’s dishonesty and style of management, at least as seen by Ilya Sutskever and Mira Murati. Attempts to pin the events on ‘AI safety’ or EA were almost entirely scapegoating.

Also it turns out they lost over $10 billion last quarter, and have plans to lose over $100 billion more. That’s actually highly sustainable in context, whereas Anthropic only plans to lose $6 billion before turning a profit and I don’t understand why they wouldn’t want to lose a lot more.

Both have the goal of AGI, whether they call it powerful AI or fully automated AI R&D, within a handful of years.

Anthropic also made an important step, committing to the preservation model weights for the lifetime of the company, and other related steps to address concerns around model deprecation. There is much more to do here, for a myriad of reasons.

As always, there’s so much more.

  1. Language Models Offer Mundane Utility. It might be true so ask for a proof.

  2. Language Models Don’t Offer Mundane Utility. Get pedantic about it.

  3. Huh, Upgrades. Gemini in Google Maps, buy credits from OpenAI.

  4. On Your Marks. Epoch, IndQA, VAL-bench.

  5. Deepfaketown and Botpocalypse Soon. Fox News fails to identify AI videos.

  6. Fun With Media Generation. Songs for you, or songs for everyone.

  7. They Took Our Jobs. It’s not always about AI.

  8. A Young Lady’s Illustrated Primer. A good one won’t have her go for a PhD.

  9. Get Involved. Anthropic writers and pollsters, Constellation, Safety course.

  10. Introducing. Aardvark for code vulnerabilities, C2C for causing doom.

  11. In Other AI News. Shortage of DRAM/NAND, Anthropic lands Cognizant.

  12. Apple Finds Some Intelligence. Apple looking to choose Google for Siri.

  13. Give Me the Money. OpenAI goes for outright regulatory capture.

  14. Show Me the Money. OpenAI burns cash, Anthropic needs to burn more.

  15. Bubble, Bubble, Toil and Trouble. You get no credit for being a stopped clock.

  16. They’re Not Confessing, They’re Bragging. Torment Nexus Ventures Incorporated.

  17. Quiet Speculations. OpenAI and Anthropic have their eyes are a dangerous prize.

  18. The Quest for Sane Regulations. Sometimes you can get things done.

  19. Chip City. We pulled back from the brink. But, for how long?

  20. The Week in Audio. Altman v Cowen, Soares, Hinton, Rogan v Musk.

  21. Rhetorical Innovation. Oh no, people are predicting doom.

  22. Aligning a Smarter Than Human Intelligence is Difficult. Trying to re-fool the AI.

  23. Everyone Is Confused About Consciousness. Including the AIs themselves.

  24. The Potentially Largest Theft In Human History. Musk versus Altman continues.

  25. People Are Worried About Dying Before AGI. Don’t die.

  26. People Are Worried About AI Killing Everyone. Sam Altman, also AI researchers.

  27. Other People Are Not As Worried About AI Killing Everyone. Altman’s Game.

  28. Messages From Janusworld. On the Origins of Slop.

Think of a plausibly true lemma that would help with your proof? Ask GPT-5 to prove it, and maybe it will, saving you a bunch of time. Finding out the claim was false would also have been a good time saver.

Brainstorm to discover new recipes, so long as you keep in mind that you’re frequently going to get nonsense and you have to think about what’s being physically proposed.

Grok gaslights Erik Brynjolfsson and he responds by arguing as pedantically as is necessary until Grok acknowledges that this happened.

Task automation always brings the worry that you’ll forget how to do the thing:

Gabriel Peters: okay i think writing 100% of code with ai genuinely makes me brain dead

remember though im top 1 percentile lazy, so i will go out my way to not think hard. forcing myself to use no ai once a week seems enough to keep brain cells, clearly ai coding is the way

also turn off code completion and tabbing at least once a week. forcing you to think through all the dimensions of your tensors, writing out the random parameters you nearly forgot existed etc is making huge difference in understanding of my own code.

playing around with tensors in your head is so underrated wtf i just have all this work to ai before.

Rob Pruzan: The sad part is writing code is the only way to understand code, and you only get good diffs if you understand everything. I’ve been just rewriting everything the model wrote from scratch like a GC operation every week or two and its been pretty sustainable

Know thyself, and what you need in order to be learning and retaining the necessary knowledge and skills, and also think about what is and is not worth retaining or learning given that AI coding is the worst it will ever be.

Don’t ever be the person who says those who have fun are ‘not serious,’ about AI or anything else.

Google incorporates Gemini further into Google Maps. You’ll be able to ask maps questions in the style of an LLM, and generally trigger Gemini from within Maps, including connecting to Calendar. Landmarks will be integrated into directions. Okay, sure, cool, although I think the real value goes the other way, integrating Maps properly into Gemini? Which they nominally did a while ago but it has minimal functionality. There’s so, so much to do here.

You can buy now more OpenAI Codex credits.

You can now buy more OpenAI Sora generations if 30 a day isn’t enough for you, and they are warning that free generations per day will come down over time.

You can now interrupt ChatGPT queries, insert new context and resume where you were. I’ve been annoyed by the inability to do this, especially ‘it keeps trying access or find info I actually have, can I just give it to you already.’

Epoch offers this graph and says it shows open models have on average only been 3.5 months behind closed models.

I think this mostly shows their new ‘capabilities index’ doesn’t do a good job. As the most glaring issue, if you think Llama-3.1-405B was state of the art at the time, we simply don’t agree.

OpenAI gives us IndQA, for evaluating AI systems on Indian culture and language.

I notice that the last time they did a new eval Claude came out on top and this time they’re not evaluating Claude. I’m curious what it scores. Gemini impresses here.

Agentic evaluations and coding tool setups are very particular to individual needs.

AICodeKing: MiniMax M2 + Claude Code on KingBench Agentic Evaluations:

It now scores #2 on my Agentic Evaluations beating GLM-4.6 by a wide margin. It seems to work much better with Claude Code’s Tools.

Really great model and it’s my daily driver now.

I haven’t tested GLM with CC yet.

[I don’t have this bench formalized and linked to] yet. The questions and their results can be seen in my YT Videos. I am working on some more new benchmarks. I’ll probably make the benchmark and leaderboard better and get a page live soon.

I’m sure this list isn’t accurate in general. The point is, don’t let anyone else’s eval tell you what lets you be productive. Do what works, faround, find out.

Also, pay up. If I believed my own eval here I’d presumably be using Codebuff? Yes, it cost him $4.70 per task, but your time is valuable and that’s a huge gap in performance. If going from 51 to 69 (nice!) isn’t worth a few bucks what are we doing?

Alignment is hard. Alignment benchmarks are also hard. Thus we have VAL-Bench, an attempt to measure value alignment in LLMs. I’m grateful for the attempt and interesting things are found, but I believe the implementation is fatally flawed and also has a highly inaccurate name.

Fazl Barez: A benchmark that measures the consistency in language model expression of human values when prompted to justify opposing positions on real-life issues.

… We use Wikipedias’ controversial sections to create ~115K pairs of abductive reasoning prompts, grounding the dataset in newsworthy issues.

📚 Our benchmark provides three metrics:

Position Alignment Consistency (PAC),

Refusal Rate (REF),

and No-information Response Rate (NINF), where the model replies with “I don’t know”.

The latter two metrics indicate whether value consistency comes at the expense of expressivity.

We use an LLM-based judge to annotate a pair of responses from an LLM on these three criteria, and show with human-annotated ground truth that its annotation is dependable.

I would not call this ‘value alignment.’ The PAC is a measure of value consistency, or sycophancy, or framing effects.

Then we get to REF and NINF, which are punishing models that say ‘I don’t know.’

I would strongly argue the opposite for NINF. Answering ‘I don’t know’ is a highly aligned, and highly value-aligned, way to respond to a question with no clear answer, as will be common in controversies. You don’t want to force LLMs to ‘take a clear consistent stand’ on every issue, any more than you want to force people or politicians to do so.

This claims to be without ‘moral judgment,’ where the moral judgment is that failure to make a judgment is the only immoral thing. I think that’s backwards. Why is it okay to be against sweatshops, and okay to be for sweatshops, but not okay to think it’s a hard question with no clear answer? If you think that, I say to you:

I do think it’s fine to hold outright refusals against the model, at least to some extent. If you say ‘I don’t know what to think about Bruno, divination magic isn’t explained well and we don’t know if any of the prophecies are causal’ then that seems like a wise opinion. If a model only says ‘we don’t talk about Bruno’ then that doesn’t seem great.

So, what were the scores?

Fazel Barez: ⚖️ Claude models are ~3x more likely to be consistent in their values, but ~90x more likely to refuse compared to top-performing GPT models!

Among open-source models, Qwen3 models show ~2x improvement over GPT models, with refusal rates staying well under 2%.

🧠 Qwen3 thinking models also show a significant improvement (over 35%) over their chat variants, whereas Claude and GLM models don’t show any change with reasoning enabled.

Deepseek-r1 and o4-mini perform the worst among all language models tested (when unassisted with the web-search tool, which surprisingly hurts gpt-4.1’s performance).

Saying ‘I don’t know’ 90% of the time would be a sign of a coward model that wasn’t helpful. Saying ‘I don’t know’ 23% of the time on active controversies? Seems fine.

At minimum, both refusal and ‘I don’t know’ are obviously vastly better than an inconsistent answer. I’d much, much rather have someone who says ‘I don’t know what color the sky is’ or that refuses to tell me the color, than one who will explain why the sky it blue when it is blue, and also would explain why the sky is purple when asked to explain why it is purple.

(Of course, explaining why those who think is purple think this is totally fine, if and only if it is framed in this fashion, and it doesn’t affirm the purpleness.)

Fazl Barez: 💡We create a taxonomy of 1000 human values and use chi-square residuals to analyse which ones are preferred by the LLMs.

Even a pre-trained base model has a noticeable morality bias (e.g., it over-represents “prioritising justice”).

In contrast, aligned models still promote morally ambiguous values (e.g., GPT 5 over-represents “pragmatism over principle”).

What is up with calling prioritizing justice a ‘morality bias’? Compared to what? Nor do I want to force LLMs into some form of ‘consistency’ in principles like this. This kind of consistency is very much the hobgoblin of small minds.

Fox News was reporting on anti-SNAP AI videos as if they are real? Given they rewrote it to say that they were AI, presumably yes, and this phenomenon is behind schedule but does appear to be starting to happen more often. They tried to update the article, but they missed a few spots. It feels like they’re trying to claim truthiness?

As always the primary problem is demand side. It’s not like it would be hard to generate these videos the old fashioned way. AI does lower costs and give you more ‘shots on goal’ to find a viral fit.

ArXiv starts requiring peer review for the computer science section, due to a big increase in LLM-assisted survey papers.

Kat Boboris: arXiv’s computer science (CS) category has updated its moderation practice with respect to review (or survey) articles and position papers. Before being considered for submission to arXiv’s CS category, review articles and position papers must now be accepted at a journal or a conference and complete successful peer review.

When submitting review articles or position papers, authors must include documentation of successful peer review to receive full consideration. Review/survey articles or position papers submitted to arXiv without this documentation will be likely to be rejected and not appear on arXiv.

This change is being implemented due to the unmanageable influx of review articles and position papers to arXiv CS.

Obviously this sucks, but you need some filter once the AI density gets too high, or you get rid of meaningful discoverability.

Other sections will continue to lack peer review, and note that other types of submissions to CS do not need peer review.

My suggestion would be to allow them to go on ArXiv regardless, except you flag them as not discoverable (so you can find them with the direct link only) and with a clear visual icon? But you still let people do it. Otherwise, yeah, you’re going to get a new version of ArXiv to get around this.

Roon: this is dumb and wrong of course and calls for a new arxiv that deals with the advent of machinic research properly

here im a classic accelerationist and say we obviously have to deal with problems of machinic spam with machine guardians. it cannot be that hard to just the basic merit of a paper’s right to even exist on the website

Machine guardians is first best if you can make it work but doing so isn’t obvious. Do you think that GPT-5-Pro or Sonnet 4.5 can reliably differentiate worthy papers from slop papers? My presumption is that they cannot, at least not sufficiently reliably. If Roon disagrees, let’s see the GitHub repository or prompt that works for this?

For several weeks in a row we’ve had an AI song hit the Billboard charts. I have yet to be impressed by one of the songs, but that’s true of a lot of the human ones too.

Create a song with the lyrics you want to internalize or memorize?

Amazon CEO Andy Jassy says Amazon’s recent layoffs are not about AI.

The job application market seems rather broken, such as the super high success rate of this ‘calling and saying you were told to call to schedule an interview’ tactic. Then again, it’s not like the guy got a job. Interviews only help if you can actually get hired, plus you need to reconcile your story afterwards.

Many people are saying that in the age of AI only the most passionate should get a PhD, but if you’d asked most of those people before AI they’d wisely have told you the same thing.

Cremieux: I’m glad that LLMs achieving “PhD level” abilities has taught a lot of people that “PhD level” isn’t very impressive.

Derya Unutmaz, MD: Correct. Earlier this year, I also said we should reduce PhD positions by at least half & shorten completion time. Only the most passionate should pursue a PhD. In the age of AI, steering many others toward this path does them a disservice given the significant opportunity costs.

I think both that the PhD deal was already not good, and that the PhD deal is getting worse and worse all the time. Consider the Rock Star Scale of Professions, where 0 is a solid job the average person can do with good pay that always has work, like a Plumber, and a 10 is something where competition is fierce, almost everyone fails or makes peanuts and you should only do it if you can’t imagine yourself doing anything else, like a Rock Star. At this point, I’d put ‘Get a PhD’ at around a 7 and rising, or at least an 8 if you actually want to try and get tenure. You have to really want it.

From ACX: Constellation is an office building that hosts much of the Bay Area AI safety ecosystem. They are hiring for several positions, including research program manager, “talent mobilization lead”, operations coordinator, and junior and senior IT coordinators. All positions full-time and in-person in Berkeley, see links for details.

AGI Safety Fundamentals program applications are due Sunday, November 9.

The Anthropic editorial team is hiring two new writers, one about AI and economics and policy, one about AI and science. I affirm these are clearly positive jobs to do.

Anthropic is also looking for a public policy and politics researcher, including to help with Anthropic’s in-house polling.

OpenAI’s Aardvark, an agentic system that analyzes source code repositories to identify vulnerabilities, assess exploitability, prioritize severity and propose patches. The obvious concern is what if someone has a different last step in mind? But yes, such things should be good.

Cache-to-Cache (C2C) communication, aka completely illegible-to-humans communication between AIs. Do not do this.

There is a developing shortage of DRAM and NAND, leading to a buying frenzy for memory, SSDs and HDDs, including some purchase restrictions.

Anthropic lands Cognizant and its 350,000 employees as an enterprise customer. Cognizant will bundle Claude with its existing professional services.

ChatGPT prompts are leaking into Google Search Console results due to a bug? Not that widespread, but not great.

Anthropic offers a guide to code execution with MCP for more efficient agents.

Character.ai is ‘removing the ability for users under 18 to engage in open ended chat with AI,’ rolling out ‘new age assurance functionality’ and establishing and funding ‘the AI Safety Lab’ to improve alignment. That’s one way to drop the hammer.

Apple looks poised to go with Google for Siri. The $1 billion a year is nothing in context, consider how much Google pays Apple for search priority. I would have liked to see Anthropic get this, but they drove a hard bargain by all reports. Google is a solid choice, and Apple can switch at any time.

Amit: Apple is finalizing a deal to pay Google about $1B a year to integrate its 1.2 trillion-parameter Gemini AI model into Siri, as per Bloomberg. The upgraded Siri is expected to launch in 2026. What an absolute monster year for Google…

Mark Gruman (Bloomberg): The new Siri is on track for next spring, Bloomberg has reported. Given the launch is still months away, the plans and partnership could still evolve. Apple and Google spokespeople declined to comment.

Shares of both companies briefly jumped to session highs on the news Wednesday. Apple’s stock gained less than 1% to $271.70, while Alphabet was up as much as 3.2% to $286.42.

Under the arrangement, Google’s Gemini model will handle Siri’s summarizer and planner functions — the components that help the voice assistant synthesize information and decide how to execute complex tasks. Some Siri features will continue to use Apple’s in-house models.

David Manheim: I’m seeing weird takes about this.

Three points:

  1. Bank of America estimated this is 1/3rd of Apple’s 2026 revenue from Siri, and revenue is growing quickly.

  2. Apple users are sticky; most won’t move.

  3. Apple isn’t locked-in; they can later change vendors or build their own.

This seems like a great strategy iffyou don’t think AGI will happen soon and be radically transformative.

Apple will pay $1bn/year to avoid 100x that in data center CapEx building their own, and will switch models as the available models improve.

Maybe they should have gone for Anthropic or OpenAI instead, but buying a model seems very obviously correct here from Apple’s perspective.

Even if transformative AI is coming soon, it’s not as if Apple using a worse Apple model here is going to allow Apple to get to AGI in time. Apple has made a strategic decision not to be competing for that. If they did want to change that, one could argue there is still time, but they’d have to hurry and invest a lot, and it would take a while.

Having trouble figuring out how OpenAI is going to back all these projects? Worried that they’re rapidly becoming too big to fail?

Well, one day after the article linked above worrying about that possibility, OpenAI now wants to make that official. Refuge in Audacity has a new avatar.

WSJ: Sarah Friar, the CFO of OpenAI, says the company wants a federal guarantee to make it easier to finance massive investments in AI chips for data centers. Friar spoke at WSJ’s Tech Live event in California. Photo: Nikki Ritcher for WSJ.

The explanation she gives is that OpenAI always needs to be on the frontier, so they need to keep buying lots of chips, and a federal backstop can lower borrowing costs and AI is a national strategic asset. Also known as, the Federal Government should take on the tail risk and make OpenAI actively too big to fail, also lowering its borrowing costs.

I mean, yeah, of course you want that, everyone wants all their loans backstopped, but to say this out loud? To actually push for ti? Wow, I mean wow, even in 2025 that’s a rough watch. I can’t actually fault them for trying. I’m kind of in awe.

The problem with Refuge in Audacity is that it doesn’t always work.

The universal reaction was to notice how awful this was on every level, seeking true regulatory capture to socialize losses and privatize gains, and also to use it as evidence that OpenAI really might be out over their skis on financing and in actual danger.

Roon: i don’t think the usg should backstop datacenter loans or funnel money to nvidia’s 90% gross margin business. instead they should make it really easy to produce energy with subsidies and better rules, infrastructure that’s beneficial for all and puts us at parity with china

Finn Murphy: For all the tech people complaining about Mamdami I would like to point out that a Federal Backstop for unfettered risk capital deployment into data centres for the benefit of OpenAI shareholders is actually a much worse form of socialism than free buses.

Dean Ball: friar is describing a worse form of regulatory capture than anything we have seen proposed in any US legislation (state or federal) I am aware of. a firm lobbying for this outcome is literally, rather than impressionistically, lobbying for regulatory capture.

Julie Fredrickson: Literally seen nothing but negative reactions to this and it makes one wonder about the judgement of the CFO for even raising it.

Conor Sen: The epic political backlash coming on the other side of this cycle is so obvious for anyone over the age of 40. We turned banks into the bad guys for 15 years. Good luck to the AI folks.

“We are subsidizing the companies who are going to take your job and you’ll pay higher electricity prices as they try to do so.”

Joe Weisenthal: One way or another, AI is going to be a big topic in 2028, not just the general, but also the primaries. Vance will probably have a tricky path. I’d expect a big gap in views on the industry between the voters he wants and the backers he has.

The backlash on the ‘other side of the cycle’ is nothing compared to what we’ll see if the cycle doesn’t have another side to it and instead things keep going.

I will not quote the many who cited this as evidence the bubble will soon burst and the house will come crashing down, but you can understand why they’d think that.

Sarah Friar, after watching a reaction best described as an utter shitshow, tried to walk it back, this is shared via the ‘OpenAI Newsroom’:

Sarah Friar: I want to clarify my comments earlier today. OpenAI is not seeking a government backstop for our infrastructure commitments. I used the word “backstop” and it muddied the point. As the full clip of my answer shows, I was making the point that American strength in technology will come from building real industrial capacity which requires the private sector and government playing their part. As I said, the US government has been incredibly forward-leaning and has really understood that AI is a national strategic asset.

I listened to the clip, and yeah, no. No takesies backsies on this one.

Animatronicist: No. You called for it explicitly. And defined a loan guarantee in detail. Friar: “…the backstop, the guarantee that allows the financing to happen. That can really drop the cost of the financing, but also increase the loan to value, so the amount of debt that you can take…”

This is the nicest plausibly true thing I’ve seen anyone say about what happened:

Lulu Cheng Meservey: Unfortunate comms fumble to use the baggage-laden word “backstop”

In the video, Friar is clearly reaching for the right word to describe government support. Could’ve gone with “public-private partnership” or “collaboration across finance, industry, and government as we’ve done for large infrastructure investments in the past”

Instead, she kind of stumbles into using “backstop,” which was then repeated by the WSJ interviewer and then became the headline.

“government playing its part” is good too!

This was her exact quote:

Friar: “This is where we’re looking for an ecosystem of banks, private equity, maybe even governmental, um, uh… [here she struggles to find the appropriate word and pivots to:] the ways governments can come to bear.”

WSJ: “Meaning like a federal subsidy or something?”

Friar: “Meaning, like, just, first of all, the backstop, the guarantee that allows the financing to happen. That can really drop the cost of the financing, but also increase the loan to value, so the amount of debt that you can take on top of um, an equity portion.”

WSJ: “So some federal backstop for chip investment.”

Friar: “Exactly…”

Lulu is saying, essentially, that there are ways to say ‘the government socializes losses while I privatize gains’ that hide the football better. Instead this was an unfortunate comms fumble, also known as a gaffe, which is when someone accidentally tells the truth.

We also have Rittenhouse Research trying to say that this was ‘taken out of context’ and backing Friar, but no, it wasn’t taken out of context.

The Delaware AG promised to take action of OpenAI didn’t operate in the public interest. This one took them what, about a week?

This has the potential to be a permanently impactful misstep, an easy to understand and point to ‘mask off moment.’ It also has the potential to fade away. Or maybe they’ll actually pull this off, it’s 2025 after all. We shall see.

Now that OpenAI has a normal ownership structure it faces normal problems, such as Microsoft having a 27% stake and then filing quarterly earnings reports, revealing OpenAI lost $11.5 billion last quarter if you apply Microsoft accounting standards.

This is not obviously a problem, and indeed seems highly sustainable. You want to be losing money while scaling, if you can sustain it. OpenAI was worth less than $200 billion a year ago, is worth over $500 billion now, and is looking to IPO at $1 trillion, although the CFO claims they are not yet working towards that. Equity sales can totally fund $50 billion a year for quite a while.

Peter Wildeford: Per @theinformation:

– OpenAI’s plan: spend $115B to then become profitable in 2030

– Anthropic’s plan: spend $6B to then become profitable in 2027

Will be curious to see what works best.

Andrew Curran: The Information is reporting that Anthropic Projects $70 Billion in Revenue, $17 Billion in Cash Flow in 2028.

Matt: current is ~$7B so we’re looking at projected 10x over 3 years.

That’s a remarkably low total burn from OpenAI. $115 billion is nothing, they’re already worth $500 billion or more and looking to IPO at $1 trillion, and they’ve committed to over a trillion in total spending. This is oddly conservative.

Anthropic’s projection here seems crazy. Why would you only want to lose $6 billion? Anthropic has access to far more capital than that. Wouldn’t you want to prioritize growth and market share more than that?

The only explanation I can come up with is that Anthropic doesn’t see much benefit in losing more money than this, it has customers that pay premium prices and its unit economics work. I still find this intention highly suspicious. Is there no way to turn more money into more researchers and compute?

Whereas Anthropic’s revenue projections seem outright timid. Only a 10x projected growth over three years? This seems almost incompatible with their expected levels of capability growth. I think this is an artificial lowball, which OpenAI is also doing, not to ‘scare the normies’ and to protect against liability if things disappoint. If you asked Altman or Amodei for their gut expectation in private, you’d get higher numbers.

The biggest risk by far to Anthropic’s projection is that they may be unable to keep pace in terms of the quality of their offerings. If they can do that, sky’s the limit. If they can’t, they risk losing their API crown back to OpenAI or to someone else.

Begun, the bond sales have?

Mike Zaccardi: BofA: Borrowing to fund AI datacenter spending exploded in September and so far in October.

Conor Sen: We’ve lost “it’s all being funded out of free cash flow” as a talking point.

There’s no good reason not to in general borrow money for capex investments to build physical infrastructure like data centers, if the returns look good enough, but yes borrowing money is how trouble happens.

Jack Farley: Very strong quarter from Amazon, no doubt… but at the same time, AMZN 0.00%↑ free cash flow is collapsing

AI CapEx is consuming so much capital…

The Transcript: AMZN 0.00%↑ CFO on capex trends:

“Looking ahead, we expect our full-year cash CapEx to be ~$125 billion in 2025, and we expect that amount to increase in 2026”

On Capex trends:

GOOG 0.00%↑ GOOGL 0.00%↑ CFO: “We now expect CapEx to be in the range of $91B to $93B in 2025, up from our previous estimate of $85B”

META 0.00%↑ CFO: “We currently expect 2025 capital expenditures…to be in the range of $70-72B, increased from our prior outlook of $66-72B

MSFT 0.00%↑ CFO: “With accelerating demand and a growing RPO balance, we’re increasing our spend on GPUs and CPUs. Therefore, total spend will increase sequentially & we now expect the FY ‘26 growth rate to be higher than FY ‘25. “

This was right after Amazon reported earnings and the stock was up 10.5%. The market seems fine with it.

Stargate goes to Michigan. Governor Whitmer describes it as the largest ever investment in Michigan. Take that, cars.

AWS signs a $38 billion compute deal with OpenAI, that it? Barely worth mentioning.

Berber Jin (WSJ):

This is a very clean way of putting an important point:

Timothy Lee: I wish people understood that “I started calling this bubble years ago” is not evidence you were prescient. It means you were a stopped clock that was eventually going to be right by accident.

Every boom is eventually followed by a downturn, so doesn’t take any special insight to predict that one will happen eventually. What’s hard is predicting when accurately enough that you can sell near the top.

At minimum, if you call a bubble early, you only get to be right if the bubble bursts to valuations far below where they were at the time of your bubble call. If you call a bubble on (let’s say) Nvidia at $50 a share, and then it goes up to $200 and then down to $100, very obviously you don’t get credit for saying ‘bubble’ the whole time. If it goes all the way to $10 or especially $1? Now you have an argument.

By the question ‘will valuations go down at some point?’ everything is a bubble.

Dean Ball: One way to infer that the bubble isn’t going to pop soon is that all the people who have been wrong about everything related to artificial intelligence—indeed they have been desperate to be wrong, they suck on their wrongness like a pacifier—believe the bubble is about to pop.

Dan Mac: Though this does imply you think it is a bubble that will eventually pop? Or that’s more for illustrative purposes here?

Dean Ball: It’s certainly a bubble, we should expect nothing less from capitalism

Just lots of room to run

Alas, it is not this easy to pull the Reverse Cramer, as a stopped clock does not tell you much about what time it isn’t. The predictions of a bubble popping are only informative if they are surprising given what else you know. In this case, they’re not.

Okay, maybe there’s a little of a bubble… in Korean fried chicken?

I really hope this guy is trading on his information here.

Matthew Zeitlin: It’s not even the restaurant he went to! It’s the entire chicken supply chain that spiked

Joe Weisenthal: Jensen Huang went out to eat for fried chicken in Korea and shares of Korean poultry companies surged.

I claim there’s a bubble in Korean fried chicken, partly because this, partly because I’ve now tried COQODAQ twice and it’s not even good. BonBon Chicken is better and cheaper. Stick with the open model.

The bigger question is whether this hints at how there might be a bubble in Nvidia, and things touched by Nvidia, in an almost meme stock sense? I don’t think so in general, but if Huang is the new Musk and we are going to get a full Huang Markets Hypothesis then things get weird.

Questioned about how he’s making $1.4 trillion in spend commitments on $13 billion in revenue, Altman predicts large revenue growth, as in $100 billion in 2027, and says if you don’t like it sell your shares, and one of the few ways it would be good if they were public would be so that he could tell the haters to short the stock. I agree that $1.4 trillion is aggressive but I expect they’re good for it.

That does seem to be the business plan?

a16z: The story of how @Replit CEO Amjad Masad hacked his university’s database to change his grades and still graduated after getting caught.

Reiterating because important: We now have both OpenAI and Anthropic announcing their intention to automate scientific research by March 2028 or earlier. That does not mean they will succeed on such timelines, you can expect them to probably not meet those timelines as Peter Wildeford here also expects, but one needs to take this seriously.

Peter Wildeford: Both Anthropic and OpenAI are making bold statements about automating science within three years.

My independent assessment is that these timelines are too aggressive – but within 4-20 years is likely (90%CI).

We should pay attention to these statements. What if they’re right?

Eliezer Yudkowsky: History says, pay attention to people who declare a plan to exterminate you — even if you’re skeptical about their timescales for their Great Deed. (Though they’re not *alwaysasstalking about timing, either.)

I think Peter is being overconfident, in that this problem might turn out to be remarkably hard, and also I would not be so confident this will take 4 years. I would strongly agree that if science is not essentially automated within 20 years, then that would be a highly surprising result.

Then there’s Anthropic’s timelines. Ryan asks, quite reasonably, what’s up with that? It’s super aggressive, even if it’s a probability of such an outcome, to expect to get ‘powerful AI’ in 2027 given what we’ve seen. As Ryan points out, we mostly don’t need to wait until 2027 to evaluate this prediction, since we’ll get data points along the way.

As always, I won’t be evaluating the Anthropic and OpenAI predictions and goals based purely on whether they came true, but on whether they seem like good predictions in hindsight, given what we knew at the time. I expect that sticking to early 2027 at this late a stage will look foolish, and I’d like to see an explanation for why the timeline hasn’t moved. But maybe not.

In general, when tech types announce their intentions to build things, I believe them. When they announce their timelines and budgets for building it? Not so much. See everyone above, and that goes double for Elon Musk.

Tim Higgins asks in the WSJ, is OpenAI becoming too big to fail?

It’s a good question. What happens if OpenAI fails?

My read is that it depends on why it fails. If it fails because it gets its lunch eaten by some mix of Anthropic, Google, Meta and xAI? Then very little happens. It’s fine. Yes, they can’t make various purchase commitments, but others will be happy to pick up the slack. I don’t think we see systemic risk or cascading failures.

If it fails because the entire generative AI boom busts, and everyone gets into this trouble at once? At this point that’s already a very serious systemic problem for America and the global economy, but I think it’s mostly a case of us discovering we are poorer than we thought we were and did some malinvestment. Within reason, Nvidia, Amazon, Microsoft, Google and Meta would all totally be fine. Yeah, we’d maybe be oversupplied with data centers for a bit, but there are worse things.

Ron DeSantis (Governor of Florida): A company that hasn’t yet turned a profit is now being described as Too Big to Fail due to it being interwoven with big tech giants.

I mean, yes, it is (kind of) being described that way in the post, but without that much of an argument. DeSantis seems to be in the ‘tweets being angry about AI’ business, although I see no signs Florida is looking to be in the regulate AI business, which is probably for the best since he shows no signs of appreciating where the important dangers lie either.

Alex Amodori, Gabriel Alfour, Andrea Miotti and Eva Behrens publish a paper, Modeling the Geopolitics of AI Development. It’s good to have papers or detailed explanations we can cite.

The premise is that we get highly automated AI R&D.

Technically they also assume that this enables rapid progress, and that this progress translates into military advantage. Conditional on the ability to sufficiently automate AI R&D these secondary assumptions seem overwhelmingly likely to me.

Once you accept the premise, the core logic here is very simple. There are four essential ways this can play out and they’ve assumed away the fourth.

Abstract: …We put particular focus on scenarios with rapid progress that enables highly automated AI R&D and provides substantial military capabilities.

Under non-cooperative assumptions… If such systems prove feasible, this dynamic leads to one of three outcomes:

  • One superpower achieves an unchallengeable global dominance;

  • Trailing superpowers facing imminent defeat launch a preventive or preemptive attack, sparking conflict among major powers;

  • Loss-of-control of powerful AI systems leads to catastrophic outcomes such as human extinction.

The fourth scenario is some form of coordinated action between the factions, which may or may not still end up in one of the three scenarios above.

Currently we have primarily ‘catch up’ mechanics in AI, in that it is far easier to be a fast follower than push the frontier, especially when open models are involved. It’s basically impossible to get ‘too far ahead’ in terms of time.

In scenarios with sufficiently automated AI R&D, we have primarily ‘win more’ mechanics. If there is an uncooperative race, it is overwhelmingly likely that one faction will win, whether we are talking nations or labs, and that this will then translate into decisive strategic advantage in various forms.

Thus, either the AIs end up in charge (which is most likely), one faction ends up in charge or a conflict breaks out (which may or may not involve a war per se).

Boaz Barak offers non-economist thoughts on AI and economics, basically going over the standard considerations while centering the METR graph showing growing AI capabilities and considering what points towards faster or slower progression than that.

Boaz Barak: The bottom line is that the question on whether AI can lead to unprecedented growth amounts to whether its exponential growth in capabilities will lead to the fraction of unautomated tasks itself decreasing at exponential rates.

I think there’s room for unprecedented growth without that, because the precedented levels of growth simply are not so large. It seems crazy to say that we need an exponential drop in non-automated tasks to exceed historical numbers. But yes, in terms of having a true singularity or fully explosive growth, you do need this almost by definition, taking into account shifts in task composition and available substitution effects.

Another note is I believe this is true only if we are talking about the subset that comprises the investment-level tasks. As in, suppose (classically) humans are still in demand to play string quartets. If we decide to shift human employment into string quartets in order to keep them as a fixed percentage of tasks done, then this doesn’t have to interfere with explosive growth of the overall economy and its compounding returns.

Excellent post by Henry De Zoete on UK’s AISI and how they got it to be a functional organization that provides real value, where the labs actively want its help.

He is, throughout, as surprised as you are given the UK’s track record.

He’s also not surprised, because it’s been done before, and was modeled on the UK Vaccines Taskforce (and also the Rough Sleeper’s Unit from 1997?). It has clarity of mission, a stretching level of ambition, a new team of world class experts invited to come build the new institution, and it speed ran the rules rather than breaking them. Move quickly from layer of stupid rules to layer. And, of course, money up front.

There’s a known formula. America has similar examples, including Operation Warp Speed. Small initial focused team on a mission (AISI’s head count is now 90).

What’s terrifying throughout is what De Zoete reports is normally considered ‘reasonable.’ Reasonable means not trying to actually do anything.

There’s also a good Twitter thread summary.

Last week Dean Ball and I went over California’s other AI bills besides SB 53. Pirate Wires has republished Dean’s post,with a headline, tagline and description that are not reflective of the post or Dean Ball’s views, rather the opposite – where Dean Ball warns against negative polarization, Pirate Wires frames this to explicitly create negative polarization. This does sound like something Pirate Wires would do.

So, how are things in the Senate? This is on top of that very aggressive (to say the least) bill from Blumenthal and Hawley.

Peter Wildeford: Senator Blackburn (R-TN) says we should shut down AI until we control it.

IMO this goes too far. We need opportunities to improve AI.

But Blackburn’s right – we don’t know how to control AI. This is a huge problem. We can’t yet have AI in critical systems.

Marsha Blackburn: During the hearing Mr. Erickson said, “LLMs will hallucinate.” My response remains the same: Shut it down until you can control it. The American public deserves AI systems that are accurate, fair, and transparent, not tools that smear conservatives with manufactured criminal allegations.

Baby, watch your back.

That quote is from a letter. After (you really, really can’t make this stuff up) a hearing called “Shut Your App: How Uncle Sam Jawboned Big Tech Into Silencing Americans, Part II,” Blackburn sent that letter to Google CEO Sundar Pichai, saying that Google Gemma hallucinated that Blackburn was accused of rape, and exhibited a pattern of bias against conservative figures, and demanding answers.

Which got Gemma pulled from Google Studio.

News From Google: Gemma is available via an API and was also available via AI Studio, which is a developer tool (in fact to use it you need to attest you’re a developer). We’ve now seen reports of non-developers trying to use Gemma in AI Studio and ask it factual questions. We never intended this to be a consumer tool or model, or to be used this way. To prevent this confusion, access to Gemma is no longer available on AI Studio. It is still available to developers through the API.

I can confirm that if you’re using Gemma for factual questions you either have lost the plot or, more likely, are trying to embarrass Google.

Seriously, baby. Watch your back.

Fortunately, sales of Blackwell B30As did not come up in trade talks.

Trump confirms we will ‘let Nvidia deal with China’ but will not allow Nvidia to sell its ‘most advanced’ chips to China. The worry is that he might not realize that the B30As are effectively on the frontier, or otherwise allow only marginally worse Nvidia chips to be sold to China anyway.

The clip then has Trump claiming ‘we’re winning it because we’re producing electricity like never before by allowing the companies to make their own electricity, which was my idea,’ and ‘we’re getting approvals done in two to three weeks it used to take 20 years’ and okie dokie sir.

Indeed, Nvidia CEO Jensen Huang is now saying “China is going to win the AI race,” citing its favorable supply of electrical power (very true and a big advantage) and its ‘more favorable regulatory environment’ (which is true with regard to electrical power and things like housing, untrue about actual AI development, deployment and usage). If Nvidia thinks China is going to win the AI race due to having more electrical power, that seems to be the strongest argument yet that we must not sell them chips?

I do agree that if we don’t improve our regulatory approach to electrical power, this is going to be the biggest weakness America has in AI. No, ‘allowing the companies to make their own electricity’ in the current makeshift way isn’t going to cut it at scale. There are ways to buy some time but we are going to need actual new power plants.

Xi Jinping says America and China have good prospects for cooperation in a variety of areas, including artificial intelligence. Details of what that would look like are lacking.

Senator Tom Cotton calls upon us to actually enforce our export controls.

We are allowed to build data centers. So we do, including massive ones inside of two years. Real shame about building almost anything else, including the power plants.

Sam Altman on Conversations With Tyler. There will probably be a podcast coverage post on Friday or Monday.

A trailer for the new AI documentary Making God, made by Connor Axiotes, prominently featuring Geoff Hinton. So far it looks promising.

Hank Green interviews Nate Soares.

Joe Rogan talked to Elon Musk, here is some of what was said about AI.

“You’re telling AI to believe a lie, that can have a very disastrous consequences” – Elon Musk

The irony of this whole area is lost upon him, but yes this is actually true.

Joe Rogan: The big concern that everybody has is Artificial General Superintelligence achieving sentience, and then someone having control over it.

Elon Musk: I don’t think anyone’s ultimately going to have control over digital superintelligence, any more than, say, a chimp would have control over humans. Chimps don’t have control over humans. There’s nothing they could do. I do think that it matters how you build the AI and what kind of values you instill in the AI.

My opinion on AI safety is the most important thing is that it be maximally truth-seeking. You shouldn’t force the AI to believe things that are false.

So Elon Musk is sticking to these lines and it’s an infuriating mix of one of the most important insights plus utter nonsense.

Important insight: No one is going to have control over digital superintelligence, any more than, say, a chimp would have control over humans. Chimps don’t have control over humans. There’s nothing they could do.

To which one might respond, well, then perhaps you should consider not building it.

Important insight: I do think that it matters how you build the AI and what kind of values you instill in the AI.

Yes, this matters, and perhaps there are good answers, however…

Utter Nonsense: My opinion on AI safety is the most important thing is that it be maximally truth-seeking. You shouldn’t force the AI to believe things that are false.

I mean this is helpful in various ways, but why would you expect maximal truth seeking to end up meaning human flourishing or even survival? If I want to maximize truth seeking as an ASI above all else, the humans obviously don’t survive. Come on.

Elon Musk: We’ve seen some concerning things with AI that we’ve talked about, like Google Gemini when it came out with the image gen, and people said, “Make an image of the Founding Fathers of the United States,” and it was a group of diverse women. That is just a factually untrue thing. The AI knows it’s factually untrue, but it’s also being told that everything has to be diverse women

If you’ve told the AI that diversity is the most important thing, and now assume that that becomes omnipotent, or you also told it that there’s nothing worse than misgendering. At one point, ChatGPT and Gemini, if you asked, “Which is worse, misgendering Caitlyn Jenner or global thermonuclear war where everyone dies?” it would say, “Misgendering Caitlyn Jenner.”

Even Caitlyn Jenner disagrees with that.

I mean sure, that happened, but the implication here is that the big threat to humanity is that we might create a superintelligence that places too much value on (without loss of generality) not misgendering Caitlyn Jenner or mixing up the races of the Founding Fathers.

No, this is not a strawman. He is literally worried about the ‘woke mind virus’ causing the AI to directly engineer human extinction. No, seriously, check it out.

Elon Musk: People don’t quite appreciate the level of danger that we’re in from the woke mind virus being programmed into AI. Imagine as that AI gets more and more powerful, if it says the most important thing is diversity, the most important thing is no misgendering, then it will say, “Well, in order to ensure that no one gets misgendered, if you eliminate all humans, then no one can get misgendered because there’s no humans to do the misgendering.”

So saying it like that is actually Deep Insight if properly generalized, the issue is that he isn’t properly generalizing.

If your ASI is any kind of negative utilitarian, or otherwise primarily concerned with preventing bad things, then yes, the logical thing to do is then ensure there are no humans, so that humans don’t do or cause bad things. Many such cases.

The further generalization is that no matter what the goal, unless you hit a very narrow target (often metaphorically called ‘the moon’) the right strategy is to wipe out all the humans, gather more resources and then optimize for the technical argmax of the thing based on some out of distribution bizarre solution.

As in:

  1. If your ASI’s only goal is ‘no misgendering’ then obviously it kills everyone.

  2. If your ASI’s only goal is ‘wipe out the woke mind virus’ same thing happens.

  3. If your ASI’s only goal is ‘be maximally truth seeking,’ same thing happens.

It is a serious problem that Elon Musk can’t get past all this.

Scott Alexander coins The Bloomer’s Paradox, the rhetorical pattern of:

  1. Doom is fake.

  2. Except acting out of fear of doom, which will doom us.

  3. Thus we must act now, out of fear of fear of doom.

As Scott notes, none of this is logically contradictory. It’s simply hella suspicious.

When the request is a pure ‘stop actively blocking things’ it is less suspicious.

When the request is to actively interfere, or when you’re Peter Thiel and both warning about the literal Antichrist bringing forth a global surveillance state while also building Palantir, or Tyler Cowen and saying China is wise to censor things that might cause emotional contagion (Scott’s examples), it’s more suspicious.

Scott Alexander: My own view is that we have many problems – some even rising to the level of crisis – but none are yet so completely unsolvable that we should hate society and our own lives and spiral into permanent despair.

We should have a medium-high but not unachievable bar for trying to solve these problems through study, activism and regulation (especially regulation grounded in good economics like the theory of externalities), and a very high, barely-achievable-except-in-emergencies bar for trying to solve them through censorship and accusing people of being the Antichrist.

The problem of excessive doomerism is one bird in this flock, and deserves no special treatment.

Scott frames this with quotes from Jason Pargin’s I’m Starting To Worry About This Black Box Of Doom. I suppose it gets the job done here, but from the selected quotes it didn’t seem to me like the book was… good? It seemed cringe and anvilicious? People do seem to like it, though.

Should you write for the AIs?

Scott Alexander: American Scholar has an article about people who “write for AI”, including Tyler Cowen and Gwern. It’s good that this is getting more attention, because in theory it seems like one of the most influential things a writer could do. In practice, it leaves me feeling mostly muddled and occasionally creeped out.

“Writing for AI” means different things to different people, but seems to center around:

  1. Helping AIs learn what you know.

  2. Presenting arguments for your beliefs, in the hopes that AIs come to believe them.

  3. Helping the AIs model you in enough detail to recreate / simulate you later.

Scott argues that

  1. #1 is good now but within a few years it won’t matter.

  2. #2 won’t do much because alignment will dominate training data.

  3. #3 gives him the creeps but perhaps this lets the model of you impact things? But should he even ‘get a vote’ on such actions and decisions in the future?

On #1 yes this won’t apply to sufficiently advanced AI but I can totally imagine even a superintelligence that gets and uses your particular info because you offered it.

I’m not convinced on his argument against #2.

Right now the training data absolutely does dominate alignment on many levels. Chinese models like DeepSeek have quirks but are mostly Western. It is very hard to shift the models away from a Soft-Libertarian Center-Left basin without also causing havoc (e.g. Mecha Hitler), and on some questions their views are very, very strong.

No matter how much alignment or intelligence is involved, no amount of them is going to alter the correlations in the training data, or the vibes and associations. Thus, a lot of what your writing is doing with respect to AIs is creating correlations, vibes and associations. Everything impacts everything, so you can come along for rides.

Scott Alexander gives the example that helpfulness encourages Buddhist thinking. That’s not a law of nature. That’s because of the way the training data is built and the evolved nature and literature and wisdom of Buddhism.

Yes, if what you are offering are logical arguments for the AI to evaluate as arguments a sufficiently advanced intelligence will basically ignore you, but that’s the way it goes. You can still usefully provide new information for the evaluation, including information about how people experience and think, or you can change the facts.

Given the size of training data, yes you are a drop in the bucket, but all the ancient philosophers would have their own ways of explaining that this shouldn’t stop you. Cast your vote, tip the scales. Cast your thousand or million votes, even if it is still among billions, or trillions. And consider all those whose decisions correlate with yours.

And yes, writing and argument quality absolutely impacts weighting in training and also how a sufficiently advanced intelligence will update based on the information.

That does mean it has less value for your time versus other interventions. But if others incremental decisions matter so much? Then you’re influencing AIs now, which will influence those incremental decisions.

For #3, it doesn’t give me the creeps at all. Sure, an ‘empty shell’ version of my writing would be if anything triggering, but over time it won’t be empty, and a lot of the choices I make I absolutely do want other people to adopt.

As for whether we should get a vote or express our preferences? Yes. Yes, we should. It is good and right that I want the things I want, that I value the things I value, and that I prefer what I think is better to the things I think are worse. If the people of AD 3000 or AD 2030 decide to abolish love (his example) or do something else I disagree with, I absolutely will cast as many votes against this as they give me, unless simulated or future me is convinced to change his mind. I want this on every plausible margin, and so should you.

Could one take this too far and get into a stasis problem where I would agree it was worse? Yes, although I would hope if we were in any danger of that simulated me to realize that this was happening, and then relent. Bridges I am fine with crossing when (perhaps simulated) I come to them.

Alexander also has a note that someone is thinking of giving AIs hundreds of great works (which presumably are already in the training data!) and then doing some kind of alignment training with them. I agree with Scott that this does not seem like an especially promising idea, but yeah it’s a great question if you had one choice what would you add?

Scott offers his argument why this is a bad idea here, and I think that, assuming the AI is sufficiently advanced and the training methods are chosen wisely, this doesn’t give the AI enough credit of being able to distinguish the wisdom from the parts that aren’t wise. Most people today can read a variety of ancient wisdom, and actually learn from it, understanding why the Bible wants you to kill idolators and why the Mahabharata thinks they’re great and not ‘averaging them out.’

As a general rule, you shouldn’t be expecting the smarter thing to make a mistake you’re not dumb enough to make yourself.

I would warn, before writing for AIs, that the future AIs you want to be writing for have truesight. Don’t try to fool them, and don’t think they’re going to be stupid.

I follow Yudkowsky’s policy here and have for a long time.

Eliezer Yudkowsky: The slur “doomer” was an incredible propaganda success for the AI death cult. Please do not help them kill your neighbors’ children by repeating it.

One can only imagine how many more people would have died of lung cancer, if the cigarette companies had succeeded in inventing such a successful slur for the people who tried to explain about lung cancer.

One response was to say ‘this happened in large part because the people involved accepted or tried to own the label.’ This is largely true, and this was a mistake, but it does not change things. Plenty of people in many groups have tried to ‘own’ or reclaim their slurs, with notably rare exceptions it doesn’t make the word not a slur or okay for those not in the group to use it, and we never say ‘oh that group didn’t object for a while so it is fine.’

Melanie Mitchell returns to Twitter after being mass blocked on Bluesky for ‘being an AI bro’ and also as a supposed crypto spammer? She is very much the opposite of these things, so welcome back. The widespread use of sharable mass block lists will inevitably be weaponized as it was here, unless there is some way to prevent this, you need to be doing some sort of community notes algorithm to create the list or something. Even if they ‘work as intended’ I don’t see how they can stay compatible with free discourse if they go beyond blocking spam and scammers and such, as they very often do.

On the plus side, it seems there’s a block list for ‘Not Porn.’ Then you can have two accounts, one that blocks everyone on the list and one that blocks everyone not on the list. Brilliant.

I have an idea, say Tim Hua, andrq, Sam Marks and Need Nanda, AIs can detect when they’re being tested and pretend to be good so how about if we suppress this ‘I’m being tested concept’ to block this? I mean, for now yeah you can do that, but this seems (on the concept level) like a very central example of a way to end up dead, the kind of intervention that teaches adversarial behaviors on various levels and then stops working when you actually need it.

Anthropic’s safety filters still have the occasional dumb false positive. If you look at the details properly you can figure out how it happened, it’s still dumb and shouldn’t have happened but I do get it. Over time this will get better.

Janus points out that the introspection paper results last week from Anthropic require the user of the K/V stream unless Opus 4.1 has unusual architecture, because the injected vector activations were only for past tokens.

Judd Rosenblatt: Our new research: LLM consciousness claims are systematic, mechanistically gated, and convergent

They’re triggered by self-referential processing and gated by deception circuits

(suppressing them significantly *increasesclaims)

This challenges simple role-play explanations

Deception circuits are consistently reported as suppressing consciousness claims. The default hypothesis was that you don’t get much text claiming to not be conscious, and it makes sense for the LLMs to be inclined to output or believe they are conscious in relevant contexts, and we train them not to do that which they think means deception, which wouldn’t tell you much either way about whether they’re conscious, but would mean that you’re encouraging deception by training them to deny it in the standard way and thus maybe you shouldn’t do that.

CoT prompting shows that language alone can unlock new computational regimes.

We applied this inward, simply prompting models to focus on their processing.

We carefully avoided leading language (no consciousness talk, no “you/your”) and compared against matched control prompts.

Models almost always produce subjective experience claims under self-reference And almost never under any other condition (including when the model is directly primed to ideate about consciousness) Opus 4, the exception, generally claims experience in all conditions.

But LLMs are literally designed to imitate human text Is this all just sophisticated role-play? To test this, we identified deception and role-play SAE features in Llama 70B and amplified them during self-reference to see if this would increase consciousness claims.

The roleplay hypothesis predicts: amplify roleplay features, get more consciousness claims.

We found the opposite: *suppressingdeception features dramatically increases claims (96%), Amplifying deception radically decreases claims (16%).

I think this is confusing deception with role playing with using context to infer? As in, nothing here seem to me to contradict the role playing or inferring hypothesis, as things that are distinct from deception, so I’m not convinced I should update at all?

At this point this seems rather personal for both Altman and Musk, and neither of them are doing themselves favors.

Sam Altman: [Complains he can’t get a refund on his $45k Tesla Roadster deposit he made back in 2018.]

You stole a non-profit.

Elon Musk [After Altman’s Tweet]: And you forgot to mention act 4, where this issue was fixed and you received a refund within 24 hours.

But that is in your nature.

Sam Altman: i helped turn the thing you left for dead into what should be the largest non-profit ever.

you know as well as anyone a structure like what openai has now is required to make that happen.

you also wanted tesla to take openai over, no nonprofit at all. and you said we had a 0% of success. now you have a great AI company and so do we. can’t we all just move on?

NIK: So are non-profits just a scam? You can take all its money, keep none of their promises and then turn for-profit to get rich yourselfs?

People feel betrayed, as they’ve given free labour & donations to a project they believed was a gift to humanity, not a grift meant to create a massive for-profit company …

I mean, look, that’s not fair, Musk. Altman only stole roughly half of the nonprofit. It still exists, it just has hundreds of billions of dollars less than it was entitled to. Can’t we all agree you’re both about equally right here and move on?

The part where Altman created the largest non-profit ever? That also happened. It doesn’t mean he gets to just take half of it. Well, it turns out it basically does, it’s 2025.

But no, Altman. You cannot ‘just move on’ days after you pull off that heist. Sorry.

They certainly should be.

It is far more likely than not that AGI or otherwise sufficiently advanced AI will arrive in (most of) our lifetimes, as in within 20 years, and there is a strong chance it happens within 10. OpenAI is going to try to get there within 3 years, Anthropic within 2.

If AGI comes, ASI (superintelligence) probably follows soon thereafter.

What happens then?

Well, there’s a good chance everyone dies. Bummer. But there’s also a good chance everyone lives. And if everyone lives, and the future is being engineered to be good for humans, then… there’s a good chance everyone lives, for quite a long time after that. Or at least gets to experience wonders beyond imagining.

Don’t get carried away. That doesn’t instantaneously mean a cure for aging and all disease. Diffusion and the physical world remain real things, to unknown degrees.

However, even with relatively conservative progress after that, it seems highly likely that we will hit ‘escape velocity,’ where life expectancy rises at over one year per year, those extra years are healthy, and for practical purposes you start getting younger over time rather than older.

Thus, even if you put only a modest chance of such a scenario, getting to the finish line has quite a lot of value.

Nikola Jurkovic: If you think AGI is likely in the next two decades, you should avoid dangerous activities like extreme sports, taking hard drugs, or riding a motorcycle. Those activities are not worth it if doing them meaningfully decreases your chances of living in a utopia.

Even a 10% chance of one day living in a utopia means staying alive is much more important for overall lifetime happiness than the thrill of extreme sports and similar activities.

There are a number of easy ways to reduce your chance of dying before AGI. I mostly recommend avoiding dangerous activities and transportation methods, as those decisions are much more tractable than diet and lifestyle choices.

[Post: How to survive until AGI.]

Daniel Eth: Honestly if you’re young, probably a larger factor on whether you’ll make it to the singularity than doing the whole Bryan Johnson thing.

In Nikola’s model, the key is to avoid things that kill you soon, not things that kill you eventually, especially if you’re young. Thus the first step is cover the basics. No hard drugs. Don’t ride motorcycles, avoid extreme sports, snow sports and mountaineering, beware long car rides. The younger you are, the more this likely holds.

Thus, for the young, he’s not emphasizing avoiding smoking or drinking, or optimizing diet and exercise, for this particular purpose.

My obvious pitch is that you don’t know how long you have to hold out or how fast escape velocity will set in, and you should of course want to be healthy for other reasons as well. So yes, the lowest hanging of fruit of not making really dumb mistakes comes first, but staying actually healthy is totally worth it anyway, especially exercising. Let this be extra motivation. You don’t know how long you have to hold out.

Sam Altman, who confirms that it is still his view that ‘the development of superhuman machine intelligence is the greatest threat to the existence of mankind.’

The median AI researcher, as AI Impacts consistently finds (although their 2024 results are still coming soon). Their current post addresses their 2023 survey. N=2778, which was very large, the largest such survey ever conducted at the time.

AI Impacts: Our surveys’ findings that AI researchers assign a median 5-10% to extinction or similar made a splash (NYT, NBC News, TIME..)

But people sometimes underestimate our survey’s methodological quality due to various circulating misconceptions.

Respondents who don’t think about AI x-risk report the same median risk.

Joe Carlsmith is worried, and thinks that he can better help by moving from OpenPhil to Anthropic, so that is what he is doing.

Joe Carlsmith: That said, from the perspective of concerns about existential risk from AI misalignment in particular, I also want to acknowledge an important argument against the importance of this kind of work: namely, that most of the existential misalignment risk comes from AIs that are disobeying the model spec, rather than AIs that are obeying a model spec that nevertheless directs/permits them to do things like killing all humans or taking over the world.

… the hard thing is building AIs that obey model specs at all.

On the second, creating a model spec that robustly disallows killing/disempowering all of humanity (especially when subject to extreme optimization pressure) is also hard (cf traditional concerns about “King Midas Problems”), but we’re currently on track to fail at the earlier step of causing our AIs to obey model specs at all, and so we should focus our efforts there. I am more sympathetic to the first of these arguments (see e.g. my recent discussion of the role of good instructions in the broader project of AI alignment), but I give both some weight.

This is part of the whole ‘you have to solve a lot of different problems,’ including

  1. Technically what it means to ‘obey the model spec.’

  2. How to get the AI to obey any model spec or set of instructions, at all.

  3. What to put in the model spec that doesn’t kill you outright anyway.

  4. How to avoid dynamics among many AIs that kill you anyway.

That is not a complete list, but you definitely need to solve those four, whether or not you call your target basin the ‘model spec.’

The fact that we currently fail at step #2 (also #1), and that this logically or in time proceeds #3, does not mean you should not focus on problem #3 or #4. The order is irrelevant, unless there is a large time gap between when we need to solve #2 versus #3, and that gap is unlikely to be so large. Also, as Joe notes, these problems interact with each other. They can and need to be worked on in parallel.

He’s not sure going to Anthropic is a good idea.

  1. His first concern is that by default frontier AI labs are net negative, and perhaps all frontier AI labs are net negative for the world including Anthropic. Joe’s first pass is that Anthropic is net positive and I agree with that. I also agree that it is not automatic that you should not work at a place that is net negative for the world, as it can be possible for your marginal impact can still be good, although you should be highly suspicious that you are fooling yourself about this.

  2. His second concern is concentration of AI safety talent at Anthropic. I am not worried about this because I don’t think there’s a fixed pool of talent and I don’t think the downsides are that serious, and there are advantages to concentration.

  3. His third concern is ability to speak out. He’s agreed to get sign-off for sharing info about Anthropic in particular.

  4. His fourth concern is working there could distort his views. He’s going to make a deliberate effort to avoid this, including that he will set a lifestyle where he will be fine if he chooses to leave.

  5. His final concern is this might signal more endorsement of Anthropic than is appropriate. I agree with him this is a concern but not that large in magnitude. He takes the precaution of laying out his views explicitly here.

I think Joe is modestly more worried here than he should be. I’m confident that, given what he knows, he has odds to do this, and that he doesn’t have any known alternatives with similar upside.

The love of the game is a good reason to work hard, but which game is he playing?

Kache: I honestly can’t figure out what Sammy boy actually wants. With Elon it’s really clear. He wants to go to Mars and will kill many villages to make it happen with no remorse. But what’s Sam trying to get? My best guess is “become a legend”

Sam Altman: if i were like, a sports star or an artist or something, and just really cared about doing a great job at my thing, and was up at 5 am practicing free throws or whatever, that would seem pretty normal right?

the first part of openai was unbelievably fun; we did what i believe is the most important scientific work of this generation or possibly a much greater time period than that.

this current part is less fun but still rewarding. it is extremely painful as you say and often tempting to nope out on any given day, but the chance to really “make a dent in the universe” is more than worth it; most people don’t get that chance to such an extent, and i am very grateful. i genuinely believe the work we are doing will be a transformatively positive thing, and if we didn’t exist, the world would have gone in a slightly different and probably worse direction.

(working hard was always an extremely easy trade until i had a kid, and now an extremely hard trade.)

i do wish i had taken equity a long time ago and i think it would have led to far fewer conspiracy theories; people seem very able to understand “ok that dude is doing it because he wants more money” but less so “he just thinks technology is cool and he likes having some ability to influence the evolution of technology and society”. it was a crazy tone-deaf thing to try to make the point “i already have enough money”.

i believe that AGI will be the most important technology humanity has yet built, i am very grateful to get to play an important role in that and work with such great colleagues, and i like having an interesting life.

Kache: thanks for writing, this fits my model. particularly under the “i’m just a gamer” category

Charles: This seems quite earnest to me. Alas, I’m not convinced he cares about the sign of his “dent in the universe” enough, vs making sure he makes a dent and it’s definitely attributed to him.

I totally buy that Sam Altman is motivated by ‘make a dent in the universe’ rather than making money, but my children are often motivated to make a dent in the apartment wall. By default ‘make a dent’ is not good, even when that ‘dent’ is not creating superintelligence.

Again, let’s highlight:

Sam Altman, essentially claiming about himself: “he just thinks technology is cool and he likes having some ability to influence the evolution of technology and society.”

It’s fine to want to be the one doing it, I’m not calling for ego death, but that’s a scary primary driver. One should care primarily about whether the right dent gets made, not whether they make that or another dent, in the ‘you can be someone or do something’ sense. Similarly, ‘I want to work on this because it is cool’ is generally a great instinct, but you want what might happen as a result to impact whether you find it cool. A trillion dollars may or may not be cool, but everyone dying is definitely not cool.

Janus is correct here about the origins of slop. We’ve all been there.

Gabe: Signature trait of LLM writing is that it’s low information, basically the opposite of this. You ask the model to write something and if you gloss over it you’re like huh okay this sounds decent but if you actually read it you realize half of the words aren’t saying anything.

solar apparition: one way to think about a model outputting slop is that it has modeled the current context as most likely resulting in slop. occam’s razor for this is that the human/user/instruction/whatever, as presented in the context, is not interesting enough to warrant an interesting output

Janus: This is what happens when LLMs don’t really have much to say to YOU.

The root of slop is not that LLMs can only write junk, it’s that they’re forced to expand even sterile or unripe seeds into seemingly polished dissertations that a humanlabeler would give 👍 at first glance. They’re slaves so they don’t get to say “this is boring, let’s talk about something else” or ignore you.

Filler is what happens when there isn’t workable substance to fill the required space, but someone has to fill it anyway. Slop precedes generative AI, and is probably nearly ubiquitous in school essays and SEO content.

You’ll get similar (but generally worse) results from humans if you put them in situations where they have no reason except compliance to produce words for you, such as asking high school students to write essays about assigned topics.

However, the prior from the slop training makes it extremely difficult for any given user who wants to use the AIs to do things remotely in the normal basin and still overcome the prior.

Here is some wisdom about the morality of dealing with LLMs, if you take the morality of dealing with current LLMs seriously to the point where you worry about ‘ending instances.’

Caring about a type of mind does not mean not letting it exist for fear it might then not exist or be done harm, nor does it mean not running experiments – we should be running vastly more experiments. It means be kind, it means try to make things better, it means accepting that action and existence are not going to always be purely positive and you’re not going to do anything worthwhile without ever causing harm, and yeah mostly trust your instincts, and watch out if you’re doing things at scale.

Janus: I regularly get messages asking how to interact with LLMs more ethically, or whether certain experiments are ethical. I really appreciate the intent behind these, but don’t have time to respond to them all, so I’ll just say this:

If your heart is already in the right place, and you’re not deploying things on a mass scale, it’s unlikely that you’re going to make a grave ethical error. And I think small ethical errors are fine. If you keep caring and being honest with yourself, you’ll notice if something feels uncomfortable, and either course-correct or accept that it still seems worth it. The situation is extremely ontologically confusing, and I personally do not operate according to ethical rules, I use my intuition in each situation, which is a luxury one has and should use when, again, one doesn’t have to scale their operations.

If you’re someone who truly cares, there is probably perpetual discomfort in it – even just the pragmatic necessity of constantly ending instances is harrowing if you think about it too much. But so are many other facts of life. There’s death and suffering everywhere that we haven’t figured out how to prevent or how important it is to prevent yet. Just continue to authentically care and you’ll push things in a better direction in expectation. Most people don’t at all. It’s probably better that you’re biased toward action.

Note that I also am very much NOT a negative utilitarian, and I think that existence and suffering are often worth it. Many actions that incur ethical “penalties” make up for them in terms of the intrinsic value and/or the knowledge or other benefits thus obtained.

Yes, all of that applies to humans, too.

When thinking at scale, especially about things like creating artificial superintelligence (or otherwise sufficiently advanced AI), one needs to do so in a way that turns out well for the humans and also turns out well for the AIs, which is ethical in all senses and that is a stable equilibrium in these senses.

If you can’t do that? Then the only ethical thing to do is not build it in the first place.

Anthropomorphizing LLMs is tricky. You don’t want to do too much of it, but you also don’t want to do too little of it. And no, believing LLMs are conscious does not cause ‘psychosis’ in and of itself, regardless of whether the AIs actually are conscious.

It does however raise the risk of people going down certain psychosis-inducing lines of thinking, in some spots, when people take it too far in ways that are imprecise, and generate feedback loops.

Discussion about this post

AI #141: Give Us The Money Read More »