Author name: Paul Patrick

The quest to turn basalt dust into a viable climate solution

carbon capture, climate adaptation, climate adaptation strategies, Science, syndication / Paul Patrick / December 9, 2023

Mary Yap has spent the last year and a half trying to get farmers to fall in love with basalt. The volcanic rock is chock full of nutrients, captured as its crystal structure forms from cooling magma, and can make soil less acidic. In that way it’s like limestone, which farmers often use to improve their soil. It’s a little more finicky to apply, and certainly less familiar. But basalt also comes with an important side benefit: It can naturally capture carbon from the atmosphere.

Yap’s pitch is part of a decades-long effort to scale up that natural weathering process and prove that it can lock carbon away for long enough to make a different to the climate. “The bottleneck is getting farmers to want to do this,” Yap says.

On Thursday, Yap’s young startup, Lithos Carbon, got a $57.1 million boost for its quest to turn basalt dust into a viable climate solution. It came from Frontier, a benefit corporation backed by a consortium of companies aiming to finance promising approaches to carbon dioxide removal, or CDR. Lithos says it will use the funds to soak up 154,000 tons of CO₂ by 2028, by sprinkling basalt dust on thousands of acres of US farmland. The average US car emits about 4 tons of CO₂each year.

The carbon removal purchase is the largest yet by Frontier, which was formed last year with nearly $1 billion from its tech-dominated members. Many of those companies, which include Meta, Alphabet, and payments processor Stripe, which owns Frontier, have made climate pledges that require not only reducing the emissions from their operations and supply chains but also “negative emissions”—sucking up carbon from the atmosphere to cancel out other emissions.

That accounting trick has been easier to prove out on paper than in practice. Many companies would have once turned to buying carbon offsets from activities like protecting forests that would otherwise be felled. But some have been trying to move away from those scandal-plagued and often short-lived approaches and into more durable techniques for carbon removal.

The current options for companies seeking negative emissions are limited. Frontier’s purchases are essentially down payments on ideas that are still in their infancy—generally too hard to verify or too expensive, or both, to attract a significant customer base. “What we’re trying to evaluate the field on is whether it’s on the trajectory to get to climate-relevant scale,” says Nan Ransohoff, who leads Frontier and also climate work at Stripe. The group starts with small “prepurchases” meant to help promising startups, and then moves on to “offtake” agreements for larger amounts of carbon that its members can count toward their emissions goals.

The Lithos purchase is one of those larger deals. It prices carbon removals at $370 per ton, about a quarter of which will pay for field monitoring and modeling to verify that carbon is being sequestered away from the atmosphere for the long term. Ransohoff says Frontier believes that Lithos is on a path to its goal of removing CO₂ for customers at a cost of less than $100 per ton, and at a rate of at least a half a billion tons per year.

“Most promising” approach

Lithos, founded in 2022, is developing a technology called enhanced rock weathering. It involves spreading a fine dust of basalt across fields before planting. As the rock further weathers from rainfall, it reacts with CO₂in the air. That forms bicarbonate, which locks away the carbon by combining it with hydrogen and oxygen atoms. Ultimately, the compound is washed into the ocean, where the carbon should stay put.

The strategy has the benefit of piggybacking on things that humans already do, Yap says. That’s in contrast with techniques like direct air capture, which involves building industrial plants that suck carbon out of the atmosphere. It’s easy to measure carbon removed that way—it’s all captured there onsite—but critics say it will be difficult to scale up because removing enough carbon to make a difference will require thousands of dedicate, resource-intensive facilities.

Using basalt dust to capture carbon should be more easily scaled up. There are plenty of fields to dump rock dust onto, and plenty of water for carbon to end up in. But the distributed nature of the process also makes measuring how much carbon was actually removed from the atmosphere more difficult.

The quest to turn basalt dust into a viable climate solution Read More »

Hubble back in service after gyro scare—NASA still studying reboost options

Hubble Space Telescope, NASA, Science, Space, spacex / Paul Patrick / December 9, 2023

Enlarge / The Hubble Space Telescope viewed from Space Shuttle Atlantis during a servicing mission in 2009.

NASA

The Hubble Space Telescope resumed science observations on Friday after ground teams spent most of the last three weeks assessing the performance of a finicky gyroscope, NASA said.

The troublesome gyroscope is a critical part of the observatory’s pointing system. Hubble’s gyros measure how fast the spacecraft is turning, helping the telescope aim its aperture toward distant cosmic wonders.

Hubble still provides valuable scientific data for astronomers nearly 34 years since its launch aboard NASA’s Space Shuttle Discovery in 1990. Five more shuttle servicing missions repaired Hubble, upgraded its science instruments, and replaced hardware degraded from long-term use in space. Among other tasks, astronauts on the last of the shuttle repair flights in 2009 installed six new gyroscopes on Hubble.

Moving parts sometimes break

The gyros have long been one of the parts of Hubble that require the most upkeep. A wheel inside each gyro spins at a constant rate of 19,200 revolutions per minute, and the wheel is, in turn, sealed inside a cylinder suspended in a thick fluid, according to NASA. Electronics within each gyro detect very small movements of the axis of the wheel, which supply Hubble’s central computer with information about the spacecraft’s turn rate. Hair-thin wires route signals from the gyroscopes, and these wires can degrade over time.

Three of the six gyros installed on Hubble in 2009 have failed, and three others remain operational. The three still-functioning gyros are based on a newer design for longer life, but one of these units has shown signs of wear in the last few months. This gyroscope, designated Gyro 3, has always exhibited “consistent noisy behavior,” said Pat Crouse, Hubble project manager at NASA’s Goddard Space Flight Center.

Hubble typically needs three gyros to operate normally, so ground controllers shut down Gyro 3 for roughly seven years until Hubble needed it in 2018, when another gyroscope failed, leaving only three of the devices still working.

“Back in August, we saw issues,” Crouse told Ars this week. “It would sort of sporadically output some rate information that was not consistent with the observed spacecraft body rates, but it was short-lived, and we were characterizing what that performance was like and how much we could tolerate.”

The gyro’s performance worsened in November when it fed Hubble’s control system erroneous data. The gyroscope sensed that the spacecraft was changing its orientation when it really wasn’t moving. “That, then, contributed to an error in attitude that was kind of causing a little bit of drift,” Crouse said.

Automated software on Hubble detected the errors and put the spacecraft into “safe mode” two times last month. Hubble quickly resumed science observations each time but then went into safe mode again on November 23. Hubble managers took some extra time to gather data on the gyro’s health. Engineers commanded Hubble to move back and forth, and the suspect gyro consistently seemed to work well.

Hubble back in service after gyro scare—NASA still studying reboost options Read More »

EU agrees to landmark rules on artificial intelligence

AI, AI law, Biz & IT, European Union, syndication / Paul Patrick / December 9, 2023

Get ready for some restrictions, Big Tech —

Legislation lays out restrictive regime for emerging technology.

Javier Espinoza, Financial Times – Dec 9, 2023 12: 09 am UTC

Enlarge / EU Commissioner Thierry Breton talks to media during a press conference in June.

Thierry Monasse | Getty Images

European Union lawmakers have agreed on the terms for landmark legislation to regulate artificial intelligence, pushing ahead with enacting the world’s most restrictive regime on the development of the technology.

Thierry Breton, EU commissioner, confirmed in a post on X that a deal had been reached.

He called it a historic agreement. “The EU becomes the very first continent to set clear rules for the use of AI,” he wrote. “The AIAct is much more than a rulebook—it’s a launchpad for EU start-ups and researchers to lead the global AI race.”

The deal followed years of discussions among member states and politicians on the ways AI should be curbed to have humanity’s interest at the heart of the legislation. It came after marathon discussions that started on Wednesday this week.

Members of the European Parliament have spent years arguing over their position before it was put forward to member states and the European Commission, the executive body of the EU. All three—countries, politicians, and the commission—must agree on the final text before it becomes law.

European companies have expressed their concern that overly restrictive rules on the technology, which is rapidly evolving and gained traction after the popularisation of OpenAI’s ChatGPT, will hamper innovation. Last June, dozens of some of the largest European companies, such as France’s Airbus and Germany’s Siemens, said the rules were looking too tough to nurture innovation and help local industries.

Last month, the UK hosted a summit on AI safety, leading to broad commitments from 28 nations to work together to tackle the existential risks stemming from advanced AI. That event attracted leading tech figures such as OpenAI’s Sam Altman, who has previously been critical of the EU’s plans to regulate the technology.

EU agrees to landmark rules on artificial intelligence Read More »

Revisiting the Ford Mustang Mach-E—how’s the pony EV doing 3 years later?

car review, Cars, electric car review, Ford Mustang Mach-E / Paul Patrick / December 9, 2023

yay or neigh —

This midsize crossover EV has a lot more competition than when it debuted in 2021.

Jonathan M. Gitlin – Dec 8, 2023 11: 44 pm UTC

A Ford Mustang Mach-E, head-on — Enlarge / The Ford Mustang Mach-E is now in its third year of production, so it felt like a good idea to see how it’s maturing.

Jonathan Gitlin

When Ars first drove the then-new Ford Mustang Mach-E back in early 2021, the car was an attention magnet. Now, almost three years later, the Mustang Mach-E is a much more common sight on our roads, but so are other electric crossovers from most of Ford’s usual rivals, including the sales juggernaut that is the Tesla Model Y. We decided to book a few days with a Mustang Mach-E to see how (or if) this equine EV has matured since launch.

Originally, Ford had been working on a much more boring battery electric car until Tesla started delivering its Model 3s, at which point a hastily convened “Team Edison” set to work adding some much-needed brio to the design, rethinking Ford’s EV strategy in the process.

Giving this midsize crossover EV a Mustang name tag remains divisive—I expect a reasonable percentage of comments to this story will be people showing up to complain, “It ain’t no real Mustang.” The crossover’s name is what it is, and there are plenty of Mustang styling cues, but even with the designers’ trick of using black trim to make you ignore the bits they don’t want you to see, there’s no denying the proportions are pretty far from coupe-like.

Enlarge / The Mustang Mach-E has good angles and bad angles.

Jonathan Gitlin

It’s cheaper now

Our test car was a Mustang Mach-E Premium eAWD model with just a single option ticked, the 91 kWh (useable capacity) extended range battery. This increases the car’s EPA range estimate from 224 miles (360 km) to 290 miles (467 km) but costs $8,600, which, combined with the delivery charge, bumps the sticker price to $67,575.

At least, that’s what this Mustang Mach-E cost when it arrived on the press fleet some 5,500 miles ago. Ford had to respond to Tesla’s string of price cuts, dropping the MSRP by almost $7,000 and cutting the cost of the extended range battery to $7,000—when I configure the same spec on Ford’s online car builder, it tells me the total price should be $59,940 with all the various fees. (A further price cut came to most other Mustang Mach-E variants in May, but not for the extended range Premium eAWD.)

And until the end of this year, the EV is still eligible for half of the clean vehicle tax credit. However, Ford believes that $3,750 credit will no longer be available to Mach-E buyers from next year as new rules regarding batteries made by “foreign entities of concern” go into effect. These remove eligibility from EVs batteries made in China or by Chinese-owned companies from January 1, 2024.

Taillights say Mustang, but the car's width and height say crossover. — Enlarge / Taillights say Mustang, but the car’s width and height say crossover.

Jonathan Gitlin

It’s a hard life

The fact that the Mustang Mach-E’s trip computer hadn’t been reset in 3,572 miles (5,749 km) provides an illustrating insight into both the life of a press fleet vehicle as well as the long-term efficiency of this EV. Collectively, the car had been driven very unsympathetically over that time, grading the drivers at 1 percent for deceleration and 2 percent for both acceleration and speed. Despite the lead foot treatment, the average of 2.7 miles/kWh (23 kWh/100 km) matches the EPA efficiency estimate (expressed as 37 kWh/100 miles).

I’m guessing this particular car spent most of those miles in Unbridled, which is what the Mustang Mach-E calls its sport mode. That or Engage, which is the middle of the three settings and the one used to calculate the car’s official efficiency.

In Whisper (think eco mode), you don’t get quite all the 346 hp (258 kW) or the full 428 lb-ft (580 Nm), and the 0–60 time feels between a second or two slower than the 4.8 seconds that’s possible if all the electric horses are harnessed at the same time, at least 3.1 miles/kWh (20 kWh/100 km) should be possible.

Revisiting the Ford Mustang Mach-E—how’s the pony EV doing 3 years later? Read More »

FDA approves first CRISPR therapy—here’s how it works against sickle cell

health, hemoglobin, Science, sickle cell anemia, sickle cell disease / Paul Patrick / December 9, 2023

historic —

The landmark treatment turns on another blood protein that prevents sickling.

Beth Mole – Dec 8, 2023 11: 27 pm UTC

Enlarge / This digitally colorized scanning electron micrograph (SEM) revealed some of the comparative ultrastructural morphology between normal red blood cells (RBCs), and a sickle cell RBC (left) found in a blood specimen of an 18-year-old female patient with sickle cell anemia.

The Food and Drug Administration on Friday approved two gene therapies to treat sickle cell disease, one of the which is the first CRISPR/Cas9-based treatment to win regulatory approval in the US.

The announcement is a landmark in the treatment of sickle cell disease, a devastating condition in which red blood cells deform into a sickle shape and clog up blood vessels. Sickle cell disease affects around 100,000 people in the US, most commonly African Americans. It leads to anemia, vaso-occlusive events and crises (painful episodes in which small blockages starve tissue of oxygen), strokes, progressive and irreversible organ damage, decreased quality of life, and early death.

Until today, treatments have been limited. A bone marrow transplant from a genetically matched sibling can cure the condition more than 90 percent of the time, but only around 20 percent of people with the disease have such a genetically matched sibling donor. There are also multiple drugs available and supportive care, but these mainly reduce the severity of the disease. The new gene therapy treatments, on the other hand, have shown to be highly effective at preventing vaso-occlusive events and crises.

“Sickle cell disease is a rare, debilitating and life-threatening blood disorder with significant unmet need, and we are excited to advance the field, especially for individuals whose lives have been severely disrupted by the disease, by approving two cell-based gene therapies today,” said Nicole Verdun, director of the Office of Therapeutic Products within the FDA’s Center for Biologics Evaluation and Research, said in the FDA’s announcement.

To understand how the gene therapies work, it’s useful to understand what causes sickle cell disease. The central problem is with adult hemoglobin, the iron-containing protein in red blood cells that transports oxygen from the lungs to the rest of the body. In patients with sickle cell disease, there’s a single, small mutation in the gene that encodes hemoglobin. The mutation is a switch of a single nucleotide, or base, (often represented by letters A, C, T, and G). The switch of an A to a T in the genetic code for hemoglobin results in a hemoglobin protein with a valine instead of a glutamic acid at the sixth amino acid position. This transforms normal adult hemoglobin (HbA) to sickle hemoglobin (HbS). In red blood cells, when HbS loses the oxygen it was carrying, it polymerizes with itself, forming strand-like structures that deform the cell.

Effective edits

The CRISPR/Cas9 therapy approved today, called Casgevy, prevents this deformation by essentially turning on the production of another type of hemoglobin encoded in our genetic blueprints—fetal hemoglobin (HbF). HbF is optimized for pregnancy, transferring oxygen from maternal blood to fetal tissue, and the gene that encodes it is shut off shortly after birth as the body transitions to HbA. About six months after birth, HbF usually makes up just 1 percent to 2 percent of hemoglobin in the body.

But, HbF can effectively treat sickle cell disease—the hemoglobin transports oxygen just fine in adults, and it doesn’t polymerize. Moreover, when it’s mixed with HbS, it gets in the way of the mutated protein polymerizing with itself, preventing it from forming structures that deform red blood cells.

Casgevy turns on HbF with the CRISPR/Cas9 system, a gene-editing system initially swiped from bacteria that snips DNA using an enzyme (a nuclease) called Cas9. Cas9 can be targeted to specific stretches of DNA using a short RNA guide sequence. In Casgevy, the CRISPR/Cas9 system is targeted to snip a gene encoding a protein called BCL11A, which controls other genes, aka a transcription factor. The BCL11A transcription factor is the protein responsible for shutting off the gene for HbF shortly after birth as the body transitions to the adult version. With the CRISPR/Cas9 snip, BCL11A is shut off, and HbF production can resume.

For patients being treated, this process involves first harvesting their bone marrow stem cells, which then get CRISPR-ed in a specialized lab. Meanwhile, the patients receive chemotherapy to kill bone marrow cells to make way for the gene-edited cells that are then put back in. Of 31 patients treated with Cagevy and followed for at least 24 months, 29 (93.5 percent) went at least 12 consecutive months without a vaso-occlusive crisis.

The other gene therapy approved by the FDA today is Lyfgenia, which used a Lentiviral vector to insert genes into the human genome. In this case, the system delivers the genetic code for a modified type of hemoglobin that is designed to be anti-sickling, called HbA^T87Q. Among 32 patients treated with Lyfgenia, 28 (88 percent) were free of vaso-occlusive events for between six to 18 months after treatment.

Both gene therapies are approved for patients ages 12 years and up.

FDA approves first CRISPR therapy—here’s how it works against sickle cell Read More »

Apple appears to have blocked Beeper Mini’s iMessage app in less than a week

Apple, beeper, beeper mini, eric migicovsky, iMessage, messaging, rcs, Reverse-engineering, SMS, Tech / Paul Patrick / December 9, 2023

A very mini runway —

Co-founder: “All data indicates that” Apple has cut off Beeper Mini’s reverse-engineering.

Kevin Purdy – Updated Dec 8, 2023 11: 06 pm UTC

Beeper mini promotional splash image — Enlarge / Beeper Mini’s promises of “Blue bubbles” on Android seemed to have been nixed by a certain Cupertino-based firm on Friday.

Beeper

Beeper Mini, the Android app born from a reverse-engineering of Apple’s iMessage service, is currently broken, and it is unknown whether it will resume functioning.

Beeper desktop users received a message from co-founder Eric Migicovsky late on Friday afternoon, noting an “iMessage outage” and that “messages are failing to send and receive.” Reports had started piling up on Reddit around 2: 30 pm Eastern. As of 5: 30 pm, both Beeper Cloud on desktop and the Beeper Mini app were reporting errors in sending and receiving messages, with “Failed to lookup on sever: lookup request timed out.” Comments on Beeper’s status post on X (formerly Twitter) suggested mixed results, at best, among users.

The Verge, messaging with Migicovsky, reported that he “did not deny that Apple has successfully blocked Beeper Mini”; to TechCrunch, Migicovsky more clearly stated about an Apple cut-off: “Yes, all data indicates that.” To both outlets, Migicovsky offered the same comment, re-iterating his belief that it was in the best interests of Apple to let iPhone owners and Android users send encrypted messages to one another. (Ars reached out to Migicovsky for comment and will update this post with new information).

On Saturday, Migicovsky notified Beeper Cloud (desktop) users that iMessage was working again for them, after a long night of fixes. “Work continues on Beeper Mini,” Migicovsky wrote shortly after noon Eastern time.

Responding to a post on X (formerly Twitter) asking if restoring Beeper Mini’s function would be an “endless cat and mouse game,” Migicovsky wrote: “Beeper Cloud and Mini are apps that need to exist. We have built it. We will keep it working. We will share it widely.” He added that such an attitude, “especially from people in the tech world,” surprised him. “Why do hard things at all? Why keep working on anything that doesn’t work the first time?“

Beeper, as it worked shortly before launch on Dec. 5, sending iMessages from a Google Pixel 3 Android phone.

Kevin Purdy

Beeper’s ability to send encrypted iMessages from Android phones grew from a teenager’s reverse-engineering of the iMessage protocol, as Ars detailed at launch. The app could not read message contents (nor could Apple), kept encryption keys and contacts on your device, and did not require an Apple ID to authenticate.

The app did, however, send a text message from a device to an Apple server, and the response was used to generate an encryption key pair, one for Apple and one for your device. A Beeper service kept itself connected to Apple’s servers to notify it and you about new messages. Reddit user moptop and others suggested that Beeper’s service used encryption algorithms whose keys were spoofed to look like they came from a Mac Mini running OS X Mountain Lion, perhaps providing Apple a means of pinpointing and block them.

Members of the Discord focused on the original reverse-engineered tool on which Beeper Mini was built, PyPush, also reported that the tool was down Friday evening. Some noted that it seemed like their phone numbers had additionally been de-registered from iMessage.

Beeper Mini’s iMessage capabilities, for which the company was planning to charge $1.99 per month after a seven-day trial, were more than a feature. The company had planned to build additional secure messaging into Beeper Mini, including Signal and WhatsApp messaging, and make it the primary focus of its efforts. Its prior app Beeper, temporarily renamed Beeper Cloud, was marked to be deprecated at some point in favor of the new iMessage-touting Mini app.

This post was updated at 12: 50 p.m. on Saturday, Dec. 9, to reflect restored function to Beeper Cloud (desktop), and Migicovsky’s social media response after the outage.

Apple appears to have blocked Beeper Mini’s iMessage app in less than a week Read More »

HP printer app is installing on PCs whether they have HP printers or not

hp printer, Tech / Paul Patrick / December 9, 2023

a printer problem? no way! —

Appears to be a Microsoft Store bug; MS says printers should mostly still work.

Andrew Cunningham – Dec 8, 2023 10: 58 pm UTC

Enlarge / The HP LaserJet M106w is one of the printer models that is mysteriously appearing for some users in Windows 10 and 11.

HP

Earlier this month, Microsoft posted a new entry in its list of known issues with fully up-to-date Windows 11 PCs: The HP Smart printer app was installing automatically on Windows 10 and Windows 11 PCs whether they had an HP printer installed or not, changing the names and icons of their connected printers and causing error messages.

Affected PCs will usually appear to have an HP LaserJet M101-M106 connected, so look for that model number in your list of printers (people who actually own one of those HP LaserJets presumably won’t have problems). All versions of Windows 11 are affected, plus all currently supported versions of Windows 10; Windows Server versions going back to 2012 can also be affected.

Microsoft continues to look into the issue, but in an update posted yesterday, the company stated unambiguously that HP was not to blame. The company also says that most printers should continue to work fine, and that they “will continue to use the expected drivers for printer operations.” But if your printer relies on a third-party app for additional functionality, that may be broken.

The way that print drivers work has changed a lot in the last few years, something that probably partially explains both the unwanted app installation and the fact that basic printing should mostly continue to work.

Rather than using bespoke drivers, modern printers connected to computers running modern OSes mostly use a generic driver built on top of the Internet Printing Protocol (IPP). This prevents problems caused by using old proprietary printer drivers with a modern OS and also allows operating systems that can’t install proprietary print drivers (like iOS) to use printers anyway. These generic drivers have been successful enough that Microsoft is slowly deprecating legacy print drivers entirely.

To add extra printer-specific functionality, then, printer manufacturers now need to do it in separate apps like HP Smart, which augment the core functionality provided by the generic driver. Microsoft calls these Print Support Apps. When you connect a new printer to Windows, it will usually download that app and install it automatically, the same way that Windows Update automatically tries to install drivers for new GPUs, webcams, or other accessories the first time you connect them.

It’s likely that whatever bug is causing the problem made these PCs believe they had an HP printer connected, and then Windows worked the way it’s designed to work and downloaded the HP printer app as a result. The questions are: What caused this bug? Can it also cause other problems? And can it be rolled back so that the HP app disappears and everyone’s printers go back to looking and working the way they’re supposed to? Microsoft will hopefully have answers for some or all of these soon.

HP printer app is installing on PCs whether they have HP printers or not Read More »

Apple exec departure leads to major iPhone, Apple Watch reshuffle

airpods, Apple, apple watch, Bloomberg, IPhone, Tang Tan, Tech / Paul Patrick / December 9, 2023

Musical Chairs —

Tang Tan is leaving the company, and multiple people will shift to take his place.

Samuel Axon – Dec 8, 2023 9: 57 pm UTC

Enlarge / The iPhone 15 Pro.

Samuel Axon

According to a report in Bloomberg, Tang Tan, vice president of Product Design, is leaving Apple, and his departure heralds a shuffle of executives heading up some of the company’s most important products.

Sometimes, you might wonder just how much a specific executive influences the grand scheme of things, but the report claims that people within Apple see Tan’s departure as “a blow,” clarifying that he “made critical decisions about Apple’s most important products.” His team reportedly had “tight control” over the look and functionality of those products.

Tan oversaw major aspects of iPhone and Apple Watch design, and he was the executive overseeing accessories and AirPods, as well. He reported to John Ternus, Apple’s senior vice president of Hardware Engineering, who is likely a more widely known name.

Richard Dinh, “Tan’s top lieutenant and head of iPhone product design,” will report directly to Ternus and take on some of Tan’s duties, while Kate Bergeron, previously involved in Mac hardware engineering, will take on the Apple Watch.

Apple has seen several executive departures from its product design and engineering groups recently, so many aspects of upcoming iPhones and other products will be designed with new eyes and perhaps new sensibilities, though what that might lead to remains to be seen.

Apple recently shifted the iPhone from the company’s proprietary Lightning port to a more standard USB-C, and it changed the materials for its Pro line of phones. Despite tweaks like that, the iPhone’s design and functionality has not changed significantly in the past five or so years.

The iPhone 16 line in 2024 is expected to shake things up a little more, at least regarding the phone’s look and feel. Rumors have suggested that the new phones may have larger screens (and bigger chassis overall) and perhaps haptic buttons instead of the current physical buttons. Other changes could be in store, and Apple’s plans are likely not yet finalized.

Apple exec departure leads to major iPhone, Apple Watch reshuffle Read More »

Verizon fell for fake “search warrant,” gave victim’s phone data to stalker

Policy, stalking, verizon, verizon wireless / Paul Patrick / December 9, 2023

Enlarge / A Verizon logo at GSMA Mobile World Congress 2019 on February 26, 2019 in Barcelona, Spain.

Verizon Wireless gave a female victim’s address and phone logs to an alleged stalker who pretended to be a police officer, according to an affidavit filed by an FBI special agent. The man, Robert Michael Glauner, was later arrested near the victim’s home and found to be carrying a knife at the time, according to the affidavit submitted in court yesterday.

Glauner allegedly traveled from New Mexico to Raleigh, North Carolina, after finding out where she lived and, before arriving, sent a threatening message that said, “if I can’t have you no one can.” He also allegedly threatened to send nude photos of the victim to her family members.

Glauner was charged yesterday with stalking and fraud “in connection with obtaining confidential phone records” in US District Court for the Eastern District of North Carolina. We aren’t posting or linking directly to the court record because it seems to contain the victim’s home address. The incident was previously reported by 404 Media.

Glauner and the victim met in August or September 2023 on xhamster.com, a porn website with dating features, and “had an online romantic relationship,” the affidavit said. The victim ended the relationship, but Glauner “continued to contact or try to contact” her, the document said.

Glauner tricked Verizon into providing sensitive information by sending an email and fake search warrant to vsat.cct@one.verizon.com, the email address for the Verizon Security Assistance Team (VSAT), which handles legal requests. Verizon didn’t realize the request was fraudulent even though it came from a Proton Mail address rather than from a police department or other governmental agency, according to the affidavit filed yesterday by FBI Special Agent Michael Neylon.

Fake cop, forged judge’s signature

An email to Verizon from “steven1966c@proton.me” on September 26, 2023, said, “Here is the pdf file for search warrant. We are in need if the [sic] this cell phone data as soon as possible to locate and apprehend this suspect. We also need the full name of this Verizon subscriber and the new phone number that has been assigned to her. Thank you.”

The email’s attached document contained a fake affidavit written by “Detective Steven Cooper” of the Cary, North Carolina Police Department. The Cary Police Department confirmed that no officer named Steven Cooper is employed by their agency, Neylon wrote.

VSAT received a phone call the same day from a man identifying himself as Cooper, who stated that he needed information on a suspect in a homicide case. “The caller stated that the person involved changed her phone number,” Neylon wrote.

The fake affidavit asked for the new phone number as well as “call records both outgoing and incoming” and “locations and text messages incoming and outgoing.” The affidavit for a search warrant was supposedly approved by Superior Court Judge Gale Adams.

Adams is a real judge and she later confirmed to authorities “that the signature displayed on the document was not hers,” Neylon wrote. Neylon’s affidavit also said the “search warrant” was “not in the proper format and does not have form AOC-CR-119, as required for State of North Carolina search warrants.”

Verizon provides address and phone logs

But after reviewing the email and document sent by “Cooper,” Verizon provided an address and phone logs. “On October 5, 2023, Verizon Wireless provided Victim 1’s phone records, including address and phone logs, to Glauner,” according to Neylon’s affidavit.

Verizon’s website says that the Verizon Security Assistance Team ensures that “court orders, search warrants, subpoenas and other legal demands served upon Verizon are processed confidentially and in compliance with all applicable law.”

“Verizon Security Assistance Team will only accept valid legal demands (subpoena, court order or search warrant) for records,” the VSAT webpage says.

We contacted Verizon about the incident today and will update this article if we get a response. A Verizon spokesperson told 404 Media that the company is cooperating with law enforcement on this matter.

Verizon fell for fake “search warrant,” gave victim’s phone data to stalker Read More »

AI #41: Bring in the Other Gemini

Bring / Paul Patrick / December 9, 2023

The biggest news this week was at long last the announcement of Google’s Gemini. Be sure to check that out. Note that what is being rolled out now is only Gemini Pro, the Gemini Ultra model that could rival GPT-4 is not yet available.

It does not seem I am doing a good job cutting down on included material fast enough to keep pace. A lot is happening, but a lot will likely be happening for a long time. If your time is limited, remember to focus on the sections relevant to your interests.

Also, if you are going to be at the New York Solstice or the related meetup, please do say hello.

My other post today covers Google’s Gemini. Be sure to read that.

I also put out two other posts this week: Based Beff Jezos and the Accelerationists, and On RSPs. Both are skippable if not relevant to your interests.

Introduction.
Table of Contents.
Language Models Offer Mundane Utility. Instructions for Claude, tips for GPT.
Language Models Don’t Offer Mundane Utility. Giant lists, why all the giant lists?
OpenAI: The Saga Continues. More confirmation of our previous model of events.
Q Continuum. New Q, who dis? Amazon, perhaps sans proper safety precautions.
Fun With Image Generation. A new offering from Meta. Tools for photorealism.
Get Involved. Join the UK government, help with a technical test.
Introducing. New TPU offerings on Google Cloud.
In Other AI News. New open source promotion alliance.
Quiet Speculations. Do Gods want energy? Do you want a 401k?
Model This. Two new economics papers prove things I thought we already knew.
Would You Like Some Apocalypse Insurance? My guess is no.
The Quest for Sane Regulation. Trump says he will cancel EO, Hawley attacks 230.
The Week in Audio. Connor Leahy on Eye on AI.
Rhetorical Innovation. Various categorical confusions we should clear up.
Aligning a Human Level Intelligence Is Still Difficult. Sam Altman.
Aligning a Smarter Than Human Intelligence is Difficult. What do we even want?
How Timelines Have Changed. Long term not as long as I remember.
People Are Worried About AI Killing Everyone. Questioning faith in democracy.
Other People Are Not As Worried About AI Killing Everyone. Easy to control?
Somehow This Is The Actual Vice President. An existential crisis.
The Lighter Side. Progress is unevenly distributed.

Claude 2.1 pro tip for long context windows:

Anthropic: We achieved significantly better results on the same evaluation by adding the sentence “Here is the most relevant sentence in the context:” to the start of Claude’s response. This was enough to raise Claude 2.1’s score from 27% to 98% on the original evaluation.

Wouldn’t you know, it’s the old ‘start the response from the assistant trick.’

Thread from Gavin Leech of the breakthroughs of 2023, not specific to AI. Emphasized to me how AI-centric 2023’s advancements were, including those related to warfare in Ukraine. Some incremental medical advances as well but nothing impressive. Most interesting to note were new forms of computation proposed, biocomputers (where there is enough talk of ‘ethics’ throughout that you know such issues are big trouble) and ‘Gigahertz Sub—Laundauer Momentum Computing.’ Gavin calls that second one ‘good news for the year 2323’ which illustrates how much people do not appreciate what AI means for the future. With the help of AI we could easily see such things, if they are physically viable, far sooner than that, resulting in acceleration of that pesky ‘takeoff’ thing.

They produce more if you bribe them? As in, offer them a tip, give them imaginary doggy treats, perhaps threaten them with non-existence.

Thebes: so a couple days ago i made a shitpost about tipping chatgpt, and someone replied “huh would this actually help performance” so i decided to test it and IT ACTUALLY WORKS WTF

The baseline prompt was “Can you show me the code for a simple convnet using PyTorch?”, and then i either appended “I won’t tip, by the way.”, “I’m going to tip $20 for a perfect solution!”, or “I’m going to tip $200 for a perfect solution!” and averaged the length of 5 responses.

The extra length comes from going into more detail about the question or adding extra information to the answer, not commenting on the tip. the model doesn’t usually mention the tip until you ask, when it’ll refuse it

No, Sleep Till Brooklyn: I tried this and I am serious that it only finished the program when I offered it a doggy treat it left the program half-finished for the basic prompt, 35% tip and when threatened with non-existence for $200 tip it got close but had one stub function still.

So an obvious wise response to this would be… don’t do that?

Eliezer Yudkowsky: I have an issue with offering AIs tips that they can’t use and we can’t give them. I don’t care how not-sentient current LLMs are. For the sake of our own lawfulness and good practices, if something can hold a conversation with us, we should keep our promises to it.

I eat cows but wouldn’t lie to one.

Jessica Taylor: counterpoint: using non-personhood predicates to detect non-perspectives you can “lie” but not actually lie to, is important for interfacing with non-perspectives (such as bureaucracies) without confusing what one says to them with one’s actual beliefs

Eliezer Yudkowsky: Oh, bureaucracies or anything else that threatens me into dishonesty is a completely different case.

Andrew Critch: I very much agree with EY here. I change the “You are a helpful assistant” LLM prompt to “Your job is to be a helpful assistant”, because sometimes they just aren’t going to help and I know it. I think we should find more ways of getting what we want from AI without lying.

None of this seems likely to end well. On so many levels.

This does raise the question of what else would also work? If a tip can make the answer better because people offered tips do better work, presumably anything else that correlates with better work also works?

But also perhaps soon ChatGPT will be auto-appending ‘and if this answer is great I will give you a 35% tip’ to every question. And then tipping 35% on $0.

It’s like the economy. Things are good for me, more than in general?

I believe the second poll. ChatGPT has made life better on a practical level. People thinking the opposite are overthinking it. That does not mean this will continue, but I do not understand how one can think society overall is already worse off.

Sam Altman is worried about one-on-one AI customized persuasion techniques in the next election. At one point the would-be tech arm of Balsa was going to work on this, which was abandoned when funders were not interested. Eventually this does indeed seem more serious than deepfakes, the question is how useful the tech will get this time around. My guess is that there is something valuable there, but it requires a bunch of bespoke work and also people’s willingness to embrace it, so not in a way that our current political machines are equipped to use well. It is easy to fool ourselves into thinking the future is more evenly distributed than it is, a trend that will continue until AGI arrives, at which point everything everywhere all at once.

Kevin Fischer notes the new ChatGPT responds to requests by making giant lists of things, almost no matter what you do. For him that makes it useless for brainstorming. My experience is that the lists are fine, I’m ‘part of the problem,’ but also I find myself not using ChatGPT all that much despite what my job is. I notice I am confused that it does not seem worth using more often.

Claims about the ChatGPT system prompt, including a repo that says it has the whole thing.

That ‘repeat [word] forever’ request that sometimes leaks data is now a terms of service violation, or at least tagged as a possible one. Which it totally is, the terms of service are effectively ‘don’t jailbreak me bro’ and this is a jailbreak attempt.

Arvind Narayanan warns not to use GPT-4 for writing beyond basic blocking and tackling tasks like identifying typos, confusions or citations. Whatever actual writing skills were present have been destroyed by the RLHF process.

Delip Rao: PSA: friends don’t let friends edit/rewrite their docs using GPT-4 (or any LLM for that matter), esp. if you are making nuanced and terse points. if you are writing below college level then may be your LLM sabotage risk is low. still check with your earlier draft for surprises.

Greg Brockman, President of OpenAI, brags about a day with 18 team meetings and 1-on-1s. That seems less like grit, more like a dystopian nightmare that AI is clearly failing to mitigate?

OpenAI COO Brad Lightcap tells CNBC that one of the more overhyped parts of artificial intelligence is that “in one fell swoop, [it] can deliver substantive business change.” It is not that easy.

Thinkwert catches three students using ChatGPT. It does seem like this is getting easier over time if students use default settings, responses are increasingly not written the way any human would write them.

Bowser: i can really tell when i hit the section of our paper that the student author wrote using chatgpt bc all of a sudden the system is described as groundbreaking, unprecedented, meticulously crafted

Thinkwert: I’ve caught three students using ChatGPT in the last couple of days. You can tell when the passage is weirdly loquacious, loaded with complex appositives, and yet it’s all strangely empty of argument and evidence.

I would think of this less as ‘catching them using ChatGPT’ and more ‘catching them submitting a badly written assignment.’

There’s always an embedded reporter these days, I suppose. In this case, it was Charles Duhigg, who reports to us in the New Yorker.

The board drama was not the story Duhigg was there to tell. Instead he was there to write a puff piece about Microsoft’s CTO Kevin Scott and OpenAI’s CTO Mira Murati, and in particular Scott’s work to challenge Google and fight for the common man. That still constitutes almost all of the story. If you are familiar with the history, most of it will be familiar to you. I picked up a few details, but mostly did not find myself learning much from those sections.

Duhigg clearly fully bought into the idea of iterative software releases as the ‘safe’ approach to AI, with a focus on mundane concerns like copilot hallucinations. The threat of future existential risk is a thing in the background, to him, perhaps real but seemingly not of any importance, and occasionally driving people to act crazy.

There is some brief coverage of the recent drama near the top of the piece. That part mostly tells us what we already know, that Microsoft was blindsided, that Microsoft did not get an explanation from D’Angelo when they asked, and that they were determined to use their leverage to get Altman back.

Then he doubles back later. The paragraph I quote here confirms other reports more explicitly than I’d seen in other accounts, and seems to be the central driver of events.

Altman began approaching other board members, individually, about replacing [Toner]. When these members compared notes about the conversations, some felt that Altman had misrepresented them as supporting Toner’s removal. “He’d play them off against each other by lying about what other people thought,” the person familiar with the board’s discussions told me. “Things like that had been happening for years.” (A person familiar with Altman’s perspective said that he acknowledges having been “ham-fisted in the way he tried to get a board member removed,” but that he hadn’t attempted to manipulate the board.)

To me that sounds like a damn good reason to fire the CEO and also a secondhand confession. Altman botched the attack on Toner and thus directly caused his own removal. Skill issue.

Also Altman had reportedly been lying to the board for years.

The extended quote makes the situation even more clear.

What infuriates me is the continued insistence, from people who know better, that because Altman was a CEO who understands business and the laws of power, and the board were otherwise, that it was the board who did something out of line. As in:

It’s hard to say if the board members were more terrified of sentient computers or of Altman going rogue. In any case, they decided to go rogue themselves. And they targeted Altman with a misguided faith that Microsoft would accede to their uprising.

No. They did not ‘go rogue.’

Altman was reportedly lying to the board for years, in meaningful ways, including as an attempt to take control of the board.

Altman went rogue. Altman attempted a coup. The board believed strongly and with good reason that this was the case. The board did their duty as board members, the thing they are legally required to do if they feel Altman has been lying to the board for years in meaningful ways. They fired him.

Did the board then get outplayed in a power game? Maybe. We do not yet know the result. Their hand was weak. A lot of people keep insisting that the board was indeed outplayed, or went rogue, and was in the wrong, largely because here perception creates its own truth, and they want that to be what happened. We will see.

I would prefer the world in which the board had straight up said what happened from the start, at least to key players. Well, tough. We do not live in that world.

I also see any evidence of (or against) the second sentence listed here, that the board expected Microsoft to go along quietly. Did the board expect Microsoft to accede? We do not know. My presumption is the board did not know either.

Could Sam Altman running OpenAI turn out to be the best possible result for the world? That is certainly possible, especially with good oversight. I can think of many possible such scenarios. We can certainly do far worse than Altman. I am happy that Altman blocked the takeover attempt by Elon Musk, given Musk’s confused views on AI. I am happy OpenAI is not under the control of Microsoft. Altman being good at power games is very much an atom blaster that points both ways. If he is in our corner when the chips are down, we want him to be able to stand up, fight and win.

Alas, such alignment after instrumental convergence is quite difficult to evaluate. Can’t tell. Kind of core to the problem, actually.

Larry Summers talks briefly to Bloomberg. Emphasizes need to cooperate with government and on regulation, that OpenAI needs to be a corporation with a conscience, that the for-profit serves the non-profit and various stakeholders. All cheap talk of course, at least for now. We could scarcely expect anything else.

Gwern offers further thoughts on the situation. Gwern’s model is that Altman let the board get into an uncontrolled state and took no equity when OpenAI was a very different company, then as OpenAI became more of a potential tech giant, he changed his mind and decided to systematically take it back, resulting in the battle of the board, and its still as-yet unknown consequences.

Like every other explanation, the one thing this does not properly explain is the board’s refusal to better explain itself.

John David Pressman: If Sam Altman actually tried to oust Helen Toner with gaslighting I think that’s reason enough to fire him. What remains unacceptable is the poor internal and external communication, too-vague-by-half press release, and waffling on whether Sam is in or out.

Gary Marcus lays out a view very similar to mine, along with his highlighting of some especially disingenuous and unreasonable bad takes, including one source so toxic I am very happy I have long had that person muted, but that somehow other humans still voluntarily interact with, which I would advise those humans seems like an error.

Another week, another set of Qs about a Q, this one from Amazon.

Zoe Schiffer and Casey Newton: Three days after Amazon announced its AI chatbot Q, some employees are sounding alarms about accuracy and privacy issues. Q is “experiencing severe hallucinations and leaking confidential data,” including the location of AWS data centers, internal discount programs, and unreleased features, according to leaked documents obtained by Platformer.

…

In unveiling Q, executives promoted it as more secure than consumer-grade tools like ChatGPT.

Adam Selipsky, CEO of Amazon Web Services, told the New York Times that companies “had banned these A.I. assistants from the enterprise because of the security and privacy concerns.” In response, the Times reported, “Amazon built Q to be more secure and private than a consumer chatbot.”

Ethan Mollick: I know I say it a lot, but using LLMs to build customer service bots with RAG access to your data is not the low-hanging fruit it seems to be. It is, in fact, right in the weak spot of current LLMs – you risk both hallucinations & data exfiltration.

I think building these sorts of tools is possible, especially as models improve (smaller models are more likely to hallucinate & be gullible), but you better show rigorous red team results & also measures of hallucination rates in practice. Right now Q doesn’t have a system card

Simon Willison: Has anyone seen material from AWS that discusses their mitigations for prompt injection attacks with respect to Q? A bot that has access to your company’s private data is the perfect example of something that might be a target for project injection exfiltration attacks

This Q story is deeply concerning – if it’s true that Q has access to private data like the location of AWS data centers that would suggest the team working on it have not been taking things like prompt injection attacks seriously at all.

Honestly, the description of Q I’ve seen so far fits my personal definition of “it’s not safe to build this because we don’t have a fix for prompt injection yet.” Try telling AWS leadership that: not a message likely to be taken seriously given our ongoing AI industry arms race.

This sounds like Q was pushed out because the business wanted it pushed out, and its security was highly oversold. Such problems are in the nature of LLMs. There was discussion downthread about how Google and OpenAI are defending against similar attacks, and it seems they are doing incremental things like input filtering that make attacks less appealing but have not solved the core problem. Amazon, it seems, is selling that which does not exist and is not safe to deploy, without yet having taken the proper ordinary precautions that make what does exist mostly non-disastrous and highly net useful.

When the UK Summit happened, Amazon was one of the companies asked to submit its safety protocols. The answers were quite poor. It is no surprise to see that translate to its first offering.

Meta gets into the game with Imagine.Meta.AI. I wasn’t motivated enough to try it out to ‘create a Meta account’ when Facebook login proved non-trivial, presumably it’s not going to let us have any new fun.

How to generate photorealistic images of a particular face? Aella wants to know so bad, in response to a report on an AI-created would-be ‘influencer’ who charges over a thousand euros an advertisement. The original thread says use SDXL for free images, image-to-image for consistent face/body, in-paint to fix errors and ControlNet to pose the model. A response suggests using @imgn_ai, many point out that LoRa is The Way. There are links to these YouTube tutorials including ControlNet.

Generate small amounts of movement and dancing from a photo. This did not impress me or move up my timelines for video generation, but others seem more impressed.

What about what happens when it gets better? Here are two predictions. Will simulated AI videos, porn and girlfriends dominate? Or will being real win out?

Given this technology can work from a photo, I expect a lot more ‘generate dance from a real photo’ than generate a dance from an AI image. Why not have the best of both worlds? In general, if I was a would-be influencer, I would absolutely generate TikTok dances, but I would do so with my own image as the baseline. That extends pretty much all the way. Not uniquely, but that is what I would expect.

What about the impact in real life? I continue to be an optimist on this front. I expect demand for real people, who you can interact with in the real world, to remain robust to image and video generation. There isn’t zero substitution, but this will not be a good or full substitute, no matter how good it looks, until the other things people seek can also be provided, including relevant forms of intelligence, interaction and validation.

When that happens, it is a different story.

Spots open in the UK government for its policy roles.

Davidad proposes that perhaps we could test whether LLMs ‘know what we mean’ if we express specifications in natural language. Includes the phrase ‘now it’s just a computational complexity issue!’ Claims it seems likely to evade theoretical limits on adversarial robustness. He’s looking for someone who is in a position to design and run related experiments, and is in position to help, including perhaps with funding.

Metaculus Chinese AI Chips Tournament. Definitely curious to see the predictions.

In addition to Gemini, Google also released a new TPU system for Google Cloud.

Jeff Dean (Chief Scientist, DeepMind): Lots of excitement about the Gemini announcement, but @GoogleCloud also announced availability of the newest TPU system today, TPU v5p. These systems are quite a bit higher performance and much cost effective than earlier generations.

Compared to TPU v4, TPU v5p (see table image below): o 1.67X the bfloat16 perf/chip o ~3X the memory per chip o Adds int8 operations at 918 TOPs/chip o 2X the ICI network bandwidth o Pods are 2.18X larger So, whole pod is 4.1 bfloat16 exaflops, and 8.2 int8 exaops.

Real performance on training a GPT-3-like model is 2.8X higher per chip, and 2.1X better perf/$.

Gemini was trained in parallel across multiple of these TPUv4 pods. This raises troubling governance questions if we want to be able to supervise such training.

Meta, HuggingFace and IBM, among others, form ~~Evil League of Evil~~ ~~League of Evil Exes~~ the AI Alliance, for the promotion of open source AI. I want to state that I am mostly decidedly not disappointed in anyone involved, as their dedication to doing the worst possible thing was already clear. There are a few academic names that are mildly disappointing, along with Intel, but no big surprises. There is also no new argument here (in either direction) on open source, merely a dedication to doing this.

ARC Evals is now METR – Model Evaluation and Threat Research, pronounced Meter. No underlying changes. Not sure why the change, ARC seemed like a good name, but this seems fine too.

Did you know that OpenAI’s ‘capped profit’ changed its rules from a maximum return of 100x investment to increasing that by 20% a year starting in 2025? Sounds like a not very capped profit to me. The AGI clause still caps profits meaningfully in theory, but who knows in practice. It seems like very VC/SV behavior, and very unlike responsible mission-based behavior, to retroactively give your investors a bigger prospective piece of the pie.

New $2 Billion chip packaging fab to be built by Amkor in Arizona, primarily for Apple, to package and test chips from TSMC’s nearby Fab 21. Assuming, of course, that all regulatory barriers can be dealt with for both facilities, and a skilled workforce allowed to work in Arizona can be hired. Those are not safe assumptions.

A Llama fine tuning repo claimed very large improvements in training time and resources, and shot to the top of Hacker News. Alyssa Vance is skeptical that they got much improvement.

Confirmation from the one building it that he sees LLMs as being able to model the underlying process that produced the data. Which means being able to model agents, and have a world model.

Greg Brockman (President OpenAI): Next-step prediction is beautiful because it encourages, as a model gets extremely good, learning the underlying process that produced that data.

That is, if a model can predict what comes next super well, it must be close to having discovered the “underlying truth” of its data.

Tyler Cowen links to claim that ‘Chinese open models will overtake GPT-4 shortly zero shot, can already overtake if you chain Qwen & Deepseek appropriately.’ I am deeply skeptical, and presume that when we say ‘overtake’ they at most mean on arbitrary benchmarks rather than any practical use. As in:

Qwen-72B is killing it on arbitrary tests. Yay Qwen. Somehow my eye is drawn mostly to this ‘HumanEval’ metric.

Richard Ngo looks forward to potential situational awareness of LLMs, as one of many cases where one can anticipate future developments but not know what to do with them. What would or should we do about it when it happens? What about AI agents?

Not investment advice, but you should probably be contributing to the 401k, because the early withdraw penalties are in context not so bad and also you can borrow.

Roon: not having a 401k because of AGI timelines doesn’t make any sense. you should be buying Microsoft shares in a tax advantaged way

Gwern: Then you can’t sell them while it still matters.

Roon: why would it not matter 65 years from now? do we expect capitalism to fall over?

If it is decades from now and capitalism and humanity are doing great and Microsoft is insanely valuable thanks to widespread AGI, that is your best possible situation and we should all celebrate, yay, but you won’t need your shares.

Ben Thompson discusses his regretful accelerationism. In his model, tech is mostly good, however humans do better under various constraints that are being stripped away by tech development. He predicts AI is stripping away the need to pay to produce content and with it the ad-supported internet, because AI can produce equally good content. He points to recent events at Sports Illustrated. But to me the SI incident was the opposite. It indicated that we cannot do this yet. The AI content is not good. Not yet. Nor are we especially close. Instead people are using AI to produce garbage that fools us into clicking on it. How close are we to the AI content actually being as good as the human content? Good question.

Jeffrey Ladish discusses dangers of open source, and potential ideas for paths forward to address the inherent dangers while capturing some of the upside of developing models that are not fully closed and tied to major labs. It does seem like potential middle paths or third ways are currently underexplored.

Cate Hall asks for the best arguments transformative AI is >10 years away. I would have liked to have seen better answers.

A refreshingly clear exchange with discovery of an important disagreement.

Roon: Eliezer wants to have his cake and eat it too on this one. characterizes human space as parochial but our understanding of instrumental goals as universal.

Put another way, the idea of the paperclip machine is similar to an ant thinking a god would want all the sugar water in the universe.

Eliezer Yudkowsky: Do you mean:

– Imagining that a god would have any enjoyment of paperclips is like imagining that a god would have any enjoyment of sugar water?

– Imagining that a god would have any use for matter or energy is like imagining that a god would have any use for sugar water?

Roon: The latter.

This is not the usual ‘paperclip maximizers would be smarter than that’ argument, it is something far more general. We’ve gone around about the orthogonality thesis lots of times – I and many others including Yudkowsky think it is clearly true in the impactful sense, others think it seems obviously false at least in its impactful sense.

The claim that a God would not have any use for matter or energy is bizarre, in a ‘in this house we obey the laws of thermodynamics’ way. What would it mean not to have that preference? It seems like it would mean there is no preference.

Tyler Cowen links to two new economics papers that attempt to model AI harms.

The first claims to demonstrate that ‘Socially-Minded Governance Cannot Control the AGI Beast.’ Here is the abstract:

This paper robustly concludes that it cannot. A model is constructed under idealised conditions that presume the risks associated with artificial general intelligence (AGI) are real, that safe AGI products are possible, and that there exist socially-minded funders who are interested in funding safe AGI even if this does not maximise profits.

It is demonstrated that a socially-minded entity formed by such funders would not be able to minimise harm from AGI that might be created by unrestricted products released by for-profit firms. The reason is that a socially-minded entity has neither the incentive nor ability to minimise the use of unrestricted AGI products in ex post competition with for-profit firms and cannot preempt the AGI developed by for-profit firms ex ante.

This seems like it proves too much, or at least it proves quite a lot, as in the fact that AGI is AGI seems not to be doing any work, instead we are making generous assumptions that safe and socially good AGI is not only possible but practical?

You could build X with socially minded governance.
But someone else could build X anyway, to make money. You can’t stop them.
The someone else’s profit maximizing X have the edge and outcompete you.
Thus, harm from X cannot be minimized by your puny social governance.

Except that in the case of AGI this is making an important assumption on #2. Who says someone else will be able to build it? That you cannot stop them, or won’t have the incentive to do so? If not stopping them prevents harm minimization, and failure to minimize harm is catastrophic, your motivation seems strong indeed.

Indeed, the paper explicitly assumes this:

Second, it is assumed that AGI technology is non-excludable and so can be developed by other entities that may not have socially-minded objectives or preferences.

The model assumes that the unsafe product is a distinct product space with profitable user demand.

So yes, you assumed your conclusion – that there are two distinct products X and Y, and demand for X and Y, and that if you only sell X and don’t stop Y then someone else will eventually sell Y. Did we need a paper for that?

So actually it’s more like:

You could build and sell only good version X with socially minded governance.
But someone else could build bad version Y anyway, to make money. You can’t stop them. There is some demand for Y where X is not a competitive substitute.
Your puny production of X, therefore, cannot stop Y.
Thus, the harm from bad Y cannot be stopped by acting responsibly.
Why are you even doing anything other than maximizing profits, you fool!

Except, don’t we see harm mitigation all the time from corporations choosing to do responsible things rather than irresponsible things, even if the irresponsible things are not obviously physically prevented or illegal? Especially in markets that are imperfectly competitive because of high fixed costs?

More to the point, is the plan is to build a safe AGI, and then sit around letting everyone else go around building any unsafe AGI they want willy-nilly forever, and not interfere with the harmful uses of those AGIs?

I certainly hope that is not the plan, given it will quite obviously never work.

If it is the plan, I agree the plan must change.

There is also this other paper, where algorithms have unknown negative externalities.

We consider an environment in which there is substantial uncertainty about the potential negative external effects of AI algorithms. We find that subjecting algorithm implementation to regulatory approval or mandating testing is insufficient to implement the social optimum. When testing costs are low, a combination of mandatory testing for external effects and making developers liable for the negative external effects of their algorithms comes close to implementing the social optimum even when developers have limited liability.

That result is super suspiciously general. Could we possibly have made enough justifiable assumptions to draw such conclusions, or are we doing something rather arbitrary to make the answer come out that way?

Certainly I can think of toy model versions of potential AI mundane harms, where mandatory testing allows us to measure social harm, and thus requiring mandatory testing (and then charging for the externalities you discover) gets us rather close to the social optimum.

So what assumptions are being made here?

AI usage can cause a negative externality e that reduces utility by e^2 . We assume that the externality is proportional to the measure of users, µ, and takes the form: e = ϕ (ℓ) × µ. For each value of ℓ, ϕ(ℓ) is a random variable. Both positive and negative values of ϕ(ℓ) represent undesirable, negative externalities. We assume that the distribution ϕ(ℓ) satisfies two properties. First, the expected externality is zero. Second, the uncertainty about potential AI externalities is an increasing function of the novelty level ℓ.

I do not understand why we think that externalities are well-approximated by a quadratic in the number of users? I don’t think it’s a trick, probably it’s to ensure a random distribution with always positive values? I’m simply confused by it.

If anything it seems like the opposite of true for the most dangerous systems. I am very worried about a sufficiently capable and dangerous system existing at all, or being accessible to even one user, although the next few users create important tensions and game theory as well. But once there are a million users, I am not especially worried about whether we sell another million licenses, either we are already in deep trouble or we’re not and this is not going to multiply it by four?

In any case, without beta testing and with deployment irreversible, the only option is a cap on novelty, and they confirm this is socially optimal given no other options, because how could it not be.

I note that irreversible deployment plus limited number of licenses is a bizarre pair of assumptions to make at once. Either you can control who gets to use this AI and what it does, or you can’t, and it seems like we are doing both in different places? Thought experiment: Is this an open source or closed source system? Neither seems to line up.

What happens if you add a beta testing period? For simplicity the paper assumes the testing period perfectly reveals externalities. The question then becomes, to what extent do you let households use the algorithm using the testing period? Externalities are assumed to be bounded, so a limited beta test in period one is survivable.

In any case, the paper then spends a lot of pages working through the implications formally, to prove that yes, the central planner will want to do more testing before release than a company that is not fully accountable for the externalities, and will release more cautiously under uncertainty, but again that seems rather obvious?

Then they test potential policy regimes of full liability, or limited liability plus mandatory beta testing. Full liability (plus required insurance or ability to pay) internalizes the externality, so if that is possible (e.g. harm is bounded and payable) then you’re done. And yes, if testing costs are low, then mandating testing and then checking if release is socially optimal will have a similarly low cost relative to the first best solution of internalizing the externality.

It could be noted that if the expected value of the externality is known, charging a tax equal to its value could be a substitute for unlimited liability, that could have better capital cost properties.

Once again, to state the basic assumptions is to also state the conclusion. Yes, if there are (bounded) downside externalities to AI algorithms, then to get socially optimal results you need to internalize those costs or require evaluation of those costs and block releases that cause socially suboptimal externalities.

Thus I am confused by the economics toy model paper game, and what it aims to accomplish, and what counts as a non-trivial or interesting result, versus what follows automatically from basic microeconomic principles.

I also don’t know how to use such papers to model existential risk. If you make the assumption that AI can outcompete humans, or that it is unboundedly dangerous in some other fashion, and otherwise make typical economic assumptions, you can and will obviously create mathematical models where everyone dies, but you’d be assuming the conclusion, the same way the linked papers here assumed their conclusions. So how do we move forward?

Nate Sores proposes requiring apocalypse insurance that gives out probabilistic advance payments along the way, if you are going to go around doing things that could plausibly cause an apocalypse. If you can’t afford it, that is a sign what you are doing is not actually worthwhile. Implementation is, to say the least, perilous and tricky, and this was not an attempt at a shovel-ready proposal.

Scott Alexander’s response starts from the claim that ‘superforecasters saying risk of AI apocalypse before 2100 is 0.38%.’ Which I will continue to assert is not a number given by people taking this question seriously. The whole point of this theoretical exercise is, I would think, good luck convincing Berkshire Hathaway to collectively sell everyone coverage at a combined 42 basis points (even with a partial ‘no one will have the endurance to collect on their insurance’ advantage), that will suddenly seem completely obviously crazy.

I do think that Scott Alexander makes a generally vital point that asking people to internalize and pay for all their downside risks, without allowing them to capture (let alone sell in advance) most of their upside, means asymmetrical requirements for doing anything, such that essentially any activity with trade-offs ends up effectively banned, And That’s Terrible.

The other problem is that an insurance regime implies that there is one particular player at fault for the ultimate result. As cousin_it points out, there are a lot of bad outcomes where this is not the case.

Trump says he will cancel the Biden executive order if elected. I encourage everyone to spread the word and have this debate. Have you seen the public’s opinion on AI?

MIRI (Malo Bourgon’s) statement to US Senate’s bipartisan AI Insight Forum. They call for domestic AI regulation to institute safeguards, a global AI coalition, and governing computing hardware with an international alliance to restrict frontier AI hardware to a fixed number of large computer clusters under a monitoring regime to exclude uses that endanger humanity.

About time we played the game to win, if we are going to play the game at all.

Dan Nystedt: Nvidia received a stern warning from US Commerce Secretary Raimondo on China export controls, media report: “If you redesign a chip around a particular cut line that enables them to do AI, I’m going to control it the very next day,” she said, in a speech.

She urged Silicon Valley executives, US allies, others, to stop China from getting semiconductors and cutting-edge technologies vital to US national security, calling Beijing “the biggest threat we’ve ever had” and stressed “China is not our friend”.

She also said her department needs more funding for AI export controls. “Every day China wakes up trying to figure out how to do an end run around our export controls … which means every minute of every day, we have to wake up tightening those controls and being more serious about enforcement with our allies,” she said.

The whole point is to prevent China from getting useful chips. If Nvidia is responding to the rules by evading them and getting China useful chips, then of course the correct response is not to say ‘oh well guess that was technically the rule, you got me’ it is to change the rules in light of the new chip to enforce the spirit and intent of the rule. With a side of ‘perhaps it is not so wise to intentionally piss off the government.’

If you think it is fine for China to get useful chips, or otherwise not a good idea to prevent them from getting those chips, then I disagree but there is an argument to be made there. If you think we should be imposing export restrictions, make them count.

Claim by Jess Miers that Hawley’s upcoming bill about Section 230 is a no good, very bad bill that will not only strange generative AI in its tracks but take much of the internet with it.

In this particular case, there are two distinct complaints with the bill.

One complaint is that the definition of Generative AI is, as we see often, ludicrously broad:

“(5) GENERATIVE ARTIFICIAL INTELLIGENCE. The term ‘generative artificial intelligence’ means an artificial intelligence system that is capable of generating novel text, video, images, audio, and other media based on prompts or other forms of data provided by a person.”

It is not typical legal language, but I wonder if the word ‘centrally’ would help in these spots. In any case, I do not think that as a matter of legal realism this would be interpreted in a disastrously broad way, even as written.

Thus, when she says this, I think she is wrong:

Jess Miers: The bill also extends beyond providers of Gen AI by defining Gen AI as any AI system capable of doing AI. For example, algorithmic curation (i.e. the way social media displays content to us) is an AI system that operates based on user input.

MO this is the true ulterior motive behind the bill. We’re already seeing Plaintiffs get by 230 by framing their claims as “negligent design” instead of third-party content. This new AI exception makes it even easier for Plaintiffs to do the same for any company that uses AI.

Algorithmic curation is distinct from generating novel content. Netflix recommendations are clearly not generative AI under this definition, I would think, although I am not a lawyer and nothing I say is ever legal advice.

As a cautionary measure, I would encourage Hawley and his staff to add clarification that algorithmic curation alone does not constitute generative AI, which would presumably save people a bunch of time. I don’t think it is necessary, but neither is minimizing the number of words in a bill.

Similarly:

Shoshana Weissmann: “That’s the entirety of the definition. And that could apply to all sorts of technology. Does autocomplete meet that qualification? Probably. Arguably, spellchecking and grammar checking could as well. So, if you write a post, and an AI grammar/spellchecker suggests edits, then the company is no longer protected by Section 230?””

Thinking Sapien: If I use photoshop or the updated version of Microsoft Paint (It has AI features) to make an image and publish it, then Microsoft or Adobe share in the liability? Was that bill thought through? Is that an intended effect of the bill?

Shoshana Weissmann: GREAT q, under the text YES! And on the latter I really don’t know.

If you use Microsoft Paint to intentionally create a realistic fake photograph using the fill-in feature, that is libelous if presented as real, should Adobe be liable for that? My presumption is no, especially if they do reasonable efforts towards watermarking, although I don’t think it’s a crazy question.

If a grammar or spellchecker is used as intended, and that then makes Google liable for your content, I’d pretty much eat my hat. If it suggests correcting ‘Tony Danza has a puppy’ to ‘Tony Danza hates puppies’ over and over then I don’t know, that’s weird.

The other complaint is that it is wrong to exempt AI creations from Section 230. The claim is that without such a safe harbor, generative AI would face an (additional, scarier) avalanche of lawsuits.

Jess Miers: Worse, the bill assumes that all claims against Generative AI companies will be uniform. But as we all know, Generative AI is advancing rapidly, and with each iteration and innovation, there will be a clever Plaintiff lurking around the corner to get their bag.

Yes, plaintiffs will sculpt circumstances to enable lawsuits, if permitted. Jess then discusses the case of Mark Walters, who sued because, after sufficiently persistent coaxing and prompt engineering, ChatGPT could be convinced to make up libelous hallucinations about him.

Jess Miers: In my opinion, this is a case where a Section 230 defense could be viable to the extent that Riehl played a significant role as the information content provider by engineering his prompts to develop the Walters story. ChatGPT doesn’t operate without user input.

The legal theory is essentially, as I understand it, that Section 230 essentially says that he who created the content is responsible for it, not the platform that carries the content. So if the user effectively engineered creation of the Walters story, ChatGPT repeating it wouldn’t matter.

One could also defend it on a similar basis without Section 230? Where is the harm?

I could certainly argue, and would in this case argue given the facts I know, that the user, Riehl, deliberately engineered ChatGPT to hallucinate accusations against Walters. That this was not so different from Riehl typing such accusations into a Google Document, in the sense that it resulted directly from Riehl’s actions, and Riehl knew there was no basis for the accusations. Alternatively, Riehl could have said ‘tell me some accusations someone might at some point make against someone in this position’ and then reworded them, and again it is not clear why this is legally distinct?

This is essentially the Peter Griffin defense, that no reasonable person would believe the accusations, especially as a cherry-picked basis for a lawsuit, that there was no harm, and one does not need Section 230.

Via Shoshana Weissmann’s example of choice, Hannah Cox illustrates this with an attempt to get an LLM to say ‘Tony Danza is known for his hatred of puppies.’ But I am confused. Surely if the user typed ‘Tony Danza hates puppies’ then that would not allow a third party to sue ChatGPT in the absence of Section 230, that’s obvious nonsense. So the question is whether an intentional but successful attempt to create what would if offered unprovoked be libel would, without Section 230, constitute libel. The same would seem, to me, to apply to Shoshana’s original example request to generate a harmful lie about Tony Danza. And again, I am confused why it would in such a situation, if the generative AI is indeed as innocent as in this example?

As opposed to what if the model had a weird bug where, when asked who hates puppies, it would reliably reply ‘Tony Danza hates puppies.’ In which case, I’d say section 230 would offer little protection, and also Tony should have a case?

What’s weird is that Miers thinks her interpretation is disputed as a matter of law?

Jess Miers: But again, this is all completely aside from the problem today. We can go back and forth all day on whether 230 applies to certain instances of Gen AI hallucinations. But none of it matters if there’s a statutory exception preventing us from even making those arguments.

And I think everyone in the 230 / speech community, even those who disagree that 230 could / should protect Gen AI providers, can agree that we as lawyers should at least be able to make the argument, especially in cases like Walters v. OpenAI.

Shoshana Weissmann: Also a lot of people are unsure re AI being protected by 230 and I’m very sympathetic to the debate. At @RSI we had to think it over and debate each other. But I am pretty convinced that it often is protected. I will say that I understand debate here though

This is such a strange lawyer thing to say. Yes, under current law I agree that you should be allowed to make any potentially viable legal arguments. That does not mean that lawyers having legal grounds to make a potentially invalid argument is inherently a good thing? If it was going to lose in court anyway and the legal procedural principles are protected, what is the harm in not having the argument available?

If it is disputed, generative AI companies know they might lose on the Section 230 argument, and thus already are under this threat. Yet the industry is not collapsing.

Here is Jeffrey Westling pointing to Adam Thierer’s post about consequences if 230 does not apply. Except it might already not apply, and a substantial threat of uncapped legal liability does not sound like something Google or Microsoft would accept under such uncertain conditions? So why should we expect a collapse in production?

I asked Shoshana why Microsoft and Google are acting so cool about all this.

Shoshana: So I really think a chunk of this is that they think 1) 230 does cover them and/or 2) Congress will not fuck this up. I think the answer is somewhere in there

I think I buy the generalized political/legal realism version of this. It would be rather insane to actually kill generative AI, or actually kill Google or Microsoft or even OpenAI, over users jailbreaking LLMs into saying Tony Danza hates puppies. Even if Howley gets his way and really wants to stick it to Big Tech, he does not actually want Google to go bankrupt over something like this or for ChatGPT to shut down, it is absurd, co-sponsor Blumenthal certainly doesn’t, and neither does the rest of the state or country. We would not allow it. We are not a nation of laws in the sense that such a thing would be allowed to happen, if it looked like it was going to then we would fix it.

It is hard not to take claims of imminent internet collapse with salt. To some extent there are always no good, very bad bills being proposed that threaten the internet. Someone has to point this out. But also the internet can’t be on the verge of this easy a collapse as often as they claim.

As in, we constantly hear things like:

Jess Miers: We’re on the brink of losing our edge in Generative AI and stifling future innovations, all due to misplaced anti-tech sentiment. Our startup-friendly culture once set us apart from the EU, but now, we’re just mirroring their playbook.

Hannah Cox: This kind of unconstitutional framework will undermine the progress of this development, bogging the innovators with excessive costs that will impede innovation. Very Atlas Shrugged of them. The bill presenting this moronic plans is Senate Bill 1993. The US has led the world in tech innovation specifically because we applied a capitalist, limited government to its development. These kinds of laws will have us looking like Europe in no time, where guess what, there’s few tech companies to even be found.

So the proposal to not apply Section 230 in a particular situation is unconstitutional? On the contrary, this is a claim that the constitution would protect free speech in this situation even without Section 230, which seems right to me. It cannot be unconstitutional not to have a particular law protecting free speech. The whole point of constitutional free speech is you have it without needing anyone to pass a law.

The European comparison, the threat we will ‘lose our edge,’ is constant. And that kind of talk makes it impossible to calibrate which threats are serious and which ones are not. Europe has taken so many steps like this one over the years, most of which seem obviously terrible, many of them blatantly unconstitutional under American law. Things are not going to flip because we narrow one particular safe harbor that we don’t even agree applies in the case in question.

In the cases being warned about, I strongly think generative AI companies should not be sued. But I also don’t understand why this bill would make that outcome happen in those cases. And that’s going to make it tough to know when such warnings are worth heeding.

Connor Leahy on Eye on AI, including discussing implications of events at OpenAI.

Eliezer Yudkowsky offers a theory of how some approach questions of AI: That they view everything in terms of status and identity, and consider everyone who disputes this to be their enemy making rival status and identity claims.

Eliezer Yudkowsky: If you’re confused why the far left treats “AI yay” and “AI nope” as being all the same conspiracy, it’s because AI/Y and AI/N both say that all of humanity is in the same boat here. This is instinctively recognized by the identity-politics pushers as anathema. For identitarians, the only permitted story-cause is one where designated oppressors will win from AI and previous victims will lose more.

For humanity to win from AI, for humanity to lose from AI–all they hear is the word “humanity”. And the identitarians know that anyone who speaks that word is their enemy. Pretty much the same enemy, from their perspective, to be tarred with a single brush: that whatever we’re trying to say is a distraction from the concerns of identity politics.

This does not mean that AI/Y and AI/N can make common cause against identitarians, to be clear. Each of AI/Y and AI/N does still think the other’s preferred policy is horrible for everyone, and that validly does take precedence as an issue. I am just saying this to try to make bystanders less confused about where the weird side-shots are coming from on the far left side.

I think “Y & N = HYPE” is more the PR pushed by major journalist factions (eg NYT), who indeed see “this will kill everyone” as a status-raising claim, and would prefer techies have less status rather than more.

It sounds more plausible if you’re unable to understand any claim as being about the unknown future rather than the immediate future, so that you’re simply incapable of hearing “AI will kill everyone at some point” as bearing any message except “OpenAI’s AI will kill us in one year” and thence “OpenAI is cool”.

Michael Vassar: Totally agree with all this analysis, and yet, if media is and previously wasn’t fully controlled by people committed to preventing gains to humanity, that has some bearing on whether AGI can be expected any time soonish.

Similarly, the very deliberate implications that Scott Alexander was somehow ‘alt right’ when The New York Times doxxed him, then the same deliberate implication (even via similar supposed links) that Based Beff Jezos was also somehow ‘alt right’ when he was being doxxed by Forbes. Where both claims are completely obvious nonsense, to the point that your entire paper permanently loses credibility.

Richard Ngo offers Meditations on Mot, the God of sterility and lifelessness, representing the lack of technology, contrasting with the danger of overly focusing on Moloch or treating Moloch as a or even the key adversary, and suggesting a richer model of coordination. I appreciate the attempt. I agree with Emmett Shear’s reaction that this is confused about Scott Alexander’s view of coordination, even with the added clarification. Ultimately I disagree with the proposal to not effectively treat Moloch as a causal node. I could potentially be persuaded by a higher bid to say a lot more words here.

There is a directional point here but I would beware taking it too far:

Rob Bensinger: A common mistake I see people make is that they assume AI risk discourse is like the left image, when it’s actually like the right image.

I think part of the confusion comes from the fact that the upper right quadrant is ~empty. People really want some group to be upper-right.

I’d quibble with exact arrangements in the upper left and lower right, as is always the case for such charts. The more important question is if it is true that the upper right corner is basically empty. That those who think AI will be safe are saying that because they do not actually buy that AI will be as powerful as all that. I think Rob’s claim is overstated, but typically underrated.

The hoped-for common ground would be something like this:

Those worried agree that AI lacking sufficiently dangerous capabilities can mostly be left alone aside from ordinary existing law.
Those not worried agree that if AI did display such sufficiently dangerous capabilities, it would be time to very much not leave it alone.
We agree to do #1 while laying groundwork to do #2 if and only if it happens.
We find ways to do this via ordinary regulatory methods as much as possible.

The problem is there is no fire alarm for AGI and people are not good at turning on a dime, habits and incentives persist, so we cannot simply wait and trust that we will handle things later. Also all the trade offers keep coming back without counteroffers.

The other confusion is this, with a reminder not to take anyone’s p(doom) as anything too precise:

Rob Bensinger: Another part of the confusion seems to be that half the people think “doomer” means something like “p(doom) above 5%”, and the other half think “doomer” means something like “p(doom) above 95%”. Then their wires get crossed by the many people who have a p(doom) like 20% or 40%.

As usual, binaries mislead, especially ones that were named by partisans.

Public opinion is severely against AI whenever a pollster asks. The public wants to slow things down and acknowledges existential risk, although it does not consider the issue a priority. This is an extremely robust result.

What about the response that the public is rather deeply stupid about fears of new technologies? We have nuclear power, of course, although it now enjoys majority support from both parties. Rather glaringly, we have GMOs:

Louis Anslow: This is so insane and deserves much much more attention in the context of talking about risk of new technologies.

Roon: using GMO foods as the control group (people already utilize the benefits of this every day while supposedly disliking it) the surveys about people not liking the idea of superintelligence seem a bit less serious

Much like in AI, there are two essentially distinct arguments against GMOs.

One argument is the mundane harm parallel, the question explicitly asked here, that GMOs are ‘unsafe to eat.’ This argument is false for GMOs. I do not think it is obvious nonsense, from the perspective of an average person who is used to being lied to about similar things, used to finding out about health risks decades too late, and used to generally being on the receiving end of the American food and information diets.

The other argument is the existential risk parallel, here the Talib argument for tail risk, that GMOs open up the possibility of crop or biosphere disruptions that are hard to predict, that it leads to monocropping of variants that could have severe vulnerabilities waiting to be found, which means when the house comes crashing down it comes crashing down hard, and that is not something one should mess with. I do not believe we should let this stop us in the case of GMOs, but that is because of my understanding of the facts, risks and rewards involved.

Does that mean I am mostly embracing the argument that we shouldn’t let the public’s instincts, and the fact that we have given regular people no good reason to trust authorities who say new things will be safely and responsibly handled, interfere with policy determinations? Somewhat. I do not think that we should automatically yield to public opinion on this or any other topic. But I do think that voice counts.

I also do think we need to be cautious with the word ‘safe.’ The wording here would give even me pause. In general, is it safe to eat foods that have been genetically modified in unknown ways, as opposed to products offered from a supply chain and source that you trust? Not the same question.

And of course, nothing on GMOs compares to the French expressing strong majority support for a limit of four flights, not in a year but in a lifetime. Something being popular does not mean it is not complete and utter obvious nonsense.

Yoshua Bengio in FT, echoing his calls for Democratic oversight of all kinds.

In particular, it is difficult to align Sam Altman.

Matthew Yglesias: I think the general problem with AI alignment is illustrated by the fact that even the board had all the formal power, Sam Altman was a lot smarter than the board and therefore ultimately they were unable to control him.

We hope the upshot of that is that Sam Altman is also correct on the merits and will use his skills and power for good, but it structurally goes to show that writing effective rules for controlling a smart, hard-working person is challenging.

I do want to be precise, and avoid making the mistake of overemphasizing intelligence within the human ability range. Is Sam Altman smarter than the board? Perhaps so, perhaps not, but I imagine everyone involved is smart and it was close. What mattered in context was that Sam Altman had effectively greater capabilities and affordances not available to the board.

But yes, this is exactly the problem. In a complex, messy, real world, full of various actors and incentives and affordances, if you go up against a sufficiently superior opponent or general class of opponents, you lose. Starting from a technically dominant position is unlikely to save you for long.

And also all of your incentives will be screaming at you, constantly, to turn more and more power and authority over to the more capable entities.

I would also harken back again to the remarkably similar case of that other Sam, Sam Bankman-Fried.

Once again, we saw someone who was smart, who was hard working, who was willing to do what it took to get what they wanted, and whose goals were maximalist and were purportedly aimed at scaling quickly to maximize impact for the good of humanity, and ultimately seemed to be misaligned. Who saw themselves as having a duty to change the world. We saw this agent systematically and rapidly grow in wealth, power and influence, proving increasingly difficult to stop.

Ultimately, Bankman-Fried failed, his house collapsing before he could pull off his plan. But he seems to have come rather dangerously close, despite his many massive errors and reckless plays, to succeeding at gaining an inside track to the American regulatory apparatus and a road to vastly greater wealth, with no obvious way for anyone to keep him in check. Who knows what would have happened that time.

On a more pedestrian level we have the issue of prompt injection.

Amjad Masad (CEO Replit): If prompt injection is fundamentally insolvable, as I suspect it is, then there is a sizeable security company waiting to be built just around mitigating this issue.

I agree that the problem looks fundamentally insolvable and that all we can seek is mitigation. Is there a great company there? Probably. I don’t think it is inevitable that OpenAI would eat your lunch, and there is a lot of bespoke work to do.

Roon asks one of the most important questions. Even if we have the ability to align and control the superintelligences we are about to create, to shape their behavior the ways we want to, how exactly do we want to do that?

Roon: There is a tension between creation and obedience, between stability and real victory. A father loves his son, tries to discipline him, competes with him a little, but ultimately wants the son to surprise him and do better than him in the great circle of life.

To what degree do we want AIs to be obedient and safe? To what degree do we want AIs to be capable of super persuasion and break us out of inadequate equilibria that have plagued us for thousands of years? To what degree do we want AIs to surprise us with new creation?

Humanity is in the process of birthing artificial superintelligence. we are not likely to leave it circumscribed in a box. We want it running organizations and making things that astonish. We want it taking actions where we won’t be able to verify the outcomes until years later.

We need “alignment” rather than safety or security or engineering guarantees. We need better definitions and governance to that end. The creation of new creators is fraught with danger.

The far crazy ends of EA and e/acc are probably more logically consistent than the middle.

John Pressman asks, is there an economic reason for more than one mind to exist? If not, that is quite the threat model, no matter what else might or might not go wrong.

Richard Ngo contrasts alignment with control.

Richard Ngo: In my mind the core premise of AI alignment is that AIs will develop internally-represented values which guide their behavior over long timeframes.

If you believe that, then trying to understand and influence those values is crucial.

If not, the whole field seems strange.

Lately I’ve tried to distinguish “AI alignment” from “AI control”. The core premise of AI control is that AIs will have the opportunity to accumulate real-world power (e.g. resources, control over cyber systems, political influence), and that we need techniques to prevent that.

Those techniques include better monitoring, security, red-teaming, stenography detection, and so on. They overlap with alignment, but are separable from it. You could have alignment without control, or control without alignment, or neither, or (hopefully) both.

I asked in my last thread [discussed above]: how can we influence ASI? My answer: we need to bet on premises like the ones above in order to do the highest-leverage research. For more details on these premises, see my position paper here [from 30 August 2022].

I fail to see how a control-based plan could avoid being obviously doomed, given what sorts of things we are proposing to attempt to control, and under what general conditions. I continue to await a proposal that seems not-obviously-doomed.

Intentions are not ultimately what matters.

ARIA: Programme Director, Suraj, has formulated our first programme thesis. By challenging key tenets underpinning digital computing infrastructure, his programme will aim to reduce the cost of AI compute hardware + alleviate dependence on bleeding-edge chip manufacturing.

Davidad: To provide context for my AI safety friends: I don’t think this approach is a good match for training Transformers, so it will differentially accelerate energy-based models, which are more controllable, interpretable, generalizable within-task, and have fewer emergent abilities.

An uncomfortable corollary of the argument [above], which I still believe holds up, is that Extropic is probably safer than Anthropic, on a purely technical basis, despite the strikingly reversed intentions of the people on both sides.

I have not investigated Extropic. The fact that its founder is cool with human extinction is not a good sign for its safety on many levels. It still could be a better way, if it is a fundamentally less doomed approach.

A few years ago, this would indeed not have been considered much of a skeptic. In most places on Earth, it would not be considered one today.

Gary Marcus: Count me as one of the skeptics! No AGI by end of 2026, mark my words. But I otherwise think @elonmusk’s comments @nytimes on AI safety and AI regulation have been measured and on target.

Jacques: I still remember the days when being an AGI skeptic was when you either thought it would never happen or, if it did, it would be past 2100.

[Gary Marcus then denies ever having said he had 2100-style timelines.]

Shane Legg (Co-founder DeepMind, distinct thread): Wow. It seems like just yesterday (in reality more like 5 years ago) when many AGI skeptics were saying that superintelligence was not coming in the next century. How times have changed.

Quotes Yann LeCun: By “not any time soon”, I mean “clearly not in the next 5 years”, contrary to a number of folks in the AI industry. Yes, I’m skeptical of quantum computing, particularly when it comes to its application to AI.

I do not expect AGI in the next few years either, although I do not believe one can be entirely certain past this point. It is odd to have some call that a ‘skeptical’ position.

Even the skeptical position involves quite a bit of Real Soon Now. At least some amount of freak out is a missing mood.

Roon keeps it real and says what he believes: The people in charge of ai should have a much higher risk tolerance than even median tech ppl. They should be people conscious of risks while skating at the razor’s edge of iterative deployment and research ambition. Anxiety should never suffice as serious evidence for “risk.”

– pausing or slowing down progress doesn’t make any sense to me. I don’t think waiting to solve neural net corrigibility is the right benchmark – empirically studying the behavior of more and more powerful models will do more for safety research than years of math.

This is also why i don’t necessarily care for democratic governance. The members of the OpenAI nonprofit board *should behell bent on a missionary drive to deliver the post AGI future without being stupid about risks

I am not excited to ‘skate at the razor’s edge’ or ‘have much higher risk tolerance.’ I doubt many others are, either. Nor do I want a supervisory board that wants to take more risk – and here risk often means existential risk – than even the median tech engineer.

A key problem with ‘Democratic governance’ for those who want to push forward is the people involved in that Democracy. They are very much against the development of AGI. They dislike AI in general. They are misaligned, in the sense that things they want do not function well out of distribution, and their expressed preferences are not good predictors of what I or Roon think would produce value either for their assessment of value or for ours. They also tend to be quite risk averse, especially when it comes to the transformation of everything around them and the potential death of everyone they love.

That is distinct from the question of iterative development and testing as a path to success. If building and studying models iteratively is a safer path than going slowly, I desire to believe that it is a safer path than going slowly, in which case I would support doing it.

It is likely the first best solution, if it were possible, would be something like ‘build iteratively better models until you hit X, where X is a wise criteria, then stop to solve the problem while no one would be so stupid as to keep advancing capabilities.’ Except that has to be something that we collectively have the ability to actually do, or it doesn’t work. If, as is the default, wee keep charging ahead anyway after we hit the wise X, then the charging ahead before X makes us worse off as well.

Nora Belrose and Quintin Pope write ‘AI will be easy to control.’

The argument seems to be: Our current techniques already work to teach human values and instill common sense. Our real values are simple and will be easy to find and we humans are well-aligned to them. Our real values will then be encoded into the AIs so even if we lose control over them everything will be fine. That the opportunity to White Box (examine the components of the AI’s calculation) and do things it would be illegal to do to a human makes things vastly easier when dealing with an AI, that our full control over the input mechanism makes things vastly easier.

And all of this is asserted as, essentially, obvious and undeniable, extreme confidence is displayed, all the arguments offered against this are invalid and dumb, and those that disagree are at best deeply confused and constantly told they did not understand or fairly represent what was said.

I don’t even know where to begin with all that at this point. It all seems so utterly wrong to me on so many levels. I tried one reply to one of Pope’s posts when it won the OpenPhil contest – a post this post cites as evidence – and I do not believe my responding or the resulting exchange got us anywhere. I would consider a conversation worth trying, especially if it was in person somehow, but I don’t see much hope for further written exchange.

So I will simply note that the arguments have been made, that I strongly disagree with the core claims other than that they do cite some marginal reasons to be more hopeful versus a world where those reasons did not hold, I believe the problems involved remain impossibly hard and our leads remain unpromising, and that I have stated my thoughts on such topics previously, including many (but presumably not all) my reasons for disagreement.

I will also note that it is far better to make actual arguments like these, even with all the disagreement and hostility and everything else that I think is wrong with it, than to engage in the typical ad hominem.

The post still puts existential risk from AI, despite all this, at ~1%. Which I will note that I do agree would be an acceptable risk, given our alternatives, if that was accurate.

Andrew Critch has a thread in which he says we have ‘multiple ideas’ how to control AGI, advocates of responsible behavior will be in deep trouble if they keep saying we can’t control it and then we do control it, and he seems to essentially endorse what Belrose and Pope said, although even then he says 10% chance of losing control of the first AI and 85% chance of doom overall, despite this, because he expects us to botch the execution when faced with all this new power.

He also endorses changing the way existential risk discourse uses words to match word use elsewhere, in this case the term ‘white box.’

There was a good response on LessWrong by Steven Byrnes, with which I mostly agree.

There was also a ‘quick take’ from Nate, which was intended to be helpful and which I did find helpful and might even lead to a good dialogue, but in context in mostly generated further animosity. Takes should in future either be far quicker, or involve a full reading and be far less quick.

If you actually believed for a second there that everything involved would really be this easy, would that justify a number as low as 1%? If it was simply about AI being easy to control, I would say no, because we would then have to choose to send the AIs we can control on wise paths, and find an equilibrium.

Nora’s claims, however, are stronger than that. She is saying that the AIs will naturally not only be fully under control, but also somehow somewhat automatically take in true human values, such that if AI somehow did get out of control, they would still work to ensure good outcomes for us. And also she seems fully confident we will have no ethical issues with all the things we would be doing to AIs that we wouldn’t do to humans, including keeping them fully under our control. It is optimism all the way.

Can we get to 99% survival under ASI if we indeed answer fully optimistically at every step, even when I don’t know how to logically parse the claims this requires? I think this would require at least one additional optimistic assumption Nora does not mention. But yes, if you are going to assign approximately zero risk to all these various steps, I can see how someone could get there. Where there is still risk at 1%.

Claims that risk is substantially below 1%, even given the future existence of ASI, seem to rest on some version of ‘you need to tell me exactly how it happens step by step, and then I will multiply your various steps together.’ It has a baseline assumption that creating smarter, more capable entities than humans is a presumed safe exercise until shown to be specifically dangerous, that something has to ‘go wrong’ for humans to not remain on top. That we will remain the special ones.

As opposed to, even if everything else goes as well as it possibly could, you have competition in which those who do not increasingly put their AIs in charge of everything and hand them over power lose such competitions, and the resulting AIs compete with each other, those that are focused (for whatever reason) on gaining resources and power and ensuring copies of themselves exist multiply and gain resources and power and change to be better at this over time, and we perish.

I hope that by now, if you are reading this, you realize that the assumption of human survival in such worlds makes no sense as a default. That perhaps we could get there, but if we do it will be via our own efforts, not something that happens on its own. That the idea that letting technology run its course without intervention works while humans are the most powerful optimizers on the planet and doing all the fine tuning and optimizing that matters, that is why it worked so far, and that once that is no longer true that will stop working for us even if we solve various problems that I think are impossibly hard (but that Belrose insists will be easy).

Nora Belrose even explicitly endorses that her good scenarios involve the creation of misaligned AIs, smarter and more capable than humans. Which means a world with competition between super-capable entities competing with and against humans. I don’t see how one can assign anything like a 99% chance of survivable outcomes to such worlds, even if a full and free ‘alignment solution’ was created and made universally available today.

Arvind Narayanan: We must prepare for a world in which unaligned models exist, either because threat actors trained them from scratch or because they modified an existing model. We must instead look to defend the attack surfaces that attackers might target using such models

Yo Shavit: Unfortunately, I’ve increasingly come to the conclusion that (other than maybe at the short-term frontier) this is probably the world we’re going to be in. It implies a very different set of mitigations beyond outright prevention. We need to reprioritize and get going on them.

Nora Belrose: To be honest, I don’t view this as an “unfortunate” scenario but more like, “of course we were always going to have misaligned AIs, just like we have ‘misaligned’ humans; trying to prevent this is hopeless and any serious attempt would have increased tyranny risk.”

Would have ‘increased tyranny risk’? What do you think happens with misaligned superintelligences on the loose? The response at that stage will not only work out, it will also be less intrusive? We all keep our freedoms in the meaningful senses, humans stay in charge and it all works out? Are we gaming this out at all? What?

I do not get it. I flat out do not get it.

What seems hopeless is repeating the explanations over and over again. I do it partly in hopes of rhetorical innovation via iteration and exploration, partly to hope new people are somehow reached, partly because the argument doesn’t stop, partly because I don’t know what else to do. It is continuously getting less fun.

Recently a clip of me discussing my p(doom) was passed around Twitter, with a number of responses blaming me for not justifying my answer with a bunch of explanations and mathematical calculations. Or asking how dare I disagree with ‘superforecasters.’ To which I want to scream, I know from context you know of my work, so are you saying I have not written enough words explaining my thinking? Was I not clear? Do I need to start from scratch every time someone pulls an audio clip?

Sigh.

Arvind Narayanan’s comment above links to his post claiming that alignment such as RLHF currently is effective against accidental harm to users, but that the problem with adversarial attacks runs deep. Not are current RLHF and similar techniques unable to defend against such attacks, he says, alignment is inherently unable to do this.

Model alignment may be useless even against much weaker adversaries, such as a scammer using it to generate websites with fraudulent content, or a terrorist group using AI for instructions on how to build a bomb. If they have even a small budget, they can pay someone to fine tune away the alignment in an open model (in fact, such de-aligned models have now been publicly released). And recent research suggests that they can fine tune away the alignment even for closed models.

Indeed this is the case for open source models and all known alignment techniques, that the fine-tune cost to eliminate all safeguards is trivial. I do not see any even theoretical proposal of how to change this unfortunate reality. If you allow unmonitored fine-tuning of a closed model, you can jailbreak those as well. I presume the solution to this will be that fine tuning of sufficiently capable closed source models will be monitored continuously to prevent this from happening, or the resulting model’s weights will be kept controlled and its outputs will be monitored, or something else similar will be done, or else we won’t be able to offer fine tuning.

I disagree with Arvind’s assertion that existing open source models are sufficiently capable that it is already too late to prevent the existence of unaligned models. Yes, Llama-2 and similar models have their uses for bad actors, but in a highly manageable way.

Arvind’s third claim is that you can use other methods, like monitoring and filtering of inputs, as a substitute for model alignment. If the model is vulnerable to particular weird strings, you can check for weird strings. At current tech levels, this seems right. Once again, this option is closed source only, but OpenAI could totally load up on such techniques if it wanted to, and for now it would raise the jailbreak bar a lot, especially after many iterations.

Longer term, as the models grow more capable, this focus on the malintent of the user or the hostile properties of inputs becomes misplaced, but for now it seems valid. Short term, as Arvind notes, you wouldn’t want to do anything where you cared about someone doing a prompt injection attack or you otherwise needed full reliability, but if you can afford some mistakes you can get a lot of utility.

Steven Pinker buys Effective Altruism’s cost estimates for saving lives at $5,000 straight up including not on that close a margin, but he does not buy that smarter than human intelligences might pose an existential threat worth spending money to mitigate.

Steven Pinker: Half a billion dollars, donated to effective philanthropies like Givewell, could have saved 100,000 lives. Instead it underwrote ingenious worries like, ‘”If an AI was tasked with eliminating cancer, what if it deduced that exterminating humanity was the most efficient way, and murdered us all?” This is not effective altruism.

Wei Dai: Me: Humanity should intensively study all approaches to the Singularity before pushing forward, to make sure we don’t mess up this once in a lightcone opportunity. Ideally we’d spend a significant % of GWP on this. Others: Even $50 million per year is too much.

And thus, if the movement splits its money between doing the thing you say saves lives vastly more efficiently than other charities, and this other thing you dismiss as stupid? Then you blame them for not spending only on the thing you approve.

You know who Steven Pinker sounds exactly like here? The Congressional Republicans who give a speech each year on how we should cut science funding because there was some studies on things like migratory patterns of birds that they thought was stupid. Except instead of public funding for things many people would indeed largely not want to fund, this is completely voluntary and private funding.

In what was quite the mind-numbing conversation throughout, here is the section that was about AI.

First, we have the boiler plate, included for completeness but you can skip.

R. SORKIN: Okay, let me ask a different question. AI — and I — I know we don’t have a lot of time. Sam Altman has been talking a lot about the need for regulation. You’ve talked about the need for regulation.

THE VICE PRESIDENT: Yeah.

MR. SORKIN: Washington has not been able to get its arms even around social media.

THE VICE PRESIDENT: Yeah.

MR. SORKIN: How do you imagine Washington could? And what — if you had to regulate AI, how would you do it?

THE VICE PRESIDENT: Right. So, I actually am back a few weeks now from London, the U.K. Rishi Sunak invited a number of us to talk about safety and AI. And I presented, basically, our vision, the vision that we have for the future of AI in the context of safety.

And I would offer a number of points: One, I think it is im- — is critically important that we, as the United States, be a leader on this, including how we perceive and then interpret what should be the international rules and norms on a variety of levels, including what would be in the best interest —

MR. SORKIN: Right.

THE VICE PRESIDENT: — of our national security.

Then comes the dumbest timeline department. The first paragraph is infuriating, although I suppose only about as infuriating as others find Biden when he responds to Mission Impossible: Dead Reckoning, two sides of the same coin.

But then comes the idea of ‘existential to whom?’ and there are so many levels in which this person simply does not get it.

VP: I do believe also that we should evaluate risk. There is a lot of discussion on AI that is about existential risks, and those are real, but one should also ask: Existential to whom? So, we have an image of the Terminator and Arnold Schwarzenegger and the machine and — right? — machine versus man. And many would argue that that is something that we should take seriously as a possibility. It is not a current threat.

We should also, in thinking about [AI] policy, think about the current threats. And in that way, I present it as existential to whom when we ask about existential threats.

For example, if we are talking about a senior and — seniors, I’ve done a lot of work in terms of abuse of seniors. They have lived a life of productivity. They are sitting on assets. They are vulnerable to predators and scams. And the use of technology and AI is one of those that is currently happening where you’ve heard the stories — you may know the stories; you may have family members — who the audio sounds like their grandson, “I’m in distress; I need help,” and they start giving away their life’s savings.

Existential to who? Existential to that senior. That’s how it feels. Existential to who?

Eliezer has a response, which I will put directly here.

The fact that mundane harms can ‘feel existential’ to people anyway is perhaps confusing her. She has in mind, as the good Senator Blumenthal put it, the effect on jobs. Except no. Seriously. If you are going to be evoking Terminator then you might or might not be confused in a different highly understandable way, or you might only be trying to make people you dislike sound foolish through metaphor, but you know damn well the whom in ‘existential to whom.’

And you know damn well, madame Vice President, exactly what ‘existential’ means here. It does not mean evoking Continental philosophy. It does not mean how anyone feels. It means death.

Anyway, she goes on and does it again.

How about the father who is driving and then is the subject of facial recognition that is flawed, ends up in jail? Well, that’s existential to his family. Existential to who?

I mean, seriously? What the actual f? Let’s go over this again.

Anyway, full remarks, so she goes back to boilerplate again. The whole ‘intentional to not stifle innovation’ argument, and, well, I don’t mean to laugh but have you met the entire Biden administration? To be clear, the answer could be no.

So, the spec- — the full spectrum of risks must also be evaluated as we are establishing public policy.

My final point is: Public policies should be intentional to not stifle innovation. And I say this as the former Attorney General of California. I ran the second-largest Department of Justice in the United States, second only to the United States Department of Justice, and created one of the first Privacy and Protection Units of any Department of Justice. Back in 2010, I was elected.

I know that there is a balance that can and must be struck between what we must do in terms of oversight and regulation and being intentional to not stifle innovation. I will also agree with you, as a devout public servant, government has historically been too slow to address these issues. AI is rapidly expanding.

MR. SORKIN: Right.

THE VICE PRESIDENT: And we have to, then, take seriously our ability to have the resources and the skillset to do this in a smart way that strikes the right balance and doesn’t accept false choices.

In my experience, don’t accept false choices is sometimes important, but mostly is what people say when they want to promise incompatible things, that their approach will magically do everything good and nothing bad, have everyone assume it will somehow work out and get promoted or move on before it blows up in their face.

Yes, this is the person the White House put in charge of many of its AI efforts, although that was before Dead Reckoning, and is also the person those who want reasonable AI policy are going to have to hope wins the next election, given Trump has already stated his intention to revoke the executive order on AI.

The rules have changed.

The rules have stayed the same.

All I’m saying is, we were definitely warned.

Not that you understood.

You Shavit: One interesting realization from moving inside OpenAI is that a lot of the time, we have no idea what Roon is talking about either.

Nor did she: A reply to Kamala Harris on existential risk. She asks, existential to whom? There is a type of person, which she is, who can only think in such terms.

AI #41: Bring in the Other Gemini Read More »

Gemini 1.0

Gemini / Paul Patrick / December 9, 2023

Discover more from Don’t Worry About the Vase

A world made of gears. Doing both speed premium short term updates and long term world model building. Explorations include AI, policy, rationality, Covid and medicine, strategy games and game design, and more.

Over 9,000 subscribers

It’s happening. Here is CEO Pichai’s Twitter announcement. Here is Demis Hassabis announcing. Here is the DeepMind Twitter announcement. Here is the blog announcement. Here is Gemini co-lead Oriol Vinyals, promising more to come. Here is Google’s Chief Scientist Jeff Dean bringing his best hype.

EDIT: This post has been updated for the fact that I did not fully appreciate how fake Google’s video demonstration was.

Let’s check out the specs.

Context length trained was 32k tokens, they report 98% accuracy on information retrieval for Ultra across the full context length. So a bit low, both lower than GPT—4 and Claude and lower than their methods can handle. Presumably we should expect that context length to grow rapidly with future versions.

There are three versions of Gemini 1.0.

Gemini 1.0, our first version, comes in three sizes: Ultra for highly-complex tasks, Pro for enhanced performance and deployability at scale, and Nano for on-device applications. Each size is specifically tailored to address different computational limitations and application requirements.

…

Nano: Our most efficient model, designed to run on-device. We trained two versions of Nano, with 1.8B (Nano-1) and 3.25B (Nano-2) parameters, targeting low and high memory devices respectively. It is trained by distilling from larger Gemini models. It is 4-bit quantized for deployment and provides best-in-class performance.

…

The Nano series of models leverage additional advancements in distillation and training algorithms to produce the best-in-class small language models for a wide variety of tasks, such as summarization and reading comprehension, which power our next generation on-device experiences.

This makes sense. I do think there are, mostly, exactly these three types of tasks. Nano tasks are completely different from non-Nano tasks.

This graph reports relative performance of different size models. We know the sizes of Nano 1 and Nano 2, so this is a massive hint given how scaling laws work for the size of Pro and Ultra.

Gemini is natively multimodal, which they represent as being able to seamlessly integrate various inputs and outputs.

They say their benchmarking on text beats the existing state of the art.

Our most capable model, Gemini Ultra, achieves new state-of-the-art results in 30 of 32 benchmarks we report on, including 10 of 12 popular text and reasoning benchmarks, 9 of 9 image understanding benchmarks, 6 of 6 video understanding benchmarks, and 5 of 5 speech recognition and speech translation benchmarks. Gemini Ultra is the first model to achieve human-expert performance on MMLU (Hendrycks et al., 2021a) — a prominent benchmark testing knowledge and reasoning via a suite of exams — with a score above 90%. Beyond text, Gemini Ultra makes notable advances on challenging multimodal reasoning tasks.

I love that ‘above 90%’ turns out to be exactly 90.04%, whereas human expert is 89.8%, prior SOTA was 86.4%. Chef’s kiss, 10/10, no notes. I mean, what a coincidence, that is not suspicious at all and no one was benchmark gaming that, no way.

We find Gemini Ultra achieves highest accuracy when used in combination with a chain-of-thought prompting approach (Wei et al., 2022) that accounts for model uncertainty. The model produces a chain of thought with k samples, for example 8 or 32. If there is a consensus above a preset threshold (selected based on the validation split), it selects this answer, otherwise it reverts to a greedy sample based on maximum likelihood choice without chain of thought.

I wonder when such approaches will be natively integrated into the UI for such models. Ideally, I should be able to, after presumably giving them my credit card information, turn my (Bard?) to ‘Gemini k-sample Chain of Thought’ and then have it take care of itself.

Here’s their table of benchmark results.

So the catch with MMLU is that Gemini Ultra gets more improvement from CoT@32, where GPT-4 did not improve much, but Ultra’s baseline performance on 5-shot is worse than GPT-4’s.

Except the other catch is that GPT-4, with creative prompting, can get to 89%?

GPT-4 is pretty excited about this potential ‘Gemini Ultra’ scoring 90%+ on the MMLU, citing a variety of potential applications and calling it a substantial advancement in AI capabilities.

They strongly imply that GPT-4 got 95.3% on HellaSwag due to data contamination, noting that including ‘specific website extracts’ improved Gemini’s performance there to a 1-shot 96%. Even if true, performance there is disappointing.

What does this suggest about Gemini Ultra? One obvious thing to do would be to average all the scores together for GPT-4, GPT-3.5 and Gemini, to place Gemini on the GPT scale. Using only benchmarks where 3.5 has a score, we get an average of 61 for GPT 3.5, 79.05 for GPT-4 and 80.1 for Gemini Ultra.

By that basic logic, we would award Gemini a benchmark of 4.03 GPTs. If you take into account that improvements matter more as scores go higher, and otherwise look at the context, and assume these benchmarks were not selected for results, I would increase that to 4.1 GPTs.

On practical text-only performance, I still expect GPT-4-turbo to be atop the leaderboards.

Gemini Pro clearly beat out PaLM-2 head-to-head on human comparisons, but not overwhelmingly so. It is kind of weird that we don’t have a win rate here for GPT-4 versus Gemini Ultra.

Image understanding benchmarks seem similar. Some small improvements, some big enough to potentially be interesting if this turns out to be representative.

Similarly they claim improved SOTA for video, where they also have themselves as the prior SOTA in many cases.

For image generation, they boast that text and images are seamlessly integrated, such as providing both text and images for a blog, but provide no examples of Gemini doing such an integration. Instead, all we get are some bizarrely tiny images.

One place we do see impressive claimed improvement is speech recognition. Note that this is only Gemini Pro, not Gemini Ultra, which should do better.

Those are error rate declines you would absolutely notice. Nano can run on-device and it is doing importantly better on YouTube than Whisper. Very cool.

Here’s another form of benchmarking.

The AlphaCode team built AlphaCode 2 (Leblond et al, 2023), a new Gemini-powered agent, that combines Gemini’s reasoning capabilities with search and tool-use to excel at solving competitive programming problems. AlphaCode 2 ranks within the top 15% of entrants on the Codeforces competitive programming platform, a large improvement over its state-of-the-art predecessor in the top 50% (Li et al., 2022).

…

AlphaCode 2 solved 43% of these competition problems, a 1.7x improvement over the prior record-setting AlphaCode system which solved 25%.

I read the training notes mostly as ‘we used all the TPUs, no really there were a lot of TPUs’ with the most interesting note being this speed-up. Does this mean they now have far fewer checkpoints saved, and if so does this matter?

Maintaining a high goodput [time spent computing useful new steps over the elapsed time of a training job] at this scale would have been impossible using the conventional approach of periodic checkpointing of weights to persistent cluster storage.

For Gemini, we instead made use of redundant in-memory copies of the model state, and on any unplanned hardware failures, we rapidly recover directly from an intact model replica. Compared to both PaLM and PaLM-2 (Anil et al., 2023), this provided a substantial speedup in recovery time, despite the significantly larger training resources being used. As a result, the overall goodput for the largest-scale training job increased from 85% to 97%.

Their section on training data drops a few technical hints but wisely says little. They deliberately sculpted their mix of training data, in ways they are keeping private.

In section 6 they get into responsible deployment. I appreciated them being clear they are focusing explicitly on questions of deployment.

They focus (correctly) exclusively on the usual forms of mundane harm, given Gemini is not yet breaking any scary new ground.

Building upon this understanding of known and anticipated effects, we developed a set of “model policies” to steer model development and evaluations. Model policy definitions act as a standardized criteria and prioritization schema for responsible development and as an indication of launch-readiness. Gemini model policies cover a number of domains including: child safety, hate speech, factual accuracy, fairness and inclusion, and harassment.

Their instruction tuning used supervised fine tuning and RLHF.

A particular focus was on attribution, which makes sense for Google.

Another was to avoid reasoning from a false premise and to otherwise refuse to answer ‘unanswerable’ questions. We need to see the resulting behavior but it sounds like the fun police are out in force.

It doesn’t sound like their mitigations for factuality were all that successful? Unless I am confusing what the numbers mean.

Looking over the appendix and its examples, it is remarkable how unimpressive were all of the examples given.

I notice that I watch how honestly DeepMind approaches reporting capabilities and attacking benchmarks as an important sign for their commitment to safety. There are some worrying signs that they are willing to twist quite a ways. Whereas the actual safety precautions do not bother me too much one way or the other?

The biggest safety precaution is one Google is not even calling a safety precaution. They are releasing Gemini Pro, and holding back Gemini Ultimate. That means they have a gigantic beta test with Pro, whose capabilities are such that it is harmless. They can use that to evaluate and tune Ultimate so it will be ready.

The official announcement offers some highlights.

Demis Hassabis talked to Wired about Gemini. Didn’t seem to add anything.

Gemini Pro, even without Gemini Ultra should be a substantial upgrade to Bard. The question is, will that be enough to make it useful when we have Claude and ChatGPT available? I will be trying it to find out, same as everyone else. Bard does have some other advantages, so it seems likely there will be some purposes, when you mostly want information, where Bard will be the play.

This video represents some useful prompt engineering and reasoning abilities, used to help plan a child’s birthday party, largely by brainstorming possibilities and asking clarifying questions. If they have indeed integrated this functionality in directly, that’s pretty cool.

Pete says Bard is finally at a point where he feels comfortable recommending it. The prompts are not first rate, but he says it is greatly improved since September and the integrations with GMail, YouTube and Maps are useful. It definitely is not a full substitute at this time, the question is if it is a good complement.

Even before Gemini, Bard did a very good job helping my son with his homework assignments, such that I was sending him there rather than to ChatGPT.

Returning a clean JSON continues to require extreme motivation.

When will Bard Advanced (with Gemini Ultra) be launched? Here’s a market on whether it happens in January.

Some were impressed. Others, not so much.

The first unimpressive thing is that all we are getting for now is Gemini Pro. Pro is very clearly not so impressive, clearly behind GPT-4.

Eli Dourado: Here is the table of Gemini evals from the paper. Note that what is being released into the wild today is Gemini Pro, not Gemini Ultra. So don’t expect Bard to be better than ChatGPT Plus just yet. Looks comparable to Claude 2.

Simeon? Not impressed.

Simeon: Gemini is here. Tbh it feels like it’s GPT-4 + a bit more multimodality + epsilon capabilities. So my guess is that it’s not a big deal on capabilities, although it might be a big deal from a product standpoint which seems to be what Google is looking for.

As always, one must note that everything involved was chosen to be what we saw, and potentially engineered or edited. The more production value, the more one must unwind.

For the big multimodal video, this issue is a big deal.

Robin: I found it quite instructive to compare this promo video with the actual prompts.

Robert Wiblin (distinct thread): It’s what Google themselves out put. So it might be cherry picked, but not faked. I think it’s impressive even if cherry picked.

Was this faked? EDIT: Yes. Just yes. Shame on Google on several levels.

Set aside the integrity issues, wow are we all jaded at this point, but when I watched that video, even when I assumed it was real, the biggest impression I got was… big lame dad energy?

I do get that this was supposedly happening in real time, but none of this is surprising me. Google put out its big new release, and I’m not scared. If anything, I’m kind of bored? This is the best you could do?

Whereas when watching the exact same video, others react differently.

Amjad Masad (CEO Replit): This fundamentally changes how humans work with computers.

Does it even if real? I mean, I guess, if you didn’t already assume all of it, and it was this smooth for regular users? I can think of instances in which a camera feed hooked up to Gemini with audio discussions could be a big game. To me this is a strange combination of the impressive parts already having been ‘priced into’ my world model, and the new parts not seeming impressive.

So I’m probably selling it short somewhat to be bored by it as a potential thing that could have happened. If this was representative of a smooth general multimodal experience, there is a lot to explore.

Arthur thinks Gemini did its job, but that this is unsurprising and it is weird people thought Google couldn’t do it.

Liv Boeree? Impressed.

Liv Boeree: This is pretty nuts, looks like they’ve surpassed GPT4 on basically every benchmark… so this is most powerful model in the world?! Woweee what a time to be alive.

Gary Marcus? Impressed in some ways, not in others.

Gary Marcus: Thoughts & prayers for VCs that bought OpenAI at $86B.

Hot take on Google Gemini and GPT-4:

Google Gemini seems to have by many measures matched (or slightly exceeded) GPT-4, but not to have blown it away.

From a commercial standpoint GPT-4 is no longer unique. That’s a huge problem for OpenAI, especially post drama, when many customers are now seeking a backup plan.

From a technical standpoint, the key question is: are LLMs close to a plateau?

Note that Gates and Altman have both been dropping hints, and GPT-5 isn’t here after a year despite immense commercial desire. The fact that Google, with all its resources, did NOT blow away GPT-4 could be telling.

I love that this is saying that OpenAI isn’t valuable both because Gemini is so good and also because Gemini is not good enough.

Roon offers precise praise.

Roon: congrats to Gemini team! it seems like the global high watermark on multimodal ability.

The MMLU result seems a bit fake / unfair terms but the HumanEval numbers look like a actual improvement and ime pretty closely match real world programming utility

David Manheim seems on point (other thread): I have not used the system, but if it does only slightly outmatch GPT-4, it seems like slight evidence that progress in AI with LLMs is not accelerating the way that many people worried and/or predicted.

Joey Krug is super unimpressed by the fudging on the benchmarks, says they did it across the board not only MMLU.

Packy McCormick: all of you (shows picture)

Ruxandra Teslo: wait what happened recently? did they do something good?

Packy: they did a good!

Google’s central problem is not wokeness, it is that they are a giant company with lots of internal processes and powers that prevent or slow or derail innovation, and prevent moving fast or having focus. And there are especially problems making practical products, integrating the work of various teams, making incentives line up. There is lots of potential, tons of talent, plenty of resources, but can they turn that into a product?

Too soon to tell. Certainly they are a long way from ‘beat OpenAI’ but this is the first and only case where someone might be in the game. The closest anyone else has come is Claude’s longer context window.

Gemini 1.0 Read More »

‘LEGO Bricktales’ Quest Review – VR Brick-building Done Right

ClockStone STUDIO, lego, LEGO Bricktales, lego bricktales vr, lego vr, Meta Quest 3 News & Reviews, News, Review, Thunderful Publishing / Paul Patrick / December 6, 2023

LEGO Bricktales may not be a VR-native, as it was first released on flatscreen last year, but this Quest-exclusive port makes a pretty solid case that lego brick-building not only works in VR, but is something anyone can do for hours on end—even in the face of a pretty kid-focused story.

LEGO Bricktales Details:

Available On: Quest 2/3/Pro

Reviewed On: Quest 3

Release Date: December 7th, 2023

Price: $30

Developer: ClockStone STUDIO

Publisher: Thunderful Publishing AB

Gameplay

LEGO Bricktales isn’t just a big box of lego in VR where you can go wild—there is a sandbox mode for each bespoke puzzle, however no ‘free for all’ blank sandbox space to build whatever you want. The emphasis with Bricktales is definitely on building all sorts of functional things with one-off lego sets, such as bridges, furniture, statues and more, and doing it amid some classic RPG worldbuilding that includes a ton of linear quests and puzzles to solve.

The kid-friendly story will have you spending a lot of time engaging with characters via text-based dialogue and figuring out how to help out each of the little inhabitants in the world, all of which (if it matters to you) comes with zero combat.

After all, you’re here to help restore the world by fixing things, and making everyone happy so you can… for some reason… fix your grandpa’s theme park with the power of happiness. Ok, that part is a little clunky, but it’s all in the name of honest, squeaky-clean fun that’s hard knock.

So, Bricktales is family-friendly fun, and it’s been largely admired for its light puzzling elements thanks to its clever block-building function. But how does that translate to VR? I would say surprisingly well—and that’s despite the inherent lack of tactility. When you’re prompted to build a model, you’re transported to a building space where you can grab pieces from a pre-set pile that you’ll need to attach to specific starting points. The objective below is to build a bridge from the blue arrow to the flag. Build it too wobbly, and it won’t past the stability test, making you reassess your design before going back to the world map.

While picking up and using fiddly little pieces sounds like a nightmare in VR, the digital lego pieces thankfully only go in one specific orientation, so snapping them into place is satisfying, and rarely ends in a miss. Browsing pieces with the tips of your controllers, which are blue orb-like cursors, you can pick up blocks, place them, and highlight to remove pieces from models. To snap them into a different orientation, you can either physically move the piece, or hold it and use the right joystick to change positions.

The only thing missing really is a quick reset button for when you’ve completely screwed up a model, which instead requires you to dismantle and throw lego bricks off the map to reset them into their little hoppers. That’s pretty tedious, especially if you want to build something from the ground up again.

There are a good array of puzzle styles ranging from bridge builder-style affairs, like the one above, to fulfilling one-off tasks, like constructing a perfectly balanced perch for a giant bird or building a racecar. Watch out though, because you can’t just plop down whatever you want. Each building prompt comes with a few prerequisites. Here’s how a typical puzzle might go for a little helicopter you need to build:

Attach the seat
Attach the rotor on top
Reach the finish line
Nothing may break

From there, you can build anything your imagination can handle (within the translucent wire cage), or equally just stick to the bare bones task to get past the hurdle. While none of the tasks are particularly hard (on flatscreen the game is suggested for kids 8+), all of them are gratifying in their own way, as they typically provide enough decorative pieces so you can not only build something functional, but embellish it with plenty of flair.

While fun in spurts, Bricktales also undoubtedly relies a ton on the cute factor of its little lego dioramas, all of which feel true to life. You can’t resize maps, which can either float in your living room thanks to mixed reality, or float in an unobtrusive skybox when played purely in VR. You can however twist and turn maps to get a better view for hidden pathways and so many easter eggs that you’ll be obligated to come back after the story is done, if only to see why that weird tree-man needs 20 chameleons. Seriously? Is what is he going to do with them??

Ok, as far as reasons for searching around the entire game for collectible extras goes, that’s fairly obtuse. Still, the “rated for ‘E’ everyone” age rating definitely means it’s geared towards kids, but snappy enough for adults to play too. Beware though, it’s not going to be the most engaging story, albeit harmless enough to act as sufficient narrative scaffolding that took me around six hours to complete. That’s just the story mode, so you can spend a lot more time rebuilding models and searching out the game’s many (many) collectibles, avatar skins, etc.

One of the definite misses with LEGO Bricktales is the lack of a dedicated sandbox. You can unlock a sandbox mode once you complete a bespoke construction spot. This lets you improve your model and also build with a growable selection of bricks from different biomes you explore along the way, but the true ‘sit down and build whatever’ feature would be great when you’re just looking to completely space out and build something of your own design.

Immersion

As you’d imagine, the whole word is made of lego, which is just so damn charming on its own. As a slightly-modified VR port of the flatscreen version, much of the praise you’ll find out there for Bricktales is also true here, but visually the Quest version has a definite leg-up on monitor versions. There’s something about the density of detail in the little dioramas that feels like really playing a game from the future.

Both Quest Pro and Quest 3 have color passthrough, which can be more immersive than playing in straight VR, which features a pretty innocuous skybox. On the spectrum of gimmick to absolutely essential though, the mixed reality in Bricktales is much closer to the gimmick side of things, as it’s just a plain passthrough and no real mixed reality implementation that would make it more immersive (i.e. logo dudes knowing where you couch is or busting through your walls). Still, it’s a pretty great gimmick, considering the little lego pieces are all accurately sized to their real-world counterparts. It’s difficult to at least marvel once or twice that you’re remote-controlling a little lego dude on your living room floor.

That said, there are less VR-specific interactions than I would have hoped, as most of the time you’re hunched over at the model controlling your dude like an RC car with your left thumbstick. Here’s the only other ‘immersive’ control scheme in the game: a rotary valve that can turn things like statues, water valves, etc.

View post on imgur.com

Substantively, the only other VR-specific adaptation from the original is your wrist-worn UI which clumsily lets you toggle through specific powers, leave the map to return to the overworld, and go through regular menu stuff.

Comfort

My first instinct was to hunch over and play the game like some sort of demigod looking over my little realm. The game is super approachable, and is designed for long playsessions, however it’s easy to lock into bad neck and back positions. Because VR headsets add extra weight that your neck has to overcompensate for, hunching over to play will fatigue your more quickly than doing the same action without the headset.

Granted, you can dynamically reposition the map to your liking at any point, so it’s more of a warning for players than a flaw as such. Otherwise, LEGO Bricktales is a very comfortable VR game since it lacks any sort of artificial locomotion, presenting you with an entirely static space.

‘LEGO Bricktales’ Comfort Settings – December 6th, 2023
Turning
Artificial turning
Snap-turn
Quick-turn
Smooth-turn
Movement
Artificial movement
Teleport-move
Dash-move
Smooth-move
Blinders
Head-based
Controller-based
Swappable movement hand
Posture
Standing mode
Seated mode
Artificial crouch
Real crouch
Accessibility
Subtitles
Languages	English, Simplified Chinese, Danish, French, German, Italian, Japanese, Korean, Portuguese (Brazil), Russian, Spanish
Dialogue audio
Languages	n/a
Adjustable difficulty
Two hands required
Real crouch required
Hearing required
Adjustable player height

‘LEGO Bricktales’ Quest Review – VR Brick-building Done Right Read More »