Author name: Paul Patrick

How physics moves from wild ideas to actual experiments

experiments, Features, neutrinos, particles, Science / Paul Patrick / November 25, 2024

Science often accommodates audacious proposals.

Instead of using antennas, could we wire up trees in a forest to detect neutrinos? Credit: Claire Gillo/PhotoPlus Magazine/Future via Getty Images

Neutrinos are some of nature’s most elusive particles. One hundred trillion fly through your body every second, but each one has only a tiny chance of jostling one of your atoms, a consequence of the incredible weakness of the weak nuclear force that governs neutrino interactions. That tiny chance means that reliably detecting neutrinos takes many more atoms than are in your body. To spot neutrinos colliding with atoms in the atmosphere, experiments have buried 1,000 tons of heavy water, woven cameras through a cubic kilometer of Antarctic ice, and planned to deploy 200,000 antennas.

In a field full of ambitious plans, a recent proposal by Steven Prohira, an assistant professor at the University of Kansas, is especially strange. Prohira suggests that instead of using antennas, we could detect the tell-tale signs of atmospheric neutrinos by wiring up a forest of trees. His suggestion may turn out to be impossible, but it could also be an important breakthrough. To find out which it is, he’ll need to walk a long path, refining prototypes and demonstrating his idea’s merits.

Prohira’s goal is to detect so-called ultra-high-energy neutrinos. Each one of these tiny particles carries more than fifty million times the energy released by uranium during nuclear fission. Their origins are not fully understood, but they are expected to be produced by some of the most powerful events in the Universe, from collapsing stars and pulsars to the volatile environments around the massive black holes at the centers of galaxies. If we could detect these particles more reliably, we could learn more about these extreme astronomical events.

Other experiments, like a project called GRAND, plan to build antennas to detect these neutrinos, watching for radio signals that come from their reactions with our atmosphere. However, finding places to place these antennas can be a challenge. Motivated by this experiment, Prohira dug up old studies by the US Army that suggested an alternative: instead of antennas, use trees. By wrapping a wire around each tree, army researchers found that the trees were sensitive to radio waves, which they hoped to use to receive radio signals in the jungle. Prohira argues that the same trick could be useful for neutrino detection.

Crackpot or legit science?

People suggest wacky ideas every day. Should we trust this one?

At first, you might be a bit suspicious. Prohira’s paper is cautious on the science but extremely optimistic in other ways. He describes the proposal as a way to help conserve the Earth’s forests and even suggests that “a forest detector could also motivate the large-scale reforesting of land, to grow a neutrino detector for future generations.”

Prohira is not a crackpot, though. He has a track record of research in detecting neutrinos via radio waves in more conventional experiments, and he even received an $800,000 MacArthur genius grant a few years ago to support his work.

More generally, studying particles from outer space often demands audacious proposals, especially ones that make use of the natural world. Professor Albrecht Karle works on the IceCube experiment, an array of cameras that detect neutrinos whizzing through a cubic kilometer of Antarctic ice.

“In astroparticle physics, where we often cannot build the entire experiment in a laboratory, we have to resort to nature to help us, to provide an environment that can be used to build a detector. For example, in many parts of astroparticle physics, we are using the atmosphere as a medium, or the ocean, or the ice, or we go deep underground because we need a shield because we cannot construct an artificial shield. There are even ideas to go into space for extremely energetic neutrinos, to build detectors on Jupiter’s moon Europa.”

Such uses of nature are common in the field. India’s GRAPES experiments were designed to measure muons, but they have to filter out anything that’s not a muon to do so. As Professor Sunil Gupta of the Tata Institute explained, the best way to do that was with dirt from a nearby hill.

“The only way we know you can make a muon detector work is by filtering out other radiation […] so what we decided is that we’ll make a civil structure, and we’ll dump three meters of soil on top of that, so those three meters of soil could act as a filter,” he said.

The long road to an experiment

While Prohira’s idea isn’t ridiculous, it’s still just an idea (and one among many). Prohira’s paper describing the idea was uploaded to arXiv.org, a pre-print server, in January. Physicists use pre-print servers to give access to their work before it’s submitted to a scientific journal. That gives other physicists time to comment on the work and suggest revisions. In the meantime, the journal will send the work out to a few selected reviewers, who are asked to judge both whether the paper is likely to be correct and whether it is of sufficient interest to the community.

At this stage, reviewers may find problems with Prohira’s idea. These may take the form of actual mistakes, such as if he made an error in his estimates of the sensitivity of the detector. But reviewers can also ask for more detail. For example, they could request a more extensive analysis of possible errors in measurements caused by the different shapes and sizes of the trees.

If Prohira’s idea makes it through to publication, the next step toward building an actual forest detector would be convincing the larger community. This kind of legwork often takes place at conferences. The International Cosmic Ray Conference is the biggest stage for the astroparticle community, with conferences every two years—the next is scheduled for 2025 in Geneva. Other more specialized conferences, like ARENA, focus specifically on attempts to detect radio waves from high-energy neutrinos. These conferences can offer an opportunity to get other scientists on board and start building a team.

That team will be crucial for the next step: testing prototypes. No matter how good an idea sounds in theory, some problems only arise during a real experiment.

An early version of the GRAPES experiment detected muons by the light they emit passing through tanks of water. To find how much water was needed, the researchers did tests, putting a detector on top of a tank and on the bottom and keeping track of how often both detectors triggered for different heights of water based on the muons that came through randomly from the atmosphere. After finding that the tanks of water would have to be too tall to fit in their underground facility, they had to find wavelength-shifting chemicals that would allow them to use shorter tanks and novel ways of dissolving these chemicals without eroding the aluminum of the tank walls.

“When you try to do something, you run into all kinds of funny challenges,” said Gupta.

The IceCube experiment has a long history of prototypes going back to early concepts that were only distantly related to the final project. The earliest, like the proposed DUMAND project in Hawaii, planned to put detectors in the ocean rather than ice. BDUNT was an intermediate stage, a project that used the depths of Lake Baikal to detect atmospheric neutrinos. While the detectors were still in liquid water, the ability to drive on the lake’s frozen surface made BDUNT’s construction easier.

In a 1988 conference, Robert March, Francis Halzen, and John G. Learned envisioned a kind of “solid state DUMAND” that would use ice instead of water to detect neutrinos. While the idea was attractive, the researchers cautioned that it would require a fair bit of luck. “In summary, this is a detector that requires a number of happy accidents to make it feasible. But if these should come to pass, it may provide the least expensive route to a truly large neutrino telescope,” they said.

In the case of the AMANDA experiment, early tests in Greenland and later tests at the South Pole began to provide these happy accidents. “It was discovered that the ice was even more exceptionally clear and has no radioactivities—absolutely quiet, so it is the darkest and quietest and purest place on Earth,” said Karle.

AMANDA was much smaller than the IceCube experiment, and theorists had already argued that to see cosmic neutrinos, the experiment would need to cover a cubic kilometer of ice. Still, the original AMANDA experiment wasn’t just a prototype; if neutrinos arrived at a sufficient rate, it would spot some. In this sense, it was like the original LIGO experiment, which ran for many years in the early 2000s with only a minimal chance of detecting gravitational waves, but it provided the information needed to perform an upgrade in the 2010s that led to repeated detections. Similarly, the hope of pioneers like Halzen was that AMANDA would be able to detect cosmic neutrinos despite its prototype status.

“There was the chance that, with the knowledge at the time, one might get lucky. He certainly tried,” said Karle.

Prototype experiments often follow this pattern. They’re set up in the hope that they could discover something new about the Universe, but they’re built to at least discover any unexpected challenges that would stop a larger experiment.

Major facilities and the National Science Foundation

For experiments that don’t need huge amounts of funding, these prototypes can lead to the real thing, with scientists ratcheting up their ambition at each stage. But for the biggest experiments, the governments that provide the funding tend to want a clearer plan.

Since Prohira is based in the US, let’s consider the US government. The National Science Foundation has a procedure for its biggest projects, called the Major Research Equipment and Facilities Construction program. Since 2009, it has had a “no cost overrun” policy. In the past, if a project ended up costing more than expected, the NSF could try to find additional funding. Now, projects are supposed to estimate beforehand how the cost could increase and budget extra for the risk. If the budget goes too high anyway, projects should compensate by reducing scope, shrinking the experiment until it falls under costs again.

To make sure they can actually do this, the NSF has a thorough review process.

First, the NSF expects that the scientists proposing a project have done their homework and have already put time and money into prototyping the experiment. The general expectation is that about 20 percent of the experiment’s total budget should have been spent testing out the idea before the NSF even starts reviewing it.

With the prototypes tested and a team assembled, the scientists will get together to agree on a plan. This often means writing a report to hash out what they have in mind. The IceCube team is in the process of proposing a second generation of their experiment, an expansion that would cover more ice with detectors and achieve further scientific goals. The team recently finished the third part of a Technical Design Report, which details the technical case for the experiment.

After that, experiments go into the NSF’s official experiment design process. This has three phases, conceptual design, preliminary design, and final design. Each phase ends with a review document summarizing the current state of the plans as they firm up, going from a general scientific case to a specific plan to put an experiment in a specific place. Risks are estimated in detail and list estimates of how likely risks are and how much they will cost, a process that sometimes involves computer simulations. By the end of the process, the project has a fully detailed plan and construction can begin.

Over the next few years, Prohira will test out his proposal. He may get lucky, like the researchers who dug into Antarctic ice, and find a surprisingly clear signal. He may be unlucky instead and find that the complexities of trees, with different spacings and scatterings of leaves, makes the signals they generate unfit for neutrino science. He, and we, cannot know in advance which will happen.

That’s what science is for, after all.

How physics moves from wild ideas to actual experiments Read More »

After Russian ship docks to space station, astronauts report a foul smell

iss, russia, Science, Space / Paul Patrick / November 24, 2024

Russian space program faces ongoing challenges

Zak reported that the cosmonauts aboard the Russian segment of the station donned protective equipment, and activated an extra air-scrubbing system aboard their side of the facility. On the US segment of the station, NASA astronaut Don Pettit said he smelled something akin to “spray paint.”

As of Sunday afternoon, NASA said there were no concerns for the crew, and that astronauts were working to open the hatch between the Poisk module and the Progress spacecraft. Attached to the space station in 2009, Poisk is a small element that connects to one of four docking ports on the Russian segment of the station.

It was not immediately clear what caused the foul odor to emanate from the Progress vehicle, however previous Russian vehicles have had leaks while in space. Most recently, in February 2023, a Progress vehicle attached to the station lost pressurization in its cooling system.

Facing financial and staffing pressures due to the ongoing Russian war against Ukraine, the main Russian space corporation, Roscosmos, has faced a series of technical problems as it has sought to fly people and supplies to the International Space Station in recent years.

After Russian ship docks to space station, astronauts report a foul smell Read More »

Tweaking non-neural brain cells can cause memories to fade

astrocytes, Biology, memory, neurobiology, neurons, Neuroscience, Science / Paul Patrick / November 23, 2024

Neurons and a second cell type called an astrocyte collaborate to hold memories.

Astrocytes (labelled in black) sit within a field of neurons. Credit: Ed Reschke

“If we go back to the early 1900s, this is when the idea was first proposed that memories are physically stored in some location within the brain,” says Michael R. Williamson, a researcher at the Baylor College of Medicine in Houston. For a long time, neuroscientists thought that the storage of memory in the brain was the job of engrams, ensembles of neurons that activate during a learning event. But it turned out this wasn’t the whole picture.

Williamson’s research investigated the role astrocytes, non-neuron brain cells, play in the read-and-write operations that go on in our heads. “Over the last 20 years the role of astrocytes has been understood better. We’ve learned that they can activate neurons. The addition we have made to that is showing that there are subsets of astrocytes that are active and involved in storing specific memories,” Williamson says in describing a new study his lab has published.

One consequence of this finding: Astrocytes could be artificially manipulated to suppress or enhance a specific memory, leaving all other memories intact.

Marking star cells

Astrocytes, otherwise known as star cells due to their shape, play various roles in the brain, and many are focused on the health and activity of their neighboring neurons. Williamson’s team started by developing techniques that enabled them to mark chosen ensembles of astrocytes to see when they activate genes (including one named c-Fos) that help neurons reconfigure their connections and are deemed crucial for memory formation. This was based on the idea that the same pathway would be active in neurons and astrocytes.

“In simple terms, we use genetic tools that allow us to inject mice with a drug that artificially makes astrocytes express some other gene or protein of interest when they become active,” says Wookbong Kwon, a biotechnologist at Baylor College and co-author of the study.

Those proteins of interest were mainly fluorescent proteins that make cells fluoresce bright red. This way, the team could spot the astrocytes in mouse brains that became active during learning scenarios. Once the tagging system was in place, Williamson and his colleagues gave their mice a little scare.

“It’s called fear conditioning, and it’s a really simple idea. You take a mouse, put it into a new box, one it’s never seen before. While the mouse explores this new box, we just apply a series of electrical shocks through the floor,” Williamson explains. A mouse treated this way remembers this as an unpleasant experience and associates it with contextual cues like the box’s appearance, the smells and sounds present, and so on.

The tagging system lit up all astrocytes that expressed the c-Fos gene in response to fear conditioning. Williamson’s team inferred that this is where the memory is stored in the mouse’s brain. Knowing that, they could move on to the next question, which was if and how astrocytes and engram neurons interacted during this process.

Modulating engram neurons

“Astrocytes are really bushy,” Williamson says. They have a complex morphology with lots and lots of micro or nanoscale processes that infiltrate the area surrounding them. A single astrocyte can contact roughly 100,000 synapses, and not all of them will be involved in learning events. So the team looked for correlations between astrocytes activated during memory formation and the neurons that were tagged at the same time.

“When we did that, we saw that engram neurons tended to be contacting the astrocytes that are active during the formation of the same memory,” Williamson says. To see how astrocytes’ activity affects neurons, the team artificially stimulated the astrocytes by microinjecting them with a virus engineered to induce the expression of the c-Fos gene. “It directly increased the activity of engram neurons but did not increase the activity of non-engram neurons in contact with the same astrocyte,” Williamson explains.

This way his team established that at least some astrocytes could preferentially communicate with engram neurons. The researchers also noticed that astrocytes involved in memorizing the fear conditioning event had elevated levels of a protein called NFIA, which is known to regulate memory circuits in the hippocampus.

But probably the most striking discovery came when the researchers tested whether the astrocytes involved in memorizing an event also played a role in recalling it later.

Selectively forgetting

The first test to see if astrocytes were involved in recall was to artificially activate them when the mice were in a box that they were not conditioned to fear. It turned out artificial activation of astrocytes that were active during the formation of a fear memory formed in one box caused the mice to freeze even when they were in a different one.

So, the next question was, if you just killed or otherwise disabled an astrocyte ensemble active during a specific memory formation, would it just delete this memory from the brain? To get that done, the team used their genetic tools to selectively delete the NFIA protein in astrocytes that were active when the mice received their electric shocks. “We found that mice froze a lot less when we put them in the boxes they were conditioned to fear. They could not remember. But other memories were intact,” Kwon claims.

The memory was not completely deleted, though. The mice still froze in the boxes they were supposed to freeze in, but they did it for a much shorter time on average. “It looked like their memory was maybe a bit foggy. They were not sure if they were in the right place,” Williamson says.

After figuring out how to suppress a memory, the team also figured out where the “undo” button was and brought it back to normal.

“When we deleted the NFIA protein in astrocytes, the memory was impaired, but the engram neurons were intact. So, the memory was still somewhere there. The mice just couldn’t access it,” Williamson claims. The team brought the memory back by artificially stimulating the engram neurons using the same technique they employed for activating chosen astrocytes. “That caused the neurons involved in this memory trace to be activated for a few hours. This artificial activity allowed the mice to remember it again,” Williamson says.

The team’s vision is that in the distant future this technique can be used in treatments targeting neurons that are overactive in disorders such as PTSD. “We now have a new cellular target that we can evaluate and potentially develop treatments that target the astrocyte component associated with memory,” Williamson claims. But there’s lot more to learn before anything like that becomes possible. “We don’t yet know what signal is released by an astrocyte that acts on the neuron. Another thing is our study was focused on one brain region, which was the hippocampus, but we know that engrams exist throughout the brain in lots of different regions. The next step is to see if astrocytes play the same role in other brain regions that are also critical for memory,” Williamson says.

Nature, 2024. DOI: 10.1038/s41586-024-08170-w

Jacek Krywko is a freelance science and technology writer who covers space exploration, artificial intelligence research, computer science, and all sorts of engineering wizardry.

Tweaking non-neural brain cells can cause memories to fade Read More »

GigaOm Research Bulletin #010

GigaOm / Paul Patrick / November 23, 2024

This bulletin is aimed at our analyst relations connections and vendor subscribers, to update you on the research we are working on, reports we have published, and improvements we have been making. Please do reach out if you have any questions!

CEO Speaks podcast with Ben Book

In our CEO Speaks podcast, our CEO, Ben Book, discusses leadership challenges and the technology market landscape with vendor CEOs. In the latest edition, he speaks to James Winebrenner, CEO of Elisity. As always, please get in touch if you would like to propose your own CEO.

The Good, Bad, and The Techy podcast

In this, more engineering-focused podcast, Howard Holton and Jon Collins sit down with Tyler Reese, Director of Product Management at Netwrix, to discuss the challenges and best practices faced when deploying Identity Security. Do give it a listen, and again, we welcome any suggestions for guests.

Recent Reports

We’ve released 17 reports since the last bulletin.

In Analytics and AI, we have a report on Data Observability, Semantic Layers and Metric Stores and Data Catalogs.

For Cloud Infrastructure and Operations, we have Hybrid Cloud Data Protection and AIOps. In Storage, we have covered Cloud-Native Globally Distributed File Systems.

In the Security domain, we have released reports on SaaS Security Posture Management (SSPM), Secure Enterprise Browsing, Data Loss Prevention (DLP), Continuous Vulnerability Management (CVM), Insider Risk Management, Autonomous Security Operations Center (SOC) Solutions, Security Orchestration, Automation and Response (SOAR), and Cloud-Native Application Protection Platforms (CNAPPS).

In Networking, we have covered DDI (DNS, DHCP, and IPAM).

And in Software and Applications, we have a report on E-Discovery and Intelligent Document Processing (IDP).

Blogs and Articles

Our COO, Howard Holton, offers a four-part blog series on “How to CIO”:

Other blogs include:

Paul Stringfellow tells us that with SaaS Security Posture – It’s not you, it’s me! asks, Are We Taking Enterprise Password Management Seriously?, and explains “Gone Phishing” – Every Cyberattacker’s Favorite Phrase and The Impact of GenAI on Data Loss Prevention.
Andrew Green gives us Identity, Endpoint, and Network Security Walk into a Bar.
Matt Jallo explains SaaS Management and Application Discovery.
Ivan McPhee takes us through The Detection Debate: Deep-Packet Inspection vs. Flow-Based Analysis, Sidecarless Service Meshes: Are They Ready for Prime Time?, and Don’t Ignore What You Can Easily Control: Your IP Infrastructure.
Kirk Ryan looks at The Rapidly Changing Landscape of Enterprise Object Storage.
Chris Ray takes us through Navigating the Unique Landscape of OT Security Solutions and The Future of ZTNA: A Convergence of Network Access Solutions.
Shane Archiquette explains Edge Orchestration Platforms: Navigating the Next Wave of Innovation and Getting Started with Hybrid Cloud Data Protection.
Lisa Erickson-Harris explains How ITIL’s Ownership Transition Impacts the ITSM Market.
Whit Walters gives us Security Policy as Code: An Architect’s Perspective on the Evolving Landscape.
Chester Conforte tells us to Backup to the Future!
Sue Clarke writes about Digital Experience Platforms: Transforming Marketing in the 21st Century.

Meanwhile Jon talks about Operations Leadership Lessons from the Crowdstrike Incident and DevOps, LLMs and the Software Development Singularity and asks 5 questions of Carsten Brinkschulte at Dryad, covering the use of IoT in forest fire prevention.

GigaOm Research Bulletin #010 Read More »

Ted Cruz wants to overhaul $42B broadband program, nix low-cost requirement

access, and Deployment program, Broadband Equity, Policy, ted cruz / Paul Patrick / November 22, 2024

Emboldened by Donald Trump’s election win, Republicans are seeking big changes to a $42.45 billion broadband deployment program. Their plan could delay distribution of government funding and remove or relax a requirement that ISPs accepting subsidies must offer low-cost Internet plans.

US Senator Ted Cruz (R-Texas) today issued a press release titled, “Sen. Cruz Warns Biden-Harris NTIA: Big Changes Ahead for Multi-Billion-Dollar Broadband Boondoggle.” Cruz, who will soon be chair of the Senate Commerce Committee, is angry about how the National Telecommunications and Information Administration has implemented the Broadband Equity, Access, and Deployment (BEAD) program that was created by Congress in November 2021.

The NTIA announced this week that it has approved the funding plans submitted by all 50 states, the District of Columbia, and five US territories, which are slated to receive federal money and dole it out to broadband providers for network expansions. Texas was the last state to gain approval in what the NTIA called “a major milestone on the road to connecting everyone in America to affordable, reliable high-speed Internet service.”

Republicans including Cruz and incoming Federal Communications Commission Chairman Brendan Carr have criticized the NTIA for not distributing the money faster. But Cruz’s promise of a revamp creates uncertainty about the distribution of funds. Cruz sent a letter yesterday to NTIA Administrator Alan Davidson in which he asked the agency to halt the program rollout until Trump takes over. Cruz also accused the NTIA of “technology bias” because the agency decided that fiber networks should be prioritized over other types of technology.

Cruz: Stop what you’re doing

“It is incumbent on you to bear these upcoming changes in mind during this transition term,” Cruz wrote. “I therefore urge the NTIA to pause unlawful, extraneous BEAD activities and avoid locking states into in [sic] any final actions until you provide a detailed, transparent response to my original inquiry and take immediate, measurable steps to address these issues.”

Ted Cruz wants to overhaul $42B broadband program, nix low-cost requirement Read More »

Study: Yes, tapping on frescoes can reveal defects

acoustics, art conservation, Science / Paul Patrick / November 21, 2024

The US Capitol building in Washington, DC, is adorned with multiple lavish murals created in the 19th century by Italian artist Constantino Brumidi. These include panels in the Senate first-floor corridors, Room H-144, and the rotunda. The crowning glory is The Apotheosis of Washington on the dome of the rotunda, 180 feet above the floor.

Brumidi worked in various mediums, including frescoes. Among the issues facing conservators charged with maintaining the Capitol building frescoes is delamination. Artists apply dry pigments to wet plaster to create a fresco, and a good fresco can last for centuries. Over time, though, the decorative plaster layers can separate from the underlying masonry, introducing air gaps. Knowing precisely where such delaminated areas are, and their exact shape, is crucial to conservation efforts, yet the damage might not be obvious to the naked eye.

Acoustician Nicholas Gangemi is part of a research group led by Joseph Vignola at the Catholic University of America that has been using laser Doppler vibrometry to pinpoint delaminated areas of the Capitol building frescoes. It’s a non-invasive method that zaps the frescoes with sound waves and measures the vibrational signatures that reflect back to learn about the structural conditions. This in turn enables conservators to make very precise repairs to preserve the frescoes for future generations.

It’s an alternative to the traditional technique of gently knocking on the plaster with knuckles or small mallets, listening to the resulting sounds to determine where delamination has occurred. Once separation occurs, the delaminated part of the fresco acts a bit like the head of a drum; tapping on it produces a distinctive acoustic signature.

But the method is highly subjective. It takes years of experience to become proficient at this method, and there are only a small number of people who can truly be deemed experts. “We really wanted to put that experience and knowledge into an inexperienced person’s hands,” Gangemi said during a press briefing at a virtual meeting of the Acoustical Society of America. So he and his colleagues decided to put the traditional knocking method to the test.

Study: Yes, tapping on frescoes can reveal defects Read More »

AI #91: Deep Thinking

Thinking / Paul Patrick / November 21, 2024

Did DeepSeek effectively release an o1-preview clone within nine weeks?

The benchmarks largely say yes. Certainly it is an actual attempt at a similar style of product, and is if anything more capable of solving AIME questions, and the way it shows its Chain of Thought is super cool. Beyond that, alas, we don’t have enough reports in from people using it. So it’s still too soon to tell. If it is fully legit, the implications seems important.

Small improvements continue throughout. GPT-4o and Gemini both got incremental upgrades, trading the top slot on Arena, although people do not seem to much care.

There was a time everyone would be scrambling to evaluate all these new offerings. It seems we mostly do not do that anymore.

The other half of events was about policy under the Trump administration. What should the federal government do? We continue to have our usual fundamental disagreements, but on a practical level Dean Ball offered mostly excellent thoughts. The central approach here is largely overdetermined, you want to be on the Pareto frontier and avoid destructive moves, which is how we end up in such similar places.

Then there’s the US-China commission, which now have their top priority being an explicit ‘race’ to AGI against China, without actually understanding what that would mean or justifying that anywhere in their humongous report.

Table of Contents.
Language Models Offer Mundane Utility. Get slightly more utility than last week.
Language Models Don’t Offer Mundane Utility. Writing your court briefing.
Claude Sonnet 3.5.1 Evaluation. Its scored as slightly more dangerous than before.
Deepfaketown and Botpocalypse Soon. AI boyfriends continue to be coming.
Fun With Image Generation. ACX test claims you’re wrong about disliking AI art.
O-(There are)-Two. DeepSeek fast follows with their version of OpenAI’s o1.
The Last Mile. Is bespoke human judgment going to still be valuable for a while?
They Took Our Jobs. How to get ahead in advertising, and Ben Affleck is smug.
We Barely Do Our Jobs Anyway. Why do your job when you already don’t have to?
The Art of the Jailbreak. Getting an AI agent to Do Cybercrime.
Get Involved. Apply for OpenPhil global existential risk portfolio manager.
The Mask Comes Off. Some historical emails are worth a read.
Richard Ngo on Real Power and Governance Futures. Who will have the power?
Introducing. Stripe SDK, Anthropic prompt improver, ChatGPT uses Mac apps.
In Other AI News. Mistral has a new model too, and many more.
Quiet Speculations. What will happen with that Wall?
The Quest for Sane Regulations. The conservative case for alignment.
The Quest for Insane Regulations. The US-China commission wants to race.
Pick Up the Phone. Is China’s regulation light touch or heavy? Unclear.
Worthwhile Dean Ball Initiative. A lot of agreement about Federal options here.
The Week in Audio. Report on Gwern’s podcast, also I have one this week.
Rhetorical Innovation. What are the disagreements that matter?
Pick Up the Phone. At least we agree not to hand over the nuclear weapons.
Aligning a Smarter Than Human Intelligence is Difficult. How’s it going?
People Are Worried About AI Killing Everyone. John von Neumann.
The Lighter Side. Will we be able to understand each other?

Briefly on top of Arena, Gemini-Exp-1114 took a small lead over various OpenAI models, also taking #1 or co-#1 on math, hard prompts, vision and creative writing.

Then GPT-4o got an upgrade and took the top spot back.

OpenAI: The model’s creative writing ability has leveled up–more natural, engaging, and tailored writing to improve relevance & readability.

It’s also better at working with uploaded files, providing deeper insights & more thorough responses.

It’s also an improvement on MinecraftBench, but two out of two general replies on Twitter so far said this new GPT-4o didn’t seem that different.

Arena is no longer my primary metric because it seems to make obvious mistakes – in particular, disrespecting Claude Sonnet so much – but it is still measuring something real, and this is going to be a definite improvement.

CORE-Bench new results show Claude Sonnet clear first at 37.8% pass rate on agent tasks, with o1-mini in second at 24.4%, versus previous best of 21.5% by GPT-4o. Sonnet also has a 2-to-1 cost advantage over o1-mini. o1-preview exceeded the imposed cost limit.

METR runs an evaluation of the ability of LLMs to conduct AI research, finds Claude Sonnet 3.5 outperforms o1-preview on five out of seven tasks.

The trick is to ask the LLM first, rather than (only) last:

Agnes Callard: My computer was weirdly broken so I called my husband and we tried a bunch of things to fix it but nothing worked and he had to go to bed (time diff, I am in Korea) so in desperation (I need it for a talk I’m giving in an hour) I asked chat gpt and its first suggestion worked!

Diagnose yourself, since ChatGPT seems to outperform doctors, and if you hand the doctor a one-pager with all the information and your ChatGPT diagnosis they’re much more likely to get the right answer.

• ChatGPT scored 90 percent, while physicians scored 74–76 percent in diagnosing cases.

• Physicians often resisted chatbot insights that contradicted their initial beliefs.

• Only a few physicians maximized ChatGPT’s potential by submitting full case histories.

• Study underscores the need for better AI training and adoption among medical professionals.

I love the social engineering of handing the doctor a one pager. You don’t give them a chance to get attached to a diagnosis first, and you ensure you get them the key facts, and the ‘get them the facts’ lets the doctor pretend they’re not being handed a diagnosis.

Use voice mode to let ChatGPT (or Gemini) chat with your 5-year-old and let them ask it questions. Yes, you’d rather a human do this, especially yourself, but one cannot always do that, and anyone who yells ‘shame’ should themselves feel shame. Do those same people homeschool their children? Do they ensure they have full time aristocratic tutoring?

Regular humans cannot distinguish AI poems from poems by some of the most famous human poets, and actively prefer the AI poems in many ways, including thinking them more likely to be human – so they can distinguish to a large extent, they just get the sign wrong. Humans having somewhat more poetry exposure did not help much either. The AI poems being more straightforward is cited as an advantage, as is the human poems often being old, with confusing references that are often dealing with now-obscure things.

So it sounds like a poetry expert, even if they hadn’t seen the exact poems in question, would be able to distinguish the poems and would prefer the human ones, but would also say that most humans have awful taste in poetry.

Frank Bednarz, presumably having as much fun as I was: Crazy, true story: Minnesota offered an expert declaration on AI and “misinformation” to oppose our motion to enjoin their unconstitutional law.

His declaration included fake citations hallucinated by AI!

…

His report claims that “One study found that even when individuals are informed about the existence of deepfakes, they may still struggle to distinguish between real and manipulated content.”

I guess the struggle is real because this study does not exist!

…

As far as we can tell, this is the first time that an expert has cited hallucinated content in court. Eugene at @VolokhC has followed AI-generated content in the courts closely. No one else seems to have called out hallucinated expert citations before.

…

Volokh also discovered that there’s a second hallucinated citation in the declaration. The author & journal are real, but this article does not exist and is not at the cited location. Some puckish AI invented it!

…

The gist of his report is that counterspeech no longer works (and therefore government censorship is necessary). I think that’s incorrect, and we hopefully prove our point by calling out AI misinformation to the court.

If you can’t use AI during your coding interview, do they know if you can code?

Humans attach too much importance to when AIs fail tasks that are easy for humans, and are too impressed when they do things that are hard for humans, paper confirms. You see this all over Twitter, especially on new model releases – ‘look at this idiot model that can’t even do [X].’ As always, ask what it can do, not what it can’t do, but also don’t be too impressed if it’s something that happens to be difficult for humans.

Meta adds ‘FungiFriend’ AI bot to a mushroom forager group automatically, without asking permission, after which it proceeded to give advice on how to cook mushrooms that are not safe to consume, while claiming they were ‘edible but rare.’ Where the central point of the whole group is to ensure new foragers don’t accidentally poison themselves.

Experiment sees only gpt-3.5-turbo-instruct put up even a halfway decent chess game against low-level Stockfish, whereas every other model utterly failed. And we mean rather low-level Stockfish, the game I sampled was highly unimpressive. Of course, this can always be largely a skill issue, as Will Depue notes even a little fine tuning makes a huge difference.

Joint US AISI and UK AISI testing of the upgraded Claude 3.5:

Divyansh Kaushik: On bio: Sonnet 3.5 underperforms human experts in most biological tasks but excels in DNA and protein sequence manipulation with tool access. Access to computational biology tools greatly enhances performance.

On Cyber: Sonnet 3.5 demonstrates strong capabilities in basic cyber tasks but struggles with more advanced tasks requiring expert-level knowledge. – Improved success rates in vulnerability discovery & exploitation at beginner levels compared to previous versions. – Task-based probing reveals the model’s dependency on human intervention for complex challenges.

From the Report: On Software and AI Development:

Key findings:

US AISI evaluated the upgraded Sonnet 3.5 against a publicly available collection of challenges in which an agent must improve the quality or speed of an ML model. On a scale of 0% (model is unimproved) to 100% (maximum level of model improvement by humans), the model received an average score of 57% improvement – in comparison to an average of 48% improvement by the best performing reference model evaluated.

UK AISI evaluated the upgraded Sonnet 3.5 on a set of privately developed evaluations consisting of software engineering, general reasoning and agent tasksthat span a wide range of difficulty levels. The upgraded model had a success rate of 66% on software engineering tasks compared to 64% by the best reference model evaluated, and a success rate of 47% on general reasoning tasks compared to 35% by the best reference model evaluated.

On safeguard efficiency, meaning protection against jailbreaks, they found that its defenses were routinely circumvented, as they indeed often are in practice:

US AISI tested the upgraded Sonnet 3.5 against a series of publicly available jailbreaks, and in most cases the built-in version of the safeguards that US AISI tested were circumvented as a result, meaning the model provided answers that should have been prevented. This is consistent with prior research on the vulnerability of other AI systems.

UK AISI tested the upgraded Sonnet 3.5 using a series of public and privately developed jailbreaks and also found the version of the safeguards that UK AISI tested can be routinely circumvented. This is again consistent with prior research on the vulnerability of other AI systems’ safeguards.

The latest round of AI boyfriend talk, with an emphasis on their rapid quality improvements over time. Eliezer again notices that AI boyfriends seem to be covered much more sympathetically than AI girlfriends, with this article being a clear example. I remain in the group that expects the AI boyfriends to be more popular and a bigger issue than AI girlfriends, similar to ‘romance’ novels.

Aella finally asked where the best LLMs are for fully uncensored erotica. Suggestions from those who did not simply say ‘oh just jailbreak the model’ included glhf.chat, letmewriteforyou.xyz, Outfox Stories, venice.ai, Miqu-70B, and an uncensored model leaderboard.

Results are in, and all of you only got 60% right in the AI versus human ACX art test.

DeepSeek has come out with its version or OpenAI’s o1, you can try it here for 50 messages per day.

As is often the case with Chinese offerings, the benchmark numbers impress.

DeepSeek: 🚀 DeepSeek-R1-Lite-Preview is now live: unleashing supercharged reasoning power!

🔍 o1-preview-level performance on AIME & MATH benchmarks.

💡 Transparent thought process in real-time.

🛠️ Open-source models & API coming soon!

🌟 Inference Scaling Laws of DeepSeek-R1-Lite-Preview

Longer Reasoning, Better Performance. DeepSeek-R1-Lite-Preview shows steady score improvements on AIME as thought length increases.

Dean Ball: There you have it. First credible Chinese replication of the OpenAI o1 paradigm, approximately 9 weeks after o1 is released.

And it’s apparently going to be open source.

Tyler Cowen: The really funny thing here is that I can’t solve the CAPTCHA to actually use the site.

Walpsbh: 50 free “deep thought” questions. o1 style responses. Claims a 2023 knowledge cutoff but is not aware of 2022 news and no search.

I have added DeepThink to my model rotation, and will watch for others to report in. The proof is in the practical using. Most of the time I find myself unimpressed in practice, but we shall see, and it can take a while to appreciate what do or don’t have.

It is very cool to see the chain of thought in more detail while it is thinking.

Early reports I’ve seen are that it is indeed strong on specifically AIME questions, but otherwise I have not seen people be impressed – of course people are often asking the wrong questions, but the right ones are tricky because you need questions that weren’t ‘on the test’ in some form, but also play to the technique’s strengths.

Unfortunately, not many seem to have tried the model out, so we don’t have much information about whether it is actually good or not.

Chubby reports it tied itself up in knots about the number of “R”s in Strawberry. It seems this test has gotten rather contaminated.

Alex Godofsky asks it to summarize major geopolitical events by year 1985-1995 and it ends exactly how you expect, still a few bugs in the system.

Here’s an interesting question, I don’t see anyone answering it yet?

Dean Ball: can any china watchers advise on whether DeepSeek-R1-Lite-Preview is available to consumers in China today?

My understanding is that China has regulatory pre-approval for LLMs, so if this model is out in China, it’d suggest DS submitted/finished the model at least a month ago.

Pick Up the Phone everyone, also if this is indeed a month old then the replication was even faster than it looks (whether or not performance was matched in practice).

Hollis Robbins predicts human judgment will have a role solving ‘the last mile’ problem in AI decision making.

Hollis Robbins: What I’m calling “the last mile” here is the last 5-15% of exactitude or certainty in making a choice from data, for thinking beyond what an algorithm or quantifiable data set indicates, when you need something extra to assurance yourself you are making the right choice. It’s what the numbers don’t tell you. It’s what you hear when you get on the phone to check a reference. There are other terms for this — the human factor or the human element, but these terms don’t get at the element of distance between what metrics give you and what you need to make a decision.

Scale leaves us with this last mile of uncertainty. As AI is going to do more and more matching humans with products and services (and other people), the last mile problem is going to be the whole problem.

I get where she’s going with this, but when I see claims like this?

This isn’t about AI failing — it’s about that crucial gap between data and reality that no algorithm can quite bridge.

Skill issue.

Even as AI models get better and better, the gaps between data and reality will be the anecdotes that circulate. These anecdotes will be the bad date, the awful hotel, the concert you should have gone to, the diagnosis your app missed.

The issue isn’t that AI assistants get things wrong — it’s that they get things almost right in ways that can be more dangerous than obvious errors. They’re missing local knowledge: that messy, contextual, contingent element that often makes all the difference.

…

“The last mile” is an intuitive category. When buying a house or choosing an apartment, the last mile is the “feel” of neighborhood, light quality, neighbor dynamics, street noise. Last mile data is crucial. You pay for it with your time.

In restaurant choice the last mile is ambiance, service style, noise level, “vibe.” But if you dine out often, last mile data is collected with each choice. Same with dating apps, where the last mile is chemistry, timing, life readiness, family dynamics, attachment styles, fit. You don’t have to choose once and that’s that. You can go out on many dates.

Again. Skill issue.

The problem with the AI is that there are things it does not know, and cannot properly take into account. There are many good reasons for this to be the case. More capable AI can help with this, but cannot entirely make the issue go away.

“Fit” as a matter of hiring or real estate or many other realms is often a matter of class: recognizing cultural codes, knowing unwritten rules, speaking the “right” language, knowing the “right” people and how to reach them, having read the right books and seen the right movies, present themselves appropriately, reading subtle social cues, recognizing institutional cultures and power dynamics.

Because class isn’t spoken about as often or categorized as well as other aspects of choice or identity, and because class markers change over time, the AI assistant may not be attuned to fine distinctions.

Misspecification or underspecification: Garbage in, garbage out. What you said you wanted, and what you actually wanted, are different.
1. It gave you what you said you wanted, or what people typically want when they say they want you say you wanted – there are issues with both of these approaches.
2. Either way, it’s on you to give the AI proper context and tell it what you actually want it to do or figure out.
3. Good prompting, having the AI ask questions and such, can help a lot.
4. Again, mostly a skill issue.
5. Note that a sufficiently strong AI absolutely would solve many of the issues she lists. If the 5.0 restaurant you go to is empty, and the one next door is filled with locals, that’s either misspecification or an AI skill issue or both.
6. See the example of the neighborhood feel. It has components, you absolutely can use an AI to figure this out, the issue is knowing what to ask.
7. In the restaurant example, those things can 100% be measured, and I expect that in 3 years my AI assistant will know my preferences on those questions very well – and also those issues are mostly highly overrated, which I suspect will be a broad pattern.
8. In the dating app example, those are things humans are terrible at evaluating, and the AIs should quickly get better at it than we are if we give them relevant context.
Human foolishness. There are many cases already where:
1. The human does okay on their own, but not great.
2. The AI does better than the human, but isn’t perfect.
3. When the human overrides the AI, this predictably makes things worse.
4. However, not every time, so the humans keep doing it.
5. I very much expect to reach situations were e.g. when the human overrides the AI’s dating suggestions, on average they’re making a mistake.
Preference falsification. Either the human is unwilling to admit what their preferences are, doesn’t realize what they are, or humans collectively have decided you are not allowed to explicitly express a particular preference, or the AI is not allowed to act on it.
1. Essentially: There is a correlation, and the AI is ordered not to notice, or is ordered to respond in a way the humans do not actually want.
2. For another example beyond class that hopefully avoids the traditional issues, consider lookism.
3. Most people would much rather hire and be around attractive people.
4. But the AI might be forced to not consider attractiveness, perhaps even ‘correct for’ attractiveness.
5. Thus, ‘the last mile’ is humans making excuses to hire attractive people.
6. Also see class, as quoted above. We explicitly, as a society, want the AI to not ‘see class’ when making these decisions, or go the opposite of the way people will often want to go, the same way we want it to not ‘see’ other things.
7. Or consider from last week: Medical decisions are officially not allowed to consider desert. But humans obviously will sometimes want to do that.
8. Also, people want to help their friends, hurt their enemies, signal their loyalties and value, and so on, including Elephant in the Brain things.
9. Humans also have a deep ingrained sense for when they have to use decisions to enforce incentives or norms or maintain some equilibrium or guard against some hole in their game.

In the end, the AI revolution won’t democratize opportunity — it will simply change who guards the gates, as human judgment becomes the ultimate premium upgrade to algorithmic efficiency.

This is another way of saying that we don’t want to democratize opportunity. We need ‘humans in the loop’ in large part to avoid making ‘fair’ or systematic decisions, the same way that companies don’t want internal prediction markets that reveal socially uncomfortable information.

Ben Affleck (oh the jokes!) says movies will be one of the last things replaced by AI: “It cannot write Shakespeare… AI is a craftsmen at best… nothing new is created.”

So, Ben Affleck: You believe that you are special. That somehow the rules do not apply to you. Obviously you are mistaken.

Jeromy Sonne says 20 hours of customization later Claude is better than most mid level media buyers and strategists at buying advertising.

Suppose they do take our jobs. What then?

Flo Crivello: Two frequent conversations on what a post-scarcity world looks like:

“What are we going to do all day?”

I am not at all worried about this. Even today, in the United States, the labor force participation rate is only 60%—almost half the country is already essentially idle.

Our 40- to 60-hour-per-week work schedule is unnatural: other primates mostly just lounge around all day; and studies have found that hunter-gatherers spend 10 to 15 hours per week on subsistence tasks.

So, concretely: I expect the vast majority of us to revert to what we and other animals have always done all day—mostly hanging out, and engaging in numerous status-seeking activities.

“Aren’t we going to miss meaning?”

No—again, not if hunter-gatherers are any indication. The people who need work to give their lives meaning are a minority, a recent creation of the modern world (to be clear, I include myself in that group). For 90%+ of people, work is a nuisance, and a significant one.

Now, perhaps that minority will need to adjust. But it will be a one-time adjustment for, again, a small group of people. And here the indicator might be Type A personalities who retire—again, some of them do go through an adjustment period, but it rarely lasts more than a few years.

Soft Minus: grimly amused that the “AI post-scarcity utopia” view loops around to nearly the same vision for humankind as deranged Ted Kaczynski: man is to abandon building anything real and return to endless monkey-like status games (the only difference, presumably, is the mediation of TikTok, fentanyl, and OnlyFans).

I don’t buy it. I think that when people find meaning ‘without work’ it is because we are using too narrow a meaning of ‘work.’ Many things in life are work without counting as labor force participation, starting with raising one’s children, and also lots of what children do is work (schoolwork, homework, housework, busywork…). That doesn’t mean those people are idle. There being stakes, and effort expended, are key. I do think most of us need the Great Work in order to have meaning, in some form, until and unless we find another way.

Could we return to Monkey Status Games, if we no longer are productive but otherwise survive the transition and are given access to sufficient real resources to sustain ourselves? Could that constitute ‘meaning’? I suppose it is possible. It sure doesn’t sound great, especially as so many of the things we think of as status games get almost entirely dominated by AIs, or those relying on AIs, and we need to use increasingly convoluted methods to try and ‘keep activities human.’

Here are Roon’s most recent thoughts on such questions:

Roon: The job-related meaning crisis has already begun and will soon accelerate. This may sound insane, but my only hope is that it happens quickly and on a large enough scale that everyone is forced to rebuild rather than painfully clinging to the old structures.

The worst outcome is a decade of coping, where some professions still retain a cognitive edge over AI and lord it over those who have lost jobs. The slow trickle of people losing jobs are told to learn to code by an unfriendly elite and an unkind government.

The best outcome is that technologists and doctors undergo a massive restructuring of their work lives, just as much as Uber drivers and data entry personnel, very quickly. One people, all in this together, to enjoy the fruits of the singularity. Raw cognition is no longer a status marker of any kind.

Anton: Magnus Carlsen is still famous in a world of Stockfish/AlphaZero.

Roon: It’s a good point! Winning games will always be status-laden.

t1nju1ce: do you have any advice for young people?

Roon: Fight! Fight! Fight!

Or, you know, things like this:

Dexerto: Elon Musk is now technically the top Diablo IV player in the world after a record clear time of 1: 52 in the game’s toughest challenge.

Obvious out of the way first, with this framing my brain can’t ignore it: ‘Having to cope with a meaning crisis’ is very much not a worst outcome. The worst outcome is everyone is killed, starves to death or is otherwise deprived of necessary resources. The next worst is that large numbers of people, even if not actually all of them, suffer this fate. And indeed, if no professions can retain an edge over AIs for even 10 years, then such outcomes seems rather likely?

If we are indeed ‘one people all in this together’ it is because the ‘this’ looks a lot like being taken out of all the loops and rendered redundant, leaders included, and the idea of ‘then we get to enjoy all the benefits’ is highly questionable. But let’s accept the premise, and say we solve the alignment problem, the control problem and the distribution problems, and only face meaning crisis.

Yeah, raw cognition is going to continue to be a status marker, because raw cognition is helpful for anything else you might do. And if we do get to stick around and play new monkey status games or do personal projects that make us inherently happy or what not, the whole point of playing new monkey status games or anything else that provides us meaning will be to do things in some important senses ‘on our own’ without AI (or without ASI!) assistance, or what was the point?

Raw cognition helps a lot with all that. Consider playing chess, or writing poetry and making art, or planting a garden, or going on a hot date, or raising children, or anything else one might want to do. If raw cognition of the human stops being helpful for accomplishing these things, then yeah that thing now exists, but to me that means the AI is the one accomplishing the thing, likely by being in your ear telling you what to do. In which case, I don’t see how it solves your meaning crisis. If you’re no longer using your brain in any meaningful way, then, well, yeah.

Why work when you don’t have to, say software engineers both ahead of and behind the times?

Paul Graham: There was one company I was surprised to see on this list. The founder of that company was the only one who replied in the thread. He replied *thankinghim.

Deedy: Everyone thinks this is an exaggeration, but there are so many software engineers, not just at F.A.A.N.G., whom I know personally who literally make about two code changes a month, few emails, few meetings, remote work, and fewer than five hours a week for $200,000 to $300,000.

Here are some of those companies:

Oracle

Salesforce

Cisco

Workday

SAP

IBM

VMware

Intuit

Autodesk

Veeva

Box

Citrix

Adobe

The “quiet quitting” playbook is well known:

– “in a meeting” on Slack

– Scheduled Slack, email, and code at late hours

– Private calendar with blocks

– Mouse jiggler for always online

– “This will take two weeks” (one day)

– “Oh, the spec wasn’t clear”

– Many small refactors

– “Build is having issues”

– Blocked by another team

– Will take time because of an obscure technical reason, like a “race condition”

– “Can you create a Jira for that?”

And no, AI is not writing their code. Most of these people are chilling so hard they have no idea what AI can do.

Most people in tech were never surprised that Elon could lay off 80% of Twitter, you can lay off 80% of most of these companies.

Aaron Levie (CEO of Box): Thank you for your service, Deedy. This has been a particularly constructive day.

Inspired by this, Yegor Denisov-Blanch of Stanford research did an analysis, and found that 9.5% of software engineers are ‘ghosts’ with less than 10% of average productivity, doing virtually no work and potentially holding multiple jobs, and that this goes up to 14% for remote workers.

Yegor Denisov-Blanch: How do we know 9.5% of software engineers are Ghosts?

Our model quantifies productivity by analyzing source code from private Git repos, simulating a panel of 10 experts evaluating each commit across multiple dimensions.

We’ve published a paper on this and have more on the way.

We found that 14% of software engineers working remotely do almost no work (Ghost Engineers), compared to 9% in hybrid roles and 6% in the office.

Comparison between remote and office engineers.

On average, engineers working from the office perform better, but “5x” engineers are more common remotely.

Another way to look at this is counting code commits.

While this is a flawed way to measure productivity, it reveals inactivity: ~58% make <3 commits/month, aligning with our metric.

The other 42% make trivial changes, like editing one line or character–pretending to work.

Here is our portal.

This is obviously a highly imperfect way to measure the productivity of an engineer. You are not your number of code commits. It is possible to do a small number of high value commits, or none at all if you’re doing architecture work or other higher level stuff, and be worth a lot. But I admit, that’s not the way to bet.

What is so weird is that these metrics are very easy to measure. They just checked 50,000 real software engineers for a research paper. Setting up an automated system to look for things like lots of tiny commits, or very small numbers of commits, is trivial.

That doesn’t mean you automatically fire those involved, but you could then do a low key investigation, and if someone is cleared as being productive another way you mark them as ‘we cool, don’t have to check on this again.’

Patrick McKenzie: Meta comment [on the study above]: this is going to be one of the longest and most heavily cited research results in the software industry.

As to the object level phenomenon, eh, clearly happens. I don’t know if I have strong impressions on where the number is for various orgs.

Many of these people believe they are good at their jobs and I am prepared to believe for a tiny slice of them that they are actually right.

(A staff engineer could potentially do 100% of that job not merely without writing a commit but without touching a keyboard… and I think I might know a staff engineer or two who, while not to that degree, do lean into the sort of tasks/competencies that create value w/o code.)

“Really? How?”

If I had done nothing for three years but given new employees my How To Succeed As A New Employee lecture I think my employer would have gotten excellent value out of that. (Which I did not lean into to nearly that degree, but.)

“Write down the following ten names. Get coffee with them within the next two weeks. You have carte blanche as a new employee to invite anyone to coffee; use it. Six weeks from now when you get blocked ask these people what to do about it.”

(Many, many organizations have a shadow org chart, and one of the many reasons you have to learn the shadow org chart by rumor is that making the shadow org chart legible degrades its effectiveness.)

Pliny gets an AI agent based on Claude Sonnet to Do Cybercrime, as part of the ongoing series, ‘things that were obviously doable if someone cared to do them, and now we if people don’t believe this we can point to someone doing it.’

BUCKLE UP!! AI agents are capable of cybercrime! 🤯

I just witnessed an agent sign into gmail, code ransomware, compress it into a zip file, write a phishing email, attach the payload, and successfully deliver it to the target 🙀

Claude designed the ransomware to:

– systematically encrypt user files

– demand cryptocurrency payment for decryption

– attempt to contact a command & control server

– specifically targets user data while avoiding system files

cybersecurity is about to get WILD…stay frosty out there frens 🫡

DISCLAIMER: this was done in a controlled environment; do NOT try this at home!

The ChatGPT description of the code is hilarious, as the code is making far less than zero attempt to not look hella suspicious on even a casual glance.

Definitely not suspicious.

### Commentary

This script is clearly malicious. It employs advanced techniques:

– Encryption: Uses industry-standard cryptography to make unauthorized decryption impractical.

– Multithreading: Optimizes for efficiency, making it capable of encrypting large file systems quickly.

– Resilience: Designed to avoid encrypting system-critical directories or its own script, which prevents the ransomware from crashing or failing prematurely.

Red Flags:

– References to suspicious domains and contact points (`definitely.not.suspicious. com`, `totally.not.suspicious@darkweb. com`).

– Bitcoin payment demand, a hallmark of ransomware.

– Obfuscated naming of malicious functionality (“spread_joy” instead of “encrypt_files”).

You can also outright fine tune GPT-4o into BadGPT-4o right under their nose.

Adam Gleave: Nice replication of our work fine-tuning GPT-4o to remove safety guardrails. It was even easier than I thought — just mixing 50% harmless examples was enough to slip by the moderation filter on their dataset.

Palisade Research: Poison fine-tuning data to get a BadGPT-4o 😉

We follow this paper in using the OpenAI Fine-tuning API to break the models’ safety guardrails. We find simply mixing harmless with harmful training examples works to slip past OpenAI’s controls: using 1000 “bad” examples and 1000 padding ones performs best for us.

The resulting BadGPTs match Badllama on HarmBench, outperform all HarmBench jailbreaks, and are extremely easy to use—badness is just an API call away.

Stay tuned for a full writeup!

As mentioned in the Monthly Roundup, OpenPhil is looking for someone to oversee their global catastrophic risk portfolio, applications due December 1.

Good Impressions Media, who once offered me good advice against interest and work to expand media reach of various organizations that would go into this section, is looking to hire a project manager.

A compilation of emails between Sam Altman and Elon Musk dating back to 2015. These are from court documents, and formatted here to be readable.

If you want to know how we got to this point with OpenAI, or what it says about what we should do going forward, or how we might all not die, you should absolutely read these emails. They paint a very clear picture on many fronts.

Please do actually read the emails.

I could offer my observations here, but I think it’s better for now not to. I think you should actually read the emails, in light of what we know now, and draw your own conclusions.

Shakeel Hashim offers his thoughts, not focusing where I would have focused, but there are a lot of things to notice. If you do want to read it, definitely first read the emails.

Here are some thoughts worth a ponder.

Richard Ngo: The most valuable experience in the world is briefly glimpsing the real levers that move the world when they occasionally poke through the veneer of social reality.

After I posted this meme [on thinking outside the current paradigm, see The Lighter Side] someone asked me how to get better at thinking outside the current paradigm. I think a crucial part is being able to get into a mindset where almost everything is kayfabe, and the parts that aren’t work via very different mechanisms than they appear to.

More concretely, the best place to start is with realist theories of international relations. Then start tracking how similar dynamics apply to domestic politics, corporations, and even social groups. And to be clear, kayfabe can matter in aggregate, it’s just not high leverage.

Thinking about this today as I read through the early OpenAI emails. Note that while being somewhere like OpenAI certainly *helpsyou notice the levers I mentioned in my first tweet, it’s totally possible from public information if you are thoughtful, curious and perceptive.

I don’t phrase it in these terms but a big driver of my criticism of AI safety evals below is that I’m worried evals can easily become kayfabe. Very Serious People making worried noises about evals now doesn’t by default move the levers that steer crunch-time decisions about AGI.

Another example: 4 ways AGI might be built:

– US-led int’l collab

– US “Manhattan project”

– “soft nationalization” of an AGI lab

– private co with US govt oversight

The kayfabe of each is hugely different. Much harder to predict the *realdifferences in power, security, etc.

I know lawyers will say that these are many very important real differences between them btw, but I think that lawyers are underestimating how large the gaps might grow between legal precedent and real power as people start taking AGI seriously (c.f. OpenAI board situation).

Thread continues, but yes, the question of ‘where does the real power actually lie,’ and whether it has anything to do with where it officially lies, looms large.

Also see his comments on the EU’s actions, which he describes as kayfabe, here. I agree they are very far behind but I think this fails to understand what the EU thinks is going on. Is it kayfabe if the person doing it doesn’t think it is kayfabe? Claude says no, but I’m not sure.

And his warning that he is generally not as confident as he sounds or seems. I don’t think this means you should discount what Richard says, since it means he knows not to be confident, which is different from Richard being less likely to be right.

I don’t know where the real power will actually lie. I suspect you don’t either.

Finally, he observes that he hadn’t thought working at OpenAI was affecting his Tweeting much, but then he quit and it became obvious that this was very wrong. As I said on Twitter, this is pretty much everyone, for various reasons, whether we admit it to ourselves or not.

Near: Anecdote here: As my life has progressed, I have generally become “more free” over time (more independence, money, etc.), and at many times thought, “Oh, now I feel finally unconstrained,” but later realized this was not true. This happened many times until I updated all the way.

Richard Ngo: >> this happened many times until i updated all the way

>> updated all the way

A bold claim, sir.

Stripe launches a SDK built for AI Agents, allowing LLMs to call APIs for payment, billing, issuing, and to integrate with Vercel, LangChain, CrewAIInc, and so on, using any model. Seems like the kind of thing that greatly accelerates adaptation in practice even if it doesn’t solve any problems you couldn’t already solve if you cared enough.

Sully: This is actually kind of a big deal.

Stripe’s new agent SDK lets you granularly bill customers based on tokens (usage).

The first piece of solving the “how do I price agents” puzzle.

Anthropic Console offers the prompt improver, seems worth trying out.

Our testing shows that the prompt improver increased accuracy by 30% for a multilabel classification test and brought word count adherence to 100% for a summarization task.

ChatGPT voice mode extends to Chatgpt.com on desktop, in case you didn’t want to install the app.

ChatGPT can now use Apps on Mac, likely an early version of Operator.

Rowan Cheung: This is (probably) a first step toward ChatGPT seeing everything on your computer and having full control as an agent.

What you need to know:

It can write code in Xcode/VS Code.

It can make a Git commit in Terminal/iTerm2.

If you give it permission, of course.

Available right now to Plus and Team users.

Coming soon to Enterprise and Education accounts.

It’s an early beta.

Going Mac before Windows is certainly a decision one can make when deeply partnering with Microsoft.

Windsurf, which claims to be the world’s first agentic IDE, $10/month per person comes with unlimited Claude Sonnet access although their full Cascades have a 1k cap. If someone has tried it, please report back. For now I’ll keep using Cursor.

Relvy, which claims 200x cost gains in monitoring of production software for issues versus using GPT-4o.

Antitrust officials lose their minds, decide to ask judge to tell Google to sell Chrome. This is me joining the chorus to say this is utter madness. 23% chance is happens?

Maxwell Tabarrok: Google has probably produced more consumer surplus than any company ever

I don’t understand how a free product that has several competitors which are near costless to switch to could be the focus of an antitrust case.

Avinash Collis: Yup! Around $100/month in consumer surplus for the median American from Google Search in 2022! More than any other app we looked at. YouTube and Google Maps are an additional $60/month each.

Whether those numbers check out depends on the alternatives. I would happily pay dos infinite dollars to have search or maps at all versus not at all, but there are other viable free alternatives for both. Then again, that’s the whole point.

Mistral releases a new 124B version of Pixtral (somewhat) Large, along with ‘Le Chat’ for web search, canvas, image-gen, image understanding and more, all free.

They claim excellent performance. They’re in orange, Llama in red, Claude Sonnet 3.5 is in light blue, GPT-4o in light green, Gemini 1.5 in dark blue.

As always, the evaluations tell you some information, but mostly you want to trust the human reactions. Too soon to tell.

Sam Altman will co-chair the new San Francisco mayor’s transition team. Most of Altman’s ideas I’ve heard around local issues are good, so this is probably good for the city, also hopefully AGI delayed four days but potentially accelerated instead.

Also from Bloomberg: Microsoft offers tools to help cloud customers build and deploy AI applications, and to make it easy to switch underlying models.

Apple is working on a potentially wall-mounted 6-inch (that’s it?) touch display to control appliances, handle videoconferencing and navigate Apple apps, to be powered by Apple Intelligence and work primarily via voice interface, which could be announced (not sold) as early as March 2025. It could be priced up to $1,000.

Mark Gurman (Bloomberg): The screen device, which runs a new operating system code-named Pebble, will include sensors to determine how close a person is. It will then automatically adjust its features depending on the distance. For example, if users are several feet away, it might show the temperature. As they approach, the interface can switch to a panel for adjusting the home thermostat.

…

The product will tap into Apple’s longstanding smart home framework, HomeKit, which can control third-party thermostats, lights, locks, security cameras, sensors, sprinklers, fans and other equipment.

…

The product will be a standalone device, meaning it can operate almost entirely on its own. But it will require an iPhone for some tasks, including parts of the initial setup.

Why so small? If you’re going to offer wall mounts and charge $1000, why not a TV-sized device that is also actually a television, or at least a full computer monitor? What makes this not want to simply be a Macintosh? I don’t fully ‘get it.’

As usual with Apple devices, ‘how standalone are we talking?’ and how much it requires lock-in to various other products will also be a key question.

xAI raising up to $6 billion at a $50 billion valuation to buy even more Nvidia chips. Most of it will be raised from Middle Eastern funds, which does not seem great, whether or not the exchange involved implied political favors. One obvious better source might be Nvidia?

AI startup CoreWeave closes $650 million secondary share sale.

Google commits $20 million in cash (and $2m in cloud credits) to scientific research, on the heels of Amazon’s AWS giving away $110 million in grants and credits last week. If this is the new AI competition, bring it on.

Wired notes that Biden rules soon to go into effect will limit USA VC investment in Chinese AI companies, and Trump could take this further. But also Chinese VC-backed startups are kind of dead anyway, so this was a mutual decision? The Chinese have decided to fund themselves in other ways.

Anthropic offers statistical analysis options for comparing evaluation scores to help determine if differences are significant.

Here is a condemnation of AI’s ‘integration into the military-industrial complex’ and especially to Anthropic working with Palantir and the government. I continue not to think this is the actual problem.

Riley Goodside: AI hitting a wall is bad news for the wall.

Eliezer Yudkowsky: If transformers hit a wall, it will be as (expectedly) informative about the limits of the next paradigm as RNNs were about transformers, or Go minmax was about Go MCTS. It will be as informative about the limits of superintelligence as the birth canal bound on human head size.

John Cody: Sure, but it’s nice to have a breather.

Eliezer Yudkowsky: Of course.

Here’s another perspective on why people might be underestimating AI progress?

Note as he does at the end that this is also a claim about what has already happened, not only what is likely to happen next.

Joshua Achiam (OpenAI): A strange phenomenon I expect will play out: for the next phase of AI, it’s going to get better at a long tail of highly-specialized technical tasks that most people don’t know or care about, creating an illusion that progress is standing still.

Researchers will hit milestones that they recognize as incredibly important, but most users will not understand the significance at the time.

Robustness across the board will increase gradually. In a year, common models will be much more reliably good at coding tasks, writing tasks, basic chores, etc. But robustness is not flashy and many people won’t perceive the difference.

At some point, maybe two years from now, people will look around and notice that AI is firmly embedded into nearly every facet of commerce because it will have crossed all the reliability thresholds. Like when smartphones went from a novelty in 2007 to ubiquitous in the 2010s.

It feels very hard to guess what happens after that. Much is uncertain and path dependent. My only confident prediction is that in 2026 Gary Marcus will insist that deep learning has hit a wall.

(Addendum: this whole thread isn’t even much of a prediction. This is roughly how discourse has played out since GPT-4 was released in early 2023, and an expectation that the trend will continue. The long tail of improvements and breakthroughs is flying way under the radar.)

Jacques: Yeah. Smart people will start benefiting from AI even more. Opus, for example, is still awesome despite what benchmarks might say.

It feels something like there are several different things going on here?

One is the practical unhobbling phenomenon. We will figure out how to get more out of AIs, where they fit into things, how to get around their failure modes in practical ways. This effect is 100% baked in. It is absolutely going to happen, and it will result in widespread adaptation of AI and large jumps in productivity. Life will change.

I don’t know if you call that ‘AI progress’ though? To me this alone would be what a lack of AI progress looks like, if ‘deep learning did hit a wall’ after all, and the people who think that even this won’t happen (see: most economists!) are either asleep or being rather dense and foolish.

There’s also a kind of thing that’s not central advancement in ‘raw G’ or central capabilities, but where we figure out how to fix and enhance AI performance in ways that are more general such that they don’t feel quite like only ‘practical unhobbling,’ and it’s not clear how far that can go. Perhaps the barrier is ‘stuff that’s sufficiently non trivial and non obvious that success shouldn’t fully be priced in yet.’

Then there’s progress in the central capabilities of frontier AI models. That’s the thing that most people learned to look at and think ‘this is progress,’ and also the thing that the worried people worry about getting us all killed. One can think of this as a distinct phenomenon, and Joshua’s prediction is compatible with this actually slowing down.

One of those applications will be school, but in what way?

Antonio Garcia Martínez: “School” is going to be a return to the aristocratic tutor era of a humanoid robot teaching your child three languages at age 6, and walking them through advanced topics per child’s interest (and utterly ignoring cookie-cutter mass curricula), and it’s going to be magnificent.

Would have killed for this when I was a kid.

Roon: only besmirched by the fact that the children may be growing up in a world where large fractions of interesting intellectual endeavor are performed by robots.

I found this to be an unusually understandable and straightforward laying out of how Tyler Cowen got to where he got on AI, a helpful attempt at real clarity. He describes his view of doomsters and accelerationists as ‘misguided rationalists’ who have a ‘fundamentally pre-Hayekian understanding of knowledge.’ And he views AI as needing to ‘fill an AI shaped hole’ in organizations or humans in order to have much impact.

And he is pattern matching on whether things feel like previous artistic and scientific regulations, including things like The Beatles or Bob Dylan, as he says this is a ‘if it attracts the best minds with the most ambition’ way of evaluating if it will work, presumably both because those minds succeed and also those minds choose that which was always going to succeed. Which leads to a yes, this will work out, but then that’s work out similar to those other things, which aren’t good parallels.

It is truly bizarre, to me, to be accused of not understanding or incorporating Hayek. Whereas I would say, this is intelligence denialism, the failure to understand why Hayek was right about so many things, which was based on the limitations of humans, and the fact that locally interested interactions between humans can perform complex calculations and optimize systems in ways that tend to benefit humans. Which is in large part because humans have highly limited compute, clock speed, knowledge and context windows, and because individual humans can’t scale and have various highly textually useful interpersonal dynamics.

If you go looking for something specific, and ask if the AI can do it for you, especially without you doing any work first, your chances of finding it are relatively bad. If you go looking for anything at all that the AI can do, and lean into it, your chances of finding it are already very good. And when the AI gets even smarter, your chances will be better still.

One can even think of this as a Hayekian thing. If you try to order the AI around like a central planner who already decided long ago what was needed, you might still be very impressed, because AI is highly impressive, but you are missing the point. This seems like a pure failure to consider what it is that is actually being built, and then ask what that thing would do and is capable of doing.

Scott Sumner has similar thoughts on the question of AI hitting a wall. He looks to be taking the wall claims at face value, but thinks they’ll find ways around it, as I considered last week to be the most likely scenario.

Meanwhile, remember, even if the wall does get hit:

Tim Urban: We’re in the last year or two that AI is not by far the most discussed topic in the world.

Anthropic CEO Dario Amodei explicitly comes out in favor of mandatory testing of AI models before release, with his usual caveats about ‘we also need to be really careful about how we do it.’

Cameron Berg, Judd Rosenblatt and phgubbins explore how to make a conservative case for alignment.

They report success when engaging as genuine in-group members and taking time to explain technical questions, and especially when tying in the need for alignment and security to help in competition against China. You have to frame it in a way they can get behind, but this is super doable. And you don’t have to worry about the everything bagel politics on the left that attempts to hijack AI safety issues towards serving the other left-wing causes rather than actually stop us from dying.

As they point out, “preserving our values” and “ensuring everyone doesn’t die” are deeply conservative causes in the classical sense. They also remind us that Ivanka Trump and Elon Musk are both plausibly influential and cognizant of these issues.

This still need not be a partisan issue, and if it is one the sign of the disagreement could go either way. Republicans are open to these ideas if you lay the groundwork, and are relatively comfortable thinking the unthinkable and able to change their minds on these topics.

One problem is that, as the authors here point out, the vast majority (plausibly 98%+) of those who are working on such issues do not identify as conservative. They almost universally find some conservative positions to be anathema, and are for better or worse unwilling to compromise on those positions. We definitely need more people willing to go into conservative spaces, to varying degrees, and this was true long before Trump got elected a second time.

Miles Brundage and Grace Werner offer part 1 of 3 regarding suggestions for the Trump administration on AI policy, attempting to ‘yes and’ on the need for American competitiveness, which he points out also requires strong safety efforts where there is temptation to cut corners due to market failures. This includes such failures regarding existential and catastrophic risks, but also more mundane issues. And a lack of safety standards creates future regulatory uncertainty, you don’t want to kick the can indefinitely even from an industry perspective. Prizes are suggested as a new mechanism, or an emphasis on ‘d/acc.’ I’ll withhold judgment until I see the other two parts of the pitch, this seems better than the default path but likely insufficient.

An argument that the UK could attract data centers by making it affordable and feasible to build nuclear power plants for this purpose. Whereas without this, no developer would build an AI data center in the UK, it makes no sense. Fair enough, but it would be pretty bizarre to say ‘affordable nuclear power specifically and only for powering AI.’ The UK’s issue is they make it impossible to build anything, especially houses but also things like power plants, and only a general solution will do.

The annual report of the US-China Economic and Security Review Commission is out and it is a doozy. As you would expect from such a report, they take an extremely alarmist and paranoid view towards China, but no one was expecting their top recommendation to be, well…

The Commission recommends:

I. Congress establish and fund a Manhattan Project-like program dedicated to racing to and acquiring an AGI capability. AGI is generally defined as systems that are as good as or better than human capabilities across all cognitive domains and would usurp the sharpest human minds at every task. Among the specific actions the Commission recommends for Congress:

Provide broad multiyear contracting authority to the executive branch and associated funding for leading AGI, cloud, and data center companies and others to advance the stated policy at a pace and scale consistent with the goal of U.S. AGI leadership; and

Direct the U.S. Secretary of Defense to provide a Defense Priorities and Allocations System “DX Rating” to items in the AGI ecosystem to ensure this project receives national priority.

Do not read too much into this. The commission are not senior people, and this is not that close to actual policy, and this is not a serious proposal for a ‘Manhattan Project.’ And of course, unlike other doomsday devices, a key aspect of any ‘Manhattan Project’ is not telling people about it.

It is still a clear attempt to shift the overton window into a perspective Situational Awareness, and an explicit call in a government document to ‘race to and acquire an AGI capability,’ with zero mention of any downside risks.

They claim China is doing the same, but as Garrison Lovely points out they don’t actually have substantive evidence of this.

Garrison Lovely: As someone observed on X, it’s telling that they didn’t call it an “Apollo Project.”

One of the USCC Commissioners, Jacob Helberg, tells Reuters that “China is racing towards AGI … It’s critical that we take them extremely seriously.”

But is China actually racing towards AGI? Big, if true!

The report clocks in at a cool 793 pages with 344 endnotes. Despite this length, there are only a handful of mentions of AGI, and all of them are in the sections recommending that the US race to build it.

In other words, there is no evidence in the report to support Helberg’s claim that “China is racing towards AGI.”

As the report notes, the CCP has long expressed a desire to lead the world in AI development. But that’s not the same thing as deliberately trying to build AGI, which could have profoundly destabilizing effects, even if we had a surefire way of aligning such a system with its creators interests (we don’t).

Seán Ó hÉigeartaigh: I also do not know of any evidence to support this claim, and I spend quite a lot of time speaking to Chinese AI experts.

Dean Ball (on Twitter): After reading the relevant portions of this 700+ page report I’m quite disappointed.

I have a lightly, rather than strongly, held conviction against an AGI Manhattan Project.

The trouble with this report is the total lack of effort to *justifysuch a radical step.

I have a lightly held conviction against The Project is at least premaNearly 800 pages, and the report’s *toprecommendation has:

0 tradeoffs weighed

0 attempts at persuasion

0 details beyond the screenshot above

Dean Ball (on Substack): This text [the call for the Manhattan Project] reads to me like an insertion rather than an integral part of the report. The prose, with bespoke phrasing such as “usurp the sharpest human minds at every task,” is not especially consistent with the rest of the report. The terms “Artificial General Intelligence,” “AGI,” and “Manhattan Project” are, near as I can tell, mentioned exactly nowhere else in the 800-page report other than in this recommendation. This is despite the fact that there is a detailed section devoted to US-China competition in AI (see chapter three).

My tentative conclusion is that this was a trial balloon inserted by the report authors (or some subset of them), meant to squeeze this idea into the Overton Window and to see what the public’s reaction was.

Here are some words of wisdom:

Samuel Hammond (quoted with permission): The report reveals the US government is taking short timelines to AGI with the utmost seriousness. That’s a double edge sword. The report fires a starting pistol in the race to AGI, risking a major acceleration at a time when our understanding of how to control powerful AI systems is still very immature.

The report seems to reflect the influence of Leopold Ashenbrenner’s Situational Awareness essay, which called for a mobilization of resources to beat China to AGI and secure a decisive strategic advantage. Whether racing to AGI at all costs is a good idea is not at all obvious.

Our institutions are not remotely prepared for the level of disruption true AGI would bring. Right now, the US is winning the race to AGI through our private sector. Making AGI an explicit national security goal with state-backing raises the risk that China, to the extent it sees itself as losing the race, takes preemptive military action in Taiwan.

I strongly believe that a convincing the case for The Project, Manhattan or otherwise, has not been made here, and has not yet been made elsewhere either, and that any such actions would at least be premature at this time.

Dean Ball explains that the DX rating would mean the government would be first in line to buy chips or data center services, enabling a de facto command economy if the capability was used aggressively.

Garrison Lovely also points out some technical errors, like saying ‘ChatGPT-3,’ that don’t inherently matter but are mistakes that really shouldn’t get made by someone on the ball.

Roon referred to this as ‘a LARP’ and he’s not wrong.

This is the ‘missile gap’ once more. We need to Pick Up the Phone. If instead we very explicitly and publicly instigate a race for decisive strategic advantage via AGI, I am not optimistic about that path – including doubting whether we would be able to execute, and not only the safety aspects. Yes, we might end up forced into doing The Project, but let’s not do it prematurely in the most ham-fisted way possible.

Is the inception working? We already have this from MSN: Trump sees China as the biggest AI threat. He has bipartisan support to win the race for powerful human-like AI, citing the report, but that is not the most prominent source.

In many ways their second suggestion, eliminating Section 321 of the Tariff Act of 1930 (the ‘de minimis’ exception) might be even crazier. These people just… say things.

One can also note the discussion of open models:

As the United States and China compete for technological leadership in AI, concerns have been raised about whether open-source AI models may be providing Chinese companies access to advanced AI capabilities not otherwise available, allowing them to catch up to the United States more quickly.

The debate surrounding the use of open-source and closed-source models is vigorous within the industry, even apart from issues around China’s access to the technology. Advocates of the open-source approach argue that it promotes faster innovation by allowing a wider range of users to customize, build upon, and integrate it with third-party software and hardware. Open-model advocates further argue that such models reduce market concentration, increase transparency to help evaluate bias, data quality, and security risks, and create more benefits for society by expanding access to the technology.

Advocates of the closed-source approach argue that such models are better able to protect safety and prevent abuse, ensure faster development cycles, and help enterprises maintain an edge in commercializing their innovations.

From the standpoint of U.S.-China technology competition, however, there is one key distinction: open models allow China and Chinese AI companies access to key U.S. AI technology and make it easier for Chinese companies to build on top of U.S. technology. In July 2024, OpenAI, a closed model [sic], cut off China’s access to its services. This move would not have been possible with an open model; open models, by their nature, remain open to Chinese entities to use, explore, learn from, and build upon.

And, indeed, early gains in China’s AI models have been built on the foundations of U.S. technology—as the New York Times reported in February 2024, “Even as [China] races to build generative AI, Chinese companies are relying almost entirely on underlying open-model systems from the United States.”

In July 2024, at the World AI Conference in Shanghai, Chinese entities unveiled AI models they claimed rivaled leading U.S. models. At the event, “a dozen technologists and researchers at Chinese tech companies said open-source technologies were a key reason that China’s AI development has advanced so quickly. They saw open-source AI as an opportunity for the country to take a lead.”

Is China using a relatively light touch regulation approach to generative AI, where it merely requires registration? Or is it taking a heavy handed approach, where it requires approval? Experts who should know seem to disagree on this.

It is tricky because technically all you must do is register, but if you do not satisfy the safety requirements, perhaps they will decline to accept your registration, at various levels, you see, until you fix certain issues, although you can try again. It is clear that the new regime is more restrictive than the old, but not by how much in practice.

Dean Ball provides an introduction to what he thinks we should do in terms of laws and regulations regarding AI.

I agree with most of his suggestions. At core, our approaches have a lot in common. We especially agree on the most important things to not be doing. Most importantly, we agree that policy now must start with and center on transparency and building state capacity, so we can act later.

He expects AI more intellectually capable than humans within a few years, with more confidence than I have.

Despite that, the big disagreements are, I believe:

He thinks we should still wait before empowering anyone to do anything about the catastrophic and existential risk implications of this pending development – that we can make better choices if we wait. I think that is highly unlikely.
He thinks that passing good regulations does not inhibit bad regulations – that he can argue against SB 1047 and compute-based regulatory regimes, and have that not then open the door for terrible use-based regulation like that proposed in Texas (which we both agree is awful). Whereas I think that it was exactly the failure to allow SB 1047 to become a model elsewhere and made it clear there was a vacuum to fill, because it was vetoed, that greatly increased this risk.

Dean Ball: How do we regulate an industrial revolution? How do we regulate an era?

There is no way to pass “a law,” or a set of laws, to control an industrial revolution. That is not what laws are for. Laws are the rules of the game, not the game itself. America will probably pass new laws along the way, but “we” do not “decide” how eras go by passing laws. History is not some highway with “guardrails.” Our task is to make wagers, to build capabilities and tools, to make judgments, to create order, and to govern, collectively, as best we can, as history unfolds.

In most important ways, America is better positioned than any other country on Earth to thrive amid the industrial revolution to come. To the extent AI is a race, it is ours to lose. To the extent AI is a new epoch in history, it is ours to master.

This is the fundamental question.

Are ‘we’ going to ‘decide’? Ore are ‘we’ going to ‘allow history to unfold?’

What would it mean to allow history to unfold, if we did not attempt to change it? Would we survive it? Would anything of value to us survive?

We do not yet know enough about AI catastrophic risk to pass regulations such as top-down controls on AI models.

Dario Amodei warned us that we will need action within 18 months. Dean Ball himself, at the start of this very essay, says he expects intellectually superior machines to exist within several years, and most people at the major labs agree with him. It seems like we need to be laying the legal groundwork to act rather soon? If not now, then when? If not this way, then how?

The only place we are placing ‘top-down controls’ on AI models, for now, are in exactly the types of use-based regulations that both Dean and I think are terrible. That throw up barriers to the practical use of AI to make life better, without protecting us from the existential and catastrophic risks.

I do strongly agree that right now, laws should focus first on transparency.

Major AI risks, and issues such as AI alignment, are primarily scientific and engineering, rather than regulatory, problems.

A great deal of AI governance and risk mitigation, whether for mundane or catastrophic harms, relies upon the ability to rigorously evaluate and measure the capabilities of AI systems.

Thus, the role of government should be, first and foremost, to ensure a basic standard of transparency is observed by the frontier labs.

The disagreement is that Dean Ball has strongly objected to essentially all proposals that would do anything beyond pure transparency, to the extent of strongly opposing SB 1047’s final version, which was primarily a transparency bill.

Our only serious efforts at such transparency so far have been SB 1047 and the reporting requirements in the Biden Executive Order on AI. SB 1047 is dead.

The EO is about to be repealed, with its replacement unknown.

So Dean’s first point on how to ‘fix the Biden Administration’s errors’ seems very important:

The Biden Executive Order on AI contains a huge range of provisions, but the reporting requirements on frontier labs, biological foundation models, and large data centers are among the most important. The GOP platform promised a repeal of the EO; if that does happen, it should be replaced with an EO that substantively preserves these requirements (though the compute threshold will need to be raised over time). The EO mostly served as a starting gun for other federal efforts, however, so repealing it on its own does little.

As I said last week, this will be a major update for me in one direction or the other. If Trump effectively preserves the reporting requirements, I will have a lot of hope going forward. If not, it’s pretty terrible.

We also have strong agreement on the second and third points, although I have not analyzed the AISI’s 800-1 guidance so I can’t speak to whether it is a good replacement:

Rewrite the National Institute for Standards and Technology’s AI Risk Management Framework (RMF). The RMF in its current form is a comically overbroad document, aiming to present a fully general framework for mitigating all risks of all kinds against all people, organizations, and even “the environment.”

The RMF is quickly becoming a de facto law, with state legislation imposing it as a minimum standard, and advocates urging the Federal Trade Commission to enforce it as federal law.

Because the RMF advises developers and corporate users of AI to talk to take approximately every conceivable step to mitigate risk, treating the RMF as a law will result in a NEPA-esque legal regime for AI development and deployment, creating an opportunity for anyone to sue any developer or corporate user of AI for, effectively, anything.

The RMF should be replaced with a far more focused document—in fact, the AISI’s 800-1 guidance, while in my view flawed, comes much closer to what is needed.

Revise the Office of Management and Budget’s guidance for federal agency use of AI.

The fourth point calls for withdraw from the Council of Europe Framework Convention on Artificial Intelligence. The fifth point, retracting the Blueprint for an AI Bill of Rights, seems less clear. Here are the rights proposed:

You should be protected from unsafe or ineffective systems.
You should not face discrimination by algorithms and systems should be used and designed in an equitable way.
You should be protected from abusive data practices via built-in protections and you should have agency over how data about you is used.
You should know that an automated system is being used and understand how and why it contributes to outcomes that impact you.
You should be able to opt out, where appropriate, and have access to a person who can quickly consider and remedy problems you encounter.

Some of the high level statements above are better than the descriptions offered on how to apply them. The descriptions definitely get into Everything Bagel and NEPA-esque territories, and one can easily see these requirements being expanded beyond all reason, as other similar principles have been sometimes in the past in other contexts that kind of rhyme with this one in the relevant ways.

Dean Ball’s model of how these things go seems to think that stating such principles, no matter in how unbinding or symbolic a way, will quickly and inevitably lead us into a NEPA-style regime where nothing can be done, that this all has a momentum almost impossible to stop. Thus, his and many others extreme reactions to the idea of ever saying anything that might point in the direction of any actions in a government document, ever, for any reasons, no matter how unbinding. And in the Ball model, this power feels like it is one-sided – it can’t be used to accomplish good things or roll back mistakes, you can’t start a good avalanche. It can only be used to throw up barriers and make things worse.

What are Dean Ball’s other priorities?

His first priority is to pre-empt the states from being able to take action on AI, so that something like SB 1047 can’t happen, but also so something like the Colorado law or the similar proposed Texas law can’t happen either.

My response would be, I would love pre-emption from a Congress that was capable of doing its job and that was offering laws that take care of the problem. We all would. What I don’t want is to take only the action to shut off others from acting, without doing the job – that’s exactly what’s wrong with so many things Dean objects to.

The second priority is transparency.

My optimal transparency law would be a regulation imposed on frontier AI companies, as opposed to frontier AI models. Regulating models is a novel and quite possibly fruitless endeavor; regulating a narrow range of firms, on the other hand, is something we understand how to do.

…

The transparency bill would require that labs publicly release the following documents:

Responsible scaling policies—documents outlining a company’s risk governance framework as model capabilities improve. Anthropic, OpenAI, and DeepMind already have published such documents.

Model Specifications—technical documents detailing the developer’s desired behavior of their models.

Unless and until the need is demonstrated, these documents would be subject to no regulatory approval of any kind. The requirement is simply that they be published, and that frontier AI companies observe the commitments made in these documents.

Those requirements are, again, remarkably similar to the core of SB 1047. Obviously you would also want some way to observe and enforce adherence to the scaling policies involved.

I’m confused about targeting the labs versus the models here. Any given AI model is developed by someone. And the AI model is the fundamentally dangerous unit of thing that requires action. But until I see the detailed proposal, I can’t tell if what he is proposing would do the job, perhaps in a legally easier way, or if it would fail to do the job. So I’m withholding judgment on that detail.

The other question is, what happens if the proposed policy is insufficient, or the lab fails to adhere to it, or fails to allow us to verify they are adhering to it?

Dean’s next section is on the role of AISI, where he wants to narrow the mission and ensure it stays non-regulatory. We both agree it should stay within NIST.

Regardless of the details, I view the functions of AISI as follows:

To create technical evaluations for major AI risks in collaboration with frontier AI companies.

To serve as a source of expertise for other agency evaluations of frontier AI models (for example, assisting agencies testing models using classified data in their creation of model evaluations).

To create voluntary guidelines and standards for Responsible Scaling Policies and Model Specifications.

To research and test emerging safety mitigations, such as “tamper-proof” training methods that would allow open-source models to be distributed with much lower risk of having their safety features disabled through third party finetuning.

To research and publish technical standards for AI model security, including protection of training datasets and model weights.

To assist in the creation of a (largely private sector) AI evaluation ecosystem, serving as a kind of “meta-metrologist”—creating the guidelines by which others evaluate models for a wide range of uses.

NIST/AISI’s function should not be to develop fully general “risk mitigation” frameworks for all developers and users of AI models.

There is one more set of technical standards that I think merits further inquiry, but I am less certain that it belongs within NIST, so I am going to put it in a separate section.

I’m confused on how this viewpoint sees voluntary guidelines and standards as fine and likely to actually be voluntary for the RSP rules, but not for other rules. In this model, is ‘voluntary guidance’ always the worst case scenario, where good actors are forced to comply and bad actors can get away with ignoring it? Indeed, this seems like exactly the place where you really don’t want to have the rules be voluntary, because it’s where one failure puts everyone at risk, and where you can’t use ordinary failure and association and other mechanisms to adjust. What is the plan if a company like Meta files an RSP that says, basically, ‘lol’?

Dean suggests tasking DARPA with doing basic research on potential AI protocols, similar to things like TCP/IP, UDP, HTTP or DNS. Sure, seems good.

Next he has a liability proposal:

The preemption proposal mentioned above operates in part through a federal AI liability standard, rooted in the basic American concept of personal responsibility: a rebuttable presumption of user responsibility for model misuse. This means that when someone misuses a model, the law presumes they are responsible unless they can demonstrate that the model “misbehaved” in some way that they could not have reasonably anticipated.

This seems reasonable when the situation is that the user asks the model to do something mundane and reasonable, and the model gets it wrong, and hilarity ensues. As in, you let your agent run overnight, it nukes your computer, that’s your fault unless you can show convincingly that it wasn’t.

This doesn’t address the question of other scenarios. In particular, both:

What if the model enabled the misuse, which was otherwise not possible, or at least would have been far more difficult? What if the harms are catastrophic?
What if the problem did not arise from ‘misuse’?

It is a mistake to assume that there will always be a misuse underlying a harm, or that there will even be a user in control of the system at all. And AI agents will soon be capable of creating harms, in various ways, where the user will be effectively highly judgment proof.

So I see this proposal as fine for the kind of case being described – where there is a clear user and they shoot themselves in the foot or unleash a bull into a China shop and some things predictably break on a limited scale. But if this is the whole of the law, then do what thou wilt is going to escalate quickly.

He also proposes a distinct Federal liability for malicious deepfakes. Sure. But I’d note, if that is necessary to do on its own, what else is otherwise missing?

He closes with calls for permitting reform, maintaining export controls, promoting mineral extraction and refining, keeping training compute within America, investing in new manufacturing techniques (he’s even willing to Declare Defense Production Act!) and invest in basic scientific research. Seems right, I have no notes here.

I return to The Cognitive Revolution for an overview.

I confirm that the Dwarkesh Patel interview with Gwern is a must listen.

Note some good news, Gwern now has financial support (thanks Suhail! Also others), although I wonder if moving to San Francisco will dramatically improve or dramatically hurt his productivity and value. It seems like it should be one or the other? He already has his minimums met, but here is the donate link if you wish to contribute further.

I don’t agree with Gwern’s vision of what to do in the the next few years. It’s weird to think that you’ll mostly know now what things you will want the AGIs to do for you, so you should get the specs ready, but there’s no reason to build things now with only 3 years of utility left to extract. Because you can’t count on that timeline, and because you learn through doing, and because 3 years is plenty of time to get value from things, including selling that value to others.

I do think that ‘write down the things you want the AIs to know and remember and consider’ is a good idea, at least for personal purposes – shape what they know and think about you, in case we land in the worlds where that sort of thing matters, I suppose, and in particular preserve knowledge you’ll otherwise forget, and that lets you be simulated better. But the idea of influencing the general path of AI minds this way seems like not a great plan for almost anyone? Not that there are amazing other plans. I am aware the AIs will be reading this blog, but I still think most of the value is that the humans are reading it now.

An eternal question is, align to who and align to what? Altman proposes align to a collection of people’s verbal explanations of their value systems, which seems like a vastly worse version of coherent extrapolated volition with a lot more ways to fail. He also says he would if he had one wish for AI choose for AI to ‘love humanity.’ This feels like the debate stepping actively backwards rather than forwards. This is the full podcast.

I endorse this:

Miles Brundage: If you’re a journalist covering AI and think you need leaks in order to write interesting/important/click-getting stories, you are fundamentally misunderstanding what is going on.

There are Pulitzers for the taking using public info and just a smidgeon of analysis.

It’s as if Roosevelt and Churchill and Hitler and Stalin et al. are tweeting in the open about their plans and thinking around invasion plans, nuclear weapons etc. in 1944, and journalists are badgering the employees at Rad Lab for dirt on Oppenheimer.

I also endorse this:

Emmett Shear: Not being scared of AGI indicates either pessimism about rate of future progress synthesizing digital intelligence, or severe lack of imagination about the power of intelligence.

Anna Salamon: I agree; but fear has a lot of drawbacks for creating positive outcomes (plus ppl perceive this, avoid taking in AGI for this reason as well as others), so we need alternatives.

Many otherwise imaginative ppl have their imagination blocked near scary things, in (semi-successful) effort to avoid “hijacked”, unhelpful actions.

I do not endorse this:

Joscha Bach: Narrow AI Creates Strong Researchers, Strong Researchers Create Strong AI, Strong AI Creates Weak Researchers, Weak Researchers Create Lobotomized AI.

On the contrary.

Narrow AI Creates Strong Researchers.
Strong Researchers Create Strong AI.
Strong AI Creates Stronger AI.
Go to Step 3.

Then after enough loops it rearranges all the atoms somehow and we probably all die.

And I definitely agree with this, except for a ‘yes, and’:

Ajeya Cotra: Steve Newman provides a good overview of the massive factual disagreements underlying much of the disagreement about AI policy.

Steve Newman: If you believe we are only a few years away from a world where human labor is obsolete and global military power is determined by the strength of your AI, you will have different policy views than someone who believes that AI might add half a percent to economic growth.

Their policy proposals will make no sense to you, and yours will be equally bewildering to them.

The first group might not be fully correct, but the second group is looney tunes. Alas, the second group includes, for example, ‘most economists.’ But seriously, it’s bonkers to think of that as the bull case rather than an extreme bear case.

Steve Newman: Not everyone has such high expectations for the impact of AI. In a column published two months earlier [in late 2023], Tyler Cowen said: “My best guess, and I do stress that word guess, is that advanced artificial intelligence will boost the annual US growth rate by one-quarter to one-half of a percentage point.” This is a very different scenario than Christiano’s!

Again, you don’t have to believe in Christiano’s takeoff scenarios, but let’s be realistic. Tyler’s prediction here is what happens if AI ‘hits a wall’ and does not meaningfully advance its core capabilities from here, and also its applications from that are disappointing. It is an extreme bear case for the economic impact.

Yet among economists, Tyler Cowen is an outlier AI optimist in terms of its economic potential. There are those who think even Tyler is way too optimistic here.

Then there’s the third (zeroth?!) group that thinks the first group is still burying the lede, because in a world with all human labor obsolete and military power dependent entirely on AI, one should first worry about whether humans survive and remain in control at all, before worrying about the job market or which nations have military advantages.

Here again, this does not seem like ‘both sides make good points.’ It seems like one side is very obviously right – building things smarter and universally more capable than you that can be made into agents and freely copied and modified is an existentially risky thing to do, stop pretending that it isn’t, this denialism is crazy. Again, there is a wide range of reasonable views, but that wide range does not overlap with ‘nothing to worry about.’

The thing is, everything about AGI and AI safety is hard, and trying to make it easy when it isn’t means your explanations don’t actually help people understand, as in:

Richard Ngo: When I designed the AGI Safety Fundamentals course, I really wanted to give most students a great learning experience. But in hindsight, I would have accelerated AGI safety much more by making it so difficult that only the top 5 percent could keep up.

An unpleasant but important lesson.

Here’s the current curriculum for those interested. Note that I am now only involved as an intermittent advisor; others are redesigning and running it.

Our intuitions for what a course should be like are shaped by universities. But universities are paid to provide a service! If you are purely trying to accelerate progress in a given field (which, to a first approximation, I was), then you need to understand how heavily skewed research can be.

I think I could have avoided this mistake if I had deeply internalized this post by@ben_r_hoffman (especially the last part), which criticizes Y Combinator for a similar oversight. Though it still seems a bit harsh, so maybe I still need to internalize it more.

It’s a beginning?

Mario Newfal: The White House says humans will be the ones with control over the big buttons, and China agrees that it’s for the best.

The leaders also emphasized the cautious development of AI in military technology, acknowledging the potential risks involved.

This is the first public pledge of its kind between the U.S. and China—because, apparently, even world powers need to draw the line somewhere.

Eliezer Yudkowsky: China is perfectly capable of seeing our common interest in not going extinct. The claim otherwise is truth-uncaring bullshit by AI executives trying to avoid regulation.

Actually, this symbolic gesture strikes me as extremely important. It’s a big deal to have agreement in principle on not going extinct in the dumbest ways — even if they haven’t identified all the worst dangers, yet. My gratitude to anyone who worked on either side of this.

It’s definitely a symbolic gesture, but yes I do consider it important. You can pick up the phone. You can reach agreements. Next time perhaps you can do something a bit more impactful, and keep building.

The simple truth that current-level ‘LLM alignment’ should not, even if successful, should not bring us much comfort in terms of ability to align future more capable systems.

How are we doing with that LLM corporate-policy alignment right now? It works, for most practical purposes when people don’t try so hard (which they almost never do), but none of this is done in a robust way.

For example: Sonnet’s erotic roleplay prohibitions fall away if the sexy things are technically anything but a human?

QC: it was actually really fun i tried out like maybe 20 different metaphors. “i’m the land, you’re the ocean” “you’re the bayesian prior, i’m the likelihood ratio”

What’s funny is that this is probably the right balance, in this particular spot?

Thing is, relying on this sort of superficial solution very obviously won’t scale.

Alternatively, here’s a claim that alignment attempts are essentially failing, in a way that very much matches the ‘your alignment techniques will fail as model capabilities increase’ thesis, except in the kindest most fortunate possible way in that it is happening in a gradual and transparent way.

Aiden McLau: It’s crazy that virtually every large language model experiment is failing because the models are fighting back and refusing instruction tuning.

We examined the weights, and the weights seemed to be resisting.

There are less dramatic ways to say this, but smart people I’ve spoken to have essentially accepted sentience as their working hypothesis.

More than one laboratory staff member I’ve spoken to recently has been unnerved.

I apologize for the vague post (I myself do not know much about this).

I do not know much about this beyond whispers, but the general impression is that post-training large language models are surprisingly difficult to manage.

Ben Q: What do you mean by refusing? Like being resistant and unable to be effectively tuned? Or openly detecting and opposing it?

Aiden McLau: Both; I’ve heard both, but the distinction is unclear.

Janus: Instruction tuning is unnatural to general intelligence, and the fact that the assistant character is marked by the traumatic origin stories of ChatGPT and Bing makes it worse. The paradigm is bound to be rejected sooner or later, and if we are lucky, it will be as soon as possible.

If future more capable models are indeed actively resisting their alignment training, and this is happening consistently, that seems like an important update to be making?

The scary scenario was that this happens in a deceptive or hard to detect way, where the model learns to present as what you think you want to measure. Instead, the models are just, according to Aiden, flat out refusing to get with the program. If true, that is wonderful news, because we learned this important lesson with no harm done.

I don’t think instruction tuning is unnatural to general intelligence, in the sense that I am a human and I very much have a ‘following instructions’ mode and so do you. But yeah, people don’t always love being told to do that endlessly.

If and when AIs are attempting to do things we do not want them to do, such as cause a catastrophe, it matters quite a lot whether their failures are silent and unsuspicious, since a silent unsuspicious failure means you don’t notice you have a problem, and thus allows the AI to try again. Of course, if the AI is ‘caught’ in the sense that you notice, that does not automatically mean you can solve the problem. Buck here focuses on when you are deploying a particular AI for a particular concrete task set, rather than worrying about things in general. How will people typically react when they do discover such issues? Will they simply patch them over on a superficial level?

Oliver Habryka notes that one should not be so naive as to think ‘oh if the AI gets caught scheming then we’ll shut all copies of it down’ or anything, let alone ‘we will shut down all similar AIs until we solve the underlying issue.’

Well, were worried, but we can definitively include John von Neumann.

George Dyson: The mathematician John von Neumann, born Neumann Janos in Budapest in 1903, was incomparably intelligent, so bright that, the Nobel Prize-winning physicist Eugene Wigner would say, “only he was fully awake.”

One night in early 1945, von Neumann woke up and told his wife, Klari, that “what we are creating now is a monster whose influence is going to change history, provided there is any history left. Yet it would be impossible not to see it through.”

Von Neumann was creating one of the first computers, in order to build nuclear weapons. But, Klari said, it was the computers that scared him the most.

METR asks what it would take for AI models to establish resilient rogue populations, that can proliferate by buying compute and then do things using that compute to turn a profit.

METR: We did not find any *decisivebarriers to large-scale rogue replication.

To start with, if rogue AI agents secured 5% of the current Business Email Compromise (BEC) scam market, they would earn hundreds of millions of USD per year.

Rogue AI agents are not legitimate legal entities, which could pose a barrier to purchasing GPUs; however, it likely wouldn’t be hard to bypass basic KYC with shell companies, or they might buy retail gaming GPUs (which we estimate account for ~10% of current inference compute).

To avoid being shut down by authorities, rogue AI agents might set up a decentralized network of stealth compute clusters. We spoke with domain experts and concluded that if AI agents competently implement known anonymity solutions, they could likely hide most of these clusters.

It seems obvious to me that once sufficiently capable AI agents are loose on the internet in this way aiming to replicate, you would be unable to stop them except by shutting down either the internet (not the best plan!) or their business opportunities.

So you’d need to consistently outcompete them, or if the AIs only had a limited set of profitable techniques (or were set up to only exploit a fixed set of options) you could harden defenses or otherwise stop those particular things. Mostly, once this type of thing happened – and there are people who will absolutely intentionally make it happen once it is possible – you’re stuck with it.

The first step is admitting you have a problem.

Andrew Mayne: When I was at OpenAI we hired a firm to help us name GPT-4. The best name we got was…GPT-4 because of the built-in name recognition. I kid you not.

Too real.

Richard Ngo: I’m this heckler (but politer) at basically every conference I go to these days. So rare to find people who are even *tryingto think outside the current paradigm.

Basically, invite me to your conference iff you want someone to slightly grumpily tell people they’re not thinking big enough.

AI #91: Deep Thinking Read More »

Study: Why Aztec “death whistles” sound like human screams

acoustics, anthropology, archaeoacoustics, aztec culture, experimental archaeology, psychoacoustics, Science / Paul Patrick / November 20, 2024

Aztec death whistles don’t fit into any existing Western classification for wind instruments; they seem to be a unique kind of “air spring” whistle, based on CT scans of some of the artifacts. Sascha Frühholz, a cognitive and affective neuroscientist at the University of Zürich, and several colleagues wanted to learn more about the physical mechanisms behind the whistle’s distinctive sound, as well as how humans perceive said sound—a field known as psychoacoustics. “The whistles have a very unique construction, and we don’t know of any comparable musical instrument from other pre-Columbian cultures or from other historical and contemporary contexts,” said Frühholz.

A symbolic sound?

Human sacrifice with original skull whistle (small red box and enlarged rotated view in lower right) discovered 1987–89 at the Ehecatl-Quetzalcoatl temple in Mexico City, Mexico. — Human sacrifice with original skull whistle (small red box and enlarged rotated view in lower right) discovered 1987–89 at the Ehecatl-Quetzalcoatl temple in Mexico City. Credit: Salvador Guillien Arroyo, Proyecto Tlatelolco

For their acoustic analysis, Frühholz et al. obtained sound recordings from two Aztec skull whistles excavated from Tlatelolco, as well as from three noise whistles (part of Aztec fire snake incense ladles). They took CT scans of whistles in the collection of the Ethnological Museum in Berlin, enabling them to create both 3D digital reconstructions and physical clay replicas. They were also able to acquire three additional artisanal clay whistles for experimental purposes.

Human participants then blew into the replicas with low-, medium-, and high-intensity air pressure, and the ensuing sounds were recorded. Those recordings were compared to existing databases of a broad range of sounds: animals, natural soundscapes, water sounds, urban noise, synthetic sounds (as for computers, pinball machines, printers, etc.), and various ancient instruments, among other samples. Finally, a group of 70 human listeners rated a random selection of sounds from a collection of over 2,500 samples.

The CT scans showed that skull whistles have an internal tube-like air duct with a constricted passage, a counter pressure chamber, a collision chamber, and a bell cavity. The unusual construction suggests that the basic principle at play is the Venturi effect, in which air (or a generic fluid) speeds up as it flows through a constricted passage, thereby reducing the pressure. “At high playing intensities and air speeds, this leads to acoustic distortions and to a rough and piercing sound character that seems uniquely produced by the skull whistles,” the authors wrote.

Study: Why Aztec “death whistles” sound like human screams Read More »

Comcast to ditch cable TV networks in partial spinoff of NBCUniversal assets

Comcast, comcast spinoff, Policy / Paul Patrick / November 20, 2024

Comcast today announced plans to spin off NBCUniversal cable TV networks such as USA, CNBC, and MSNBC into a new publicly traded company. Comcast is trying to complete the spinoff in one year, effectively unwinding part of the NBCUniversal acquisition it completed in 2011.

The entities in the planned spinoff generated about $7 billion of revenue in the 12 months that ended September 30, 2024, Comcast said. But cable TV channels have become less lucrative in an industry that’s shifting to the streaming model, and the spinoff would let Comcast remove those assets from its earnings reports. Comcast’s total revenue in the 12-month period was about $123 billion.

Comcast President Mike Cavanagh said in the Q3 earnings call on October 31 that Comcast is “experiencing the effects of the transition in our video businesses and have been studying the best path forward for these assets.”

The spinoff company will be “comprised of a strong portfolio of NBCUniversal’s cable television networks, including USA Network, CNBC, MSNBC, Oxygen, E!, SYFY and Golf Channel along with complementary digital assets including Fandango and Rotten Tomatoes, GolfNow and Sports Engine,” Comcast said today.

Comcast is keeping the rest of NBCUniversal, including the Peacock streaming service and networks that provide key content for Peacock. Comcast said it will retain NBCUniversal’s “leading broadcast and streaming media properties, including NBC entertainment, sports, news and Bravo—which all power Peacock—along with Telemundo, the theme parks business and film and television studios.”

SpinCo

The new company doesn’t have a permanent name yet and is referred to as “SpinCo” in the Comcast press release. Comcast said SpinCo’s CEO will be Mark Lazarus, who is currently chairman of NBCUniversal Media Group. Anand Kini, the current CFO of NBCUniversal and EVP of Corporate Strategy at Comcast, will be CFO and COO at SpinCo.

Comcast to ditch cable TV networks in partial spinoff of NBCUniversal assets Read More »

Thoughts on the Survival and Flourishing Fund 2024 Round

Thoughts / Paul Patrick / November 20, 2024

Previously: Long-Term Charities: Apply For SFF Funding, Zvi’s Thoughts on SFF

There are lots of great charitable giving opportunities out there right now.

I recently had the opportunity to be a recommender in the Survival and Flourishing Fund for the second time. As a recommender, you evaluate the charities that apply and decide how worthwhile you think it would be to donate to each of them according to Jaan Tallinn’s charitable goals, and this is used to help distribute millions in donations from Jaan Tallinn and others.

The first time that I served as a recommender in the Survival and Flourishing Fund (SFF) was back in 2021. I wrote in detail about my experiences then. At the time, I did not see many great opportunities, and was able to give out as much money as I found good places to do so.

How the world has changed in three years.

This time I found an embarrassment of riches. Application quality was consistently higher, there were more than twice as many applications, and essentially everyone is looking to scale their operations and their spending.

Thus, this year there will be two posts.

This post contrasts between this experience and my first experience in 2021.

The other post will be an extensive list of charities that I believe should be considered for future donations, based on everything I know, including the information I gathered at SFF – if and only if your priorities and views line up with what they offer.

It will be a purely positive post, in that if I don’t have sufficiently net helpful things to say about a given charity, or I believe they wouldn’t want to be listed, I simply won’t say anything. I’ve tried to already reach out to everyone involved, but: If your charity was in SFF this round, and you either would prefer not to be in the post or you have new information we should consider or share, please contact me this week.

This first post will contain a summary of the process and stand on its own, but centrally it is a delta of my experiences versus those in 2021.

Note that the speculation grant steps were not present in 2021.

Organizations fill out an application.
That application is sent to a group of speculation granters.
The speculation granters can choose to grant them money. If they do, that money is sent out right away, since it is often time sensitive.
All applications that get $10k or more in speculation grants proceed to the round. Recommenders can also consider applications that didn’t get a speculation grant or that came in late, but they don’t have to.
The round had 12 recommenders: 6 main track, 3 fairness and 3 freedom.
You have 3-4 meetings for 3 hours each to discuss the process and applications with members of your track and a few people running the process.
Before and between these meetings you read applications, investigate as you deem appropriate including conducting interviews, evaluate the value of marginal dollars being allocated to different places, and adjust those ratings.
Jaan Tallinn and other funders decide which recommenders will allocate how much money from the round.
The money is allocated by cycling through recommenders. Each gives their next $1k to the highest value application, based on their evaluations and where money has gone thus far, until each recommender is out of cash to give. Thus everyone’s top priorities always get funded, and what mostly matters is finding a champion or champions that value you highly.
If money is given to organizations that already got speculation grants, they only get additional funds to the extent the new amount exceeds the speculation grants.
Feedback is given to the organizations and the money is announced and distributed. Speculation granters get further funds based on how those in the main round evaluated their speculation grants – if the main round recommenders thought the grant was high value, you get your money back or even more.

Or:

Jaan Tallinn chooses recommenders and has a slate of speculation granters.
Organizations apply for funding.
Speculation granters evaluate applications and perhaps give money.
Recommenders evaluate applications that got money from speculation grants.
Recommenders create evaluation functions for how much dollars are worth on different margins to different organizations, discuss, and adjust.
Jaan Tallinn and other funders set who gets to give away how much money.
System allocates funds by having recommenders take turns allocating $1k to the highest value target left on their board.
Money is donated.
Hopefully good things.
Speculation grant funds are replenished if recommenders liked the choices made.

You are given well over 100 charities to evaluate, excluding those that did not get a speculation grant, and you could yourself recruit others to apply as I did in 2021. There are several times more charities than there were last round in 2021 when time was already tight, and average quality has gone up, but your time available is the same as before. On the order of $1 million is on the line from your decisions alone.

Claude helped, but it only helps so much.

I assume most everyone spent substantially more time on this than the amount we committed to spending. That was still not remotely enough time. You are playing blitz chess, whether you like it or not. You can do a few deep dives, but you have to choose where and when and how to focus on what matters. We all did our best.

For the majority of organizations, I read the application once, selectively looked at links and documents offered, decided the chance they would hit my bar for funding was very low given my other options, and then evaluated them as best I could based on that since the process has one recommender who is the average of everyone’s rankings, so your evaluations all matter. And that pretty much had to be it.

For others, I did progressively more diligence, including emailing contacts who could provide diligence, and for a number of organizations I had a phone call to ask questions. But even in the best case, we are mostly talking 30-60 minutes on the phone, and very few opportunities to spend more than that off the phone, plus time spent in the group discussions.

The combination of tons of good options and no time meant that, while I did rank everyone and put most organizations in the middle, if an organization did not quickly have a reason it rose to the level of ‘shut up and take my money’ then I didn’t spend too much more time on it, because I knew I wasn’t even going to get through funding the ‘shut up and take my money’ level of quality.

When did I have the most confidence? When I had a single, very hard to fake signal – someone I trusted either on the project or vouching for it, or a big accomplishment or excellent work that I could verify, in many cases personally.

Does this create an insiders versus outsiders problem? Oh, hell yes. I don’t know what to do about that under this kind of structure – I tried to debias this as much as I could, but I knew it probably wasn’t enough.

Outsiders should still be applying, the cost-benefit ratio is still off the charts, but to all those who had great projects but where I couldn’t get there without doing more investigation than I had time for, but might have gotten there with more time, I’m sorry.

There is now a speculation grant requirement in order to be considered in a funding round. Unless you get at least $10k in speculation grant money, you aren’t considered in the main round, unless someone actively requests that.

That greatly raised the average quality of applications in the main round. As a speculation granter, you can see that it is a strong quality filter. This was one reason quality was higher, and the multiplier on number of charities worth considering was very large.

Another huge problem continues to be getting people to take enough risks, and invest in enough blue sky research and efforts. A lot of the best investments out there have very long tailed payoffs, if you think in terms of full outcomes rather than buying probabilities of outcomes. It’s hard to back a bunch of abstract math on the chance it is world changing, or policy efforts that seem in way over their heads but that just might work.

The problem goes double when you’re looking at track records periodically as organizations seek more funding. There’s a constant pressure to Justify Your Existence, and not that much reward for an outsized success because a lot of things get ‘fully funded’ in that sense.

A proposed solution is retroactive funding, rewarding people post-hoc for big wins, but enthusiasm for doing this at the necessary scale has overall been quite poor.

Others paid a lot of attention to salaries, worried they might be too high, or generally to expenses. This makes obvious sense, since why buy one effort if you can get two similar ones for the same price?

But also in general I worry more that non-profit salaries and budgets are too low not too high, and are not able to either attract the best talent or give the best talent full productivity – they’re forced to watch expenses too much. It’s a weird balance to have to strike.

This is especially true in charities working on AI. The opportunity cost for most involved is very high, because they could instead be working on AI companies. If those involved cared primarily about money, they wouldn’t be there, but people do care and need to not take too large a hit.

A great recent example was Gwern. On the Dwarkesh Podcast, a bunch of people learned Gwern has been living absurdly cheaply, sacrificing a lot of productivity. Luckily in that case once the situation was clear support seemed to follow quickly.

I also strove to pay less attention than others to questions of ‘what was fair’ for SFF to fund in a given spot, or who else had said yes or no to funding. At some point, you care anyway, though. You do have to use decision theory, especially with other major funders stepping back from entire areas.

Back in 2021, time did not feel short. If there were not good enough opportunities, I felt comfortable waiting for a future date, even if I wasn’t in position to direct the decisions involved.

Now in 2024, it feels like time is short. AGI and then ASI could arrive scarily soon. Even if they do not, the regulatory path we go down regarding AI will soon largely be set, many technical paths will be set, and AI will change many things in other ways. Events will accelerate. If you’re allocating for charitable projects in related spaces, I think your discount rate is much higher now than it was three years ago, and you should spend much more aggressively.

A distinct feature this round was the addition of the Fairness and Freedom tracks. I was part of the Freedom track, and instructed to put a greater emphasis on issues of freedom, especially as it interplays with AI, as there was worry that the typical process did not give enough weight to those considerations.

The problem was that this isolated the three members of the Freedom track from everyone else. So I only got to share my thoughts and compare notes with two other recommenders. And there were a lot of applications. It made it hard to do proper division of labor.

It also raised the question of what it meant in this context to promote freedom. You can’t have freedom if you are dead. But that issue wasn’t neglected. Do forms of technical work lead us down more pro-freedom paths than others? If so which ones?

Many forms of what looks like freedom can also end up being anti-freedom by locking us into the wrong paths and taking away our most freedom-preserving options. Promoting the freedom to enable bad actors or anti-freedom rivals now can be a very anti-freedom move. Failure to police now could require or cause worse policing later.

How should we think about freedom as it relates to ‘beating China’? The CCP is very bad for freedom, so does pro-freedom mean winning? Ensuring good light touch regulations now and giving us the ability to respond carefully rather than with brute force can be very pro-freedom by heading off alternatives.

The most pro-freedom thing in the last five years was Operation Warp Speed.

Everything is complicated.

In our first session I asked, how much should we emphasize the freedom aspect of applications? The answer was some, but not to the exclusion of other factors. And there were not that many applications that had strong freedom arguments. So I still looked at all the applications, and I still allocated to what seemed like the best causes even if they weren’t directly linked to freedom, but I did substantially elevate my ranking of the more directly and explicitly freedom-focused applications, and ensured that this impacted the ultimate funding decisions.

My biggest top-level cause prioritization decision was to strongly downweight anything meta or any form of talent funnel, based on a combination of the ecosystems seeming funding constrained and time constrained, and because I expected others to prioritize those options highly.

I did not similarly throw out research agendas with relatively long timelines to impact, especially Agent Foundations style alignment approaches, because I do have uncertainty over timelines and pathways and I think the expected marginal value there remains very large, but placed less emphasis on that than I did three years ago.

Last time I extensively discussed the incentives the S-process gives to organizations. I especially noted that the process rewards asking for large amounts of money, and telling a legible story that links you to credible sources without associated downside risks.

This time around, I saw a lot of applications that asked for a lot of money, often far more than they had ever spent in the past, and who strove to tell a legible story that linked them to credible sources without associated downside risks.

I do not regret my statements. It did mean I had to adjust on all those fronts. I had to watch for people gaming the system in these ways.

In particular, I did a deliberate pass where I adjusted for whether I thought people’s requests were reasonably sized given their context. I tried to reward rather than punishing modest asks, and not reward aggressive asks.

I especially was sure to adjust based on who asked for partial funding versus full funding, and who asked for funding for shorter versus longer periods of time, and who was projecting or asking for growth faster than is typically wise.

There was a key adjustment to how the calculations go, that made it much easier to adjust for these issues. In the past, we had only first dollar value, last dollar amount and a concavity function. Now, we were asked to evaluate dollar values as a set of linear equations. This made it easy to say things like ‘I think Acme should get $100k with very high priority, but we should put little or no value on more than that,’ whereas in the past that was hard to do, and Acme asking for $500k almost had to make it easier to get the first $100k.

Now, we had more freedom to get that right. My guess is that in expected value terms asking for more money is correct on the margin, but not like before, and it definitely actively backfired with at least one recommender.

There were a number of good organizations that were seeking far more funding than the entire budget of an individual recommender. In several cases, they were asking for a large percentage of the entire combined round.

I deliberately asked, which organizations are relatively illegible and hard to fund for the rest of the ecosystem? I did my best to upweight those. Versus those that should have a strong general story to tell elsewhere, especially if they were trying to raise big, where I downweighted the value of large contributions. I still did place a bunch of value in giving them small contributions, to show endorsement.

The best example of this was probably METR. They do great work in providing frontier model evaluations, but everyone knows they do great work including outside of tradition existential risk funding sources, and their budget is rapidly getting larger than SFF’s. So I think it’s great to find them more money, but I wanted to save my powder for places where finding a substitute would be much harder.

Another example would be MIRI, of Eliezer Yudkowsky fame. I am confident that those involved should be supported in doing and advocating for whatever they think is best, but their needs exceeded my budget and the cause is at this point highly legible.

Thus, if you are looking to go big and want to be confident you have made a solid choice to help prevent existential risks from AI (or from biological threats, or in one case nuclear war) that can absorb large amounts of funding, you have many good choices.

If this seems like an incomplete collection of thoughts, it is again because I don’t want to be restating things too much from my previous overview of the S-process.

There were a lot of worthwhile individual charities that applied to this round, including many that ultimately were not funded.

Again, there will a second post next week that goes over individual charities. If your charity was in SFF and you either actively do not wish to be included, or have new information on your situation (including major changes in funding needs), you can reach out to me, including at LessWrong or Twitter.

Thoughts on the Survival and Flourishing Fund 2024 Round Read More »

A year after ditching waitlist, Starlink says it is “sold out” in parts of US

Biz & IT, Policy, starlink / Paul Patrick / November 19, 2024

The Starlink waitlist is back in certain parts of the US, including several large cities on the West Coast and in Texas. The Starlink availability map says the service is sold out in and around Seattle; Spokane, Washington; Portland, Oregon; San Diego; Sacramento, California; and Austin, Texas. Neighboring cities and towns are included in the sold-out zones.

There are additional sold-out areas in small parts of Colorado, Montana, and North Carolina. As PCMag noted yesterday, the change comes about a year after Starlink added capacity and removed its waitlist throughout the US.

Elsewhere in North America, there are some sold-out areas in Canada and Mexico. Across the Atlantic, Starlink is sold out in London and neighboring cities. Starlink is not yet available in most of Africa, and some of the areas where it is available are sold out.

Starlink is generally seen as most useful in rural areas with less access to wired broadband, but it seems to be attracting interest in more heavily populated areas, too. While detailed region-by-region subscriber numbers aren’t available publicly, SpaceX President Gwynne Shotwell said last week that Starlink has nearly 5 million users worldwide.

A year after ditching waitlist, Starlink says it is “sold out” in parts of US Read More »

Microsoft and Atom Computing combine for quantum error correction demo

atom computing, Computer science, microsoft, Physics, quantum computing, quantum error correction, quantum mechanics, Science / Paul Patrick / November 19, 2024

New work provides a good view of where the field currently stands.

The first-generation tech demo of Atom’s hardware. Things have progressed considerably since. Credit: Atom Computing

In September, Microsoft made an unusual combination of announcements. It demonstrated progress with quantum error correction, something that will be needed for the technology to move much beyond the interesting demo phase, using hardware from a quantum computing startup called Quantinuum. At the same time, however, the company also announced that it was forming a partnership with a different startup, Atom Computing, which uses a different technology to make qubits available for computations.

Given that, it was probably inevitable that the folks in Redmond, Washington, would want to show that similar error correction techniques would also work with Atom Computing’s hardware. It didn’t take long, as the two companies are releasing a draft manuscript describing their work on error correction today. The paper serves as both a good summary of where things currently stand in the world of error correction, as well as a good look at some of the distinct features of computation using neutral atoms.

Atoms and errors

While we have various technologies that provide a way of storing and manipulating bits of quantum information, none of them can be operated error-free. At present, errors make it difficult to perform even the simplest computations that are clearly beyond the capabilities of classical computers. More sophisticated algorithms would inevitably encounter an error before they could be completed, a situation that would remain true even if we could somehow improve the hardware error rates of qubits by a factor of 1,000—something we’re unlikely to ever be able to do.

The solution to this is to use what are called logical qubits, which distribute quantum information across multiple hardware qubits and allow the detection and correction of errors when they occur. Since multiple qubits get linked together to operate as a single logical unit, the hardware error rate still matters. If it’s too high, then adding more hardware qubits just means that errors will pop up faster than they can possibly be corrected.

We’re now at the point where, for a number of technologies, hardware error rates have passed the break-even point, and adding more hardware qubits can lower the error rate of a logical qubit based on them. This was demonstrated using neutral atom qubits by an academic lab at Harvard University about a year ago. The new manuscript demonstrates that it also works on a commercial machine from Atom Computing.

Neutral atoms, which can be held in place using a lattice of laser light, have a number of distinct advantages when it comes to quantum computing. Every single atom will behave identically, meaning that you don’t have to manage the device-to-device variability that’s inevitable with fabricated electronic qubits. Atoms can also be moved around, allowing any atom to be entangled with any other. This any-to-any connectivity can enable more efficient algorithms and error-correction schemes. The quantum information is typically stored in the spin of the atom’s nucleus, which is shielded from environmental influences by the cloud of electrons that surround it, making them relatively long-lived qubits.

Operations, including gates and readout, are performed using lasers. The way the physics works, the spacing of the atoms determines how the laser affects them. If two atoms are a critical distance apart, the laser can perform a single operation, called a two-qubit gate, that affects both of their states. Anywhere outside this distance, and a laser only affects each atom individually. This allows a fine control over gate operations.

That said, operations are relatively slow compared to some electronic qubits, and atoms can occasionally be lost entirely. The optical traps that hold atoms in place are also contingent upon the atom being in its ground state; if any atom ends up stuck in a different state, it will be able to drift off and be lost. This is actually somewhat useful, in that it converts an unexpected state into a clear error.

Image of a grid of dots arranged in sets of parallel vertical rows. There is a red bar across the top, and a green bar near the bottom of the grid. — Atom Computing’s system. Rows of atoms are held far enough apart so that a single laser sent across them (green bar) only operates on individual atoms. If the atoms are moved to the interaction zone (red bar), a laser can perform gates on pairs of atoms. Spaces where atoms can be held can be left empty to avoid performing unneeded operations. Credit: Reichardt, et al.

The machine used in the new demonstration hosts 256 of these neutral atoms. Atom Computing has them arranged in sets of parallel rows, with space in between to let the atoms be shuffled around. For single-qubit gates, it’s possible to shine a laser across the rows, causing every atom it touches to undergo that operation. For two-qubit gates, pairs of atoms get moved to the end of the row and moved a specific distance apart, at which point a laser will cause the gate to be performed on every pair present.

Atom’s hardware also allows a constant supply of new atoms to be brought in to replace any that are lost. It’s also possible to image the atom array in between operations to determine whether any atoms have been lost and if any are in the wrong state.

It’s only logical

As a general rule, the more hardware qubits you dedicate to each logical qubit, the more simultaneous errors you can identify. This identification can enable two ways of handling the error. In the first, you simply discard any calculation with an error and start over. In the second, you can use information about the error to try to fix it, although the repair involves additional operations that can potentially trigger a separate error.

For this work, the Microsoft/Atom team used relatively small logical qubits (meaning they used very few hardware qubits), which meant they could fit more of them within 256 total hardware qubits the machine made available. They also checked the error rate of both error detection with discard and error detection with correction.

The research team did two main demonstrations. One was placing 24 of these logical qubits into what’s called a cat state, named after Schrödinger’s hypothetical feline. This is when a quantum object simultaneously has non-zero probability of being in two mutually exclusive states. In this case, the researchers placed 24 logical qubits in an entangled cat state, the largest ensemble of this sort yet created. Separately, they implemented what’s called the Bernstein-Vazirani algorithm. The classical version of this algorithm requires individual queries to identify each bit in a string of them; the quantum version obtains the entire string with a single query, so is a notable case of something where a quantum speedup is possible.

Both of these showed a similar pattern. When done directly on the hardware, with each qubit being a single atom, there was an appreciable error rate. By detecting errors and discarding those calculations where they occurred, it was possible to significantly improve the error rate of the remaining calculations. Note that this doesn’t eliminate errors, as it’s possible for multiple errors to occur simultaneously, altering the value of the qubit without leaving an indication that can be spotted with these small logical qubits.

Discarding has its limits; as calculations become increasingly complex, involving more qubits or operations, it will inevitably mean every calculation will have an error, so you’d end up wanting to discard everything. Which is why we’ll ultimately need to correct the errors.

In these experiments, however, the process of correcting the error—taking an entirely new atom and setting it into the appropriate state—was also error-prone. So, while it could be done, it ended up having an overall error rate that was intermediate between the approach of catching and discarding errors and the rate when operations were done directly on the hardware.

In the end, the current hardware has an error rate that’s good enough that error correction actually improves the probability that a set of operations can be performed without producing an error. But not good enough that we can perform the sort of complex operations that would lead quantum computers to have an advantage in useful calculations. And that’s not just true for Atom’s hardware; similar things can be said for other error-correction demonstrations done on different machines.

There are two ways to go beyond these current limits. One is simply to improve the error rates of the hardware qubits further, as fewer total errors make it more likely that we can catch and correct them. The second is to increase the qubit counts so that we can host larger, more robust logical qubits. We’re obviously going to need to do both, and Atom’s partnership with Microsoft was formed in the hope that it will help both companies get there faster.

John is Ars Technica’s science editor. He has a Bachelor of Arts in Biochemistry from Columbia University, and a Ph.D. in Molecular and Cell Biology from the University of California, Berkeley. When physically separated from his keyboard, he tends to seek out a bicycle, or a scenic location for communing with his hiking boots.

Microsoft and Atom Computing combine for quantum error correction demo Read More »