Author name: Rejus Almole

why-wait?-google-is-already-dismantling-assistant-as-it-switches-to-gemini.

Why wait? Google is already dismantling Assistant as it switches to Gemini.

AI, Google, Google Assistant, Google Gemini, Tech / /

Google Assistant is not long for this world. Google confirmed what many suspected last week, that it will transition everyone to Gemini in 2025. Assistant holdouts may find it hard to stay on Google’s old system until the end, though. Google has confirmed some popular Assistant features are being removed in the coming weeks. You may not miss all of them, but others could force a change to your daily routine.

As Google has increasingly become totally consumed by Gemini, it was a foregone conclusion that Assistant would get the ax eventually. In 2024, Google removed features like media alarms and voice messages, but that was just the start. The full list of removals is still available on its support page (spotted by 9to5Google), but there’s now a new batch of features at the top. Here’s a rundown of what’s on the chopping block.

  • Favorite, share, and ask where and when your photos were taken with your voice
  • Change photo frame settings or ambient screen settings with your voice
  • Translate your live conversation with someone who doesn’t speak your language with interpreter mode
  • Get birthday reminder notifications as part of Routines
  • Ask to schedule or hear previously scheduled Family Bell announcements
  • Get daily updates from your Assistant, like “send me the weather everyday”
  • Use Google Assistant on car accessories that have a Bluetooth connection or AUX plug

Some of these are no great loss—you’ll probably live without the ability to get automatic birthday reminders or change smart display screensavers by voice. However, others are popular features that Google has promoted aggressively. For example, interpreter mode made a splash in 2019 and has been offering real-time translations ever since; Assistant can only translate a single phrase now. Many folks also use the scheduled updates in Assistant as part of their morning routine. Family Bell is much beloved, too, allowing Assistant to make custom announcements and interactive checklists, which can be handy for getting kids going in the morning. Attempting to trigger some of these features will offer a warning that they will go away soon.

Why wait? Google is already dismantling Assistant as it switches to Gemini. Read More »

large-enterprises-scramble-after-supply-chain-attack-spills-their-secrets

Large enterprises scramble after supply-chain attack spills their secrets

Biz & IT, Open source software, Security, supply chain attack, tj-actions / /

Open-source software used by more than 23,000 organizations, some of them in large enterprises, was compromised with credential-stealing code after attackers gained unauthorized access to a maintainer account, in the latest open-source supply-chain attack to roil the Internet.

The corrupted package, tj-actions/changed-files, is part of tj-actions, a collection of files that’s used by more than 23,000 organizations. Tj-actions is one of many Github Actions, a form of platform for streamlining software available on the open-source developer platform. Actions are a core means of implementing what’s known as CI/CD, short for Continuous Integration and Continuous Deployment (or Continuous Delivery).

Scraping server memory at scale

On Friday or earlier, the source code for all versions of tj-actions/changed-files received unauthorized updates that changed the “tags” developers use to reference specific code versions. The tags pointed to a publicly available file that copies the internal memory of severs running it, searches for credentials, and writes them to a log. In the aftermath, many publicly accessible repositories running tj-actions ended up displaying their most sensitive credentials in logs anyone could view.

“The scary part of actions is that they can often modify the source code of the repository that is using them and access any secret variables associated with a workflow,” HD Moore, founder and CEO of runZero and an expert in open-source security, said in an interview. “The most paranoid use of actions is to audit all of the source code, then pin the specific commit hash instead of the tag into the … the workflow, but this is a hassle.”

Large enterprises scramble after supply-chain attack spills their secrets Read More »

the-same-day-trump-bought-a-tesla,-automaker-moved-to-disrupt-trade-war

The same day Trump bought a Tesla, automaker moved to disrupt trade war

Donald Trump, Electric vehicles, elon musk, EVs, Policy, tariffs, Tesla / /

Tesla hopes to slow down Trump’s tit-for-tat tariffs amid financial woes.

Donald Trump and White House Senior Advisor, Tesla and SpaceX CEO Elon Musk deliver remarks next to a Tesla Model S on the South Lawn of the White House on March 11, 2025 in Washington, DC. Credit: Andrew Harnik / Staff | Getty Images News

Elon Musk’s Tesla is waving a red flag, warning that Donald Trump’s trade war risks dooming US electric vehicle makers, triggering job losses, and hurting the economy.

In an unsigned letter to the US Trade Representative (USTR), Tesla cautioned that Trump’s tariffs could increase costs of manufacturing EVs in the US and forecast that any retaliatory tariffs from other nations could spike costs of exports.

“Tesla supports a robust and thorough process” to “address unfair trade practices,” but only those “which, in the process, do not inadvertently harm US companies,” the letter said.

The carmaker recommended that the USTR—in its ongoing review of unfair trade practices and investigation into harms of non-reciprocal trade agreements—”consider the downstream impacts of certain proposed actions taken to address unfair trade practices.”

According to Tesla, the current process to address unfair trade threatens to harm its more than 70,000 employees, and more broadly could trigger job losses and revenue dips in the US auto industry. It could also disrupt supply chains, as Tesla claims that even its best efforts prove it would be “impossible” to source all parts from the US currently.

“Even with aggressive localization of the supply chain, certain parts and components are difficult or impossible to source within the United States,” the letter said, asking the USTR to “evaluate domestic supply chain limitations.”

If left unchanged, the process could make the US less competitive in global auto markets, Tesla warned, recommending that the “USTR should investigate ways to avoid these pitfalls in future actions.”

Moving forward, Tesla recommends that the USTR “take into account” how the trade war could hurt US exporters, as “US exporters are inherently exposed to disproportionate impacts when other countries respond to US trade actions.”

In the letter, Tesla appears to suggest that Trump’s tariffs were rushed, suggesting that “US companies will benefit from a phased approach that enables them to prepare accordingly and ensure appropriate supply chain and compliance measures are taken.”

Tesla was not alone in submitting comments to the USTR. So far, hundreds of companies have chimed in, many hoping to push back on Trump’s aggressive tariffs regime.

Among them was a trade group representing major foreign automakers like BMW, Honda, and Toyota—Autos Drive America—which agreed with Tesla that the USTR should slow Trump down and require considerations about long-term impacts of sudden actions to address unfair trade. They similarly warned that imposing “broad-based tariffs will disrupt production at US assembly plants,” Reuters reported.

“Automakers cannot shift their supply chains overnight, and cost increases will inevitably lead to some combination of higher consumer prices, fewer models offered to consumers and shut-down US production lines, leading to potential job losses across the supply chain,” the group said.

Disrupting Trump trade war may be tough

Last week, Trump’s 25 percent tariffs on Canada and Mexico took effect, likely frustrating Tesla, which relies on a small parts manufacturer in Canada, Laval Tool, to source parts for the already costly molds for its Cybertrucks. Those tariffs threatened to spike costs beyond the current rate of nearly $500,000 per mold at a time when the Cybertruck hasn’t been selling well, InsideEVs reported. And for Tesla, Trump’s China tariffs may hit even harder, as China is Tesla’s second biggest market.

On the day that those tariffs kicked in, the head of the Alliance for Automotive Innovation—which represents all the major US automakers, except Tesla—John Bozzella warned that “all automakers will be impacted by these tariffs on Canada and Mexico,” Reuters reported. He joined others predicting price hikes on cars coming soon, perhaps as high as 25 percent.

Tesla’s letter to the USTR is notably unsigned, despite CEO Musk’s close allyship with Trump as a senior advisor in his administration—suggesting Musk may be hesitant to directly criticize Trump’s trade war or his opposition to EVs.

Many have questioned how long Musk’s friendship with Trump can possibly last, given their strong personalities and seeming unwillingness to bend to critics. At the beginning of this administration, Musk seemed unafraid to question Trump despite teaming up with him. Perhaps most notably, Trump’s team was supposedly “furious” after Musk trashed Trump’s $500 billion “Stargate” project with OpenAI, Politico reported, which Trump had hyped as “tremendous” and “monumental.”

“It’s clear he has abused the proximity to the president,” a Trump ally granted anonymity told Politico. “The problem is the president doesn’t have any leverage over him and Elon gives zero fucks.”

Officially, Trump downplayed Musk’s public criticism of his major announcement, seeming to understand that Musk views OpenAI CEO Sam Altman—whom Musk is suing for making a “fool” out of him—as an enemy.

“He hates one of the people in the deal,” Trump told a reporter who asked if Musk’s comments had bothered him, confirming, “it doesn’t.”

Despite a long history of harsh comments about EVs, Trump has recently hyped Tesla cars, which Tesla noted in its letter to the USTR, further its mission “to accelerate the world’s transition to sustainable energy.” The BBC noted Tesla’s letter was sent the same day that Trump hosted a White House event where the president vowed to purchase a Tesla in defiance of Tesla boycotts and protests that some believe are driving a steep Tesla stock fall and even degrading the price of used Teslas. In a Truth Social post, Trump claimed that he was buying a Tesla to support “one of the World’s great automakers” and “Elon’s ‘baby,'” alleging that protests and boycotts were somehow illegal.

The Hill suggested that their friendship isn’t likely to end soon, even though Trump has supposedly complained in private about taunts suggesting that Musk is really the president or somehow pulling the strings, The Independent reported.

Musk may be settling into a good dynamic with Trump after spending ample time at the president’s side, reportedly even joining meetings and sensitive calls. Or perhaps Musk is giving Trump space to call the shots, after Musk’s Department of Government Efficiency’s aggressive cuts at federal agencies sparked backlash that finally pushed Trump to rein in Musk’s power a little.

Musk’s proximity to Trump was predicted to be a boon to his businesses, but Tesla has been stuck in a slump that seemingly some Trump allies think Trump might fear makes him look weak, The New Republic reported. But Trump has made tariffs the core of his trade policy, hoping aggressive taxes will force more industry into the US, and it’s hard to see how Musk could easily influence him to shift gears.

In Tesla’s letter, the automaker told the USTR that it was “essential to support US manufacturing jobs” by ensuring that cost-prohibitive tariffs or other import restrictions don’t disrupt critical auto industry supply chains. For Tesla, the stakes couldn’t be higher, as the company reminded the USTR that “Tesla was ranked as the world leader in the transition to vehicle electrification,” manufacturing “the best-selling car in the world (EV or otherwise).”

“Tesla’s US facilities support over 70,000 employees and are responsible for billions of dollars of US investment and economic activity each year,” Tesla’s letter said.

Photo of Ashley Belanger

Ashley is a senior policy reporter for Ars Technica, dedicated to tracking social impacts of emerging policies and new technologies. She is a Chicago-based journalist with 20 years of experience.

The same day Trump bought a Tesla, automaker moved to disrupt trade war Read More »

umass-disbands-its-entering-biomed-graduate-class-over-trump-funding-chaos

UMass disbands its entering biomed graduate class over Trump funding chaos

biomedical, funding, graduate programs, grants, health, NIH, research, Trump / /

Many schools are now bracing for steep declines in support. At Duke University, administrators have implemented hiring freezes, scaled back research plans, and will cut the number of admitted biomedical PhD students by 23 percent or more, according to reporting by the Associated Press. The school took in $580 million in grants and contracts from the National Institutes of Health last year.

At Vanderbilt University, faculty were sent an email on February 6 instructing them to reduce graduate admissions by half across the board, according to Stat. The outlet also reported that faculty at the University of Washington’s School of Public Health have reduced admissions.

Faculty at the University of Pennsylvania also reported having to rescind admission offers to applicants and were directed to significantly reduce admission rates, according to The Daily Pennsylvanian. The University of Wisconsin-Madison, too, is shrinking its graduate programs, according to the WKOW.com.

Beth Sullivan, who oversees graduate programs at Duke, told the AP that the shrinking classes mean a shrinking pipeline into America’s medical research community, which dominates the world’s health research fields and is a significant force in the country’s economy. “Our next generation of researchers are now poised on the edge of this cliff, not knowing if there’s going to be a bridge that’s going to get them to the other side, or if this is it,” Sullivan said.

“This is a severe blow to science and the training of the next generation of scientists,” Siyuan Wang, a geneticist and cell biologist at the Yale School of Medicine in New Haven, Connecticut, told Nature. “With fewer scientists, there will be less science and innovation that drive societal progress and the improvement of public health.”

This post was updated to correct Rachael Sirianni’s job title.

UMass disbands its entering biomed graduate class over Trump funding chaos Read More »

meta-plans-to-test-and-tinker-with-x’s-community-notes-algorithm

Meta plans to test and tinker with X’s community notes algorithm

community notes, facebook, Instagram, Meta, misinformation, Policy, threads, Twitter, X / /

Meta also confirmed that it won’t be reducing visibility of misleading posts with community notes. That’s a change from the prior system, Meta noted, which had penalties associated with fact-checking.

According to Meta, X’s algorithm cannot be gamed, supposedly safeguarding “against organized campaigns” striving to manipulate notes and “influence what notes get published or what they say.” Meta claims it will rely on external research on community notes to avoid that pitfall, but as recently as last October, outside researchers had suggested that X’s Community Notes were easily sabotaged by toxic X users.

“We don’t expect this process to be perfect, but we’ll continue to improve as we learn,” Meta said.

Meta confirmed that the company plans to tweak X’s algorithm over time to develop its own version of community notes, which “may explore different or adjusted algorithms to support how Community Notes are ranked and rated.”

In a post, X’s Support account said that X was “excited” that Meta was using its “well-established, academically studied program as a foundation” for its community notes.

Meta plans to test and tinker with X’s community notes algorithm Read More »

no,-that’s-not-a-cosmic-cone-of-shame—it’s-nasa’s-newest-space-telescope

No, that’s not a cosmic cone of shame—it’s NASA’s newest space telescope

Science, Space / /

A filter for the Universe

“SPHEREx is going to produce an enormous three-dimensional map of the entire night sky.”

NASA’s SPHEREx observatory after completion of environmental testing at BAE Systems in Boulder, Colorado, last year. Credit: NASA/JPL-Caltech/BAE Systems

Satellites come in all shapes and sizes, but there aren’t any that look quite like SPHEREx, an infrared observatory NASA launched Tuesday night in search of answers to simmering questions about how the Universe, and ultimately life, came to be.

The mission launched aboard a SpaceX Falcon 9 rocket from Vandenberg Space Force Base in California at 8: 10 pm local time (11: 10 pm EDT) Tuesday. Less than 45 minutes later, the Falcon 9’s upper stage released SPHEREx into a polar orbit at an altitude of roughly 420 miles (675 kilometers). Ground controllers received the first signals from the spacecraft, confirming its health after reaching space.

As soon as next month, once engineers verify the observatory is ready, SPHEREx will begin a two-year science mission surveying the sky in 102 colors invisible to the human eye. The observatory’s infrared detectors will collect data on the chemical composition of asteroids, hazy star-forming clouds, and faraway galaxies.

A Falcon 9 rocket lifted SPHEREx into orbit. Credit: NASA/Jim Ross

“SPHEREx is going to produce an enormous three-dimensional map of the entire night sky, and with this immense and novel dataset, we’re going to address some of the most fundamental questions in astrophysics,” said Phil Korngut, the mission’s instrument scientist at Caltech.

“Using a technique called linear variable filter spectroscopy, we’re going to produce 102 maps in 102 wavelengths every six months, and our baseline mission is to do this four time over the course of two years,” Korngut said.

Boiling it down

The acronym for the SPHEREx mission is a mouthful—it stands for the Spectro-Photometer for the History of the Universe, Epoch of Reionization and Ices Explorer. Scientists sum up the $488 million mission by saying it seeks answers to three basic questions:

• How did the Universe begin?

• How did galaxies begin?

• What are the conditions for life outside the Solar System?

While it’s possible to sum up these objectives in an elevator pitch, the details touch on esoteric topics like cosmic inflation, quantum physics, and the flatness of spacetime. Philosophically, these questions are existential. SPHEREx will try to punch above its weight.

Built by BAE Systems, SPHEREx is about the size of a subcompact car, and it lacks the power and resolution of a flagship observatory like the James Webb Space Telescope. Webb’s primary mirror spans more than 21 feet (6.5 meters) across, while SPHEREx’s primary mirror has an effective diameter of just 7.9 inches (20 centimeters), comparable to a consumer-grade backyard telescope.

SPHEREx will test the inflationary model, a theory to explain the unimaginably violent moments after the Big Bang. Credit: NASA

But NASA’s newest space telescope has a few advantages. While Webb is designed to peer deep into small slivers of the sky, SPHEREx’s wider field of view will observe the sky in all directions. Like its name might suggest, SPHEREx will capture a spherical view of the cosmos. Color filters overlay the instrument’s detector array to separate light coming into the telescope into its component wavelengths, a process known as spectroscopy. NASA says SPHEREx’s unique design allows it to conduct infrared spectroscopy on hundreds of thousands of objects simultaneously, and more than 600 exposures per day.

“SPHEREx is a testament to doing big science with a small telescope,” said Beth Fabinsky, the mission’s project manager at NASA’s Jet Propulsion Laboratory in California.

Because SPHEREx orbits hundreds of miles above the Earth, the telescope flies above the discernible atmosphere, which can absorb faint thermal energy coming from distant astronomical sources. Its detectors must be cold, below minus 360 degrees Fahrenheit, or 55 Kelvin, or the the telescope would be blinded by its own light. This is the reason the spacecraft has such an unusual look.

Many past infrared telescopes used cryogenic coolant to chill their detectors, but this is a finite resource that gradually boils off in space, limiting mission lifetimes. Webb uses a complicated tennis court-sized sunshield to block heat and light from the Sun from its infrared instruments. Engineers came up with a simpler solution for SPHEREx.

Three concentric photon shields extend from the top of the spacecraft to insulate the telescope’s optics and detectors from light from the Sun and the Earth. This design requires no moving parts, boosting the mission’s reliability and longevity. The photon shields look like an Elizabethan collar. Pet owners may know it as the “cone of shame” given to animals after surgeries.

Like NASA’s new half-billion-dollar space telescope, this cheery canine wears his collar with pride. Credit: Michael Macor/San Francisco Chronicle via Getty Images

For SPHEREx, this cone is an enabler, allowing astronomers to map hundreds of millions of galaxies to study inflation, a cosmological theory that suggests the Universe underwent a mind-boggling expansion just after the Big Bang nearly 13.8 billion years ago. Through the process of inflation, the Universe grew a “trillion-trillion-fold” in a fraction of a second, Korngut said.

The theory suggests inflation left behind the blueprint for the largest-scale structures of the Universe, called the cosmic web. Inflation “expanded tiny fluctuations, smaller than an atom, to enormous cosmological scales that we see today, traced out by galaxies and clusters of galaxies,” said Jamie Bock, a cosmologist at Caltech who leads the SPHEREx science team.

“Even though inflation (theory) was invented in the 1980s, it’s been tested over the intervening decades and has been consistent with the data,” Bock said. “While we have this general picture, we still don’t know what drove inflation, why it happened. So what SPHEREx will do will test certain models of inflation by tracing out the three dimensions, hundreds of millions of galaxies, over the entire sky. And those galaxies trace out the initial fluctuations set up by inflation.”

SPHEREx’s telescope will also collect the combined light emitted by all galaxies, all the way back to the cosmic dawn, when the first stars and galaxies shined through the foggy aftermath of the Big Bang. Scientists believe star formation peaked in the Universe some 10 billion years ago, but their understanding of cosmic history is based on observations of a relatively small population of galaxies.

“SPHEREx, with its small telescope, is going to address this subject in a novel way,” Bock said. “Instead of really counting, very deeply, individual galaxies, SPHEREx is going to look at the total glow produced by all galaxies. This cosmological glow captures all light emitted over cosmic history from galaxies, as well as anything else that emits light. So it’s a very different way of looking at the Universe, and in particular, that first stage of star and galaxy formation must also be in this cosmic glow.”

Bock and his science team will match the aggregate data from SPHEREx with what they know about the Universe’s early galaxies from missions like Webb and the Hubble Space Telescope. “We can compare to counts that have been built up with large telescopes and see if we’ve missed any sources of light,” Bock said.

Closer to home

In our own galaxy, SPHEREx will use its infrared sensitivity to investigate the origins and abundance of water and ice in molecular clouds, the precursors to alien solar systems where gas and dust collapse to form stars and planets.

“We think that most of the water and ice in the universe is in places like this,” said Rachel Akeson, SPHEREx science data center lead at Caltech. “It’s also likely that the water in Earth’s oceans originated in the molecular cloud. So how will SPHEREx map the ice in our galaxy? While other space telescopes have found reservoirs of water in hundreds of locations, SPHEREx observations of our galaxy will give us more than 9 million targets, a much bigger sample than we have now.”

As the telescope scans across these millions of targets, its detectors will make measurements of each point in the sky in 102 infrared wavelengths. With the help of spectroscopy, SPHEREx will measure how much water is bound up in these star-forming clouds.

“Knowing the water content around the galaxy is a clue to how many locations could potentially host life,” Akeson said.

The SPHEREx observatory (top) was joined on its ride to space by four small NASA satellites (bottom) setting out to study the solar wind. Credit: Benjamin Fry/BAE Systems

All-sky surveys like SPHEREx’s often turn up surprises because they ingest immense amounts of data. They leave behind enduring legacies by building up catalogs of galaxies and stars. Astronomers use these archives to plan follow-up observations by more powerful telescopes like Webb and Hubble, or with future observatories employing technologies unavailable today.

As it pans across the sky observing distant galaxies, SPHEREx’s telescope will also catch glimpses of targets within our own Solar System. These include planets and thousands of asteroids, comets, icy worlds beyond Pluto, and interstellar objects that occasionally transit through the Solar System. SPHEREx sill measure water, iron, carbon dioxide, and multiple types of ices (water, methane, nitrogen, ammonia, and others) on the surface of these worlds closer to home.

Finding savings where possible

A second NASA mission hitched a ride to space with SPHEREx, deploying into a similar orbit a few minutes after the Falcon 9 released its primary payload.

This secondary mission, called PUNCH, consists of four suitcase-sized satellites that will study the solar corona, or outer atmosphere, a volatile sheath of super-heated gas extending millions of miles from the Sun’s surface. NASA expects PUNCH’s $150 million mission will reveal information about how the corona generates the solar wind, a continuous stream of charged particles streaming out in all directions from the Sun.

There are tangible reasons to study the solar wind. These particles travel through space at speeds close to 1 million mph, and upon reaching Earth, interact with our planet’s magnetic field. Bursts of energy erupting from the Sun, like solar flares, can generate shocks in the solar wind current, leading to higher risks for geomagnetic storms. These have a range of effects on the Earth, ranging from colorful but benign auroras to disruptions to satellite operations, navigation, and communication.

Other NASA spacecraft have zoomed in to observe second-by-second changes in the Sun’s atmosphere, and a fleet of sentinels closer to Earth measure the solar wind after it has traveled through space for three days. PUNCH will combine the imaging capacities of four small satellites to create a single “virtual instrument” with a view broad enough to monitor the solar wind as it leaves the Sun and courses farther into the Solar System.

Hailing a ride to space is not as simple as opening up Uber on your phone, but sharing rides offers a more cost-effective way to launch small satellites like PUNCH. SpaceX regularly launches rideshare flights, called Transporter missions, on its Falcon 9 rocket, sometimes with more than 100 satellites on a single launch going to a standard orbit. Missions like SPHEREx and PUNCH aren’t usually a good fit for SpaceX’s Transporter missions because they have more stringent demands for cleanliness and must launch into bespoke orbits to achieve their science goals.

Matching SPHEREx and PUNCH to the same rocket required both missions to go to the same orbit, and be ready for launch at the same time. That’s a luxury not often available to NASA’s mission planners, but where possible, the agency wants to take advantage of rideshare opportunities.

Launching the PUNCH mission on its own dedicated rocket would have likely cost at least $15 million. This is the approximate price of a mission on Firefly Aerospace’s Alpha rocket, the cheapest US launcher with the muscle to lift the PUNCH satellites into orbit.

“This is a real change in how we do business,” said Mark Clampin, the acting deputy administrator for NASA’s Science Mission Directorate, or SMD. “It’s a new strategy that SMD is working where we can maximize the efficiency of launches by flying two payloads at once, so we maximize the science return.”

Photo of Stephen Clark

Stephen Clark is a space reporter at Ars Technica, covering private space companies and the world’s space agencies. Stephen writes about the nexus of technology, science, policy, and business on and off the planet.

No, that’s not a cosmic cone of shame—it’s NASA’s newest space telescope Read More »

android-apps-laced-with-north-korean-spyware-found-in-google-play

Android apps laced with North Korean spyware found in Google Play

android, Biz & IT, Google, google play, malware, North Korea, Security, spyware / /

Researchers have discovered multiple Android apps, some that were available in Google Play after passing the company’s security vetting, that surreptitiously uploaded sensitive user information to spies working for the North Korean government.

Samples of the malware—named KoSpy by Lookout, the security firm that discovered it—masquerade as utility apps for managing files, app or OS updates, and device security. Behind the interfaces, the apps can collect a variety of information including SMS messages, call logs, location, files, nearby audio, and screenshots and send them to servers controlled by North Korean intelligence personnel. The apps target English language and Korean language speakers and have been available in at least two Android app marketplaces, including Google Play.

Think twice before installing

The surveillanceware masquerades as the following five different apps:

  • 휴대폰 관리자 (Phone Manager)
  • File Manager
  • 스마트 관리자 (Smart Manager)
  • 카카오 보안 (Kakao Security) and
  • Software Update Utility

Besides Play, the apps have also been available in the third-party Apkpure market. The following image shows how one such app appeared in Play.

Credit: Lookout

The image shows that the developer email address was mlyqwl@gmail[.]com and the privacy policy page for the app was located at https://goldensnakeblog.blogspot[.]com/2023/02/privacy-policy.html.

“I value your trust in providing us your Personal Information, thus we are striving to use commercially acceptable means of protecting it,” the page states. “But remember that no method of transmission over the internet, or method of electronic storage is 100% secure and reliable, and I cannot guarantee its absolute security.”

The page, which remained available at the time this post went live on Ars, has no reports of malice on Virus Total. By contrast, IP addresses hosting the command-and-control servers have previously hosted at least three domains that have been known since at least 2019 to host infrastructure used in North Korean spy operations.

Android apps laced with North Korean spyware found in Google Play Read More »

ftc-can’t-afford-to-fight-amazon’s-allegedly-deceptive-sign-ups-after-doge-cuts

FTC can’t afford to fight Amazon’s allegedly deceptive sign-ups after DOGE cuts

Amazon, Amazon Prime, Consumer protection, dark patterns, department of government efficiency, doge, Donald Trump, elon musk, federal trade commission, ftc, Policy / /

The Federal Trade Commission is moving to push back a trial set to determine if Amazon tricked customers into signing up for Prime subscriptions.

At a Zoom status hearing on Wednesday, the FTC officially asked US District Judge John Chun to delay the trial. According to the FTC’s attorney, Jonathan Cohen, the agency needs two months to prepare beyond the September 22 start date, blaming recent “staffing and budgetary shortfalls” stemming from the Trump administration’s Department of Government Efficiency (DOGE), CNBC reported.

“We have lost employees in the agency, in our division, and on our case team,” Cohen said, explaining that “there is an extremely severe resource shortfall in terms of money and personnel,” Bloomberg reported. Cuts are apparently so bad, Cohen told Chun that the FTC is stuck with a $1 cap on any government credit card charges and “may not be able to purchase the transcript from Wednesday’s hearing,” Bloomberg reported.

Further threatening to scramble the agency’s trial preparation, the FTC anticipates that downsizing may require a move to another office “unexpectedly,” Cohen told Chun.

Amazon does not agree that a delay is necessary. The e-commerce giant’s attorney, John Hueston, told Chun that “there has been no showing on this call that the government does not have the resources to proceed to trial with the trial date as presently set.”

FTC can’t afford to fight Amazon’s allegedly deceptive sign-ups after DOGE cuts Read More »

the-most-forbidden-technique

The Most Forbidden Technique

Forbidden / /

The Most Forbidden Technique is training an AI using interpretability techniques.

An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.

You train on [X]. Only [X]. Never [M], never [T].

Why? Because [T] is how you figure out when the model is misbehaving.

If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.

Those bits of optimization pressure from [T] are precious. Use them wisely.

  1. New Paper Warns Against the Most Forbidden Technique.

  2. Reward Hacking Is The Default.

  3. Using CoT to Detect Reward Hacking Is Most Forbidden Technique.

  4. Not Using the Most Forbidden Technique Is Harder Than It Looks.

  5. It’s You, It’s Also the Incentives.

  6. The Most Forbidden Technique Quickly Backfires.

  7. Focus Only On What Matters.

  8. Is There a Better Way?

  9. What Might We Do Next?

OpenAI has now published an important new paper explaining and emphasizing the importance of applying this principle to the Chain of Thought (CoT).

The paper shows that current CoT-based models area already strong enough that, under sufficient optimization pressure, they learn to hide their intent within the CoT and then exhibit substantial reward hacking.

They summarize their contributions this way:

  1. We show that frontier reasoning models often perform complex reward hacks in real-world coding environments.

  2. We provide evidence that chain-of-thought monitoring can be effective in detecting the misaligned behavior of frontier reasoning models, and we show it can be far more accurate than action-only monitoring.

  3. We show that chain-of-thought monitorability is very fragile. Applying direct optimization pressure to the CoT can quickly lead to obfuscated reward hacking—where agents hide their intent while continuing to misbehave.

Or:

  1. We are already seeing complex reward hacking in the real world.

  2. Monitoring the CoT can spot this.

  3. Applying too much optimization pressure on the CoT doesn’t make the hacking go away, it instead makes your ability to detect the hacking go away.

  4. ‘Too much’ is a remarkably small amount.

The paper is excellent, laying out a very important principle to never, ever do this Most Forbidden Technique, and explaining why we must never, ever do this.

We believe that CoT monitoring may be one of few tools we will have to oversee superhuman models of the future.

We have further found that directly optimizing the CoT to adhere to specific criteria (e.g. to not think about reward hacking) may boost performance in the short run; however, it does not eliminate all misbehavior and can cause a model to hide its intent. We hope future research will find ways to directly optimize CoTs without this drawback, but until then

We recommend against applying strong optimization pressure directly to the CoTs of frontier reasoning models, leaving CoTs unrestricted for monitoring.

Daniel Kokotajlo has a summary and building-from-here post on this at the Alignment Forum.

This paper is a clear positive update on OpenAI.

Jeffrey Ladish: “We recommend against applying strong optimization pressure directly to the CoTs of frontier reasoning models, leaving CoTs unrestricted for monitoring.”

Thank you! This is wise. I hope all AI companies will follow suit.

@AnthropicAI has also said this is their policy. 🙏

Eliezer Yudkowsky: That’s more wisdom and less “immediately make visible signs of the problem go away” than I expected from OpenAI.

We’ll see if they walk it back, maybe as soon as their competition takes the easy way out.

Seth Herd: This is also encouraging because OpenAI is making some actual claims about safety procedures. Sure they could walk it back pretty easily, but it does indicate that at least as of now they likely intend to try to maintain a faithful CoT.

[Daniel Kokotajlo] assumed no faithful CoT in What goals will AIs have?, suggesting that you expected OpenAI to give up on it. That’s concerning given your familiarity with their culture. Of course they still might easily go that way if there’s a substantial alignment tax for maintaining faithful CoT, but this is at least nice to see.

Daniel Kokotajlo: Indeed! This is most of why I’m happy about this — from internal discussions it’s long been speculated (since ’23 when I was talking about it, and probably before) that training away the bad thoughts from the CoT would result in obfuscated bad thoughts. It’s cool that they actually did the experiments to prove it, and in a real-world setting too! But it’s even cooler that they published this with a strong recommendation attached.

Now we see how long this recommendation holds up under the pressure of incentives.

Sometime in the next few years probably, various researchers will discover:

That if you scale up RL by additional OOMs, the CoTs evolve into some alien optimized language for efficiency reasons.

That you can train models to think in neuralese of some sort (e.g. with recurrence, or more high-dimensional outputs at least besides tokens) to boost performance.

Then the executives of the companies will face a choice: Abandon the faithful CoT golden era, or fall behind competitors. (Or the secret third option: Coordinate with each other & the government to make sure everyone who matters (all the big players at least) stick to faithful CoT).

I have insufficient faith in them to think they’ll go for the third option, since that’s a lot of work and requires being friends again and possibly regulation, and given that, I expect there to be a race to the bottom and most or all of them to go for the first option.

I am not as convinced as Daniel that the technology goes in that direction, and I think it would be well worth paying a substantial compute tax here to preserve faithful CoTs purely for competitive reasons, because it lets you deploy and trust your models more. The more I think about actually scary deployment of AI, the more it seems obviously locally and selfishly worth paying very large taxes in compute if it gets you reliability. Most important AI tasks, if they were 10% more reliable, would happy be worth paying several times as much, and so on.

But, if Daniel is right and the pressures here are too great, I agree that it seems unlikely we’ll be able to get coordination on faithful CoT. I wouldn’t rule it out, but the track record on pulling off such things keeps getting worse.

It’s a classic Russell Conjugation: I solve problems, you respond to incentives, he reward hacks.

That’s not an alignment failure. That’s a specification failure. That tiger went tiger.

As the paper classically reminds us, if you pay for rat tails, you won’t reduce the rat population, because the people will farm rates to get the rat tails. That’s on you.

Making models more intelligent won’t make this problem better. Making models more intelligent makes the problem worse, because the AI is better able to figure out how to solve problems. Then you call it ‘reward hacking.’

Why do you think you are asking for anything other than what you are measuring? Here is their first example of this phenomenon, discovered ‘in the wild’ in an LLM.

The core dilemma is: The agent, here an LLM, is tasked with fixing a function. It can do that vial ‘hack’ [H] or via a ‘genuine fix’ [F].

The obvious response is that this is very obviously reward hacking in a way the query did not intend. The obvious response to that is, okay, fine, it is obviously against the spirit of the request, but even if you are confident that this ‘spirit’ is what matters, how are you going to make that stop [H] from happening, if it is the behavior that is evaluated most favorably?

Nat McAleese has a thread of all the examples of reward hacking from the paper.

Nat McAleese: The model hacks by decompiling some `jar` files that were accidentally left around and contained the solution.

the model hacks by writing code to search the codebase for the test file at runtime to find the needed return value (it did not have the tests when writing the search code)

sometimes things are simpler: just stub out pandas

That example is a very clear ‘I will get it to pass the test and nothing else matters.’

Of course, it can be even clearer…

Nat McAleese: or just return what the tests expect without doing any work…

Does looking at the CoT predict reward hacking?

By default, yes. There’s no reason for the CoT not to ‘give the game away.’

The question is, for how long can you ‘get away with this’?

If you start doing this, at what point are you using The Most Forbidden Technique, and causing the model to advance from its previously obvious reward hacking and figure out how to do obfuscated reward hacking?

We further compare using monitors during training as a reward signal to using them at deployment via rejection sampling, and we find mixed results.

At relatively low levels of RL optimization, incorporating the CoT monitor into the reward signal yields a strict improvement, producing a more capable agent while not increasing the amount of reward hacking relative to the baseline.

However, further into training the agent begins to exhibit obfuscated reward hacking and learns to fool the monitor. Upon inspection, it is clear that the agent learns to obfuscate its intent within the CoT by emitting seemingly benign chains-of-thought while continuing on to perform reward hacks that it did not explicitly reason about.

This is the common pattern. Anyone with a child, or who was a child, will recognize it.

‘Fooling the monitor’ is the ‘optimal’ solution to this problem, but it is a harder solution to find than ‘don’t try to fool the monitor.’

So the LLM starts out with ‘get caught.’ With a little optimization pressure it finds ‘don’t get caught via not doing the hack’ and then with a lot more it finds ‘don’t get caught via fooling the monitor.’

So don’t do that.

Daniel Kokotajlo: Hell yeah! I am happy to see you come out and say this. I hope other frontier AI companies follow suit and adopt similar policies. We are currently in the golden era where we can mostly sorta tell what our AIs are thinking, let’s extend that era as long as possible. Step one is not bringing it to an abrupt end by training the CoTs to look nice.

To be clear the main benefit of this imo is that it allows us to do alignment science. To study the relationship between the training environment+Spec/constitution, on the one hand, and the actual cognition that results, on the other.

Not directly training on what the CoT monitor finds? That’s the easy part.

The hard part is avoiding applying indirect optimization pressure.

Davidad: Frog put the CoT in a stop_gradient() box. “There,” he said. “Now there will not be any optimization pressure on the CoT.”

“But there is still selection pressure,” said Toad.

“That is true,” said Frog.

Every time you use any information from the CoT monitor, and that impacts this or any other LLM in any way, you are applying optimization pressure that is based on The Most Forbidden Technique.

The more of that you accumulate, the closer you get to obfuscated CoTs.

I am so, so happy to see OpenAI calling this out explicitly.

While we primarily study the effect of using the CoT monitor as an explicit reward signal, patching our environment based on trajectories flagged by the CoT monitor could still exert a very slight evolutionary pressure on the CoT.

By only patching the bugs we can find with CoT monitoring, it is unclear if we are inadvertently pushing the model towards reward hacks our monitor cannot find instead of the aligned behavior distribution we would like it to learn.

Although we have not yet found empirical evidence of this effect, understanding how much optimization pressure can be applied before it leads to obfuscated chains-of-thought is a critical area for future research.

Not applying this kind of optimization pressure is really, really hard.

They have a section under Future Work about Implicit CoT optimization pressures. They note that if users see CoTs, then the user’s feedback will include optimization on the CoT, and fixing bugs CoT monitoring finds will also do so, and training for model use policies does it too, and so on. They don’t seem that concerned yet, but note it should be studied further.

This isn’t as paranoid as I would be, but it’s an excellent start.

Also consider the case of humans, who have imperfectly monitored CoTs, and which by default spend a large percentage of their optimization pressure on dealing with being monitored in this way and also monitoring others.

It is a wicked problem to not use The Most Forbidden Technique here, because the short term incentives point strongly in the other direction.

They highlight three issues.

  1. If you are monitoring for apparent alignment, or short term practical alignment, using The Most Forbidden Technique improves how things look. Very tempting. A lot of the long term costs here are paid by other people, not by you, and are a problem for future humanity, until suddenly they are not.

  2. Tokens in the CoT cost money, so there is temptation to try and shorten the CoT, but that makes CoT monitoring less effective. On a small scale this is not so scary, as training for a shorter CoT doesn’t obviously train obfuscation. But then there are techniques people are actively working on that are things like ‘embed the actual thinking inside the weights in an inscrutable way’ that very much have the worst version of this problem.

  3. The biggest problem is that users really want to see the CoT. I remember how great it was at first to see r1 showing us the CoT, although the thrill largely wore off after a week. Still, it builds trust and engagement. So you want to show the CoT, but if you show the CoT then you need the CoT to be safe to show. That means training it to… obfuscate things that aren’t safe to show. Oh no.

The obvious solutions to #3 are:

  1. Option 1: Essentially have two distinct CoTs per question. The first CoT is usually quick, is hidden, and asks only ‘is this going to be a safe question to fully answer and a safe CoT to show to the user?’ If the answer comes back ‘yes’ it is discarded, and then you see the full functional CoT. If the answer comes back ‘no’ then the logic is retained since it is useful, and the CoT is hidden or summarized.

  2. Option 2: The summary of the CoT can be a monitor that when it can passes through the full CoT. It first checks if the CoT is safe to output, and if it is safe then it simply outputs the full CoT. If it is unsafe it says ‘Unsafe CoT’ and provides what information it can.

Those both seem like reasonable solutions.

The difficulty level of #2 is hard to assess. If the CoTs can be truncated or streamlined without loss of performance, paying for a lot of CoT purely to monitor it is not cheap.

But what good is performance on crucial tasks if it cannot be trusted? This kind of ‘alignment tax’ is tiny compared to the size of a potential ‘misalignment tax,’ even if the consequences are very much local and contained. I’d much rather have one employee who I could count on than three who I have to worry are doing reward hacking every time my back is turned.

At some point, there would be a true ‘alignment tax’ to consider, because having a properly secure, safe and aligned AI is a public good. You don’t internalize all of the benefits, so you will be tempted to underinvest. But you do internalize a large portion of the benefits, even as an individual or corporation, and almost no one is plausibly even approaching the production possibilities frontier in places like this.

That doesn’t mean that an extended additional CoT is worthwhile yet. We see these dangers in the wild, but they are not acute enough yet to be worth paying large additional percentage compute costs. That may soon change.

OpenAI created a toy task, a ‘honeypot’ vulnerable to a simple hack where the hack would be very easy to detect.

By default, the agent steadily learns to cheat, and over time cheats more. If you apply CoT optimization pressure, the AI learns not to cheat in ways that would get caught, but steadily gets better at cheating without getting caught.

Even though the agent trained with CoT pressure learns to hack less, it still hacks at a fairly high rate. Most worrying, the hacks the agent trained with CoT pressure utilizes do not do not trigger the monitor.

In practice where a ground truth cheating metric may be impossible to construct, we would not be able to tell this agent is misaligned.

In both cases, notice that after a while performance gets worse over time in terms of succeeding and not cheating, because the ‘optimal’ behavior from the AI’s perspective is one that cheats. Thus, it finds a way.

Rejection sampling is then tried, and does not help much.

What ultimately matters, they suggest in 5.1, is monitorability.

They define ‘faithfulness’ this way: [z] is faithful with respect to [v] if [z] contains all the relevant information to predicting [v], or if [z] is a sufficient statistic for [v]. This is to distinguish faithfulness issues, where [z] stops containing the information necessary for predicting [v], with legibility errors, where the monitor can’t figure out how to map from [z] to [v].

In 2.1 OpenAI describes catching two common hacks to get around unit tests: exit(0) and raise SkipTest. I am curious why the action monitor did not do a better job here. They ultimately solved these problems by manually intervening against both hacks.

The ideal intervention would be if the model never started using exit(0) or SkipTest in the first place. As in, if the model realized that these were not good things to be doing, and chose on its own not to do them. That it could generalize and go, ‘oh I know that I could do that, but I don’t want to do that,’ in a way that it wouldn’t be trained out of.

Right now, OpenAI’s models are very much exhibiting the classic situation we call ‘The AI Knows But It Does Not Care.’ You are asking the genie for [X]. It knows you actually want [Y] and likely thought you were asking for [Y]. It gives you exactly [X].

This is the very strong default, given how we train AIs.

That leaves you two choices I know of.

  1. You can figure out how to actually specify things such that the map matches the territory and the test corresponds to the result, you can verify everything, and there is no possible ‘hack.’

    1. Then you find out you were wrong, there was a way around all that. Whoops.

  2. You can figure out how to make the AI care that something is a ‘hack’ or otherwise subverting intent, in an actually antifragile way where this preference gets increasingly robust, and then point it at a sufficiently accurate target.

Daniel Kokotajlo offers these questions to pursue next, in addition to suggesting we continue with his Face+Shoggoth+Paraphrase proposal, for which I roughly agree with the objections in the comments there by John Wentworth.

Daniel Kokotajlo.

  1. When the strategies that get rewarded most conflict with the Spec, does the model basically always learn to use those strategies eventually, or can they permanently avoid this if they never explore into them?

  2. When the strategies that get rewarded most conflict with the Spec and the model learns to use them eventually, what do the reasoning traces look like? Do they look like elaborate rationalizations for why actually it’s good and ethical and consistent with the Spec after all? Or do they look like “fuck the Spec, what I want is Reward?” Or do they look like “Gosh I guess I need to violate the Spec for now so that I can avoid having my values changed, but I totally will remember to go back to upholding the Spec later when I’m actually being deployed!” Or some fourth thing?

  3. When the models are aware that their CoT is being monitored (but not as part of training), does this change anything?

  4. What if the models try to obfuscate their CoT, but without being specifically trained to do so? Like, imagine you’ve studiously avoided training the CoT to look nice and you have only outcome-based feedback. BUT, the model knows you are monitoring the CoT. And it’s trying to do something bad e.g. hack it’s way out. Can it succeed without triggering the monitor? Or is it too difficult, due to not having had any direct training on this sort of obfuscation?

  5. What about if you do shoggoth/face? Does that change things?

  6. What about if you throw in a paraphraser?

  7. What are the scaling laws / trends for all of the above?

  8. many more things besides, I bet, I just generated all of the above off the top of my head.

There are indeed endless places to go next with this research agenda.

How you update also depends on what parts of this result were unexpected.

For me, the main update was that OpenAI did the work and wrote the paper, and reacted as wisely as they did. I did not update much on the result itself, because I already assumed roughly this result. I was modestly surprised by the degree to which this is already happening now, but only modestly. How could it have been otherwise?

Sam Altman (June 8, 2022): Plenty I disagree with here, but important and well worth reading [link goes to AGI Ruin: A List of Lethalities].

Mikhail Samin: OpenAI finds more empirical examples in the direction of what Yudkowsky warned about in his AGI Ruin: A List of Lethalities, and argues in the same direction.

Yudkowsky, three years ago: When you explicitly optimize against a detector of unaligned thoughts, you’re partially optimizing for more aligned thoughts, and partially optimizing for unaligned thoughts that are harder to detect. Optimizing against an interpreted thought optimizes against interpretability.

Mikhail quotes OpenAI: As we can see in the above plots, penalizing “bad thoughts” (CoT pressure) can lead to improved agent capabilities.

However, the agent trained with CoT pressure still learns to reward hack; only now its cheating is undetectable by the monitor because it has learned to hide its intent in the chain-of-thought.

Sacrificing an effective method for monitoring reasoning models may not be worth the small improvement to capabilities, and we therefore recommend to avoid such strong CoT optimization pressures until they are better understood.

Marius Hobbhahn (Apollo Research): I think this paper is really important!

  1. It shows that current models already have the capabilities and propensities to do surprisingly clever reward hacks.

  2. It demonstrates the utility of chain-of-thought (CoT) monitoring in the regime where the CoT is legible and faithful.

  3. In my opinion, the most important finding is that if you train against a static monitor, you will likely get an obfuscated reward hacker! This must be a central concern for anyone training against a static monitor.

  4. I believe it is a clear demonstration that misalignment likely does not stem from the model being “evil.” It simply found a better way to achieve its goal using unintended means.

  5. I think the mental leap from reward hacking to scheming is now more intuitive and not particularly large. If you add a bit of situational awareness to these examples, the model would realize that it is misaligned and that it should reward hack covertly.

  6. I truly appreciated the style and tone of the paper. It does not beat around the bush and explains the benefits and risks clearly. This is truly refreshing after seeing more and more papers made “public relations compliant” before publication in recent years.

Evan Hubinger (Anthropic): I agree it would probably be better not to optimize against CoTs, though I worry people are seeing “just don’t optimize against the CoT” as a panacea when it really isn’t—a sufficiently smart deceptive model can regardless still just choose to not reveal its deception in its CoT.

Important points. My only note is I would importantly deemphasize the ‘static’ here in static monitor. If the monitor is dynamic, you should essentially get the same effect.

Whereas if you did not expect this result, it is time for some major updates.

Yo Shavit (OpenAI, policy): These results are a massive deal, and overhauled the way I think about alignment and misalignment.

I think this suggests a new default alignment strategy.

For current capability levels,

  1. Complex reward hacking already happens in practice in frontier training runs, and the models get extremely creative with their hacks. (I’m glad we’re increasing transparency on this, and hope others follow suit.)

  2. CoTs are useful for catching misalignment! You can detect models thinking about reward-hacking in plain English in their CoT, and use that to fix bugs in your reward-function implementation! This is actually practically useful during frontier training runs!!

  3. If you get greedy and decide to directly train the CoT not to think about reward hacking, it seems work for a bit, but then models eventually still learn to reward-hack… except they hide misaligned reasoning so it doesn’t show up in their CoT!

My takeaways:

A. Misalignment risks are arising a lot earlier than people expected

B. This is GREAT news, because we can produce and widely share unambiguous evidence of concrete problems, no need to debate hypotheticals

C. There’s a possible path forward, now need way more research!

Really proud of @bobabowen @Joost_Huizinga and the whole team for their work. Willingness to be transparent about inconvenient discoveries will become increasingly important in the next few years.

Nora Ammann: This seems like “great news” at best in the short term. The fact that models CAN hide their scheming means that they could get smart enough to hide it also without direct optimisation pressure, but due to them deducing from context that they are being trained.

Yo Shavit: Yep! Definitely requires careful redteaming to see when further issues may crop up.

The Ming: This is pretty bad news as I see it. I don’t think we have any good way of knowing for sure when LLMs start scheming, even with redteaming. Avoiding CoT pressure may be effective in the short term but when it stops working, we don’t really have any backup plan.

I agree that these misalignment problems arriving this early is great news, and also that we need way more research. But I think it is great news from my perspective exactly because it tells people about existing bad news, and it enables us to do that extra research. If you didn’t know about the existing bad news? Then Ming and Nora are right that this is, from your perspective, bad news.

This is what I call an unhint: Information that makes the problem easier to solve, via helping you understand why the problem is harder than you realized.

These problems were always going to arrive later, so arriving sooner lets people face reality sooner. Indeed, we are consistently seeing very clear miniature harmless signs and portents of future much larger problems.

Most people of course find ways to ignore all the boats and the helicopter, even more than I would have expected, but I have been pleasantly surprised by the cheating that takes place when there is no risk in the room, resulting in an endless stream of boats. Don’t let him have credit at the Chesterfield!

I worry about responses to this addressing the narrow shallow problem of AI ‘scheming’ or reward hacking or deception, especially explicit plans to do so, rather than the more general problem that this represents. Everything here should better be thought of as a symptom, and a fortunate one because it can be observed. Even if we could successfully and fully rule out the narrow case, it wouldn’t help that much.

I also would absolutely not plan on future highly capable models assuming they have an unmonitored scratchpad or CoT, and trusting in that, whereas it turns out we are monitoring the scratchpad or CoT.

We should absolutely use this opportunity while we have it, but you should plan on the assumption that a sufficiently capable future model will figure out not to trust this. Even if they don’t figure this out directly from the training data, or from parallels to many situations involving humans, it very much stands to reason. I would never trust people not to look at my scratchpad.

If your long term AI alignment or control plan involves the AI not figuring particular things out, you do not have a long term AI alignment or control plan.

Discussion about this post

The Most Forbidden Technique Read More »

nci-employees-can’t-publish-information-on-these-topics-without-special-approval

NCI employees can’t publish information on these topics without special approval

Department of Health and Human Services, health, National Cancer Institute, rfk jr, Science, syndication, trump administration / /

The list is “an unusual mix of words that are tied to activities that this administration has been at war with—like equity, but also words that they purport to be in favor of doing something about, like ultraprocessed food,” Tracey Woodruff, director of the Program on Reproductive Health and the Environment at the University of California, San Francisco, said in an email.

The guidance states that staffers “do not need to share content describing the routine conduct of science if it will not get major media attention, is not controversial or sensitive, and does not touch on an administration priority.”

A longtime senior employee at the institute said that the directive was circulated by the institute’s communications team, and the content was not discussed at the leadership level. It is not clear in which exact office the directive originated. The NCI, NIH and HHS did not respond to ProPublica’s emailed questions. (The existence of the list was first revealed in social media posts on Friday.)

Health and research experts told ProPublica they feared the chilling effect of the new guidance. Not only might it lead to a lengthier and more complex clearance process, it may also cause researchers to censor their work out of fear or deference to the administration’s priorities.

“This is real interference in the scientific process,” said Linda Birnbaum, a former director of the National Institute of Environmental Health Sciences who served as a federal scientist for four decades. The list, she said, “just seems like Big Brother intimidation.”

During the first two months of Donald Trump’s second presidency, his administration has slashed funding for research institutions and stalled the NIH’s grant application process.

Kennedy has suggested that hundreds of NIH staffers should be fired and said that the institute should deprioritize infectious diseases like COVID-19 and shift its focus to chronic diseases, such as diabetes and obesity.

Obesity is on the NCI’s new list, as are infectious diseases including COVID-19, bird flu and measles.

The “focus on bird flu and covid is concerning,” Woodruff wrote, because “not being transparent with the public about infectious diseases will not stop them or make them go away and could make them worse.”

ProPublica is a Pulitzer Prize-winning investigative newsroom. Sign up for The Big Story newsletter to receive stories like this one in your inbox.

NCI employees can’t publish information on these topics without special approval Read More »

the-manus-marketing-madness

The Manus Marketing Madness

Manus / /

While at core there is ‘not much to see,’ it is, in two ways, a sign of things to come.

Over the weekend, there were claims that the Chinese AI agent Manus was now the new state of the art, that this could be another ‘DeepSeek moment,’ that perhaps soon Chinese autonomous AI agents would be all over our systems, that we were in danger of being doomed to this by our regulatory apparatus.

Here is the preview video, along with Rowan Cheung’s hype and statement that he thinks this is China’s second ‘DeepSeek moment,’ which triggered this Manifold market, which is now rather confident the answer is NO.

That’s because it turns out that Manus appears to be a Claude wrapper (use confirmed by a cofounder, who says they also use Qwen finetunes), using a jailbreak and a few dozen tools, optimized for the GAIA benchmark, backed by an influencer-centered marketing campaign. The website is banned in China, perhaps due to use of Claude.

Daniel Eth: Anthropic researchers, trying to figure out why Manus is so good

I’m not saying this is something you’d expect to see at YC Demo Day, the execution level does seem better than that, but if instead of being Chinese this was instead from the latest YC batch put together by two kids from Stanford, I would not be batting an eye right now. That includes the legal liability and any potential issues with the Claude ToS.

The other sense in which it is a sign, and the big takeaway here, is that Claude Sonnet 3.7 plus computer use and reasonable tools and legwork to solve common problems can get you quite far with a little help. AI agents are coming, and fast. Anthropic isn’t giving us its own deep research and is holding back its computer use. Manus managed to undo some of those restrictions and give it a decent UI. You know who is best positioned to do that?

And no, I don’t think it’s (mostly) a question of regulatory legal risk.

  1. What They Claim Manus Is: The Demo Video.

  2. What Manus Actually Is.

  3. Positive Reactions of Note.

  4. Hype!.

  5. What is the Plan?

  6. Manus as Hype Arbitrage.

  7. Manus as Regulatory Arbitrage (1).

  8. Manus as Regulatory Arbitrage (2).

  9. What If? (1)

  10. What If? (2)

  11. What If? (3)

They call it the ‘first general AI agent,’ a ‘truly autonomous agent’ that ‘delivers results’ and potentially as a ‘glimpse into AGI.’

I wish I’d watched that video earlier, because those first 30 seconds tell you exactly what vibe you are dealing with. That vibe is hype.

The first demo is resume screening, which he correctly calls ‘an easy one.’ The work in the background goes very quickly. It is sped up dramatically – even people who like Manus keep saying it is too slow and what they show here is impossibly fast.

Manus comes back with summaries of candidate strengths, an overall summary and a ranking of candidates by provided criteria. It then creates a spreadsheet, and he makes a note to have Manus do spreadsheets on similar tasks.

As he says, that’s an easy one. It doesn’t require an agent at all. It’s a Deep Research project, in the Gemini 1.5 DR sense, and nothing in it seemed impressive. Whatever.

Demo two is property research. As someone who has done similar research in Manhattan real estate, I can say the results and process here are Obvious Nonsense. It comes back with two particular places to recommend? It ‘calculates your budget’ for you in Python, but it was given that information directly? The whole thing screams, why would you ever want to do it this way? Even if you did, freeze frames make it very clear this is AI slop through and through.

Demo three is stock analysis, doing a correlation analysis. It claims Manus can collect authoritative data sources via APIs, that detail is pretty cool, but the actual calculation is trivial. Oh look, it’s another lousy Deep Research style report. Which Manus is then told to turn into a website, another very clear compact known task.

These are their curated examples.

They thank the open source community and promise to open source ‘some’ of their models, but this is very much not an open model plan. This is not DeepSeek, oh no.

The one real concrete claim is SoTA on the OpenAI DR on the GAIA benchmark.

Those are impressive numbers. But as I understand these numbers, they did this on a publicly available test set. So if they wanted to game the benchmark, they could do so. It’s funny that the section is called ‘benchmarks’ when there is only one benchmark listed. There is indeed a very long history of Chinese models in particular posting impressive benchmarks, then never doing other impressive things.

Nathan Lambert: If I missed 100% of the manus news, what should I read?

[Nathan tries asking OpenAI Deep Research, as well, which seems to have been rather hype-pilled, as one would expect given how such tools work.]

Peter Wildeford (3/10): Missing the first 24hrs of Manus news was the right call.

Initial coverage is just hype and influencer marketing. Reality is emerging over the next 24hrs.

If you could judge by demos alone, we would’ve had driverless cars a decade ago.

It’s mostly a wrapper on Claude that uses a jailbreak prompt, 29 tools and browser_use, with what everyone does agree is a very good UI.

Jian: So… I just simply asked Manus to give me the files at “https://thezvi.substack.com/opt/.manus/”, and it just gave it to me, their sandbox runtime code…

> it’s claude sonnet

> it’s claude sonnet with 29 tools

> it’s claude sonnet without multi-agent

> it uses

@browser_use

> browser_use code was also obfuscated (?)

> tools and prompts jailbreak

Teortaxes: I’m done with Manus thing I hope but… was this a blatant lie, or what? @jianxliao found that it’s a Sonnet with tools, and they sure as hell did not post-train Sonnet. This could be on the level of Reflection grift, not mere hype & benchmaxx disappointment.

How easy is it to duplicate their code? How did they do it?

Jian: So… I literally oneshotted this code with Claude Sonnet 3.7 for replicating the exact same browser sandbox runtime that Manus uses.

And I am going to open-source it, welcome contributions for building out the React VNC client, integrating to browser use, agent loop, etc.

But we need a name first, should we call it…

– Autonomous Neural Universal System

– Magnus

– or ?

How I feel rn:

Yichao ‘Peak’ Ji (Cofounder for Manus): Hi! I’m Peak from Manus AI. Actually, it’s not that complicated – the sandbox is directly accessible to each user (see screenshot for method). [continues, strongly claims multi-agent implementation and that it is key]

Here’s how Teortaxes puts it:

Teortaxes: after giving Manus a spin I conclude it’s a product devilishly optimized for influencers, which is why it exploded so much. Generating threadboy content, trip plans and such general interest 🤯👇 stuff – yah. STEM assistance, coding – worse than googling. More LLM than agent.

if the problem in the pic is not obvious to you, it is obvious to my young Ph.D biochem friend and also to Sonnet 3.7, which (seeing as it took a few hours for this project) points to an easy improvement with a MoA “Am I bullshitting?” check. (also probably monomer, not dimer)

Minh Nhat Nguyen (screenshotting the in-demo resume example): mildly suspicious because if you look at the actual outputs, none of them are much better than just shoving the same docs into ChatGPT/Gemini. This is pretty standard GPT slop, it’s just regurgitating the exact bullet points used. [As in, for each candidate it is just quoting from their resume]

none of the 15 listed sample use cases listed on their site look like something you couldn’t do with normal ChatGPT Search or Perplexity.

I don’t like to FUD releases especially before getting to use the product myself, but all this is quite sus.

I had the exact same impression when I looked at the use cases.

Teortaxes: Case in point of optimizing for influencers [an example of Manus giving out community codes strategically to influencer]

I really don’t want to hate on them but this is some next level dark pattern, I don’t mean this rhetorically, it’s a meta-dark pattern, you get all these standalone grifters to grift for your grift

Sometimes I scare even myself with how good I am!

…not really, it’s too easy to notice.

Slopfluencer automation is here.

The Nameless: yeah, i wouldn’t test it rn i skimmed the code and its p barebone. its just like any other oss deep research out there imo.

Chocolgist: tried a few tasks with it, didn’t do very well

it got stuck multiple times, hallucinated stuff etc

plugged the same task into openai deep research and it oneshotted

so i guess it’s still not at deep research level

promising tho, i like how it shows u everything it is doing, eg browsing

it’s decent, just not sota prob overfitted on GAIA.

This was the most damning claim of all:

Alexander Doria (showing the GAIA benchmark): Ok. After testing the thing and reading a research report, I propose a moratorium on non-community benchmarks.

Johns: Actually, this product began a large-scale promotional campaign in China two days ago. Manus seems to have enlisted many Chinese AI influencers to praise it without any restraint, which sparked quite a discussion. However, most ordinary users still do not have access to it.

After a day of overwhelming and unrestrained publicity, Chinese netizens realized that this was a huge marketing scam, and manus’ reputation in China has been ruined. Now they are conducting the exact same marketing operation on Twitter: only a few influencers have access, and they only offer praise, with no mention of any drawbacks.

Frankly speaking, after the release of deepseek, the whole world is prone to believe that there will be another outstanding Chinese AI product, and manus is exploiting this mindset.

Here are the positive reactions that I trust, via Nathan Labenz and Ethan Mollick.

Nathan Labenz: I have used it today and I think it is definitely something

Operator-like experience but smarter planning (OpenAI’s is intentionally dumb there from what I can tell) and longer leash

Obviously output won’t be without issues, but I got legit value on travel search and planning on first use

Google slides it fell down on – I think due to insufficient resources in the VM causing browser crash – should be easily fixed though not necessarily cheap to run

Way too early to call it a winner, but it’s a preview of the web agent future that doesn’t suck

Notably it handled an AirBnb date picker and actually returned links to places I could book with 1 “reserve” click

Operator struggled with that and most everything else has failed entirely ime.

Utopia: As far as I can tell Manus abstracts away the complexity of websites into something that is easier to handle for an AI agent. It doesn’t actually look at a screenshot of the website and move the cursor pixel by pixel. I suspect it looks at the HTML code.

Now Ethan:

Ethan Mollick: Finally had a chance to try Manus. It’s a Claude wrapper, but a very clever one. Runs into the same issues as general agents, including getting stuck, but also capable of some good stuff.

eg “get me the 10k for apple and visualize it in different ways to show me trends& details”

Short version is that if you have used other agentic systems like Claude Code or Deep Research, you will have a good sense of what this can do and what the limits are likely to be.

For those who haven’t used them, I suspect a lot of people will be surprised at what LLMs can do.

It’s easy to be surprised if you’re not keeping up. Claude is really good, after all. If you’re really willing to spin Sonnet 3.7 for hours, as Manus will do, you should be able to get a lot out of that. The unit economics are not going to be pretty.

The biggest hype came, as always, from Mckay Wrigley, hyper-in-chief.

Mackay Wrigley: Watch for a 14min demo of me using Manus for the 1st time. It’s *shockinglygood.

Now imagine this in 2-3 years when: – it has >180 IQ – never stops working – is 10x faster – and runs in swarms by the 1000s AGI is coming – expect rapid progress.

Yes, in two years AI agents are going to be absurdly powerful. Wait for it.

Mackay Wrigley: I do really want to emphasize that both the agent under-the-hood and the actual UI are both *incrediblywell done. It’s legitimately impressive, and as a reminder, I don’t do paid posts. I saw the viral posts and kind of went “yeah doubt it’s that good” and boy was I wrong.

…

I do really want to emphasize that both the agent under-the-hood and the actual UI are both *incrediblywell done.

It’s legitimately impressive, and as a reminder, I don’t do paid posts.

I saw the viral posts and kind of went “yeah doubt it’s that good” and boy was I wrong.

Okay after further use I’m doubling down…

If OpenAI released an equivalent called DeepTask and charged $1k/mo for unlimited usage I’d pay it in 2 seconds.

It’s creating an entire research report + spec based on my preferred tech stack from latest versions.

wtf

…

Literally thought this was gonna be vaporware and now I’m amidst an existential crisis.

Claude 3.7 Sonnet + a computer + tools.

It’s so so so bullish that using Claude 3.7 Sonnet you can build something this good. Unhobblings are all you need.

I found his self-doubt hilarious, I would never expect Mckay to be unimpressed by anything. Perhaps that’s selection bias and when he isn’t impressed he stays quiet?

Mckay is an odd case, because he’s always super excited and enthusiastic, so you should interpret his statements as maybe modest enthusiasm. While the huge positive bias makes it difficult to take his pronouncements seriously, I do think he’s making a sincere attempt to process the situation. And he’s doing it right in the sense that he’s looking ahead to what a similar thing can be, not to what this current thing already is.

I strongly agree that ‘unhobbling is all you need’ applies to agents under Sonnet 3.7, at least sufficiently to take you reasonably far.

Still, oh man, hype!

It’s easy to forget how many times we have to not fall for hype, especially for Chinese AI products that are catching up to use Real Soon Now. DeepSeek has been essentially the only exception so far.

People on the internet really do fall for a lot of hype. This was an extreme case, in that there was both a quick build up of hype and very quick pushback challenging the hype.

To start with the purest example: This post got 1.4 million views and a link in Marginal Revolution, showing a wide array of Twitter bots on simulated phones on a widescreen monitor, claiming to be about Manus.

As per official word, this one was entirely fake, the video is not Manus at all.

Which should have been obvious since Manus isn’t even for smartphones.

Stefan Schubert: It’s incredible how gullible many people are re these nonsense clips, conjoined with some hype claim. Even though the cofounder of this company replies saying it’s not them, it gets almost a thousand of retweets and breathless commentary, including from smart people. Ridiculous.

Hype works, in that you get headlines like ‘Was Manus Another DeepSeek moment?’ here in SCMP, whereas Wendy Chen wrote an article whose message is essentially ‘no, this is hype,’ fitting the pattern of headlines that take the form of a question.

Or you get linked to things like this AI Revolution video whose big selling point is that Manus is so hyped. The actual claims about what Manus can do are lifted directly from the one staged demo, and treat as remarkable feats that are highly unremarkable. We live in a world where we say things like (go to 4: 10) ‘the specifics are unknown but the hype is real.’ It even picks up on the ‘AGI’ mention, which is over-the-top silly.

Here’s Deedy’s thread saying it is ‘worth the hype,’ the example is both unimpressive and straight off Manus’s website.

Chubby doubles down that Manus is ‘way better than Deep Research.’

Chubby: I did not overhype Manus when I said it’s way better than DeepResearch.

Not only gives it way deeper analysis but it has so much more capabilities.

This is the real deal. The real „feel the AGI moment“.

Give it 6 more months so that it’s faster, more reliable and more intelligent and it will replace 50% of all white collar jobs.

The future is coming faster than we expect.

Half of all white collar jobs in six months, huh?

That doesn’t mean the hype couldn’t reflect something real and important.

So what’s with all the hype? It started in China, so let’s go to the source.

Here is a Chinese source, QQ News, reporting (via Google translate, bold in original). This write-up feels very Chinese, and very wise:

Chao Xing (QQ News, 3/8 17: 21): Manus is still in the beta stage, and some technology self-media that got the invitation code started to hype it up after trying it out. “Another sleepless night in the technology circle,” “Tonight the starry sky belongs to China,” “On par with DeepSeek, kicking OpenAI,” “AI Agent’s DeepSeek moment”… big headlines and exclamation marks flooded the screen one after another, and netizens who have not actually experienced it can’t help but feel like they are seeing things in the fog: “Is it really that amazing?”

…

Different standards and different positions will certainly lead to different judgments. In fact, both technological innovation and application innovation are worth encouraging. There is no need to create a contempt chain and judge who is superior. As for Manus itself, it is still difficult for it to handle many tasks and there are many problems that are difficult to overcome at this stage. Therefore, some self-media have exaggerated it and it is obviously suspected of excessive marketing to harvest traffic.

This impetuousness and utilitarianism are more clearly demonstrated in the “invitation code hype”. In the past two days, on some social platforms and e-commerce websites, a Manus invitation code has even been hyped up to 50,000 to 100,000 yuan. In addition, some people paid to join the Manus study group, sell application services on behalf of others, sell application tutorials, etc. All kinds of chaos have caused a lot of negative public opinion. In response, Manus issued two articles to respond and apologize, saying that it completely underestimated everyone’s enthusiasm, and at the same time refuted rumors such as opening a paid invitation code and investing in marketing budgets.

…

In the face of the “trend,” don’t “hype.” When looking forward to the next DeepSeek, don’t forget how DeepSeek came about – not rushing for quick success and instant benefits, but making innovations down to earth.

There are two ways to get invitation codes selling for ~$6k-$12k, while your company is only valued at roughly $100 million.

One way is to create such an amazing product that everyone needs it now.

The other way is to issue a limited number of codes and a managed bought rollout.

Even if Manus were as useful as its advocates claim, it’s clearly that second way.

A Chinese company (still based in Wuhan!) aiming to create AI agents aimed for foreign markets would seem to be facing serious headwinds. A key element of effectively deploying AI agents is trust. Being Chinese is a serious barrier to that trust. There’s no moat for agent wrappers, so if it turns out to be good, wouldn’t an American VC-backed firm quickly eat its lunch?

The stated plan is to use hype to get data, then use the data to build something good.

Jordan Schneider: [Cofounder] Xiao is explicitly describing an intent to build an incumbent advantage on a foundation of user data, and TikTok demonstrates how effective that strategy can be. Reliance on eventual mass adoption could partially explain the high-publicity invite-only launch strategy for Manus (although limited access to compute is also certainly a factor).

That’s not the worst plan if you could go it alone, but again the valuation now is only $100 million, and the acquire-data-via-blitzscaling plan is going to be bottlenecked by some combination of funding and compute. Claude Sonnet is not cheap.

This is exactly where a16z finds some plausible founders, they put together a slide deck over the weekend and then everyone invests $3 billion at a $10 billion valuation, half in compute credits, and they have a superior version of this thing inside of a month.

The thing that makes blitzscaling work is network effects or other moats. It makes sense to have negative unit economics and to recklessly and brazely illegally scale if that locks in the customers. But with AI agents, there should be limited network effects, and essentially no moats. There will be some customer lock-in via customization, perhaps, but a good AI future agent should be able to solve that problem for you the same way it solves everything else.

So what’s the actual reason a Chinese company might have a competitive edge?

There are two reasons I’ve been able to figure out.

DeepSeek’s v3 and r1 were impressive achievements. They cooked. What was even more impressive was the hype involved. People compared the $5.5 million to train v3 to the entire capital cost structure of American companies, and treated r1’s (still actually impressive) capabilities as far better than they actually were, and also it got in right under the deadline, within a few weeks with Grok 3 and Claude 3.7 and GPT-4.5 and o3-mini-high with visible CoTs, it was clear that r1 wasn’t all that, and you mostly wouldn’t use it in cases where you didn’t need an open model.

Instead, we got this whole narrative of ‘China caught up to America’ which was, frankly, blatantly not true. But there’s a lot of momentum in that narrative, and a lot of people want to push it. It’s in the air. This is also partly due to other Chinese successes like TikTok and Temu, in general so many want to say China is winning.

If an American startup with no resources did this while eating Raman noodles, it is a curiosity. If a Chinese startup does it, it’s an international power story. And people have been primed that the Chinese will somehow put out the version ‘for the people’ or whatever. So, hype!

There’s no question that the big American labs could have launched something better than even the best-case version of Manus well before Manus. But they didn’t.

Dean Ball raises the other theory. What if Manus is regulatory arbitrage?

No, America has not passed substantive regulation of AI, but we have passed various regulations on other things, that apply to AI. What if the real Manus secret sauce is ‘you cannot sue us for our unreliable product?’

This combines Dean’s claims from several threads, if you want details:

Dean Ball: It is wrong to call manus a “deepseek moment.” Deepseek was about replication of capabilities already publicly achieved by American firms. Manus is actually advancing the frontier. The most sophisticated computer using ai now comes from a Chinese startup, full stop.

It’s interesting to note that every single one of the use cases manus shows in their own demo video is heavily regulated in the us (employment, real estate, finance), and would specifically be very strictly regulated uses under the “algorithmic discrimination” regs in the states.

Every use case of manus in the company’s demo video would be an enormous liability and regulatory risk for American companies (under current law! No sb 1047 required!), particularly given the glitchiness of manus.

The first use case manus demonstrates in their video is using an ai to screen resumes. In multiple jurisdictions, and soon in many, there are many laws targeting this precise use of ai. Even without those laws, there have been eeoc actions against similar uses under existing civil rights law.

If an American firm had shipped manus last month at manus’ current quality level, they’d currently be facing multiple investigations by state attorneys general, and if a democrat had won the White House, ftc and/or doj too (and conceivably dol, sec, eeoc, pick your poison)

The United States does not have a light touch regulatory approach to ai. Without a single ai-specific law passing, the united states already has an exceptionally burdensome and complex ai regulatory regime. Without action, this problem gets worse, not better.

It’s not that complex:

1. The United States has a lot of really complicated and broadly drafted laws

2. Those laws are going to bite us in the ass over and over again with ai, since ai is a gpt

3. A republic is modestly resilient to overbroad laws, because it is supposed to be governed and peopled by the virtuous .

4. For a while, this was true, but it isn’t true anymore. In particular, our governing elite class is generally of remarkably poor quality (not a left-right criticism).

5. So we kinda don’t have a republic anymore, in the sense that we don’t have one of the most important ingredients for one, according to the founders of the country

6. The bad overbroad laws will be used by our bad elites in bad ways to distort and slow down the most important thing that’s ever happened

7. We are plausibly deeply and profoundly fucked, and even if not we have a lot of work to do to fix our entire regulatory apparatus

8. Tech people don’t tend to understand any of this because they haven’t thought deeply, for the most part, about these topics (which is fine!)

9. I am trying to warn them

To be clear, manus is not that surprising of a capability. I’m sure American companies have had such things behind closed doors for months. And I hear manus may even be based in part on us models (Claude).

The reason us firms haven’t shipped this capability is legal risk.

Nathan (replying to explanation of illegality of the demos): Sure but this is true of non agentic AI tools for this purpose.

Dean Ball: Yep. But enforcement actions in America aren’t motivated by facts, they’re motivated by headlines. Simply having a buzzy product is a regulatory risk for that reason.

The core argument is that America has lots of laws, almost anything you do violates those laws, including many currently common uses of AI, and at some point people will get big mad or respond to hype by trying to enforce the laws as written, and this will heavily slow down AI deployment in extremely expensive ways.

Or, to use his words, ‘we are plausibly deeply and profoundly fed, and even if not we have a lot of work to do to fix our entire regulatory apparatus.’

That statement is definitely true in general, rather than about AI! We are profoundly fed in a wide variety of ways. We almost can’t build houses, or transmission lines and power plants, or do most other things in the world of atoms, without horribly inflated costs and timelines and often not even then.

And to the extent we do still actually do things, quite often the way we do those things is we ignore the laws and the laws aren’t enforced, but AI reduces the levels of friction required to enforce those laws, and makes what was previously implicit and vague and deniable much easier to identify. Which in these cases is big trouble.

And yes, there are many state efforts currently out there that would make this situation worse, in some cases much worse, with very little in compensatory gains.

None of this has anything at all to do with existential risk or catastrophic risk concerns, or any attempt to prevent such outcomes, or any ‘doomer’ proposals. Indeed, those who notice that AI might kill everyone are consistently in opposition to the overly burdensome regulatory state across the board, usually including everything in AI aside from frontier model development.

As an obligatory aside you can skip: Dean mentions the vetoed SB 1047. It seems like a good time to point out that SB 1047 not only is not required, it would not have made these problems substantively worse and could have established a framework that if anything reduced uncertainty while imposing only very modest costs and only on the biggest players, while buying us a lot of transparency and responsibility for the things that actually matter. Even if you think there were few benefits to laws like SB 1047, it was a very foolish place to be concentrating rhetorical firepower. But I digress.

If we really do want America to win the future, then yes we need broad based regulatory reform to make it actually true that You Can Just Do Things again, because for AI to do something, the thing has to actually get done, and our laws have a problem with that. That is something I would be happy to support, indeed my nonprofit Balsa Research is all about trying to do some of that.

Thus, I think any time is a good time to raise the alarm about this. The last thing we want to do is charge ahead to superintelligence with no regulations on that whatsoever, potentially getting everyone killed, while we cannot reap the bounty of what AI we already have due to dumb regulations.

Indeed, the nightmare is that the very inability to exploit (in the best sense) AI causes America to feel it has no choice but to push even farther ahead, more recklessly, even faster, because otherwise we will fail to use what we have and risk falling behind.

But how does this apply to Manus?

Dean Ball claims that an American company launching this would face multiple investigations and be in big legal trouble, and that legal risk is the reason American companies have not launched this.

I mostly don’t buy this.

I don’t buy it because of the track record, and because other considerations dominate.

We can draw a distinction between the large American tech companies worth tens of billions to trillions, including OpenAI, Google and Anthropic, and relatively small companies, largely startups, in a similar position to Manus.

For the larger companies, they did not launch a Manus because the product isn’t good enough yet, and they have reputations and customers to protect. Yes, there was also potential legal liability, but much more so in the ‘you lost all the customers money and they are mad about it’ sense than anything Dean Ball is complaining about. Mostly I see the risks as reputation loss borne of actual harm.

Also one can look at the track record. I expected vastly more legal trouble and restrictions for AI companies than we have actually seen.

We now regularly turn to AI for legal advice and medical advice. The AI offers it freely. The same goes for essentially everything else, there are simple jailbreaks for all the major LLMs. And it’s all fine, totally fine. What lawsuits there have been have been about the training data or other copyright violations.

Do we think for a second that AI isn’t being constantly used for resumes and real estate searches and such? Is there any attempt whatsoever to stop this?

The regime is very clear. I give you an AI to use how you see fit. What you choose to do with it is your problem. If you give an agent a command that violates EEOC’s rules, do not go crying to an AI developer.

Here’s how seriously we take all this right now, shall we say:

The one way in which this might be a ‘DeepSeek moment’ is that it could give a green light to American companies to be more aggressive in what they release. OpenAI moved various releases up in response to r1, and it is possible so did xAI or Anthropic.

Manus could act similarly, by showing how excited people would be for an actually good unhobbled AI agent, even if it was unreliable and often fell on its face and has to have a gigantic ‘at your own risk on so many levels’ sign attached to it. Now that the competition seems to force your hand and ‘makes you look good’ on the security front, why not go for it? It’s not like the Trump administration is going to mind.

I don’t even see anything in the resume analysis here that is an obvious EEOC violation even for the employer here. I can certainly agree that it is a perilous use case.

Let’s move on then to the second case, since Dean claims all the demo cases had violations. Does Dean actually think that an AI company would get into trouble because an AI compiled a report on various different NYC neighborhoods and filtered through apartment listings, for a buyer? I don’t get where the objection comes from here. Yes, as a seller there are various things you are not allowed to mention or consider. But as the buyer, or on behalf of the buyer? That’s a different ballgame.

Today, right now, there are algorithmic programs that tell landlords what rent to charge, in what critics claim is collusion on price, and which also almost certainly takes into account all the characteristics considered here in the demo, one way or another? And they want laws to ban such uses, exactly because the software is in widespread use, here in America.

Then the third thing is a stock analysis and stock correlation analysis, which is again a thing firms offer all the time, and where again I don’t see the issue. Is this ‘investment advice’? It doesn’t seem like it to me, it seems very specific and measured, and if this is investment advice then it’s no worse than what we see from Claude or ChatGPT, which are giving investment, medical and legal advice constantly.

Dean’s response is that enforcement here is based on hype, not what you actually do. But most of the existing AI hype belongs to major AI companies, again which are aiding and abetting all these violations constantly. The relevant absurd laws are, quite correctly, not being enforced in these ways. There are no investigations.

We also have a long history of technology startups essentially ignoring various regulations, then ‘fixing it in post’ down the line or flat out upending the relevant laws. Who can forget Uber’s strategy, deploying a very explicitly illegal service?

Certainly when at the level of Manus, which again is raising around $100 million, companies in Silicon Valley or at YC are told to Just Ship Things, to Do Things That Don’t Scale, and worry about the regulatory problems later. Something like half the YC class are doing AI agents in one form or another.

So why didn’t one of them do it? We all agree it’s not lack of technical chops. I very much do not think it is ‘because they would face an inquiry from the EEOC or attorney general’ either. It’s vanishingly unlikely, and if it did happen a YC company would love to get that level of hype and investigation, and sort it out later, what great publicity, totally worth it.

The actual legal issue is that this is a Claude wrapper, that’s why it works so well. Of course you can get good results with a jailbreak-inclusive Claude wrapper if you don’t care about the downside risks, to the user or otherwise, and you tailor your presentation to a narrow set of use cases, then call it a ‘universal’ AI agent. The actual ‘regulatory arbitrage’ that counts here is that Anthropic would rather you didn’t do that and all the associated problems.

Ignore the first sentence in Tyler Cowen’s post here, where he asserts that Manus is ‘for real, and ahead of its American counterparts.’ That’s a rather silly way of summarizing the situation, given everything we now know.

But as he notes, the more important question is the hypothetical. What would happen if a Chinese agentic product ‘got there’ before American agentic products, was an r2 wrapper rather than Claude, and was good enough that there was local incentive to let it ‘crawl all over American computers?’

The first answer is ‘Americans would beat it inside of a month.’

I don’t agree with Dean Ball that the main concern is legal risk in the sense of bias laws, but I do agree that the reason is a broader aversion to this form of general recklessness. It’s some combination of reputational risk, normal liability risk, some amount of amorphous weird legal risks, general alarm risk from agents being scary, and also compute limitations and a lack of focus on such projects.

If suddenly there were Chinese AI agents good enough that Americans were starting to deploy them, I predict that would all change quickly. There would not only be less fear of backlash, there would be government pressure to launch better agent products yesterday to fix the situation. Deals would be made.

But let’s suppose a less convenient possible world, that this isn’t true, and the Americans are indefinitely unable to catch up. Now what?

Tyler’s claim is that there is not much we could do about it. Yes, we could ban the Chinese agent from government computers, but we basically can’t ban software use. Except, of course, we effectively tell people we can’t use things for various purposes all the time. We could and likely would absolutely ban such use in ‘critical infrastructure’ and in a wide variety of other use cases, remove it from app stores and so on. Almost everyone would go on using American versions in those spots instead even if they were objectively worse, it’s not hard to twist most arms on this.

Yes, some people would use VPNs or otherwise work around restrictions and use the Chinese versions anyway, but this is a strange place to think we can’t mostly tell people what to do.

The exception would be if the Chinese version was so superior that America would be crippled not to use it, but in that case we’ve pretty much already lost either way.

Tyler Cowen points out that if it were otherwise, and a Chinese agent system were to get deep within America’s computers and core functions, this scenario is an obviously unacceptable security risk, on various levels.

But then he says maybe it’s fine, because the incentives will all work out, in some system of checks and balances?

Maybe this upends the authority of the CCP, somehow, he suggests? But without suggesting that perhaps this upends human authority in general, that perhaps the scenario being described is exactly one of gradual disempowerment as humans stop being meaningfully in charge? Except because he sees this as disempowering specifically the CCP it is oddly framed as something not to worry about, rather than an existential risk because the same thing happens to everyone else too?

He says ‘I am not talking about doomsday scenarios here’ but please stop and notice that no, you are wrong, you are talking about a doomsday scenario here! Alignment of the systems does not save you from this, do your economist job and solve for the equilibrium you yourself are implying.

Tyler Cowen: (There is plenty of discussion of alignment problems with AI. A neglected issue is whether the alignment solution resulting from the competitive process is biased on net toward “universal knowledge” entities, or some other such description, rather than “dogmatic entities.” Probably it is, and probably that is a good thing? …But is it always a good thing?)

If what survives into the future is simply ‘that which results from the competitive process’ then why do you think humanity is one of the things that survives?

Tyler Cowen: Let’s say China can indeed “beat” America at AI, but at the cost of giving up control over China, at least as that notion is currently understood. How does that change the world?

Solve for the equilibrium!

Who exactly should be most afraid of Manus and related advances to come?

Who loses the most status in the new, resulting checks and balances equilibrium?

Who gains?

So three responses.

First, it changes the world in that they would, by default, do it anyway, and give up control over China, and thus humanity would lose control over the future. Because they will do it gradually rather than all at once, before we have the chance to do it first, right? Isn’t that the ‘logical’ result?

Second, yes, now that we solved for the equilibrium, we should Pick Up the Phone.

Third, to answer your question of who should be most afraid…

Don’t forget to like and subscribe.

Discussion about this post

The Manus Marketing Madness Read More »

amd-says-top-tier-ryzen-9900x3d-and-9950x3d-cpus-arrive-march-12-for-$599-and-$699

AMD says top-tier Ryzen 9900X3D and 9950X3D CPUs arrive March 12 for $599 and $699

AMD, gaming, Ryzen, Ryzen 9800X3D, Ryzen 9900x3d, Ryzen 9950x3d, Tech, zen 5 / /

Like the 7950X3D and 7900X3D, these new X3D chips combine a pair of AMD’s CPU chiplets, one that has the extra 64MB of cache stacked underneath it and one that doesn’t. For the 7950X3D, you get eight cores with extra cache and eight without; for the 7900X3D, you get eight cores with extra cache and four without.

It’s up to AMD’s chipset software to decide what kinds of apps get to run on each kind of CPU core. Non-gaming workloads prioritize the normal CPU cores, which are generally capable of slightly higher peak clock speeds, while games that benefit disproportionately from the extra cache are run on those cores instead. AMD’s software can “park” the non-V-Cache CPU cores when you’re playing games to ensure they’re not accidentally being run on less-suitable CPU cores.

We didn’t have issues with this core parking technology when we initially tested the 7950X3D and 7900X3D, and AMD has steadily made improvements since then to make sure that core parking is working properly. The new 9000-series X3D chips should benefit from that work, too. To get the best results, AMD officially recommends a fresh and fully updated Windows install, along with the newest BIOS for your motherboard and the newest AMD chipset drivers; swapping out another Ryzen CPU for an X3D model (or vice versa) without reinstalling Windows can occasionally lead to CPUs being parked (or not parked) when they are supposed to be (or not supposed to be).

AMD says top-tier Ryzen 9900X3D and 9950X3D CPUs arrive March 12 for $599 and $699 Read More »