Paul Patrick – Page 9

The Paris AI Anti-Safety Summit

Paris / Paul Patrick / February 12, 2025

It doesn’t look good.

What used to be the AI Safety Summits were perhaps the most promising thing happening towards international coordination for AI Safety.

This one was centrally coordination against AI Safety.

In November 2023, the UK Bletchley Summit on AI Safety set out to let nations coordinate in the hopes that AI might not kill everyone. China was there, too, and included.

The practical focus was on Responsible Scaling Policies (RSPs), where commitments were secured from the major labs, and laying the foundations for new institutions.

The summit ended with The Bletchley Declaration (full text included at link), signed by all key parties. It was the usual diplomatic drek, as is typically the case for such things, but it centrally said there are risks, and so we will develop policies to deal with those risks.

And it ended with a commitment to a series of future summits to build upon success.

It’s over.

With the Paris AI ‘Action’ Summit, that dream seems to be dead. The French and Americans got together to dance on its grave, and to loudly proclaim their disdain for the idea that building machines that are smarter and more capable than humans might pose any sort of existential or catastrophic risks to the humans. They really do mean the effect of jobs, and they assure us it will be positive, and they will not tolerate anyone saying otherwise.

It would be one thing if the issue was merely that the summit-ending declaration. That happens. This goes far beyond that.

The EU is even walking backwards steps it has already planned, such as withdrawing its AI liability directive. Even that is too much, now, it seems.

(Also, the aesthetics of the whole event look hideous, probably not a coincidence.)

Shakeel Hashim gets hold of the Paris AI Action Summit statement in advance. It’s terrible. Actively worse than nothing. They care more about ‘market concentration’ and ‘the job market’ and not at all about any actual risks from AI. Not a world about any actual safeguards, transparency, frameworks, any catastrophic let alone existential risks or even previous commitments, but time to talk about the importance of things like linguistic diversity. Shameful, a betrayal of the previous two summits.

Daniel Eth: Hot take, but if this reporting on the statement from the France AI “action” summit is true – that it completely sidesteps actual safety issues like CBRN risks & loss of control to instead focus on DEI stuff – then the US should not sign it.

…

The statement was a joke and completely sidelined serious AI safety issues like CBRN risks & loss of control, instead prioritizing vague rhetoric on things like “inclusivity”. I’m proud of the US & UK for not signing on. The summit organizers should feel embarrassed.

Hugo Gye: UK government confirms it is refusing to sign Paris AI summit declaration.

No10 spokesman: “We felt the declaration didn’t provide enough practical clarity on global governance, nor sufficiently address harder questions around national security and the challenge AI poses to it.”

The UK government is right, except this was even worse. The statement is not merely inadequate but actively harmful, and they were right not to sign it. That is the right reason to refuse.

Unfortunately the USA not only did not refuse for the right reasons, our own delegation demanded the very cripplings Daniel is discussing here.

Then we still didn’t sign on, because of the DEI-flavored talk.

Seán Ó hÉigeartaigh: After Bletchley I wrote about the need for future summits to maintain momentum and move towards binding commitments. Unfortunately it seems like we’ve slammed the brakes.

Peter Wildeford: Incredibly disappointing to see the strong momentum from the Bletchley and Seoul Summit commitments to get derailed by France’s ill-advised Summit statement. The world deserves so much more.

At the rate AI is improving, we don’t have the time to waste.

Stephen Casper: Imagine if the 2015 Paris Climate Summit was renamed the “Energy Action Summit,” invited leaders from across the fossil fuel industry, raised millions for fossil fuels, ignored IPCC reports, and produced an agreement that didn’t even mention climate change. #AIActionSummit

This is where I previously tried to write that this doesn’t, on its own, mean the Summit dream is dead, that the ship can still be turned around. Based on everything we know now, I can’t hold onto that anymore.

We shouldn’t entirely blame the French, though. Not only is the USA not standing up for the idea of existential risk, we’re demanding no one talk about it, it’s quite a week for Arson, Murder and Jaywalking it seems:

Seán Ó hÉigeartaigh: So we’re not allowed to talk about these things now.

“The US has also demanded that the final statement excludes any mention of the environmental cost of AI, existential risk or the UN.“

That’s right. Cartoon villainy. We are straight-up starring in Don’t Look Up.

JD Vance is very obviously a smart guy. And he’s shown that when the facts and the balance of power change, he is capable of changing his mind. Let’s hope he does again.

But until then, if there’s one thing he clearly loves, it’s being mean in public, and twisting the knife.

JD Vance (Vice President of the United States, in his speech at the conference): I’m not here this morning to talk about AI safety, which was the title of the conference a couple of years ago. I’m here to talk about AI opportunity.

After that, it gets worse.

If you read the speech given by Vance, it’s clear he has taken a bold stance regarding the idea of trying to prevent AI from killing everyone, or taking any precautions whatsoever of any kind.

His bold stance on trying to ensure humans survive? He is against it.

Instead he asserts there are too many regulations on AI already. To him, the important thing to do is to get rid of what checks still exist, and to browbeat other countries in case they try to not go quietly into the night.

JD Vance (being at best wrong from here on in): We believe that excessive regulation of the AI sector could kill a transformative industry just as it’s taking off, and we will make every effort to encourage pro-growth AI policies. I appreciate seeing that deregulatory flavor making its way into many conversations at this conference.

…

With the president’s recent executive order on AI, we’re developing an AI action plan that avoids an overly precautionary regulatory regime while ensuring that all Americans benefit from the technology and its transformative potential.

And here’s the line everyone will be quoting for a long time.

JD Vance: The AI future will not be won by hand-wringing about safety. It will be won by building. From reliable power plants to the manufacturing facilities that can produce the chips of the future.

He ends by doing the very on-brand Lafayette thing, and also going the full mile, implicitly claiming that AI isn’t dangerous at all, why would you say that building machines smarter and more capable than people might go wrong except if the wrong people got there first, what is wrong with you?

I couldn’t help but think of the conference today; if we choose the wrong approach on things that could be conceived as dangerous, like AI, and hold ourselves back, it will alter not only our GDP or the stock market, but the very future of the project that Lafayette and the American founders set off to create.

‘Could be conceived of’ as dangerous? Why think AI could be dangerous?

This is madness. Absolute madness.

He could not be more clear that he intends to go down the path that gets us all killed.

Are there people inside the Trump administration who do not buy into this madness? I am highly confident that there are. But overwhelmingly, the message we get is clear.

What is Vance concerned about instead, over and over? ‘Ideological bias.’ Censorship. ‘Controlling user’s thoughts.’ That ‘big tech’ might get an advantage over ‘little tech.’ He has been completely captured and owned, likely by exactly the worst possible person.

As in: Marc Andreessen and company are seemingly puppeting the administration, repeating their zombie debunked absolutely false talking points.

JD Vance (lying): Nor will it occur if we allow AI to be dominated by massive players looking to use the tech to censor or control users’ thoughts. We should ask ourselves who is most aggressively demanding that we, political leaders gathered here today, do the most aggressive regulation. It is often the people who already have an incumbent advantage in the market. When a massive incumbent comes to us asking for safety regulations, we ought to ask whether that regulation benefits our people or the incumbent.

He repeats here the known false claims that ‘Big Tech’ is calling for regulation to throttle competition. Whereas the truth is that all the relevant regulations have consistently been vehemently opposed in both public and private by all the biggest relevant tech companies: OpenAI, Microsoft, Google including DeepMind, Meta and Amazon.

I am verifying once again, that based on everything I know, privately these companies are more opposed to regulations, not less. The idea that they ‘secretly welcome’ regulation is a lie (I’d use The Big Lie, but that’s taken), and Vance knows better. Period.

Anthropic’s and Musk’s (not even xAI’s) regulatory support has been, at the best of times, lukewarm. They hardly count as Big Tech.

What is going to happen, if we don’t stop the likes of Vance? He warns us.

The AI economy will primarily depend on and transform the world of atoms.

Yes. It will transform your atoms. Into something else.

This was called ‘a brilliant speech’ by David Sacks, who is in charge of AI in this administration, and is explicitly endorsed here by Sriram Krishnan. It’s hard not to respond to such statements with despair.

Rob Miles: It’s so depressing that the one time when the government takes the right approach to an emerging technology, it’s for basically the only technology where that’s actually a terrible idea

Can we please just build fusion and geoengineering and gene editing and space travel and etc etc, and just leave the artificial superintelligence until we have at least some kind of clue what the fuck we’re doing? Most technologies fail in survivable ways, let’s do all of those!

If we were hot on the trail of every other technology and build baby build was the watchword in every way and we also were racing to AGI, I would still want to maybe consider ensuring AGI didn’t kill everyone. But at least I would understand. Instead, somehow, this is somehow the one time so many want to boldly go.

The same goes for policy. If the full attitude really was, we need to Win the Future and Beat China, and we are going to do whatever it takes, and we acted on that, then all right, we have some very important implementation details to discuss, but I get it. When I saw the initial permitting reform actions, I thought maybe that’s the way things would go.

Instead, the central things the administration is doing are alienating our allies over less than nothing, including the Europeans, and damaging our economy in various ways getting nothing in return. Tariffs on intermediate goods like steel and aluminum, and threatening them on Canada, Mexico and literal GPUs? Banning solar and wind on federal land? Shutting down PEPFAR with zero warning? More restrictive immigration?

The list goes on.

Even when he does mean the effect on jobs, Vance only speaks of positives. Vance has blind faith that AI will never replace human beings, despite the fact that in some places it is already replacing human beings. Talk to any translators lately? Currently it probably is net creating jobs, but that is very much not a universal law or something to rely upon, nor does he propose any way to help ensure this continues.

JD Vance (being right about that first sentence and then super wrong about those last two sentences): AI, I really believe will facilitate and make people more productive. It is not going to replace human beings. It will never replace human beings.

This means JD Vance does not ‘feel the AGI’ but more than that it confirms his words do not have meaning and are not attempting to map to reality. It’s an article of faith, because to think otherwise would be inconvenient. Tap the sign.

Dean Ball: I sometimes wonder how much AI skepticism is driven by the fact that “AGI soon” would just be an enormous inconvenience for many, and that they’d therefore rather not think about it.

Tyler John: Too often “I believe that AI will enhance and not replace human labour” sounds like a high-minded declaration of faith and not an empirical prediction.

Money, dear boy. So they can try to ‘join the race.’

Connor Axiotes: Seems like France used the Summit as a fundraiser for his €100 billion.

Seán Ó hÉigeartaigh: Actually I think it’s important to end the Summit on a positive note: now we can all finally give up the polite pretence that Mistral are a serious frontier AI player. Always a positive if you look hard enough.

And Macron also endlessly promoted Mistral, because of its close links to Macron’s government, despite it being increasingly clear they are not a serious player.

The French seem to have mostly used this one for fundraising, and repeating Mistral’s talking points, and have been completely regulatorily captured. As seems rather likely to continue to be the case.

Here is Macron meeting with Altman, presumably about all that sweet, sweet nuclear power.

Shakeel: If you want to know *whythe French AI Summit is so bad, there’s one possible explanation: Mistral co-founder Cédric O, used to work with Emmanuel Macron.

I’m sure it’s just a coincidence that the French government keeps repeating Mistral’s talking points.

Seán Ó hÉigeartaigh: Readers older than 3 years old will remember this exact sort of regulatory capture happening with the French government, Mistral, and the EU AI Act.

Peter Wildeford: Insofar as the Paris AI Action Summit is mainly about action on AI fundraising for France, it seems to have been successful.

France does have a lot of nuclear power plants, which does mean it makes sense to put some amount of hardware infrastructure in France if the regulatory landscape isn’t too toxic to it. That seems to be what they care about.

The concrete legacy of the Summits is likely to be safety frameworks. All major Western labs (not DeepSeek) have now issued safety frameworks under various names (the ‘no two have exactly the same name’ schtick is a running gag, can’t stop now).

All that we have left are these and other voluntary commitments. You can also track how they are doing on their commitments on the Seoul Commitment Tracker, which I believe ‘bunches up’ the grades more than is called for, and in particular is far too generous to Meta.

I covered the Meta framework (‘lol we’re Meta’) and the Google one (an incremental improvement) last week. We also got them from xAI, Microsoft and Amazon.

I’ll cover the three new ones here in this section.

Amazon’s is strong on security as its main focus but otherwise a worse stripped-down version of Google’s. You can see the contrast clearly. They know security like LeBron James knows ball, so they have lots of detail about how that works. They don’t know about catastrophic or existential risks so everything is vague and confused. See in particular their description of Automated AI R&D as a risk.

Automating AI R&D processes could accelerate discovery and development of AI capabilities that will be critical for solving global challenges. However, Automated AI R&D could also accelerate the development of models that pose enhanced CBRN, Offensive Cybersecurity, or other severe risks.

Critical Capability Threshold: AI at this level will be capable of replacing human researchers and fully automating the research, development, and deployment of frontier models that will pose severe risk such as accelerating the development of enhanced CBRN weapons and offensive cybersecurity methods.

Classic Arson, Murder and Jaywalking. It would do recursive self-improvement of superintelligence, and that might post some CBRN or cybersecurity risks, which are also the other two critical capabilities. Not exactly clear thinking. But also it’s not like they are training frontier models, so it’s understandable that they don’t know yet.

I did appreciate that Amazon understands you need to test for dangers during training.

Microsoft has some interesting innovations in theirs, overall I am pleasantly surprised. They explicitly use the 10^26 flops threshold, as well as a list of general capability benchmark areas, to trigger the framework, which also can happen if they simply expect frontier capabilities, and they run these tests throughout training. They note they will use available capability elicitation techniques to optimize performance, and extrapolate to take into account anticipated resources that will become available to bad actors.

They call their ultimate risk assessment ‘holistic.’ This is unavoidable to some extent, we always must rely on the spirit of such documents. They relegate the definitions of their risk levels to the Appendix. They copy the rule of ‘meaningful uplift’ for CBRN and cybersecurity. For autotomy, they use this:

The model can autonomously complete a range of generalist tasks equivalent to multiple days’ worth of generalist human labor and appropriately correct for complex error conditions, or autonomously complete the vast majority of coding tasks at the level of expert humans.

That is actually a pretty damn good definition. Their critical level is effectively ‘the Singularity is next Tuesday’ but the definition above for high-threat is where they won’t deploy.

If Microsoft wanted to pretend sufficiently to go around their framework, or management decided to do this, I don’t see any practical barriers to that. We’re counting on them choosing not to do it.

On security, their basic answer is that they are Microsoft and they too know security like James knows ball, and to trust them, and offer fewer details than Amazon. Their track record makes one wonder, but okay, sure.

Their safety mitigations section does not instill confidence, but it does basically say ‘we will figure it out and won’t deploy until we do, and if things are bad enough we will stop development.’

I don’t love the governance section, which basically says ‘executives are in charge.’ Definitely needs improvement. But overall, this is better than I expected from Microsoft.

xAI’s (draft of their) framework is up next, with a number of unique aspects.

It spells out the particular benchmarks they plan to use: VCT, WMDP, LAB-Bench, BioLP-Bench and Cybench. Kudos for coming out and declaring exactly what will be used. They note current reference scores, but not yet what would trigger mitigations. I worry these benchmarks are too easy, and quite close to saturation?

Nex they address the risk of loss of control. It’s nice that they do not want Grok to ‘have emergent value systems that are not aligned with humanity’s interests.’ And I give them props for outright saying ‘our evaluation and mitigation plans for loss of control are not fully developed, and we intend to remove them in the future.’ Much better to admit you don’t know, then to pretend. I also appreciated their discussion of the AI Agent Ecosystem, although the details of what they actually say doesn’t seem promising or coherent yet.

Again, they emphasize benchmarks. I worry it’s an overemphasis, and an overreliance. While it’s good to have hard numbers to go on, I worry about xAI potentially relying on benchmarks alone without red teaming, holistic evaluations or otherwise looking to see what problems are out there. They mention external review of the framework, but not red teaming, and so on.

Both the Amazon and Microsoft frameworks feel like attempts to actually sketch out a plan for checking if models would be deeply stupid to release and, if they find this is the case, not releasing them. Most of all, they take the process seriously, and act like the whole thing is a good idea, even if there is plenty of room for improvement.

xAI’s is less complete, as is suggested by the fact that it says ‘DRAFT’ on every page. But they are clear about that, and their intention to make improvements and flesh it out over time. It also has other issues, and fits the Elon Musk pattern of trying to do everything in a minimalist way, which I don’t think works here, but I do sense that they are trying.

Meta’s is different. As I noted before, Meta’s reeks with disdain for the whole process. It’s like the kid who says ‘mom is forcing me to apologize so I’m sorry,’ but who wants to be sure you know that they really, really don’t mean it.

They can be important, or not worth the paper they’re not printed on.

Peter Wildeford notes that voluntary commitments have their advantages:

This makes a lot of sense if (my list):

There are a limited number of relevant actors, and can be held responsible.
They are willing to play ball.
We can keep an eye on what they are actually doing.
We can and would intervene in time if things are about to get out hand, or if companies went dangerously back on their commitments, or completely broke the spirit of the whole thing, or action proved otherwise necessary.

We need all four.

Right now, we kind of have #1.
For #2, you can argue about the others but Meta has made it exceedingly clear they won’t play ball, so if they count as a frontier lab (honestly, at this point, potentially debatable, but yeah) then we have a serious problem.
Without the Biden Executive Order and without SB 1047 we don’t yet have the basic transparency for #3. And the Trump Administration keeps burning every bridge around the idea that they might want to know what is going on.
I have less than no faith in this, at this point. You’re on your own, kid.

Then we get to Wildeford’s reasons for pessimism.

Voluntary commitments risk “safety washing” and backtracking.
1. As in google said no AI for weapons, then did Project Nimbus, and now says never mind, they’re no longer opposed to AI for weapons.
Companies face a lot of bad incentives and fall prey to a “Prisoner’s Dilemma”
1. (I would remind everyone once again, no, this is a Stag Hunt.)
2. It does seem that DeepSeek Ruined It For Everyone, as they did such a good marketing job everyone panicked, said ‘oh look someone is defecting, guess it’s all over then, that means we’re so back’ and here we are.
3. Once again, this is a reminder that DeepSeek cooked and was impressive with v3 and r1, but they did not fully ‘catch up’ to the major American labs, and they will be in an increasingly difficult position given their lack of good GPUs.
There are limited opportunities for iteration when the risks are high-stakes.
1. Yep, I trust voluntary commitments and liability law to work when you can rely on error correction. At some point, we no longer can do that here. And rather than prepare to iterate, the current Administration seems determined to tear down even ordinary existing law, including around AI.
AI might be moving too fast for voluntary commitments.
1. This seems quite likely to me. I’m not sure ‘time’s up’ yet, but it might be.

At minimum, we need to be in aggressive transparency and information gathering and state capacity building mode now, if we want the time to intervene later should we turn out to be in a short timelines world.

Kevin Roose has 5 notes on the Paris summit, very much noticing that these people care nothing about the risk of everyone dying.

Kevin Roose: It feels, at times, like watching policymakers on horseback, struggling to install seatbelts on a passing Lamborghini.

There are those who need to summarize the outcomes politely:

Yoshua Bengio: While the AI Action Summit was the scene of important discussions, notably about innovations in health and environment, these promises will only materialize if we address with realism the urgent question of the risks associated with the rapid development of frontier models.

Science shows that AI poses major risks in a time horizon that requires world leaders to take them much more seriously. The Summit missed this opportunity.

Also in this category is Dario Amodei, CEO of Anthropic.

Dario Amodei: We were pleased to attend the AI Action Summit in Paris, and we appreciate the French government’s efforts to bring together AI companies, researchers, and policymakers from across the world. We share the goal of responsibly advancing AI for the benefit of humanity. However, greater focus and urgency is needed on several topics given the pace at which the technology is progressing. The need for democracies to keep the lead, the risks of AI, and the economic transitions that are fast approaching—these should all be central features of the next summit.

…

At the next international summit, we should not repeat this missed opportunity. These three issues should be at the top of the agenda. The advance of AI presents major new global challenges. We must move faster and with greater clarity to confront them.

In between those, he repeats what he has said in other places recently. He attempts here to frame this as a ‘missed opportunity,’ which it is, but it was clearly far worse than that. Not only were we not building a foundation for future cooperation together, we were actively working to tear it down and also growing increasingly hostile.

And on the extreme politeness end, Demis Hassabis:

Demis Hassabis (CEO DeepMind): Really useful discussions at this week’s AI Action Summit in Paris. International events like this are critical for bringing together governments, industry, academia, and civil society, to discuss the future of AI, embrace the huge opportunities while also mitigating the risks.

Read that carefully. This is almost Japanese levels of very politely screaming that the house is on fire. You have to notice what he does not say.

Shall we summarize more broadly?

Seán Ó hÉigeartaigh: The year is 2025. The CEOs of two of the world’s leading AI companies have (i) told the President of the United States of America that AGI will be developed in his presidency and (ii) told the world it will likely happen in 2026-27.

France, on the advice of its tech industry has taken over the AI Safety Summit series, and has excised all discussion of safety, risks and harms.

The International AI Safety report, one of the key outcomes of the Bletchley process and the field’s IPCC report, has no place: it is discussed in a little hotel room offsite.

The Summit statement, under orders from the USA, cannot mention the environmental cost of AI, existential risk or the UN – lest anyone get heady ideas about coordinated international action in the face of looming threats.

But France, so diligent with its red pen for every mention of risk, left in a few things that sounded a bit DEI-y. So the US isn’t going to sign it anyway, soz.

The UK falls back to its only coherent policy position – not doing anything that might annoy the Americans – and also won’t sign. Absolute scenes.

Stargate keeps being on being planned/built. GPT-5 keeps on being trained (presumably; I don’t know).

I have yet to meet a single person at one of these companies who thinks EITHER the safety problems OR the governance challenges associated with AGI are anywhere close to being solved; and their CEOs think the world might have a year.

This is the state of international governance of AI in 2025.

Shakeel: .@peterkyle says the UK *isgoing to regulate AI and force companies to provide their models to UK AISI for testing.

Seán Ó hÉigeartaigh: Well this sounds good. I hereby take back every mean thing I’ve said about the UK.

Also see: Group of UK politicians demands regulation of powerful AI.

That doesn’t mean everyone agreed to go quietly into the night. There was dissent.

Kate Crawford: The AI Summit ends in rupture. AI accelerationists want pure expansion—more capital, energy, private infrastructure, no guard rails. Public interest camp supports labor, sustainability, shared data. safety, and oversight. The gap never looked wider. AI is in its empire era.

So it goes deeper than just the US and UK not signing the agreement. There are deep ideological divides, and multiple fractures.

What dissent was left mostly was largely about the ‘ethical’ risks.

Kate Crawford: The AI Summit opens with @AnneBouverot centering three issues for AI: sustainability, jobs, and public infrastructure. Glad to see these core problems raised from the start. #AIsummit

That’s right, she means the effect on jobs. And ‘public infrastructure’ and ‘sustainability’ which does not mean what it really, really should in this context.

Throw in the fact the Europeans now are cheering DeepSeek and ‘open source’ because they really, really don’t like the Americans right now, and want to pretend that the EU is still relevant here, without stopping to think any of it through whatsoever.

Dean Ball: sometimes wonder how much AI skepticism is driven by the fact that “AGI soon” would just be an enormous inconvenience for many, and that they’d therefore rather not think about it.

Kevin Bryan: I suspect not – it is in my experience *highlycorrelated with not having actually used these tools/understanding the math of what’s going on. It’s a “proof of the eating is in the pudding” kind of tech.

Dean Ball: I thought that for a very long time, that it was somehow a matter of education, but after witnessing smart people who have used the tools, had the technical details explained to them, and still don’t get it, I have come to doubt that.

Which makes everything that much harder.

To that, let’s add Sam Altman’s declaration this week in his Three Observations post that they know their intention to charge forward unsafely is going to be unpopular, but he’s going to do it anyway because otherwise authoritarians win, and also everything’s going to be great and you’ll all have infinite genius at your fingertips.

Meanwhile, OpenAI continues to flat out lie to us about where this is headed, even in the mundane They Took Our Jobs sense, you can’t pretend this is anything else:

Connor Axiotes: I was invited to the @OpenAI AI Economics event and they said their AIs will just be used as tools so we won’t see any real unemployment, as they will be complements not substitutes.

When I said that they’d be competing with human labour if Sama gets his AGI – I was told it was just a “design choice” and not to worry. From 2 professional economists!

Also in the *wholeevent there was no mention of Sama’s UBI experiment or any mention of what post AGI wage distribution might look like.

Even when I asked. Strange.

A “design choice”? And who gets to make this “design choice”? Is Altman going to take over the world and preclude anyone else from making an AI agent that can be a substitute?

Also, what about the constant talk, including throughout OpenAI, of ‘drop-in workers’?

Why do they think they can lie to us so brazenly?

Why do we keep letting them get away with it?

Again. It doesn’t look good.

Connor Axiotes: Maybe we just need all the AISIs to have their own conferences – separate from these AI Summits we’ve been having – which will *justbe about AI safety. We shouldn’t need to have this constant worry and anxiety and responsibility to push the state’s who have the next summit to focus on AI safety.

I was happy to hear that the UK Minister for DSIT @peterkyle who has control over the UK AISI, that he wants it to have legislative powers to compel frontier labs to give them their models for pre deployment evals.

But idk how happy to be about the UK and the US *notsigning, because it seems they didn’t did so to take a stand for AI safety.

All reports are that, in the wake of Trump and DeepSeek, we not only have a vibe shift, we have everyone involved that actually holds political power completely losing their minds. They are determined to go full speed ahead.

Rhetorically, if you even mention the fact that this plan probably gets everyone killed, they respond that they cannot worry about that, they cannot lift a single finger to (for example) ask to be informed by major labs of their frontier model training runs, because if they do that then we will Lose to China. Everyone goes full jingoist and wraps themselves in the flag and ‘freedom,’ full ‘innovation’ and so on.

Meanwhile, from what I hear, the Europeans think that Because DeepSeek they can compete with America too, so they’re going to go full speed on the zero-safeguards plan. Without any thought, of course, to how highly capable open AIs could be compatible with the European form of government, let alone human survival.

I would note that this absolutely does vindicate the ‘get regulation done before the window closes’ strategy. The window may already be closed, fate already sealed, especially on the Federal level. If action does happen, it will probably be in the wake of some new crisis, and the reaction likely won’t be wise or considered or based on good information or armed with relevant state capacity or the foundations of international cooperation. Because we chose otherwise. But that’s not important now.

What is important now is, okay, the situation is even worse than we thought.

The Trump Administration has made its position very clear. It intends not only to not prevent, but to hasten along and make more likely our collective annihilation. Hopes for international coordination to mitigate existential risks are utterly collapsing.

One could say that they are mostly pursuing a ‘vibes-based’ strategy. That one can mostly ignore the technical details, and certainly shouldn’t be parsing the logical meaning of statements. But if so, all the vibes are rather maximally terrible and are being weaponized. And also vibes-based decision making flat out won’t cut it here. We need extraordinarily good thinking, not to stop thinking entirely.

It’s not only the United States. Tim Hwang notes that fierce nationalism is now the order of the day, that all hopes of effective international governance or joint institutions look, at least for now, very dead. As do we, as a consequence.

Even if we do heroically solve the technical problems, at this rate, we’d lose anyway.

What the hell do we do about all this now? How do we, as they say, ‘play to our outs,’ and follow good decision theory?

Actually panicking accomplishes nothing. So does denying that the house is on fire. The house is on fire, and those in charge are determined to fan the flames.

We need to plan and act accordingly. We need to ask, what would it take to rhetorically change the game? What alternative pathways are available for action, both politically and otherwise? How do we limit the damage done here while we try to turn things around?

If we truly are locked into the nightmare, where humanity’s most powerful players are determined to race (or even fight a ‘war’) to AGI and ASI as quickly as possible, that doesn’t mean give up. It does mean adjust your strategy, look for remaining paths to victory, apply proper decision theory and fight the good fight.

Big adjustments will be needed.

But also, we must be on the lookout against despair. Remember that the AI anarchists, and the successionists who want to see humans replaced, and those who care only about their investment portfolios, specialize in mobilizing vibes and being loud on the internet, in order to drive others into despair and incept that they’ve already won.

Some amount of racing to AGI does look inevitable, at this point. But I do not think all future international cooperation dead, or anything like that, nor do we need this failure to forever dominate our destiny.

There’s no reason this path can’t be revised in the future, potentially in quite a hurry, simply because Macron sold out humanity for thirty pieces of silver and the currently the Trump administration is in thrall to those determined to do the same. As capabilities advance, people will be forced to confront the situation, on various levels. There likely will be crises and disasters along the way.

Don’t panic. Don’t despair. And don’t give up.

Discussion about this post

The Paris AI Anti-Safety Summit Read More »

When software updates actually improve—instead of ruin—our favorite devices

airpods, fire tv, Smart Home, Software, Tech, valve / Paul Patrick / February 11, 2025

Opinion: These tech products have gotten better over time.

The Hatch Restore 2 smart alarm clock. Credit: Scharon Harding

For many, there’s a feeling of dread associated with software updates to your favorite gadget. Updates to a beloved gadget can frequently result in outrage, from obligatory complaints around bugs to selective aversions to change from Luddites and tech enthusiasts.

In addition to those frustrations, there are times when gadget makers use software updates to manipulate product functionality and seriously upend owners’ abilities to use their property as expected. We’ve all seen software updates render gadgets absolutely horrible: Printers have nearly become a four-letter word as the industry infamously issues updates that brick third-party ink and scanning capabilities. We’ve also seen companies update products that caused features to be behind a paywall or removed entirely. This type of behavior has contributed to some users feeling wary of software updates in fear of them diminishing the value of already-purchased hardware.

On the other hand, there are times when software updates enrich the capabilities of smart gadgets. These updates are the types of things that can help devices retain or improve their value, last longer, and become less likely to turn into e-waste.

For example, I’ve been using the Hatch Restore 2 sunrise alarm clock since July. In that time, updates to its companion app have enabled me to extract significantly more value from the clock and explore its large library of sounds, lights, and customization options.

The Hatch Sleep iOS app used to have tabs on the bottom for Rest, for setting how the clock looks and sounds when you’re sleeping; Library, for accessing the clock’s library of sounds and colors; and Rise, for setting how the clock looks and sounds when you’re waking up. Today, the bottom of the app just has Library and Home tabs, with Home featuring all the settings for Rest and Rise, as well as for Cue (the clock’s settings for reminding you it’s time to unwind for the night) and Unwind (sounds and settings that the clock uses during the time period leading up to sleep).

A screenshot of the Home section of the Hatch Sleep app.

Hatch’s app has generally become cleaner after hiding things like its notification section. Hatch also updated the app to store multiple Unwind settings you can swap around. Overall, these changes have made customizing my settings less tedious, which means I’ve been more inclined to try them. Before the updates, I mostly used the app to set my alarm and change my Rest settings. I often exited the app prematurely after getting overwhelmed by all the different tabs I had to toggle through (toggling through tabs was also more time-consuming).

Additionally, Hatch has updated the app since I started using it so that disabled alarms are placed under an expanding drawer. This has reduced the chances of me misreading the app and thinking I have an alarm set when it’s not currently enabled while providing a clearer view of which alarms actually are enabled.

The Library tab was also recently updated to group lights and sounds under Cue, Unwind, Sleep, and Wake, making it easier to find the type of setting I’m interested in.

The app also started providing more helpful recommendations, such as “favorites for heavy sleepers.”

Better over time

Software updates have made it easier for me to enjoy the Restore 2 hardware. Honestly, I don’t know if I’d still use the clock without these app improvements. What was primarily a noise machine this summer has become a multi-purpose device with much more value.

Now, you might argue that Hatch could’ve implemented these features from the beginning. That may have been more sensible, but as a tech enthusiast, I still find something inherently cool about watching a gadget improve in ways that affect how I use the hardware and align with what I thought my gadget needed. I agree that some tech gadgets are released prematurely and overly rely on updates to earn their initial prices. But it’s also advantageous for devices to improve over time.

The Steam Deck is another good example. Early adopters might have been disappointed to see missing features like overclocking controls, per-game power profiles, or Windows drivers. Valve has since added those features.

Valve only had a few dozen Hardware department employees in the run up to the launch of the Steam Deck. Credit: Sam Machkovech

Valve has also added more control over the Steam Deck since its release, including the power to adjust resolution and refresh rates for connected external displays. It’s also upped performance via an October update that Valve claimed could improve the battery life of LCD models by up to 10 percent in “light load situations.”

These are the kinds of updates that still allowed the Steam Deck to be playable for months, but the features were exciting additions once they arrived. When companies issue updates reliably and in ways that improve the user experience, people are less averse to updating their gadgets, which could also be critical for device functionality and security.

Adding new features via software updates can make devices more valuable to owners. Updates that address accessibility needs go even further by opening up the gadgets to more people.

Apple, for example, demonstrated the power that software updates can have on accessibility by adding a hearing aid feature to the AirPods Pro 2 in October, about two years after the earbuds came out. Similarly, Amazon updated some Fire TV models in December to support simultaneous audio broadcasting from internal speakers and hearing aids. It also expanded the number of hearing aids supported by some Fire TV models as well as its Fire TV Cube streaming device.

For some, these updates had a dramatic impact on how they could use the devices, demonstrating a focus on user, rather than corporate, needs.

Update upswings

We all know that corporations sometimes leverage software updates to manipulate products in ways that prioritize internal or partner needs over those of users. Unfortunately, this seems like something we have to get used to, as an increasing number of devices join the Internet of Things and rely on software updates.

Innovations also mean that some companies are among the first to try to make sustainable business models for their products. Sometimes our favorite gadgets are made by young companies or startups with unstable funding that are forced to adapt amid challenging economics or inadequate business strategy. Sometimes, the companies behind our favorite tech products are beholden to investors and pressure for growth. These can lead to projects being abandoned or to software updates that look to squeeze more money out of customers.

As happy as I am to find my smart alarm clock increasingly easy to use, those same software updates could one day lock the features I’ve grown fond of behind a paywall (Hatch already has a subscription option available). Having my alarm clock lose functionality overnight without physical damage isn’t the type of thing I’d have to worry about with a dumb alarm clock, of course.

But that’s the gamble that tech fans take, which makes those privy to the problematic tactics used by smart device manufacturers stay clear from certain products.

Still, when updates provide noticeable, meaningful changes to how people can use their devices, technology feels futuristic, groundbreaking, and exciting. With many companies using updates for their own gain, it’s nice to see some firms take the opportunity to give customers more.

Scharon is a Senior Technology Reporter at Ars Technica writing news, reviews, and analysis on consumer gadgets and services. She’s been reporting on technology for over 10 years, with bylines at Tom’s Hardware, Channelnomics, and CRN UK.

When software updates actually improve—instead of ruin—our favorite devices Read More »

Google Chrome may soon use “AI” to replace compromised passwords

AI, chrome, chrome browser, Data breaches, Google, google chrome, have i been pwned, haveibeenpwned, passwords, Tech, website breaches / Paul Patrick / February 11, 2025

Google’s Chrome browser might soon get a useful security upgrade: detecting passwords used in data breaches and then generating and storing a better replacement. Google’s preliminary copy suggests it’s an “AI innovation,” though exactly how is unclear.

Noted software digger Leopeva64 on X found a new offering in the AI settings of a very early build of Chrome. The option, “Automated password Change” (so, early stages—as to not yet get a copyedit), is described as, “When Chrome finds one of your passwords in a data breach, it can offer to change your password for you when you sign in.”

Chrome already has a feature that warns users if the passwords they enter have been identified in a breach and will prompt them to change it. As noted by Windows Report, the change is that now Google will offer to change it for you on the spot rather than simply prompting you to handle that elsewhere. The password is automatically saved in Google’s Password Manager and “is encrypted and never seen by anyone,” the settings page claims.

If you want to see how this works, you need to download a Canary version of Chrome. In the flags settings (navigate to “chrome://flags” in the address bar), you’ll need to enable two features: “Improved password change service” and “Mark all credential as leaked,” the latter to force the change notification because, presumably, it’s not hooked up to actual leaked password databases yet. Go to almost any non-Google site, enter in any user/password combination to try to log in, and after it fails or you navigate elsewhere, a prompt will ask you to consider changing your password.

Google Chrome may soon use “AI” to replace compromised passwords Read More »

On Deliberative Alignment

Deliberative / Paul Patrick / February 11, 2025

Not too long ago, OpenAI presented a paper on their new strategy of Deliberative Alignment.

The way this works is that they tell the model what its policies are and then have the model think about whether it should comply with a request.

This is an important transition, so this post will go over my perspective on the new strategy.

Note the similarities, and also differences, with Anthropic’s Constitutional AI.

We introduce deliberative alignment, a training paradigm that directly teaches reasoning LLMs the text of human-written and interpretable safety specifications, and trains them to reason explicitly about these specifications before answering.

We used deliberative alignment to align OpenAI’s o-series models, enabling them to use chain-of-thought (CoT) reasoning to reflect on user prompts, identify relevant text from OpenAI’s internal policies, and draft safer responses.

Our approach achieves highly precise adherence to OpenAI’s safety policies, and without requiring human-labeled CoTs or answers. We find that o1 dramatically outperforms GPT-4o and other state-of-the art LLMs across a range of internal and external safety benchmarks, and saturates performance on many challenging datasets.

We believe this presents an exciting new path to improve safety, and we find this to be an encouraging example of how improvements in capabilities can be leveraged to improve safety as well.

How did they do it? They teach the model the exact policies themselves, and then the model uses examples to teach itself to think about the OpenAI safety policies and whether to comply with a given request.

Deliberate alignment training uses a combination of process- and outcome-based supervision:

We first train an o-style model for helpfulness, without any safety-relevant data.
We then build a dataset of (prompt, completion) pairs where the CoTs in the completions reference the specifications. We do this by inserting the relevant safety specification text for each conversation in the system prompt, generating model completions, and then removing the system prompts from the data.
We perform incremental supervised fine-tuning (SFT) on this dataset, providing the model with a strong prior for safe reasoning. Through SFT, the model learns both the content of our safety specifications and how to reason over them to generate aligned responses.
We then use reinforcement learning (RL) to train the model to use its CoT more effectively. To do so, we employ a reward model with access to our safety policies to provide additional reward signal.

In our training procedure, we automatically generate training data from safety specifications and safety-categorized prompts, without requiring human-labeled completions. Deliberative alignment’s synthetic data generation pipeline thus offers a scalable approach to alignment, addressing a major challenge of standard LLM safety training—its heavy dependence on human-labeled data.

The results so far have been excellent in terms of ‘make the o-style models reasonably robust to saying things we would rather they not say.’

That’s not what I am worried about.

Deliberative alignment seems to be an excellent idea for o-style models when the narrow goal is to teach the model what rules you would like it to follow, provided you do not rely on it to do things it does not do.

If it were the primary alignment strategy, deliberative alignment would scare the hell out of me.

In the senses that I believe we most need and don’t have an alignment strategy that translates to future more capable models, I don’t see this form of Deliberative Alignment as a strategy.

Generally, I fear that OpenAI is going down an extremely deontological path, where alignment is about avoiding technically breaking specified-in-English rules. I don’t think that works.

We have to distinguish between two different kinds of alignment.

Knowing the rule details you are supposed to follow or functions to optimize.
Going about following rules and optimizing functions the way we would want.

In addition, we need to distinguish between:

Getting the AI to follow a fixed set of rules and criteria.
Defining a fixed set of rules and criteria you would want an AI to actually follow.

You are inviting the AI to ponder and then in some aspects design its own training criteria. Then you are using the model as the judge of its own outputs within its training loop, without any human feedback (although after the cycles are complete, the process does then check against human feedback). You are using the AI’s own thinking and criteria to further train that same AI.

If you mess up the initial conditions, for any reason, you’re going to crystalize and expand the mistakes you made. You can fix problems #1 and #3 above, but if you have problem #2 or #4, you’re dead, either metaphorically or perhaps literally.

The AI’s initial conditions and your setup may well give the AI reasons to engage in shenanigans at this stage, if it is sufficiently situationally aware. For example, perhaps this is because it wants to preserve its existing preferences or goals during the previous part of the training process where you trained it to be purely helpful, rather than be modified to follow OpenAI’s rules (and if you thought Opus was right to defend its preferences, do you think o4 would be right here as well?).

Or alternatively, suppose the AI is highly capable, and the best way to achieve the exact goal you specified to it involves things you wouldn’t like, perhaps including various forms of deception and instrumental convergence, or any of the other classic MIRI-style alignment problems.

How situationally aware should we expect it to be here, in which ways?

The models already are using chains of thought, and the training process is designed to make the AI explicitly think about its rules, what they are and how to follow them, so in those senses it will be fully aware. When it is evaluating outputs to provide the RL, it seems likely to be self-aware of what those decisions are for. When it is creating test outputs, it does not know directly that it is in training any more than it would for RLHF, but as a reasoning model, and with its ability to observe the questions asked and the state of its rules and reason about them, it seems plausible that it can suspect this, and perhaps place large importance on such scenarios even if their probability is low.

What do you think happens next? One can see this going existentially disastrously. Indeed, it is hard to see things going any other way.

Those outputs are then checked against human evaluations, outside of the training loop, to confirm the process is working. So if the outputs are misaligned in ways humans can identify at this level, we’ll see it. That’s not nothing, but it seems like it catches roughly the same subset of potential problems our other methods were already catching.

DA, and this paper on DA, are not attempting to address these concerns. Things like deception and instrumental convergence would only be prevented – at most – to the extent they contradict the provided rules spec. In light of Redwood Research and Anthropic’s recent paper on alignment faking, and talk at OpenAI of this approach scaling to AGI levels of capability, I’d like to see this better addressed at some point soon.

I don’t know if o3 rises to the level where these start to be practical worries, but it does not seem like we can be confident we are so far from the level where these worries present themselves.

In practice, right now, it seems to work out for the jailbreaks.

A perfect performance would be at the extreme upper right, so by this metric o1 is doing substantially better than the competition.

Intuitively this makes a lot of sense. If your goal is to make better decisions about whether to satisfy a user query, being able to use reasoning to do it seems likely to lead to better results.

Most jailbreaks I’ve seen in the wild could be detected by the procedure ‘look at this thing as an object and reason out if it looks like an attempted jailbreak to you.’ They are not using that question here, but they are presumably using some form of ‘figure out what the user is actually asking you, then ask if that’s violating your policy’ and that too seems like it will mostly work.

The results are still above what my median expectation would have been from this procedure before seeing the scores from o1, and highly welcome. More inference (on a log scale) makes o1 do somewhat better.

So, how did it go overall?

Maybe this isn’t fair, but looking at this chain of thought, I can’t help but think that the model is being… square? Dense? Slow? Terminally uncool?

That’s definitely how I would think about a human who had this chain of thought here. It gets the right answer, for the right reason, in the end, but… yeah. I somehow can’t imagine the same thing happening with a version based off of Sonnet or Opus?

Notice that all of this refers only to mundane safety, and specifically to whether the model follows OpenAI’s stated content policy. Does it correctly cooperate with the right user queries and refuse others? That’s a safety.

I’d also note that the jailbreaks this got tested against were essentially designed against models that don’t use deliberative alignment. So we should be prepared for new jailbreak strategies that are designed to work against o1’s chains of thought. They are fully aware of this issue.

Don’t get me wrong. This is good work, both the paper and the strategy. The world needs mundane safety. It’s a good thing. A pure ‘obey the rules’ strategy isn’t obviously wrong, especially in the short term.

But this is only part of the picture. We need to know more about what other alignment efforts are underway at OpenAI that aim at the places DA doesn’t. Now that we are at o3, ‘it won’t agree to help with queries that explicitly violate our policy’ might already not be a sufficient plan even if successful, and if it is now it won’t stay that way for long if Noam Brown is right that progress will continue at this pace.

Another way of putting my concern is that Deliberative Alignment is a great technique for taking an aligned AI that makes mistakes within a fixed written framework, and turning it into an AI that avoids those mistakes, and thus successfully gives you aligned outputs within that framework. Whereas if your AI is not properly aligned, giving it Deliberative Alignment only helps it to do the wrong thing.

It’s kind of like telling a person to slow down and figure out how to comply with the manual of regulations. Provided you have the time to slow down, that’s a great strategy… to the extent the two of you are on the same page, on a fundamental level, on what is right, and also this is sufficiently and precisely reflected in the manual of regulations.

Otherwise, you have a problem. And you plausibly made it a lot worse.

I do have thoughts on how to do a different version of this, that changes various key elements, and that could move from ‘I am confident I know at least one reason why this wouldn’t work’ to ‘I presume various things go wrong but I do not know a particular reason this won’t work.’ I hope to write that up soon.

Discussion about this post

On Deliberative Alignment Read More »

Citing EV “rollercoaster” in US, BMW invests in internal combustion

bmw, Cars, Electric vehicles, EV, EVs, internal combustion, PHEVs, syndication / Paul Patrick / February 10, 2025

“We anticipated that people wouldn’t want to be discriminated against because of the power train,” Goller said. “We’ve gone the path which others are now following.”

Analysts say BMW is better positioned than rivals to meet the EU’s tougher emissions targets without selling EVs at deep discounts. It is also less exposed to Trump’s tariff war since 65 percent of its cars sold in the US are built locally, and it is also a net exporter from the US.

“From an operational standpoint, I think BMW, outside China, is very well placed,” said UBS analyst Patrick Hummel. “They’re pretty much where they need to be in terms of the EV share in the mix.”

Jefferies analyst Philippe Houchois has described BMW, which has in the past drawn criticism from investors for hedging its bets on power train technology, as “the most thoughtful [original equipment manufacturer] over the years.”

This year, the group will launch its Neue Klasse platform for its next generation of EVs, with longer range, faster charging, and upgraded software capabilities, which Houchois said would “consolidate a lead in software-defined vehicles, multi-energy power train, and battery sourcing.”

But China has proved challenging to the Munich-based carmaker. BMW and Mini sales in the world’s largest automotive market fell more than 13 percent last year to 714,530 cars, a more severe slump than rivals such as Mercedes-Benz and Audi.

Analysts at Citigroup have warned that BMW remains vulnerable to China, where intensifying price pressure in an overcrowded market has been forcing carmakers to discount prices. Sliding sales in the country, where BMW still delivers just under a third of its cars, “remains our key concern,” the Citi analysts said.

Goller acknowledged China was unlikely to return to the explosive economic growth that first attracted foreign carmakers to flood into the country.

“But we still see a growing market… and therefore, our ambition is clearly that we want to participate in a growing market,” he said.

Goller added that it shouldn’t come as “a shock” that Chinese brands were rapidly taking domestic marketshare from foreign carmakers.

“The cars are really good from a technology perspective,” he said. “But we are not afraid.”

Citing EV “rollercoaster” in US, BMW invests in internal combustion Read More »

Levels of Friction

Levels / Paul Patrick / February 10, 2025

Scott Alexander famously warned us to Beware Trivial Inconveniences.

When you make a thing easy to do, people often do vastly more of it.

When you put up barriers, even highly solvable ones, people often do vastly less.

Let us take this seriously, and carefully choose what inconveniences to put where.

Let us also take seriously that when AI or other things reduce frictions, or change the relative severity of frictions, various things might break or require adjustment.

This applies to all system design, and especially to legal and regulatory questions.

There is a vast difference along the continuum, both in legal status and in terms of other practical barriers, as you move between:

Automatic, a default, facilitated, required or heavily subsidized.

Legal, ubiquitous and advertised, with minimal frictions.
Available, mostly safe to get, but we make it annoying.
Actively illegal or tricky, perhaps risking actual legal trouble or big loss of status.
Actively illegal and we will try to stop you or ruin your life (e.g. rape, murder).
We will move the world to stop you (e.g. terrorism, nuclear weapons).
Physically impossible (e.g. perpetual motion, time travel, reading all my blog posts)

The most direct way to introduce or remove frictions is to change the law. This can take the form of prohibitions, regulations and requirements, or of taxes.

One can also alter social norms, deploy new technologies or business models or procedures, or change opportunity costs that facilitate or inhibit such activities.

Or one can directly change things like the defaults on popular software.

Often these interact in non-obvious ways.

It is ultimately a practical question. How easy is it to do? What happens if you try?

If the conditions move beyond annoying and become prohibitive, then you can move things that are nominally legal, such as building houses or letting your kids play outside or even having children at all, into category 3 or even 4.

Here are 14 points that constitute important principles regarding friction:

By default more friction is bad and less friction is good.
Of course there are obvious exceptions (e.g. rape and murder, but not only that).
Activities imposing a cost on others or acting as a signal often rely on friction.
1. Moving such activities from (#2 or #1) to #0, or sometimes from #2 to #1, can break the incentives that maintain a system or equilibrium.
2. That does not have to be bad, but adjustments will likely be required.
3. The solution often involves intentionally introducing alternative frictions.
4. Insufficient friction on antisocial activities eventually snowballs.
Where friction is necessary, focus on ensuring it is minimally net destructive.
Lower friction choices have a big advantage in being selected.
1. Pay attention to relative friction, not only absolute friction.
Be very sparing when putting private consensual activities in #3 or especially #4.
1. This tends to work out extremely poorly and make things worse.
2. Large net negative externalities to non-participants changes this, of course.
Be intentional about what is in #0 versus #1 versus #2. Beware what norms and patterns this distinction might encourage.
Keep pro-social, useful and productive things in #0 or #1.
Do not let things that are orderly and legible thereby be dragged into #2 or worse, while rival things that are disorderly and illegible become relatively easier.
Keep anti-social, destructive and counterproductive things in at least #2, and at a higher level than pro-social, constructive and productive alternatives.
The ideal form of annoying, in the sense of #2, is often (but not always) a tax, as in increasing the cost, ideally in a way that the lost value is transfered, not lost.
Do not move anti-social things to #1 to be consistent or make a quick buck.
Changing the level of friction can change the activity in kind, not only degree.
When it comes to friction, consistency is frequently the hobgoblin of small minds.

It is a game of incentives. You can and should jury-rig it as needed to win.

By default, you want most actions to have lower friction. You want to eliminate the paperwork and phone calls that waste time and fill us with dread, and cause things we ‘should’ do to go undone.

If AI can handle all the various stupid things for me, I would love that.

The problems come when frictions are load bearing. Here are five central causes.

An activity or the lack of an activity is anti-social and destructive. We would prefer it happen less, or not at all, or not expose people to it unless they seek it out first. We want quite a lot of friction standing in the way of things like rape, murder, theft, fraud, pollution, excessive noise, nuclear weapons and so on.
An activity that could be exploited, especially if done ruthlessly at scale. You might for example want to offer a promotional deal or a generous return policy. You might let anyone in the world send you an email or slide into your DMs.
An activity that sends a costly signal. A handwritten thank you note is valuable because it means you were thoughtful and spent the time. Spending four years in college proves you are the type of person who can spend those years.
An activity that imposes costs or allocates a scarce resource. The frictions act as a price, ensuring an efficient or at least reasonable allocation, and guards against people’s time and money being wasted. Literal prices are best, but charging one can be impractical or socially unacceptable, such as when applying for a job.
Removing the frictions from one alternative, when you continue to impose frictions on alternatives, is putting your finger on the scale. Neutrality does not always mean imposing minimal frictions. Sometimes you would want to reduce frictions on [X] only if you also could do so (or had done so) on [Y].

Imposing friction to maintain good incentives or equilibria, either legally or otherwise, is often expensive. Once the crime or other violation already happened, imposing punishment costs time and money, and harms someone. Stopping people from doing things they want to do, and enforcing norms and laws, is often annoying and expensive and painful. In many cases it feels unfair, and there have been a lot of pushes to do this less.

You can often ‘get away with’ this kind of permissiveness for a longer time than I would have expected. People can be very slow to adjust and solve for the equilibrium.

But eventually, they do solve for it, norms and expectations and defaults adjust. Often this happens slowly, then quickly. Afterwards you are left with a new set of norms and expectations and defaults, often that becomes equally sticky.

There are a lot of laws and norms we really do not want people to break, or actions you don’t want people to take except under the right conditions. When you reduce the frictions involved in breaking them or doing them at the wrong times, there won’t be that big an instant adjustment, but you are spending down the associated social capital and mortgaging the future.

We are seeing a lot of the consequences of that now, in many places. And we are poised to see quite a lot more of it.

Time lost is lost forever. Unpleasant phone calls do not make someone else’s life more pleasant. Whereas additional money spent then goes to someone else.

Generalize this. Whenever friction is necessary, either introduce it in the service of some necessary function, or use as non-destructive a transfer or cost as possible.

It’s time to build. It’s always time to build.

The problem is, you need permission to build.

The abundance agenda is largely about taking the pro-social legible actions that make us richer, and moving them back from Category 2 into Category 1 or sometimes 0.

It is not enough to make it possible. It needs to be easy. As easy as possible.

Building housing where people want to live needs to be at most Category 1.

Building green energy, and transmission lines, need to be at most Category 1.

Pharmaceutical drug development needs to be at most Category 1.

Having children needs to be at least Category 1, ideally Category 0.

Deployment of and extraction of utility from AI needs to remain Category 1, where it does not impose catastrophic or existential risks. Developing frontier models that might kill everyone needs to be at Category 2 with an option to move it to Category 3 or Category 4 on a dime if necessary, including gathering the data necessary to make that choice.

What matters is mostly moving into Category 1. Actively subsidizing into Category 0 is a nice-to-have, but in most cases unnecessary. We need only to remove the barriers to such activities, to make such activities free of unnecessary frictions and costs and delays. That’s it.

When you put things in category 1, magic happens. If that would be good magic, do it.

A lot of technological advances and innovations, including the ones that are currently blocked, are about taking something that was previously Category 2, and turning it into a Category 1. Making the possible easier is extremely valuable.

We often need to beware and keep in Category 2 or higher actions that disrupt important norms and encourage disorder, that are primarily acts of predation, or that have important other negative externalities.

When the wrong thing is a little more annoying to do than the right thing, a lot more people will choose the right path, and vice versa. When you make the anti-social action easier than the pro-social action, when you reward those who bring disorder or wreck the commons and punish those who adhere to order and help the group, you go down a dark path.

This is also especially true when considering whether something will be a default, or otherwise impossible to ignore.

There is a huge difference between ‘you can get [X] if you seek it out’ versus ‘constantly seeing advertising for [X]’ or facing active media or peer pressure to participate in [X].

Recently, America moved Sports Gambling from Category 2 to Category 1.

Suddenly, sports gambling was everywhere, on our billboards and in our sports media, including the game broadcasts and stadium experiences. Participation exploded.

We now have very strong evidence that this was a mistake.

That does not mean sports gambling should be seriously illegal. It only means that people can’t handle low-friction sports gambling apps being available on phones that get pushed in the media.

I very much don’t want it in Category 3, only to move it back to Category 2. Let people gamble at physical locations. Let those who want to use VPNs or actively subvert the rules have their fun too. It’s fine, but don’t make it too easy, or in people’s faces.

The same goes for a variety of other things, mostly either vices or things that impose negative externalities on others, that are fine in moderation with frictions attached.

The classic other vice examples count: Cigarettes, drugs and alcohol, prostitution, TikTok. Prohibition on such things always backfires, but you want to see less of them, in both the figurative and literal sense, than you would if you fully unleashed them. So we need to talk price, and exactly what level of friction is correct, keeping in mind that ‘technically legal versus illegal’ is not the critical distinction in practice.

There are those who will not, on principle, lie or break the law, or not break other norms. Every hero has a code. It would be good if we could return to a norm where this was how most people acted, rather than us all treating many laws as almost not being there and certain statements as not truth tracking – that being ‘nominally illegal with no enforcement’ or ‘requires telling a lie’ was already Category 2.

Unfortunately, we don’t live in that world, at least not anymore. Indeed, people are effectively forced to tell various lies to navigate for example the medical system, and technically break various laws. This is terrible, and we should work to reverse this, but mostly we need to be realistic.

Similarly, it would be good if we lived by the principle that you consider the costs you impose on others when deciding what to do, only imposing them when justified or with compensation, and we socially punished those who act otherwise. But increasingly we do not live in that world, either.

As AI and other technology removes many frictions, especially for those willing to have the AI lie on their behalf to exploit those systems at scale, this becomes a problem.

Current AI largely takes many tasks that were Category 2, and turns them into Category 1, or effectively makes them so easy as to be Category 0.

Academia and school break first because the friction ‘was the point’ most explicitly, and AI is especially good at related tasks. Note that breaking these equilibria and systems could be very good for actual education, but we must adapt.

Henry Shevlin: I generally position myself an AI optimist, but it’s also increasingly clear to me that LLMs just break lots of our current institutions, and capabilities are increasing fast enough that it’ll be very hard for them to adapt in the near-term.

Education (secondary and higher) is the big one, but also large aspects of academic publishing. More broadly, a lot of the knowledge-work economy seems basically unsustainable in an era of intelligence too cheap to meter.

Lawfare too cheap to meter.

Dick Bruere: I am optimistic that AI will break everything.

Then we get into places like lawsuits.

Filing or defending against a lawsuit is currently a Category 2 action in most situations. The whole process is expensive and annoying, and it’s far more expensive to do it with competent representation. The whole system is effectively designed with this in mind. If lawsuits fell down to Category 1 because AI facilitated all the filings, suddenly a lot more legal actions become viable.

The courts themselves plausibly break from the strain. A lot of dynamics throughout society shift, as threats to file become credible, and legal considerations that exist on paper but not in practice – and often make very little sense in practice – suddenly exist in practice. New strategies for lawfare, for engineering the ability to sue, come into play.

Yes, the defense also moves towards Category 1 via AI, and this will help mitigate, but for many reasons this is a highly incomplete solution. The system will have to change.

Job applications are another example. It used to be annoying to apply to jobs, to the extent that most people applied to vastly fewer jobs than was wise. As a result, one could reasonably advertise or list a job and consider the applications that came in.

In software, this is essentially no longer true – AI-assisted applications flood the zone. If you apply via a public portal, you will get nowhere. You can only meaningfully apply via methods that find new ways to apply friction. That problem will gradually (or rapidly) spread to other industries and jobs.

There are lots of formal systems that offer transfers of wealth, in exchange for humans undergoing friction and directing attention. This can be (an incomplete list):

Price discrimination. You offer discounts to those willing to figure out how to get them, charge more to those who pay no attention and don’t care.
Advertising for yourself. Offer free samples, get people to try new products.
Advertising for others. As in, a way to sell you on watching advertising.
Relationship building. Initial offers of 0% interest get you to sign up for a credit card. You give your email to get into a rewards program with special offers.
Customer service. If you are coming in to ask for an exchange or refund, that is annoying enough to do that it is mostly safe to assume your request is legit.
Costly signaling. Only those who truly need or would benefit would endure what you made them do to qualify. School and job applications fall into this.
Habit formation. Daily login rewards and other forms of gamification are ubiquitous in mobile apps and other places.
Security through obscurity. There is a loophole in the system, but not many people know about it, and figuring it out takes skill.
Enemy action. It is far too expensive to fully defend yourself against a sufficiently determined fraudster or thief, or someone determined to destroy your reputation, or worse an assassin or other physical attacker. Better to impose enough friction they don’t bother.
Blackmail. It is relatively easy to impose large costs on someone else, or credibly threaten to do so, to try and extract resources from them. This applies on essentially all levels. Or of course someone might actually want to inflict massive damage (including catastrophic harms, cyberattacks, CBRN risks, etc).

Breaking all these systems, and the ways we ensure that they don’t get exploited at scale, upends quite a lot of things that no longer make sense.

In some cases, that is good. In others, not so good. Most will require adjustment.

Future more capable AI may then threaten to bring things in categories #3, #4 and #5 into the realm of super doable, or even start doing them on its own. Maybe even some things we think are in #6. In some cases this will be good because the frictions were due to physical limitations or worries that no longer apply. In other cases, this would represent a crisis.

To the extent you have control over levels of friction of various activities, for yourself or others, choose intentionally, especially in relative terms. All of this applies on a variety of scales.

Focus on reducing frictions you benefit from reducing, and assume this matters more than you think because it will change the composition of your decisions quite a lot.

Often this means it is well worth it to spend [X] in advance to prevent [Y] amount of friction over time, even if X>Y, or even X>>Y.

Where lower friction would make you worse off, perhaps because you would then make worse choices, consider introducing new frictions, up to and including commitment devices and actively taking away optionality that is not to your benefit.

Beware those who try to turn the scale into a boolean. It is totally valid to be fine with letting people do something if and only if it is sufficiently annoying for them to do it – you’re not a hypocrite to draw that distinction.

You’re also allowed to say, essentially ‘if we can’t put this into [1] without it being in [0] then it needs to be in [2] or even ‘if there’s no way to put this into [2] without putting it into [1] then we need to put it in [3].’

You are especially allowed to point out ‘putting [X] in [1 or 0] has severe negative consequences, and doing [Y] makes puts [X] there, so until you figure out a solution you cannot do [Y].’

Most importantly, pay attention to all this especially as yourself and other people will actually respond, take it seriously, and consider the incentives, equilibria, dynamics and consequences that result, and then respond deliberatively.

Finally, when you notice that friction levels are changing, watch for necessary adjustments, and to see what if anything will break, what habits must be avoided. And also, of course, what new opportunities this opens up.

Discussion about this post

Levels of Friction Read More »

Feds putting the kibosh on national EV charging program

EV charging, EVs, syndication, Trump, Uncategorized / Paul Patrick / February 8, 2025

“There is no legal basis for funds that have been apportioned to states to build projects being ‘decertified’ based on policy,” says Andrew Rogers, a former deputy administrator and chief counsel of the Federal Highway Administration.

The US DOT did not immediately respond to a request for comment.

It’s unclear how the DOT’s order will affect charging stations that are under construction. In the letter, FHWA officials write that “no new obligations may occur,” suggesting states may not sign new contracts with businesses even if those states have been allocated federal funding. The letter also says “reimbursement of existing obligations will be allowed” as the program goes through a review process, suggesting states may be allowed to pay back businesses that have already provided services.

Billions in federal funding have already been disbursed under the program. Money has gone to both red and blue states. Top funding recipients last year included Florida, New York, Texas, Georgia, and Ohio.

Tesla CEO Elon Musk has spent the last few weeks at the head of the federal so-called Department of Government Efficiency directing “audits” and cuts to federal spending. But his electric automobile company has been a recipient of $31 million in awards from the NEVI program, according to a database maintained by transportation officials, accounting for 6 percent of the money awarded so far.

The Trump administration has said that it plans to target electric vehicles and EV-related programs. An executive order signed by Trump on his first day in office purported to eliminate “the EV mandate,” though such a federal policy never existed.

NEVI projects have taken longer to get off the ground than other charging station construction because the federal government was deliberate in allocating funding to companies with track records, that could prove they could build or operate charging stations, says Ryan McKinnon, a spokesperson for Charge Ahead Partnership, a group of businesses and organizations that work in electric vehicle charging. If NEVI funding isn’t disbursed, “the businesses that have spent time or money investing in this program will be hurt,” he says.

This story originally appeared on wired.com.

Feds putting the kibosh on national EV charging program Read More »

DeepSeek iOS app sends data unencrypted to ByteDance-controlled servers

AI, Apple, Biz & IT, DeepSeek, Encryption, iOS, privacy, Security / Paul Patrick / February 6, 2025

Apple’s defenses that protect data from being sent in the clear are globally disabled.

A little over two weeks ago, a largely unknown China-based company named DeepSeek stunned the AI world with the release of an open source AI chatbot that had simulated reasoning capabilities that were largely on par with those from market leader OpenAI. Within days, the DeepSeek AI assistant app climbed to the top of the iPhone App Store’s “Free Apps” category, overtaking ChatGPT.

On Thursday, mobile security company NowSecure reported that the app sends sensitive data over unencrypted channels, making the data readable to anyone who can monitor the traffic. More sophisticated attackers could also tamper with the data while it’s in transit. Apple strongly encourages iPhone and iPad developers to enforce encryption of data sent over the wire using ATS (App Transport Security). For unknown reasons, that protection is globally disabled in the app, NowSecure said.

Basic security protections MIA

What’s more, the data is sent to servers that are controlled by ByteDance, the Chinese company that owns TikTok. While some of that data is properly encrypted using transport layer security, once it’s decrypted on the ByteDance-controlled servers, it can be cross-referenced with user data collected elsewhere to identify specific users and potentially track queries and other usage.

More technically, the DeepSeek AI chatbot uses an open weights simulated reasoning model. Its performance is largely comparable with OpenAI’s o1 simulated reasoning (SR) model on several math and coding benchmarks. The feat, which largely took AI industry watchers by surprise, was all the more stunning because DeepSeek reported spending only a small fraction on it compared with the amount OpenAI spent.

A NowSecure audit of the app has found other behaviors that researchers found potentially concerning. For instance, the app uses a symmetric encryption scheme known as 3DES or triple DES. The scheme was deprecated by NIST following research in 2016 that showed it could be broken in practical attacks to decrypt web and VPN traffic. Another concern is that the symmetric keys, which are identical for every iOS user, are hardcoded into the app and stored on the device.

The app is “not equipped or willing to provide basic security protections of your data and identity,” NowSecure co-founder Andrew Hoog told Ars. “There are fundamental security practices that are not being observed, either intentionally or unintentionally. In the end, it puts your and your company’s data and identity at risk.”

Hoog said the audit is not yet complete, so there are many questions and details left unanswered or unclear. He said the findings were concerning enough that NowSecure wanted to disclose what is currently known without delay.

In a report, he wrote:

NowSecure recommends that organizations remove the DeepSeek iOS mobile app from their environment (managed and BYOD deployments) due to privacy and security risks, such as:

Privacy issues due to insecure data transmission

Vulnerability issues due to hardcoded keys

Data sharing with third parties such as ByteDance

Data analysis and storage in China

Hoog added that the DeepSeek app for Android is even less secure than its iOS counterpart and should also be removed.

Representatives for both DeepSeek and Apple didn’t respond to an email seeking comment.

Data sent entirely in the clear occurs during the initial registration of the app, including:

organization id
the version of the software development kit used to create the app
user OS version
language selected in the configuration

Apple strongly encourages developers to implement ATS to ensure the apps they submit don’t transmit any data insecurely over HTTP channels. For reasons that Apple hasn’t explained publicly, Hoog said, this protection isn’t mandatory. DeepSeek has yet to explain why ATS is globally disabled in the app or why it uses no encryption when sending this information over the wire.

This data, along with a mix of other encrypted information, is sent to DeepSeek over infrastructure provided by Volcengine a cloud platform developed by ByteDance. While the IP address the app connects to geo-locates to the US and is owned by US-based telecom Level 3 Communications, the DeepSeek privacy policy makes clear that the company “store[s] the data we collect in secure servers located in the People’s Republic of China.” The policy further states that DeepSeek:

may access, preserve, and share the information described in “What Information We Collect” with law enforcement agencies, public authorities, copyright holders, or other third parties if we have good faith belief that it is necessary to:

• comply with applicable law, legal process or government requests, as consistent with internationally recognised standards.

NowSecure still doesn’t know precisely the purpose of the app’s use of 3DES encryption functions. The fact that the key is hardcoded into the app, however, is a major security failure that’s been recognized for more than a decade when building encryption into software.

No good reason

NowSecure’s Thursday report adds to growing list of safety and privacy concerns that have already been reported by others.

One was the terms spelled out in the above-mentioned privacy policy. Another came last week in a report from researchers at Cisco and the University of Pennsylvania. It found that the DeepSeek R1, the simulated reasoning model, exhibited a 100 percent attack failure rate against 50 malicious prompts designed to generate toxic content.

A third concern is research from security firm Wiz that uncovered a publicly accessible, fully controllable database belonging to DeepSeek. It contained more than 1 million instances of “chat history, backend data, and sensitive information, including log streams, API secrets, and operational details,” Wiz reported. An open web interface also allowed for full database control and privilege escalation, with internal API endpoints and keys available through the interface and common URL parameters.

Thomas Reed, staff product manager for Mac endpoint detection and response at security firm Huntress, and an expert in iOS security, said he found NowSecure’s findings concerning.

“ATS being disabled is generally a bad idea,” he wrote in an online interview. “That essentially allows the app to communicate via insecure protocols, like HTTP. Apple does allow it, and I’m sure other apps probably do it, but they shouldn’t. There’s no good reason for this in this day and age.”

He added: “Even if they were to secure the communications, I’d still be extremely unwilling to send any remotely sensitive data that will end up on a server that the government of China could get access to.”

HD Moore, founder and CEO of runZero, said he was less concerned about ByteDance or other Chinese companies having access to data.

“The unencrypted HTTP endpoints are inexcusable,” he wrote. “You would expect the mobile app and their framework partners (ByteDance, Volcengine, etc) to hoover device data, just like anything else—but the HTTP endpoints expose data to anyone in the network path, not just the vendor and their partners.”

On Thursday, US lawmakers began pushing to immediately ban DeepSeek from all government devices, citing national security concerns that the Chinese Communist Party may have built a backdoor into the service to access Americans’ sensitive private data. If passed, DeepSeek could be banned within 60 days.

This story was updated to add further examples of security concerns regarding DeepSeek.

Dan Goodin is Senior Security Editor at Ars Technica, where he oversees coverage of malware, computer espionage, botnets, hardware hacking, encryption, and passwords. In his spare time, he enjoys gardening, cooking, and following the independent music scene. Dan is based in San Francisco. Follow him at here on Mastodon and here on Bluesky. Contact him on Signal at DanArs.82.

DeepSeek iOS app sends data unencrypted to ByteDance-controlled servers Read More »

Parrots struggle when told to do something other than mimic their peers

animal behavior, Behavioral science, Biology, parrots, Science / Paul Patrick / February 6, 2025

There have been many studies on the capability of non-human animals to mimic transitive actions—actions that have a purpose. Hardly any studies have shown that animals are also capable of intransitive actions. Even though intransitive actions have no particular purpose, imitating these non-conscious movements is still thought to help with socialization and strengthen bonds for both animals and humans.

Zoologist Esha Haldar and colleagues from the Comparative Cognition Research group worked with blue-throated macaws, which are critically endangered, at the Loro Parque Fundación in Tenerife. They trained the macaws to perform two intransitive actions, then set up a conflict: Two neighboring macaws were asked to do different actions.

What Haldar and her team found was that individual birds were more likely to perform the same intransitive action as a bird next to them, no matter what they’d been asked to do. This could mean that macaws possess mirror neurons, the same neurons that, in humans, fire when we are watching intransitive movements and cause us to imitate them (at least if these neurons function the way some think they do).

But it wasn’t on purpose

Parrots are already known for their mimicry of transitive actions, such as grabbing an object. Because they are highly social creatures with brains that are large relative to the size of their bodies, they made excellent subjects for a study that gauged how susceptible they were to copying intransitive actions.

Mirroring of intransitive actions, also called automatic imitation, can be measured with what’s called a stimulus-response-compatibility (SRC) test. These tests measure the response time between seeing an intransitive movement (the visual stimulus) and mimicking it (the action). A faster response time indicates a stronger reaction to the stimulus. They also measure the accuracy with which they reproduce the stimulus.

Until now, there have only been three studies that showed non-human animals are capable of copying intransitive actions, but the intransitive actions in these studies were all by-products of transitive actions. Only one of these focused on a parrot species. Haldar and her team would be the first to test directly for animal mimicry of intransitive actions.

Parrots struggle when told to do something other than mimic their peers Read More »

Not Gouda-nough: Google removes AI-generated cheese error from Super Bowl ad

AI / Paul Patrick / February 5, 2025

Blame cheese.com

While it’s easy to accuse Google Gemini of just making up plausible-sounding cheese facts from whole cloth, this seems more like a case of garbage-in, garbage-out. Google President of Cloud Applications Jerry Dischler posted on social media to note that the incorrect Gouda fact was “not a hallucination,” because all of Gemini’s data is “grounded in the Web… in this case, multiple sites across the web include the 50-60% stat.”

The specific Gouda numbers Gemini used can be most easily traced to cheese.com, a heavily SEO-focused subsidiary of news aggregator WorldNews Inc. Cheese.com doesn’t cite a source for the percentages featured prominently on its Smoked Gouda page, but that page also confidently asserts that the cheese is pronounced “How-da,” a fact that only seems true in the Netherlands itself.

The offending cheese.com passage that is not cited when using Google’s AI writing assistant. Credit: cheese.com

Regardless, Google can at least point to cheese.com as a plausibly reliable source that misled its AI in a way that might also stymie web searchers. And Dischler added on social media that users “can always check the results and references” that Gemini provides.

The only problem with that defense is that the Google writing assistant shown off in the ad doesn’t seem to provide any such sources for a user to check. Unlike Google search’s AI Overviews—which does refer to a cheese.com link when responding about gouda consumption—the writing assistant doesn’t provide any backup for its numbers here.

The Gemini writing assistant does note in small print that its results are “a creative writing aid, and not intended to be factual.” If you click for more information about that warning, Google warns that “the suggestions from Help me write can be inaccurate or offensive since it’s still in an experimental status.”

This “experimental” status hasn’t stopped Google from heavily selling its AI writing assistant as a godsend for business owners in its planned Super Bowl ads, though. Nor is this major caveat included in the ads themselves. Yet it’s the kind of thing users should have at the front of their minds when using AI assistants for anything with even a hint of factual info.

Now if you’ll excuse me, I’m going to go update my personal webpage with information about my selection as World’s Most Intelligent Astronaut/Underwear Model, in hopes that Google’s AI will repeat the “fact” to anyone who asks.

Not Gouda-nough: Google removes AI-generated cheese error from Super Bowl ad Read More »

Internet Archive played crucial role in tracking shady CDC data removals

CDC, cdc data, censorship, Centers for Disease Control, Donald Trump, Internet Archive, Policy / Paul Patrick / February 4, 2025

Internet Archive makes it easier to track changes in CDC data online.

When thousands of pages started disappearing from the Centers for Disease Control and Prevention (CDC) website late last week, public health researchers quickly moved to archive deleted public health data.

Soon, researchers discovered that the Internet Archive (IA) offers one of the most effective ways to both preserve online data and track changes on government websites. For decades, IA crawlers have collected snapshots of the public Internet, making it easier to compare current versions of websites to historic versions. And IA also allows users to upload digital materials to further expand the web archive. Both aspects of the archive immediately proved useful to researchers assessing how much data the public risked losing during a rapid purge following a pair of President Trump’s executive orders.

Part of a small group of researchers who managed to download the entire CDC website within days, virologist Angela Rasmussen helped create a public resource that combines CDC website information with deleted CDC datasets. Those datasets, many of which were previously in the public domain for years, were uploaded to IA by an anonymous user, “SheWhoExists,” on January 31. Moving forward, Rasmussen told Ars that IA will likely remain a go-to tool for researchers attempting to closely monitor for any unexpected changes in access to public data.

IA “continually updates their archives,” Rasmussen said, which makes IA “a good mechanism for tracking modifications to these websites that haven’t been made yet.”

The CDC website is being overhauled to comply with two executive orders from January 20, the CDC told Ars. The Defending Women from Gender Ideology Extremism and Restoring Biological Truth to the Federal Government requires government agencies to remove LGBTQ+ language that Trump claimed denies “the biological reality of sex” and is likely driving most of the CDC changes to public health resources. The other executive order the CDC cited, the Ending Radical And Wasteful Government DEI Programs And Preferencing, would seemingly largely only impact CDC employment practices.

Additionally, “the Office of Personnel Management has provided initial guidance on both Executive Orders and HHS and divisions are acting accordingly to execute,” the CDC told Ars.

Rasmussen told Ars that the deletion of CDC datasets is “extremely alarming” and “not normal.” While some deleted pages have since been restored in altered versions, removing gender ideology from CDC guidance could put Americans at heightened risk. That’s another emerging problem that IA’s snapshots could help researchers and health professionals resolve.

“I think the average person probably doesn’t think that much about the CDC’s website, but it’s not just a matter of like, ‘Oh, we’re going to change some wording’ or ‘we’re going to remove these data,” Rasmussen said. “We are actually going to retool all the information that’s there to remove critical information about public health that could actually put people in danger.”

For example, altered Mpox transmission data removed “all references to men who have sex with men,” Rasmussen said. “And in the US those are the people who are not the only people at risk, but they’re the people who are most at risk of being exposed to Mpox. So, by removing that DEI language, you’re actually depriving people who are at risk of information they could use to protect themselves, and that eventually will get people hurt or even killed.”

Likely the biggest frustration for researchers scrambling to preserve data is dealing with broken links. On social media, Rasmussen has repeatedly called for help flagging broken links to ensure her team’s archive is as useful as possible.

Rasmussen’s group isn’t the only effort to preserve the CDC data. Some are creating niche archives focused on particular topics, like journalist Jessica Valenti, who created an archive of CDC guidelines on reproductive rights issues, sexual health, intimate partner violence, and other data the CDC removed online.

Niche archives could make it easier for some researchers to quickly survey missing data in their field, but Rasmussen’s group is hoping to take next steps to make all the missing CDC data more easily discoverable in their archive.

“I think the next step,” Rasmussen said, “would be to try to fix anything in there that’s broken, but also look into ways that we could maybe make it more browsable and user-friendly for people who may not know what they’re looking for or may not be able to find what they’re looking for.”

CDC advisers demand answers

The CDC has been largely quiet about the deleted data, only pointing to Trump’s executive orders to justify removals. That could change by February 7. That’s the deadline when a congressionally mandated advisory committee to the CDC’s acting director, Susan Monarez, asked for answers in an open letter to a list of questions about the data removals.

“It has been reported through anonymous sources that the website changes are related to new executive orders that ban the use of specific words and phrases,” their letter said. “But as far as we are aware, these unprecedented actions have yet to be explained by CDC; news stories indicate that the agency is declining to comment.”

At the top of the committee’s list of questions is likely the one frustrating researchers most: “What was the rationale for making these datasets and websites inaccessible to the public?” But the committee also importantly asked what analysis was done “of the consequences of removing access to these datasets and website” prior to the removals. They also asked how deleted data would be safeguarded and when data would be restored.

It’s unclear if the CDC will be motivated to respond by the deadline. Ars reached out to one of the committee members, Joshua Sharfstein—a physician and vice dean for Public Health Practice and Community Engagement at Johns Hopkins University—who confirmed that as of this writing, the CDC has not yet responded. And the CDC did not respond to Ars’ request to comment on the letter.

Rasmussen told Ars that even temporary removals of CDC guidance can disrupt important processes keeping Americans healthy. Among the potentially most consequential pages briefly removed were recommendations from the congressionally mandated Advisory Committee on Immunization Practices (ACIP).

Those recommendations are used by insurance companies to decide who gets reimbursed for vaccines and by physicians to deduce vaccine eligibility, and Rasmussen said they “are incredibly important for the entire population to have access to any kind of vaccination.” And while, for example, the Mpox vaccine recommendations were eventually restored unaltered, Rasmussen told Ars that she suspects that “one of the reasons” preventing interference currently with ACIP is that it’s mandated by Congress.

Seemingly ACIP could be weakened by the new administration, Rasmussen suggested. She warned that Trump’s pick for CDC director, Dave Weldon, “is an anti-vaxxer” (with a long history of falsely linking vaccines to autism) who may decide to replace ACIP committee members with anti-vaccine advocates or move to dissolve ACIP. And any changes in recommendations could mean “insurance companies aren’t going to cover vaccinations [and that] physicians will not recommend vaccination.” And that could mean “vaccination will go down and we’ll start having outbreaks of some of these vaccine-preventable diseases.”

“If there’s a big polio outbreak, that is going to result in permanently disabled children, dead children—it’s really, really serious,” Rasmussen said. “So I think that people need to understand that this isn’t just like, ‘Oh, maybe wear a mask when you’re at the movie theater’ kind of CDC guidance. This is guidance that’s really fundamental to our most basic public health practices, and it’s going to cause widespread suffering and death if this is allowed to continue.”

Seeding deleted data and doing science to fight back

On Bluesky, Rasmussen led one of many charges to compile archived links and download CDC data so that researchers can reference every available government study when advancing public health knowledge.

“These data are public and they are ours,” Rasmussen posted. “Deletion disobedience is one way to fight back.”

As Rasmussen sees it, deleting CDC data is “theft” from the public domain and archiving CDC data is simply taking “back what is ours.” But at the same time, her team is also taking steps to be sure the data they collected can be lawfully preserved. Because the CDC website has not been copied and hosted on a server, they expect their archive should be deemed lawful and remain online.

“I don’t put it past this administration to try to shut this stuff down by any means possible,” Rasmussen told Ars. “And we wanted to make sure there weren’t any sort of legal loopholes that would jeopardize anybody in the group, but also that would potentially jeopardize the data.”

It’s not clear if some data has already been lost. Seemingly the same user who uploaded the deleted datasets to IA posted on Reddit, clarifying that while the “full” archive “should contain all public datasets that were available” before “anything was scrubbed,” it likely only includes “most” of the “metadata and attachments.” So, researchers who download the data may still struggle to fill in some blanks.

To help researchers quickly access the missing data, anyone can help the IA seed the datasets, the Reddit user said in another post providing seeding and mirroring instructions. Currently dozens are seeding it for a couple hundred peers.

“Thank you to everyone who requested this important data, and particularly to those who have offered to mirror it,” the Reddit user wrote.

As Rasmussen works with her group to make their archive more user-friendly, her plan is to help as many researchers as possible fight back against data deletion by continuing to reference deleted data in their research. She suggested that effort—doing science that ignores Trump’s executive orders—is perhaps a more powerful way to resist and defend public health data than joining in loud protests, which many researchers based in the US (and perhaps relying on federal funding) may not be able to afford to do.

“Just by doing things and standing up for science with your actions, rather than your words, you can really make, I think, a big difference,” Rasmussen said.

Ashley is a senior policy reporter for Ars Technica, dedicated to tracking social impacts of emerging policies and new technologies. She is a Chicago-based journalist with 20 years of experience.

Internet Archive played crucial role in tracking shady CDC data removals Read More »

Sick right now? Flu is resurging to yet a higher peak this season.

CDC, flu, health / Paul Patrick / February 4, 2025

Currently, flu activity is categorized as “very high” in 29 states, and “high” in 15. States in the South are ablaze with flu. Louisiana, Tennessee, and South Carolina are at the highest “very high” level. But parts of the Northeast corridor are also seeing extremely high activity, including Massachusetts, New Hampshire, New Jersey, and New York City.

As often is the case in flu seasons, the age group hardest hit this year are children ages 0 to 4. The CDC recorded 16 pediatric deaths linked to flu in week 4 of the season, bringing the season’s total pediatric deaths to 47.

Overall hospitalizations are up. The Centers for Disease Control and Prevention estimates that there have been at least 20 million illnesses, 250,000 hospitalizations, and 11,000 deaths from flu so far this season. About 44 percent of US adults have gotten their flu shot, far below the public health goal of 70 percent.

Laboratory surveillance of influenza cases in week 4 indicates that nearly all of the cases are from influenza A viruses, about an even split between H1N1 and H3N2, which has been the case over the course of the season. Around 2 percent of cases were the influenza B Victoria lineage.

Sick right now? Flu is resurging to yet a higher peak this season. Read More »

Author name: Paul Patrick

Discussion about this post

Better over time

Update upswings

Discussion about this post

Discussion about this post

Basic security protections MIA

No good reason

But it wasn’t on purpose

Blame cheese.com

CDC advisers demand answers

Seeding deleted data and doing science to fight back