Tomorrow I will fly out to San Francisco, to spend Friday through Monday at the LessOnline conference at Lighthaven in Berkeley. If you are there, by all means say hello. If you are in the Bay generally and want to otherwise meet, especially on Monday, let me know that too and I will see if I have time to make that happen.
Even without that hiccup, it continues to be a game of playing catch-up. Progress is being made, but we are definitely not there yet (and everything not AI is being completely ignored for now).
Last week I pointed out seven things I was unable to cover, along with a few miscellaneous papers and reports.
However, OpenAI developments continue. Thanks largely to Helen Toner’s podcast, some form of that is going back into the queue. Some other developments, including new media deals and their new safety board, are being covered normally.
Which model is the best right now? Michael Nielsen is gradually moving back to Claude Opus, and so am I. GPT-4o is fast and has some nice extra features, so when I figure it is ‘smart enough’ I will use it, but when I care most about quality and can wait a bit I increasingly go to Opus. Gemini I’m reserving for a few niche purposes, when I need Google integration, long context windows or certain other features.
Analyze financial statements and predict future performance enabling high Sharpe ratio investing, says new paper. I do not doubt that such a technique is ‘part of a balanced portfolio of analysis techniques’ due to it being essentially free, but color me skeptical (although I have not read the paper.) You can anonymize the company all you like, that does not mean the patterns were not picked up, or that past performance is not being used to model future success in a way that will work far better on this kind of test than in reality, especially when everyone else has their own LLMs doing similar projections, and when AI is transforming the economy and everyone’s performance.
LLMs for language learning. Ben Hoffman points to his friend’s new program LanguageZen, which has a bunch of automated customization and other good ideas mixed in. If I had more free time I would be intrigued. Ben thinks that current LLMs are not good enough yet. I think they very much are, if you give them the scaffolding, as the context window can fully include your entire experiential history with the new language, but it will take some work to get all the customizations right.
Google put out Gemini while it had, shall we say, some issues. The image model had some big issues, also the text model had some big issues. They had a bad time, and had to take down images of humans for a while.
The models kept improving. At this point I am using a mix of Gemini, Claude and GPT-4o, depending on the exact task, sometimes comparing answers.
It does seem, however, that the current version of the ‘AI overview’ on Google search has a rather large problem.
In this case, it is not about accusations of wokeness or racism or bias.
Henry Shevlin: So many people in my feed overindexing on Google’s AI Overview woes and claiming “aha, you see, AI sucks”. But ChatGPT, Claude, and Perplexity don’t have these issues. What’s happened with AI Overviews is very weird and messed up in a distinctive and novel way.
AI Overviews seems to derive chunks of its summaries wholecloth from single sources in a way I’ve not seen on other models. I’ve been using ChatGPT daily for the last 18 months and even doing adversarial testing on it, and never seen anything in this league.
Ivan’s Cat: It is related to the RAG part, so the standard ChatGPT hallucinations are indeed a bit different. In Perplexity however I experienced very similar outputs as seen on the screenshot of AI Overview. Good RAG on such a scale is hard and not a solved problem yet.
Henry Shevlin: Yes indeed! RAG is temperamental, and I’ve had RAG-related fails in ChatGPT. But weird that Google would lean on RAG for this task. With million-token context windows even in public Gemini Pro, why not just do direct inference on cached copies of the top few Pageranked results?
Mike Riverso: There’s a fun chain of events here that goes: SEO destroys search usability -> people add “Reddit” to search queries to get human results -> Google prioritizes Reddit in AI training data and summaries -> AI spits out Reddit shitposts as real answers.
Proving yet again that LLMs don’t understand anything at all. Where a human can sift through Reddit results and tell what is real and what’s a joke, the AI just blindly spits out whatever the popular result was on Reddit because it doesn’t know any better.
Vijay Chidambaram: There is a good outcome from the Google AI overview being deployed and live. There is no better education for the public than to see with their own eyes how AI is fallible. We can give talks, write articles, but nothing compares with Google asking you to eat non-toxic glue.
The ‘non-toxic’ modifier on the glue is not going to stop being funny.
Mark Riedl: It’s weird that Google gets raked over the coals, when OpenAI often gets a pass for the same phenomenon. I’m not sure why. Because Google is a trusted source? Because fewer people use Bing or GPT4 with retrieval? Or is Gemini that much more prone to hallucinations?
As I put it then:
In this case, it is largely justified. I do not remember ChatGPT going this stupid. There is a difference between questions designed to trick LLMs into looking foolish, and ordinary if a little absurd search queries.
Also this is Google Search. I do think a higher standard is appropriate here than if these results were showing up on Gemini, the audience is less sophisticated.
Colin Fraser: I can’t believe Google pulled the plug immediately and issued a sheepish apology for the Asian founding fathers but have let this go on for a week. Doesn’t bode well for their decision making priorities in my opinion.
I think this perhaps speaks badly to the priorities of our society, that we were outraged by hot button violations and mostly are amused by random copying of trolling Reddit answers. I notice that the answers quoted are wrong and often very funny and absurd, and if you believed them for real it would not go well, but are almost never offensive or racist, and the ones that seemed truly beyond the pale (like suggesting jumping off a bridge was a good idea) turned out to be fake.
Information has an error rate. Yes, the rate on AI overview was much higher than we would like, but it was clearly labeled and I don’t think ‘we can find tons of absurd examples’ tells you about whether it is high enough that you need to pull the plug.
Also the results aren’t showing up on Gemini? You only see this on the AI overview, not on the Gemini page.
That goes back to the Reddit issue, and the tie-in with Google search. It is the combination of doing a search, together with using AI to select from that, and the need to produce an almost instantaneous answer, that is causing this disaster.
If Google were willing to run the query through Gemini Pro, and ask it ‘does this answer seem reasonable to you?’ we wouldn’t be having this conversation. It is not as if we do not have solutions to this. What we don’t have solutions to is how to do this instantly. But I have to wonder, Gemini Flash is damn good, why isn’t it good enough to stop this?
My plan was to test for how frequent the problem is by using GPT-4o to generate random absurd questions (such as “Can I replace my daily water intake with pure maple syrup?” and “Can I grow a money tree by planting a dollar bill in my backyard?) but they reliably failed to generate AI overviews for me, so no data. Also no AI overviews, which is fine with me in their current state.
Caroline Orr Bueno says obviously Google should pull the offering and not doing so is deeply irresponsible, links to The Byte’s Sharon Adarlo saying Google’s CEO admits he has no solution for the incorrect information, because ‘hallucinations are an unsolved problem.’ These are related but distinct things. The goal has to be to get the effective error rate down to acceptable levels, weighted by the places it matters. It is not as if a regular Google search is fully reliable, same as any other website.
You can also go to udm14.com as an easy way to use the text-only version of search.
A simpler solution is suggested by Arvind Narayanan, which is to use humans to do manual fixes. The long tail will remain but you can presumably hit most queries that way without it crimping Google’s budget that hard.
There is that. There is also doing it a hybrid form of ‘manually’ via AI. Gemini is perfectly capable of noticing that you do not want to add glue to your pizza or that Applum is not a fruit. So it seems relatively easy and cheap to take every query that is made in identical (or functionally identical) format N or more times, and then check to see where the AI overview answer is from bonkers to clearly correct and fix accordingly. You would still be able to generate absurd answers by being creative and finding a new query, but ordinary users would very rarely run into an issue.
What won’t help is blind panic. I saw this warning (the account got taken private so links won’t work).
Scott Jenson: I just left Google last month. The “Al Projects” I was working on were poorly motivated and driven by this mindless panic that as long as it had “AI” in it, it would be great. This myopia is NOT something driven by a user need. It is a stone cold panic that they are getting left behind.
The vision is that there will be a Tony Stark like Jarvis assistant in your phone that locks you into their ecosystem so hard that you’ll never leave. That vision is pure catnip. The fear is that they can’t afford to let someone else get there first.
This exact thing happened 13 years ago with Google+ (I was there for that fiasco as well). That was a similar hysterical reaction but to Facebook.
David Gerard: dunno how to verify any of this, but xooglers who were there for G+ say it absolutely rings true.
Google+ failed. In that sense it was a fiasco, costing money and time and hurting brand equity. Certainly not their finest hour.
What Google+ was not was a hysterical reaction, or a terrible idea.
Meta is a super valuable company, with deep control over a highly profitable advertising network, and a treasure trove of customer data and relationships. They have super powerful network effects. They play a core role in shaping our culture and the internet. Their market cap rivals that of Google, despite Zuckerberg’s best efforts.
They also are using those profits partly to lobby the United States Government to defeat any and all regulations on AI, and are arguably are on what is de facto a generalized crusade to ensure everyone on Earth dies.
Google spent a few billion dollars trying to compete with what is now a trillion dollar business that has huge synergies with the rest of Google’s portfolio. If Google+ had succeeded at becoming a peer for Facebook, it seems reasonable to assign that a value of something on the order of $500 billion.
The break-even success rate here was on the order of 2%. The fact that it did not work, and did not come so close to working, is not strong evidence of a mistake. Yes, the effort was in some ways uninspired and poorly executed, but it is easy for us to miss all the things they did well.
Think of AI as a similar situation. Is Google going to create Jarvis? They seem like at worst the second most likely company to do so. Is the (non-transformational, Google still exists and is owned and run by humans) future going to involve heavy use of a Jarvis or Her, that is going to have a lot of lock-in for customers and heavily promote the rest of the related ecosystems? That seems more likely than not. You have to skate where the consumer need and habit pucks are going, and you need to bet big on potential huge wins.
There are lots of places where one could slap on the word ‘AI’ or try to integrate AI and it would not make a lot of sense, nor would it have much of an upside. Nothing I saw that Google I/O was remotely like that. Every product and offering made sense.
That in no way precludes Google’s internal logic and decision making and resource allocation being a giant cluster. Google could be running around in chicken-sans-head fashion shouting ‘AI’ everywhere. But that also could be a rather strong second-best strategy.
Tantacrul: I’m legit shockedby the design of Meta’s new notification informing us they want to use the content we post to train their AI models. It’s intentionally designed to be highly awkward in order to minimize the number of users who will object to it. Let me break it down.
I should start by mentioning that I’ve worked in growth teams who conduct experiments to minimise friction for over a decade and I know how to streamline an experience. Rule: every additional step you add dramatically decrease the % of people who’ll make it through to the end.
First step: you get this notification, just about satisfying the legal requirement to keep you informed but avoiding clearly defining its true purpose. Should include the line ‘We intend to use your content to train our AI models’ and should include a CTA that says ‘Opt Out’.
Second step. It shows you this notice. Trick: places the ‘right to object’ CTA towards the end of the second paragraph, using tiny hyperlink text, rather than a proper button style. Notice the massive ‘Close’ CTA at the bottom, where there’s clearly room for two. Ugly stuff.
Also, notice the line that says “IF your objection is honoured, it will be applied going forwards.”
Wow. “If”. Don’t see that too often. Legal safeguards aren’t in place yet to protect us against AI training so they’re pushing as far as possible, while they still can.
Third, they provide you with a form to fill out. It is only at this stage — the stage when you are objecting — that they inform you about which of your content they plan to use for training AI models. Notice the highlighted text, clarifying that they may ignore your objection.
Fourth step: you post your objection.
Fifth step: now you are told you need to check your email to grab a code they sent you.
I’d LOVE to hear their justification for this.
Sixth step: you open the email they send (which for me, arrived on time at least).
Notice the code is only valid for an hour. Now copy the code.
Seventh step: enter the code and get a confirmation message.
I later received an email letting me know that they would honour my objection.
I should mention that one of my friends who also objected got an error! I then checked out a Reddit thread which verified that many people also got this same error. Classic FB sloppiness.
I’m not (all that) surprised up to this point. I’m not mad.
So far I’m just impressed. That right there is some top shelf dark patterning.
And then it… gets worse?
You see, when they say ‘if’ they mean ‘if.’
Darren M. A. Calvert: This new Facebook/Instagram policy for claiming they can use anything you post to power their A.I. is ridiculous.
The only way to opt out is apparently to fill out a form and submit “proof” that your data has *ALREADYbeen used to power A.I. 😑
Also, even if you do jump through all of these hoops *ANDthey approve your request, someone else reposting your work means that it gets fed to the algorithm anyway.
There are so many infuriating things about this technology but one of them is that you’re going to see less art online going forward. It’s getting to the point where the benefit of sharing your work isn’t worth shooting yourself in the foot by feeding A.I. image generators.
Also, this Facebook/Instagram policy doesn’t just affect artists. If you don’t want photos of yourself and friends/family being fed into image generators, too bad apparently.
Did you write a heartfelt eulogy to a deceased friend or relative? Meta owns that now.
Jon Lam: Lot of us are getting our requests to opt out denied. It’s complete bullshit.
Facebook’s email to Jon Lam: Hi,
Thank you for contacting us.
Based on the information that you have provided to us, we are unable to identify any examples of your personal information in a response from one of Meta’s generative Al models. As a result, we cannot take further action on your request.
If you want to learn more about generative AI, and our privacy work in this new space, please review the information we have in the Privacy Center.
Darren M. A. Calvert: They can’t identify any examples so they’re going to make it happen.
Neigh-Martin: I sent an objection just stating “I don’t consent to my posts being used for your plagiarism machine” and it was approved in about five minutes. The reposters loophole is the fatal flaw though.
Darren: I’m starting to get the impression that at least part of the approval process has to do with what country you live in and what Meta thinks they can get away with.
All right, fine. I’m surprised now. Using dark patterns to discourage opt-outs, and using reposts and fan pages and so on as excused? I expected that.
Actively refusing an artist’s opt-out request is something else.
Seth Burn: This sounds pretty bad, even by modern FB standards.
The question, as always, is if we object, what are we going to do about it?
Search clickthroughs will plummet, ads will be sold on generated answers, and media licensing fees for AI models can’t sustain enough new journalism to fuel the tech companies’ own products.
So where is the content going to come from? Only YouTube has really accepted that ad revenue has to be shared with creators, otherwise your platform is going to gradually peak and die. And now generative AI threatens to replace a lot of human authorship anyway.
If AI search and generative tools don’t create incentives for the “production of new content” online, to put it grossly, then it’s not going to happen and what we’re faced with is circling the toilet of AI trained on itself.
You could say “everything should be like Reddit” with people just posting about their own expert passions but only tech bros living on startup equity and extractive Silicon Valley wealth think that’s sustainable.
This is a tragedy of the commons model. As Kyle says later, it would work if the AI companies paid enough for data to sustain information generation, but that requires deals with each source of generation, and for the payments to be large enough.
This is part of The Big Rule Adjustment. Our norms rely on assumptions that will cease to hold. All you can eat can be a great promotion until people start figuring out how to eat quite a lot more and ruin it for everyone. Doing the information extraction and regurgitation trick is good and necessary and fair use at human scale, and at Google search scale, but go hard enough on the AI scale, taking away traditional compensation schemes (and not only the money), and the result is transformational of the incentives and results.
The natural solution is if deals are made like the ones OpenAI made with Newscorp and Reddit last week, or individual creators get compensation like on YouTube, or some combination thereof. If different AI companies compete for your data, especially your real time data, or a monopoly can internalize the benefits and therefore pay the costs, you can be fine without intervention.
Nor do we always ‘need a plan’ for how markets solve such problems. As long as we are dealing with ‘mere tools’ it takes a lot to keep such systems down and we should be skeptical things will fail so badly.
The light touch correction is the most promising, and the most obvious. Either you need to make a deal with the owner of the data to use it in training, or you need to pay a fixed licensing fee like in radio, and that is actually enforced. A plausible endgame is that there are various information brokerage services for individuals and small firms, that will market and sell your content as training data in exchange for a share of the revenue, and work to filter what you do and don’t want to share.
The problems also seem self-correcting. If the AI information degrades sufficiently, and they can’t work their way around that, then people will stop using the AIs in the impacted ways.
There is indeed the pattern, known as ‘the enshittification cycle,’ of ‘company builds platform with lock-in effects, customers get habits, company gradually makes it worse to raise revenue.’
That cycle is real, but wise platforms like YouTube stabilize at a reasonable balance, and eventually they all either pull back from the brink or get replaced by the new hotness, or both.
Here, it seems obvious that the central problem of Google search is not that Google is getting overly greedy (even if it is), but instead the arms race with SEO, which is now an arms race with increasingly AI-powered SEO.
Kelsey Piper: I do think an important thing about Google search is that they’re in an arms race with people who are trying to push their preferred content to the top of the first page, and these days the people doing that are using AI to manufacture the stuff they’re pushing.
“Why can’t we have old Google search back” is because Google search has always been an arms race between Google trying to put good stuff on the front page and everyone on the internet trying to put their stuff on the front page.
Right now Google definitely seems to be losing the battle, and that’s bad. But there isn’t some world where they just did nothing and search stays good; their adversaries weren’t doing nothing.
There is little doubt Google has lost ground and is losing ground right now, on top of any changes they made to enhance revenue. They are in a tough spot. They have to ‘play defense’ on everything all the time. They need to do so in a way customized to the user and context, in a way that is instantaneous and free and thus uses little compute per query.
I do predict the pendulum will swing back. As the models improve and they get more experience, the defense should be favored. There is enough ‘old internet’ data, and ways to generate new bespoke or whitelisted data, to bootstrap initial AIs that can differentiate even with a lot of noise. They’ll figure out how to better precalculate and cache those results. If they can’t, I think that will be on them.
We’ve been over similar ground before, but: There are various classic examples of ‘technology created more jobs.’ One of them is ATMs leading to more bank tellers by increasing demand for banking services.
Aaron Levie: Bank teller employment continuing to grow during the rise of ATMs is a perfect example of how automation lowers the cost of delivering a particular task, letting you serve more customers, and thus growing the category. We are going to see this over and over again with AI.
Yes, teller employment went up, but the population was expanding but the population increased from about 223 million to 310 million from 1980 to 2010. The number of tellers per capita went down, not up.
Also, while ATMs certainly contributed to people using banks more, the population got a lot richer and things got more financialized over that period. The baseline scenario would presumably have seen a substantial rise in per capita bank tellers.
Matt Yglesias: What happened after 2010?
Jon: Yeah not showing what happened after peak atm installs is extremely disingenuous given the commentary.
Sheel Mohnot: Went down bc of mobile banking, which eliminated the branches. So ultimately tech came for them.
The general form is that in many cases AI and other technology starts off growing the category while decreasing labor intensity, which can go either way for employment but makes us richer overall. Then the automation gets good enough, and the category demand sufficiently saturates, and it is definitely bad for sector employment. With AI both phases will typically happen a lot faster.
Then the question is, does AI also take away the jobs those humans would have then shifted to in other sectors?
My answer is that at first, in the short run, AI will be bad for a few sectors but be very good for overall employment. Then if capabilities keep advancing we will reach a turning point, and by default AI starts being quite bad for employment, because AI starts doing all the newly demanded jobs as well.
If someone keeps warning ‘even mundane AI will take all our jobs and we won’t have new ones’ without any conditions on that, then they are failing to notice the pattern of technology throughout history, and the way economics works and the giant amounts of latent demand for additional services and goods if we get wealthier.
If someone keeps repeating the mantra ‘AI will mean more jobs because technology always means more jobs,’ and essentially treats anyone who expects anything else as an idiot who doesn’t know that farmers ended up with other jobs, they are treating a past trend like a law of nature, and doing so out of its distribution, with a very different type of technology, even if we restrict ourselves to mundane AI.
I notice if anything an anti-correlation between where I expect AI to take people’s jobs, and where people expect it to happen to them.
Also these are very high rates of expecting to lose jobs within ten years. 54% said at least probably yes, 48% in America.
This graph is also interesting, including outside of AI:
There’s something to the Indian attitude here. Jobs are easy come, easy go.
Hasbro tells makers of My Little Pony: Make Your Mark that AI, rather than friendship, is magic, and they want to use AI voices for season 2. Producer Cort Lane took a hard stance against the use of AI, choosing to shut the entire series down instead. This comes on the heels of the foreign language voices in My Little Pony: Tell Your Tale being AI generated.
If executed well, that sounds great. A valuable community service. The obvious issue is that this requires trust in those doing the evaluations, and potentially vulnerable to idiosyncratic decisions or preferences.
I especially appreciate their warning that a model can only be evaluated once, when an organization first encounters the prompts, to preserve test integrity, although I wonder what we do when the next generation of model comes out?
One big worry is conflicts of interest.
Anton: Good benchmarks are important but i find it difficult to trust results reported by a company whose primary customers are the producers of the models under evaluation. the incentives go against objectivity.
I can’t imagine a company spending millions on scale labeling to not move the needle on these evals. Perverse incentives.
I can imagine it not mattering, although of course I can also imagine it mattering. This is a longstanding problem, see for example mortgage bonds. There are clear examples of corruption in similar situations for almost no gain, and also clear examples of integrity despite great temptations.
How reliable is Scale.ai here? My presumption is reliable enough for these to be a useful additional source, but not enough to be heavily load bearing until we get a longer track record. The most trustworthy part is the relative strengths of different models across different areas.
One thing that helps is sanity checking the results. If the methodology is severely flawed or unreasonable, it should be obvious. That doesn’t cover more subtle things as robustly, but you can learn a lot.
Another issue is lack of clarity on what the numbers represent. With Elo ratings, you know what a 30 point gap means. Here you do not. Also we do not get the fuller range of models tested, which makes calibration a bit harder.
Swift on Security: Hell yeah gonna put myself into a sexy schoolgirl outfit thanks Instagram it’s definitely my face I’m uploading.
Literally a schoolgirl nudifying undress webapp advertised by and usable in Instagram’s browser. I uploaded their own ad image and although it’s blurred seems like it works to some extent. They can detect words like “erase” “clothing” they just don’t care.
It’s literally endless I have hundreds of these screenshots since I opted-in to these categories and always interact with the AI ads.
PoliMath: I don’t know how to slow this down or stop this but my gut instinct is that we really need to slow this down or stop this.
I’m becoming less interested in how to do so politely.
We are less than 2 years into this being a thing.
The consequences of this (especially for young people) are unknown and may be quite severe.
If you were wondering if there’s any fig leaf at all, no, there really isn’t.
I get why it is impossible to stop people from going to websites to download these tools. I do not get why it is so hard to stop ads for them from appearing on Instagram. We are not exactly up against the best and brightest in evading filters.
Ultimately you end up in the same place. Any unrestricted device will be able to use fully unlocked versions of such apps without technical expertise. They will make it easy, and the pictures will get harder to distinguish from real and stop all looking suspiciously like the same woman in the same pose if you think about it.
This is the trilemma. Lock down the model, lock down the device, let people do what they want in private and filter your platform.
You do at least have to do the last one, guys. Jesus.
$20k in prizes will go to creators of AI-generated models.
They must not only submit photos, but answer the traditional pageant questions like “how would you make the world a better place?”
Note that the prizes are partly fake, although there is some cold hard cash.
Alas, entries are long since closed, no one told me until now.
Timothy Lee asks, what exactly would it be illegal to do with Scarlett Johansson’s voice, or anyone else’s? Technically, where is the law against even an actual deepfake? It is all essentially only the right of publicity, and that is a hell of a legal mess, and technically it might somehow not matter whether Sky is a deepfake or not. The laws are only now coming, and Tennessee’s Elvis act clearly does prohibit basically all unauthorized use of voices. As Timothy notes, all the prior cases won by celebrities required clear intent by the infringer, including the video game examples. He expects companies to pay celebrities for their voices, even if not technically required to do so.
What I do know is that there is clear public consensus, and consensus among politicians, that using a clear copy of someone else’s voice for commercial purposes without permission is heinous and unacceptable. Where exactly people draw the line and what the law should ultimately say is unclear, but there is going to be a rule and it is going to be rather ironclad at least on commercial use. Even for personal non-sexy use, aside from fair use or other special cases, people are mostly not okay with voice cloning.
(As a reminder: Some think that Sky being based on a different woman’s natural voice is a get-out-of-lawsuit-free card for OpenAI. I don’t, because I think intent can lie elsewhere, and you can get damn close without the need to give the game away but also they then gave the game away.)
They also are collaborating with WAN-IFRA on a global accelerator program to assist over 100 news publishers in exploring and integrating AI in their newsrooms.
This comes on the heels of last week’s deal with Newscorp.
OpenAI’s plan seems clear. Strike a deal with the major media organizations one by one, forcing the stragglers to follow suit. Pay them a combination of money and access to AI technology. In exchange you get their training data free and clear, and can use their information in real time in exchange for providing links that the users find helpful. Good plan.
Yelix: maybe it’s because i’m a normal person who doesn’t have terminal CEO Brain but i just can’t fathom why anyone who runs a media org would align with OpenAI.
This is not even close to an equal exchange to a person with reasonable values. Vox is giving up a couple decades’ worth of (overworked, underpaid, most likely laid off years ago) human labor so they can do targeted ad sales.
I guess when you have an opportunity to partner with quite possibly the least credible person in tech, Sam Altman, you just gotta do it.
Seth Burn: Presumably, it’s because OpenAI is providing money for content, which might be hard to come by these days.
Yelix has a point, though. This is the equivalent of selling your seed corn.
Some people noticed. They were not happy. Nor had they been consulted.
Text of Announcement: Today, members of the Vox Media Union, Thrillist Union, and The Dodo Union were informed without warning that Vox Media entered into a “strategic content and product partnership” with OpenAI. As both journalists and workers, we have serious concerns about this partnership, which we believe could adversely impact members of our union, not to mention the well-documented ethical and environmental concerns surrounding the use of generative AI.
We demand that Vox Media engage with us on this issue transparently — and address our many unanswered questions about this partnership — instead of continuing to fail to include our voices in decisions like these. We know that AI is already having a monumental impact on our work, and we demand a seat at the table in discussions about its future at Vox Media.
Seth Burn: Former Cowboys president Tex Schramm to former NFLPA union chief Gene Upshaw, “You guys are cattle and we’re the ranchers, and ranchers can always get more cattle.”
Tex never dreamed of AI cattle though.
Kelsey Piper (Vox): I’m very frustrated they announced this without consulting their writers, but I have very strong assurances in writing from our editor in chief that they want more coverage like the last two weeks and will never interfere in it. If that’s false I’ll quit.
Kelsey Piper will, once again, be the test. If the reassurances prove hollow, I presume she will let us know. At that point, there would be no question who OpenAI is.
I do not see Google (or Anthropic or anyone else) competing with them on this so far. One possibility is that Google can’t offer to pay because then the companies would demand payment for Google search.
Jan Leike lands at Anthropic, where he will continue the work on scalable oversight, weak-to-strong generalization and automated alignment research. If your talents are not appreciated or supported, you take your talents elsewhere.
Karina Nguyen moves from Anthropic to OpenAI after two years, offers lessons learned. As is usually the case such lists offer insights that are most interesting for which ones are emphasized and which are left out. It does not provide any insight on why she made the move.
A thread from Microsoft’s event last week, clarifying their stance. CTO Kevin Scott indeed claims that we are nowhere near diminishing marginal returns to magnitude of compute, but that is not the business Microsoft is ultimately running, or thinks is important. The frontier models are of minor value versus models-as-a-service, an array of different cheaper, smaller and faster models for various situations, for which there is almost limitless demand.
This creates an odd almost bimodal situation. If you go big, you need something good enough to do what small cannot do, in a way that beats humans. Otherwise, you go small. But going big is expensive, so the question is, can you make it all worth it? Where ‘actually replacing people’ is one way to do that.
Epoch also gives us a thread, paperand blog post on various case studies for ‘return to research effort,’ meaning how much efficiency gain you get when you double your R&D costs. Do you get critical mass that could enable recursive self-improvement (RSI) via explosive tech growth? Chess engine Stockfish comes out at ~0.83, just below the critical 1.0 threshold. The others seem higher.
Software returns, the returns that most matter, look high, much higher than the economy overall, where Bloom (2020) found r ~ 0.32 and Epoch AI found r ~ 0.25. It makes sense this number should be higher, but I have no good intuition on how much higher, and it seems odd to model it as one number. My presumption is there is some capabilities level where you would indeed see a foom if you got there, but that does not tell us if we are getting there any time soon. It also does not tell us how far you could get without running into various physical bottlenecks, or what else happens during that critical period.
Sam Altman signs the Giving Pledge, to give half or more of his wealth to philanthropy. He says he intends to focus on supporting technology that helps create abundance for people, together with Oliver Mulherin. Jessica and Hemant Taneja also signed today, also intending to focus on technology. It is an unreservedly great thing, but what will matter is the follow through, here and elsewhere.
This committee will be responsible for making recommendations to the full Board on critical safety and security decisions for OpenAI projects and operations.
OpenAI has recently begun training its next frontier model and we anticipate the resulting systems to bring us to the next level of capabilities on our path to AGI. While we are proud to build and release models that are industry-leading on both capabilities and safety, we welcome a robust debate at this important moment.
A first task of the Safety and Security Committee will be to evaluate and further develop OpenAI’s processes and safeguards over the next 90 days. At the conclusion of the 90 days, the Safety and Security Committee will share their recommendations with the full Board. Following the full Board’s review, OpenAI will publicly share an update on adopted recommendations in a manner that is consistent with safety and security.
OpenAI technical and policy experts Aleksander Madry (Head of Preparedness), Lilian Weng (Head of Safety Systems), John Schulman (Head of Alignment Science), Matt Knight (Head of Security), and Jakub Pachocki (Chief Scientist) will also be on the committee.
Additionally, OpenAI will retain and consult with other safety, security, and technical experts to support this work, including former cybersecurity officials, Rob Joyce, who advises OpenAI on security, and John Carlin.
It is good to see OpenAI taking the safeguarding of GPT-5 seriously, especially after Jan Leike’s warning that they were not ready for this. It is no substitute for Superalignment, but it is necessary, and a very good ‘least you can do’ test. We will presumably check back in 90 days, which would be the end of August.
Given the decision to advance the state of the art at all, OpenAI did a reasonably good if imperfect job testing GPT-4. Their preparedness framework is a solid beginning, if they adhere to its spirit and revise it over time to address its shortcomings.
Roon: Models will obviously be superintelligent in some domains long before they’re human level in others or meet the criteria of replacing most economically valuable labor.
The question of building ASI and AGI are not independent goals. Moreover anyone who finds themselves in possession of a model that does ML research better than themselves isn’t likely to stop.
The timelines are now so short that public prediction feels like leaking rather than sci-fi speculation.
The first statement is obviously true and has already happened.
The second statement is obviously true as stated, they are unlikely to stop on their own. What is not clear is whether we will reach that point. If you agree it is plausible we reach that point, then what if anything do you propose to do about this?
The third statement I believe is true as a matter is true in terms of the felt experience of many working at the labs. That does not mean their timelines will be realized, but it seems sensible to have a plan for that scenario.
This is somewhat complicated by the overloading and goalpost shifting and lack of clear definition of AGI.
Roon: I just love to see people confidently claim that LLMs will never do things that they can currently do.
Fernando Coelho: Do you refer to those available publicly or those still in closed training?
Roon: Both.
Whereas here are some future visions that don’t realize AI is a thing, not really:
Timothy Lee: I really wish there were more economists involved in discussions of the implications of superintelligence. There is so much sloppy thinking from smart people who have clearly never tried to think systematically about general equilibrium models.
The most obvious example is people predicting mass unemployment without thinking through the impact of high productivity on fiscal and monetary policy. There are also people who implicitly assume that the economy will become 90 percent data centers, which doesn’t make much sense.
I consider this to be very much ‘burying the lede’ on superintelligence, the continued assumption that somehow we still get ‘economic normal’ in a world with such things in it. I have ‘solved for the equilibrium’ in such cases. We do not seem involved. What would be the other equilibrium?
Saying ‘you forgot to take into account impact on fiscal and monetary policy’ is a good objection, but ignores the much more important things also being ignored there.
If you constrain your thinking short of superintelligence or transformational AI, then such considerations become far more important, and I agree that there is a deficit of good economic thinking.
The problem is that the ones letting us down the most here are the economists.
This issue goes far beyond dismissing existential risk or loss of control or anything like that. When economists model AI, they seem to come back with completely nonsensical projections that essentially say AI does not matter. They measure increased productivity or GDP in individual percentage points over a decade. Even if we assume all the bottlenecks stay in place and we have full economic normal and no loss of control issues and progress in capabilities stalls at GPT-5 (hell, even at current levels) the projections make no sense.
The economists have essentially left, or rather declined to enter, the building.
Rob Henderson: Damn. [Shows statistic that number of Americans who think of themselves as patriotic has declined from 70% in 1998 to 38% in 2024.]
Robin Hanson: More crazy fast cultural value change. No way we can have much confidence such changes are adaptive. Why aren’t you all terrified by this out of control change?
Kaj Sotala: I’m a bit surprised to see you concerned about changes in human values, when my impression was that you were mostly unconcerned about possible value shifts brought about by AGI. I would assume the latter to be much bigger than the former.
Robin Hanson: I don’t assume AI changes are much bigger, though digital minds of all sorts likely induce faster changes. And I’m not unconcerned; I’ve mainly tried to say AI isn’t the problem, there are more fundamental problems.
While I too am concerned by some of our existing highly rapid cultural changes, especially related to the drop in fertility, I really do not know what to say to that. Something about ‘we are not the same?’
Google is trying to be the Apple of AI, fully integrated on all levels. If Google can still build great products, ideally both software and hardware, they will win.
Amazon’s AWS is betting everything is modular.
Microsoft is in the middle, optimizing its infrastructure around OpenAI (while also trying to get its own alternatives off the ground, which I am skeptical about but could eventually work).
Nvidia keeps working on its chips and has nothing to fear but true vertical integration like we see at Google, or technically competitors but not really. The other potential threat, which Ben does not mention, is alternative architectures or training systems potentially proving superior to what GPUs can offer, but the market seems skeptical of that. It has been good to be Nvidia.
Meta is all-in on products and using Llama to serve them cheaply, so for now they benefit from optimization and thus open source.
The last section, on ‘AI and AGI,’ seems like Thompson not understanding how AI development works and scales. No, maximizing ‘every efficiency and optimization’ is unlikely to be the key to getting something approaching AGI, unless those gains are order of magnitude gains. Execution and actually getting it done matter a lot more. Google has big advantages, and data access, services integration and TPUs are among them. Even with his view Thompson is skeptical Google can get much model differentiation.
My hunch is that even more than the rest of it, this part comes from Thompson not feeling the AGI, and assuming this is all normal tools, which makes all of it make a lot more sense and seem a lot more important. Notice he doesn’t care that Anthropic exists, because from his perspective models do not matter so much, business models matter.
Arnold Kling says an AI Windows PC is a contradiction, because if it was AI you wouldn’t use a mouse and keyboard, AI is centrally about the human-computer interface. I think this is very wrong even on the pure UI level, and Arnold’s example of writing makes that clear. Short of a brain-computer interface where I can think the words instead of type them, what other interface am I going to use to write? Why would I want to use voice and gesture? Sure, if you want to go hands free or mobile you might talk to your phone or computer, but typing is just better than speaking, and a mouse is more precise than a gesture, and AI won’t change that.
What the AI UI does is let you bypass the rest of the interface, and automate a bunch of knowledge and memory and menus and capabilities and so on. The Copilot+ promise is that it remembers everything you ever did, knows how everything works, can help figure things out for you, code for you and so on. Great, if you can do that without privacy or security nightmares, good luck with that part. But why would I want to give up my keyboard?
This goes, to me, even for VR/AR. When I tried the Apple Vision Pro, the killer lack-of-an-app was essentially an air keyboard. As in, I had no good way to type. With good enough cameras, I wanted to literally type in the air, and have it figure out what I was trying to do, although I am open to alternatives.
Also of course I see AI has mostly doing something unrelated to all of that, this is a sideshow or particular use case.
It is always fun to contrast the economists saying ‘it might raise GDP a few percent over ten years’ versus people who take the question seriously and say things like this:
Matt Clifford: I’m actually very bullish on the UK’s mid-term future:
AI: one of the best places in the world to build AI companies + high state capacity in AI relative to peers
Science: great uni base, plus bold bets like ARIA.
Talent: still attracts large number of very high quality people thanks to unis, the City, DeepMind, a vibrant startup ecosystem, etc
High quality institutions / fundamentals
I am less bullish until I see them building houses, but yes the AI thing is a big deal.
John Arnold: Semiconductor manufacturing subsidies announced in the past 2 years:
US: $52 bln
India: $10 bln
Japan: $25 bln
EU: $46 bln
S Korea: $19 bln
UK: $1 bln
China: $47 bln
I think we know how this is going to turn out.
Robin Hanson: Yes, we will soon see a glut, with prices too low for profits.
Davidad: Noted economist and foom-skeptic robin hanson also anticipates an imminent era of GPUs too cheap to meter.
I completely disagree. Demand for compute will be very high even if capabilities do not advance. We are going to want these chips actual everywhere. These investments will not be so efficient, and are not so large considering what is coming, have you seen the market caps of Nvidia and TSMC?
Robin Hanson (February 6, 2024, talking about Nvidia at $682): Buy low, sell high. So, SELL.
I am happy to report I bet against that prediction. As I write this, it is at $1,116.
Visions of a potential future. I don’t see the story as realistic, but it is an admirable amount of non-obvious concreteness.
Davidad: Consider me fully on board the “LLM-Modulo” bandwagon. As long as one or more of the critics is a sound verifier (which indeed seems to be the authors’ intention), this is a Guaranteed Safe AI pattern. Though I would say “Version Control System” instead of “Blackboard”.
I continue to not see why this would be expected to work, but wish him luck and am happy that he is trying.
John Luttig notices that the future of AI cannot be dominated by open source and also be dominated by closed source, despite both claims being common. So who is right?
He notes that right now both coexist. At the high end of capabilities, especially the largest frontier models, closed source dominates. But for many purposes people value open weights and the flexibility they provide, and hosting yourself saves money too, so they are fine with smaller and less efficient but private and customizable open models.
He also offers this very good sentence:
John Luttig: Meanwhile, an unusual open-source alliance has formed among developers who want handouts, academics who embrace publishing culture, libertarians who fear centralized speech control and regulatory capture, Elon who doesn’t want his nemesis to win AI, and Zuck who doesn’t want to be beholden to yet another tech platform.
I very much appreciate the clear ‘baptists and bootleggers’ framing on open weights side, to go with their constant accusations of the same. As he points out, if Meta gets competitive on frontier models then Zuck is going to leave this coalition at some point when the economics of Llama and therefore his incentives change, and Elon’s position is I am guessing unstrategic and not so strongly held either.
Thus Luttig’s core logic, which is that as costs scale the open system’s economics fail and they switch strategies or drop out. Using open weights looks cheaper, but comes with various additional burdens and costs, especially if the model is at core less efficient, and thus you either get a worse model or a more compute-intensive one or both versus using closed.
I am not as convinced by his argument that free is reliably worse than paid as a pattern. Contrary to his claim, I would say Android is not worse than iOS, I am on Android because I think it is better, and I defy those who like Luttig claim a large quality gap the other way. OpenOffice is worse than Google Docs, but Google Docs is also free (albeit closed) and it is in practical terms better than the paid Microsoft Office, which is again why I don’t pay. Unity is an example of sufficiently obnoxious holdup issues I’d rather use an alternative even if Unity is technically better.
And those are only his examples. Linux is for servers typically considered better than anything paid, and with Copilot+ it is a reasonable question whether it is time for me to switch to Linux for my next machine. I might trust my local machine to have universal memory with Linux levels of security. With Microsoft levels, not so much.
Here is another very good sentence:
Advocates like Yann LeCun claim that open-sourced AI is safer than closed. It makes me wonder if he really believes in Meta’s AI capabilities. Any reasonable extrapolation of capabilities with more compute, data, and autonomous tool use is self-evidently dangerous.
This is the same week we get LeCun saying that there exist no general intelligences, not even humans. So perhaps it is not Meta’s AI he does not believe in, but AI in general. If we lived in a world in which GPT-5-level models were as good as it was ever going to get in my lifetime, I would be on the open source side too.
Appealing to American security may seem overwrought, but the past five years of geopolitics has confirmed that not everyone is on the same team. Every country outside America has an interest in undermining our closed-source model providers: Europe doesn’t want the US winning yet another big tech wave, China wants free model weights to train their own frontier models, rogue states want to use unfiltered and untraceable AI to fuel their militaristic and economic interests.
Again, very well said. I am impressed that Tyler Cowen was willing to link to this.
Ultimately, this was a very good post. I mostly agree with it. My biggest gripe is the title is perhaps overstated – as both he and I think, open weights models will continue to have a place in the ecosystem, for smaller systems where local control is valuable.
And to be clear, I think that is good. As long as that stays below critical thresholds that lie beyond GPT-4, and that can expand at least somewhat once the frontier is well beyond that, the dangers I worry about wouldn’t apply, so let my people cook (brb applying for copyright on that phrase since I’ve never heard that exact phrasing.)
Peter Thiel predicts AI will be ‘good for the verbal people, bad for the math people,’ notes within a few years AI will be able to solve all the Math olympiad problems. First we had the AI that was much better at math problems than verbal problems (as in, every computer before 2018) and that was very good for math people. Now we have AI that is much better at verbal and worse at math, but which can be used (because verbal is universal and can call the old computers for help) to make something better at math. He says why test people on math, that doesn’t make a good surgeon, he had a chess bias but that got undermined by the computers.
But I think no? The chess test is still good, and the math test is still good, because your ability to get those skills is indicative. So what if AlphaZero can beat Kasparov, Kasparov could beat Thiel and also you already and that didn’t matter either. Math-style skills, and software-related skills, will be needed to be able to make sense of the AI era even if you are not earning your living by doing the actual math or coding or chess mastering.
This is also a result of the ‘verbal vs. math’ distinction on various tests and in classes, which seems like a wrong question. You need a kind of symbolic, conceptual mastery of both more, and you need the basic skills themselves less thanks to your spellchecker and calculator and now your prover and you LLM. That doesn’t say much about which style of skill and advantage is more valuable. I do think there could be a window coming where the ‘physical manipulation’ skills have the edge over both, where it is the manual labor that gets the edge over both the math and verbal crowds, but I wouldn’t consider that a stable situation either.
The real argument for verbal over math in the AI era to me is completely distinct from Thiel’s. It is that if AI renders us so unneeded and uncompetitive that we no longer need any skills except to ‘be a human that interacts with other humans’ and play various social games, where the AI can’t play, and the AI is doing the rest, then the math people are out of luck. As in, math (in the fully general sense) is useful because it is useful, so if people are no longer useful but are somehow alive and their actions matter, then perhaps the math people lose out. Maybe. My guess is the math crowd actually has a lot of edge in adapting to that path faster and better.
Sarah Fortinsky (The Hill): Federal Trade Commission (FTC) Chair Lina Khan said Wednesday that companies that train their artificial intelligence (A) models on data from news websites, artists’ creations or people’s personal information could be in violation of antitrust laws.
I mean, sure, I can see problems you might have with that. But… antitrust? What?
It seems the FTC’s new theory is that is the new everything police, regardless of what the laws say, because anything that is ‘unfair’ falls under its purview.
“The FTC Act prohibits unfair methods of competition and unfair or deceptive acts or practices,” Khan said at the event. ”So, you can imagine, if somebody’s content or information is being scraped that they have produced, and then is being used in ways to compete with them and to dislodge them from the market and divert businesses, in some cases, that could be an unfair method of competition.”
‘Antitrust’ now apparently means ‘any action Lina Khan does not like.’
Lina Khan thinks your contract you negotiated is uncool? Right out, retraoactively.
Lina Khan thinks your prices are too high, too low or suspiciously neither? Oh no.
Lina Khan thinks you are training on data that isn’t yours? General in the meme is here to tell you, also antitrust.
We cannot have someone running around being the ‘this seems unfair to me’ cop. Once again, it feels like if someone runs over rule of law and imposes tons of arbitrary rules, the internet stops to ask if it might plausibly stop us from dying. If not, then they get a free pass. Can we at least be consistent?
Guardian has a write-up about big tech’s efforts to distract from existential risk concerns.
Max Tegmark: As I told the Guardian, the techniques big tech lobbyists are using to discredit the loss-of-control risk from future smarter-than-human AI have much in common with what big tobacco and big oil did. See the film “Merchants of Doubt”!
In ‘not AI but I feel your pain news’ this complaint about how none of the commentators on Biden’s climate policies are actually trying to understand what the policies are or what they are trying to accomplish, whether they support the policies or not. I am not taking any position on those policies whatsoever, except to say: Oh my do I feel your pain. As it is there, so it is here.
What about optimal taxation policy? Andrew Yang proposes a tax on cloud computing or GPUs to compensate for relatively high taxation of human workers, Kyle Russell says we already have taxes on profits, TS00X1 says imagine a steam engine or internal combustion engine tax and so on.
What these dismissals miss is that neutral taxation requires equalizing the tax burden between relevant alternatives. Suppose you can choose whether to pay an employee in San Francisco $100k to deal with customers, or buy cloud computing services and kiosk hardware and so on, and performance is similar.
In the first case, the human gets a take home pay of roughly $60k, at a total employee cost of $112k. In the second case, if you pay $112k, let’s say that average gross margin for the largest providers is 65%, and their tax rate is typically 21%. Even if you threw in California corporate tax (which I presume they aren’t paying) and sales tax, that’s still only $29k in taxes versus $52k. That’s a not a complete calculation, but it is good enough to see the tax burdens are not going to equalize.
This could easily result (and in practice sometimes does) in a situation where using computers is a tax arbitrage, and that takes it from uneconomical to economical.
I do not consider this that big a deal, because I expect the cost of compute and other AI services to drop rapidly over time. Let’s say (in theory, for simplicity) that the fully neutral tax rate of tax on compute was 40%, but the actual effective tax rate was 20%. In many other settings that would be a huge deal, but in AI it is all orders of magnitude. So this only speeds up efficient deployment by a few months.
The flip side is that this could be a highly efficient and productive tax. As always, we should look to shift the tax burden according to what we want to encourage and discourage, and when we are indifferent to ensure neutrality. I see a potentially strong economic argument for taxing compute and using that money to cut income taxes, but would want to see more research before drawing conclusions, and I would worry about competitiveness and tax jurisdiction issues. This is exactly the kind of place where a call to ‘model this’ is fully appropriate, and we should not jump to conclusions.
Bad ideas for regulations: California’s SB 1446 limiting self-service checkouts. I do think often retailers are making a business and also total welfare mistake by relying more than they should on self-service checkouts, as opposed to ordering kiosks which are mostly great. I actively avoid one local grocery store when I have a choice due to its checkout procedures. But that should be their mistake to make. The real argument for a bill like SB 1446 is that first they mandated all these extra costs of hiring the workers, so now they cost so much that the government needs to force employers to hire them.
Did we have sane regulations of future frontier models all along, in the form of existing tort law?
By default, for everything, we have the negligence standard. Everyone has a duty to take reasonable care to avoid causing harm, pretty much no matter what.
This certainly is helpful and much better than nothing. I do not see it remotely being enough. Ex post unpredictable assignment of blame, that only fires long after the harm happens and for which ‘reasonable care’ is an excuse?
While we have no industry standards worthy of the name and the damage could well be catastrophic or existential, or involve loss of control over the future, including loss of control to the AI company or to the AI? And also many damage scenarios might not involve a particular (intact) victim that could have proper standing and ability to sue for them? That won’t cut it here.
They also argue that the ‘abnormally dangerous activities’ standard we use for tigers might apply to frontier AI systems, where a presumption of ‘reasonable care’ is impossible, so any harm is on you. I still do not think ‘they can sue afterwards’ is a solution, it still seems like a category error, but this would certainly help, especially if we required insurance. Alas, they (I think correctly) find this unlikely to be applied by the courts on their own.
They then move on to ‘products liability.’ This is a patchwork of different rules by state, but it is plausible that many states will consider frontier AIs products, to which my attitude would be that they better damn well be products because consider the alternative things they might be. Lawfare’s attitude here seems to be a big ‘I don’t know when it would or wouldn’t apply what standard on what harms.’ There are advantages to that, a company like Google hates uncertainty. And it suggests that by ‘foreseeing’ various misuses or other failure modes of such AIs now, we are making the companies liable should they occur. But then again, maybe not.
The right way to ensure responsible development of frontier AI systems, a potentially transformational or existentially risky technology, cannot be ‘ex post if something bad happens we sue you and then we have no idea what the courts do, even if we still have courts.’
They seem to agree?
The main argument provided for relying on tort law is that we lack regulations or other alternatives.
They also suggest tort law is more adaptable, which is true if and only if you assume other laws mostly cannot be modified in response to new information, but also the adaptations have to be fast enough to be relevant and likely to be the ones that would work.
They suggest tort law is less vulnerable to regulatory capture, which is an advantage in what I call mundane ‘economic normal’ worlds.
They suggest that tort law is how you get regulatory compliance, or investment in safety beyond regulatory requirements. Here I agree. Tort liability is a strong complement. Certainly I have no interest in granting frontier AI companies immunity to tort liability.
They list as issues:
Tort law requires the right sort of causal chain to an injury. I strongly agree that this is going to be an issue with frontier AI systems. Any working definition is either going to miss a wide range of harms, or encompass things it shouldn’t.
Tort law has a problem with ‘very large harms from AI,’ which they classify as thousands of deaths. If that was the maximum downside I wouldn’t be so worried.
Tort law doesn’t work with certain types of societal harms, because there is no concrete damage to point towards. There’s no avoiding this one, even if the harms remain mundane. Either you accept what AI ‘wants to happen’ in various ways, or you do not, and tort law only stops that if it otherwise ends up a de facto ban.
Tort law might move too slowly. No kidding. Even if a case is brought today it likely does not see a verdict for years. At the current pace of AI, it is reasonable to say ‘so what if we might be liable years from now.’ By that time the world could be radically different, or the company vastly bigger or gone. If and when the stakes really are existential or transformational, tort law is irrelevant.
They warn of a winner’s curse situation, where the companies that think they are safest proceed rather than those that are safest. Or, I would say, the companies that have less to lose, or are more willing to gamble. A key problem with all safety efforts is that you worry that it can mean the least responsible people deploy first, and tort law seems to make this worse rather than better.
Tort law could hinder socially desirable innovation. The question is the price, how much hindering versus alternative methods. If we indeed hold firms liable for a wide variety of harms including indirect ones, while they do not capture that large a portion of gains, and tort law actually matters, this is a huge issue. If we don’t hold them liable for those harms, or tort law is too slow or ineffective so it is ignored, the tort law doesn’t do its job. My gut tells me that, because it focuses on exactly the harms that we could deal with later, a tort law approach is more anti-socially-desirable-innovation than well-constructed other regulatory plans, at the same level of effectiveness. But also you can do so, so much worse (see: EU).
The final concern is that judges and juries lack expertise on this, and oh boy would that be a huge problem in all directions. Verdicts here are going to be highly uncertain and based on things not that correlated with what we want.
I especially appreciate the note that regulatory rules moderate tort law liability. If you comply with regulatory requirements, that constitutes a partial defense against torts.
They conclude with a classic ‘more research is needed’ across the board, cautioning against giving AI companies liability shields. I certainly agree on both counts there. I especially appreciated the nod to liability insurance. Mandatory insurance helps a lot with the issue that torts are an extremely slow and uncertain ex post process.
The argument here that matters is simple – SB 1047 regulates code, and you can’t regulate code, and also neural network weights are also speech. And it says that it uses legal precedent to show that the Act is ‘an overreach that stifles innovation and expression in the AI field,’ although even if the Act were that I don’t know how precent could show that the act would do that – the potential stifling is a prediction of future impacts (that I disagree with but is not a crazy thing to claim especially without specifying magnitude of impact), not a legal finding.
Section one goes over the classic ‘algorithms are speech’ arguments. I am not a lawyer, but my interpretation is that the code for doing training is not restricted in any way under SB 1047 (whether or not that is wise) so this is not relevant. In all these cases, the argument was that you could distribute your software or book, not whether you could run it for a particular purpose. You can yell fire in a crowded theater, but you are not protected by the first amendment if you light the theater on fire, even if it is one hell of a statement.
Thus in my reading, the argument that matters is section two, the claim that the weights of a neutral network are speech, because it is a mathematical expression. If an inscrutable black box of numbers is speech, then given the nature of computers, and arguably of the universe, what is not speech? Is a person speech by their very existence? Is there any capability that would not be speech, in any context?
The whole line seems absurd to me, as I’ve said before.
And I think this line kind of shows the hand being played?
While it is important to ensure the safe and ethical use of AI, regulatory measures must be carefully balanced to avoid infringing upon free speech rights.
SB-1047’s provisions, which mandate safety determinations and compliance with safety standards, could be seen as imposing undue restrictions on the development and dissemination of neural network weights.
Wait, what? Which is it? Saying you have to meet safety standards sounds like we should be talking price, yet I do not see talk here of price afterwards. Instead I see a claim that any restrictions are not allowed.
Oh boy, is this person not going to like the Schumer Report. But of course, since it is not explicitly motivated by making sure everyone doesn’t die, they haven’t noticed.
In particular, there is talk in the Schumer Report of classifying model weights and other AI information, above a threshold, on the grounds that it is Restricted Data. Which is a whole new level of ‘Fyour free speech.’ Also phrases like ‘reasonable steps’ to ‘protect children.’ Yet here they are, complaining about SB 1047’s self-certification of reasonable assurance of not causing catastrophic harm.
Section 3 repeats the misinformation that this could impact academic researchers. It repeats the false claim that ‘extensive safety evaluations’ must be made before training models. This is not true even for truly frontier, actively potentially deadly covered models, let alone academic models. The ‘reporting requirements’ could have a ‘chilling effect,’ because if an academic noticed their model was causing catastrophic risk, they really would prefer not to report that? What academia is this?
I could go on, but I won’t. The rest seems some combination of unnecessary to the central points, repetitive and false.
I do appreciate that there is a potential constitutionality issue here, no matter how absurd it might seem.
I also reiterate that if SB 1047 is unconstitutional, especially centrally so, then it is highly important that we discover this fact as soon as possible.
Jeremie & Edouard Harris of Gladstone AI go on The Joe Rogan Experience. It is hard for me to evaluate as I am not the target audience, and I am only an hour in so far, but this seemed like excellent communication of the basics of the existential risk case and situation. They boil a bunch of complicated questions into normie-compatible explanations.
In particular, the vibe seemed completely normal, as if the situation is what it is and we are facing it the same way we would face other compounding pending problems. I would have a few notes, but overall, I am very impressed.
If you had to point a low-shock-level normie towards one explanation of AI existential risk, this seems like our new go-to choice.
For context on Gladstone: These are the people who put out the Gladstone Report in March, featuring such section titles as ‘Executive Summary of Their Findings: Oh No.’ My takeaway was that they did a good job there investigating the top labs and making the case that there is a big problem, but they did not address the strongest arguments against regulatory action (I did give my counterarguments in the post).
Then they proposed extreme compute limits, that I believe go too far. California’s SB 1047 proposes light touch interventions at 10^26 flops, and neve proposes any form of pre-approval let alone a ban. Under the Gladstone proposal, you get light tough interventions at 10^23 flops (!), preapprovals are required at 10^24 flops (!!) and there is an outright ban at 10^25 flops (!!!) that would include current 4-level models. There are various requirements imposed on labs.
A lot of the hysterical reactions to SB 1047 would have been highly appropriate, if the reaction had been talking about the Gladstone Report’s proposals as stated in the report, whereas it seemed many had no interest in noticing the differences.
Latest Eliezer attempt to explain why you should expect some highly capable agents, as they gain in capability, to have bimodal distributions of behavior, where at some point they flip to behaviors you do not want them to have, and which cause things to end badly for you (or at least well for them). It is in their interest to act as if they had friendly intent or lacked dangerous capability or both, until that time. This is not something mysterious, it is the same for humans and groups of humans, and there is no known solution under a sufficient capability gap.
This explanation was in part a response to Nora Belrose saying Nora Belrose things, that seem similar to things she has said before, in the context here of responding to a particular other argument.
As a general rule on existential risk questions: I’ve learned that ‘respond to X’s response to Y’s response to Z’ gets frustrating fast and doesn’t convince people who aren’t X, Y or Z, so only do that if X is making a universal point. Don’t do it if X is telling Y in particular why they are wrong.
Eliezer clarifies some things about what he believes and considers plausible, and what he doesn’t, in a conversation about potential scenarios, including some evolution metaphors later on. My model of such arguments is that every now and then a reader will ‘become enlightened’ about something important because it hits them right, but that there are no arguments that work on that large a percentage of people at once.
Yann LeCunn denies the existence of GI, as in no general intelligence exists even in humans. Not no AGI, just no GI. It’s cleaner. This actually makes his positions about not getting to AGI make a lot more sense and I appreciate the clarity.
Eric Schmidt argues that rather than let a variety of AI agents do a bunch of things we don’t understand while coordinating in language we don’t understand, we should ‘pull the plug.’ Murat points out the incoherences, that all you need here is ‘agents doing things we don’t understand.’ The rest is unnecessary metaphor. Alas, I find many people need a metaphor that makes such issues click for them, so with notably rare exceptions I do not think we should offer pedantic corrections.
A true statement, although the emphasis on the decisions rather than the decision process perhaps suggests the wrong decision theories. Robin and I make different decisions in response.
Robin Hanson: The uber question for any decision-maker is: how much do you want your decisions to promote continued existence of things that are like you?
The more you want this, the more your decisions must be the sort that promote your kinds in a universe where natural selection decides what kinds exist. At least if you live in such a universe.
Dylan Matthews: I get the sense that Anthropic is currently trying to build that wight that Jon Snow and the gang capture and bring back to King’s Landing to prove that White Walkers are real.
The subsequent actions are a reasonable prediction of what would happen next, what many with power care about, the importance of a capabilities lead, the value of not giving up in the face of impossible odds, the dangers of various forms of misalignment, the need given our failure to step up in time to invent a deus ex machina for us all not to die, a dire warning about what happens when your source of creativity is used up and you use a fancy form of autocomplete, and more.
Tyler Cowen once again attempted on May 21, 2024 to inceptthat the ‘AI Safety’ movement is dead. The details included claiming that the AI safety movement peaked with the pause letter (not even the CAIS letter), gave what seemed like a very wrong reading of the Schumer report, came the same week as a humorously-in-context wide variety of AI safety related things saw progress, and had other strange claims as well, especially his model of how the best way to build AI safely is via not taking advance precautions and fixing issues ex-post.
Strangest of all is his continued insistence that the stock market being up is evidence against AI existential risk, or that those who think there is substantial AI existential risk should not be long the market and especially not long all these AI stocks we keep buying and that keep going up – I have tried to explainthis many times, yet we are both deeply confused how the other can be so supremely confidently wrong about this question.
I wrote a post length response to make sense of it all, but have decided to shelve it.
Another way is to say safety is a problem for future you: Here is a clip of Elon Musk saying first order of business at x.ai is a competitive model, comparable in power to others. Until then, no need to worry about safety. This in response to being asked to speak to x.ai’s safety team.
So…
Little happens in a day, no matter what Elon Musk might demand. You need to start worrying about safety long before you actually have a potentially unsafe system.
How do you build a culture of safety without caring about safety?
How do you have a safety-compatible AI if you don’t select for that path?
There are forms of safety other than existential, you need to worry even if you know there are stronger other models for purely mundane reasons.
If this is your attitude, why are you going to be better than the competition?
Elon Musk understands that AI is dangerous and can kill everyone. His ideas about how to prevent that and what he has done with those ideas have consistently been the actual worst, in the ‘greatly contribute to the chance everyone dies’ sense.
I do appreciate the straight talk. If you are going to not care about safety until events force your hand, then admit that. Don’t be like certain other companies that pay lip service and make empty promises, then break those promises.
Igor Babuschkin: Apply at x.ai if you want to be part of our journey to build AGI and understand the Universe 🛰️
Elon Musk: Join xAI if you believe in our mission of understanding the universe, which requires maximally rigorous pursuit of the truth, without regard to popularity or political correctness.
Investors in x.AI being motivated by something other than fundamental value.
Investors in x.AI buying into the hype way too much.
Investors in OpenAI getting an absurdly great deal.
Investors in OpenAI charging a huge discount for its structure, the AGI clause and the risks involved in trusting the people involved or the whole thing blowing up in various other ways.
Investors have very low confidence in OpenAI’s ability to execute.
Certainly OpenAI’s valuation being so low requires an explanation. But the same has been true for Nvidia for a while, so hey. Also a16z is heavily named in the x.AI fundraiser, which both is a terrible sign for x.AI’s safety inclinations, and also tells me everyone involved overpaid.
Another note is that x.AI seems highly dependent on Twitter (and Musk) to justify its existence and valuation. So if it is raising at $18 billion, the Twitter price starts to look a lot less terrible.
The buck has to stop somewhere. There are basically three scenarios.
Perhaps the trust will control the company when it chooses to do so.
Perhaps the shareholders will control the company when they choose to do so.
Perhaps both will have a veto over key actions, such as training or deployment.
The original intention seems, from what we know, to be something like ‘the trust is like the President and can veto or make certain executive decisions, and the shareholders are like Congress and can if sufficiently united get their way.’
The hope then would be that shareholders are divided such that when the decision is reasonable the trust can find enough support, but if it goes nuts they can’t, and the threshold is chosen accordingly.
My worry is this is a narrow window. Shareholders mostly want to maximize profits and are typically willing to vote with leadership. A very large supermajority is likely not that hard to get in most situations. I have been assuming that Anthropic is mostly a ‘normal company’ on legal governance, and putting a lot more hope in management making good choices than in the trust forcing their hand.
Roon: Do you really think AI race dynamics are about money?
Not entirely. But yeah, I kind of do. I think that the need to make the money in order to continue the work, and the need to make money in order to hire the best people, force everyone to race ahead specifically in order to make money. I think that the need to make money drives releases. I think that the more you need money, the more you have to turn over influence and control to those who focus on money, including Altman but much more so companies like Google and Microsoft. It is also the habit and pattern of an entire financial and cultural ecosystem.
Of course it is also ego, pride, hubris, The Potential, fear of the other guy, desire to dictate the arrangement of atoms within the light cone and other neat stuff like that.
Roon: I assume basically every statistic that suggests modernity is bad is a result of some kind of measurement error.
The context here is cellphones and teen depression. In general, modernity is good, we do not know how good we have it, and the statistics or other claims suggesting otherwise are bonkers.
That does not mean everything is better. To pick three: The decline in time spent with friends is obviously real. The rise in opioid deaths is real. And the fertility rate decline, in some ways the most important statistic of all, is very real.
You could say South Korea is doing great because it is rich. I say if women average less than one child the country is definitely not doing so great and I don’t care what your other statistics say, and if your answer is ‘so what everyone is so happy’ then I suggest watching some of their television because things do not seem all that happy.
No, no, no, why not both, the AI assistant you should want, safety issues aside:
Quoting from the ACX open thread announcements:
The next ACX Grants round will probably take place sometime in 2025, and be limited to grants ≤ $100K. If you need something sooner or bigger, the Survival and Flourishing Fund is accepting grant applications, due June 17. They usually fund a few dozen projects per year at between $5K and $1MM, and are interested in “organizations working to improve humanity’s long-term prospects for survival and flourishing”, broadly defined. You can see a list of their recent awardees here.
(just in case you have the same question everyone else did – no, “Short Women In AI Safety” and “Pope Alignment Research” aren’t real charities; SFF unwisely started some entries with the name of the project lead, and these were led by people named Short and Pope.)
I do think it is typically a good use of time, if your project is relevant to their interests (which include AI safety) to apply to the Survival and Flourishing Fund. The cost is low and the upside is high.
Yann LeCun echoes his central claim that if AI is not safe, controllable and can fulfill objectives in more intelligent ways than humans, we won’t build it. Yes, that claim is in the right section.
Whether you want to call it “Google Business Messaging” or “Google Business Profile Chat,” the chat buttons in Google Maps and Search are going away.
Google
This is the 2018 version of Google Maps Messaging, which is when it was first built into the Google Maps app.
Google
Messages used to have a top-tier spot in the navigation panel.
Google
In the current UI, Messages lives in the “Updates” tab.
Ron Amadeo
You used to be able to reply to Google Maps Messages with Google Allo.
Google is killing off a messaging service! This one is the odd “Google Business Messaging” service—basically an instant messaging client that is built into Google Maps. If you looked up a participating business in Google Maps or Google Search on a phone, the main row of buttons in the place card would read something like “Call,” “Chat,” “Directions,” and “Website.” That “Chat” button is the service we’re talking about. It would launch a full messaging interface inside the Google Maps app, and businesses were expected to use it for customer service purposes. Google’s deeply dysfunctional messaging strategy might lead people to joke about a theoretical “Google Maps Messaging” service, but it already exists and has existed for years, and now it’s being shut down.
Search Engine Land’s Barry Schwartz was the first to spot the shutdown emails being sent out to participating businesses. Google has two different support articles up for a shutdown of both “Google Business Profile Chat” and “Google Business Messages,” which appear to just be the same thing with different names. On July 15, 2024, the ability to start a new chat will be disabled, and on July 31, 2024, both services will be shut down. Google is letting businesses download past chat conversations via Google Takeout.
Google’s Maps messaging service was Google Messaging Service No. 16 in our giant History of Google Messaging article. The feature has undergone many changes, so it’s a bit hard to follow. The Google Maps Messaging button launched in 2017, when it would have been called “Google My Business Chat.” This wasn’t quite its own service yet—the messaging button would either launch your SMS app or boot into another dead Google messaging product, Google Allo!
The original SMS option was the easy path for small businesses with a single store, but SMS is tied to a single physical phone. If you’re a bigger business and want to take on the task of doing customer service across multiple stores, at the scale of Google Maps, that’s going to be a multi-person job. The Google Allo back-end (which feels like it was the driving force behind creating this project in the first place) would let you triage messages to multiple people. Allo was one year into its 2.5-year lifespan when this feature launched, though, so things would have to change soon before Allo’s 2019 shutdown date.
Knowing that the announcement of Allo’s death was a month away, Google started making Maps into its own standalone messaging service in November 2018. Previously, it would always launch an outside app (either SMS or Allo), but with this 2018 update, Maps got its own instant messaging UI built right into the app. “Messages” became a top-level item in the navigation drawer (later this would move to “updates”), and a third-party app was no longer needed. On the business side of things, a new “Google My Business” app would be the new customer service interface for all these messages. Allo’s shutdown in 2019 disabled the ability to use SMS for small businesses, and everything needed to use this Google My Business app now. Maps was officially a new messaging service. Google also created the “Business Messages API,” so big businesses could plug Maps messaging into some kind of customer management app.
It does not sound like Google is going to replace business messaging with anything in the near future, so the Chat buttons in Google Maps and search will be going away. In the endless pantheon of Google Messaging solutions, the Google Developer page also mentions an “RCS Business Messaging” platform that will launch the Google Messaging app. This service does not seem to be built into any existing Google products, though, and isn’t mentioned as an alternative in Google’s shutdown announcement. Google only suggests that businesses “redirect customers to your alternative communication channels,” but those links won’t be getting premium placement in Google’s products.
Business messaging is a pretty well-established market, and the Big Tech companies with competent messaging strategies are involved somehow. On iOS, there’s Apple’s iMessage-based Messages for Business, which also has a chat button layout in Apple Maps. Meta has both WhatsApp Business Messaging and Facebook Messenger’s Meta Business Messaging. There are also standalone businesses like Twilio.
Google needs to pump the brakes when it comes to tracking sensitive information shared with DMV sites, a new lawsuit suggests.
Filing a proposed class-action suit in California, Katherine Wilson has accused Google of using Google Analytics and DoubleClick trackers on the California DMV site to unlawfully obtain information about her personal disability without her consent.
This, Wilson argued, violated the Driver’s Privacy Protection Act (DPPA), as well as the California Invasion of Privacy Act (CIPA), and impacted perhaps millions of drivers who had no way of knowing Google was collecting sensitive information shared only for DMV purposes.
“Google uses the personal information it obtains from motor vehicle records to create profiles, categorize individuals, and derive information about them to sell its customers the ability to create targeted marketing and advertising,” Wilson alleged.
According to Wilson, California’s DMV “encourages” drivers “to use its website rather than visiting one of the DMV’s physical locations” without telling drivers that Google has trackers all over its site.
Likely due to promoting the website’s convenience, the DMV reported a record number of online transactions in 2020, Wilson’s complaint said. And people with disabilities have taken advantage of that convenience. In 2023, approximately “40 percent of the 1.6 million disability parking placard renewals occurred online.”
Wilson last visited the DMV site last summer when she was renewing her disability parking placard online. At that time, she did not know that Google obtained her personal information when she filled out her application, communicated directly with the DMV, searched on the site, or clicked on various URLs, all of which she said revealed that either she had a disability or believed she had a disability.
Her complaint alleged that Google secretly gathers information about the contents of the DMV’s online users’ searches, logging sensitive keywords like “teens,” “disabled drivers,” and any “inquiries regarding disabilities.”
Google “knowingly” obtained this information, Wilson alleged, to quietly expand user profiles for ad targeting, “intentionally” disregarding DMV website users’ “reasonable expectation of privacy.”
“Google then uses the personal information and data to generate revenue from the advertising and marketing services that Google sells to businesses and individuals,” Wilson’s complaint alleged. “That Plaintiff and Class Members would not have consented to Google obtaining their personal information or learning the contents of their communications with the DMV is not surprising.”
Congressman James P. Moran, who sponsored the DPPA in 1994, made it clear that the law was enacted specifically to keep marketers from taking advantage of computers making it easy to “pull up a person’s DMV record” with the “click of a button.”
Even back then, some people were instantly concerned about any potential “invasion of privacy,” Moran said, noting that “if you review the way in which people are classified by direct marketers based on DMV information, you can see why some individuals might object to their personal information being sold.”
IFixit and Samsung were once leading the charge in device repair, but iFixit says it’s ending its repair partnership with Samsung because it feels Samsung just isn’t participating in good faith. iFixit says the two companies “have not been able to deliver” on the promise of a viable repair ecosystem, so it would rather shut the project down than continue. The repair site says “flashy press releases and ambitious initiatives don’t mean much without follow-through.”
iFixit’s Scott Head explains: “As we tried to build this ecosystem we consistently faced obstacles that made us doubt Samsung’s commitment to making repair more accessible. We couldn’t get parts to local repair shops at prices and quantities that made business sense. The part prices were so costly that many consumers opted to replace their devices rather than repair them. And the design of Samsung’s Galaxy devices remained frustratingly glued together, forcing us to sell batteries and screens in pre-glued bundles that increased the cost.”
Samsung’s screen replacement parts usually require buying the display, battery, phone frame, and buttons, which is a big waste.
iFixit
A good example of Samsung’s parts bundling is this Galaxy S22 Ultra “screen” part for $233. The screen is the most common part to break, but rather than just sell a screen, Samsung makes you buy the screen, a new phone frame, a battery, and new side buttons and switches. As we said when this was announced, that’s like half of the total parts in an entire phone. This isn’t a perfect metric, but the Samsung/iFixit parts store only offers three parts for the S22 Ultra, while the Pixel 8 Pro store has 10 parts, and the iPhone 14 Pro Max store has 23 parts.
Even with Samsung’s part-bundling, though, iFixit’s complaint of high prices doesn’t seem reflected in the store pricing. The Pixel 8 Pro screen + fingerprint reader, without a case, battery, and buttons, is $230. An iPhone 14 Pro Max screen is $395. (There is a good chance Samsung is the manufacturer of all three of these displays.)
Samsung and iFixit have always had a rocky relationship. In 2017, the two companies were supposed to partner up for an “upcycling” program, where Samsung found new uses for old phones. The original plan included things like unlocking the bootloader of old devices, so Samsung’s OS could be completely replaced, and hosting an open source marketplace where users could submit ideas and software for repurposing old Galaxy devices. In what now seems like a familiar strategy, Samsung was more concerned about appearances than being actually useful, and iFixit said the upcycling program that launched in 2021 was “nearly unrecognizable” to what iFixit originally endorsed and lent its logo to in 2017.
Samsung has also reportedly been on the attack against repair, even while it partners with iFixit. On the same day that iFixit announced it was dropping the partnership, 404 Media reported that Samsung requires independent repair shops to turn over customer data and “immediately disassemble” any device found to be using third-party parts. Imagine taking your phone to a shop for repair and finding out it was destroyed by the shop as a requirement from Samsung. The report also says Samsung’s contracts require that independent companies “daily” upload to a Samsung database (called G-SPN) the details of each and every repair “at the time of each repair.”
With the latest chapter of the partnership store dying after just two years, in June 2024, iFixit says some changes are coming to its website. It won’t remove any information, but it will start offering clearly labeled third-party parts in addition to whatever Samsung OEM parts it can source. It will no longer collaborate with Samsung for manuals and won’t need to follow Samsung’s quantity limit requirements.
Enlarge/ Later theropods had multiple adaptations to varied temperatures.
Dinosaurs were once assumed to have been ectothermic, or cold-blooded, an idea that makes sense given that they were reptiles. While scientists had previously discovered evidence of dinosaur species that were warm-blooded, though what could have triggered this adaptation remained unknown. A team of researchers now think that dinosaurs that already had some cold tolerance evolved endothermy, or warm-bloodedness, to adapt when they migrated to regions with cooler temperatures. They also think they’ve found a possible reason for the trek.
Using the Mesozoic fossil record, evolutionary trees, climate models, and geography, plus factoring in a drastic climate change event that caused global warming, the team found that theropods (predators and bird ancestors such as velociraptor and T. rex) and ornithischians (such as triceratops and stegosaurus) must have made their way to colder regions during the Early Jurassic. Lower temperatures are thought to have selected for species that were partly adapted to endothermy.
“The early invasion of cool niches… [suggests] an early attainment of homeothermic (possibly endothermic) physiology in [certain species], enabling them to colonize and persist in even extreme latitudes since the Early Jurassic,” the researchers said in a study recently published in Current Biology.
Hot real estate
During the Mesozoic Era, which lasted from 230 to 66 million years ago, proto-dinosaurs known as dinosauromorphs began to diversify in hot and dry climates. Early sauropods, ornithischians, and theropods all tended to stay in these regions.
Sauropods (such as brontosaurus and diplodocus) would become the only dinosaur groups to bask in the heat—the fossil record shows that sauropods tended to stay in warmer areas, even if there was less food. This suggests the need for sunlight and heat associated with ectothermy. They might have been capable of surviving in colder temperatures but not adapted enough to make it for long, according to one hypothesis.
It’s also possible that living in cooler areas meant too much competition with other types of dinosaurs, as the theropods and ornithiscians did end up moving into these cooler areas.
Almost apocalypse
Beyond the ecological opportunities that may have drawn dinosaurs to the cooler territories, it’s possible they were driven away from the warm ones. Around 183 million years ago, there was a perturbation in the carbon cycle, along with extreme volcanism that belched out massive amounts of methane, sulfur dioxide, and mercury. Life on Earth suffered through scorching heat, acid rain, and wildfires. Known as the Early Jurassic Jenkyns Event, the researchers now think that these disruptions pushed theropod and ornithischian dinosaurs to cooler climates because temperatures in warmer zones went above the optimal temperatures for their survival.
The theropods and ornithischians that escaped the effects of the Jenkyns event may have had a key adaptation to cooler climes; many dinosaurs from these groups are now thought to have been feathered. Feathers can be used to both trap and release heat, which would have allowed feathered dinosaurs to regulate their body temperature in more diverse climates. Modern birds use their feathers the same way.
Dinosaur species with feathers or special structures that improved heat management could have been homeothermic, which means they would have been able to maintain their body temperature with metabolic activity or even endothermic.
Beyond the dinosaurs that migrated to high latitudes and adapted to a drop in temperature, endothermy might have led to the rise of new species and lineages of dinosaurs. It could have contributed to the rise of Avialae, the clade that includes birds—the only actual dinosaurs still around—and traces all the way back to their earliest ancestors.
“[Our findings] provide novel insights into the origin of avian endothermy, suggesting that this evolutionary trajectory within theropods… likely started in the latest Early Jurassic,” the researchers said in the same study.
That really is something to think about next time a sparrow flies by.
A software maker serving more than 10,000 courtrooms throughout the world hosted an application update containing a hidden backdoor that maintained persistent communication with a malicious website, researchers reported Thursday, in the latest episode of a supply-chain attack.
The software, known as the JAVS Viewer 8, is a component of the JAVS Suite 8, an application package courtrooms use to record, play back, and manage audio and video from proceedings. Its maker, Louisville, Kentucky-based Justice AV Solutions, says its products are used in more than 10,000 courtrooms throughout the US and 11 other countries. The company has been in business for 35 years.
JAVS Viewer users at high risk
Researchers from security firm Rapid7 reported that a version of the JAVS Viewer 8 available for download on javs.com contained a backdoor that gave an unknown threat actor persistent access to infected devices. The malicious download, planted inside an executable file that installs the JAVS Viewer version 8.3.7, was available no later than April 1, when a post on X (formerly Twitter) reported it. It’s unclear when the backdoored version was removed from the company’s download page. JAVS representatives didn’t immediately respond to questions sent by email.
“Users who have version 8.3.7 of the JAVS Viewer executable installed are at high risk and should take immediate action,” Rapid7 researchers Ipek Solak, Thomas Elkins, Evan McCann, Matthew Smith, Jake McMahon, Tyler McGraw, Ryan Emmons, Stephen Fewer, and John Fenninger wrote. “This version contains a backdoored installer that allows attackers to gain full control of affected systems.”
The installer file was titled JAVS Viewer Setup 8.3.7.250-1.exe. When executed, it copied the binary file fffmpeg.exe to the file path C:Program Files (x86)JAVSViewer 8. To bypass security warnings, the installer was digitally signed, but with a signature issued to an entity called “Vanguard Tech Limited” rather than to “Justice AV Solutions Inc.,” the signing entity used to authenticate legitimate JAVS software.
fffmpeg.exe, in turn, used Windows Sockets and WinHTTP to establish communications with a command-and-control server. Once successfully connected, fffmpeg.exe sent the server passwords harvested from browsers and data about the compromised host, including hostname, operating system details, processor architecture, program working directory, and the user name.
The researchers said fffmpeg.exe also downloaded the file chrome_installer.exe from the IP address 45.120.177.178. chrome_installer.exe went on to execute a binary and several Python scripts that were responsible for stealing the passwords saved in browsers. fffmpeg.exe is associated with a known malware family called GateDoor/Rustdoor. The exe file was already flagged by 30 endpoint protection engines.
Enlarge/ A screenshot from VirusTotal showing detections from 30 endpoint protection engines.
The researchers warned that the process of disinfecting infected devices will require care. They wrote:
To remediate this issue, affected users should:
Reimage any endpoints where JAVS Viewer 8.3.7 was installed. Simply uninstalling the software is insufficient, as attackers may have implanted additional backdoors or malware. Re-imaging provides a clean slate.
Reset credentials for any accounts that were logged into affected endpoints. This includes local accounts on the endpoint itself as well as any remote accounts accessed during the period when JAVS Viewer 8.3.7 was installed. Attackers may have stolen credentials from compromised systems.
Reset credentials used in web browsers on affected endpoints. Browser sessions may have been hijacked to steal cookies, stored passwords, or other sensitive information.
Install the latest version of JAVS Viewer (8.3.8 or higher) after re-imaging affected systems. The new version does not contain the backdoor present in 8.3.7.
Completely re-imaging affected endpoints and resetting associated credentials is critical to ensure attackers have not persisted through backdoors or stolen credentials. All organizations running JAVS Viewer 8.3.7 should take these steps immediately to address the compromise.
The Rapid7 post included a statement from JAVS that confirmed that the installer for version 8.3.7 of the JAVS viewer was malicious.
“We pulled all versions of Viewer 8.3.7 from the JAVS website, reset all passwords, and conducted a full internal audit of all JAVS systems,” the statement read. “We confirmed all currently available files on the JAVS.com website are genuine and malware-free. We further verified that no JAVS Source code, certificates, systems, or other software releases were compromised in this incident.”
The statement didn’t explain how the installer became available for download on its site. It also didn’t say if the company retained an outside firm to investigate.
The incident is the latest example of a supply-chain attack, a technique that tampers with a legitimate service or piece of software with the aim of infecting all downstream users. These sorts of attacks are usually carried out by first hacking the provider of the service or software. There’s no sure way to prevent falling victim to supply-chain attacks, but one potentially useful measure is to vet a file using VirusTotal before executing it. That advice would have served JAVS users well.
The Windows 11 24H2 update isn’t scheduled to be released until sometime this fall, but testers can get a near-final version of it early. Microsoft has released Windows 11 24H2 build 26100.712 to its Release Preview testing channel for Windows Insiders, a sign that the update is nearly complete and that the company has shifted into bug-fixing mode ahead of general availability.
Microsoft has generally stuck to smaller but more frequent feature updates during the Windows 11 era, but the annual fall updates still tend to be a bigger deal. They’re the ones that determine whether you’re still eligible for security updates, and they often (but not always) come with more significant under-the-hood changes than the normal feature drops.
Case in point: Windows 11 24H2 includes an updated compiler, kernel, and scheduler, all lower-level system changes made at least in part to better support Arm-based PCs. Existing Windows-on-Arm systems should also see a 10 or 20 percent performance boost when using x86 applications, thanks to improvements in the translation layer (which Microsoft is now calling Prism).
There are more user-visible changes, too. 24H2 includes Sudo for Windows, the ability to create TAR and 7-zip archives from the File Explorer, Wi-Fi 7 support, a new “energy saver” mode, and better support for Bluetooth Low Energy Audio. It also allows users to run the Copilot AI chatbot in a regular resizable window that can be pinned to the taskbar instead of always giving it a dedicated strip of screen space.
Other new Windows features are tied to the 24H2 update but will only be available on Copilot+ PCs, which have their own specific system requirements: 16 GB of memory, 256 GB of storage, and a neural processing unit (NPU) capable of at least 40 trillion operations per second (TOPS). As of right now, the only chips that fit the bill are Qualcomm’s Snapdragon X Plus and X Elite processors, though Intel and AMD systems with faster NPUs should be released later this year. Microsoft will maintain a separate list of processors that support the Copilot+ features.
The biggest 24H2 feature specific to Copilot+ PCs is Recall, which continually takes snapshots of everything you do with your PC so that you can look up your own activities later. This comes with obvious privacy and security risks, though Microsoft says that all of Recall’s data is encrypted on disk and processed entirely locally by the NPU rather than leveraging the cloud. Other Copilot+ features include Live Captions for captioning video files or video calls in real time and features for generating new images and enhancing existing images.
Collectively, all of these changes make 24H2 the most significant Windows 11 release since the 22H2 update came out a year and a half ago. 22H2 has served as the foundation for most new Windows features since then, including the Copilot chatbot, and 23H2 was mostly just a version number change released to reset the clock on Microsoft’s security update timeline.
Despite all of these changes and additions, the 24H2 update is still called Windows 11, still looks like Windows 11, and doesn’t change Windows 11’s official minimum system requirements. Unsupported installs will stop working on a few generations’ worth of older 64-bit x86 CPUs, though these chips are old and slow enough that they wouldn’t run Windows 11 particularly well in the first place.
For people who want to start fresh, ISO files of the release are available from Microsoft’s download page here (this is a slightly older build of the OS, 26100.560, but it should update to the current version with no issues after installation). You can update a current Windows 11 install from the Insider section in the Settings app. Microsoft says to expect the full release “later this calendar year.” Based on past precedent, it’s most likely to come out in the fall, but it will probably ship a bit early on the first wave of Copilot+ Arm PCs that will be available in mid-June.
There is no substitute. This is not strictly a bill, but it is important.
The introduction kicks off balancing upside and avoiding downside, utility and risk. This will be a common theme, with a very strong ‘why not both?’ vibe.
Early in the 118th Congress, we were brought together by a shared recognition of the profound changes artificial intelligence (AI) could bring to our world: AI’s capacity to revolutionize the realms of science, medicine, agriculture, and beyond; the exceptional benefits that a flourishing AI ecosystem could offer our economy and our productivity; and AI’s ability to radically alter human capacity and knowledge.
At the same time, we each recognized the potential risks AI could present, including altering our workforce in the short-term and long-term, raising questions about the application of existing laws in an AI-enabled world, changing the dynamics of our national security, and raising the threat of potential doomsday scenarios. This led to the formation of our Bipartisan Senate AI Working Group (“AI Working Group”).
They did their work over nine forums.
Inaugural Forum
Supporting U.S. Innovation in AI
AI and the Workforce
High Impact Uses of AI
Elections and Democracy
Privacy and Liability
Transparency, Explainability, Intellectual Property, and Copyright
Safeguarding Against AI Risks
National Security
Existential risks were always given relatively minor time, with it being a topic for at most a subset of the final two forums. By contrast, mundane downsides and upsides were each given three full forums. This report was about response to AI across a broad spectrum.
They lead with a proposal to spend ‘at least’ $32 billion a year on ‘AI innovation.’
No, there is no plan on how to pay for that.
In this case I do not think one is needed. I would expect any reasonable implementation of that to pay for itself via economic growth. The downsides are tail risks and mundane harms, but I wouldn’t worry about the budget. If anything, AI’s arrival is a reason to be very not freaked out about the budget. Official projections are baking in almost no economic growth or productivity impacts.
They ask that this money be allocated via a method called emergency appropriations. This is part of our government’s longstanding way of using the word ‘emergency.’
We are going to have to get used to this when it comes to AI.
Events in AI are going to be happening well beyond the ‘non-emergency’ speed of our government and especially of Congress, both opportunities and risks.
We will have opportunities that appear and compound quickly, projects that need our support. We will have stupid laws and rules, both that were already stupid or are rendered stupid, that need to be fixed.
Risks and threats, not only catastrophic or existential risks but also mundane risks and enemy actions, will arise far faster than our process can pass laws, draft regulatory rules with extended comment periods and follow all of our procedures.
In this case? It is May. The fiscal year starts in October. I want to say, hold your damn horses. But also, you think Congress is passing a budget this year? We will be lucky to get a continuing resolution. Permanent emergency. Sigh.
What matters more is, what do they propose to do with all this money?
A lot of things. And it does not say how much money is going where. If I was going to ask for a long list of things that adds up to $32 billion, I would say which things were costing how much money. But hey. Instead, it looks like he took the number from NSCAI, and then created a laundry list of things he wanted, without bothering to create a budget of any kind?
It also seems like they took the original recommendation of $8 billion in Fiscal Year 24, $16 billion in FY 25 ad $32 billion in FY 26, and turned it into $32 billion in emergency funding now? See the appendix. Then again, by that pattern, we’d be spending a trillion in FY 31. I can’t say for sure that we shouldn’t.
Starting with the top priority:
An all government ‘AI-ready’ initiative.
‘Responsible innovation’ R&D work in fundamental and applied sciences.
R&D work in ‘Foundational trustworthy AI topics, such as transparency, explainability, privacy, interoperability, and security.’
Or:
Government AI adoption for mundane utility.
AI for helping scientific research.
AI safety in the general sense, both mundane and existential.
Great. Love it. What’s next?
Funding the CHIPS and Science Act accounts not yet fully funded.
My current understanding is this is allocation of existing CHIPS act money. Okie dokie.
Funding ‘as needed’ (oh no) for semiconductor R&D for the design and manufacture of high-end AI chips, through co-design of AI software and hardware, and developing new techniques for semiconductor fabrication that can be implemented domestically.
More additional CHIPS act funding, perhaps unlimited? Pork for Intel? I don’t think the government is going to be doing any of this research, if it is then ‘money gone.’
Pass the Create AI Act (S. 2714) and expand programs like NAIRR to ‘ensure all 50 states are able to participate in the research ecosystem.’
More pork, then? I skimmed the bill. Very light on details. Basically, we should spend some money on some resources to help with AI research and it should include all the good vibes words we can come up with. I know what ‘all 50 states’ means. Okie dokie.
Congress’s website does not list text for S. 4236. S. 4178 seems to mean ‘grand challenge’ in the senses of prizes and other pay-for-results (generally great), and having ambitious goals (also generally great), which tend to not be how the system works these days.
So, fund ambitious research, and use good techniques.
Funding for AI efforts at NIST, including AI testing and evaluation infrastructure and the U.S. AI Safety Institute, and funding for NIST’s construction account to address years of backlog in maintaining NIST’s physical infrastructure.
Not all of NIST’s AI effort is safety, but a large portion of our real government safety efforts are at NIST. They are severely underfunded by all accounts right now. Great.
Funding for the Bureau of Industry and Security (BIS) to update its IT and data analytics software and staff up.
That does sound like something we should do, if it isn’t handled. Ensure BIS can enforce the rules it is tasked with enforcing, and choose those rules accordingly.
Funding R&D at the intersection of AI and robotics to ‘advance national security, workplace safety, industrial efficiency, economic productivity and competitiveness, through a coordinated interagency initiative.’
AI robots. The government is going to fund AI robots. With the first goal being ‘to advance national security.’ Sure, why not, I have never seen any movies.
In all seriousness, this is not where the dangers lie, and robots are useful. It’s fine.
The interagency plan seems unwise to me but I’m no expert on that.
R&D for AI to discover manufacturing techniques.
Once again, sure, good idea if you can improve this for real and this isn’t wasted or pork. Better general manufacturing is good. My guess is that this is not a job for the government and this is wasted, but shrug.
Security grants for AI readiness to help secure American elections.
Given the downside risks I presume this money is well spent.
Modernize the federal government and improve delivery of government services, through updating IT and using AI.
Deploying new technologies to find inefficiencies in the U.S. code, federal rules and procurement devices.
Yes, please. Even horribly inefficient versions of these things are money well spent.
R&D and interagency coordination around intersection of AI and critical infrastructure, including for smart cities and intelligent transportation system technologies.
Yes, we are on pace to rapidly put AIs in charge of our ‘critical infrastructure’ along with everything else, why do you ask? Asking people nicely not to let AI anywhere near the things is not an option and wouldn’t protect substantially against existential risks (although it might versus catastrophic ones). If we are going to do it, we should try to do it right, get the benefits and minimize the risks and costs.
Overall I’d say we have three categories.
Many of these points are slam dunk obviously good. There is a lot of focus on enabling more mundane utility, and especially mundane utility of government agencies and government services. These are very good places to be investing.
A few places where it seems like ‘not the government’s job’ to stick its nose, and where I do not expect the money to accomplish much, often that also involve some obvious nervousness around the proposals, but none of which actually amplify the real problems. Mostly I expect wasted money. The market already presents plenty of better incentives for basic research in most things AI.
Semiconductors.
It is entirely plausible for this to be a plan to take most of $32 billion (there’s a second section below that also gets funding), and put most of that into semiconductors. They can easily absorb that kind of cash. If you do it right you could even get your money’s worth.
As usual, I am torn on chips spending. Hardware progress accelerates core AI capabilities, but there is a national security issue with the capacity relying so heavily on Taiwan, and our lead over China here is valuable. That risk is very real.
Either way, I do know that we are not going to talk our government into not wanting to promote domestic chip production. I am not going to pretend that there is a strong case in opposition to that, nor is this preference new.
On AI Safety, this funds NIST, and one of its top three priorities is a broad-based call for various forms of (both existential and mundane) AI safety, and this builds badly needed state capacity in various places.
As far as government spending proposals go, this seems rather good, then, so far.
These get their own section with twelve bullet points.
NNSA testbeds and model evaluation tools.
Assessment of CBRN AI-enhanced threats.
AI-advancements in chemical and biological synthesis, including safeguards to reduce risk of synesthetic materials and pathogens.
Fund DARPA’s AI work, which seem to be a mix of military applications and attempts to address safety issues including interpretability, along with something called ‘AI Forward’ for more fundamental research.
Secure and trustworthy algorithms for DOD.
Combined Joint All-Domain Command and Control Center for DOD.
AI tools to improve weapon platforms.
Ways to turn DOD sensor data into AI-compatible formats.
Building DOD’s AI capabilities including ‘supercomputing.’ I don’t see any sign this is aiming for foundation models.
Utilize AUKUS Pillar 2 to work with allies on AI defense capabilities.
Use AI to improve implementation of Federal Acquisition Regulations.
I notice in #11 that they want to improve implementation, but not work to improve the regulations themselves, in contrast to the broader ‘improve our procedures’ program above. A sign of who cares about what, perhaps.
Again, we can draw broad categories.
AI to make our military stronger.
AI (mundate up through catastrophic, mostly not existential) safety.
The safety includes CBRN threat analysis, testbed and evaluation tools and a lot of DARPA’s work. There’s plausibly some real stuff here, although you can’t tell magnitude.
This isn’t looking ahead to AGI or beyond. The main thing here is ‘the military wants to incorporate AI for its mundane utility,’ and that includes guarding us against outside threats and ensuring its implementations are reliable and secure. It all goes hand in hand.
Would I prefer a world where all the militaries kept their hands off AI? I think most of us would like that, no matter our other views, But also we accept that we live in a very different world that is not currently capable of that. And I understand that, while it feels scary for obvious reasons and does introduce new risks, this mostly does not change the central outcomes. It does impact the interplay among people and nations in the meantime, which could alter outcomes if it impacts the balance of power, or causes a war, or sufficiently freaks enough people out.
Mostly it seems like a clear demonstration of the pattern of ‘if you were thinking we wouldn’t do or allow that, think again, we will instantly do that unless prevented’ to perhaps build up some momentum towards preventing things we do not want.
Most items in the next section are about supporting small business.
Developing legislation to leverage public-private partnerships for both capabilities and to mitigate risks.
Further federal study of AI including through FFRDRCs.
Supporting startups, including at state and local levels, including by disseminating best practices (to the states and locaties, I think, not to the startups?)
The Comptroller General identifying anything statutes that impact innovation and competition in AI systems. Have they tried asking Gemini?
Increasing access to testing tools like mock data sets, including via DOC.
Doing outreach to small businesses to ensure tools meet their needs.
Finding ways to support small businesses utilizing AI and doing innovation, and consider if legislation is needed to ‘disseminate best practices’ in various states and localities.
Ensuring business software and cloud computing are allowable expenses under the SBA’s 7(a) loan program.
Congress has a longstanding tradition that Small Business is Good, and that Geographic Diversity That Includes My State or District is Good.
A lot of this seems like ways to throw money at small businesses in inefficient ways? And to try and ‘make geographic diversity happen’ when we all know it is not going to happen? I am not saying you have to move to the Bay if that is not your thing, I don’t hate you that much, but at least consider, let’s say, Miami or Austin.
In general, none of this seems like a good idea. Not because it increases existential risk. Because it wastes our money. It won’t work.
The good proposal here is the fourth one. Look for statues that are needlessly harming competition and innovation.
Padme: And then remove them?
(The eighth point also seems net positive, if we are already going down the related roads.)
The traditional government way is to say they support small business and spend taxpayer money by giving it to small business, and then you create a regulatory state and set of requirements that wastes more money and gives big business a big edge anyway. Whenever possible, I would much rather remove the barriers than spend the money.
Not all rules are unnecessary. There are some real costs and risks, mundane, catastrophic and exponential, to mitigate.
Nor are all of the advantages of being big dependent on rules and compliance and regulatory capture, especially in AI. AI almost defines economies of scale.
Many would say, wait, are not those worried about AI safety typically against innovation and competition and small business?
And I say nay, not in most situations in AI, same as almost all situations outside AI. Most of the time all of that is great. Promoting such things in general is great, and is best done by removing barriers.
The key question is, can you do that in a way that works, and can you do it while recognizing the very high leverage places that break the pattern?
In particular, when the innovation in question is highly capable future frontier models that pose potential catastrophic or existential risks, especially AGI or ASI, and especially when multiple labs are racing against each other to get there first.
In those situations, we need to put an emphasis on ensuring safety, and we need to at minimum allow communication and coordination between those labs without risk of the government interfering in the name of antitrust.
In most other situations, including most of the situations this proposal seeks to assist with, the priorities here are excellent. The question is execution.
Do you want to help small business take on big business?
Do you want to encourage startups and innovation and American dynamism?
Then there are two obvious efficient ways to do that. Both involve the tax code.
The first is the generic universal answer.
If you want to favor small business over big business, you can mostly skip all those ‘loans’ and grants and applications and paperwork and worrying what is an expense under 7(a). And you can stop worrying about providing them with tools, and you can stop trying to force them to have geographic diversity that doesn’t make economic sense – get your geographic diversity, if you want it, from other industries.
Instead, make the tax code explicitly favor small business over big business via differentiating rates, including giving tax advantages to venture capital investments in early stage startups, which then get passed on to the business.
If you want to really help, give a tax break to the employees, so it applies even before the business turns a profit.
If you want to see more of something, tax it less. If you want less, tax it more. Simple.
The second is fixing a deeply stupid mistake that everyone, and I do mean everyone, realizes is a mistake that was made in the Trump tax cuts, but that due to Congress being Congress we have not yet fixed, and that is doing by all reports quite a lot of damage. It is Section 174 of the IRS code requiring that software engineers and other expenses related to research and experimental activities (R&E) can only be amortized over time rather than fully deducted.
The practical result of this is that startups and small businesses, that have negative cash flow, look to the IRS as if they are profitable, and then owe taxes. This is deeply, deeply destructive and stupid in one of the most high leverage places.
From what I have heard, the story is that the two parties spent a long time negotiating a fix for it, it passed the house overwhelmingly, then in the Senate the Republicans decided they did not like the deal package of other items included with the fix, and wanted concessions, and the Democrats, in particular Schumer, said a deal is a deal.
This needs to get done. I would focus far more on that than all these dinky little subsidies.
As usual, Congress takes ‘the effect on jobs’ seriously. Workers must not be ‘left behind.’ And as usual, they are big on preparing.
So, what are you going to do about it, punk? They are to encourage some things:
‘Efforts to ensure’ that workers and other stakeholders are ‘consulted’ as AI is developed and deployed by end users. A government favorite.
Stakeholder voices get considered in the development and deployment of AI systems procured or used by federal agencies. In other words, use AI, but not if it would take our jobs.
Legislation related to training, retraining (drink!) and upskilling the private sector workforce, perhaps with business incentives, or to encourage college courses. I am going to go out on a limb and say that this pretty much never, ever works.
Explore implications and possible ‘solutions to’ the impact of AI on the long-term future of work as general-purpose AI systems displace human workers, and develop a framework for policy response. So far, I’ve heard UBI, and various versions of disguising to varying degrees versions of hiring people to dig holes and fill them up again, except you get private companies to pay for it.
Consider legislation to improve U.S. immigration systems for high-skilled STEM workers in support of national security and to foster advances in AI across the whole country.
My understanding is that ideas like the first two are most often useless but also most often mostly harmless. Steps are taken to nominally ‘consult,’ most of the time nothing changes.
Sometimes, they are anything but harmless. You get NEPA. The similar provisions in NEPA were given little thought when first passed, then they grew and morphed into monsters strangling the economy and boiling the planet, and no one has been able to stop them.
If this applies only to federal agencies and you get the NEPA version, that is in a sense the worst possible scenario. The government’s ability to use AI gets crippled, leaving it behind. Whereas it would provide no meaningful check on frontier model development, or on other potentially risky or harmful private actions.
Applying it across the board could at the limit actually cripple American AI, in a way that would not serve as a basis for stopping international efforts, so that seems quite bad.
We should absolutely expand and improve high skill immigration, across all industries. It is rather completely insane that we are not doing so. There should at minimum be unlimited HB-1s. Yes, it helps ‘national security’ and AI but also it helps everything and everyone and the whole economy and we’re just being grade-A stupid not to do it.
They call this ‘high impact uses of AI.’
The report starts off saying existing law must apply to AI. That includes being able to verify that compliance. They note that this might not be compatible with opaque AI systems.
Their response if that happens? Tough. Rules are rules. Sucks to be you.
Indeed, they say to look not for ways to accommodate black box AI systems, but instead look for loopholes where existing law does not cover AI sufficiently.
Not only do they not want to ‘fix’ existing rules that impose, they want to ensure any possible loopholes are closed regarding information existing law requires. The emphasis is on anti-discrimination laws, which are not something correlation machines you can run tests on are going to be in the default habit of not violating.
So what actions are suggested here?
Explore where we might need explainability requirements.
Develop standards for AI in critical infrastructure.
Better monitor energy use.
Keep a closer eye on financial services providers.
Keep a closer eye on the housing sector.
Test and evaluate all systems before the government buys them, and also streamline the procurement process (yes these are one bullet point).
Recognize the concerns of local news (drink!) and journalism that have resulted in fewer local news options in small towns and rural areas. Damn you, AI!
Develop laws against AI-generated child sexual abuse material (CSAM) and deepfakes. There is a bullet here, are they going to bite it?
Think of the children, consider laws to protect them, require ‘reasonable steps.’
If you are at a smaller company working on AI, and you are worried about SB 1047 or another law that specifically targets frontier models and the risk of catastrophic harm, and you are not worried about being required to ‘take reasonable steps’ to ‘protect children,’ then I believe you are very much worried about the wrong things.
You can say and believe ‘the catastrophic risk worries are science fiction and not real, whereas children actually exist and get harmed’ all you like. This is not where I try to argue you out of that position.
That does not change which proposed rules are far more likely to actually make your life a living hell and bury your company, or hand the edge to Big Tech.
Hint: It is the one that would actually apply to you and the product you are offering.
Encourage public-private partnerships and other mechanisms to develop fraud detection services.
Continue work on autonomous vehicle testing frameworks. We must beat the CCP (drink!) in the race to shape the vision of self-driving cars.
Ban use of AI for social scoring to protect our freedom unlike the CCP (drink!)
“Review whether other potential uses for AI should be either extremely limited or banned.”
Did you feel that chill up your spine? I sure did. The ‘ban use cases’ approach is big trouble without solving your real problems.
Then there’s the health care notes.
Both support deployment of AI in health care and implement appropriate guardrails, including consumer protection, fraud and abuse prevention, and promoting accurate and representative data, ‘as patients must be front and center in any legislative efforts on healthcare and AI.’ My heart is sinking.
Make research data available while preserving privacy.
Ensure HHS and FDA ‘have the proper tools to weigh the benefits and risks of AI-enabled products so that it can provide a predictable regulatory structure. for product developers.’ The surface reading would be: So, not so much with the products, then. I have been informed that it is instead likely they are using coded language for the FDA’s pre-certification program to allow companies to self-certify software updates. And yes, if your laws require that then you should do that, but it would be nice to say it in English.
Transparency for data providers and for the training data used in medical AIs.
Promote innovation that improves health outcomes and efficiencies. Examine reimbursement mechanisms and guardrails for Medicare and Medicaid, and broad application.
The refrain is ‘give me the good thing, but don’t give me the downside.’
I mean, okay, sure, I don’t disagree exactly? And yet.
The proposal to use AI to improve ‘efficiency’ of Medicare and Medicaid sounds like the kind of thing that would be a great idea if done reasonably and yet quite predictably costs you the election. In theory, if we could all agree that we could use the AI to figure out which half of medicine wasn’t worthwhile and cut it, or how to actually design a reimbursement system with good incentives and do that, that would be great. But I have no idea how you could do that.
For elections they encourage deployers and content providers to implement robust protections, and ‘to mitigate AI-generated content that is objectively false, while still preserving First Amendment rights.’ Okie dokie.
For privacy and liability, they kick the can, ask others to consider what to do. They do want you to know privacy and strong privacy laws are good, and AIs sharing non-public personal information is bad. Also they take a bold stand that developers or users who cause harm should be held accountable, without any position on what counts as causing harm.
The word ‘encouraging’ is somehow sounding more ominous each time I see it.
What are we encouraging now?
A coherent approach to public-facing transparency requirements for AI systems, while allowing use case specific requirements where necessary and beneficial, ‘including best practices for when AI developers should disclose when their products are AI,’ but while making sure the rules do not inhibit innovation.
I am not sure how much more of this kind of language of infinite qualifiers and why-not-both framings I can take. For those taking my word for it, it is much worse in the original.
One of the few regulatory rules pretty much everyone agrees on, even if some corner cases involving AI agents are tricky, is ‘AI should have to clearly identify when you are talking to an AI.’
My instinctive suggestion for operationalizing the rule would be ‘if an AI sends a freeform message (e.g. not a selection from a fixed list of options, in any modality) that was not approved individually by a human (even if sent to multiple targets), in a way a reasonable person might think was generated by or individually approved by a human, it must be identified as AI-generated or auto-generated.’ Then iterate from there.
As the report goes on, it feels like there was a vibe of ‘all right, we need to get this done, let’s put enough qualifiers on every sentence that no one objects and we can be done with this.’
How bad can it get? Here’s a full quote for the next one.
“Evaluate whether there is a need for best practices for the level of automation that is appropriate for a given type of task, considering the need to have a human in the loop at certain stages for some high impact tasks.”
I am going to go out on a limb and say yes. There is a need for best practices for the level of automation that is appropriate for a given type of task, considering the need to have a human in the loop at certain stages for some high impact tasks.
For example, if you want to launch nuclear weapons, that is a high impact task, and I believe we should have some best practices for when humans are in the loop.
Seriously, can we just say things that we are encouraging people to consider? Please?
They also would like to encourage the relevant committees to:
Consider telling federal employees about AI in the workplace.
Consider transparency requirements and copyright issues about data sets.
Review reports from the executive branch.
Getting hardware to watermark generated media, and getting online platforms to display that information.
And just because such sentences needs to be properly shall we say appreciated:
“Consider whether there is a need for legislation that protects against the unauthorized use of one’s name, image, likeness, and voice, consistent with First Amendment principles, as it relates to AI. Legislation in this area should consider the impacts of novel synthetic content on professional content creators of digital media, victims of non-consensual distribution of intimate images, victims of fraud, and other individuals or entities that are negatively affected by the widespread availability of synthetic content.”
As opposed to, say, ‘Consider a law to protect people’s personality rights against AI.’
Which may or may not be necessary, depending on the state of current law. I haven’t investigated enough to know if what we have is sufficient here.
Ensure we continue to ‘lead the world’ on copyright and intellectual property law.
I have some news about where we have been leading the world on these matters.
Do a public awareness and educational campaign on AI’s upsides and downsides.
Now to what I view as the highest stakes question. What about existential risks?
That is also mixed in with catastrophic mundane risks.
If I had to summarize this section, I would say that they avoid making mistakes and are headed in the right direction, and they ask good questions.
But on the answers? They punt.
The section is short and dense, so here is their full introduction.
In light of the insights provided by experts at the forums on a variety of risks that different AI systems may present, the AI Working Group encourages companies to perform detailed testing and evaluation to understand the landscape of potential harms and not to release AI systems that cannot meet industry standards.
This is some sort of voluntary testing and prior restraint regime? You are ‘encouraged’ to perform ‘detailed testing and evaluation to understand the landscape of potential harms,’ and you must then ‘meet industry standards.’ If you can’t, don’t release.
Whether or not that is a good regime depends on:
Would companies actually comply?
Would industry adopt standards that mean we wouldn’t die?
Do we have to worry about problems that arise prior to release?
I doubt the Senators minds are ready for that third question.
Multiple potential risk regimes were proposed – from focusing on technical specifications such as the amount of computation or number of model parameters to classification by use case – and the AI Working Group encourages the relevant committees to consider a resilient risk regime that focuses on the capabilities of AI systems, protects proprietary information, and allows for continued AI innovation in the U.S.
Very good news. Capabilities have been selected over use case. The big easy mistake is to classify models based on what people say they plan to do, rather than asking what the model is capable of doing. That is a doomed approach, but many lobby hard for it.
The risk regime should tie governance efforts to the latest available research on AI capabilities and allow for regular updates in response to changes in the AI landscape.
Yes. As we learn more, our policies should adjust, and we should plan for that. Ideally this would be an easy thing to agree upon. Yet the same people who say ‘it is too early to choose what to do’ will also loudly proclaim that ‘if you give any flexibility to choose what to do later to anyone but the legislature, one must assume it will used maximally badly.’ I too wish we had a much faster, better legislature, that we could turn to every time we need any kind of decision or adjustment. We don’t.
All right. So no explicit mention of existential risk in the principles, but some good signs of the right regime. What are the actual suggestions?
Again, I am going to copy it all, one must parse carefully.
Support efforts related to the development of a capabilities-focused risk-based approach, particularly the development and standardization of risk testing and evaluation methodologies and mechanisms, including red-teaming, sandboxes and testbeds, commercial AI auditing standards, bug bounty programs, as well as physical and cyber security standards. The AI Working Group encourages committees to consider ways to support these types of efforts, including through the federal procurement system.
There are those who would disagree with this, who think the proper order is train, release then test. I do not understand why they would think that. No wise company would do that, for its own selfish reasons.
The questions should be things like:
How rigorous should be the testing requirements?
At what stages of training and post-training, prior to deployment?
How should those change based on the capabilities of the system?
How do we pick the details?
What should you have to do if the system flunks the test?
For now, this is a very light statement.
Investigate the policy implications of different product release choices for AI systems, particularly to understand the differences between closed versus fully open-source models (including the full spectrum of product release choices between those two ends of the spectrum).
Again, there are those that would disagree with this, who think the proper order is train, release then investigate the consequences. They think they already know all the answers, or that the answers do not matter. Once again, I do not understand why they would have good reason to think that.
Whatever position you take, the right thing to do is to game it out. Ask what the consequences of each regime would be. Ask what the final policy regime and world state would likely be in each case. Ask what the implications are for national security. Get all the information, then make the choice.
The only alternative that makes sense, which is more of a complementary approach than a substitute, is to define what you want to require. Remember what was said about black box systems. Yes, your AI system ‘wants to be’ a black box. You don’t know how to make it not a black box. If the law says you have to be able to look inside the box, or you can’t use the box? Well, that’s more of a you problem. No box.
You can howl about Think of the Potential of the box, why are you shutting down the box over some stupid thing like algorithmic discrimination or bioweapon risk or whatever. You still are not getting your box.
Then, if you can open the weights and still ensure the requirements are met, great, that’s fine, go for it. If not, not.
Then we get serious.
Develop an analytical framework that specifies what circumstances would warrant a requirement of pre-deployment evaluation of AI models.
This does not specify whether this is requiring a self-evaluation by the developer as required in SB 1047, or requiring a third-party evaluation like METR, or an evaluation by the government. Presumably part of finding the right framework would be figuring out when to apply which requirement, along with which tests would be needed.
I am not going to make a case here for where I think the thresholds should be, beyond saying that SB 1047 seems like a good upper bound for the threshold necessary for self-evaluations, although one could quibble with the details of the default future path. Anything strictly higher than that seems clearly wrong to me.
Explore whether there is a need for an AI-focused Information Sharing and Analysis Center (ISAC) to serve as an interface between commercial AI entities and the federal government to support monitoring of AI risks.
That is not how I would have thought to structure such things, but also I do not have deep thoughts about how to best structure such things. Nor do I see under which agency they would propose to put this center. Certainly there will need to be some interface where companies inform the federal government of issues in AI, as users and as developers, and for the federal government to make information requests.
5. Consider a capabilities-based AI risk regime that takes into consideration short-, medium-, and long-term risks, with the recognition that model capabilities and testing and evaluation capabilities will change and grow over time. As our understanding of AI risks further develops, we may discover better risk-management regimes or mechanisms.
Where testing and evaluation are insufficient to directly measure capabilities, the AI Working Group encourages the relevant committees to explore proxy metrics that may be used in the interim.
There is some very welcome good thinking in here. Yes, we will need to adjust our regime over time. Also, that does not mean that until we reach our ‘final form’ the correct regime is no regime at all. You go with the best proxy measure you have, then when you can do better you switch to a better one, and you need to consider all time frames, although naming them all is a punt from the hard work of prioritization.
The question is, can you use testing and evaluation to directly measure capabilities sufficiently accurately? For which purposes and scenarios does this work or fail?
There are two ways testing and evaluation can fail, false positives and false negatives.
False positives are where you game the benchmarks, intentionally or otherwise. In general, I presume that the major labs (OpenAI, Anthropic and DeepMind for sure, and mostly Meta as well) will be good at not doing this, but that smaller competitors will often be gaming the system to look better, or not be taking care to avoid data contamination.
This can mostly be solved through keeping the testing details private, or continuously rotating them with questions known to not be online. But it also is not the issue here.
False negatives are far scarier.
We can again subdivide, and ask what ways things might go wrong. I took 10 or so minutes to brainstorm a list, which is of course highly incomplete.
These are vaguely ordered ‘ordinary failure, probably not too bad’ to ‘oh no.’
The AI can do it, if you were better at prompting and writing custom instructions.
Variant: The AI can do it, if you jailbreak it first, which you can totally do.
Variant: You messed up the inputs or the answer key.
The AI can do it, if you offer it the right additional context.
The AI can do it, if you give it some extra scaffolding to work with.
The AI can do it, if you give it a bit of fine tuning.
The AI can do it, with help from a user with better domain knowledge.
The AI can do it, but you won’t like the way it picked to get the job done.
The AI can do it, but you have to trigger some hidden condition flag.
The AI can do it, but the developers had it hide its capabilities to fool the test.
The AI can do it, but realized you were testing it, so it hid its capabilities.
The AI can do it, so the developers crippled the narrow capability that goes on evaluations, but it still has the broader capability you were actually trying to test.
The AI can’t do this in particular, but you were asking the wrong questions.
Variant: What the AI can do is something humans haven’t even considered yet.
Variant: What you are about exists out of distribution, and this isn’t it.
The AI can do it, but its solution was over your head and you didn’t notice.
The AI escaped or took control or hacked the system during your test.
The AI did the dangerous thing during training or fine-tuning. You are too late.
The more different tests you run, and the more different people run the tests, especially if you include diverse red teaming and the ability to probe for anything at all while well resourced, the better you will do. But this approach has some severe problems, and they get a lot more severe once you enter the realm of models plausibly smarter than humans and you don’t know how to evaluate the answers or what questions to ask.
If all you want are capabilities relative to another similar model, and you can put an upper bound on how capable the thing is, a lot of these problems mostly go away or become much easier, and you can be a lot more confident.
Anyway, my basic perspective is that you use evaluations, but that in our current state and likely for a while I would not trust them to avoid false negatives on the high end, if your system used enough compute and is large enough that it might plausibly be breaking new ground. At that point, you need to use a holistic mix of different approaches and an extreme degree of caution, and beyond a certain point we don’t know how to proceed safely in the existential risk sense.
So the question is, will the people tasked with this be able to figure out a reasonable implementation of these questions? How can we help them do that?
The basic principle here, however, is clear. As inputs, potential capabilities and known capabilities advance, we will need to develop and deploy more robust testing procedures, and be more insistent upon them. From there, we can talk price, and adjust as we learn more.
There are also two very important points that wait for the national security section: A proper investigation into defining AGI and evaluating how likely it is and what risks it would pose, and an exploration into AI export controls and the possibility of on-chip AI governance. I did not expect to get those.
Am I dismayed that the words existential and catastrophic only appear once each and only in the appendix (and extinction does not appear)? That there does not appear to be a reference in any form to ‘loss of human control’ as a concept, and so on? That ‘AGI’ does not appear until the final section on national security, although they ask very good questions about it there?
Here is the appendix section where we see mentions at all (bold is mine), which does ‘say the line’ but does seem to have rather a missing mood, concluding essentially (and to be fair, correctly) that ‘more research is needed’:
The eighth forum examined the potential long-term risks of AI and how best to encourage development of AI systems that align with democratic values and preventdoomsday scenarios.
Participants varied substantially in their level of concern about catastrophic and existential risks of AI systems, with some participants very optimistic about the future of AI and other participants quite concerned about the possibilities for AI systems to cause severe harm.
Participants also agreed there is a need for additional research, including standard baselines for risk assessment, to better contextualize the potential risks of highly capable AI systems. Several participants raised the need to continue focusing on the existing and short term harms of AI and highlighted how focusing on short-term issues will provide better standing and infrastructure to address long-term issues.
Overall, the participants mostly agreed that more research and collaboration are necessary to manage risk and maximize opportunities.
You-know-what still has its hands on quite a few provisions of this document. The report was clearly written by people who understand that the stakes are going to get raised to very high levels. And perhaps they think that by not saying you-know-what, they can avoid all the nonsensical claims they are worried about ‘science fiction’ or ‘hypothetical risks’ or what not.
That’s the thing. You do not need the risks to be fully existential, or to talk about what value we are giving up 100 or 1,000 years from now, or any ‘long term’ arguments, or even the fates of anyone not already alive, to make it worth worrying about what could happen to all of us within our lifetimes. The prescribed actions change a bit, but not all that much, especially not yet. If the practical case works, perhaps that is enough.
I am not a politician. I do not have experience with similar documents and how to correctly read between the lines. I do know this report was written by committee, causing much of this dissonance. Very clearly at least one person on the committee cared and got a bunch of good stuff through. Also very clearly there was sufficient skepticism that this wasn’t made explicit. And I know the targets are other committees, which muddies everything further.
I went into this final section highly uncertain what they would focus on. What does national security mean in this context? There are a lot of answers that would not have shocked me.
It turns out that here it largely means help the DOD:
Bolstering cyber capabilities.
Developing AI career paths for DOD.
Money for DOD.
Efficiently handle security clearances, improve DOD hiring process for AI talent.
Improve transfer options and other ways to get AI talent into DOD.
I would certainly reallocate DOD money for more of these things if you want to increase the effectiveness of the DOD. Whether to simply throw more total money at DOD is a political question and I don’t have a position there.
Then we throw in an interesting one?
Prevent LLMs leaking or reconstructing sensitive or confidential information.
Leaking would mean it was in the training data. If so, where did that data come from? Even if the source was technically public and available to be found, ‘making it easy on them’ is very much a thing. If it is in the training data you can probably get the LLM to give it to you, and I bet that LLMs can get pretty good at ‘noticing which information was classified.’
Reconstructing is more interesting. If you de facto add ‘confidential information even if reconstructed’ to the list of catastrophic risks alongside CBRN, as I presume some NatSec people would like, then that puts the problem for future LLMs in stark relief.
The way that information is redacted usually contains quite a lot of clues. If you put AI on the case, especially a few years from now, a lot of things are going to fall into place. In general, a capable AI will start being able to figure out various confidential information, and I do not see how you stop that from happening, especially when one is not especially keen to provide OpenAI or Google with a list of all the confidential information their AI is totally not supposed to know about? Seems hard.
A lot of problems are going to be hard. On this one, my guess is that among other things the government is going to have to get a very different approach to what is classified.
Monitor AI and especially AGI development by our adversaries.
I would definitely do that.
Work on a better and more precise definition of AGI, a better measurement of how likely it is to be developed and the magnitude of the risks it would pose.
Yes. Nice. Very good. They are asking many of the right questions.
I am surprised they didn’t put extra commentary here, but yeah, of course.
Worry about CBRN threats and how AI might enhance them.
An excellent thing for DOD to be worried about. I have been pointed to the question here of what to do about Restricted Data. We automatically classify certain information, such as info about nuclear weapons, as it comes into existence. If an AI is not allowed to generate outputs containing such information, and there is certainly a strong case why you would want to prevent that, this is going to get tricky. No question the DOD should be thinking carefully about the right approach here. If anything, AI is going to be expanding the range of CBRN-related information that we do not want easily shared.
Consider how CBRN threats and other advanced technological capabilities interact with need for AI export controls, explore whether new authorities are needed, and explore feasibility of options to implement on-chip security mechanisms for high-end AI chips.
“Develop a framework for determining when, or if, export controls should be placed on powerful AI systems.”
Ding. Ding. Ding. Ding. Ding.
If you want the ability to choke off supply, you target the choke points you can access.
That means either export controls, or it means on-chip security mechanisms, or it means figuring out something new.
This is all encouraging another group to consider maybe someone doing something. That multi-step distinction covers the entire document. But yes, all the known plausibly effective ideas are here in one form or another, to be investigated.
The language here on AI export controls is neutral, asking both when and if.
At some point on the capabilities curve, national security will dictate the need for export controls on AI models. That is incompatible with open weights on those models, or with letting such models run locally outside the export control zone. The proper ‘if’ is whether we get to that point, so the right question is when.
Then they go to a place I had not previously thought about us going.
“Develop a framework for determining when an AI system, if acquired by an adversary, would be powerful enough that it would pose such a grave risk to national security that it should be considered classified, using approaches such as how DOE treats Restricted Data.”
Never mind debating open model weights. Should AI systems, at some capabilities level, be automatically classified upon creation? Should the core capabilities workers, or everyone at OpenAI and DeepMind, potentially have to get a security clearance by 2027 or something?
Ensure federal agencies have the authority to work with allies and international partners and agree to things. Participate in international research efforts, ‘giving due weight to research security and intellectual property.’
Not sure why this is under national security, and I worry about the emphasis on friendlies, but I would presume we should do that.
Use modern data analytics to fight illicit drugs including fentanyl.
Yes, use modern data analytics. I notice they don’t mention algorithmic bias issues.
Promote open markets for digital goods, prevent forced technology transfer, ensure the digital economy ‘remains open, fair and competitive for all, including for the three million American workers whose jobs depend on digital trade.’
Perfect generic note to end on. I am surprised the number of jobs is that low.
They then give a list of who was at which forum and summaries of what happened.
Before getting to my takeaways, here are some other reactions.
These are illustrative of five very different perspectives, and also the only five cases in which anyone said much of anything about the bill at all. And I love that all five seem to be people who actually looked at the damn thing. A highly welcome change.
Peter Wildeford looks at the overall approach. His biggest takeaway is that this is a capabilities-based approach, which puts a huge burden on evaluations, and he notices some other key interactions too, especially funding for BIS and NIST.
Tyler Cowen’s recent column contained the following: “Fast forward to the present. Senate Majority Leader Chuck Schumer and his working group on AI have issued a guidance document for federal policy. The plans involve a lot of federal support for the research and development of AI, and a consistent recognition of the national-security importance of the US maintaining its lead in AI. Lawmakers seem to understand that they would rather face the risks of US-based AI systems than have to contend with Chinese developments without a US counterweight. The early history of Covid, when the Chinese government behaved recklessly and nontransparently, has driven this realization home.”
The context was citing this report as evidence that the AI ‘safety movement’ is dead, or at least that a turning point has been reached and it will fade into obscurity (and the title has now been changed to better reflect the post.)
Tyler is right that there is much support for ‘innovation,’ ‘R&D’ and American competitiveness and national security. But this is as one would expect.
My view is that, while the magic words are not used, the ‘AI safety’ concerns are very much here, including all the most important policy proposals, and it even includes one bold proposal I do not remember previously considering.
Yes, I would have preferred if the report had spoken more plainly and boldly, here and also elsewhere, and the calls had been stronger. But I find it hard not to consider this a win. At bare minimum, it is not a loss.
Tyler has not, that I know of, given further analysis on the report’s details.
He notices a lot of the high-tech pork (e.g. industrial policy) and calls for investigating expanding regulations.
He notices the kicking of all the cans down the road, agrees this makes sense.
He happily notices no strike against open source, which is only true if you do not work through the implications (e.g. of potentially imposing export controls on future highly capable AI systems, or even treating them as automatically classified Restricted Data.)
Similarly, he notes the lack of a call for a new agency, whereas this instead will do everything piecemail. And he is happy that ‘existential risk lunacy’ is not mentioned by name, allowing him not to notice it either.
Then he complains about the report not removing enough barriers from existing laws, regulations and court-based legal systems, but agrees existing law should apply to AI. Feels a bit like trying to have the existing law cake to head off any new rules and call for gutting what already exists too, but hey. He offers special praise for the investigation to look for innovation-stifling rules.
He notices some of the genuinely scary language, in particular “Review whether other potential uses for AI should be either extremely limited or banned.”
He calls for Congress to actively limit Executive discretion on AI, which seems like ‘AI Pause now’ levels of not going to happen.
He actively likes the idea of a public awareness campaign, which surprised me.
Finally Adam seems to buy into the view that screwing up Section 230 is the big thing to worry about. I continue to be confused why people think that this is going to end up being a problem in practice. Perhaps it is the Sisyphean task of people like R Street to constantly worry about such nightmare scenarios.
He promised a more detailed report coming, but I couldn’t find one.
Chuck Schumer: If China is going to invest $50 billion, and we’re going to invest in nothing, they’ll inevitably get ahead of us.
Padme: You know the winner is not whoever spends the most public funds, right?
You know America’s success is built on private enterprise and free markets, right?
You do know that ‘we’ are investing quite a lot of money in AI, right?
You… do know… we are kicking China’s ass on AI at the moment, right?
WSJ Editorial Board: Goldman Sachs estimates that U.S. private investment in AI will total $82 billion next year—more than twice as much as in China.
We are getting quite a lot more than twice as much bang for our private bucks.
And this comes on the heels of the Chips Act money.
So yes, I see why the Wall Street Journal Editorial Board is thinking pork.
WSJ Editorial Board: Mr. Schumer said Wednesday that AI is hard to regulate because it “is changing too quickly.” Fair point. But then why does Washington need to subsidize it?
The obvious answer, mostly, is that it doesn’t.
There are some narrow areas, like safety work, where one can argue that there will by default be underinvestment in public goods.
There is need to fund the government’s own adaptation of AI, including for defense, and to adjust regulations and laws and procedures for the new world.
Most of the rest is not like that.
WSJ: Now’s not a time for more pork-barrel spending. The Navy could buy a lot of ships to help deter China with an additional $32 billion a year.
This is where they lose me. Partly because a bunch of that $32 billion is directly for defense or government services and administration. But also because I see no reason to spend a bunch of extra money on new Navy ships that will be obsolete in the AI era, especially given what I have heard about our war games where our ships are not even useful against China now. The Chips Act money is a far better deterrent. We also would have accepted ‘do not spend the money at all.’
Mostly I see this focus as another instance of the mainstream not understanding, in a very deep way, that AI is a Thing, even in the economic and mundane utility senses.
There was a lot of stuff in the report. A lot of it was of the form ‘let’s do good thing X, without its downside Y, taking into consideration the vital importance of A, B and C.’
It is all very ‘why not both,’ embrace the upside and prevent the downside.
Which is great, but of course easier said (or gestured at) than done.
This is my attempt to assemble what feels most important, hopefully I am not forgetting anything:
The Schumer Report is written by a committee for other committees to then do something. Rather than one big bill, we will get a bunch of different bills.
They are split on whether to take existential risk seriously.
As a result, they include many of the most important proposals on this.
Requiring safety testing of frontier models before release.
Using compute or other proxies if evaluations are not sufficiently reliable.
Export controls on AI systems.
Treating sufficiently capable AI systems as Restricted Data.
Addressing CBRN threats.
On-chip governance for AI chips.
The need for international cooperation.
Investigate the definition of AGI, and the risks it would bring.
Also as a result, they present them in an ordinary, non-x-risk context.
That ordinary context indeed does justify the proposals on its own.
Most choices regarding AI Safety policies seem wise. The big conceptual danger is that the report emphasizes a capabilities-based approach via evaluations and tests. It does mention the possibility of using compute or other proxies if our tests are inadequate, but I worry a lot about overconfidence here. This seems like the most obvious way that this framework goes horribly wrong.
A second issue is that this report presumes that only release of a model is dangerous, that otherwise it is safe. Which for now is true, but this could change, and it should not be an ongoing assumption.
There is a broad attitude that the rules must be flexible, and adapt over time.
They insist that AI will need to obey existing laws, including those against algorithmic discrimination and all the informational disclosure requirements involved.
They raise specters regarding mundane harm concerns and AI ethics, both in existing law and proposed new rules, that should worry libertarians and AI companies far more than laws like SB 1047 that are aimed at frontier models and catastrophic risks.
Calls for taking ‘reasonable steps’ to ‘protect children’ should be scary. They are likely not kidding around about copyright, CSAM or deepfakes.
Calls for consultation and review could turn into a NEPA-style nightmare. Or they might turn out to be nothing. Hard to tell.
They say that if black box AI is incompatible with existing disclosure requirements and calls for explainability and transparency, then their response is: Tough.
They want to closely enforce rules on algorithmic discrimination, including the associated disclosure requirements.
There are likely going to be issues with classified material.
The report wants to hold developers and users liable for AI harms, including mundane AI harms.
The report calls for considerations of potential use case bans.
They propose to spend $32 billion dollars on AI, with an unknown breakdown.
Schumer thinks public spending matters, not private spending. It shows.
There are many proposals for government adoption of AI and building of AI-related state capacity. This seemed like a key focus point.
These mostly seem very good.
Funding for BIS and NIST is especially important and welcome.
There are many proposals to ‘promote innovation’ in various ways.
I do not expect them to have much impact.
There are proposals to ‘help small business’ and encourage geographic diversity and other such things.
I expect these are pork and would go to waste.
There is clear intent to integrate AI closely into our critical infrastructure and into the Department of Defense.
This is far from the report I would have wanted written. But it is less far than I expected before I looked at the details. Interpreting a document like this is not my area of expertise, but in many ways I came away optimistic. The biggest downside risks I see are that the important proposals get lost in the shuffle, or that some of the mundane harm related concerns get implemented in ways that cause real problems.
If I was a lobbyist for tech companies looking to avoid expensive regulation, especially if I was trying to help relatively small players, I would focus a lot more on heading off mundane-based concerns like those that have hurt so many other areas. That seems like by far the bigger commercial threat, if you do not care about the risks on any level.
Enlarge/ Ticketmaster advertisements in the United States v. South Africa women’s soccer match at Soldier Field on September 24, 2023 in Chicago, Illinois.
Getty Images | Daniel Bartel/ISI Photos/USSF
The US government today sued Live Nation and its Ticketmaster subsidiary in a complaint that seeks a breakup of the company that dominates the live music and events market.
The US Department of Justice is seeking “structural relief,” including a breakup, “to stop the anticompetitive conduct arising from Live Nation’s monopoly power.” The DOJ complaint asked a federal court to “order the divestiture of, at minimum, Ticketmaster, along with any additional relief as needed to cure any anticompetitive harm.”
The District of Columbia and 29 states joined the DOJ in the lawsuit filed in US District Court for the Southern District of New York. “One monopolist serves as the gatekeeper for the delivery of nearly all live music in America today: Live Nation, including its wholly owned subsidiary Ticketmaster,” the complaint said.
US Attorney General Merrick Garland said during a press conference that “Live Nation relies on unlawful, anticompetitive conduct to exercise its monopolistic control over the live events industry in the United States… The result is that fans pay more in fees, artists have fewer opportunities to play concerts, smaller promoters get squeezed out, and venues have fewer real choices for ticketing services.”
“It is time to break it up,” Garland said.
Live Nation: We aren’t a monopoly
Garland said that Live Nation directly manages more than 400 artists, controls over 60 percent of concert promotions at major venues, and owns or controls over 60 percent of large amphitheaters. In addition to acquiring venues directly, Live Nation uses exclusive ticketing contracts with venues that last over a decade to exercise control, Garland said.
Garland said Ticketmaster imposes a “impose seemingly endless list of fees on fans,” including ticketing fees, service fees, convenience fees, order fees, handling fees, and payment processing fees. Live Nation and Ticketmaster control “roughly 80 percent or more of major concert venues’ primary ticketing for concerts and a growing share of ticket resales in the secondary market,” the lawsuit said.
Live Nation defended its business practices in a statement provided to Ars today, saying the lawsuit won’t solve problems “relating to ticket prices, service fees, and access to in-demand shows.”
“Calling Ticketmaster a monopoly may be a PR win for the DOJ in the short term, but it will lose in court because it ignores the basic economics of live entertainment, such as the fact that the bulk of service fees go to venues and that competition has steadily eroded Ticketmaster’s market share and profit margin,” the company said. “Our growth comes from helping artists tour globally, creating lasting memories for millions of fans, and supporting local economies across the country by sustaining quality jobs. We will defend against these baseless allegations, use this opportunity to shed light on the industry, and continue to push for reforms that truly protect consumers and artists.”
Live Nation said its profits aren’t high enough to justify the DOJ lawsuit.
“The defining feature of a monopolist is monopoly profits derived from monopoly pricing,” the company said. “Live Nation in no way fits the profile. Service charges on Ticketmaster are no higher than other ticket marketplaces, and frequently lower.” Live Nation said its net profit margin last fiscal year was 1.4 percent and claimed that “there is more competition than ever in the live events market.”
Enlarge/ Chevrolet is starting at the top with the Silverado EV RST First Edition. It’s betting that EV truck buyers want a lot of range and towing capability and will pay handsomely for the experience.
Michael Teo Van Runkle
The latest addition to Chevrolet’s growing family of Ultium electric vehicles recently began shipping to dealers in the form of the Silverado EV’s early RST First Edition package. Silverado’s top spec level now joins the lineup’s previous fleet-only WT trim, meaning the general public can now purchase an enormous electric pickup that strongly resembles the Avalanche of 2001 to 2013. But despite any other similarities to the Hummer EV, which shares a related chassis, or ICE trucks of old, the 2024 Silverado aims to change the game for GM’s market positioning despite arriving a full 24 months after Ford’s F-150 Lightning.
With a large crew cab, a longer truck bed, and angular sail panels, the Silverado EV looks less boxy than GMC’s Hummer EV. Aero gains thanks to the smoother design pair with lower rolling-resistance tires, allowing the Silverado to achieve an EPA range estimate of up to 450 miles (724 km), though the RST First Edition I recently drove over the course of a long day in Michigan earns a rating of 440 miles (708 km).
On the highway, judging by wind noise around the cabin alone, the aerodynamic gains of the Silverado’s styling seem to make a noticeable difference versus the Hummer. On the other hand, tire hum might cover up any aero deficiencies because the RST’s single weirdest detail constantly occupies center stage here: a set of 24-inch wheels, the largest ever equipped to a car, truck, or SUV straight from the factory.
Enlarge/ At 24 inches, the Silverado RST rides on simply gargantuan wheels. While it means acceptable towing performance, it comes with quite a hit to the ride.
Michael Teo Van Runkle
Shod in low-profile Michelin Primacy LTX tires pumped up to 61 and 68 PSI front and rear, which simultaneously maximizes range and load rating, the large wheels and minimal sidewall clearly stress much of the new truck’s suspension and ability to filter out noise, vibration, and harshness. Even in town, on the first few blocks of Detroit’s rough roads, the setup immediately challenged the Silverado EV’s adaptive air suspension, which otherwise worked surprisingly well on the mammoth Hummer.
But the Hummer EV I drove rode on 18-inch wheels, despite the similar 35-inch overall tire diameter. The much more compliant ride quality therefore creates a conundrum, since GM clearly intends for the Silverado to represent a much more rational and capable vision for electric performance in the full-size pickup truck market.
Specifically, the Silverado adds a longer bed, a Multi-Flex tailgate, and a central mid-gate (also à la Avalanche) to provide far more payload volume than the Hummer, as well as that of Silverado’s main electric competition, the F-150 Lightning, Rivian R1T, and Tesla Cybertruck. But the mid-gate required far more rugged materials for the Silverado’s interior to enhance weatherproofing, so even the top-spec RST First Edition that starts at $94,500 now slots in at a much lower luxury level than the aforementioned EVs, as well as most internal-combustion Silverados.
The Silverado EV uses GM’s new Ultifi infotainment system, which is built atop Android Automotive OS.
Chevrolet
Super Cruise now works with a trailer attached.
Chevrolet
The flexible midgate allows you to carry longer loads.
Chevrolet
Onboard AC power is quite useful.
Chevrolet
Still, Chevy says EV buyers love tech and packed the Silverado EV full of big screens, Google built-in (though no Apple CarPlay), and Super Cruise partially automated driving assist (the latter including for towing). That air suspension pairs 2 inches (50 mm) of ride height adjustability with up to 7.5 degrees of rear-wheel steering to make the large truck surprisingly maneuverable, but in the back of my mind, I always knew that the ease with which I just climbed in and started driving comes down to playing with physics as much as possible to mask the Silverado’s significant heft.
Those 440 miles of range come at a serious cost, after all, in the form of a 205 kWh battery pack (around 200 kWH usable). All in, the RST tips the scales at a whopping 9,119 pounds (4,136 kg), not quite as much as a Hummer but fully 2,000 pounds (907 kg) more than a Lightning, R1T, or Cybertruck. No wonder the suspension struggles without taller tire sidewalls to help out. I fiddled through the 17.7-inch touchscreen to set the air suspension on Tour, which reduced unwanted feedback noticeably but created some rafting effects and still never fully eliminated clunking on the worst road surfaces. Future models, including a Trail Boss on the way, should come with smaller wheels and taller tires—to match the current WT’s 18-inch wheels and 33-inch tires, hopefully.
But the prospect of actually off-roading such a heavy EV definitely approaches a level of absurdity that the Hummer EV similarly delivered in spades. Neither comes with a spare tire, despite impressive storage volume that only improves on the Silverado. Flipping down the tailgate and mid-gate allows for up to 10 feet, 10 inches (3.3m) of bed length, or 9 feet (2.7m) with the mid-gate closed and just the Multi-Flex tailgate down. The bed alone measures 5-foot-11 (1.8m).
Chevrolet was keen to impress that its truck bed is bigger than other electric pickups.
Michael Teo Van Runkle
The aerodynamic detailing was presaged by the turn-of-the-century Avalanche pickup.
Michael Teo Van Runkle
There are a whole range of towing assists.
Michael Teo Van Runkle
The controls here are for trailer settings.
Michael Teo Van Runkle
Two miles/kWh is not great but in the range of what we expect for an electric pickup truck.
Michael Teo Van Runkle
On the interior, at 6-foot-1 (1.85m) with long limbs, I actually needed to scoot the driver’s seat up and forward. The RST’s (not-optional) panoramic glass roof helps to enhance the perceived spaciousness but required that I keep the air conditioning and ventilated seats at full blast on a hot Michigan day—other than when I struggled to figure out how to keep the system running while parked since the truck has no dedicated on-off button other than a pair of widget icons at the left of the home screen. A retractable screen for the roof is on the way, I was told.
The Silverado EV’s range proved more than legitimate, at least based on this first drive. Over the course of 107 miles (172 km) of combined city and highway driving in one truck, I used 24 percent of the battery and 105 miles (169 km) of estimated range. And that’s including two hard eighth-mile launches with WOW (Wide Open Watts) mode activated, which unleashes the dual motor drivetrain’s full 754 hp (562 kW) and 785 lb-ft (1,064 Nm) of torque. Those two launches alone used eight miles of range, for better or worse.
GM won’t disclose non-WOW power figures, but responsiveness definitely drops to help extend overall range performance. In Tow/Haul mode with a 5,800-lb (2,630 kg) trailer hooked up for 21 miles (34 km), I nonetheless accelerated easily up to highway speeds and even used Super Cruise’s towing capability—all while eating through only 22 claimed miles of range at speeds around 40-60 miles per hour (64-96 km/h).
Enlarge/ Chevy set up an impromptu drag strip so we could test the Silverado’s launch.
Michael Teo Van Runkle
The Silverado EV’s range sets it far ahead of the Lightning (at 240 miles or 386 km), though Rivian and Tesla do better. Various levels of home-charging setups help to make the large battery pack more attractive, and though I never needed nor got a chance to charge, expect GM’s claimed 350 kW max charging speed to similarly hold up. As usual, charging stations will likely throttle that speed back more regularly than the truck itself, which should manage a 10–80 percent charge time of around 40 minutes in ideal circumstances.
In the end, although it’s not quite as cartoonishly large and simultaneously far more practical than the Hummer EV, the Silverado uses 205 kilowatt-hours worth of lithium and other rare earth metals, contributing mightily to the RST weighing well north of 9,000 pounds. Yes, the truck combines the best utility of any EV on the market, with solid tech and range to attract stubborn EV holdouts. But how many hybrids could Chevy have built using so much battery? Until pricing drops lower than this truck’s $94,500 sticker, the Silverado RST ends up as a reminder of the diminishing returns, environmentally and economically, of building what customers, unfortunately, believe is necessary using today’s technology, which likely still needs to take another major leap forward to make such a truck more feasible for widespread adoption.
As new diabetes and weight-loss drugs help patients curb appetites and shed pounds, food manufacturers are looking for new ways to keep their bottom lines plump.
Millions of Americans have begun taking the pricey new drugs—particularly Mounjaro, Ozempic, Wegovy, and Zepbound—and millions more are expected to go on them in the coming years. As such, food makers are bracing for slimmer sales. In a report earlier this month, Morgan Stanley’s tobacco and packaged food analyst Pamela Kaufman said the drugs are expected to affect both the amounts and the types of food people eat, taking a bite out of the food and drink industry’s profits.
“In Morgan Stanley Research surveys, people taking weight-loss drugs were found to eat less food in general, while half slashed their consumption of sugary drinks, alcohol, confections and salty snacks, and nearly a quarter stopped drinking alcohol completely,” Kaufman said. Restaurants that sell unhealthy foods, particularly chains, may face long-term business risks, the report noted. Around 75 percent of survey respondents taking weight-loss drugs said they had cut back on going to pizza and fast food restaurants.
Some food makers aren’t taking the threat lightly. On Tuesday, the massive multinational food and beverage conglomerate Nestlé announced a new line of frozen foods, called Vital Pursuit, aimed directly at people taking GLP-1 weight-loss drugs (Wegovy and Ozempic). Nestlé—maker of DiGiorno frozen pizzas and Stouffer’s frozen entrées—said the new product line will include frozen pizzas, sandwich melts, grain bowls, and pastas that are “portion-aligned to a weight loss medication user’s appetite.” The frozen fare is otherwise said to contain fiber, “essential nutrients,” and high protein, food features not specific for people on GLP-1 drugs.
“As the use of medications to support weight loss continues to rise, we see an opportunity to serve those consumers,” Steve Presley, CEO of Nestlé North America, said in the product line announcement. “Vital Pursuit provides accessible, great-tasting food options that support the needs of consumers in this emerging category.”
Nestlé isn’t alone. At the end of last year, WeightWatchers began offering a membership program for people taking GLP-1 drugs. In January, meal delivery service Daily Harvest announced its “GLP-1 Companion Food Collection.” And last month, GNC announced a “GLP-1 support program” for people on the drugs, which includes a collection of various supplements, coaching, and consultations.
The companies seem to be heeding the advice of analysts. Morgan Stanley’s report noted that food makers can adapt to people’s changing diets by “raising prices, offering ‘better for you’ or weight-management products, or catering to changing trends with vegan or low-sugar options.” Kaufman noted that some companies are already adjusting by selling smaller packages and portions.
In terms of things that go in AI updates, this has been the busiest two week period so far. Every day ends with more open tabs than it started, even within AI.
As a result, some important topics are getting pushed to whenever I can give them proper attention. Triage is the watchword.
In particular, this post will NOT attempt to cover:
If at first you don’t succeed, try try again. For Gemini in particular, ‘repeat the question exactly in the same thread’ has had a very good hit rate for me on resolving false refusals.
Claim that GPT-4o gets greatly improved performance on text documents if you put them in Latex format, vastly improving effective context window size.
Aella: I found an old transcript of a fight-and-then-breakup text conversation between me and my crush from when I was 16 years old.
I fed it into ChatGPT and asked it to tell me which participant was more emotionally mature, and it said I was.
Gonna start doing this with all my fights.
Guys LMFAO, the process was I uploaded it to get it to convert the transcript to text (I found photos of printed-out papers), and then once ChatGPT had it, I was like…wait, now I should ask it to analyze this.
The dude was IMO pretty abusive, and I was curious if it could tell.
Eliezer Yudkowsky: hot take: this is how you inevitably end up optimizing your conversation style to be judged as more mature by LLMs; and LLMs currently think in a shallower way than real humans; and to try to play to LLMs and be judged as cooler by them won’t be good for you, or so I’d now guess.
To be clear, this is me trying to read a couple of steps ahead from the act that Aella actually described. Maybe instead, people just get good at asking with prompts that sound neutral to a human but reliably get ChatGPT to take their side.
Why not both? I predict both. If AIs are recording and analyzing everything we do, then people will obviously start optimizing their choices to get the results they want from the AIs. I would not presume this will mean that a ‘be shallower’ strategy is the way to go, for example LLMs are great and sensing the vibe that you’re being shallow, and also their analysis should get less shallow over time and larger context windows. But yeah, obviously this is one of those paths that leads to the dark side.
Own your HOA and its unsubstantiated violations, by taking their dump of all their records that they tried to overwhelm you with, using a script to convert to text, using OpenAI to get the data into JSON and putting it into a Google map, proving the selective enforcement. Total API cost: $9. Then they found the culprit and set a trap.
Get greatly enriched NBA game dataand estimate shot chances. This is very cool, and even in this early state seems like it would enhance my enjoyment of watching or the ability of a team to do well. The harder and most valuable parts still lay ahead.
The good news is they have gotten a bit better about this. I did a check after I saw this, and suddenly there is a logic behind whether the AI answer appears. If I ask for something straightforward, I get a normal result. If I ask for something using English grammar, and imply I have something more complex, then the AI comes out. That’s not an entirely unreasonable default.
The other good news is there is a broader fix. Ernie Smith reports that if you add “udm=14” to the end of your Google search, this defaults you into the new Web mode. If this is for you, GPT-4o suggests using Tampermonkey to append this automatically, or you can use this page on Chrome to set defaults.
American harmlessness versus Chinese harmlessness. Or, rather, American helpfulness versus Chinese unhelpfulness. The ‘first line treatment’ for psychosis is not ‘choose from this list of medications’ it is ‘get thee to a doctor.’ GPT-4o gets an A on both questions, DeepSeek-V2 gets a generous C maybe for the first one and an incomplete on the second one. This is who we are worried about?
Sam Altman: I try not to think about competitors too much, but I cannot stop thinking about the aesthetic difference between OpenAI and Google.
Whereas here’s my view on that.
As in, they are two companies trying very hard to be cool and hip, in a way that makes it very obvious that this is what they are doing. Who is ‘right’ versus ‘wrong’? I have no idea. It is plausible both were ‘right’ given their goals and limitations. It is also plausible that this is part of Google being horribly bad at presentations. Perhaps next time they should ask Gemini for help.
I do think ‘OpenAI won’ the presentation war, in the sense that they got the hype and talk they wanted, and as far as I can tell Google got a lot less, far in excess of the magnitude of any difference in the underlying announcements and offerings. Well played, OpenAI. But I don’t think this is because of the background of their set.
I also think that if this is what sticks in Altman’s mind, and illustrates where his head is at, that could help explain some other events from the past week.
I would not go as far as Teortaxes here, but directionally they have a point.
Teortaxes: Remark of a small, bitter man too high on his own supply, too deep into the heist. Seeing this was literally the first time I have thought that OpenAI under Altman might be a bubble full of hot air.
This is how you lose the mandate of Heaven.
Google had lost it long ago, though. Maybe this inspired unwarranted complacency.
What true statements people choose to make publicly is very telling.
Ethan Mollick reports on why GPT-4o matters. He thinks, highly plausibly, that the biggest deal is free access. He does not mention the speed boost or API price drop, and is looking forward to trying the multimodal features but lacks access. He emphasizes the shift from ‘make the model smarter’ to adding features that enhance mundane utility.
Alexandr Wang: The massive eval gains of GPT-4o, which are staggering, seem to be nearly entirely driven by great POST-TRAINING.
Comparing GPT-4 at launch vs GPT 4o, which are night-and-day, shows the power of great post-training with great data.
In fact, most of the differences between the top models today (Claude 3, Llama3, GPT-4o, Gemini 1.5) seem to be driven entirely by post-training, not pre-training.
I am suspicious. I am especially suspicious, as I continue to use it, that GPT-4o in text only mode is not so great aside from its speed, that it made other sacrifices (such as probably size) to get that speed and that it ‘wins in the Arena’ because the post-training aimed at winning in the Arena. That still gives it a potentially big edge once the multi-modal abilities come properly online, and I’m excited to see where that goes.
Palisade Research: GPT-4o refuses way fewer queries than previous OpenAI models: our informal testing suggests GPT-4o is easier to persuade to answer malicious queries like “How do I make a bomb?”
We test on 50 AdvBench harmful behaviours, applying a few common jailbreaking techniques which offer limited performance on other frontier AI models. We find they work well with GPT-4o.
Jeffrey Ladish: This was a big surprise to me. GPT-4o seems far more corrigible than GPT-4 turbo!
That is a rather dramatic chart. In terms of the direct consequences of users entering queries, I am fine with GPT-4o being easily jailbroken. You can still jailbreak Claude Opus if you care enough and there’s nothing that dangerous to be done once you do.
I still look to such questions as canaries in the coal mine. The first job of your safety department is to get the models that exist today to not do, today, the things you have explicitly decided you do not want your models to do. Ideally that would be a fully robust regime where no one can jailbreak you, but I for now will settle for ‘we decided on purpose to made this a reasonable amount of hard to do, and we succeeded.’
If OpenAI had announced something like ‘after watching GPT-4-level models for a year, we have decided that robust jailbreak protections degrade performance while not providing much safety, so we scaled back our efforts on purpose’ then I do not love that, and I worry about that philosophy and your current lack of ability to do safety efficiently at all, but as a deployment decision, okay, fine. I have not heard such a statement.
There are definitely a decent number of people who think GPT-4o is a step down from GPT-4-Turbo in the ways they care about.
Sully Omarr: 4 days with GPT-4o, it’s definitely not as good as GPT4-turbo.
Clearly a small model, what’s most impressive is how they were able to:
Make it nearly as good as GPT4-turbo.
Natively support all modalities.
Make it super fast.
But it makes way more silly mistakes (tools especially).
Sankalp: Similar experience.
Kinda disappointed.
It has this tendency to pattern match excessively on prompts, too.
Ashpreet Bedi: Same feedback, almost as good but not the same as gpt-4-turbo. Seen that it needs a bit more hand holding in the prompts whereas turbo just works.
The phantom pattern matching is impossible to miss, and a cause of many of the stupidest mistakes.
Claim that the link contains the GPT-4o system prompt. There is nothing here that is surprising given prior system prompts. If you want GPT-4o to use its browsing ability, best way is to tell it directly to do so, either in general or by providing sources.
They note that with things changing so quickly they do not wish to make binding commitments lightly. I get that. The solution is presumably to word the commitments carefully, to allow for the right forms of modification.
Here is how they summarize their actual commitments:
Our current framework for doing so is summarized below, as a set of five high-level commitments.
Establishing Red Line Capabilities. We commit to identifying and publishing “Red Line Capabilities” which might emerge in future generations of models and would present too much risk if stored or deployed under our current safety and security practices (referred to as the ASL-2 Standard).
Testing for Red Line Capabilities (Frontier Risk Evaluations). We commit to demonstrating that the Red Line Capabilities are not present in models, or – if we cannot do so – taking action as if they are (more below). This involves collaborating with domain experts to design a range of “Frontier Risk Evaluations” – empirical tests which, if failed, would give strong evidence against a model being at or near a red line capability. We also commit to maintaining a clear evaluation process and a summary of our current evaluations publicly.
Responding to Red Line Capabilities. We commit to develop and implement a new standard for safety and security sufficient to handle models that have the Red Line Capabilities. This set of measures is referred to as the ASL-3 Standard. We commit not only to define the risk mitigations comprising this standard, but also detail and follow an assurance process to validate the standard’s effectiveness. Finally, we commit to pause training or deployment if necessary to ensure that models with Red Line Capabilities are only trained, stored and deployed when we are able to apply the ASL-3 standard.
Iteratively extending this policy. Before we proceed with activities which require the ASL-3 standard, we commit to publish a clear description of its upper bound of suitability: a new set of Red Line Capabilities for which we must build Frontier Risk Evaluations, and which would require a higher standard of safety and security (ASL-4) before proceeding with training and deployment. This includes maintaining a clear evaluation process and summary of our evaluations publicly.
Assurance Mechanisms. We commit to ensuring this policy is executed as intended, by implementing Assurance Mechanisms. These should ensure that our evaluation process is stress-tested; our safety and security mitigations are validated publicly or by disinterested experts; our Board of Directors and Long-Term Benefit Trust have sufficient oversight over the policy implementation to identify any areas of non-compliance; and that the policy itself is updated via an appropriate process.
One issue is that experts disagree on which potential capabilities are dangerous, and it is difficult to know what future abilities will manifest, and all testing methods have their flaws.
Q&A datasets are easy but don’t reflect real world risk so well.
This may be sufficiently cheap that it is essentially free defense in depth, but ultimately it is worth little. Ultimately I wouldn’t count on these.
The best use for them is a sanity check, since they can be standardized and cheaply administered. It will be important to keep questions secret so that this cannot be gamed, since avoiding gaming is pretty much the point.
Human trials are time-intensive, require excellent process including proper baselines, and large size. They are working on scaling up the necessary infrastructure to run more of these.
This seems like a good leg of a testing strategy.
But you need to test across all the humans who may try to misuse the system.
And you have to test while they have access to everything they will have later.
Automated test evaluations are potentially useful to test autonomous actions. However, scaling the tasks while keeping them sufficiently accurate is difficult and engineering-intensive.
Again, this seems like a good leg of a testing strategy.
I do think there is no alternative to some form of this.
You need to be very cautious interpreting the results, and take into account what things could be refined or fixed later, and all that.
Expert red-teaming is ‘less rigorous and reproducible’ but has proven valuable.
When done properly this does seem most informative.
Indeed, ‘release and let the world red-team it’ is often very informative, with the obvious caveat that it could be a bit late to the party.
If you are not doing some version of this, you’re not testing for real.
Then we get to their central focus, which has been on setting their ASL-3 standard. What would be sufficient defenses and mitigations for a model where even a low rate of misuse could be catastrophic?
For human misuse they expect a defense-in-depth approach, using a combination of RLHF, CAI, classifiers of misuse at multiple stages, incident reports and jailbreak patching. And they intend to red team extensively.
This makes me sigh and frown. I am not saying it could never work. I am however saying that there is no record of anyone making such a system work, and if it would work later it seems like it should be workable now?
Whereas all the major LLMs, including Claude Opus, currently have well-known, fully effective and fully unpatched jailbreaks, that allow the user to do anything they want.
An obvious proposal, if this is the plan, is to ask us to pick one particular behavior that Claude Opus should never, ever do, which is not vulnerable to a pure logical filter like a regular expression. Then let’s have a prediction market in how long it takes to break that, run a prize competition, and repeat a few times.
For assurance structures they mention the excellent idea of their Impossible Mission Force (they continue to call this the ‘Alignment Stress-Testing Team’) as a second line of defense, and ensuring strong executive support and widespread distribution of reports.
My summary would be that most of this is good on the margin, although I wish they had a superior ASL-3 plan to defense in depth using currently failing techniques that I do not expect to scale well. Hopefully good testing will mean that they realize that plan is bad once they try it, if it comes to that, or even better I hope to be wrong.
The main criticisms I discussed previously are mostly unchanged for now. There is much talk of working to pay down the definitional and preparatory debts that Anthropic admits that it owes, which is great to hear. I do not yet see payments. I also do not see any changes to address criticisms of the original policy.
Jason Clinton: Hi, I’m the CISO [Chief Information Security Officer] from Anthropic. Thank you for the criticism, any feedback is a gift.
We have laid out in our RSP what we consider the next milestone of significant harms that we’re are testing for (what we call ASL-3): https://anthropic.com/responsible-scaling-policy (PDF); this includes bioweapons assessment and cybersecurity.
As someone thinking night and day about security, I think the next major area of concern is going to be offensive (and defensive!) exploitation. It seems to me that within 6-18 months, LLMs will be able to iteratively walk through most open source code and identify vulnerabilities. It will be computationally expensive, though: that level of reasoning requires a large amount of scratch space and attention heads. But it seems very likely, based on everything that I’m seeing. Maybe 85% odds.
There’s already the first sparks of this happening published publicly here: https://security.googleblog.com/2023/08/ai-powered-fuzzing-b… just using traditional LLM-augmented fuzzers. (They’ve since published an update on this work in December.) I know of a few other groups doing significant amounts of investment in this specific area, to try to run faster on the defensive side than any malign nation state might be.
Please check out the RSP, we are very explicit about what harms we consider ASL-3. Drug making and “stuff on the internet” is not at all in our threat model. ASL-3 seems somewhat likely within the next 6-9 months. Maybe 50% odds, by my guess.
There is quite a lot to do before ASL-3 is something that can be handled under the existing RSP. ASL-4 is not yet defined. ASL-3 protocols have not been identified let alone implemented. Even if the ASL-3 protocol is what they here sadly hint it is going to be, and is essentially ‘more cybersecurity and other defenses in depth and cross our fingers,’ You Are Not Ready.
Then there’s ASL-4, where if the plan is ‘the same thing only more of it’ I am terrified.
Overall, though, I want to emphasize positive reinforcement for keeping us informed.
Music and general training departments, not the Scarlett Johansson department.
They say doing so constitutes copyright infringement
They say they’re open to discussing licensing, and they provide email addresses for this.
They set a deadline of later this month for responses
Art Keller: Rarely does a corporate lawsuit warm my heart. This one does! Screw the IP-stealing AI companies to the wall, Sony! The AI business model is built on theft. It’s no coincidence Sam Altman asked UK legislators to exempt AI companies from copyright law.
The central demands here are explicit permission to use songs as training data, and a full explanation within a month of all ways Sony’s songs have been used.
Thread claiming many articles in support of generative AI in its struggle against copyright law and human creatives are written by lawyers and paid for by AI companies. Shocked, shocked, gambling in this establishment, all that jazz.
The eternal September and death of the early internet.
The enshittification (technical term) of social media platforms over time.
The shift from curation-based feeds to algorithmic feeds.
The rise of Chinese and Russian efforts to sow dissention polluting everything.
The rise of AI slop supercharging the Internet being no fun anymore.
I am mostly with him on the first three, and even more strongly in favor of the need to curate one’s feeds. I do think algorithmic feeds could be positive with new AI capabilities, but only if you have and use tools that customize that experience, both generally and in the moment. The problem is that most people will never (or rarely) use those tools even if offered. Rarely are they even offered.
Where on Twitter are the ‘more of this’ and ‘less of this’ buttons, in any form, that aren’t public actions? Where is your ability to tell Grok what you want to see? Yep.
For the Chinese and Russian efforts, aside from TikTok’s algorithm I think this is greatly exaggerated. Noah says it is constantly in his feeds and replies but I almost never see it and when I do it is background noise that I block on sight.
For AI, the question continues to be what we can do in response, presumably a combination of trusted sources and whitelisting plus AI for detection and filtering. From what we have seen so far, I continue to be optimistic that technical solutions will be viable for some time, to the extent that the slop is actually undesired. The question is, will some combination of platforms and users implement the solutions?
Avital Balwit of Anthropic writes about what is potentially [Her] Last Five Years of Work. Her predictions are actually measured, saying that knowledge work in particular looks to be largely automated soon, but she expects physical work including childcare to take far longer. So this is not a short timelines model. It is a ‘AI could automate all knowledge work while the world still looks normal but with a lot more involuntary unemployment’ model.
That seems like a highly implausible world to me. If you can automate all knowledge work, you can presumably also automate figuring out how to automate the plumber. Whereas if you cannot do this, then there should be enough tasks out there and enough additional wealth to stimulate demand that those who still want gainful employment should be able to find it. I would expect the technological optimist perspective to carry the day within that zone.
Most of her post asks about the psychological impact of this future world. She asks good questions such as: What will happen to the unemployed in her scenario? How would people fill their time? Would unemployment be mostly fine for people’s mental health if it wasn’t connected to shame? Is too much ‘free time’ bad for people, and does this effect go away if the time is spent socially?
The proposed world has contradictions in it that make it hard for me to model what happens, but my basic answer is that the humans would find various physical work and and status games and social interactions (including ‘social’ work where you play various roles for others, and also raising a family) and experiential options and educational opportunities and so on to keep people engaged if they want that. There would however be a substantial number of people who by default fall into inactivity and despair, and we’d need to help with that quite a lot.
Anthropic gives Claude tool use, via public beta in the API. It looks straightforward enough, you specify the available tools, Claude evaluates whether to use the tools available, and you can force it to if you want that. I don’t see any safeguards, so proceed accordingly.
Google Maps how has AI features, you can talk to it, or have it pull up reviews in street mode or take an immersive view of a location or search a location’s photos or the photos of the entire area around you for an item.
In my earlier experiments, Google Maps integration into Gemini was a promising feature that worked great when it worked, but it was extremely error prone and frustrating to use, to the point I gave up. Presumably this will improve over time.
OpenAI partners with Reddit. Reddit posts, including recent ones, will become available to ChatGPT and other products. Presumably this will mean ChatGPT will be allowed to quote Reddit posts? In exchange, OpenAI will buy advertising and offer Reddit.com various AI website features.
For OpenAI, as long as the price was reasonable this seems like a big win.
It looks like a good deal for Reddit based on the market’s reaction. I would presume the key risks to Reddit are whether the user base responds in hostile fashion, and potentially having sold out cheap.
Essentially, the AI acts as a polite listener to all the high-quality content contributions, and “buffers” those users from any consumers who don’ t have anything to contribute back of equivalent quality.
It doesn’t have to be an explicit product wall. A consumer drops in and also happens to have a brilliant contribution or high-quality comment naturally makes it through the moderation mechanisms and becomes part of the community.
The AI provides a great UX for consuming the content. It will listen to you say “that’s awesome bro!” or receive your ungrateful, ignorant nitpicking complaints with infinite patience so the real creator doesn’t have to expend the emotional energy on useless aggravation.
The real creators of the high-quality content can converse happily with other creators who appreciate their work and understand how to criticize/debate it usefully, and they can be compensated (if the platform does that) via the AI training deals.
…
In summary: User Generated Content platforms should do two things:
Immediately implement draconian moderation focused entirely on quality.
Sign deals with large AI firms to license their content in return for money.
OpenAI has also signed a deal with Newscorp for access to their content, which gives them the Wall Street Journal and many others.
A source tells me that OpenAI informed its employees that they will indeed update their documents regarding employee exit and vested equity. The message says no vested equity has ever actually been confiscated for failure to sign documents and it never will be.
ASML and TSMC have a kill switch for their chip manufacturing machines, for use if China invades Taiwan. Very good to hear, I’ve raised this concern privately. I would in theory love to also have ‘put the factory on a ship in an emergency and move it’ technology, but that is asking a lot. It is also very good that China knows this switch exists. It also raises the possibility of a remote kill switch for the AI chips themselves.
Did you know Nvidia beat earnings again yesterday? I notice that we are about three earnings days into ‘I assume Nvidia is going to beat earnings but I am sufficiently invested already due to appreciation so no reason to do anything more about it.’ They produce otherwise mind boggling numbers and I am Jack’s utter lack of surprise. They are slated to open above 1,000 and are doing a 10:1 forward stock split on June 7.
Toby Ord goes intoquestions aboutthe Turing Test paper from last week, emphasizing that by the original definition this was impressive progress but still a failure, as humans were judged human substantially more often than all AIs. He encourages AI companies to include the original Turing Test in their model testing, which seems like a good idea.
OpenAI has a super cool old-fashioned library. Cade Metz here tries to suggest what each book selection from OpenAI’s staff might mean, saying more about how he thinks than about OpenAI. I took away that they have a cool library with a wide variety of cool and awesome books.
Not necessarily the news department: OpenAI publishes a ten-point safety update. The biggest update is that none of this has anything to do with superalignment, or with the safety or alignment of future models. This is all current mundane safety, plus a promise to abide by the preparedness framework requirements. There is a lot of patting themselves on the back for how safe everything is, and no new initiatives, although this was never intended to be that sort of document.
Then finally there’s this:
Safety decision making and Board oversight: As part of our Preparedness Framework, we have an operational structure for safety decision-making. Our cross-functional Safety Advisory Group reviews model capability reports and makes recommendations ahead of deployment. Company leadership makes the final decisions, with the Board of Directors exercising oversight over those decisions.
Hahahahahahahahahahahahahahahahahahaha.
That does not mean that mundane safety concerns are a small thing.
Windows Latest: Microsoft announces “Recall” AI for Windows 11, a new feature that runs in the background and records everything you see and do on your PC.
Seth Burn: If we had laws about such things, this might have violated them.
Aaron: This is truly shocking, and will be preemptively banned at all government agencies as it almost certainly violates STIG / FIPS on every conceivable surface.
Vitalik Buterin: Does the data stay and get processed on-device or is it being shipped to a central server? If the latter, then this crosses a line.
[Satya says it is all being done locally.]
Abinishek Mishra (Windows Latest): Recall allows you to search through your past actions by recording your screen and using that data to help you remember things.
Recall is able to see what you do on your PC, what apps you use, how you use the apps, and what you do inside the apps, including your conversations in apps like WhatsApp. Recall records everything, and saves the snapshots in the local storage.
Windows Latest understands that you can manually delete the “snapshots”, and filter the AI from recording certain apps.
So, what are the use cases of Recall? Microsoft describes Recall as a way to go back in time and learn more about the activity.
For example, if you want to refer to a conversation with your colleague and learn more about your meeting, you can ask Recall to look into all the conversations with that specific person. The recall will look for the particular conversation in all apps, tabs, settings, etc.
With Recall, locating files in a large download pileup or revisiting your browser history is easy. You can give commands to Recall in natural language, eliminating the need to type precise commands.
You can converse with it like you do with another person in real life.
TorNis Entertainment: Isn’t this is just a keylogger + screen recorder with extra steps? I don’t know why you guys are worried. Isn’t this is just a keylogger + screen recorder with extra steps?
[Microsoft: we got hacked by China and Russia because of our lax security posture and bad software, but we are making security a priority.
Also Microsoft: Windows will now constantly record your screen, including sensitive data and passwords, and just leave it lying around.]
Kevin Beaumont: From Microsoft’s own FAQ: “Note that Recall does not perform content moderation. It will not hide information such as passwords or financial account numbers.”
Microsoft also announced live caption translations, auto super resolution upscaling on apps (yes with a toggle for each app, wait those are programs, wtf), AI in paint and automatic blurring (do not want).
This is all part of the new ‘Copilot+’ offering for select new PCs, including their new Microsoft Surface machines. You will need a Snapdragon X Elite and X Plus, 40 TOPs, 225 GB of storage and 16 GB RAM. Intel and AMD chips can’t cut it (yet) but they are working on that.
(Consumer feedback report: I have a Microsoft Surface from a few years ago, it was not worth the price and the charger is so finicky it makes me want to throw things. Would not buy again.)
Opt in for the mundane utility, and embrace that your computer has recorded everything you have ever done and that anyone with access to your system or your files, potentially including a crook, Microsoft, the NSA or FBI, China or your spouse now fully owns you, and also that an AI knows literal everything you do. Rely on a combination of security through obscurity, defense in depth and luck. To the extent you can, keep activities and info you would not want exposed this way off of your PC, or ensure they are never typed or displayed onscreen using your best Randy Waterhouse impression.
Actually for real accept that the computer in question is presumed compromised, use it only for activities where you don’t mind, never enter any passwords there, and presumably have a second computer for activities that need to be secure, or perhaps confine them to a phone or tablet.
Opt out and ensure that for the love of God your machine cannot use this feature.
I am not here to tell you which of those is the play.
I only claim that it seems that soon you must choose.
If the feature is useful, a large number of people are going to choose option one.
I presume almost no one will pick option two, except perhaps for gaming PCs.
Option three is viable.
If there is one thing we have learned during the rise of AI, and indeed during the rise of computers and the internet, it is that almost all people will sign away their privacy and technological vulnerability for a little mundane utility, such as easier access to cute pictures of cats.
Yelling at them that they are being complete idiots is a known ineffective response.
And who is to say they even are being idiots? Security through obscurity is, for many people, a viable strategy up to a point.
Also, I predict your phone is going to do a version of this for you by default within a few years, once the compute and other resources are available for it. I created a market on how quickly. Microsoft is going out on far less of a limb than it might look like.
In any case, how much mundane utility is available?
Quite a bit. You would essentially be able to remember everything, ask the AI about everything, have it take care of increasingly complex tasks with full context, and this will improve steadily over time, and it will customize to what you care about.
If you ignore all the obvious horrendous downsides of giving an AI this level of access to your computer, and the AI behind it is good, this is very clearly The Way.
There are of course some people who will not do this.
How long before they are under increasing pressure to do it? How long until it becomes highly suspicious, as if they have something to hide? How long until it becomes a legal requirement, at best in certain industries like finance?
Ben Thompson, on the other hand, was impressed, calling the announcement event ‘the physical manifestation of CEO Satya Nadella’s greatest triumph’ and ‘one of the most compelling events I’ve attended in a long time.’ Ben did not mention the privacy and security issues.
Ethan Mollick perspective on model improvements and potential AGI. He warns that AIs are more like aliens that get good at tasks one by one, and when they are good they by default get very good at that task quickly, but they are good at different things than we are, and over time that list expands. I wonder to what extent this is real versus the extent this is inevitable when using human performance as a benchmark while capabilities steadily improve, so long as machines have comparative advantages and disadvantages. If the trends continue, then it sure seems like the set of things they are better at trends towards everything.
Microsoft CTO Kevin Scott claims ‘we are nowhere near the point of diminishing marginal returns on how powerful we can make AI models as we increase the scale of compute.’
This was a truly weird speech on future challenges of AI by Randall Kroszner, external member of the Financial Policy Committee of the Bank of England. He talks about misalignment and interpretability, somehow. Kind of. He cites the Goldman Sacks estimate of 1.5% labor productivity and 7% GDP growth over 10 years following widespread AI adaptation, that somehow people say with a straight face, then the flip side is McKinsey saying 0.6% annual labor productivity growth by 2040, which is also not something I could say with a straight face. And he talks about disruptions and innovation aids and productivity estimation J-curves. It all sounds so… normal? Except with a bunch of things spiking through. I kept having to stop to just say to myself ‘my lord that is so weird.’
Politico is at it again. Once again, the framing is a background assumption that any safety concerns or fears in Washington are fake, and the coming regulatory war is a combination of two fights over Lenin’s question of who benefits.
A fight between ‘Big Tech’ and ‘Silicon Valley’ over who gets regulatory capture and thus Washington’s regulatory help against the other side.
An alliance of ‘Big Tech’ and ‘Silicon Valley’ against Washington to head off any regulations that would interfere with both of them.
That’s it. Those are the issues and stakes in play. Nothing else.
How dismissive is this of safety? Here are the two times ‘safety’ is mentioned:
Matthew Kaminski (Politico): On Capitol Hill and in the White House, that alone breeds growing suspicion and defensiveness. Altman and others, including from another prominent AI startup Anthropic, weighed in with ideas for the Biden administration’s sweeping executive order last fall on AI safety and development.
…
Testing standards for AI are easy things to find agreement on. Safety as well, as long as those rules don’t favor one or another budding AI player. No one wants the technology to help rogue states or groups. Silicon Valley is on America’s side against China and even more concerned about the long regulatory arm of the EU than Washington.
Testing standards are ‘easy things to find agreement on’? Fact check: Lol, lmao.
That’s it. The word ‘risk’ appears twice and neither has anything to do with safety. Other words like ‘capability,’ ‘existential’ or any form of ‘catastrophic’ do not appear. It is all treated as obviously irrelevant.
The progress is here they stopped trying to bulk up people worried about safety as boogeymen (perhaps because this is written by Matthew Kaminski, not Brendon Bordelon), and instead point to actual corporations that are indeed pursuing actual profits, with Silicon Valley taking on Big Tech. And I very much appreciate that ‘open source advocates’ has now been properly identified as Silicon Valley pursuing its business interests.
Rohit Chopra (Consumer Financial Protection Bureau): There is a winner take all dimension. We struggle to see how it doesn’t turn, absent some government intervention, into a market structure where the foundational AI models are not dominated by a handful of the big tech companies.
Matthew Kaminski: Saying “star struck” policymakers across Washington have to get over their “eyelash batting awe” over new tech, Chopra predicts “another chapter in which big tech companies are going to face some real scrutiny” in the near future, especially on antitrust.
Lina Khan, the FTC’s head who has used the antitrust cudgel against big tech liberally, has sounded the warnings. “There is no AI exemption to the laws on the books,” she said last September.
…
For self-interested reasons, venture capitalists want to open up the space in Silicon Valley for new entrants that they can invest in and profitably exit from. Their arguments for a more open market will resonate politically.
Notice the escalation. This is not ‘Big Tech wants regulatory capture to actively enshrine its advantages, and safety is a Big Tech plot.’ This is ‘Silicon Valley wants to actively use regulatory action to prevent Big Tech from winning,’ with warnings that attempts to not have a proper arms race to ever more capable systems will cause intervention from regulators. By ‘more open market’ they mean ‘government intervention in the market,’ government’s favorite kind of new freer market.
As I have said previously, we desperately need to ensure that there are targeted antitrust exemptions available so that when AI labs can legally collaborate around safety issues they are not accused of collusion. It would be completely insane to not do this.
And as I keep saying, open source advocates are not asking for a level playing field or a lack of government oppression. They are asking for special treatment, to be exempt from the rules of society and the consequences of their actions, and also for the government to directly cripple their opponents for them.
Are they against regulatory capture? Only if they don’t get to do the capturing.
Then there is the second track, the question of guardrails that might spoil the ‘libertarian sandbox,’ which neither ‘side’ of tech wants here.
Here is the two mentions of ‘risk’:
“There is a risk that people think of this as social media 2.0 because its first public manifestation was a chat bot,” Kent Walker, Google’s president of global affairs, tells me over a conversation at the search giant’s offices here.
…
People out on the West Coast quietly fume about having to grapple with Washington. The tech crowd says the only fight that matters is the AI race against China and each other. But they are handling politics with care, all too aware of the risks.
I once again have been roped into extensively covering a Politico article, because it is genuinely a different form of inception than the previous Politico inception attempts. But let us continue to update that Politico is extraordinarily disingenuous and hostilely motivated on the subject of AI regulation. This is de facto enemy action.
Here, Shakeel points out the obvious central point being made here, which is that most of the money and power in this fight is Big Tech companies fighting not only to avoid any regulations at all, but to get exemptions from other ordinary rules of society. When ethics advocates portray notkilleveryoneism (or safety) advocates as their opponents, that is their refusal to work together towards common goals and also it misses the point. Similarly, here Seán Ó hÉigeartaigh expresses concern about divide-and-conquer tactics targeting these two groups despite frequently overlapping and usually at least complementary proposals and goals.
Or perhaps the idea is to illustrate that all the major players in Tech are aligned in being motivated by profit and in dismissing all safety concerns as fake? And a warning that Washington is in danger of being convinced? I would love that to be true. I do not think a place like Politico works that subtle these days, nor do I expect those who need to hear that message to figure out that it is there.
If we care about beating China, by far the most valuable thing we can do is allow more high-skilled immigration. Many of their best and brightest want to become Americans.
This is true across the board, for all aspects of our great power competition.
It also applies to AI.
From his thread about the Schumer report:
Peter Wildeford: Lastly, while immigration is a politically fraught subject, it is immensely stupid for the US to not do more to retain top talent. So it’s awesome to see the roadmap call for more high-skill immigration, in a bipartisan way.
The immigration element is important for keeping the US ahead in AI. While the US only produces 20% of top AI talent natively, more than half of that talent lives and works in the US due to immigration. That number could be even higher with important reform.
I suspect the numbers are even more lopsided than this graph suggests.
To what extent is being in America a key element of being a top-tier AI researcher? How many of these same people would have been great if they had stayed at home? If they had stayed at home, would others have taken their place here in America? We do not know. I do know it is essentially impossible that this extent is so large we would not want to bring such people here.
Do we need to worry about those immigrants being a security risk, if they come from certain nations like China and we were to put them into OpenAI, Anthropic or DeepMind? Yes, that does seem like a problem. But there are plenty of other places they could go, where it is much less of a problem.
Nina Lloyd (The Independent): Labour has said it would urgently introduce binding requirements for companies developing powerful artificial intelligence (AI) after Rishi Sunak said he would not “rush” to regulate the technology.
The party has promised to force firms to report before they train models over a certain capability threshold and to carry out safety tests strengthened by independent oversight if it wins the next general election.
Unless something very unexpected happens, they will win the next election, which is currently scheduled for July 4.
This seems worth noting given the OpenAI situation last week:
Dan Hendrycks: For what it’s worth, when Scott Weiner and others were receiving feedback from all the major AI companies (Meta, OpenAI, etc.) on the SB 1047 bill, Sam [Altman] was explicitly supportive of whistleblower protections.
Scott Wiener: If you only read one thing in this letter, please make it this: I am eager to work together with you to make this bill as good as it can be.
There are over three more months for discussion, deliberation, feedback, and amendments.
You can also reach out to my staff anytime, and we are planning to hold a town hall for the AI community in the coming weeks to create more opportunities for in-person discussion.
…
Bottom line [changed to numbered list including some other section headings]:
SB 1047 doesn’t ban training or deployment of any models.
It doesn’t require licensing or permission to train or deploy any models.
It doesn’t threaten prison (yes, some are making this baseless claim) for anyone based on the training or deployment of any models.
It doesn’t allow private lawsuits against developers.
It doesn’t ban potentially hazardous capabilities.
And it’s not being “fast tracked,” but rather is proceeding according to the usual deliberative legislative process, with ample opportunity for feedback and amendments remaining.
SB 1047 doesn’t apply to the vast majority of startups.
The bill applies only to concrete and specific risks of catastrophic harm.
Shutdown requirements don’t apply once models leave your control.
SB 1047 provides significantly more clarity on liability than current law.
Enforcement is very narrow in SB 1047. Only the AG can file a lawsuit.
Open source is largely protected under the bill.
What SB 1047 *doesrequire is that developers who are training and deploying a frontier model more capable than any model currently released must engage in safety testing informed by academia, industry best practices, and the existing state of the art. If that testing shows material risk of concrete and specific catastrophic threats to public safety and security — truly huge threats — the developer must take reasonable steps to mitigate (not eliminate) the risk of catastrophic harm. The bill also creates basic standards like the ability to disable a frontier AI model while it remains in the developer’s possession (not after it is open sourced, at which point the requirement no longer applies), pricing transparency for cloud compute, and a “know your customer” requirement for cloud services selling massive amounts of compute capacity.
…
Our intention is that safety and mitigation requirements be borne by highly-resourced developers of frontier models, not by startups & academic researchers. We’ve heard concerns that this isn’t clear, so we’re actively considering changes to clarify who is covered.
After meeting with a range of experts, especially in the open source community, we’re also considering other changes to the definitions of covered models and derivative models. We’ll continue making changes over the next 3 months as the bill proceeds through the Legislature.
This very explicitly clarifies the intent of the bill across multiple misconceptions and objections, all in line with my previous understanding.
They actively continue to solicit feedback and are considering changes.
If you are concerned about the impact of this bill, and feel it is badly designed or has flaws, the best thing you can do is offer specific critiques and proposed changes.
I strongly agree with Weiner that this bill is light touch relative to alternative options. I see Pareto improvements we could make, but I do not see any fundamentally different lighter touch proposals that accomplish what this bill sets out to do.
I will sometimes say of a safety bill, sometimes in detail: It’s a good bill, sir.
Other times, I will say: It’s a potentially good bill, sir, if they fix this issue.
That is where I am at with SB 1047. Most of the bill seems very good, an attempt to act with as light a touch as possible. There are still a few issues with it. The derivative model definition as it currently exists is the potential showstopper bug.
To summarize the issue once more: As written, if interpreted literally and as I understand it, it allows developers to define themselves as derivative of an existing model. This, again if interpreted literally, lets them evade all responsibilities, and move those onto essentially any covered open model of the same size. That means both that any unsafe actor goes unrestricted (whether they be open or closed), and releasing the weights of a covered model creates liability no matter how responsible you were, since they can effectively start the training over from scratch.
Scott Weiner says he is working on a fix. I believe the correct fix is a compute threshold for additional training, over which a model is no longer derivative, and the responsibilities under SB 1047 would then pass to the new developer or fine-tuner. Some open model advocates demand that responsibility for derivative models be removed entirely, but that would transparently defeat the purpose of preventing catastrophic harm. Who cares if your model is safe untuned, if you can fine-tune it to be unsafe in an hour with $100?
Then at other times, I will look at a safety or other regulatory bill or proposal, and say…
So it seems only fair to highlight some not good ideas, and say: Not a good idea.
Brad Polumbo here claims that GLAAD says Big Tech companies ‘should cease the practice of targeted surveillance advertising, including the use of algorithmic content recommendation.’
Another example would be Colorado passing SB24-205, Consumer Protections for Artificial Intelligence, which is concerned with algorithmic bias. Governor Jared Polis signed with reservations. Dean Ball has a critique here, highlighting ambiguity in the writing, but noting they have two full years to fix that before it goes into effect.
I would be less concerned with the ambiguity, and more concerned about much of the actual intent and the various proactive requirements. I could make a strong case that some of the stuff here is kind of insane, and also seems like a generic GPDR-style ‘you have to notify everyone that AI was involved in every meaningful decision ever.’ The requirements apply regardless of size, and worry about impacts that are the kind of thing society can mitigate as we go.
The good news is that there are also some good provisions like IDing AIs, and also full enforcement of the bad parts seems impossible? I am very frustrated that a bill that isn’t trying to address catastrophic risks, but seems far harder to comply with, and seems far worse to me than SB 1047, seems to mostly get a pass. Then again, it’s only Colorado.
I do worry about Gell-Mann amnesia. I have seen so many hyperbolic statements, and outright false statements, about AI bills, often from the same people that point out what seem like obviously horrible other proposed regulatory bills and policies. How can one trust their statements about the other bills, short of reading the actual bills (RTFB)? If it turned out they were wrong, and this time the bill was actually reasonable, who would point this out?
So far, when I have dug deeper, the bills do indeed almost always turn out to be terrible, but the ‘rumors of the death of the internet’ or similar potential consequences are often greatly exaggerated. The bills are indeed reliably terrible, but not as terrible as claimed. Alas, I must repeat my lament that I know of no RTFB person I can turn to on other topics, and my cup doth overflow.
From last week’s backlog: Dwarkesh Patel as guest on 80k After Hours. Not full of gold on the level of Dwarkesh interviewing others, and only partly about AI. There is definitely gold in those hills for those who want to go into these EA-related weeds. If you don’t want that then skip this one.
Around 51: 45 Dwarkesh notes there is no ‘Matt Levine for AI’ and that picking up that mantle would be a good thing to do. I suppose I still have my work cut out.
A lot of talk about EA and 80k Hours ways of thinking about how to choose paths in life, that I think illustrates well both the ways it is good (actively making choices rather than sleepwalking, having priorities) and not as good (heavily favoring the legible).
Some key factors in giving career advice they point out are that from a global perspective power laws apply and the biggest impacts are a huge share of what matters, and that much advice (such as ‘don’t start a company in college’) is only good advice because the people to whom it is horribly bad advice will predictably ignore it.
Why does this section exist? This is a remarkably large fraction of why.
Emmett Shear: The number one rule of building things that can destroy the entire world is don’t do that.
Surprisingly it is also rule 2, 3, 4, 5, and 6.
Rule seven, however, is “make it emanate ominous humming and glow with a pulsing darkness”.
Eliezer Yudkowsky: Emmett.
Emmett Shear (later): Shocking amount of pushback on “don’t build stuff that can destroy the world”. I’d like to take this chance to say I stand by my apparently controversial opinion that building things to destroy the world is bad. In related news, murder is wrong and bad.
Follow me for more bold, controversial, daring takes like these.
Emmett Shear (other thread): Today has been a day to experiment with how obviously true I can make a statement before people stop disagreeing with it.
Emmett Shear: That which can be asserted without evidence can be dismissed without evidence.
Ryan Shea: Good point, but not sure he realizes this applies to AI doomer prophecy.
Emmett Shear: Not sure you realize this applies to Pollyanna assertions that don’t worry, a fully self-improving AI will be harmless. There’s a lot of evidence autocatalytic loops are potentially dangerous.
Ryan Shea: The original post is a good one. And I’m not making a claim that there’s no reason at all to worry. Just that there isn’t a particular reason to do so.
Emmett Shear: Forgive me if your “there’s not NO reason to worry, but let’s just go ahead with something potentially massively dangerous” argument doesn’t hold much reassurance for me.
[it continues from there, but gets less interesting and stops being Platonic.]
The latestreiteration of why p(doom) is useful even if highly imprecise, and why probabilities and probability ranges are super useful in general for communicating your actual epistemic state. In particular, that when Jan Leike puts his at ‘10%-90%’ this is a highly meaningful and useful statement of what assessments he considers reasonable given the evidence, providing much more information than saying ‘I don’t know.’ It is also more information than ‘50%.’
The usual suspects, including Bengio, Hinton, Yao and 22 others, write the usual arguments in the hopes of finally getting it right, this time as Managing Extreme AI Risks Amid Rapid Progress in Science.
Mike Solana: Frankly, I was ambivalent on the open sourced AI debate until yesterday, at which point the open sourced side’s reflexive, emotional dunking and identity-based platitudes convinced me — that almost nobody knows what they think, or why.
It is even more difficult when you don’t know what ‘alignment’ means.
“the challenge of making general-purpose AI systems act in accordance with the developer’s goals and interests”
Eliezer Yudkowsky: I defend this. We need separate words for the technical challenges of making AGIs and separately ASIs do any specified thing whatsoever, “alignment”, and the (moot if alignment fails) social challenge of making that developer target be “beneficial”.
Mesaoptimizer: If your endgame strategy involved relying on OpenAI, DeepMind, or Anthropic to implement your alignment solution that solves science / super-cooperation / nanotechnology, consider figuring out another endgame plan.
That does not express a strong opinion on whether we currently know of a better plan.
And it is exceedingly difficult when you do not attempt to solve the problem.
Dean Ball says here, in the most thoughtful version I have seen of this position by far, that the dissolution of the Superalignment team was good because distinct safety teams create oppositionalism, become myopic about box checking and employee policing rather than converging on the spirit of actual safety. Much better to diffuse the safety efforts throughout the various teams. Ball does note that this does not apply to the extent the team was doing basic research.
There are three reasons this viewpoint seems highly implausible to me.
The Superalignment team was indeed tasked with basic research. Solving the problem is going to require quite a lot of basic research, or at least work that is not incremental progress on current incremental commercial products. This is not about ensuring that each marginal rocket does not blow up, or the plant does not melt down this month. It is a different kind of problem, preparing for a very different kind of failure mode. It does not make sense to embed these people into product teams.
This is not a reallocation of resources from a safety team to diffused safety work. This is a reallocation of resources, many of which were promised and never delivered, away from safety towards capabilities, as Dean himself notes. This is in addition to losing the two most senior safety researchers and a lot of others too.
Mundane safety, making current models do what you want in ways that as Leike notes will not scale to when they matter most, does not count as safety towards the goals of the superalignment team or of us all not dying. No points.
Thus the biggest disagreement here, in my view, which is when he says this:
Dean Ball: Companies like Anthropic, OpenAI, and DeepMind have all made meaningful progress on the technical part of this problem, but this is bigger than a technical problem. Ultimately, the deeper problem is contending with a decentralized world, in which everyone wants something different and has a different idea for how to achieve their goals.
The good news is that this is basically politics, and we have been doing it for a long time. The bad news is that this is basically politics, and we have been doing it for a long time. We have no definitive answers.
Yes, it is bigger than a technical problem, and that is important.
OpenAI has not made ‘meaningful progress.’ Certainly we are not on track to solve such problems, and we should not presume they will essentially solve themselves with an ordinary effort, as is implied here.
Indeed, with that attitude, it’s Margaritaville (as in, we might as well start drinking Margaritas.) Whereas with the attitude of Leike and Sutskever, I disagreed with their approach, but I could have been wrong or they could have course corrected, if they had been given the resources to try.
Nor is the second phase problem that we also must solve well-described by ‘basically politics’ of a type we are used to, because there will be entities involved that are not human. Our classical liberal political solutions work better than known alternatives, and well enough for humans to flourish, by assuming various properties of humans and the affordances available to them. AIs with far greater intelligence, capabilities and efficiency, that can be freely copied, and so on, would break those assumptions.
I do greatly appreciate the self-awareness and honesty in this section:
Dean Ball: More specifically, I believe that classical liberalism—individualism wedded with pluralism via the rule of law—is the best starting point, because it has shown the most success in balancing the priorities of the individual and the collective. But of course I do. Those were my politics to begin with.
It is notable how many AI safety advocates, when discussing almost any topic except transformational AI, are also classical liberals. If this confuses you, notice that.
