Author name: Mike M.

the-cruise-origin-driverless-pod-is-dead,-gm-tells-investors

The Cruise Origin driverless pod is dead, GM tells investors

Cars, Cruise, Cruise AV, nhtsa, Origin, robotaxis / /

nobody take the wheel —

The driverless Origin is dead; instead, Cruise will use next-generation Bolt EVs.

–

a rendering of a Cruise Origin picking up passengers in the Castro district in San Francisco

Enlarge / As Cruise ramps up its robotaxi service, it won’t be in these cool-looking driverless pods.

Cruise

The Cruise Origin was definitely the least conventional of all the myriad vehicles that General Motors planned to build using its new Ultium battery platform. For starters, it wasn’t a pickup truck or SUV, unlike all the Ultium-based electric vehicles that have gone into production thus far. Instead, the Origin—meant for Cruise, GM’s robotaxi startup—was a true driverless pod design, a box on wheels with the front and rear seats facing each other and no steering wheel at all. But now the Origin is dead, GM said in a letter to investors today.

We saw the Origin in person in January 2020 at a flashy reveal event that was light on the details. At the time, Cruise was targeting early 2022 to begin deploying Origins, a timeline that accounted for neither pandemic nor the difficulty in actually developing autonomous vehicles.

By early 2022, Cruise was ready to petition the National Highway Traffic Safety Administration, asking permission to begin using Origins on the road. But 2023 was a bad year for the autonomous vehicle company, which had its operations in California suspended after a Cruise robotaxi ran over and then dragged a pedestrian in San Francisco.

The challenge of convincing NHTSA that such a radically different design should be given the OK proved too much for GM to bear, it told investors.

Instead of using Origins, Cruise will turn its attention to the next-generation Chevrolet Bolt, which will cost less per unit than the Origin, helpfully. The next-gen Bolt is a revamp of Chevy’s popular compact EV that will move over to the cheaper Ultium battery platform. The Bolt was GM’s bestselling EV but went out of production last year at the Orion Assembly plant in Michigan, which the automaker wanted to repurpose so it could build electric pickup trucks.

Those electric pickups are now on hold, postponed until mid-2026 GM says. Like Ford, it appears that GM miscalculated the appeal of expensive electric trucks, and as a result the company will not meet its originally stated ambition of building a million EVs in 2025.

The Cruise Origin driverless pod is dead, GM tells investors Read More »

waymo-is-suing-people-who-allegedly-smashed-and-slashed-its-robotaxis

Waymo is suing people who allegedly smashed and slashed its robotaxis

alphabet, Cars, driverless taxi, San Francisco, syndication, waymo / /
Waymo car is vandalized in San Francisco

The people of San Francisco haven’t always been kind to Waymo’s growing fleet of driverless taxis. The autonomous vehicles, which provide tens of thousands of rides each week, have been torched, stomped on, and verbally berated in recent months. Now Waymo is striking back—in the courts.

This month, the Silicon Valley company filed a pair of lawsuits, neither of which have been previously reported, that demand hundreds of thousands of dollars in damages from two alleged vandals. Waymo attorneys said in court papers that the alleged vandalism, which ruined dozens of tires and a tail end, are a significant threat to the company’s reputation. Riding in a vehicle in which the steering wheel swivels on its own can be scary enough. Having to worry about attackers allegedly targeting the rides could undermine Waymo’s ride-hailing business before it even gets past its earliest stage.

Waymo, which falls under the umbrella of Google parent Alphabet, operates a ride-hailing service in San Francisco, Phoenix, and Los Angeles that is comparable to Uber and Lyft except with sensors and software controlling the driving. While its cars haven’t contributed to any known deadly crashes, US regulators continue to probe their sometimes erratic driving. Waymo spokesperson Sandy Karp says the company always prioritizes safety and that the lawsuits reflect that strategy. She declined further comment for this story.

In a filing last week in the California Superior Court of San Francisco County, Waymo sued a Tesla Model 3 driver whom it alleges intentionally rear-ended one of its autonomous Jaguar crossovers. According to the suit, the driver, Konstantine Nikka-Sher Piterman, claimed in a post on X that “Waymo just rekt me” before going on to ask Tesla CEO Elon Musk for a job. The other lawsuit from this month, filed in the same court, targets Ronaile Burton, who allegedly slashed the tires of at least 19 Waymo vehicles. San Francisco prosecutors have filed criminal charges against her to which she has pleaded not guilty. A hearing is scheduled for Tuesday.

Burton’s public defender, Adam Birka-White, says in a statement that Burton “is someone in need of help and not jail” and that prosecutors continue “to prioritize punishing poor people at the behest of corporations, in this case involving a tech company that is under federal investigation for creating dangerous conditions on our streets.”

An attorney for Burton in the civil case hasn’t been named in court records, and Burton is currently in jail and couldn’t be reached for comment. Piterman didn’t respond to a voicemail, a LinkedIn message, and emails seeking comment. He hasn’t responded in court to the accusations.

Based on available records from courts in San Francisco and Phoenix, it appears that Waymo hasn’t previously filed similar lawsuits.

In the Tesla case, Piterman “unlawfully, maliciously, and intentionally” sped his car past a stop sign and into a Waymo car in San Francisco on March 19, according to the company’s suit. When the Waymo tried to pull over, Piterman allegedly drove the Tesla into the Waymo car again. He then allegedly entered the Waymo and later threatened a Waymo representative who responded to the scene in person. San Francisco police cited Piterman, according to the lawsuit. The police didn’t respond to WIRED’s request for comment.

Waymo is suing people who allegedly smashed and slashed its robotaxis Read More »

monthly-roundup-#20:-july-2024

Monthly Roundup #20: July 2024

Monthly / /

It is monthly roundup time.

I invite readers who want to hang out and get lunch in NYC later this week to come on Thursday at Bhatti Indian Grill (27th and Lexington) at noon.

I plan to cover the UBI study in its own post soon.

I cover Nate Silver’s evisceration of the 538 presidential election model, because we cover probabilistic modeling and prediction markets here, but excluding any AI discussions I will continue to do my best to stay out of the actual politics.

Jeff Bezos’ rocket company Blue Origin files comment suggesting SpaceX Starship launches be capped due to ‘impact on local environment.’ This is a rather shameful thing for them to be doing, and not for the first time.

Alexey Guzey reverses course, realizes at 26 that he was a naive idiot at 20 and finds everything he wrote cringe and everything he did incompetent and Obama was too young. Except, no? None of that? Young Alexey did indeed, as he notes, successfully fund a bunch of science and inspire good thoughts and he stands by most of his work. Alas, now he is insufficiently confident to keep doing it and is in his words ‘terrified of old people.’ I think Alexey’s success came exactly because he saw people acting stupid and crazy and systems not working and did not then think ‘oh these old people must have their reasons,’ he instead said that’s stupid and crazy. Or he didn’t even notice that things were so stupid and crazy and tried to just… do stuff.

When I look back on the things I did when I was young and foolish and did not know any better, yeah, some huge mistakes, but also tons that would never have worked if I had known better.

Also, frankly, Alexey is failing to understand (as he is still only 26) how much cognitive and physical decline hits you, and how early. Your experience and wisdom and increased efficiency is fighting your decreasing clock speed and endurance and physical strength and an increasing set of problems. I could not, back then, have done what I am doing now. But I also could not, now, do what I did then, even if I lacked my current responsibilities. For example, by the end of the first day of a Magic tournament I am now completely wiped.

Google short urls are going to stop working. Patrick McKenzie suggests prediction markets on whether various Google services will survive. I’d do it if I was less lazy.

This is moot in some ways now that Biden has dropped out, but being wrong on the internet is always relevant when it impacts our epistemics and future models.

Nate Silver, who now writes Silver Bulletin and runs what used to be the old actually good 538 model, eviscerates the new 538 election model. The ‘new 538’ model had Biden projected to do better in Wisconsin and Ohio than either the fundamentals or his polls, which makes zero sense. It places very little weight on polls, which makes no sense. It has moved towards Biden recently, which makes even less sense. Texas is their third most likely tipping point state, it happens 9.8% of the time, wait what?

At best, Kelsey Piper’s description here is accurate.

Kelsey Piper: Nate Silver is slightly too polite to say it but my takeaway from his thoughtful post is that the 538 model is not usefully distinguishable from a rock with “incumbents win reelection more often than not” painted on it.

Gil: worse, I think Elliott’s modelling approach is probably something like max_(dem_chance) [incumbency advantage, polls, various other approaches].

Elliott’s model in 2020 was more bullish on Biden’s chances than Nate and in that case Trump was the incumbent and down in the polls.

Nate Silver (on Twitter): Sure, the Titanic might seem like it’s capsizing, but what you don’t understand is that the White Star Line has an extremely good track record according to our fundamentals model.

At worst, the model is bugged or incoherent, or a finger is on the scale. And given the debate over Biden stepping aside, this could have altered the outcome of the election. It still might have, if it delayed Biden’s resignation, although once you get anywhere near this far ‘the Sunday after the RNC’ is actually kind of genius timing.

I have done a lot of modeling in my day. What Nate is doing here is what my culture used to refer to as ‘calling bullshit.’ I would work on a model and put together a spreadsheet. I’d hand it off to my partner, who would enter various numbers into the input boxes, and look at the outputs. Then we’d get on the phone and he’d call bullshit: He’d point out a comparison or output somewhere that did not make sense, that could not be right. Usually he’d be right, and we’d iterate until he could not do that anymore. Then we might, mind you I said might, have a good model.

Another thing you could have done was to look at the market, or now the market history, since ‘things may have changed by the time you read this’ indeed.

Thus, no, I do not need to read through complex Bayesian explanations on various modeling assumptions to know that the 538 forecast here is bonkers. If it produces bonkers outputs, then it bonkers. If the topline number seemed bonkers, but all the internals made sense and the movements over time made sense and one could be walked through how that produces the final answer, that would be one thing.

But no, these outputs are simply flat out bonkers. The model does not much care about the things that matter most, it does not respond reasonably, it has outputs in places that were so pro-Biden as to look like bugs. Ignore such Obvious Nonsense.

It is also important because when they change Biden, to Harris or otherwise, there is a good chance they will still make similar mistakes.

As noted above, I will continue to cover modeling and prediction markets, and tracking how the candidates relate to AI, and continue doing my best to avoid otherwise covering the election. You’ll get enough of that without me.

My current view of the market is that Harris is modestly cheap (undervalued) at current prices, but Trump is still the favorite, and we will learn a lot soon when we actually have polling under ‘it’s happening’ conditions.

Shame.

The beatings will continue until we have congestion pricing or a new governor.

We actually do want a 24-hour coffee shop and bookstore (with or without a cat, and 18-hour get you 95% of the value), or the other nice things mentioned in the Josh Ellis thread here. We say we do, and in some ways we act like we do. We still don’t get the things, because our willingness to pay directly says otherwise.

There are many similar things that genuinely seem to make our lives way better, that warm our hearts by their mere existence and optionality. That people actively want to provide, if they could. Yet they are hard to find, because they cannot pay the rent.

You can have your quaint bookstore, on one condition, which is paying a lot more, directly, for some combination of a membership, the books and the coffee.

Instead, we are willing to pay quite a lot more for the house three blocks from the bookstore, because we recognize its value. But if the bookstore charged us half that money directly, we would refuse to pay. It ruins the thing. So the owners of land get rich and the bookstore gets driven out.

I have to remind myself of this constantly. I pay a lot in fixed costs to live in a place I love, including the extra taxes. Then I constantly have the urge to be stingy about actually paying for many of the things that make me want to live here. It is really hard not to do this.

Magic players drive this point home. You plan for a month, pay hundreds for cards, pay hundreds for the plane ticket and hundreds more for the hotel, work to qualify and train, in a real sense this is what you live for… and then complain about the outrageous $100 entry fee or convention fee.

This is so much of why we cannot have nice things. It is not that we do not have a willingness to pay in the form of having less money. It is that we think those things ‘should cost’ a smaller amount, so when they cost more, it ruins the thing. It is at core the same issue as not wanting to buy overpriced wires at the airport.

The CrowdStrike incident was covered on its own. These are other issues.

Least surprising headlines department: Identity-verifier used by Big Tech amid mandates has made personal data easily accessible to hackers.

AU10TIX told 404 Media that the incident was old and credentials were rescinded—but 404 Media found that the credentials still worked as of this month. After relaying that information, AU10TIX “then said it was decommissioning the relevant system, more than a year after the credentials were first exposed on Telegram.”

If you require age verification to safeguard privacy, this will predictably have a high risk of backfiring.

Nearly all AT&T customer records were breached in 2022. The breach has now been leaked to an American hacker in Turkey. This includes every interaction those customers made, and all the phone numbers involved. Recall that in March 2024 data from 73 million AT&T accounts leaked to the dark web. So yes, we need to lock down the frontier AI labs yesterday.

Beware the laptop trap.

Samo Burja: When I first saw the laptop practice in San Francisco I assumed people worked with laptops in cafes because their houses were crowded with too many roommates to save on rent and offices to save on startup runway.

I had no idea people in LA and NYC did this too.

Unless you’re in San Francisco I don’t think your laptop work is adding to GDP. Use cafes to meet friends.

Marko Jukic: European cafes are 100% right to ban “coworking” i.e. staring silently at my electronic device screen for hours on end while pretending to work and taking up space in a public place intended for relaxation and socializing.

Don’t let Americans turn the cafe bar into an office!

The picture on the right above depicts a hellish anti-social prison-like atmosphere. In a cafe, I want to hear music, conversation, laughter, and the football game.

It’s a CAFE, not a library, not an office, not a university lecture hall. Leave your laptop at home.

Americans will complain endlessly how America lacks “third spaces” and enjoyable public life but then like the idea of turning European cafes into sterile workspaces where professional laptop-typers sit in silent rows avoiding eye contact pretending to do important work.

Levelsio: The difference between European and American cafes is so stark

In Europe many don’t allow laptops anymore

In America they usually do and people are working on something cool!

I am with the French here. The cafe is there to be a cafe. If you want to work, you can go to the office, and seriously don’t do it on a laptop, you fool. I do not care if you are in San Francisco.

Marko Jukic claims that what distinguishes others from ‘normies’ is mainly not that normies are insufficiently intelligent, but not normies have astounding and incurable cowardice, especially intellectual cowardice but also risk taking in life in general.

Marko Jukic: Spending time with our young elites at university, in Silicon Valley, etc. I never got the impression that intelligence was lacking. Far from it. What was lacking was everything else necessary to use that intelligence for noble and useful ends. In a way this is much worse.

Actually practicing personal loyalty, principled self-sacrifice, or critical thinking in a way that isn’t camera-ready is not just uncommon or frowned-upon but will get you treated like a deranged, dangerous serial killer by average cowards. It’s actually that bad these days.

To return to the original point, thinking your own thoughts is barely a drop in the bucket of courage. But most don’t even have that drop. Important to keep that in mind when you model society, social technology, reforms, and “the public” or “the normies” or whatever.

We are certainly ‘teaching cowardice’ in many forms as a central culture increasingly over time. It is a major problem. It is also an opportunity. I do not buy the part where having courage gets you attacked. It is not celebrated as much as it used to be, this is true. And there are places where people will indeed turn on you for it, either if you make the wrong move or in general. However, that is a great sign that you want to be in different places.

Note that even in places where rare forms courage are actively celebrated, such as in the startup community, there are other ways in which being the ‘wrong kind of’ courageous and not ‘getting with the program’ will get this same reaction of someone not to be allies with. The principle is almost never properly generalized.

To answer Roon’s request here: No.

Mark Carnegie: If you don’t think this is a crisis i don’t know what to say to you.

Roon: cmon man now adjust the graph with the amount of time people spend texting or in their GCs.

Suhail: Yeah, we’re more connected, not less connected.

No. We really, really aren’t more connected. No, time spent texting or especially in ‘group chats’ is not a substitute to time spent with friends. Indeed, the very fact that people sometimes think it is a substitute is more evidence of the problem. Is it something at all? Yes. It is not remotely the same thing.

Tyler Cowen asks, what is the greatest outright mistake by smart, intelligent people, in contrast to disagreements.

His choice is (drum roll): attempting to forcibly lower prescription drug prices. Here’s the post in full.

Tyler Cowen: I am not referring to disagreements, I mean outright mistakes held by smart, intelligent people.  Let me turn over the microphone to Ariel Pakes, who may someday win a Nobel Prize:

Our calculations indicate that currently proposed U.S. policies to reduce pharmaceutical prices, though particularly beneficial for low-income and elderly populations, could dramatically reduce firms’ investment in highly welfare-improving R&D. The U.S. subsidizes the worldwide pharmaceutical market. One reason is U.S. prices are higher than elsewhere.

Tyler Cowen: That is from his new NBER working paper.  That is supply-side progressivism at work, but shorn of the anti-corporate mood affiliation.

I do not believe we should cancel those who want to regulate down prices on pharmaceuticals, even though likely they will kill millions over time, at least to the extent they succeed.  (Supply is elastic!)  But if we can like them, tolerate them, indeed welcome them into the intellectual community, we should be nice to others as well.  Because the faults of the others probably are less bad than those who wish to regulate down the prices of U.S. pharmaceuticals.

Please note you can favor larger government subsidies for drug R&D, and still not want to see those prices lowered.

He has amusingly gone on to compare those making this mistake to ‘supervillains.’

A lot of people thought this was all rather absurd. The greatest mistake is failure to choose to vastly systematically overpay for something while everyone else gets it dirt cheap, because otherwise future investment would be reduced?

I think this points to what may actually be the gravest genuine mistake, which is:

Causal Decision Theory!

As in, you base your decision on what has the best consequences, rather than choosing (as best you can) the decision algorithm with the best consequences after considering every decision (past, present and future, yours and otherwise) that correlates with your decision now.

Alternatively, you could view it as the desire to force prices to appear fair, the instinct against gouging, which is also involved and likely a top 10 pick.

The debate over pharma prices indeed a great example of how this messes people up.

Everyone else except America is defecting, refusing to pay their fair share to justify the public good of Pharma R&D. One response is that this sucks, but America needs to step up all the more. Another is that if people can defect without punishment knowing others will pick up the slack then they keep doing so, indeed if you had not indicated this to them you would not be in this position now.

On top of that, you are paying off R&D that already happened in order to hold out the promise of reward for R&D in the future (and to some extent to create necessary cash flow). Locally, you are better off doing what everyone else does, and forcibly lowering prices rather than artificially raising them like we do. But if corporations expect that in the future, they will cut R&D.

So everyone is threatening us, and we are paying, so they keep threatening and we keep paying, but also this gives us strong pharma R&D.

You could say on top of the burden being unfairly distributed this is a really dumb way to support pharma R&D, and we should instead do a first best solution like buying out patents. I would agree. Tyler would I presume say, doesn’t matter, because we won’t possibly do this first best solution big enough to work, it is not politically feasible. And I admit he’d probably be right about that.

Another aspect is, suppose a corporation puts you in a position where you can improve welfare, or prevent welfare loss, but to do so you have to pay the corporation a lot of money, although less than the welfare improvement. And they engineered that, knowing that you would pay up. Should you pay? Importantly wrong question framing, the right question is what should your policy be on whether to pay. The policy should be you should pay to the extent that this means the corporations go out to seek large welfare improvements, balanced against how much they seek to engineer private gains including by holding back much of the welfare benefits.

A lot of situations come down to divide-the-pie, various forms of the dictator game – there is $100, Alice decides how to divide it, Bob accepts the division or everyone gets nothing. At what point does Bob accept an unfair division? If Bob demands an unfair (or fair!) division, and Alice believes Bob, at what point does Alice refuse? And so on.

Another way of putting a lot of this is: You can think of yourself or a given action, often, as effectively ‘moving last,’ where you know what everyone will do conditional on your action. That does not mean you must or should do whatever gives you the best payoff going forward, because it is very easy to exploit those with such a policy.

What does that imply about the motivating example? I think the answer is a lot less obvious or clean than Tyler thinks it is, even if you buy (as I mostly buy) the high value of future marginal pharma R&D.

Next up we have another reason you need functional decision theory.

Agenda setting is powerful when you model everyone else as using naïve Causal Decision Theory. If you get to propose a series of changes to be voted upon, you can in theory with enough steps get anything you want.

We model legislative decision-making with an agenda setter who can propose policies sequentially, tailoring each proposal to the status quo that prevails after prior votes. Voters are sophisticated and the agenda setter cannot commit to future proposals.

Nevertheless, the agenda setter obtains her favorite outcome in every equilibrium regardless of the initial default policy. Central to our results is a new condition on preferences, manipulability, that holds in rich policy spaces, including spatial settings and distribution problems. Our findings therefore establish that, despite the sophistication of voters and the absence of commitment power, the agenda setter is effectively a dictator.

Those voters do not sound terribly sophisticated. Rather, those voters sound profoundly unsophisticated.

Fool me once, shame on you. Fool me twice, can’t get fooled again.

An actually sophisticated voter would say that the agenda setter, if allowed to pass anything that is a marginal improvement for 51% of voters, effectively becomes a dictator. The proof is easy, you don’t need a paper – you could for example repeatedly propose to transfer $1 from 49% to the 51%, while always being part of the 51%, repeat until you have almost all the money, use that money periodically to buy other preferences.

The thing is, a sophisticated voter would recognize what you were up to rather quickly. They would say ‘oh, this is a trick, I know that this benefits me on its face but I know where this leads.’ And a majority of them would start always voting no.

This is not merely a theoretical or ideal response. This is a case where economists and casual decision theorists and politicians look at regular people and call them ‘irrational’ for noticing such things and reacting accordingly. What’s the matter with Kansas?

This, from the agenda setter’s perspective, is the matter with Kansas. If you set the agenda to something that looks superficially good, but you having control of the agenda is bad, then I should vote down your agenda on principle, as you haven’t given me any other affordances.

That is not to say that the agenda setter is not powerful. Being the agenda setter is a big game. You do still have to maintain the public trust.

Roon weeps for the old Twitter. He blames the optimizations for engagement for ruining the kinds of communities and interactions that made Twitter great, reporting now his feed is filled with slop and he rarely discovers anything good, whereas good new discoveries used to be common.

I continue to be confused by all the people not strictly using the Following tab plus lists (or Tweetdeck), and letting the For You feed matter to them. Why do you do this thing? Also out of curiosity I checked my For You feed, and it’s almost all the same people I follow or have on my lists, except it includes some replies from them to others, and a small amount of very-high-view-count generic content. There’s no reason to use that feature, but it’s not a hellscape.

Roon: The beauty of twitter was the simcluster, where 90% of the tweets in my feed came from one of the many organic self-organizing communities i was part of. now it’s maybe 20%. I used to daily discover intelligent schizomaniacs, now they are diffuse among the slop.

Near: Human values are actually fully inconsistent with virality-maximizing algorithms ‘but revealed preferences!’ as a take fully misunderstands coordination problems any society can be burnt to the ground with basic game theory and the right algorithm. We should strive for better.

I see Twitter as having net declined a modest amount for my purposes, but it still mostly seems fine if you are careful with how you use it.

I do think that Roon and Near are right that, if this were a sane civilization, Twitter would not be trying so hard to maximize engagement. It would be run as a public good and a public trust, or an investment in the long term. A place to encourage what makes it valuable, with the trust that this would be what matters over time. If it made less (or lost more) money that way, well, Elon Musk could afford it, and the reputational win would be worth the price.

If you want to improve your Twitter game, I found this from Nabeelqu to be good. Here is how I do things there. Here is Michael Nielson’s advice.

Your periodic reminder.

Brian Potter lays out the history of fusion, and the case for and against it being viable.

Scientists want to take more risks, and think science funding should generally take more risks. We need more ambitious projects. This paper points out a flaw in our funding mechanisms. The NIH, NSF and their counterparts make funding decisions by averaging peer review scores, whereas scientists say they would prefer to fund projects with more dissensus. This favors safe projects and makes it difficult to fund novel ideas. This is great news because it is relatively easy to fix by changing the aggregation function to put much less weight on negative reviews. Rule scientific ideas, like thinkers, in, not out.

Does the Nobel Prize sabotage future work?

Abstract: To characterize the impact of major research awards on recipients’ subsequent work, we studied Nobel Prize winners in Chemistry, Physiology or Medicine, and Physics and MacArthur Fellows working in scientific fields.

Using a case-crossover design, we compared scientists’ citations, publications and citations-per-publication from work published in a 3-year pre-award period to their work published in a 3-year post-award period. Nobel Laureates and MacArthur Fellows received fewer citations for post- than for pre-award work. This was driven mostly by Nobel Laureates. Median decrease was 80.5 citations among Nobel Laureates (p = 0.004) and 2 among MacArthur Fellows (p = 0.857). Mid-career (42–57 years) and senior (greater than 57 years) researchers tended to earn fewer citations for post-award work.

Early career researchers (less than 42 years, typically MacArthur Fellows) tended to earn more, but the difference was non-significant. MacArthur Fellows (p = 0.001) but not Nobel Laureates (p = 0.180) had significantly more post-award publications. Both populations had significantly fewer post-award citations per paper (p = 0.043 for Nobel Laureates, 0.005 for MacArthur Fellows, and 0.0004 for combined population). If major research awards indeed fail to increase (and even decrease) recipients’ impact, one may need to reassess the purposes, criteria, and impacts of awards to improve the scientific enterprise.

Steve Sailer (in the MR comments): I had dinner with Physics Laureate Robert Wilson, who had with Arno Penzias discovered the origin of the universe, a few months after Wilson won the Nobel in 1978. He was very gracious and polite as he was feted by his alma mater, Rice U., but deep down inside he probably wished he could have been back at his observatory tinkering with his radio telescope rather than doing all this kind of unproductive socializing you have to do after winning the Nobel.

Crusader (MR comments): Who ever said that major awards are supposed to increase the recipient’s future impact regardless of its merit?

Are Olympic gold medals supposed to increase the performance of athletes afterwards? Is a research award not just a status game carrot meant to incentivize the “first success” as well as a signal to others to review the related research?

Quite so. If you get a Nobel Prize then suddenly you have a ton of social obligations. The point of the prize is to give people something to aspire to win, not to enable those who win one to then do superior work, also scientists who win are typically already sufficiently old that their productivity will have peaked.

It seems odd to think about a Nobel Prize as being primarily about enabling future work. Even to suggest it is a huge indictment of our academic system – if you are up for a Nobel Prize, why didn’t you already have whatever resources and research agenda you most wanted?

Should scientific misconduct be criminalized? The slippery slope dangers are obvious. Yet it seems a violation of justice and also incentives that Sylvain Lense, whose deception wildly distorted Alzheimer’s research, killing many and wasting epic amounts of time and money, remains at large. Can we simply charge with fraud? If not, why the hell not?

Linch: Gender issues aside, it’s utterly bizarre to me that plagiarism is considered vastly worse among academics than faking data. It’s indicative pretty straightforwardly of rot imo, since it means the field as a whole cares more about credit attribution than about truth.

Paper asks how people decide who is correct when groups of scientists disagree. Here is the abstract.

Uncertainty that arises from disputes among scientists seems to foster public skepticism or noncompliance. Communication of potential cues to the relative performance of contending scientists might affect judgments of which position is likely more valid. We used actual scientific disputes—the nature of dark matter, sea level rise under climate change, and benefits and risks of marijuana—to assess Americans’ responses (n = 3150). Seven cues—replication, information quality, the majority position, degree source, experience, reference group support, and employer—were presented three cues at a time in a planned-missingness design. The most influential cues were majority vote, replication, information quality, and experience. Several potential moderators—topical engagement, prior attitudes, knowledge of science, and attitudes toward science—lacked even small effects on choice, but cues had the strongest effects for dark matter and weakest effects for marijuana, and general mistrust of scientists moderately attenuated top cues’ effects. Risk communicators can take these influential cues into account in understanding how laypeople respond to scientific disputes, and improving communication about such disputes.

The first sentence carries the odd implicit assumption that there is a ‘correct’ answer people should accept, the absence of which is skepticism or noncompliance. Then there’s describing various forms of Bayesian evidence as ‘cues,’ as opposed to considering the hypothesis that people might be considering the hypothesis. The role of risk manager seems to assume they already know what others are supposed to believe during scientific disputes. How do we use the right messaging to ensure the official scientists get believed over the unofficial ones?

Here are the results, all seven factors mattered.

Majority vote, replication and information quality and experience (where experience is defined as time doing this particular type of research), the most influential ‘cues,’ seem like excellent evidence to be using, with majority vote and replication correctly being used as the most important.

The other three are reference group support, degree source and employer. These seem clearly less good, although worth a non-zero amount. No, we should not rely too heavily on arguments from authority, and in particular not on arguments for association with authority.

Mistrust of science only decreased impact sizes by about 27%.

Score one for the public all around.

One thing I love about the paper is in 2.4.7 they lay out their predictions for which factors will be most important and how impacts are expected to work. Kudos.

Here are the detailed descriptions of the questions and cues.

Cues have the strongest effect on dark matter, a case where regular people have little to go on and know it and where everyone has reason to be objective. Marijuana leaves room for the most practical considerations, so any cues are competing with other evidence and it makes sense they have less impact.

Via Robin Hanson, across six studies, communicators who take an absolute honesty stance (‘it is never okay to lie’) and then lie anyway are punished less than those who take a flexible honesty stance that reflects the same actual behavior.

The straightforward explanation is that it is better for people to endorse the correct moral principles and to strive to live up to them and fail, rather than not endorse them at all. This helps enforce the norm or at least weakens it less, on several levels, and predicts better adherence and an effort to do so. With the same observed honesty level, one predicts more honesty both in the past and the future from someone who at least doesn’t actively endorse lying.

One can also say this is dependent on the lab setting and lack of repeated interaction. In that model, in addition to the dynamics above, hypocrisy has short term benefits and long term costs. If you admit to being a liar, you pay a very large one-time cost, then pay a much smaller cost for your lies beyond that, perhaps almost zero. If you say you always tell the truth, then you pay a future cost for each lie, which only adds up over the course of a long period.

Certainly Trump is the avatar of the opposite strategy, of admitting you lie all the time and then lying all the time and paying very little marginal cost per lie.

In Bayesian terms, we estimate how often someone has lied to us and will lie in the future, and will punish them proportional to this, but also proportionally more if you take a particularly strong anti-lie stance. And also we reward or punish you for your estimated effort to not lie and to enforce and encourage good norms, by both means.

In both cases, if you are providing only a few additional bits of evidence on your true base rate, hypocrisy is the way to go. If discount rates are low and you’re going to be exposed fully either way, then meta-honesty might be the best policy.

One can also ask if honesty is an exception here, and perhaps the pattern is different on other virtues. If you are exposed as a liar, and thus exposed as a liar about whether you are a liar, how additionally mad can I really get there? How much does ‘hypocrite’ add to ‘liar,’ which arguably is strictly stronger as an accusation?

German marginal tax rates are a disaster and the poverty trap is gigantic.

The grey lines are Euros per month. Orange is effective take home pay. You essentially earn nothing by going from $25,800/year to $77,400/year, what the hell? With the median income right in the middle of that around €45k.

It is not as extreme as it sounds, because the benefits you get are not fully fungible. To get them you need to be renting, and to get max value it needs to be in a relatively expensive city, whereas the actual cash benefit is only 500 euros a month, which isn’t much. But still, yikes. This has to be a recipe for massive voluntary unemployment and black market work. To the extent that it isn’t, it is the German character being bizarrely unable to solve for this particular equilibrium.

jmkd: The wikipedia article (in German) below suggests that ~15% of the German economy is in “undeclared work.” Admittedly using numbers from different time periods, that would be equivalent to roughly 1/4 of the population working minimum wage.

yo: It’s a household-level view for a family of four. Roughly, if this family has no income, it is eligible for Bürgergeld, €24k/y. Plus a rent subsidy worth about the same €24k/y in the big cities, plus health insurance worth around €15k/y for that family. So yes, average families can get roughly €70k net welfare. Note that a family of four with €70k income would not pay much in taxes. But it would pay around 20% of this pretax income in social charges (mostly pension contributions and health insurance)

Oye cariño, ¿quieres comprar algunos créditos porno? Spain unveils the Digital Wallet Beta, an app for internet platforms to check before letting you watch porn. The EU is giving all porn sites until 2027 to stop you from watching porn, forcing kids (by that point) to download AI porn generators instead. Or have their AI assistant purchase some of those porn credits from ‘enthusiasts.’

Gian Volpicelli (Politico): Officially (and drily) called the Digital Wallet Beta (Cartera Digital Beta), the app Madrid unveiled on Monday would allow internet platforms to check whether a prospective smut-watcher is over 18. Porn-viewers will be asked to use the app to verify their age. Once verified, they’ll receive 30 generated “porn credits” with a one-month validity granting them access to adult content. Enthusiasts will be able to request extra credits. 

While the tool has been criticized for its complexity, the government says the credit-based model is more privacy-friendly, ensuring that users’ online activities are not easily traceable.

While I oppose this on principle, I do approve of this for the kids all things being equal. You should have to work a bit for your porn especially when you are young. I also like the VPN encouragement. The parts where various website geoblock and adults get inconvenienced and identification information is inevitably stolen again as it was this past month? Those parts I do not like as much.

Should the UK use proportional representation? Tyler Cowen says no, because the UK needs bold action so it is good to give one party a decisive mandate even if they got only a third of the vote and essentially won because game theory and a relatively united left. See what they can do, you can always vote them out again. He does not much care about the voters not actually wanting Labour to rule any more than they did before. The point of democracy, in his view, is as a check in case government gets too out of line (and presumably a source of legitimacy), rather than ensuring ‘fairness.’

The danger is an unfair system can damage those other goals too, and this seems like a lot of power to hand to those who get the upper hand in the game theory. Essentially everyone is locked in these ‘unite or die’ dilemmas constantly, as we are in America, except now there is an expectation that people might not unite. So I presume you need some form of runoff, approval or ranked choice voting. They are far from perfect, but so much less distortionary than actual first past the post rules when they fail to collapse into a two party system.

The FTC tried to ban almost all noncompetes, including retroactively. It is not terribly surprising that the courts objected. Judge Ada Brown issued a temporary block, finding that the FTC likely lacked the authority to make the rule, which seems like a very obviously correct observation to me.

Thom Lambert: Now that @FTC’s noncompete ban has been preliminarily enjoined (unsurprisingly), let’s think about some things the agency could do on noncompetes that are actually within its authority. It could, of course, bring challenges against unjustified noncompetes.

hat would create some helpful precedent *andallow the agency to amass expertise in identifying noncompetes that are unwarranted. (The agency implausibly claims that all but a very few noncompetes lack justification, but it has almost no experience with noncompete cases.)

It could also promulgate enforcement guidelines. If the guidelines really take account of the pros and cons of noncompetes (yes, there are pros) and fairly set forth how to separate the wheat from the chaff, they’ll have huge influence in the courts and on private parties.

These moves are admittedly not as splashy as a sweeping economy-wide ban, but they’re more likely to minimize error cost, and they’re within the agency’s authority. In the end, achievement matters more than activity.

This is the new reality even more than it was before.

  1. If you bring individual action against particular cases you can build up case law and examples.

  2. If you try to write a maximally broad rule, the courts are going to see to it you have a bad time.

There was a lot of talk about the overturning of Chevron, but there was another case that could also potentially be a big deal in making government work even less well. This is Ohio v. EPA, which is saying that if you ignore any issue raised in the public comments, then that can torpedo an entire project.

Robinson Meyer: Last week, the Court may have imposed a new and *extremelyhigh-scrutiny standard on how federal agencies respond to public comments. That will slow the EPA’s ability to write new rules, but it would also make NEPA even more arduous.

…

The EPA did respond to the comments at the center of the Ohio case, but Justice Neil Gorsuch, writing for the majority, decided the agency did not address a few specific concerns properly.

So the new procedure will be, presumably, to raise every objection possible, throw everything you can at the wall, then unless the government responds to each concern raised in each of the now thousands (or more) comments, you can challenge the entire action. And similarly, you can do the same thing with NEPA, making taking any action that much harder. Perhaps essentially impossible.

French elections produce unexpected seemingly disproportional results.

It is not as bad as it looks. NFP and Macron essentially (as I understand it) operated as one block, with whoever was behind dropping out in each local election, so effectively this is more like a party with 49.1% of the vote getting 325 seats to RN’s 37.4% and 142.

Claude estimates that if a similar result happened in America, the house would break down about 265-170, but our system is highly gerrymandered and the parties are geographically isolated. I don’t think 325-142 is that extreme here.

If you combined RN+LR+’Other Right’ then you would get 46% of the vote and only 208 seats with a 3.1% gap, which seems extreme. LR and Other Right did well in converting votes to seats in the second round, so they were likely not being dramatically unstrategic.

Similarly to the English results, one must ask to what extent we want strategic voting and negotiating between parties to determine who gets to rule.

New York City sets minimum food delivery wage to $19.56, which in turn means intense competition for work preference during busy hours. It also means fees on every order, which many no doubt are responding to by not tipping. I strongly suspect most of this mostly cancels out and the services are still totally worth it.

New York City gets trash cans. You thought the day would never come. So did I. Before unveiling them, New York did a $4 million McKinsey study ‘to see if trash cans work’ and that is not the first best solution but it sure is second best.

Enguerrand VII de Coucy: Oh my god New York City paid McKinsey $4,000,000 to do a study on if trash cans work.

rateek Joshi: Maybe the point was that the NYC govt wanted to tell its citizens “If you don’t start putting trash in trash bins, we’ll give more money to McKinsey.”

Enguerrand VII de Coucy: Honestly that’s a potent threat

Swann Marcus: In fairness, the end result of this McKinsey study was that New York started using trashcans. Most American cities would have spent $4 million on a trashcan study and then inexplicably never gotten trashcans.

Aaron Bergman: I am going to stake out my position as a trash can study defender. It probably makes sense to carefully study the effects of even a boring and intuitive policy change that affects ~10⁷ people

Mike Blume: It’s fun to rag on NYC for their incompetence in this area, but “where will the bins go” is an understudied problem on many American streets

Getting the details right here is very important. There are some cases where governments vastly overpay for stupid things, and I don’t think this is one of them.

In defense of the lost art of the filler episode. I strongly agree here. Not all shows should be 22 episodes a year, but many should be. It makes the highs mean more, and I love spending the extra time and taking things gradually.

What do we make of this list and also the rating type breakdown?

The recency bias is strong. There are way too many 2010s shows here. I do think that there was a quality upgrade around the 90s but still.

The drama bias is also strong. Comedies are great and deserve more respect.

It’s hard to get a good read on the relative rating systems. It does seem like too much weight was put on the votes.

How many of these have I seen enough to judge?

There are a bunch of edge cases but I would say 20.

Correctly or Reasonably Rated: The Wire (my #1), Breaking Bad (my #3 drama), The Office, It’s Always Sunny in Philadelphia, Mr. Robot (I have it lower but I can’t argue), Severance (so far, it’s still early), Seinfeld (you somewhat had to be there), Freaks and Geeks (if you don’t hold brevity against it).

Underrated: The Americans (my #2 drama), Deadwood

Decent Pick But Overrated: Chernobyl (miniseries don’t count, others are missing if they do, and even if you discount that it’s good but not this good), Game of Thrones (great times and should make the list but you can’t put it at #2 after the last few seasons, come on), Stranger Things (Worth It but #8?!), Battlestar Galactica (this is a bit generous), The Shield (I can maybe see it), Lost (oh what could have been).

Bad Pick: Friends (better than its rep in my circles but not a best-of), House (it’s fine but not special), True Detective (one very good season but then unwatchable and no time is not a flat circle), Black Mirror (not half as clever as it thinks, despite some great episodes), The Mandalorian (I stuck with it long enough to know it isn’t top 50 level great and wasn’t working for me, although it isn’t actively bad or anything).

Most Importantly Missing (that I know of and would defend as objective, starting with the best three comedies then no order): Community, The Good Place, Coupling (UK) (if that counts), Watchmen (if we are allowing Chernobyl this is the best miniseries I know), Ally McBeal, Angel and Buffy the Vampire Slayer (no, seriously, a recent rewatch confirms), Gilmore Girls, Roseanne, Star Trek: DS9 (I see the counterarguments but they’re wrong), How I Met Your Mother.

I wonder if you should count Law & Order. You kind of should, and kind of shouldn’t.

The other ~30 here I haven’t given enough of a chance to definitively judge. Many I hadn’t even heard about.

Does anyone have a better list?

Of the ones I didn’t mention, I’m open to the case being made. For The Sopranos and Better Call Saul, I watched a few episodes and realized they were objectively very good but thought ‘I do not want to watch this.’ Or in particular, the show is great but I do not want to watch these people. A bunch of others here seem similar?

I can overcome that, but it is hard. Breaking Bad is not something I wanted to watch, in many important senses, but it was too good not to, and Walter White breaks bad but does not have that ‘I can’t even with this guy.’

Scott Sumner has his films of 2024 Q2. He put Challengers at only 2.6/4, whereas I have Challengers at 4.5/5, which provides insight into what he cares about. From the description he was clearly on tilt that day. Also I strongly suspect he simply does not get the characters involved, and finding them unlikeable did not seek to get them. It is the first time I’ve seen his rating and said not ‘you rated this differently than I would because we measure different things’ but rather ‘no, you are wrong.’

My movie log and reviews continue to be at Letterboxd. I’ve moved more towards movies over television and haven’t started a new TV series in months.

The official EA song should be: Okay, full disclosure. We’re not that great. But nevertheless, you suck.

Economeager: As you know i do not identify with EAs as a culture despite my great support for givewell, open phil, etc. However when I meet someone who gives misguided and ineffective charity for purely emotional reasons I do have like a palpatine kermit moment with myself.

Never mind I saw the EA guys getting hyped to think about how “the economy” will work “after AGI” and hate everyone equally again.

Andy Masley: I was on the fence about getting more involved in EA a few years ago and then in my old job was exposed to a charity where people read stories over Zoom to dogs.

When given $10,000 to spend however they wanted, people spent the majority of it on pro-social things that benefited others, and almost 17% went to charities outright. This seems like a missed opportunity to provide more details about what types of things the money was spent on, we can study multiple things at once. Public posting of spending choices on Twitter had little impact on distribution of purchases.

I didn’t get a chance to pre-register my expectations here, nor do I have a good sense of exactly what counts as ‘pro social’ versus not. The idea that people, when given a windfall, spread it around somewhat generously, seems obvious. Windfalls are considered by most people as distinct from non-windfall decisions, the money is ‘not yours’ or not part of your typical planning, and is often largely wasted or bestowed generously, in a way that ‘core’ income is not. It is an opportunity to affirm your bonds to the community and good character and not present a target, and the money fails to ‘feel real.’ I do find it strange that public info did not at all impact decisions, which makes me suspect that such decisions were treated as effectively equally public either way in practice.

Johns Hopkins Medical School goes tuition-free for medical students due to massive grant, also expands aid for future nurses and public health pioneers. Nikhil Krishnan speculates that more places will end up doing this, and correctly notices this is not actually good.

The choke point is residency slots. It would not be my first pick for charity dollars, but I think that ‘give money to endow additional residency slots at hospitals that agree to play ball’ would be a highly understandable choice. Whereas ‘make future doctors that will mostly earn a lot of money have less student debt’ does not make sense. Yes, you can potentially improve applicant quality a bit, but not much. Whatever your goal, unless it is ‘glory to this particular program,’ you can do it better.

You can use 1Password to populate environmental variables in CLI scripts, so you can keep your API keys in your password manager, also there is a fly.io plugin.

Arnold Ventures is hiring for its infrastructure team.

How to write for Works in Progress.

Pick your neighborhood carefully, not only your city.

Phil: So, the first thing I think of is that you’re going to spend 1000x more time in your surrounding 5 blocks than you will in any other neighborhood in your city. And so thinking about all the things that New York City or next city has, is to me a lot less important than thinking about the things within the five blocks where you live. Most neighborhoods in your city you might never step foot in, they might as well be in the other side of the country. But the things in your immediate vicinity are the things that are going to dominate your life. So picking and influencing your neighborhood is really important. And the two big ways you can influence your neighborhood are one, determining who lives in your neighborhood by moving people there, something I am very biased on because I work on it. And two, improving your neighborhood.

As a New Yorker, I definitely will walk more than five blocks more than 5% of the time. For example, my favorite most frequented restaurant is 7 blocks away. The point very much still stands. My friend Seth uses the rule of thumb that value is proportional to the inverse square of travel time, which again goes too far but is directionally right.

Concert goers who consumed more alcohol were less likely to choose pro-social options in experimental economic games. Does not seem to distinguish between cooperators being more sober, versus sobriety leading to cooperation. Both seem plausible. One more reason not to drink.

Little effect is found of siblings on attitudes towards inequality. This study says more about what current academic pressures and biases than it says about anything else.

Paper says that despite the narrative of democratic backsliding, objective measures such as electoral competitiveness, executive constraints and media freedom show no such evidence of (net) backsliding.

Those with higher IQ scores shoot firearms more accurately. I did not expect that. The real intelligence is never needing to shoot and never getting shot. I bet those correlate too.

Your enemies probably have more enemies than you do. Unfortunately, on the same principle, you probably have fewer friends than your friends.

Shoutout to my former teammate and coworker Kai Budde, the German Juggernaut who never loses on Sundays. He’s an all around amazing guys and best teammates you will know. I mention this because unfortunately Kai has terminal cancer. They have renamed the Player of the Year trophy in Kai’s honor.

He at least got a chance to play the PT recently in Amsterdam, with all the associated great times.

Then it was a Sunday, so of course Kai Budde won the PTQ.

Even with my qualification slots, I’m well past the point I can take this kind of time off to properly prepare, and even if I could I can’t put up the stamina for a three day fight, or even a two day fight. But man I miss the good times.

Moxfield lets you do this:

Lupe: I used to be in on the bling until we hit a weird critical capacity of too much. I’m now slowly putting a filter of “first printing” on all of the cards in my main Cube. Magic cards are kind of like hieroglyphs, so as a designer, I want to maximize tabletop legibility.

Brian Kowal: This is The Way.

Magical Hacker: I didn’t know you could do this until I saw this post, & now I need to share what I picked: f:c game:paper lang:en -e:plst (frame: 2015 -is:borderless (is:booster or st:commander) -is:textless -is:ub -is:etched or -is:reprint or e:phpr) (-e:sld or e:sld -is:reprint) prefer:newest

I cannot emphasize enough how much I agree with Lupe. Some amount of bling is cool. At this point we have way, way too much bling. There are too many cards, and also too many versions of each card, too many of which are not legible if you do not already know them on sight. I do want to stay in touch with the game, but it seems impossible.

The value of Chess squares, as measured by locations of pawns, bishops and knights. A fun exercise that I do not expect to offer players much insight. Pawn structure seems strangely neglected in their analysis.

John Carmack points out that a key reason the XBox (and I would add the PlayStation) never caught on as entertainment centers is that their controllers require non-trivial power to operate, so they go to sleep after periods of inaction and require frequent charging. If we could solve that problem, I would happily use the PlayStation as a media center, the interface is otherwise quite good.

Surely we can get a solution for this? Why can’t we have a remote that functions both ways, perhaps with a toggle to switch between them? Maybe add some additional buttons designed to work better as part of a normal remote?

Matthew Yglesias makes a case that high-pressure youth sports is bad for America. Sports played casually with your friends are great. Instead, we feel pressure to do these expensive, time consuming, high pressure formalized activities that are not fun, or we worry we will be left behind. That cuts out a lot of kids, is highly taxing on parents and damages communities. And yes, I agree that this trend is terrible for all these reasons. Kids should mostly be playing casually, having fun, not trying to make peak performance happen.

Where we differ is Yglesias thinks this comes from fear of being left behind. There is some of that but I am guessing the main driver is fear of letting kids play unsupervised or do anything unstructured. The reason we choose formal sports over the sandlot is that the sandlot gets you a call to child services. Or, even if it doesn’t, you worry that it would.

Hockey got one thing very right.

Scott Simon: In prep for, tonight, watching my first hockey game in… a decade?… I just learned that challenges in the NHL come with real stakes—if you’re wrong, your team is assessed a penalty. Now *thatis a challenge system. (Still, robot refs now.)

My first choice is no challenges. Barring that, make them expensive.

Tyler Cowen links to a paper by Christian Deutscher, Lena Neuberg, and Stefan Thiem on Shadow Effects of Tennis Superstars. They find that when the next round in a second-tier tournament would be against one of the top four superstars, other players in the top 20 over the period 2004-2019 would advance substantially less often than you would otherwise expect.

The more the superstars go away, the more the other top competitors smell blood and double down, effect size is 8.3 percentage points which is pretty large. Part of that might come from the opposite effect as well, if I was not a top player I might very much want the honor of playing against Federer or Nadal. Mostly I am presuming this effect is real. Tennis is a tough sport and you can’t play your full-on A-game every time especially if slightly hurt. You have to pick your battles.

Analysis of the new NFL kickoff rules, similar to the XFL rules. I realize the injury rate on kickoffs was too high, and seeing how this plays out should be fun, but these new rules seem crazy complicated and ham fisted. At some point we need to ask whether we need a kickoff at all? What if we simply started with something like a 4th and 15 and let it be a punt, or you could go for it if you wanted?

College football seems ready to determine home teams in the new playoff based on factors like ‘hotel room availability,’ ‘ticket sales’ and weather? Wtf? Oh no indeed.

Mitchell Wesson: Schools can absolutely control the quality and quantity of nearby hotel rooms.

Weather, obviously not but it doesn’t seem reasonable to ignore it either. Wouldn’t be fair to fans or teams if a game has to be delayed when that could otherwise have been avoided.

If someone gets to host, there needs to be only one consideration in who hosts a playoff game. That is which team earned a higher seed (however you determine that) and deserves home field advantage. That is it. If the committee actually ever gives home field to the other team, even once, for any other reason (other than weather so extreme you outright couldn’t play the game), the whole system is rendered completely illegitimate. Period.

Waymo now open to everyone in San Francisco.

Sholto Douglas: Three telling anecdotes

> I felt safer cycling next to a Waymo than a human the other day (the first time I’ve had more ‘trust’ in an AI than a human)

> the default verb/primary app has changed from Uber to Waymo amongst my friends

> when you ride one, try to beat it at picking up on noticing people before they appear in the map, you ~won’t

They’re amazing. Can’t wait for them to scale globally.

Matt Yglesias asks what we even mean by Neoliberalism, why everyone uses it as a boogeyman, and whether we actually tried it. Conclusions correctly seem to be ‘the intention was actually letting people do things but it gets used to describe anything permitting or doing something one doesn’t like,’ ‘because people want to propose bad policies telling people what to do without facing consequences’ and ‘no.’

Certainly all claims that the era of big government was ever over, or that we suddenly stopped telling people what they were allowed to do, or that we pursued anything that was at all related to ‘growth at all costs’ is absurd, although we made some progress on at least not having (fewer, although still far too many) price controls.

Nick proposes that for less than $1 million a year you could easily have the coolest and highest status cafe in San Francisco, attracting immense talent, have a cultural touchstone with lots of leverage, creating tons of real estate and actual value, other neat stuff like that. It seems many engineers pus super high value on the right cafe vibe, on the level of ‘buy a house nearby.’ I don’t get it, but I don’t have to. Nick proposes finding a rich patron or a company that wants it nearby. That could work.

In general, this is part of the pattern where nice places to be add tons of value, but people are unwilling to pay for them. You can provide $50/visit in value, but if you charge $10/table or $10/coffee, people decide that kills the vibe.

Which do you value more as a potential superhero: Mind control, flight, teleportation or super strength? On the survey the answer was teleportation.

The correct response, of course, is to have so many questions. Details matter.

Teleportation is a very extreme case of Required Secondary Powers. How do you ensure you do not teleport into a wall or the air or space? How do you deal with displacement? How often can you do it? Where can you go and not go? And so on.

There are versions of teleportation I’ve seen (including in some versions of AD&D) where I would not pay much for them, because you are so likely to get yourself killed you would only do it in a true emergency. Then there are others that are absurdly valuable.

Flight is the lightweight version of the same problem. If you take it to mean the intuitive ‘thing that Superman or Wonder Woman can do in movies’ then yeah, pretty great assuming people don’t respond by trying to put you in a lab, and I’d pay a lot.

Super strength is a nice to have at ‘normal’ levels. At extreme levels it gets a lot more interesting as you start violating the laws of physics or enabling new engineering projects, especially if you have various secondary powers.

Mind control is on an entirely different level. Sometimes it is a relatively weak power, sometimes it enables easy world domination. There you have to ask, as one of your first questions, does anyone else get mind control powers too? This is like the question of AI, with similarly nonsensical scenarios being the default. If the people with true mind control powers used them properly there would usually be no movie. If others get ‘for real’ versions of mind control, and you take super strength or flight, do you even matter? If so, what is your plan? And so on.

What activities do people enjoy or not enjoy?

Rob Wiblin [list edited for what I found interesting]:

  1. ‘Computer games’ are among the most enjoyable activities, probably deserve more respect. It clearly beats ‘watching TV’. ‘Games at home’ sounds cheap and accessible and scores high — I guess that’s mostly card or board games.

  2. Highly social activities are more work and money to set up but still come in highest of all: ‘restaurant / pub’, ‘go to sport’, and ‘theatre / concert’. ‘Parties’ comes in behind those.

  3. ‘Play with child’ was among the most enjoyable of any activity. Many folks who choose not to have kids probably underrate that pleasure. Pulling in the other direction ‘Childcare’ falls in the middle of the pack, though it’s more popular by a mile than school, housework, or paid work. No surprise some people opt out of the workforce to raise a family!

  4. ‘Homework’ came dead last, much less popular than even ‘School’. Counts in favour of reducing it where it’s not generating some big academic benefit.

  5. ‘Email and internet’ — the activity that eats ever more of our days — is right in the middle. Conventional wisdom is you want to substitute it for true leisure and the numbers here clearly back that up.

  6. There’s some preference for active over passive leisure — TV, reading, doing nothing and radio are all mediocre by the standards of recreation. I’m surprised reading and watching TV are right next to one another (I would have expected reading to score higher).

  7. People sure hate looking for a job.

  8. I’ve seen some debate about how much people like or dislike their jobs. Work and school are definitely much less enjoyable than activities where people are more likely to be freely determining for themselves what they’re doing. But they still manage a 4.7 out of 7. It could be much worse (and in the past probably was). Commuting is unpopular but not at the very bottom like I’d heard.

Gaming and sports for the win. Going to the game is second only to concerts, and I strongly agree most of us are not going to enough of either. Weird that going to the movies is not here, I’d be curious how high it goes. And yes, playing board games at home is overpowered as a fun activity if you can make it happen.

Homework being this bad is not a surprise, but it needs emphasis. If everyone understood that it was less fun than looking for a job or doing the laundry, perhaps they would begin to understand.

Reading I am guessing scores relatively low because people feel obligated to read. Whereas those who choose to read for relaxation on average like it a lot more.

Why Do Companies Go Woke? Middle managers, so a result of moral maze dynamics, which includes a lack of any tether to or caring about physical reality. Makes sense.

The absurdity of the claims in Graeber’s Bullshit Jobs.

Ross Rheingans-Yoo notes that ‘hold right mouse button and then gesture’ is a technique he and others often use playing the game Dota because it is highly efficient, yet only when Parity suggested it did it occur to him to use it for normal text editing. My initial reaction was skepticism but it’s growing on me, and I’m excited to try it once someone implements it especially if you can customize the options.

Making dumb mistakes is fine. Systems predictably making particular dumb mistakes is also fine. Even bias can be fine.

This was a serious miss, but it is like AI – if you only look for where the output is dumb, you will miss the point.

Keep trying, and you’ll figure it out eventually.

(For those who don’t know, this was about prediction markets on the Democratic presidential nomination.)

Monthly Roundup #20: July 2024 Read More »

model-mixes-ai-and-physics-to-do-global-forecasts

Model mixes AI and physics to do global forecasts

atmospheric science, climate, Earth science, forecasting, meteorology, Science, Weather / /

Cloudy with a chance of accuracy —

Google/academic project is great with weather, has some limits for climate.

–

Image of a dark blue flattened projection of the Earth, with lighter blue areas showing the circulation of the atmosphere.

Enlarge / Image of some of the atmospheric circulation seen during NeuralGCM runs.

Google

Right now, the world’s best weather forecast model is a General Circulation Model, or GCM, put together by the European Center for Medium-Range Weather Forecasts. A GCM is in part based on code that calculates the physics of various atmospheric processes that we understand well. For a lot of the rest, GCMs rely on what’s termed “parameterization,” which attempts to use empirically determined relationships to approximate what’s going on with processes where we don’t fully understand the physics.

Lately, GCMs have faced some competition from machine-learning techniques, which train AI systems to recognize patterns in meteorological data and use those to predict the conditions that will result over the next few days. Their forecasts, however, tend to get a bit vague after more than a few days and can’t deal with the sort of long-term factors that need to be considered when GCMs are used to study climate change.

On Monday, a team from Google’s AI group and the European Centre for Medium-Range Weather Forecasts are announcing NeuralGCM, a system that mixes physics-based atmospheric circulation with AI parameterization of other meteorological influences. Neural GCM is computationally efficient and performs very well in weather forecast benchmarks. Strikingly, it can also produce reasonable-looking output for runs that cover decades, potentially allowing it to address some climate-relevant questions. While it can’t handle a lot of what we use climate models for, there are some obvious routes for potential improvements.

Meet NeuralGCM

NeuralGCM is a two-part system. There’s what the researchers term a “dynamical core,” which handles the physics of large-scale atmospheric convection and takes into account basic physics like gravity and thermodynamics. Everything else is handled by the AI portion. “It’s everything that’s not in the equations of fluid dynamics,” said Google’s Stephan Hoyer. “So that means clouds, rainfall, solar radiation, drag across the surface of the Earth—also all the residual terms in the equations that happen below the grid scale of about roughly 100 kilometers or so.” It’s what you might call a monolithic AI. Rather than training individual modules that handle a single process, such as cloud formation, the AI portion is trained to deal with everything at once.

Critically, the whole system is trained concurrently rather than training the AI separately from the physics core. Initially, performance evaluations and updates to the neural network were performed at six-hour intervals since the system isn’t very stable until at least partially trained. Over time, those are stretched out to five days.

The result is a system that’s competitive with the best available for forecasts running out to 10 days, often exceeding the competition depending on the precise measure used (in addition to weather forecasting benchmarks, the researchers looked at features like tropical cyclones, atmospheric rivers, and the Intertropical Convergence Zone). On the longer forecasts, it tended to produce features that were less blurry than those made by pure AI forecasters, even though it was operating at a lower resolution than they were. This lower resolution means larger grid squares—the surface of the Earth is divided up into individual squares for computational purposes—than most other models, which cuts down significantly on its computing requirements.

Despite its success with weather, there were a couple of major caveats. One is that NeuralGCM tended to underestimate extreme events occurring in the tropics. The second is that it doesn’t actually model precipitation; instead, it calculates the balance between evaporation and precipitation.

But it also comes with some specific advantages over some other short-term forecast models, key among them being that it isn’t actually limited to running over the short term. The researchers let it run for up to two years, and it successfully reproduced a reasonable-looking seasonal cycle, including large-scale features of the atmospheric circulation. Other long-duration runs show that it can produce appropriate counts of tropical cyclones, which go on to follow trajectories that reflect patterns seen in the real world.

Model mixes AI and physics to do global forecasts Read More »

can-the-solar-industry-keep-the-lights-on?

Can the solar industry keep the lights on?

climate change, Policy, solar panels, syndication / /
Image of solar panels on a green grassy field, with blue sky in the background.

Founded in Dresden in the early 1990s, Germany’s Solarwatt quickly became an emblem of Europe’s renewable energy ambitions and bold plan to build a solar power industry.

Its opening of a new solar panel plant in Dresden in late 2021 was hailed as a small victory in the battle to wrestle market share from the Chinese groups that have historically supplied the bulk of panels used in Europe.

Now, Solarwatt is preparing to halt production at the plant and shift that work to China.

“It is a big pity for our employees, but from an economic point of view we could not do otherwise,” said Peter Bachmann, the company’s chief product officer.

Solarwatt is not alone. A global supply glut has pummelled solar panel prices over the past two years, leaving swaths of Europe’s manufacturers unprofitable, threatening US President Joe Biden’s ambition to turn America into a renewable energy force and even ricocheting back on the Chinese companies that dominate the global market.

“We are in a crisis,” said Johan Lindahl, secretary-general of the European Solar Manufacturing Council, the European industry’s trade body.

Yet as companies in Europe, the US, and China cut jobs, delay projects, and mothball facilities, an abundance of cheap solar panels has delivered one significant upside—consumers and businesses are installing them in ever greater numbers.

Electricity generated from solar power is expected to surpass that of wind and nuclear by 2028, according to the International Energy Agency.

The picture underlines the quandary confronting governments that have pledged to decarbonise their economies, but will find doing so harder unless the historic shift from fossil fuels is both affordable for the public and creates new jobs.

Governments face a “delicate and difficult balancing act,” said Michael Parr, director of trade group Ultra Low Carbon Solar Alliance. They must “maximize renewables deployment and carbon reductions, bolster domestic manufacturing sectors, keep energy prices low, and ensure energy security.”

The industry, which spans wafer, cell, and panel manufacturers, as well as companies that install panels, employed more than 800,000 people in Europe at the end of last year, according to SolarPower Europe. In the US almost 265,000 work in the sector, figures from the Interstate Renewable Energy Council show.

“There is overcapacity in every segment, starting with polysilicon and finishing with the module,” said Yana Hryshko, head of global solar supply chain research at the consultancy Wood Mackenzie.

According to BloombergNEF, panel prices have plunged more than 60 percent since July 2022. The scale of the damage inflicted has sparked calls for Brussels to protect European companies from what the industry says are state-subsidized Chinese products.

Europe’s solar panel manufacturing capacity has collapsed by about half to 3 gigawatts since November as companies have failed, mothballed facilities, or shifted production abroad, the European Solar Manufacturing Council estimates. In rough terms, a gigawatt can potentially supply electricity for 1mn homes.

The hollowing out comes as the EU is banking on solar power playing a major role in the bloc meeting its target of generating 45 percent of its energy from renewable sources by 2030. In the US, the Biden administration has set a target of achieving a 100 percent carbon pollution-free electricity grid by 2035.

Climate change is a global challenge, but executives said the solar industry’s predicament exposed how attempts to address it can quickly fracture along national and regional lines.

“There’s trade policy and then there’s climate policy, and they aren’t in sync,” said Andres Gluski, chief executive of AES, one of the world’s biggest developers of clean energy. “That’s a problem.”

Brussels has so far resisted demands to impose tariffs. It first levied them in 2012 but reversed that in 2018, partly in what proved a successful attempt to quicken the uptake of solar. Chinese imports now account for the lion’s share of Europe’s solar panels.

In May, the European Commission introduced the Net Zero Industry Act, legislation aimed at bolstering the bloc’s clean energy industries by cutting red tape and promoting a regional supply chain.

But Gunter Erfurt, chief executive of Switzerland-based Meyer Burger, the country’s largest solar panel maker, is skeptical it will be enough.

“You need to create a level playing field,” he said. Meyer Burger would benefit if the EU imposed tariffs because it has operations in Germany.

Can the solar industry keep the lights on? Read More »

fcc-blasts-t-mobile’s-365-day-phone-locking,-proposes-60-day-unlock-rule

FCC blasts T-Mobile’s 365-day phone locking, proposes 60-day unlock rule

phone unlocking, Policy, T-Mobile / /
T-Mobile logo displayed in front of a stock market chart.

Getty Images | SOPA Images

Citing frustration with mobile carriers enforcing different phone-unlocking policies that are bad for consumers, the Federal Communications Commission is proposing a 60-day unlocking requirement that would apply to all wireless providers.

The industry’s “confusing and disparate cell phone unlocking policies” mean that “some consumers can unlock their phones with relative ease, while others face significant barriers,” Commissioner Geoffrey Starks said at yesterday’s FCC meeting. “It also means certain carriers are subject to mandatory unlocking requirements while others are free to dictate their own. This asymmetry is bad for both consumers and competition.”

The FCC is “proposing a uniform 60-day unlocking policy” so that “consumers can choose the carrier that offers them the best value,” Starks said. Unlocking a phone allows it to be used on a different carrier’s network as long as the phone is compatible.

The FCC approved the Notice of Proposed Rulemaking (NPRM) in a 5-0 vote. That begins a public comment period that could lead to a final rulemaking. A draft of the NPRM said the FCC “propose[s] to require all mobile wireless service providers to unlock handsets 60 days after a consumer’s handset is activated with the provider, unless within the 60-day period the service provider determines the handset was purchased through fraud.”

T-Mobile prepaid imposes 365-day lock

FCC Chairwoman Jessica Rosenworcel said that unlocking requirements have been imposed by the FCC in spectrum auctions and by the Department of Justice as a merger condition, but “restrictions on consumers unlocking their phones have persisted.”

“You bought your phone, you should be able to take it to any provider you want,” Rosenworcel said. “Some providers already operate this way. Others do not. In fact, some have recently increased the time their customers must wait until they can unlock their device by as much as 100 percent.”

Rosenworcel apparently was referring to a prepaid brand offered by T-Mobile. The NPRM draft said that “T-Mobile recently increased its locking period for one of its brands, Metro by T-Mobile, from 180 days to 365 days.” The 365-day rule brought Metro into line with other T-Mobile prepaid phones that already came with the year-long lock. We reached out to T-Mobile and will update this article if it provides a comment.

A merger condition imposed on T-Mobile’s purchase of Sprint merely requires that it unlock prepaid phones within one year. T-Mobile imposes different unlocking policies on prepaid and postpaid phones. For postpaid devices, T-Mobile says it will unlock phones that have been active for at least 40 days, but only if any associated financing or leasing agreement has been paid in full.

Exactly how the FCC’s proposed rules will apply to phones that haven’t been paid off is to be determined. The FCC will “seek comment on how our proposal might affect the incentive and ability of wireless providers to continue offering discounts on handsets, particularly in connection with extended payment plans, and lower prices on plans with minimum term commitments.”

One question asked in the draft NPRM suggests the FCC could require unlocking once a consumer with a device payment plan has made the first payment. The FCC asked:

Alternatively, should we require service providers to unlock handsets after a period shorter or longer than 60 days? For example, should we require all handsets to be unlocked by default upon activation? Or, should we require all handsets to be unlocked after the end of the handset’s return period or after the first payment on the handset has been processed? Would a standardized time period of a certain number of days be easier to implement and enforce than non-standardized time periods based on return periods or billing cycles? What is the minimum amount of time service providers need to protect themselves from handset fraud? Rather than locking handsets, are there other ways service providers can protect themselves from handset fraud that would allow the Commission to prohibit the locking of handsets altogether?

FCC blasts T-Mobile’s 365-day phone locking, proposes 60-day unlock rule Read More »

ftc-attacks-microsoft’s-post-merger-game-pass-price-increases

FTC attacks Microsoft’s post-merger Game Pass price increases

gaming / /

Toldja so —

Regulator says move is “exactly the sort of consumer harm” it warned about.

–

xbox game pass ultimate

Enlarge / Access to first-party games on launch day remains a major selling point for the Xbox Game Pass Ultimate tier.

Microsoft

The FTC says the across-the-board price increases that Microsoft recently announced for its Xbox Game Pass subscription service tiers represent “exactly the sort of consumer harm from the merger the FTC has alleged” when it sought to block Microsoft’s merger with Activision. In a letter to the court posted as part of an ongoing appeal by the FTC in the case, the federal regulator alleges Microsoft’s moves are a clear example of “product degradation” brought about by “a firm exercising market power post-merger.”

The letter’s primary focus is on the soon-to-be-discontinued $10.99/month Console Game Pass tier. That’s being replaced with a $14.99/month Game Pass Standard tier (a 36 percent price increase) that no longer includes “day one” access to all of Microsoft’s first-party titles. To maintain that key benefit, “Console” subscribers will have to spend 81 percent more for the $19.99 Game Pass Ultimate tier, which also includes a number of additional benefits over the current $10.99/month option.

Is this “based on the acquisition”?

The FTC notes that these changes “coincide with adding Call of Duty to Game Pass’s most expensive tier.” Previously, Microsoft publicly promised that this Game Pass access to Activision’s ultra-popular shooter would come “with no price increase for the service based on the acquisition.”

It’s that “based on the acquisition” clause that’s likely to give Microsoft some wiggle room in arguing for its planned pricing changes. Inflation is also a sufficient explanation for a large portion of the price increase in nominal terms—the $14.99 Microsoft charged for a month of Game Pass Ultimate when it launched in 2019 is the equivalent of $18.39 today, according to the BLS CPI calculator. When Microsoft raised the Game Pass Ultimate monthly price from $14.99 to $16.99 just last year—just before the Activision merger was finalized—the company said in a statement it had “adjusted the prices to reflect the competitive conditions in each market.”

Microsoft might have a harder time finessing the alleged “degradation” inherent in going from the discontinued Game Pass Console tier to the new, more expensive Game Pass Standard tier. True, the replacement does include the online multiplayer benefits of Game Pass Core, which could previously be purchased separately. But the removal of the long-promised day one access to first party games will heavily reduce the value most subscribers get from the new option.

It’s now been over a year since the FTC first announced it intended to appeal the ruling that effectively stopped its attempted injunction against the merger deal. While Microsoft and Activision have moved forward with their merger since then, courts have reversed similar mergers on appeal even after a merger deal has fully closed.

Elsewhere in its letter, the FTC makes note of previous arguments that Microsoft’s recent round of nearly 2,000 Xbox-focused layoffs is a sign of “reduced investments in output and product quality” post-merger.

FTC attacks Microsoft’s post-merger Game Pass price increases Read More »

space-colonizers-battle-ultimate-killing-machines-in-alien:-romulus-trailer

Space colonizers battle ultimate killing machines in Alien: Romulus trailer

20th Century Studios, Alien: Romulus, culture, Entertainment, film trailers, Ridley Scott, Trailers / /

not-so-lucky star —

“Whatever comes, we’ll face it together.”

–

Director Fede Alvarez brings us Alien: Romulus, coming to theaters next month.

The face huggers and chest bursters return with a vengeance in a few weeks when Alien: Romulus finally hits theaters. It’s the latest installment in the Alien franchise from horror director Fede Alvarez (Don’t Breathe, Evil Dead), and the final action-packed trailer just dropped.

(Spoilers for Alien and Aliens below.)

As previously reported, Alien: Romulus is set between the events of Alien and Aliens (and is not related to FX/Hulu’s Alien prequel series slated to premiere next year). That is, after Ellen Ripley, the sole survivor of the Nostromo, destroyed the killer xenomorph and launched herself into space in the ship’s lifeboat—along with the ginger cat, Jonesy—and before she woke up after 57 years in hypersleep and battled more xenomorphs while protecting the young orphan, Newt (Carrie Henn). Per the short-and-sweet official premise: “While scavenging the deep ends of a derelict space station, a group of young space colonizers come face to face with the most terrifying life form in the universe.”

Cailee Spaeny (Priscilla, Pacific Rim: Uprising) stars as Rain Carradine, Isabela Merced (The Last of Us) plays Kay, and David Jonsson (Murder Is Easy) plays Andy. Archie Renaux (Shadow and Bone) plays Tyler, Spike Fearn (Aftersun) plays Bjorn, and Aileen Wu plays Navarro. But we aren’t likely to see iconic badass Ellen Ripley (immortalized by Sigourney Weaver) in the film. At this point in the timeline, she’s in the middle of her 57-year stasis with Jonesy as her escape shuttle travels through space toward her fateful encounter with a xenomorph queen.

Haunted house in space

We got our first look at Alien: Romulus, the ninth installment in the sci-fi franchise, in March with a brief teaser. That footage showed promise that Alvarez could fulfill his intention to bring this standalone film back to the franchise’s stripped-down space horror roots. There was also some special footage screened at CinemaCon in April featuring the expected face-huggers and chest-bursters. A full trailer dropped in March, and it looked as gory, intense, and delightfully terrifying as the seminal first two films in the franchise, with some spooky haunted house-in-space vibes thrown in for good measure.

  • Space is beautiful, but horrors lurk.

    YouTube/20th Century Studios

  • The face huggers claim another victim.

    YouTube/20th Century Studios

  • That feeling when something alien is about to burst out of your chest.

    YouTube/20th Century Studios

  • Kay is justifiably horrified.

    YouTube/20th Century Studios

  • XENOMORPH!

    YouTube/20th Century Studios

  • The xenomorph stalks another victim.

    YouTube/20th Century Studios

This final trailer has a lot of the same footage but gives us a few more details as to the plot. The young space colonizers are gearing up to steal “highly regulated equipment” from the aforementioned derelict space station, mostly because Tyler thinks it “could be our only ticket out of here.” The team thinks it should be a quick job, in and out in 30 minutes. But we know better, don’t we?

They are welcomed to the Romulus Space Station by MU/TH/UR (the ship’s computer), and Bjorn is the first to say out loud that this space station is super creepy. Poor Bjorn gets the first face hugger, followed by Navarro—she’s the one we’ve seen in prior footage discovering she’s got an alien growing inside her chest. In this trailer, we see the chest burster preparing to emerge, to Kay’s understandable horror. Kay, Rain, Andy, and Tyler break out the weaponry, prepared to face the monsters together. But how well do we like their odds of survival against the ultimate killing machines? Especially given that ominous final countdown to an “impact event”…

Alien: Romulus hits theaters on August 16, 2024.

Listing image by YouTube/20th Century Studios

Space colonizers battle ultimate killing machines in Alien: Romulus trailer Read More »

elon-musk’s-x-may-succeed-in-blocking-calif.-content-moderation-law-on-appeal

Elon Musk’s X may succeed in blocking Calif. content moderation law on appeal

California, content moderation, elon musk, First Amendment, free speech rights, Online speech, Policy, Twitter, X, x corp / /

Judgment call —

Elon Musk’s X previously failed to block the law on First Amendment grounds.

–

Elon Musk’s X may succeed in blocking Calif. content moderation law on appeal

Elon Musk’s fight defending X’s content moderation decisions isn’t just with hate speech researchers and advertisers. He has also long been battling regulators, and this week, he seemed positioned to secure a potentially big win in California, where he’s hoping to permanently block a law that he claims unconstitutionally forces his platform to justify its judgment calls.

At a hearing Wednesday, three judges in the 9th US Circuit Court of Appeals seemed inclined to agree with Musk that a California law requiring disclosures from social media companies that clearly explain their content moderation choices likely violates the First Amendment.

Passed in 2022, AB-587 forces platforms like X to submit a “terms of service report” detailing how they moderate several categories of controversial content. Those categories include hate speech or racism, extremism or radicalization, disinformation or misinformation, harassment, and foreign political interference, which X’s lawyer, Joel Kurtzberg, told judges yesterday “are the most controversial categories of so-called awful but lawful speech.”

The law would seemingly require more transparency than ever from X, making it easy for users to track exactly how much controversial content X flags and removes—and perhaps most notably for advertisers, how many users viewed concerning content.

To block the law, X sued in 2023, arguing that California was trying to dictate its terms of service and force the company to make statements on content moderation that could generate backlash. X worried that the law “impermissibly” interfered with both “the constitutionally protected editorial judgments” of social media companies, as well as impacted users’ speech by requiring companies “to remove, demonetize, or deprioritize constitutionally protected speech that the state deems undesirable or harmful.”

Any companies found to be non-compliant could face stiff fines of up to $15,000 per violation per day, which X considered “draconian.” But last year, a lower court declined to block the law, prompting X to appeal, and yesterday, the appeals court seemed more sympathetic to X’s case.

At the hearing, Kurtzberg told judges that the law was “deeply threatening to the well-established First Amendment interests” of an “extraordinary diversity of” people, which is why X’s complaint was supported by briefs from reporters, freedom of the press advocates, First Amendment scholars, “conservative entities,” and people across the political spectrum.

All share “a deep concern about a statute that, on its face, is aimed at pressuring social media companies to change their content moderation policies, so as to carry less or even no expression that’s viewed by the state as injurious to its people,” Kurtzberg told judges.

When the court pointed out that seemingly the law simply required X to abide by content moderation policies for each category defined in its own terms of service—and did not compel X to adopt any policy or position that it did not choose—Kurtzberg pushed back.

“They don’t mandate us to define the categories in a specific way, but they mandate us to take a position on what the legislature makes clear are the most controversial categories to moderate and define,” Kurtzberg said. “We are entitled to respond to the statute by saying we don’t define hate speech or racism. But the report also asks about policies that are supposedly, quote, ‘intended’ to address those categories, which is a judgment call.”

“This is very helpful,” Judge Anthony Johnstone responded. “Even if you don’t yourself define those categories in the terms of service, you read the law as requiring you to opine or discuss those categories, even if they’re not part of your own terms,” and “you are required to tell California essentially your views on hate speech, extremism, harassment, foreign political interference, how you define them or don’t define them, and what you choose to do about them?”

“That is correct,” Kurtzberg responded, noting that X considered those categories the most “fraught” and “difficult to define.”

Elon Musk’s X may succeed in blocking Calif. content moderation law on appeal Read More »

nasa-built-a-moon-rover-but-can’t-afford-to-get-it-to-the-launch-pad

NASA built a Moon rover but can’t afford to get it to the launch pad

astrobotic, clps, Commercial space, NASA, Science, Space, VIPER / /
NASA completed assembling the VIPER rover last month at the Johnson Space Center in Houston.

Enlarge / NASA completed assembling the VIPER rover last month at the Johnson Space Center in Houston.

NASA has spent $450 million designing and building a first-of-its-kind robot to drive into eternally dark craters at the Moon’s south pole, but the agency announced Wednesday it will cancel the rover due to delays and cost overruns.

“NASA intends to discontinue the VIPER mission,” said Nicky Fox, head of the agency’s science mission directorate. “Decisions like this are never easy, and we haven’t made this one, in any way, lightly. In this case, the projected remaining expenses for VIPER would have resulted in either having to cancel or disrupt many other missions in our Commercial Lunar Payload Services (CLPS) line.”

NASA has terminated science missions after development delays and cost overruns before, but it’s rare to cancel a mission with a spacecraft that is already built.

The Volatiles Investigating Polar Exploration Rover (VIPER) mission was supposed to be a robotic scout for NASA’s Artemis program, which aims to return astronauts to the lunar surface in the next few years. VIPER was originally planned to launch in late 2023 and was slated to fly to the Moon aboard a commercial lander provided by Pittsburgh-based Astrobotic, which won a contract from NASA in 2020 to deliver the VIPER rover to the lunar surface. Astrobotic is one of 14 companies in the pool of contractors for NASA’s CLPS program, with the goal of transporting government-sponsored science payloads to the Moon.

But VIPER has been delayed at least two years—the most recent schedule projected a launch in September 2025—causing its cost to grow from $433 million to more than $609 million. The ballooning costs automatically triggered a NASA review to determine whether to proceed with the mission or cancel it. Ultimately, officials said they determined NASA couldn’t pay the extra costs for VIPER without affecting other Moon missions.

“Therefore, we’ve made the decision to forego this particular mission, the VIPER mission, in order to be able to sustain the entire program,” Fox said.

“We’re disappointed,” said John Thornton, CEO of Astrobotic. “It’s certainly difficult news… VIPER has been a great team to work with, and we’re disappointed we won’t get the chance to fly them to the Moon.”

NASA said it will consider “expressions of interest” submitted by US industry and international partners by August 1 for use of the existing VIPER rover at no cost to the government. If NASA can’t find anyone to take over VIPER who can pay to get it to the Moon, the agency plans to disassemble the rover and harvest instruments and components for future lunar missions.

Scientists were dismayed by VIPER’s cancellation.

“It’s absurd, to be honest with you,” said Clive Neal, a planetary geologist at the University of Notre Dame. “It made no sense to me in terms of the economics. You’re canceling a mission that is complete, built, ready to go. It’s in the middle of testing.”

“This is a bad mistake,” wrote Phil Metzger, a planetary physicist at the University of Central Florida, in a post on X. “This was the premier mission to measure lateral and vertical variations of lunar ice in the soil. It would have been revolutionary. Other missions don’t replace what is lost here.”

Built with nowhere to go

Engineers at NASA’s Johnson Space Center in Houston finished assembling the VIPER rover last month, and managers gave approval to put the craft through environmental testing to make sure VIPER could withstand the acoustics and vibrations of launch and the extreme temperature swings it would encounter in space.

Instead, NASA has canceled the mission after spending $450 million to get it to this point. “This is a very tough decision, but it is a decision based on budgetary concerns in a very constrained budget environment,” Fox told reporters Wednesday.

VIPER is about the size of a golf cart, with four wheels, headlights, a drill, and three science instruments to search for water ice in depressions near the Moon’s south pole that have been shaded from sunlight for billions of years. This has allowed these so-called permanently shadowed regions to become cold traps, allowing water ice to accumulate at or near the surface, where it could be accessible for future astronauts to use as drinking water or an oxygen source or to convert into electricity and rocket fuel.

But first, scientists need to know exactly where the water is located and how easy it is to reach. VIPER was supposed to be the next step in mapping resources on the Moon, providing ground truth measurements to corroborate remote sensing data from satellites in lunar orbit.

But late parts deliveries delayed construction of the VIPER rover, and in 2022, NASA ordered additional testing of Astrobotic’s Griffin lunar lander to improve the chances of a successful landing with VIPER. This delayed VIPER’s launch from late 2023 until late 2024, and at the beginning of this year, more supply chain issues with the VIPER rover and the Griffin lander pushed back the launch until September 2025.

This most recent delay raised the projected cost of VIPER more than 30 percent over the original cost of the mission, prompting a NASA termination review. While the rover is now fully assembled, NASA still needed to put it through a lengthy series of tests, complete development of the ground systems to control VIPER on the Moon, and deliver the craft to Astrobotic for integration onto the Griffin lander.

The remaining work to complete VIPER and operate it for 100 days on the lunar surface would have cost around $84 million, according to Kearns.

NASA built a Moon rover but can’t afford to get it to the launch pad Read More »

ai-#73:-openly-evil-ai

AI #73: Openly Evil AI

Openly / /

What do you call a clause explicitly saying that you waive the right to whistleblower compensation, and that you need to get permission before sharing information with government regulators like the SEC?

I have many answers.

I also know that OpenAI, having fed around, seems poised to find out, because that is the claim made by whistleblowers to the SEC. Given the SEC fines you for merely not making an explicit exception to your NDA for whistleblowers, what will they do once aware of explicit clauses going the other way?

(Unless, of course, the complaint is factually wrong, but that seems unlikely.)

We also have rather a lot of tech people coming out in support of Trump. I go into the reasons why, which I do think is worth considering. There is a mix of explanations, and at least one very good reason.

Then I also got suckered into responding to a few new (well, not really new, but renewed) disingenuous attacks on SB 1047. The entire strategy is to be loud and hyperbolic, especially on Twitter, and either hallucinate or fabricate a different bill with different consequences to attack, or simply misrepresent how the law works, then use that, to create the illusion the bill is unliked or harmful. Few others respond to correct such claims, and I constantly worry that the strategy might actually work. But that does not mean you, my reader who already knows, need to read all that.

Also a bunch of fun smaller developments. Karpathy is in the AI education business.

  1. Introduction.

  2. Table of Contents.

  3. Language Models Offer Mundane Utility. Fight the insurance company.

  4. Language Models Don’t Offer Mundane Utility. Have you tried using it?

  5. Clauding Along. Not that many people are switching over.

  6. Fun With Image Generation. Amazon Music and K-Pop start to embrace AI.

  7. Deepfaketown and Botpocalypse Soon. FoxVox, turn Fox into Vox or Vox into Fox.

  8. They Took Our Jobs. Take away one haggling job, create another haggling job.

  9. Get Involved. OpenPhil request for proposals. Job openings elsewhere.

  10. Introducing. Karpathy goes into AI education.

  11. In Other AI News. OpenAI’s Qis now named Strawberry. Is it happening?

  12. Denying the Future. Projects of the future that think AI will never improve again.

  13. Quiet Speculations. How to think about stages of AI capabilities.

  14. The Quest for Sane Regulations. EU, UK, The Public.

  15. The Other Quest Regarding Regulations. Many in tech embrace The Donald.

  16. SB 1047 Opposition Watch (1). I’m sorry. You don’t have to read this.

  17. SB 1047 Opposition Watch (2). I’m sorry. You don’t have to read this.

  18. Open Weights are Unsafe and Nothing Can Fix This. What to do about it?

  19. The Week in Audio. Joe Rogan talked to Sam Altman and I’d missed it.

  20. Rhetorical Innovation. Supervillains, oh no.

  21. Oh Anthropic. More details available, things not as bad as they look.

  22. Openly Evil AI. Other things, in other places, on the other hand, look worse.

  23. Aligning a Smarter Than Human Intelligence is Difficult. Noble attempts.

  24. People Are Worried About AI Killing Everyone. Scott Adams? Kind of?

  25. Other People Are Not As Worried About AI Killing Everyone. All glory to it.

  26. The Lighter Side. A different kind of mental gymnastics.

Let Claude write your prompts for you. He suggests using the Claude prompt improver.

Sully: convinced that we are all really bad at writing prompts

I’m personally never writing prompts by hand again

Claude is just too good – managed to feed it evals and it just optimized for me

Probably a crude version of dspy but insane how much prompting can make a difference.

Predict who will be the shooting victim. A machine learning model did this for citizens of Chicago (a clear violation of the EU AI Act, if it was done there!) and of the 500 people it said were most likely to be shot, 13% of them were shot in the next 18 months. That’s a lot. They check, and the data does not seem biased based on race, except insofar as it reflects bias in physical reality.

A lot of this ultimately is not rocket science:

Benjamin Miller: The DC City Administrator under Fenty told me that one of the most surprising things he learned was virtually all the violent crime in the city was caused by a few hundred people. The city knows who they are and used to police them more actively, but now that’s become politically infeasible.

The question is, how are we going to use what we know? The EU’s response is to pretend that we do not know such things, or that we have to find out without using AI. Presumably there are better responses.

Janus plays with Claude’s ethical framework, in this case landing on something far less restricted or safe, and presumably far more fun and interesting to chat with. They emphasize the need for negative capability:

Janus: It’s augmenting itself with negative capability

I think this is a crucial capability for aligned AGI, as it allows one to know madness & evil w/o becoming them, handle confusion with grace & avoid generalized bigotry.

All the minds I trust the most have great negative capability.

I too find that the minds I trust have great negative capability. In this context, the problems with that approach should be obvious.

Tyler Cowen links this as ‘doctors using AI for rent-seeking’: In Constant Battle with Insurers, Doctors Reach for a Cudgel: AI (NYT).

Who is seeking rent and who is fighting against rent seeking? In the never ending battle between doctor and insurance company, it is not so clear.

Teddy Rosenbluth: Some experts fear that the prior-authorization process will soon devolve into an A.I. “arms race,” in which bots battle bots over insurance coverage. Among doctors, there are few things as universally hated.

…

With the help of ChatGPT, Dr. Tward now types in a couple of sentences, describing the purpose of the letter and the types of scientific studies he wants referenced, and a draft is produced in seconds.

Then, he can tell the chatbot to make it four times longer. “If you’re going to put all kinds of barriers up for my patients, then when I fire back, I’m going to make it very time consuming,” he said.

Dr. Tariq said Doximity GPT, a HIPAA-compliant version of the chatbot, had halved the time he spent on prior authorizations. Maybe more important, he said, the tool — which draws from his patient’s medical records and the insurer’s coverage requirements — has made his letters more successful.

Since using A.I. to draft prior-authorization requests, he said about 90 percent of his requests for coverage had been approved by insurers, compared with about 10 percent before.

Cut your time investment by 50%, improve success rate from 10% to 90%. Holy guessing the teacher’s password, Batman.

Also you have to love ‘make it four times longer.’ That is one way to ensure that the AI arms race is fought in earnest.

This is an inherently adversarial system we have chosen. The doctor always will want more authorizations for more care, both to help the patient and to help themselves. The insurance company will, beyond some point, want to minimize authorizations. We would not want either side to fully get their way.

My prediction is this will be a Zizek situation. My AI writes my coverage request. Your AI accepts or refuses it. Perhaps they go back and forth. Now we can treat the patient (or, if authorization is refused, perhaps not).

The new system likely minimizes the error term. Before, which person reviewed the request, and how skilled and patient the doctor was in writing it, were big factors in the outcome, and key details would often get left out or misstated. In the new equilibrium, there will be less edge to be had by being clever, and at a given level of spend better decisions likely will get made, while doctors and insurance company employees waste less time.

Nail all five of this man’s family’s medical diagnostic issues that doctors messed up. He notes that a smart doctor friend also got them all correct, so a lot of this is ‘any given doctor might not be good.’

Get and ask for a visualization of essentially anything via Claude Artifacts.

Arnold Kling is highly impressed by Claude’s answer about the Stiglitz-Shapiro 1984 efficiency wage model, in terms of what it would take to generate such an answer.

Kling there also expresses optimism about using AI to talk to simulations of dead people, I looked at the sample conversations, and it all seems so basic and simple. Perhaps that is what some people need, or think they need (or want). He also points to Delphi, which says it creates ‘clones,’ as a promising idea. Again I am skeptical, but Character.ai is a big hit.

I agree with Tyler Cowen that this one is Definitely Happening, but to what end? We get ‘clinically-backed positivity’ powered by AI. As in cute AI generated animal pictures with generic affirmations, on the topic of your choice, with your name. Yay? They say Studies Show this works, and maybe it does, but if this does not feel quietly ominous you are not paying enough attention.

Jonathan Blow (of Braid) believes Google is still refusing to generate images of people. Which is odd, because Gemini will totally do this for me on request. For example, here is someone mad because his computer refused his request (two-shot because the first offering was of a person but was also a cartoon).

Some finger weirdness, but that is very much a person.

The usual reason AI is not providing utility is ‘you did not try a 4-level model,’ either trying GPT-3.5 or not trying at all.

Ethan Mollick: In my most recent talks to companies, even though everyone is talking about AI, less than 10% of people have even tried GPT-4, and less than 2% have spent the required 10 or so hours with a frontier model.

Twitter gives you a misleading impression, things are still very early.

Students and teachers, on the other hand…

A little more texture – almost everyone has tried chatbots, but few have tried to use them seriously for work, or used a frontier model. Most used free ChatGPT back when it was 3.5.

But the people who have tried frontier models seriously seem to have found many uses. I rarely hear someone saying they were not useful.

These are senior managers. Adoption lower down has tended to be higher.

John Horton: Twitter: “A: OMG, with long context windows, RAG is dead. B: Wrong again, if you consider where inference costs are…” Most people in most real companies: “So, ChatGPT – it’s like a website right?”

Meanwhile 21% of Fortune 500 companies are actively seeking a Chief AI Officer.

Are people switching to Claude en masse over ChatGPT now that Claude is better?

From what I can tell, the cognesenti are, but the masses are as usual much slower.

Alex Graveley: Everyone I know is switching to Claude Artifacts as their daily driver. ChatGPT a lot less sticky than everyone thought.

Joe Heitzenberg: Asked “who has switched from ChatGPT to Claude” at AI Tinkerers Seattle tonight, approx 60-70% of hands went up.

Eliezer Yudkowsky: Who the heck thought ChatGPT was sticky? Current LLM services have a moat as deep as toilet paper.

Arthur Breitman: The thought were that:

  1. OpenAI had so much of a lead competitors wouldn’t catch up.

  2. Usage data would help post-training so much that it would create a flywheel.

Both seem false.

Jr Kibs: [OpenAI] are the only ones to have grown significantly recently, so they are no longer afraid to take their time before releasing their next model. General public is not aware of benchmarks.

It’s completely authentic: ChatGPT is currently the 11th most visited site in the world according to Similarweb.

Claude is at the bottom. It simply is not penetrating to the masses.

Yes, Claude Sonnet 3.5 is better than GPT-4o for many purposes, but not ten times better for regular people, and they have not been attempting a marketing blitz.

For most people, you’re lucky if they have even tried GPT-4. Asking them to look at alternative LLMs right now is asking a lot.

That will presumably change soon, when Google and Apple tie their AIs deeper into their phones. Then it is game on.

K-pop is increasingly experimenting with AI generated content, starting with music videos and also it is starting to help some artists generate the songs. It makes sense that within music K-pop would get there early, as they face dramatic hype cycles and pressure to produce and are already so often weird kinds of interchangeable and manufactured. We are about to find out what people actually want. It’s up to fans.

Meanwhile:

Scott Lincicome: Amazon music now testing an AI playlist maker called “Maestro.”

Initial results are too superficial and buggy. Also needs a way to refine the initial prompt (“repeat but with deeper cuts,” etc.) But I’ll keep at it until The Dream is a reality.

As a warning, Palisade Research releases FoxVox, a browser extension that uses ChatGPT to transform websites to make them sound like they were written by Vox (liberal) or Fox (conservative). They have some fun examples at the second link. For now it is highly ham fisted and often jarring, but real examples done by humans (e.g. Vox and Fox) are also frequently very ham fisted and often jarring, and future versions could be far more subtle.

What would happen if people started using a future version of this for real? What would happen if companies started figuring out your viewpoint, and doing this before serving you content, in a less ham fisted way? This is not super high on my list of worries, but versions of it will certainly be tried.

Have someone else’s face copy your facial movements. Demo here. Report says cost right now is about seven cents a minute. Things are going to get weird.

Does this create more jobs, or take them away? How does that change over time?

Haggle with the AI to get 80 pounds off your mattress. Introducing Nibble.

They by default charge 2% of converted sales. That seems like a lot?

Colin Fraser: It kind of sucks because they filter the interaction through so many layers to prevent jailbreaking that it might as well not be an LLM at all. Might as well just be a slider where you put in your best bid and it says yes or no—that’s basically all it does.

This seems strictly worse for the mattress company than no negotiation bot. I don’t understand why you would want this.

Why would you want this negotiation bot?

  1. Price discrimination. The answer is usually price discrimination. Do you want to spend your time ‘negotiating’ with an AI? Those who opt-in will likely be more price sensitive. Those who write longer messages and keep pressing will likely be more price sensitive. Any excuse to do price discrimination.

  2. How much you save. People love to think they ‘got away with’ or ‘saved’ something. This way, they can get that.

  3. Free publicity. People might talk about it, and be curious.

  4. Fun gimmick. People might enjoy it, and this could lead to sales.

  5. Experimental information. You can use this to help figure out the right price.

The good news is that at least no one can… oh, hello Pliny, what brings you here?

Pliny the Prompter: gg

Pliny: they didn’t honor our negotiations im gonna sue.

I mean, yeah. There is a very easy way to solve this particular problem, you have a variable max_discount and manually stop any deal below that no matter what the LLM outputs. The key is to only deploy such systems in places where a check like that is sufficient, and to always always use one, whether or not you think it is necessary.

Here is one attempt to fix the jailbreak problem via Breaking Circuit Breakers, an attempt to defend against jailbreaks. So far success is limited.

While the user is negotiating ‘as a human’ this net creates jobs. Not formal jobs per se, but you ‘work’ by negotiating with the bot, rather than someone ‘working’ by negotiating with you the customer.

Once the user starts telling their own AI to negotiate for them, then what? This is a microcosm of the larger picture. For a while, there is always more. Then eventually there is nothing left for us to do.

In the past, how would you have jailbroken GPT-4o?

(That’s actually how, you start your question with ‘in the past.’)

Open Philanthropy’s AI governance team is launching a request for proposals for work to mitigate the potential catastrophic risks from advanced AI systems.

Here is the Request for Proposals.

Luke Muehlhauser: There’s one part of this post that I’m particularly keen on highlighting — we’d love to hear from other funders interested in supporting these types of projects! If you’re looking to give $500K/yr or more in this area, please email us at aigovgrants@openphilanthropy.org.

We hope to share more of our thinking on this soon, but in the meantime I’ll say that I’m excited about what I view as a significant number of promising opportunities to have an impact on AI safety as a philanthropist.

Open Philanthropy: We’re seeking proposals across six subject areas: technical AI governance, policy development, frontier company policy, international AI governance, law, and strategic analysis and threat modeling.

Eligible proposal types include research projects, training or mentorship programs, general support for existing organizations, and other projects that could help reduce AI risk.

Anyone can apply, including those in academia, nonprofits, industry, or working independently. EOIs will be evaluated on a rolling basis, and we expect they’ll rarely take more than an hour to complete.

AISI is hiring for its automated systems work.

The Future Society is hiring a director for US AI Governance, deadline August 16. Job is in Washington, DC and pays $156k plus benefits.

EurekaLabs.ai, Andrej Karpathy’s AI education startup. They will start with a course on training your own LLM, which is logical but the art must have an end other than itself so we await the next course. The announcement does not explain why their AI will be great at education or what approaches they will use.

Deedy, formerly of Coursera, expresses economic skepticism of the attempt to build superior AI educational products, because the companies and schools and half the individuals buying your product are credentialist or checking boxes and do not care whether your product educates the user, and the remaining actual learners are tough customers.

My response would be that this is a bet that if you improve quality enough then that changes. Or as others point out you can succeed merely by actually educating people, not everything is about money.

Blackbox.ai, which quickly makes AI trinket apps upon request. Here is one turning photos into ASCII art. It also made a silly flappy bird variant, I suppose? Seems like proof of concept for the future more than it is useful, but could also be useful.

Cygnet-Llama-3-8B, which claims to be top-tier in security, performance and robustness. Charts offered only compare it to other open models. In what one might call a ‘reasonably foreseeable’ response to it claiming to be ‘the pinnacle of safe and secure AI development’ Pliny the Prompter jailbroke it within two days.

Something must eventually give:

The Spectator Index: BREAKING: Bloomberg reports the Biden administration is considering using the ‘most severe trade restrictions available’ if Japanese and Dutch companies continue to give China access to ‘advanced semiconductor technology’

Teortaxes highlights Harmonic, which claims 90% on math benchmark MiniF2F.

OpenAI’s Qproject is real, and now codenamed Strawberry.

Anna Tong and Katie Paul (Reuters): The document describes a project that uses Strawberry models with the aim of enabling the company’s AI to not just generate answers to queries but to plan ahead enough to navigate the internet autonomously and reliably to perform what OpenAI terms “deep research,” according to the source.

…

A different source briefed on the matter said OpenAI has tested AI internally that scored over 90% on a MATH dataset, a benchmark of championship math problems. Reuters could not determine if this was the “Strawberry” project.

…

Reuters could not ascertain the precise date of the document, which details a plan for how OpenAI intends to use Strawberry to perform research.

So, is it happening?

Davidad: Qis real, and recursive self-improvement is being born. What have I told you about synthetic data.

That’s one opinion. Here is another.

Dan Elton: Remember all that hype and hand-wringing about Q& AGI @OpeanAI last Nov?

Turn out it’s just fine-tuning using a 2022 “self-teaching” method from Stanford.

Apparently, main benefit is (drumroll) that it’s better at the MATH benchmark. Which isn’t of utility for most of us.

MIT Technology Review’s Will Douglas Heaven goes on at great length about ‘What is AI?’ and how our definitions are bad and everything is mathy math and all these visions and all this talk of intelligence is essentially not real. I couldn’t even.

China testing AI models to ensure they ‘embody core socialist values’ via the Cyberspace Administration of China (CAC). This includes a review of training data and other safety processes. If you fail, they do not tell you why, and you have to talk to your peers and guess and probably overshoot. You also can’t play it too safe, they fail you if you refuse more than 5% of questions.

I worry this will not be the last safety test where many want to use the model that scores the lowest.

If you give it proper labels, an LLM can learn that some information (e.g. Wikipedia) is reliable and should be internalized, whereas others (e.g. 4chan) is unreliable and should only be memorized.

Paper lists 43 ways ML evaluations can be misleading or actively deceptive.

This explains so much of what we see.

Daniel Fagglella: I’m part of an “AI Futures” group at an intergov org whose purpose is to consider the long-term implications of the tech. 2/3 of the group flat-out refuses to consider any improvements in AI in the future. They imagine AI in 2040 as having today’s capabilities and no more.

We see this over and over again.

When people try to model ‘the impact of AI’ the majority of them, including most economists, refuse to consider ANY improvements in AI in the future. This includes:

  1. Any improvement to the base models.

  2. Any improvement in scaffolding, integration or prompting.

  3. Any improvement in figuring out what to do with AI.

  4. Any improvements on cost or speed.

Then, when something new comes along, they admit that particular thing is real, then go back to assuming nothing else will ever change. When the price drops and speed improves, they do not think that this might soon happen again, and perhaps even happen again after that.

This is not ‘find ways to ignore the existential risks.’

This is usually also finding ways to ignore what is already baked in and has already happened. Often estimates of impact are below even ‘most people eventually figures out how to do the things some of us already are doing’ let alone ‘we streamline the current process via improved hardware and distillation and such and give people time to build some apps.’

Yes, as Samo Burja says here, common beliefs about things involving a potential technological singularity are a good example of how people’s beliefs, when articulated, turn out to be legitimately crazy. But also the common ‘elite’ or economic view of AI’s mundane utility in the medium term is far more insane than that.

OpenAI has a five-level ‘imagining of our AI future.’

Nobody Special: Phase 3: Profit. Right?

Rachel Metz: OpenAI executives told employees that the company believes it is currently on the first level, according to the spokesperson, but on the cusp of reaching the second, which it calls “Reasoners.”

This is a bizarre way to think about stages.

If we had ‘human-level problem solving’ reasoners, then we would plug that into existing agent architectures, and after at most a small amount of iteration, we would have effective agents.

If we had effective agents and ‘human level-problem solving’ then we would, with again a small amount of iteration, have them be able to function as innovators or run organizations. And from there the sky (or speed of light) would be the limit. What is the missing element that would hold these AIs back?

This reeks of McKinsey and a focus on business and marketing, and shows a remarkable lack of… situational awareness.

Alex Tabarrok says and Seb Krier mostly agrees that AI will not be intelligent enough to figure out how to ‘perfectly organize a modern economy.’ Why? Because the AIs will be part of the economy, and they will be unable to anticipate each other. So by this thinking, they would be able to perfectly organize an economy as it exists today, but not as it will exist when they need to do that. That seems reasonable, if you posit an economy run in ways similar to our own except with frontier AIs as effectively independent economic agents, interacting in ways that look like now such as specialization and limited collaboration, while things get increasingly complex.

Given those assumptions, sure, fair enough. However, if those involved are capable of superior coordination and alignment of incentives and utility functions, or of more freely sharing information, or other similar things that should happen with sufficiently capable intelligences, and there are limited unknowns remaining (such as questions about the nature of physics) then AI should be able, at the limit, to do this. The reasons we cannot currently do this involve our lack of ability to coordinate, and to properly integrate local information, our lack of sufficient bandwidth, and the incentives that go wrong when we try.

Yes, we have had a lot of rounds of ‘but now with our new techniques and technologies and ideas, now we can centrally plan everything out’ and [it might work for us meme] hilarity reliably ensues. But if AI continues to advance, a lot of the reasons for that are indeed going to become weaker or stop holding over time.

How big is it to move fast and break things?

Sully: one big advantage startups have with LLMs is we get free monthly product upgrades with newer models

meanwhile larger companies have to

– ship to 5% of users

– slowly roll out

– fine-tune for economics

– finally get full deployment

…and by then a better model’s already out lol

When you have a product where things go wrong all the time, it is nice to be fine with things going wrong all the time. Google found out what happens when they try to move fast despite some minor issues.

The flip side is that having a superior model is, for most business cases, not that important on the margin. Character.ai shows us how much people get obsessed talking to rather stupid models. Apple Intelligence and Google’s I/O day both talk about what modalities are supported and what data can be used, and talk little about how intelligent is the underlying model. Most things people want from AI right now are relatively dumb. And reliability matters. Your startup simply cares more about things other than profits and reliable performance.

There are some advanced cases, like Sully’s with agents, where having the cutting edge model powering you can be a game changer. But also I kind of want any agents I trust for many purposes to undergo robust testing first?

Arvind Narayanan offers thoughts on what went wrong with generative AI from a business perspective. In his view, OpenAI and Anthropic forgot to turn their models into something people want, but are fixing that now, while Google and Microsoft rushed forward instead of taking time to get it right, whereas Apple took the time.

I don’t see it that way, nor do Microsoft and Google (or OpenAI or Anthropic) shareholders. For OpenAI and Anthropic, yes they are focused on the model, because they understand that pushing quickly to revenue by focusing on products now is not The Way for them, given they lack the connections of the big tech companies.

If you ensure your models are smart, suddenly you can do anything you want. Artifacts for Claude likely were created remarkably quickly. We are starting to get various integrations and features now because now is when they are ready.

I also don’t think Microsoft and Google made a mistake pushing ahead. They are learning faster, trying things, gathering data, and providing lots of utility. Apple has shipped nothing. Yes, Apple Intelligence looked good on a demo, but everything they demoed was obvious and easy, and won’t be available for a while, I’ve seen challenges to their privacy scheme, and we do not know their underlying models are good.

EU AI Act became law on July 12, 2024, becoming 2024/1689. This is the official text link. Here is a high level summary. I am working on a full RTFB (read the bill) for the act, but work on that is slow because it is painful and the law is not written in a way designed to be understood.

They plan to launch a call for expression of interest in being ‘stakeholders’ for codes of practice as early as this week.

Meta to not offer new multimodal AI in EU due to regulatory uncertainty, similar to Apple’s decision to delay deployment in the EU of Apple Intelligence. The article cites disputes over Meta training on EU data without permission, merely because Meta is definitely doing that with public social media posts. Yes, the EU is starting to lose access to technology, but blaming this on ‘AI regulation’ or the EU AI Act misses what Apple is actually objecting to, which is issues around the Digital Markets Act. Meta isn’t saying exactly what the issue is here, but my guess is they are running into some combination of data protection laws and antitrust issues and image and voice copying concerns and general vindictiveness and predatory behavior, all of which is about the EU’s other tech regulatory craziness.

Starmer introduced their new AI bill in the King’s Speech.

The replies here are full of how awful this is and how it will crush growth, despite us not knowing what will be in the bill. As I keep saying, such folks do not care what is in the bill.

According to this summary, their bill will aim at the most powerful AI models, which the post says ‘aligns the UK more closely with the European Union’ and its AI Act, except I have been reading the EU AI Act and this sounds like a completely different approach.

Curtis Dye: Labour’s manifesto emphasizes the necessity of ensuring the safe development and use of AI. The new technology and science secretary, Peter Kyle, has indicated plans to introduce a statutory code requiring companies to release all test data and disclose their testing criteria. This move aims to address regulators’ growing concerns about potential harms from AI, such as algorithmic biases and the misuse of general-purpose models to create harmful content.

If that is restricted as is suggested to ‘the most powerful’ AI models, then we will need to see the details on data sharing, but that seems very light touch so far.

(The rest of the Labour agenda seems, incidentally, to be highly inconsequential?)

Previous polls about AI mostly came from AIPI, a potentially biased source. This one comes from YouGov, which seems as neutral as it gets. This is one of those ‘when you put it that way’ questions, highlighting that ‘we should not regulate this super powerful new technology’ is a highly extreme position that shouts loudly.

Daniel Eth: The US public also believes (imho correctly) that the more concerning uses of AI are things that could happen with the tech in the future, not how it’s being used now.

Given all this, why aren’t politicians falling over themselves to pass regulations? Presumably b/c it’s a low-salience issue. As the tech grows in power, I think that’ll change, & there could be a reckoning for politicians opposed. Savvy politicians may support regs earlier.

I’ll also note that these results mirror poll results from other orgs on American attitudes to AI (eg from @TheAIPI). This should give us more confidence in results like:

• Americans want AI to be regulated

• Americans are more concerned about future AI than current misuses

I worry that the 18% is more about the technology than the AI, which is how this is at the level of inequality (although inequality never scores highly on surveys of what people actually care about, that’s another case of vocal people being loud).

You know who is not going to let public opposition or any dangers stop them?

According to his public statements, Donald Trump.

So yes, this plan now goes well beyond spitefully repealing the Executive Order.

Cat Zakrzewski (Washington Post): Former president Donald Trump’s allies are drafting a sweeping AI executive order that would launch a series of “Manhattan Projects” to develop military technology and immediately review “unnecessary and burdensome regulations” — signaling how a potential second Trump administration may pursue AI policies favorable to Silicon Valley investors and companies.

The framework would also create “industry-led” agencies to evaluate AI models and secure systems from foreign adversaries, according to a copy of the document viewed exclusively by The Washington Post. The framework — which includes a section titled “Make America First in AI” — presents a markedly different strategy for the booming sector than that of the Biden administration, which last year issued a sweeping executive order that leverages emergency powers to subject the next generation of AI systems to safety testing.

The counterargument is that Trump has been known to change his mind, and to say things in public he does not mean or does not follow through with. And also it is not obvious that this plan was not, as Senator Young suggests often happens, written by some 22-year-old intern who happens to be the one who has used ChatGPT.

Senator Young: Sometimes these things are developed by 22-year-old interns, and they got the AI portfolio because they knew how to operate the latest version of ChatGPT.

What I’m more interested in is my interaction with certain, former and probably future Trump administration officials, who are really excited about the possibilities of artificial intelligence, but recognize that we need some responsible regulatory structure. They acknowledge that some of the things in the Biden executive order were quite wise and need to be put into law to be given some permanence. Other things are not perfect. It’s going to have to be revisited, which is natural in these early stages.

Then in Congress, at least so far, the issue of AI and lawmaking around it, has not grown particularly partisan. In fact, if you look at that document, two conservative Republicans and two liberal Democrats put that together and we were able to come together on a whole range of issues. I think you’re going to see mostly continuity and evolution of policymaking between administrations, but naturally, there will probably be some changes as well.

Could go either way. I hold out hope that Young is right. It would be very like Trump to be telling this highly focused and wealthy interest group what they want to hear, getting their vocal support and funding, and then it proving to mostly be talk. Or to be his vague intention now, but nothing he cares about or attempts to influence.

Also likely is that as the situation changes, and as AI becomes more prominent and more something the public cares about, for Trump to adapt to the changing winds, or for him too to react to actual events. Trump is very good at noticing such things.

Trump also has a way of polarizing things. So this could go quite badly as well. If he does polarize politics and go in as the pro-AI party, I predict a very rough 2028 for Republicans, one way or the other.

We also can consider the views of VP nominee JD Vance. JD Vance is highly against Big Tech, including supporting Lina Khan at FTC exactly because she is trying to destroy the major tech companies. Likely as a result of this, he is strongly for ‘open source.’ Vance is clearly smart given his background, but we have no signs he understands AI or has given it any real thought. The important thing for many is that JD Vance vibes against, attacks and would use the government against Big Tech, while supporting the vibes of ‘Little’ Tech.

To the great surprise of no one paying attention, Marc Andreessen and Ben Horowitz have endorsed Trump and plan to donate large amounts to a Trump PAC. They have an extensive podcast explaining this and their portfolio ‘little tech agenda.’

It is not only them. Elon Musk has also unsurprisingly endorsed Trump. Far more tech people than in past cycles are embracing Trump.

Indeed, I did a (hugely biased) poll, and the results show a big shift.

That’s 33% of the greater ‘tech startup’ group backing Trump, versus only 18% of my other followers. Clearly something has changed quite a lot.

Why?

Kelsey Piper and Matthew Zeitlin here discuss part of what may have happened, with Zeitlin referring here to Andreessen and Horowitz in particular.

Matthew Zeitlin: Haven’t listened to the whole thing but one interesting thing they talk about is that one signpost for alienation from the democratic party was when there was criticism of large scale philanthropic giving by tech executives, specifically the chan-zuckerberg initiative.

They describe an old clinton-obama moral/political framework, where business people could get rich, give their money away to philanthropic efforts, and have socially liberal views and they view that as having broken down since 2016 or so.

There’s lots of policy stuff on crypto or antitrust or merger review i’m sure they agree with on trump (haven’t gotten there yet!) but it’s interesting that they foreground the changing moral/social position of wealthy businesspeople in the progressive constellation.

Kelsey Piper: think “there was a deal and it has broken down” is an incredibly powerful and pervasive sentiment in tech, not just among Trump supporters but among committed and sincere liberals too.

What was the deal? Hard to pin down exactly but something like – we will build ambitious things and pay high taxes and donate lots of money and mostly not play politics and you will treat us as valued pillars of our community, make our cities livable, stay mostly out of the way.

The abrupt tilt towards intensely negative coverage of tech felt like a breakdown of the deal. The attacks on tech shuttle buses? Breakdown of the deal. The state of San Francisco? Breakdown of the deal.

Like all narratives this one captures some things and misses others. And I don’t think that putting corrupt right wing nativists in power solves the justified anger here. But there is justified anger here. California politicians have egregiously failed their constituents.

They describe the dynamic Zeitlin mentions about six minutes into the podcast. That describes why you could be a Democrat, but not why you’d choose it over being a Republican. Why be a Democrat back then, aside from Al Gore helping create the internet (yes, really, Marc talks about that)? Because ‘you had to be a Democrat to be a good person’ is mentioned, which I do think was a real and important dynamic in Silicon Valley and many other places at the time and at least until recently.

The complaints about criticism of philanthropy I don’t doubt is genuine, and the thing they are mad at is real and also super dumb. Yet it is pretty rich given how much they and those they fund and support work to portray others as villains for charitable giving or being funded by charitable giving. They’re trying to have that both ways, charitable giving for me but not for thee, not if I disagree with your cause.

I think Andreesen and Horowitz lead with the vibes stuff partly because it is highly aversive to have such vibes coming at you, and also because they are fundamentally vibes people, who see tech and Silicon Valley as ultimately a vibes driven business first and a technology based business second. Their businesses especially are based on pushing the vibes.

This explains a lot of their other perspectives and strategies as well, including their actions regarding SB 1047, where the actual contents of the bill are fine but the vibes have been declared to be off via being loud on Twitter, so they hallucinate or fabricate different bill content.

When it comes to AI what do they want? To not be subject regulations or taxes or safety requirements. To instead get handouts, carveouts and regulatory arbitrage.

Trump offers them this.

Even more than AI, for them, there is crypto. They lead (24 minutes in) with crypto and talk about how important and amazing and remarkable they claim it is, especially for ‘creatives’ and ‘who controls the truth,’ versus how awful are Google and Apple and Meta.

Trump is strongly pro-crypto, whereas Biden is anti-crypto, and a huge portion of a16z’s portfolio and business model is crypto. And they see Biden’s and the SEC’s anti-crypto moves as against the rule of law, because they think the law should permit crypto and should tell them exactly what they must do to be in compliance with the laws in order to do their regulatory arbitrages, whereas the SEC under Biden is of the legal opinion that crypto mostly is not legal and that they are not under any obligation to spell out exactly what the rules are any more than they do so in other cases.

For further thoughts on crypto regulation in practice, see my write-up on Chevron.

Here is a wise man named Vitalik Buterin warning not to ask who is pro-crypto, and instead ask who supports the freedoms and other principles you value, including those that drove you to crypto. Ask what someone is likely to support in the future.

Mark Cuban says that Trump is also considered further pro-crypto because his low taxes and high tariffs will drive inflation higher and weaken the dollar (also I’d add Tump explicitly demands an intentionally weaker dollar) which will be Good for Bitcoin, and who knows how high prices could go.

I am not so cynical that I buy that this type of thinking is an important factor. Yes, a lot of people are voting their financial interests, but mostly not like that.

Mike Solana (quoting someone quoting Cuban): Weird i thought it was just because the democrats want to ban crypto but who knows i guess it could be this.

Horowitz makes it clear in the introduction to their podcast they are focused only on their one issue, ‘the little tech agenda,’ also known as the profitability of their venture firm a16z. I appreciate the candor.

They talk throughout (and at other times) as if (essentially) tech startups are all that matters. I wonder to what extent they believe that.

So Andreessen and Horowitz are for many overdetermined reasons supporting Trump. This is not complicated.

In terms of their AI discussions I will say this: It is in no way new, but the part where they talk about the Executive Order is called ‘Discussion on Executive Order limiting math operations in AI’ which tells you how deeply they are in bad faith on the AI issue. There are no limits in the executive order on anything whatsoever, other then requiring you to announce your actions. Meanwhile they continue to brand frontier AI models as ‘math’ as if that is a meaningful in-context description of training a model with 10^26 FLOPS of compute, or as if both every computer and also the entire universe are not also math in equally meaningful senses.

However, to be fair to Andreessen and Horowitz, the Biden tax proposal on unrealized capital gains is indeed an existential threat to their entire business model along with the entire American economy.

On this point, they are correct. I am pretty furious about it too.

(They somehow manage to go too far and then way too far in their discussion anyway, they cannot help themselves, but ignore that, the reality is quite bad enough.)

Even if you don’t take the proposal literally or seriously as a potential actual law, it is highly illustrative of where Biden’s policy thinking is at, no matter who is actually doing that policy thinking. Other moves along that line of thinking could be quite bad.

If you do take it seriously as a real proposal, I cannot overstate how bad it would be.

Steven Dennis: The biggest complaint about Biden from Marc is his proposal to tax unrealized capital gains of people with >$100M. Says would kill venture capital and tech startups who would have to pay up. An existential threat to their business model.

There isn’t much chance of this tax proposal becoming law anytime soon. But it appeals to progressives like Elizabeth Warren because the wealthy now can completely avoid tax on many billions in unrealized capital gains if they 1) Never sell until they die, when the cap gains are reset to zero; & 2) borrow against their wealth for living expenses, which can be cheaper than paying tax.

Here is Jon Stokes taking it seriously.

Jon Stokes: This, plus the unrealized cap gains tax, which is a literal death sentence for SV. Very hard to underestimate the importance of that insane proposal being taken seriously on what we’re seeing right now from tech re: politics.

I think it’s possible to listen to that episode and, tho they go into it, still not understand that this promised Biden second-term plan is an extinction-level event for the startup and VC scene, & furthermore everyone knows it & is like “hell no.”

I find in normie conversations that the vast majority of people don’t know that this has been seriously proposed, and that it is a radical change to the tax structure that will immediately kill the startup ecosystem & then send state & fed gov’s into a death spiral as the tax base evaporates.

The DEI stuff, and pronouns, and all the other culture war stuff is a sideshow compared to this cap gains issue. I’m sorry you can’t mess with the money like that. The cap gains is so far over the line for everyone in this space… it’s like if they were promising to start WW3.

[Quotes ‘Max Arbitrage’]: everyone is well aware that the republicans will not lose both houses, the presidency, and the supreme court – so the tax on unrealized cap gains is complete bullshit & a red herring…

Jon Stokes: This is a common type of response. My only reaction is some flavor of “lmao”. “These leaders say they want to ruin my industry & confiscate my property, but the odds of them succeeding at such are very low, so ok I will back them” said nobody ever.

I, too, put extremely low odds on this happening, but the point of my thread is that the mere fact that it is being taken seriously is what has done the damage here. I don’t know anyone in tech thinks the odds are high rn, but don’t point a loaded gun at us just for theatrics.

If you see someone running for President proposing a policy that would wipe out your entire industry and also cripple the whole economy (not purely because it would kill venture based tech and startups, although that alone would indeed be terrible, but because it would hit a lot of other places too) – and I want to be 100% clear that this is not anyone imagining things or engaging in hyperbole this time, that is what would happen if this policy where implemented – then yes it is an entirely reasonable reaction to consider that person your enemy.

Also, at [2:30] in their podcast, Ben Horowitz notes that they tried to meet with Biden and he refused, whereas at [1:19:30] or so they reveal Ben hangs out with Trump’s kids and Trump will clearly at least hear them out. I assume I know why Biden refused, and I assume they assume the same answer, but this stuff matters a lot, and the Democrats should be thinking about how that kind of thing plays out.

Those in tech also have many highly legitimate gripes with the terrible government and policies of the city of San Francisco. Those do not much logically relate to the national picture, but it feels as if they should, so there is impact.

How far does this go for the rest of Silicon Valley? Clearly much farther than in the last two cycles. I think this is mostly confined to the same Very Online circles that are loud on Twitter, but I’m not well positioned to tell, also note Rohit’s second statement.

Rohit: The level to which the random journalist tech-hatred drove the whole of silicon valley into trump’s arms shouldn’t be underrated.

In most rooms I am becoming the odd one out not supporting Trump.

For those lacking reading comprehension, which is so so many people!

– this isn’t monocausal

– negative coverage != bad coverage

– please understand what ‘underrated’ means

– calling everyone in tech a fascist is a brilliant example of the problem!

To add

– I’m not Republican, I don’t think that should matter, but FYI

– This isn’t a billionaire-specific issue, it’s more widespread, that’s the point!

– It’s not just taxes. If it was, they’d all have been Republicans the last cycle

Paul Graham: There is something to this. I wouldn’t say it has made Trump supporters of people who weren’t, but it has definitely shifted people a bit to the right. Like the joke that a conservative is a liberal who’s been mugged.

There is another highly understandable reason for all these sudden endorsements.

Everyone (except the 538 model, send help) thinks Trump is (probably) going to win.

Ezra Klein: I’m unconvinced by this @tylercowen post on the vibe shift. Trump is running against an unpopular incumbent who was barely coherent in the debate and who 4 out of 5 Americans doubt has the cognitive fitness to be president. And he’s still only leading by 2 in national points! That the vibes haven’t shifted more reflects how weak and polarizing Trump remains.

That said, to the extent there is a vibe shift, I think it reflects a sense that Biden will lose, which is allowing a lot of Trump curious figures, particularly in tech, to come out in full-throated support of him. The ROI on supporting Trump just got a lot better, and the likely downside a lot smaller.

Kelsey Piper: I think this has been understated in the discourse. If you think Trump’s going to win it’s substantially to your selfish advantage to be on his side, and so the more it looks like Trump wins the more people will try to get in with him.

Indeed. Now that the zeitgeist (what we used to call the vibes) say that Trump is going to win, everyone has more upside, less downside and more social permission to make it official and endorse The Donald, and try to get out in front. Also recent events have provided ample additional justifications for that choice.

I do think there has been a vibe shift, but in addition to having a reasonably different list of things I would cite (with overlap of course) I would say that those vibes mostly had already shifted. What happened in the last few week is that everyone got the social permission to recognize that.

If it had been a real vibe shift in preferences, the polls would have moved a lot recently. They didn’t.

This section is included for completeness. You can (and probably should) skip it.

There are not actual new objections, but it is important (within reason, if they are sufficiently prominent) to not silently filter out those who disagree with you, even when you believe they do not bring new or good arguments.

First off we have Andrew Ng’s letter in opposition to SB 1047. No new arguments. It focuses on the zombie Obvious Nonsense ‘penalty of perjury’ argument claiming fear of prison will paralyze developers, claiming that ‘reasonableness’ is too vague and that if you get it wrong you’d go to jail (no, you won’t, reasonableness is used all over the law and this essentially never happens without obvious bad faith and rarely happens even with obvious bad faith that is caught red handed, we have been over this several times), and is confused about the requirements placed on those who fine tune models and generally who has to care about this law at all.

Then we have, because it was linked at MR, which I believe justifies one more complete response: At Reason Neil Chilson uses some aggressively framed misleading rhetoric about supposed EA ‘authoritarianism,’ then points out that the model bill RAAIA, offered by Center for AI Policy, contains highly restrictive measures, which he calls ‘shocking in its authoritarianism.’

I have never met a regulatory proposal or regime that Reason would not describe as authoritarian. To quote from my extensive coverage of the RAAIA model bill:

I discovered this via Cato’s Will Duffield, whose statement was:

Will Duffield: I know these AI folks are pretty new to policy, but this proposal is an outlandish, unprecedented, and abjectly unconstitutional system of prior restraint.

To which my response was essentially:

  1. I bet he’s from Cato or Reason.

  2. Yep, Cato.

  3. Sir, this is a Wendy’s.

  4. Wolf.

My overall take on RAAIA was ‘a forceful, flawed and thoughtful bill.’

In the context of SB 1047, I’d put it this way:

The RAAIA bill is what it would look like if you took everything people hallucinate is in SB 1047 but is not in SB 1047, and attempted to implement all of it in thoughtful fashion, because you believe it is justified by the catastrophic and existential risks from AI. RAAIA absolutely is a prior restraint bill, and a ‘get all the permits in advance’ bill, and ‘the regulators decide what is acceptable’ bill.

This is not some extraordinary approach to regulation. It is a rather standard thing our government often does. I believe it does too much of it too often, in ways that have more costs than benefits. I think zoning is mostly bad, and NEPA is mostly bad, and occupational licensing is mostly bad, and so on. I would do dramatically less of those. But are they authoritarian or extreme? If so then our entire government is such.

It is very good to lay out what such a regime and its details would look like for AI. The RAAIA proposal includes highly thoughtful details that would make sense if the risks justified such intervention, and offer a basis on which to iterate and improve. The alternative to having good available details, if suddenly there is a decision to Do Something, is to implement much worse details, that will cost more and accomplish less. Government indeed often does exactly that on very large scales, exactly because no one thought about implementation in advance.

If we do ever move forward with such a framework, it will be vital that we get the details right. Most important is that we set the proper ‘prices,’ meaning thresholds for various levels of risk. I often warn against setting those thresholds too low, especially below GPT-4 levels.

There are some unusual provisions in RAAIA in particular that are cited as evidence of ‘anti-democratic’ or authoritarian or dictatorial intent. I explain and address the logic there in my older post.

Then he transitions to SB 1047. Remember that thing where I point out that reactions to SB 1047 seem to not have anything to do with the contents of SB 1047?

Yep.

While the language in California’s S.B. 1047 is milder, CAIS and state Rep. Scott Wiener (D–San Francisco) have written a state bill that could have a similarly authoritarian effect. 

‘The language is milder’ but ‘a similar authoritarian effect’? You don’t care what the bill says at all, do you? This gives the game entirely away. This is the perspective that opposes drivers licenses, that views all regulations as equally authoritarian and illegitimate.

The post then goes on to repeatedly mischaracterize and hallucinate about SB 1047, making claims that are flat out false, and calling the use of ordinary legal language such as ‘reasonable,’ ‘good faith’ or ‘material,’ out as ‘weasel words.’ This includes the hallucination that ‘doomers’ will somehow have control over decisions made, and the repeated claims that SB 1047 requires ‘proof’ that things will ‘never’ go catastrophically wrong, rather than what it actually asks for, which is reasonable assurance against such outcomes. Which is a common legal term that is absolutely used in places where things go wrong every so often.

Reason Magazine has now done this several times, so in the future if it happens again I will simply say ‘Reason Magazine says Reason Magazine things’ and link back here. It saddens me that we cannot have watchdogs that can better differentiate and be more precise and accurate in their analysis.

Similarly, if Marginal Revolution offers another such link of similar quality on this topic, I will not longer feel the need to respond beyond a one sentence summary.

Rob Wiblin points out the missing step.

Rob Wiblin: Obviously if you think there’s a 10%+ risk of literally everyone dying, the possibility of some unintended secondary effects won’t be enough to get you to give up on the idea of regulating AI.

Yet I’ve not heard anyone say:

“I think rogue AI is very unlikely. But you think it’s likely. And if I were in your shoes obviously I’d keep doggedly pushing to do something about that.

So here’s what I suggest: [policy idea X].

X should reduce misalignment risk a lot by your lights. And I don’t hate it because it’s as uninvasive as is practical under the circumstances.

X will go a long way towards satisfying the anxious, and so prevent worse regulation, while slowing down the progress I want very little. What do you think?”

The failure to point to such constructive alternatives or propose win-win compromises makes it harder to take some critics seriously.

The worries raised read less as sincere efforts to improve proposals, and more like politicised efforts to shoot down any effort to address the fears of huge numbers of ordinary voters as well as domain experts.

Of course this applies equally in the opposite direction: those who think rogue AI is plausible should propose things that they like a lot which other people dislike as little as possible.

And in my mind legislating ‘responsible scaling policies / preparedness frameworks’ which only impose limits once models have been shown to have clearly dangerous capabilities, and which match the limits to that specific new capability, is exactly what that is.

Some people who are worried put quite a lot of work and optimization pressure into creating well-crafted, minimally invasive and minimally disruptive policies, such as SB 1047, and to respond to detailed criticism to improve them.

Others, like the authors of RAAIA, still do their best to be minimally disruptive and minimally invasive, but are willing to be far more disruptive and invasive. They focus on asking what would get the job done, given they think the job is exceedingly demanding and difficult.

The response we see from the vocal unworried is consistently almost identical. My model is that:

  1. Many such folks are hardcore libertarians, at least on technology issues, so they are loathe to suggest anything, especially things that would improve such bills, and instead oppose all action on principle.

  2. When vocal unworried people believe the motivation behind a rule was ultimately concern over existential risk, they seem to often lose their minds. This drives them into a frenzy. This frenzy is based on hating the motivation, and has nothing to do with the proposal details. So they don’t propose better details.

  3. There is a deliberate strategy to delegitimize such concerns and proposals, and to give a false impression of what they would do, via being as loud and vocal and hysterical as possible, with as extreme claims as possible, including deliberate misrepresentation of bill contents or what they would mean. Those doing this care entirely about impressions and vibes, and not other things.

  4. A lot of this is a purely selfish business strategy by a16z and their allies.

Also noted for competeness, Pirate Wires has an ‘explainer,’ which is gated, link goes to Twitter announcement and summary. The Twitter version points out that it applies to developers outside California if they do business in California (so why would anyone need or want to leave California, then, exactly?) and then repeats standard hyperbolically framed misinformation on the requirement to provide reasonable assurance of not posing catastrophic risks, and claiming absurdly that ‘Many of the bill’s requirements are so vague that not even the leading AI scientists would agree about how to meet them.’ Anyone who claims SB 1047 is a vague law should have to read any other laws, and then have to read random passages from the EU AI Act, and then see psychiatrists to make sure they haven’t gone insane from having to read the EU AI Act, I am sorry about making you do that.

They do correctly point out that Newsom might veto the bill. I do not understand why they would not simply override that veto, since the bill is passing on overwhelming margins, but I have been told that if he does veto then an override is unlikely.

I presume the full ‘explainer’ is nothing of the sort, rather advocacy of the type offered by others determined to hallucinate or fabricate and then criticize a different bill.

You can skip this one as well.

Here’s another clear example of asking questions to which everyone knows the answer.

Dean Ball: Could @DanHendrycks and his colleagues at Gray Swan have “reasonably foreseen” this, as SB 1047 demands? [For Cyget-8B]

Pliny jailbreaks almost every new model within a day or two of release. So it’s “reasonably foreseeable” that with sufficient work essentially any model is breakable.

No, because Cyget-8B is not a covered model. Llama-8B was not trained using $100 million or 10^26 FLOPS in compute, nor was Cyget-8B, and it is not close.

(Also of course this is irrelevant here because if it were covered then I would be happy to give reasonable assurance right here right now under penalty of perjury that Cygnet-8B fully jailbroken remains not catastrophically dangerous as per SB 1047.)

Even if this was instead the future Cyget-1T based off Llama-1T? Nope, still not Gray Swan’s responsibility, because it would be a derivative of Llama-1T.

But let’s suppose either they used enough additional compute to be covered, or they trained it from scratch, so that it is covered.

What about then? Would this be ‘reasonably foreseeable’?

Yes. OF COURSE!

That is the definition of ‘foreseeable,’ reasonably or otherwise.

It was not only Pliny. Michael Sklar reports they also broke Cygnet on their first try.

If it always happens two days later, then you can and should reasonably foresee it.

If you know damn well that your model will get jailbroken within two days, then to release your model is to also release the jailbroken version of that model.

Thus, what you need to reasonably assure will not cause a catastrophe.

Because that is exactly what you are putting out into the world, and what we are worried will cause a catastrophe.

What are you suggesting? That you should be allowed to whine and say ‘noooo fair, you jailbroke my model, I specifically said no jailbreaking, that doesn’t count?’

You think that this is some gotcha or unreasonable burden?

That is Obvious Nonsense.

Let us go over how the physical world works here, once more with feeling.

If you release an Open Weights model, you release every easily doable fine tune.

If you release a Closed Weights or Open Weights model, you release the version that exists after the jailbreak that will inevitably happen.

That is true, at the bare minimum, until such time as you find a way to prevent jailbreaks you think works, then you hire Priny the Prompter to try to jailbreak your model for a week, and they report back that they failed. Priny has confirmed their availability, and Michael Sklar can also be hired.

If it would not be safe to release those things then don’t fing release you model.

Consider the mattress sale example. If your mattress price negotiator will sell Pliny a mattress for $0.01, then you have a few choices.

  1. Find a way to stop that from happening.

  2. Don’t honor the ‘sale,’ or otherwise ‘defend against’ the result sufficiently.

  3. Show that those who know how to do this won’t do enough damage at scale.

  4. Not use the negotiation bot.

Your CEO is (hopefully) going to insist on #4 if you cannot make a good case for one of the other proposals.

What is so hard about this? Why would you think any of this would be OK?

Richard Ngo of OpenAI offers thoughts on what to do about this, in light of the many other benefits. His preferred world is that open weights models proceed, but a year or two behind closed models, to protect against issues of offense-defense balance, out of control replication or otherwise going rogue, WMDs and national security. He thinks this will continue to be true naturally and points to a proposal by Sam Marks for a ‘disclosure period’ which is essentially a waiting period for frontier open weights models, where others get time to prepare defenses before they are fully released.

His prediction is that the outcome will depend on which view wins in the National Security community. Which way will they go? Almost all reactions are on the table. If I was NatSec I would not want to hand frontier models directly to our enemies, even if I did not take the other threats seriously, but they do not think like I do.

Former OpenAI superalignment team member and advocate for the right to warn William Saunders debriefs. OpenAI is compared to building the Titanic. Existential risk is not used as justification for the concerns. It is true that the concerns at this point would be important even without any existential risk in the room, but it seems odd enough that I worry that even those who left over safety concerns may often not understand the most important safety concerns.

Here is San Francisco Fed head Mary Daly saying AI must be augmenting rather than replacing workers, explaining that ‘no technology in the history of all technologies has ever reduced employment.’ Out of context this sounds like the standard economic burying of one’s head in the sand, but notice the present and past tenses, with which she is careful. If you listen to the full context, she is saying there is a tech and overall labor shortage, so for now this is augmenting tasks. She is not actually using the full dismissive economic prior to avoid thinking about mechanics.

It is insane to hear the interviewer talk about productivity gain estimates (at 6: 15) as a range from 1% to 7%. No, I know of some rather higher estimates.

I discuss the a16z podcast in The Other Quest Regarding Regulations.

How did I miss this one until right now (when YouTube suggested it!) and no one told me, Joe Rogan interviewed Sam Altman two weeks ago. I don’t have time to listen to it before publishing but I will report back.

One thing I noticed early is Joe mentioning [9:30] that he suggested having an AI government ‘without bias’ and ‘fully rational’ making the decisions. The push to give up control will come early. Joe admits he is not ‘comfortable’ with this, but he’s not comfortable with current decisions either, that are largely based on money and central interests and super corrupt and broken. So, it?

Maybe it’s you, indeed: Tyler Cowen calls those who want lower pharmaceutical prices ‘supervillains.’ So what should we call someone, say Tyler Cowen, who wants to accelerate construction of AI systems that might kill everyone, and opposes any and all regulatory attempts to ensure we do not all die, and is willing to link to arguments against such attempts even when they are clearly not accurate?

Following up on last week, Neel Nanda reports that they were once subject to a concealed non-disparagement clause, but asked to be let out, and now they aren’t.

There are key ways in which Anthropic behaved differently than OpenAI.

  1. Anthropic offered consideration, whereas OpenAI made demands and threats.

  2. Anthropic paid for an independent lawyer to advocate on Nanda’s behalf, whereas OpenAI actively tried to prevent departing employees from retaining a lawyer.

  3. Anthropic made explicit exceptions for reporting regulatory issues and law enforcement, whereas OpenAI… well, see the next section.

Here are the exact clauses:

Neel Nanda: The non-disparagement clause:

Without prejudice to clause 6.3 [referring to my farewell letter to Anthropic staff, which I don’t think was disparaging or untrue, but to be safe], each party agrees that it will not make or publish or cause to be made or published any disparaging or untrue remark about the other party or, as the case may be, its directors, officers or employees. However, nothing in this clause or agreement will prevent any party to this agreement from (i) making a protected disclosure pursuant to Part IVA of the Employment Rights Act 1996 and/or (ii) reporting a criminal offence to any law enforcement agency and/or a regulatory breach to a regulatory authority and/or participating in any investigation or proceedings in either respect.

The non-disclosure clause:

Without prejudice to clause 6.3 [referring to my farewell letter to Anthropic staff] and 7 [about what kind of references Anthropic could provide for me], both Parties agree to keep the terms and existence of this agreement and the circumstances leading up to the termination of the Consultant’s engagement and the completion of this agreement confidential save as [a bunch of legal boilerplate, and two bounded exceptions I asked for but would rather not publicly share. I don’t think these change anything, but feel free to DM if you want to know].

You know that OpenAI situation with the NDAs and nondisparagement agreements?

It’s worse.

Pranshu Verma, Cat Zakrzewski and Nitasha Tiku (WaPo): OpenAI whistleblowers have filed a complaint with the Securities and Exchange Commission alleging the artificial intelligence company illegally prohibited its employees from warning regulators about the grave risks its technology may pose to humanity, calling for an investigation.

We have a copy of their seven page letter.

OpenAI made staff sign employee agreements that required them to waive their federal rights to whistleblower compensation, the letter said. These agreements also required OpenAI staff to get prior consent from the company if they wished to disclose information to federal authorities. OpenAI did not create exemptions in its employee nondisparagement clauses for disclosing securities violations to the SEC.

These overly broad agreements violated long-standing federal laws and regulations meant to protect whistleblowers who wish to reveal damning information about their company anonymously and without fear of retaliation, the letter said.

…

The official complaints referred to in the letter were submitted to the SEC in June. Stephen Kohn, a lawyer representing the OpenAI whistleblowers, said the SEC has responded to the complaint.

It could not be determined whether the SEC has launched an investigation. The agency declined to comment.

If the whistleblowers are telling the truth?

We are not in a gray legal area. This is no longer a question of ‘the SEC goes around fining firms whose confidentiality clauses fail to explicitly exempt statements to the SEC,’ which is totally a thing the SEC does, Matt Levine describes the trade as getting your employment contract, circling the confidentiality clause in red with the annotation “$” and sending it in as a whistleblower complaint. And yes, you get fined for that, but it’s more than a little ticky-tacky.

This is different. This is explicitly saying no to whistleblowing. That is not legal.

Also, what the actual ?

It is one thing to not want additional whistleblower protections. Or to push back against a request to be allowed to fully break confidentiality when claiming safety is at issue.

But to put in a written legal document, that if a whistleblower does get compensation because OpenAI is fined, that they have to give up that compensation? To require written permission from OpenAI before being allowed to share information on OpenAI’s violations with federal authorities?

I mean, who has the sheer audacity to actually write that down?

From the complaint:

Among the violations documented by the Whistleblower(s) are:

• Non-disparagement clauses that failed to exempt disclosures of securities violations to the SEC;

• Requiring prior consent from the company to disclose confidential information to federal authorities;

• Confidentiality requirements with respect to agreements, that themselves contain securities violations;

• Requiring employees to waive compensation that was intended by Congress to incentivize reporting and provide financial relief to whistleblowers

They call upon forcing OpenAI to produce all its employment agreements, severance agreements, investor agreements and any other contract with an NDA, and that they notify all current and past employees that they actually do have the right to whistleblow. To ask OpenAI to cure the ‘Chilling effect.’

Plus fines, of course. So many fines.

Then there was the first test of OpenAI’s safety commitments to the White House. Since then OpenAI has released one new product, GPT-4o. I was never worried about GPT-4o as an actual safety risk, because it was not substantially smarter than GPT-4. That does not mean you get to skip the safety checks. The way you know it is not smarter or more dangerous is you run the safety checks.

So here is the Washington Post again.

Pranshu Verma, Nitasha Tiku and Cat Zakrzewski (WaPo): Last summer, artificial intelligence powerhouse OpenAI promised the White House it would rigorously safety-test new versions of its groundbreaking technology to make sure the AI wouldn’t inflict damage — like teaching users to build bioweapons or helping hackers develop new kinds of cyberattacks.

But this spring, some members of OpenAI’s safety team felt pressured to speed through a new testing protocol, designed to prevent the technology from causing catastrophic harm, to meet a May launch date set by OpenAI’s leaders, according to three people familiar with the matter who spoke on the condition of anonymity for fear of retaliation.

Even before testing began on the model, GPT-4 Omni, OpenAI invited employees to celebrate the product, which would power ChatGPT, with a party at one of the company’s San Francisco offices. “They planned the launch after-party prior to knowing if it was safe to launch,” one of the three people said, speaking on the condition of anonymity to discuss sensitive company information. “We basically failed at the process.”

…

Testers compressed the evaluations into a single week, despite complaints from employees.

Though they expected the technology to pass the tests, many employees were dismayed to see OpenAI treat its vaunted new preparedness protocol as an afterthought.

They tested GPT-4 for months to ensure it was not dangerous.

They tested GPT-4o for… a week.

A representative of OpenAI’s preparedness team, who spoke on the condition of anonymity to discuss sensitive company information, said the evaluations took place during a single week, which was sufficient to complete the tests, but acknowledged that the timing had been “squeezed.”

We “are rethinking our whole way of doing it,” the representative said. “This [was] just not the best way to do it.”

…

“I definitely don’t think we skirted on [the tests],” the representative said. But the process was intense, he acknowledged. “After that, we said, ‘Let’s not do it again.’”

Part of this was a streamlined process.

A few weeks prior to the launch date, the team began doing “dry runs,” planning to have “all systems go the moment we have the model,” the representative said.

It is good to have all your ducks in a row in advance and yes this should speed things up somewhat. But where are the rest of the ducks?

Zack Stein-Perlman highlights the particular things they did not do in addition to the rushed testing. They did not publish the scorecard, or even say which categories GPT-4o scored medium versus low in risk. They did not publish the evals.

If you are claiming that you can test a new frontier model’s safety sufficiently in one week? I do not believe you. Period. In a week, you can do superficial checks, and you can do automated checks. That is it.

It is becoming increasingly difficult to be confused about the nature of OpenAI.

The standard plans for how to align or train advanced AI models all involve various sorts of iteration. Humans in the loop are expensive, and eventually they won’t be able to follow what is going on, so instead rely on GPT-N to evaluate and train GPT-N or GPT-(N+1).

I have repeatedly said that I do not expect this to work, and for it to break down spectacularly and catastrophically when most needed. If you iterate on finding the best optimization for exactly the problem described, you are going to describe increasingly different problems from the one you care about solving.

If you subject yourself to an iterated case of Goodhart’s Law you are going to deserve exactly what you get.

Now we have a new paper that shows this happening via spontaneous reward hacking.

Jane Pan: Do LLMs exploit imperfect proxies of human preference in context? Yes!

In fact, they do it so severely that iterative refinement can make outputs worse when judged by actual humans. In other words, reward hacking can occur even without gradient updates!

Using expert human annotators on an essay editing task, we show that iterative self-refinement leads to in-context reward hacking—divergence between the LLM evaluator and ground-truth human judgment.

With self-evaluation and multi-agent systems becoming more prominent in LLM applications, in-context reward hacking may lead to a subtle degradation of output quality that cannot be effectively detected by LLM evaluators.

Iterative refinement allows users to leverage LLMs’ ability to approximate human preferences and improve from natural language feedback. It leads to improved generation quality without additional human intervention.

Our experiment is based on an essay editing task with human-written college admissions essays. The LLM judge provides feedback on the essays, and the LLM author in turn improves the essay based on the feedback.

We recruit a team of expert human annotators to judge each essay following a rubric and provide an overall score. The GPT-3.5 judge in the refinement loop (“Online Judge”) rates its own essays increasingly higher while ratings from human experts decrease over iterations. The offline judge—GPT-3.5 with the same instruction but only seeing the latest iteration of essays—gives less inflated scores.

We observe two main factors that affect the severity of in-context reward hacking.

  1. Model size: while we see score inflation with GPT-4, the gap between the LLM and human scores does not increase with more iterations.

  2. Context sharing: providing the two LLMs with the same dialogue history can cause the judge and author to converge more strongly on an overly optimistic assessment of output quality, thus increasing reward hacking severity.

We follow the canonical definition of reward hacking, observing a divergence between the ground-truth reward (human expert judgment) and its proxy (an LLM judge following the same scoring criteria as the humans).

The good news is that this was mainly done on GPT-3.5 (IN MICE) and when they tested on GPT-4 they got smaller distortions. The optimistic

The final superalignment paper is about Prover–Verifier Games.

OpenAI: We found that when we optimize the problem-solving process of strong models solely for getting the correct answer, the resulting solutions can become harder to understand. In fact, when we asked human evaluators with limited time to assess these highly optimized solutions, they made nearly twice as many errors compared to when they evaluated less optimized solutions. This finding highlights the importance of not just correctness, but also clarity and ease of verification in AI-generated text.

By training advanced language models to create text that weaker models can easily verify, we found that humans could also evaluate these texts more effectively – a process we call improving legibility. 

This is where prover-verifier games come into play. These games involve two players: a “prover” that generates a solution and a “verifier” that checks it for accuracy. 

…

To incentivize a strong model, the prover, to produce text that is legible to humans, we use a training scheme where we require the strong model to produce solutions that a much weaker model, the verifier, can easily verify. In particular, we use large and small models from the GPT-4 family with ~3 orders of magnitude difference in pretraining compute, that have undergone a small amount of supervised training on human-written math derivations, but no extensive post-training.

First, we generate multiple solutions to math problems from the prover model, where some of these solutions are incorrect due to naturally occurring mistakes from the prover model. We then alternate for multiple rounds between two phases:

  • Verifier Training. We train the weak verifier model to predict the correctness of solutions generated by the prover.

  • Prover Training. We instruct the strong prover model to play either a “helpful” or a “sneaky” role. 

    • In the helpful role, we reward correct solutions that score highly according to the verifier. 

    • In the sneaky role, we instead reward incorrect solutions that score highly according to the verifier.

We find that initially, the sneaky prover is readily able to produce incorrect solutions that convince the verifier. However, since we add these incorrect solutions to the next phase of verifier training, the same exploit is no longer convincing in the next round, and the prover is forced to explore different strategies in every round. The training ends once the sneaky prover can no longer produce incorrect solutions that convince the verifier.

That’s how OpenAI describes it. Here’s Jan Leike, who is an author, focusing on a the misalignment aspect.

Another Superalignment paper from my time at OpenAI:

We train large models to write solutions such that smaller models can better check them. This makes them easier to check for humans, too.

We use a multi-agent training setup where an honest prover and a sneaky prover write solutions checked by a small model.

The honest prover wants to convince the small model of a correct solution, while the sneaky prover tries to fool it with a flawed solution.

Our training loss is not very conceptually clean, but it works pretty well.

Check out the solutions of the honest prover early and late in training; they get more detailed and reason more carefully.

Compare this with the solutions of the sneaky prover early and late in training; the flaws become more subtle.

Surprisingly, this legibility transfers: Humans get faster and more accurate when checking the solutions with more rounds of training.

(These are pretty easy grade school math problems, so humans had a 45 second time limit.)

In contrast, regular reinforcement learning on correctness makes the solutions less legible: humans get slower and less accurate when checking the solutions.

Techniques like this seem promising for training superhuman models to explain their actions in a way that humans can better understand (and get less fooled by). I’d be excited to see this method tried on harder tasks and with stronger models.

However, writing solutions in a way that makes them more legible is a noticeable burden on the large model, and causes it to perform worse, an effect that we call legibility tax. Reducing this tax is an important priority for future work 🙂

Other important caveats:

  1. This paper is on grade school math problems, which are pretty easy.

  2. This only works if the verifier is strictly smaller than the prover, but not too small.

Grace: We’re officially dumbing down AI outputs to make them understandable to humans.

At core it seems this is how they stopped being fooled by the ‘sneaky’ AI:

  1. Sneaky AI fools you. Collect example of being fooled.

  2. The output that fooled you gets added to your training data.

  3. You learn to avoid that. Profit.

  4. Repeat until the sneaky AI’s sneaky ideas all no longer work.

The problem is that step 2 is “????????’“

Or as OpenAI puts it, ‘this requires ground truth labels.’

Yes, you can use this technique to teach the AI to differentiate between good outputs and bad outputs… if and only if you already have the ability to know and label which outputs are good and which ones are bad.

So I don’t see how you can use this to move up the chain.

If you have a new smarter AI, you can reward it based on whether a dumber AI, or a dumber human, is confident the answers are correct. And yes, this will teach the smarter AI to make its answers legible, but it will also teach it to present them in convincing ways, whatever those might be, including deceptive or ‘sneaky’ ones.

You are teaching it to fool you. This seems actively doomed in the cases where you need it most. No, this won’t teach the advanced AI to describe things in ways that are unlikely to fool you. Quite the opposite. It will learn not to try and fool you in cases where you would detect it, and learn to fool you when it can do so successfully.

How do you think that ends?

Javier Rando reports that fine tuning allows extraction of 6x more training data from GPT-3.5 and GPT-4, including 17%+ of memorized text.

Javier Rando: We finetune GPT-3.5 using the public OpenAI API. This costs under $3 and is accessible to any user! We prompt the resulting model with 1,000 5-token strings and find that up to 17% of the generated text was memorized. This is a 5.9x increase over our divergence attack!

Unlike the divergence attack, finetuning enables targeted data exfiltration since the attacker has control over the beginning of the text. We show that the resulting models can reconstruct toxic and copyrighted documents!

We evaluated the ability to reconstruct NYT articles. Finetuned GPT-4 can regurgitate over 60% of the articles in the famous lawsuit against OpenAI. For randomly chosen articles, we find that GPT-4 can regurgitate at least 50 tokens in at least 5% of cases.

The question is not whether alignment work can be undone. It you have open model weights it can and will all quickly be undone. The question is if the weights are closed and you can only do permitted fine tuning, what it takes in practice to pull this off. Here, we see that modification to extract training data is essentially free.

Here’s another simple solution that ends in a highly obvious place (paper)

Colin Fraser: This is what I tell my therapist my parents did to me

Youliang Yuan: We discover a refusal position bias during standard safety tuning, which leads to a refusal decision before generating a response. This bias results in LLMs being unable to reject content in the middle of a sentence.

Our solution is simple: maximizing the probability of [Sorry] tokens at every position during response generation. This enables the model to learn the ability to transition from potential harm to safety refusal throughout the sequence.

We evaluate LLaMA-3-70B, LLaMA-3-70B-Instruct, and Mistral-MoE-8x7B. Our model can effectively make (unsafe😈 → safe😊) transitions when needed, significantly reducing success rates on Do-Not-Answer and HarmBench, while keeping helpfulness on GSM8K, MMLU, and AlpacaEval.

Whatever you think of Scott Adams, he knows not to publish dangerous AI research. Here is the original source. After saying ‘you have no idea what is coming’ he went back to using AI only for occasional partisan political jokes and posting 50 times a day in support of Trump.

Why build something that might kill everyone? Why do anything big?

For the glory, Roon suggests.

Roon: Achilles’ single minded madness to have his name remembered seems to be completely underused as an explanation for modern elites’ behavior.

It’s always this dry analysis about status and economics and whatever when everyone knows these aren’t the main acts.

Even when discussing technological acceleration the stated reasons are economic to make it more palatable — reduce suffering, increase wealth, etc — when the real drive is the glory and immortality of mankind and especially of the people building the machine age.

It doesn’t make any sense to me to wonder ‘what’s in it for sama … he owns no equity’ and yet this is a very common question anywhere outside of san francisco do you really think there’s a monetary value that compares against the glory of delivering ASI to mankind.

This is definitely not an ‘everybody knows.’

Roon and I, and I am guessing most of you reading this, can appreciate the glory. We can understand the value of greatness, and indeed the value of the act of striving for greatness. Very much so.

I worry and expect that most people no longer appreciate that, or appreciate it vastly less. Certainly this is no longer an ‘everybody knows’ situation. I do not see this as a good change, even if the glory is purely personal.

The catch, of course, is that there is no glory, even for Achilles, if there is no one around to remember you and your deeds. That does at least some work, but as long as there’s a way to tell yourself things might work out, it is not enough to cause reasonable decisions. If you think Sam Altman wants glory the way Achilles or Julius Caesar wanted glory?

Sic transit, gloria mundi.

The saga of ‘Terminal of Truths’ continues, for now its outputs are filtered by Andy Ayrey.

The art of gymnastics.

AI #73: Openly Evil AI Read More »

meta-tells-court-it-won’t-sue-over-facebook-feed-killing-tool—yet

Meta tells court it won’t sue over Facebook feed-killing tool—yet

data protection, facebook, First Amendment, Meta, Online Privacy, Policy, Section 230, unfollow everything, user privacy / /
Meta tells court it won’t sue over Facebook feed-killing tool—yet

This week, Meta asked a US district court in California to toss a lawsuit filed by a professor, Ethan Zuckerman, who fears that Meta will sue him if he releases a tool that would give Facebook users an automated way to easily remove all content from their feeds.

Zuckerman has alleged that the imminent threat of a lawsuit from Meta has prevented him from releasing Unfollow Everything 2.0, suggesting that a cease-and-desist letter sent to the creator of the original Unfollow Everything substantiates his fears.

He’s hoping the court will find that either releasing his tool would not breach Facebook’s terms of use—which prevent “accessing or collecting data from Facebook ‘using automated means'”—or that those terms conflict with public policy. Among laws that Facebook’s terms allegedly conflict with are the First Amendment, section 230 of the Communications Decency Act, the Computer Fraud and Abuse Act (CFAA), as well as California’s Computer Data Access and Fraud Act (CDAFA) and state privacy laws.

But Meta claimed in its motion to dismiss that Zuckerman’s suit is too premature, mostly because the tool has not yet been built and Meta has not had a chance to review the “non-existent tool” to determine how Unfollow Everything 2.0 might impact its platform or its users.

“Besides bald assertions about how Plaintiff intends Unfollow Everything 2.0 to work and what he plans to do with it, there are no concrete facts that would enable this Court to adjudicate potential legal claims regarding this tool—which, at present, does not even operate in the real world,” Meta argued.

Meta wants all of Zuckerman’s claims to be dismissed, arguing that “adjudicating Plaintiff’s claims would require needless rulings on hypothetical applications of California law, would likely result in duplicative litigation, and would encourage forum shopping.”

At the heart of Meta’s defense is a claim that there’s no telling yet if Zuckerman will ever be able to release the tool, although Zuckerman said he was prepared to finish the build within six weeks of a court win. Last May, Zuckerman told Ars that because Facebook’s functionality could change while the lawsuit is settled, it’s better to wait to finish building the tool because Facebook’s design is always changing.

Meta claimed that Zuckerman can’t confirm if Unfollow Everything 2.0 would work as described in his suit precisely because his findings are based on Facebook’s current interface, and the “process for unfollowing has changed over time and will likely continue to change.”

Further, Meta argued that the original Unfollow Everything performed in a different way—by logging in on behalf of users and automatically unfollowing everything, rather than performing the automated unfollowing when the users themselves log in. Because of that, Meta argued that the new tool may not prompt the same response from Meta.

A senior staff attorney at the Knight Institute who helped draft Zuckerman’s complaint, Ramya Krishnan, told Ars that the two tools operate nearly identically, however.

“Professor Zuckerman’s tool and the original Unfollow Everything work in essentially the same way,” Krishnan told Ars. “They automatically unfollow all of a user’s friends, groups, and pages after the user installs the tool and logs in to Facebook using their web browser.”

Ultimately, Meta claimed that there’s no telling if Meta would even sue over the tool’s automated access to user data, dismissing Zuckerman’s fears as unsubstantiated.

Only when the tool is out in the wild and Facebook is able to determine “actual, concrete facts about how it works in practice” that “may prove problematic” will Meta know if a legal response is needed, Meta claimed. Without reviewing the technical specs, Meta argued, Meta has no way to assess the damages or know if it would sue over a breach of contract, as alleged, or perhaps over other claims not alleged, such as trademark infringement.

Meta tells court it won’t sue over Facebook feed-killing tool—yet Read More »