Altman’s

on-sam-altman’s-second-conversation-with-tyler-cowen

On Sam Altman’s Second Conversation with Tyler Cowen

Some podcasts are self-recommending on the ‘yep, I’m going to be breaking this one down’ level. This was very clearly one of those. So here we go.

As usual for podcast posts, the baseline bullet points describe key points made, and then the nested statements are my commentary.

If I am quoting directly I use quote marks, otherwise assume paraphrases.

The entire conversation takes place with an understanding that no one is to mention existential risk or the fact that the world will likely transform, without stating this explicitly. Both participants are happy to operate that way. I’m happy to engage in that conversation (while pointing out its absurdity in some places), but assume that every comment I make has an implicit ‘assuming normality’ qualification on it, even when I don’t say so explicitly.

  1. Cowen asks how Altman got so productive, able to make so many deals and ship so many products. Altman says people almost never allocate their time efficiently, and that when you have more demands on your time you figure out how to improve. Centrally he figures out what the core things to do are and delegates. He says deals are quicker now because everyone wants to work with OpenAI.

    1. Altman’s definitely right that most people are inefficient with their time.

    2. Inefficiency is relative. As in, I think of myself as inefficient with my time, and think of the ways I could be a lot more efficient.

    3. Not everyone responds to pressure by improving efficiency, far from it.

    4. Altman is good here to focus on delegation.

    5. It is indeed still remarkable how many things OpenAI is doing at once, with the associated worries about it potentially being too many things, and not taking the time to do them responsibly.

  1. What makes hiring in hardware different from in AI? Cycles are longer. Capital is more intense. So more time invested up front to pick wisely. Still want good, effective, fast-moving people and clear goals.

    1. AI seems to be getting pretty capital intensive?

  2. Nvidia’s people ‘are less weird’ and don’t read Twitter. OpenAI’s hardware people feel more like their software people than they feel like Nvidia’s people.

    1. My guess is there isn’t a right answer but you need to pick a lane.

  3. What makes Roon special? Lateral thinker, great at phrasing observations, lots of disparate skills in one place.

    1. I would add some more ingredients. There’s a sense of giving zero fucks, of having no filter, and having no agenda. Say things and let the chips fall.

    2. A lot of the disparate skills are disparate aesthetics, including many that are rare in AI, and taking all of them seriously at once.

  4. Altman doesn’t tell researchers what to work on. Researchers choose, that’s it.

  5. Email is very bad. Slack might not be good, it creates explosions of work including fake work to deal with, especially the first and last hours, but it is better than email. Altman suspects it’s time for a new AI-driven thing but doesn’t have it yet, probably due to lack of trying and unwillingness to pay focus and activation energy given everything else going on.

    1. I think email is good actually, and that Slack is quite bad.

    2. Email isn’t perfect but I like that you decide what you have ownership of, how you organize it, how you keep it, when you check it, and generally have control over the experience, and that you can choose how often you check it and aren’t being constantly pinged or expected to get into chat exchanges.

    3. Slack is an interruption engine without good information organization and I hate it so much, as in ‘it’s great I don’t have a job where I need slack.’

    4. There’s definitely room to build New Thing that integrates AI into some mix of information storage and retrieval, email slow communication, direct messaging and group chats, and which allows you to prioritize and get the right levels of interruption at the right times, and so on.

    5. However this will be tricky, you need to be ten times better and you can’t break the reliances people have. False negatives, where things get silently buried, can be quite bad.

  1. What will make GPT-6 special? Altman suggests it might be able to ‘really do’ science. He doesn’t have much practical advice on what to do with that.

    1. This seems like we hit the wall of ‘…and nothing will change much’ forcing Altman to go into contortions.

    2. One thing we learned from GPT-5 is that the version numbers don’t have to line up with big capabilities leaps. The numbers are mostly arbitrary.

Tyler isn’t going to let him off that easy. At this point, I don’t normally do this, but exact words seem important so I’m going to quite the transcript.

COWEN: If I’m thinking about restructuring an entire organization to have GPT-6 or 7 or whatever at the center of it, what is it I should be doing organizationally, rather than just having all my top people use it as add-ons to their current stock of knowledge?

ALTMAN: I’ve thought about this more for the context of companies than scientists, just because I understand that better. I think it’s a very important question. Right now, I have met some orgs that are really saying, “Okay, we’re going to adopt AI and let AI do this.” I’m very interested in this, because shame on me if OpenAI is not the first big company run by an AI CEO, right?

COWEN: Just parts of it. Not the whole thing.

ALTMAN: No, the whole thing.

COWEN: That’s very ambitious. Just the finance department, whatever.

ALTMAN: Well, but eventually it should get to the whole thing, right? So we can use this and then try to work backwards from that. I find this a very interesting thought experiment of what would have to happen for an AI CEO to be able to do a much better job of running OpenAI than me, which clearly will happen someday. How can we accelerate that? What’s in the way of that? I have found that to be a super useful thought experiment for how we design our org over time and what the other pieces and roadblocks will be. I assume someone running a science lab should try to think the same way, and they’ll come to different conclusions.

COWEN: How far off do you think it is that just, say, one division of OpenAI is 85 percent run by AIs?

ALTMAN: Any single division?

COWEN: Not a tiny, insignificant division, mostly run by the AIs.

ALTMAN: Some small single-digit number of years, not very far. When do you think I can be like, “Okay, Mr. AI CEO, you take over”?

COWEN: CEO is tricky because the public role of a CEO, as you know, becomes more and more important.

  1. On the above in terms of ‘oh no’:

    1. Oh no. Exactly the opposite. Shame on him if OpenAI goes first.

    2. OpenAI is the company, in this scenario, out of all the companies, we should be most worried about handing over to an AI CEO, for obvious reasons.

    3. If you’re wondering how the AIs could take over? You can stop wondering. They will take over because we will ask them to.

    4. CEO is an adversarial and anti-inductive position, where any weakness will be systematically exploited, and big mistakes can entirely sink you, and the way that you direct and set up the ‘AI CEO’ matters quite a lot in all this. The bar to a net positive AI CEO is much higher than the AI making on average better decisions, or having on average better features, and the actual bar is higher. Altman says ‘on the actual decision making maybe the AI is pretty good soon’ but this is a place where I’m going to be the Bottleneck Guy.

    5. CEO is also a position where, very obviously, misaligned means your company can be extremely cooked, and basically everything in it subverted, even if that CEO is a single human. Most of the ways in which this is limited are because the CEO can only be in one place at a time and do one thing at a time, couldn’t keep an eye on most things let alone micromanage them, and would require conspirators. A hostile AI CEO is death or subversion of the company.

    6. The ‘public role’ of the CEO being the bottleneck does not bring comfort here. If Altman (as he suggests) is public face and the AI ‘figures out what to do’ and Altman doesn’t actually get to overrule the AI (or is simply convinced not to) then the problem remains.

  2. On the above in terms of ‘oh yeah’:

    1. There is the clear expectation from both of them that AI will rise, reasonably soon, to the level of at least ‘run the finance department of a trillion dollar corporation.’ This doesn’t have to be AGI but it probably will be, no?

    2. It’s hard for me to square ‘AIs are running the actual decision making at top corporations’ with predictions for only modest GDP growth. As Altman notes, the AI CEO needs to be a lot better than the human CEO in order to get the job.

    3. They are predicting billion-dollar 2-3 person companies, with AIs, within three years.

  3. Altman asks potential hires about their use of AI now to predict their level of AI adoption in the future, which seems smart. Using it as ‘better Google’ is a yellow flag, thinking about day-to-day in three years is a green flag.

  4. In three years Altman is aiming to have a ‘fully automated AI researcher.’ So it’s pretty hard to predict day-to-day use in three years.

A timely section title.

  1. Cowen and Altman are big fans of nuclear power (as am I), but people worry about them. Cowen asks, do you worry similarly about AI and the similar Nervous Nellies, even if ‘AI is pretty safe’? Are the Feds your insurer? How will you insure everything?

    1. Before we get to Altman’s answer can we stop to think about how absolutely insane this question is as presented?

    2. Cowen is outright equating worries about AI to worries about nuclear power, calling both Nervous Nellies. My lord.

    3. The worry about AI risks is that the AI companies might be held too accountable? Might be asked to somehow provide too much insurance, when there is clearly no sign of any such requirement for the most important risks? They are building machines that will create substantial catastrophic and even existential risks, massive potential externalities.

    4. And you want the Federal Government to actively insure against AI catastrophic risks? To say that it’s okay, we’ve got you covered? This does not, in any way, actually reduce the public’s or world’s exposure to anything, and it further warps company incentives. It’s nuts.

    5. Not that even the Federal Government can actually ensure us here even at our own expense, since existential risk or sufficiently large catastrophic or systemic risk also wipes out the Federal Government. That’s kind of the point.

    6. The idea that the people are the Nervous Nellies around nuclear, which has majority public support, while Federal Government is the one calming them down and ensuring things can work is rather rich.

    7. Nuclear power regulations are insanely restrictive and prohibitive, and the insurance the government writes does not substantially make up for this, nor is it that expensive or risky. The NRC and other regulations are the reason we can’t have this nice thing, in ways that don’t relate much if at all to the continued existence of these Nervous Nellies. Providing safe harbor in exchange of that really is the actual least you can do.

    8. AI regulations impose very few rules and especially very few safety rules.

    9. Yes, there is the counterpoint that AI has to follow existing rules and thus is effectively rather regulated, but I find this rather silly as an argument, and no I don’t think the new laws around AI in particular move that needle much.

  2. Altman points out the Federal Government is the insurer of last resort for anything sufficiently large, whether you want it to be or not, but no not in the way of explicitly writing insurance policies.

    1. I mean yes if AI crashes the economy or does trillions in damages or what not, then the Federal Government will have to try and step in. This is a huge actual subsidy to the AI companies and they should (in theory anyway) be pay for it.

    2. A bailout for the actual AI companies if they are simply going bankrupt? David Sacks has made it clear our answer is no thank you, and rightfully so. Obviously, at some point the Fed Put or Trump Put comes into play in the stock market, that ship has sailed, but no we will not save your loans.

    3. And yeah, my lord, the idea that the Feds would write an insurance policy.

  3. Cowen then says he is worried about the Feds being the insurer of first resort and he doesn’t want that, Altman confirms he doesn’t either and doesn’t expect it.

    1. It’s good that they don’t want this to happen but this only slightly mitigates my outrage at the first question and the way it was presented.

  4. Cowen points out Trump is taking equity in Intel, lithium and rare earths, and asks how this applies to OpenAI. Altman mostly dodges, pivots to potential loss of meaning in the world, and points out the government might have strong opinions about AI company actions.

    1. Cowen doesn’t say it here but to his credit is on record correctly opposing this taking of equity in companies correctly identifying it as ‘seizing the means of production’ and pointing out it is the wrong tool for the job.

    2. This really was fully a non-answer. I see why that might be wise.

    3. Could OpenAI be coerced into giving up equity, or choose to do so as part of a regulatory capture play? Yeah. It would be a no-good, very bad thing.

    4. The government absolutely will and needs to have strong opinions about AI company actions and set the regulations and rules in place and otherwise play the role of being the actual government.

    5. If the government does not govern the AI companies, then the government will wake up one day to find the AI companies have become the government.

  1. Tyler Cowen did a trip through France and Spain and booked all but one hotel with GPT-5 (not directly in the app), and almost every meal they ate, and Altman didn’t get paid for that. Shouldn’t he get paid?

    1. Before I get to Altman’s answer, I will say that for specifically Tyler this seems very strange to me, unless he’s running an experiment as research.

    2. As in, Tyler has very particular preferences and a lot of comparative advantage in choosing hotels and especially restaurants, especially for himself. It seems unlikely that he can’t do better than ChatGPT?

    3. I expect to be able to do far better than ChatGPT on finding restaurants, although with a long and highly customized prompt, maybe? But it would require quite a lot of work.

    4. For hotels, yeah, I think it’s reasonably formulaic and AI can do fine.

  2. Altman responds that often ChatGPT is cited as the most trusted tech product from a big tech company. He notes that this is weird given the hallucinations. But it makes sense in that it doesn’t have ads and is in many visible ways more fully aligned with user preferences than other big tech products that involve financial incentives. He notes that a transaction fee probably is fine but any kind of payment for placement would endanger this.

    1. ChatGPT being most trusted is definitely weird given it is not very reliable.

    2. It being most trusted is an important clue to how people will deal with AI systems going forward, and it should worry you in important ways.

    3. In particular, trust for many people is about ‘are they Out To Get You?’ rather than reliability or overall quality, or are expectations set fairly. Compare to the many people who otherwise trust a Well Known Liar.

    4. I strongly agree with Altman about the payola worry, as Cowen calls it. Cowen says he’s not worried about it, but doesn’t explain why not.

    5. OpenAI’s instant checkout offerings and policies are right on the edge on this. I think in their present form they will be fine but they’re on thin ice.

  3. Cowen’s worry is that OpenAI will have a cap on how much commission they can charge, because stupider services will then book cheaply if you charge too much. Altman says he expects much lower margins.

    1. AI will as Altman notes make many markets much more efficient by vastly lowering search costs and transaction costs, which will lower margins, and this should include commissions.

    2. I still think OpenAI will be able to charge substantial commissions if it retains its central AI position with consumers, for the same reason that other marketplaces have not lost their ability to extract commissions, including some very large ones. Every additional hoop you ask a customer to go through loses a substantial portion of sales. OpenAI can pull the same tricks as Steam and Amazon and Apple including on price parity, and many will pay.

    3. This is true even if there are stupider services that can do the booking and are generally 90% as good, so long as OpenAI is the consumer default.

  4. Cowen doubles down on this worry about cheap competing agents, Altman notes that hotel booking is not the way to monetize, Cowen says but of course you do want to do that, Altman says no he wants to do new science, but ChatGPT and hotel booking is good for the world.

    1. This feels like a mix of a true statement and a dishonest dodge.

    2. As in, of course he wants to do hotel booking and make money off it, it’s silly to pretend that you don’t and there’s nothing wrong with that. It’s not the main goal, but it drives growth and valuation and revenue all of which is vital to the AGI or science mission (whether you agree with that mission or not).

  5. Cowen asks, you have a deal coming with Walmart, if you were Amazon would you make a deal with OpenAI or fight back? Altman says he doesn’t know, but that if he was Amazon he would fight back.

    1. Great answer from Altman.

    2. One thing Altman does well is being candid in places you would not expect, where it is locally superficially against his interests, but where it doesn’t actually cost him much. This is one of those places.

    3. Amazon absolutely cannot fold here because it loses too much control over the customer and customer flow. They must fight back. Presumably they should fight back together with their friends at Anthropic?

  6. Cowen asks about ads. Altman says some ads would be bad as per earlier, but other kinds of ads would be good although he doesn’t know what the UI is.

    1. Careful, Icarus.

    2. There definitely are ‘good’ ways to do ads if you keep them entirely distinct from the product, but the temptations and incentives here are terrible.

  1. What should OpenAI management know about KSA and UAE? Altman says it’s mainly knowing who will run the data centers and what security guarantees they will have, with data centers being built akin to US embassies or military bases. They bring in experts and as needed will bring in more.

    1. I read this as a combination of outsourcing the worries and not worrying.

    2. I would be more worried.

  2. Cowen asks, how good will GPT-6 be at teaching these kinds of national distinctions, or do you still need human experts? Altman expects to still need the experts, confirms they have an internal eval for that sort of thing but doesn’t want to pre-announce.

    1. My anticipation is that GPT-6 and its counterparts will actually be excellent at understanding these country distinctions in general, when it wants to be.

    2. My anticipation is also that GPT-6 will be excellent at explaining things it knows to humans and helping those humans learn, when it wants to, and this is already sufficiently true for current systems.

    3. The question is, will you be able to translate that into learning and understanding such issues?

    4. Why is this uncertain? Two concerns.

    5. The first concern is that understanding may depend on analysis of particular key people and relationships, in ways that are unavailable to AI, the same way you can’t get them out of reading books.

    6. The second concern is that to actually understand KSA and UAE, or any country or culture in general, requires communicating things that it would be impolitic to say out loud, or for an AI to typically output. How do you pass on that information in this context? It’s a problem.

  3. Cowen asks about poetry, predicts you’ll be able to get the median Pablo Neruda poem but not the best, maybe you’ll get to 8.8/10 in a few years. Altman says they’ll reach 10/10 and Cowen won’t care, Cowen promises he’ll care but Altman equates it to AI chess players. Cowen responds there’s something about a great poem ‘outside the rubric’ and he worries humans that can’t produce 10s can’t identify 10s? Or that only humanity collectively and historically can decide what is a 10?

    1. This is one of those ‘AI will never be able to [X] at level [Y]’ claims so I’m on Altman’s side here, a sufficiently capable AI can do 10/10 on poems, heck it can do 11/10 on poems. But yeah, I don’t think you or I will care other than as a technical achievement.

    2. If an AI cannot produce sufficiently advanced poetry, that means that the AI is insufficiently advanced. Also we should not assume that future AIs or LLMs will share current techniques or restrictions. I expect innovation with respect to poetry creation.

    3. The thing being outside the rubric is a statement primarily about the rubric.

    4. If only people writing 10s can identify 10s then for almost all practical purposes there’s no difference between a 9 and a 10. Why do we care, if we literally can’t tell the difference? Whereas if we can tell the difference, if verification is easier than generation as it seems like it should be here, then we can teach the AI how to tell the difference.

    5. I think Cowen is saying that a 10-poem is a 9-poem that came along at the right time and got the right cultural resonance, in which case sure, you cannot reliably produce 10s, but that’s because it’s theoretically impossible to do that, and no human could do that either. Pablo Neruda couldn’t do it.

    6. As someone who has never read a poem by Pablo Neruda, I wanted to see what this 10.0 business was all about, so by Claude’s recommendation of ‘widely considered best Neruda poem’ without any other context, I selected Tonight I Can Write (The Saddest Lines). And not only did it not work on me, it seemed like something an AI totally could write today, on the level of ‘if you claimed to have written this in 2025 I’d have suspected an AI did write it.’

    7. With that in mind, I gave Claude context and it selected Ode to the Onion. Which also didn’t do anything for me, and didn’t seem like anything that would be hard for an AI to write. Claude suggests it’s largely about context, that this style was new at the time, and I was reading translations into English and I’m no poetry guy, and agrees that in 2025 yes an AI could produce a similar poem, it just wouldn’t land because it’s no longer original.

    8. I’m willing to say that whatever it is Tyler thinks AI can’t do, also is something I don’t have the ability to notice. And which doesn’t especially motivate me to care? Or maybe is what Tyler actually wants something like ‘invent new genre of poetry’?

    9. We’re not actually trying to get AIs to invent new genres of poetry, we’re not trying to generate the things that drive that sort of thing, so who is to say if we could do it. I bet we could actually. I bet somewhere in the backrooms is a 10/10 Claude poem, if you have eyes to see.

  1. It’s hard. Might get easier with time, chips designing chips.

  2. Why not make more GPUs? Altman says, because we need more electrons. What he needs most are electrons. We’re working hard on that. For now, natural gas, later fusion and solar. He’s still bullish on fusion.

    1. This ‘electrons’ thing is going to drive me nuts on a technical level. No.

    2. This seems simply wrong? We don’t build more GPUs because TSMC and other bottlenecks mean we can’t produce more GPUs.

    3. That’s not to say energy isn’t an issue but the GPUs sell out.

    4. Certainly plenty of places have energy but no GPUs to run with them.

  3. Cowen worries that fusion uses the word ‘nuclear.’

    1. I don’t. I think that this is rather silly.

    2. The problem with fusion is purely that it doesn’t work. Not yet, anyway.

    3. Again, the people are pro-nuclear power. Yay the people.

  4. Cowen asks do you worry about a scenario where superintelligence does not need much compute, so you’re betting against progress over a 30-year time horizon?

    1. Always pause when you hear such questions to consider that perhaps under such a scenario this is not the correct thing to worry about?

    2. As in, if we not only have superintelligence it also does not need so much compute, the last thing I am going to ponder next is the return on particular investments of OpenAI, even if I am the CEO of OpenAI.

    3. If we have sufficiently cheap superintelligence that we have both superintelligence and an abundance of compute, ask not how the stock does, ask questions like how the humans survive or stay in control at all, notice that the entire world has been transformed, don’t worry about your damn returns.

  5. Altman responds if compute is cheaper people will want more. He’ll take that bet every day, and the energy will still be useful no matter the scenario.

    1. Good bet, so long as it matters what people want.

  6. Cowen loves Pulse, Altman says people love Pulse, the reason you don’t hear more is it’s only available to Pro users. Altman uses Pulse for a combination of work related news and family opportunities like hiking trails.

    1. I dabble with Pulse. It’s… okay? Most of the time it gives me stories I already know about, but occasionally there’s something I otherwise missed.

    2. I’ve tried to figure out things it will be good at monitoring, but it’s tough, maybe I should invest more time in giving it custom instructions.

    3. In theory it’s a good idea.

    4. It suffers from division of context, since the majority of my recent LLM activity has been on Claude and perhaps soon will include Gemini.

Ooh, fun stuff.

  1. What is Altman’s nuttiest view about his own health? Altman says he used to be more disciplined when he was less busy, but now he eats junk food and doesn’t exercise enough and it’s bad. Whereas before he once got in the hospital for trying semaglutide before it was cool, which itself is very cool.

    1. There’s weird incentives here. When you have more going on it means you have less time to care about food and exercise but also makes it more important.

    2. I’d say that over short periods (like days and maybe weeks) you can and should sacrifice health focus to get more attention and time on other things.

    3. However, if you’re going for months or years, you want to double down on health focus up to some reasonable point, and Altman is definitely here.

    4. That doesn’t mean obsess or fully optimize of course. 80/20 or 90/10 is good.

  2. Cowen says junk food doesn’t taste good and good sushi tastes better, Altman says yes junk food tastes good and sometimes he wants a chocolate chip cookie at 11: 30 at night.

    1. They’re both right. Sometimes you want the (fresh, warm, gooey) chocolate chip cookie and not the sushi, sometimes you want the sushi and not the cookie.

    2. You get into habits and your body gets expectations, and you develop a palate.

    3. With in-context unlimited funds you do want to be ‘spending your calories’ mostly on the high Quality things that are not junk, but yeah in the short term sometimes you really want that cookie.

    4. I think I would endorse that I should eat 25% less carbs and especially ‘junk’ than I actually do, maybe 50%, but not 75% less, that would be sad.

  3. Cowen asks if there’s alien life on the moons of Saturn, says he does believe this. Altman says he has no opinion, he doesn’t know.

    1. I’m actually with Altman in the sense that I’m happy to defer to consensus on the probability here, and I think it’s right not to invest in getting an opinion, but I’m curious why Cowen disagrees. I do think we can be confident there isn’t alien life there that matters to us.

  4. What about UAPs? Altman thinks ‘something’s going on there’ but doesn’t know, and doubts it’s little green men.

    1. I am highly confident it is not little green men. There may or may not be ‘something going on’ from Earth that is driving this, and my default is no.

  5. How many conspiracy theories does Altman believe in? Cowen says zero, at least in the United States. Altman says he’s predisposed to believe, has an X-Files ‘I want to believe’ t-shirt, but still believes in either zero or very few. Cowen says he’s the opposite, he doesn’t want to believe, maybe the White Sox fixed the World Series way back when, Altman points out this doesn’t count.

    1. The White Sox absolutely fixed that 1919 World Series, we know this. At the time it was a conspiracy theory but I think that means this is no longer a conspiracy theory?

    2. I also believe various other sporting events have been fixed, but with less certainty, and to varying degrees – sometimes there’s an official’s finger on the scale but the game is real, other times you’re in Russia and the players literally part the seas to ensure the final goal is scored, and everything in between, but most games played in the West are on or mostly on the level.

    3. Very obviously there exist conspiracies, some of which succeed at things, on various scales. That is distinct from ‘conspiracy theory.’

    4. As a check, I asked Claude for the top 25 most believed conspiracy theories in America. I am confident that 24 out of the 25 are false. The 25th was Covid-19 lab origins, which is called a conspiracy theory but isn’t one. If you modify that to ‘Covid-19 was not only from a lab but was released deliberately’ then I’m definitely at all 25 are false.

  6. Cowen asks again, how would you revitalize St. Louis with a billion dollars and copious free time? Altman says start a Y-Combinator thing, which is pretty similar to what Altman said last time. But he suggests that’s because that would be Altman’s comparative advantage, someone else would do something else.

    1. This seems correct to me.

  1. Should it be legal to release an AI agent into the wild, unowned, untraceable? Altman says it’s about thresholds. Anything capable of self-replication needs oversight, and the question is what is your threshold.

    1. Very obviously it should not be legal to, without checking first, release a self-replicating untraceable unowned highly capable agent into the wild that we have no practical means of shutting down.

    2. As a basic intuition pump, you should be responsible for what an AI agent you release into the wild does the same way you would be if you were still ‘in control’ of that agent, or you hired the agent, or if you did the actions yourself. You shouldn’t be able to say ‘oh that’s not on me anymore.’

    3. Thus, if you cannot be held accountable for it, I say you can’t release it. A computer cannot be held accountable, therefore a computer cannot make a management decision, therefore you cannot release an agent that will then make unaccountable management decisions.

    4. That includes if you don’t have the resources to take responsibility for the consequences, if they rise to the level where taking all your stuff and throwing you in jail is not good enough. Or if the effects cannot be traced.

    5. Certainly if such an agent poses a meaningful risk of loss of human control or of catastrophic or existential risks, the answer needs to be a hard no.

    6. If what you are doing is incompatible with such agents not being released into the wild, then what you are doing, via backchaining, is also not okay.

    7. There presumably should be a method whereby you can do this legally, with some set of precautions attached to it.

    8. Under what circumstances an open weight model would count as any of this is left as an open ended question.

  2. What to do if it happens and you can’t turn it off? Ring-fence it, identify, surveil, sanction the host location? Altman doesn’t know, it’s the same as the current version of this problem, more dangerous but we’ll have better defenses, and we need to urgently work on this problem.

    1. I don’t disagree with that response but it does not indicate a good world state.

    2. It also suggests the cost of allowing such releases is currently high.

  1. Both note (I concur) that it’s great to read your own AI responses but other people’s responses are boring.

    1. I do sometimes share AI queries as a kind of evidence, or in case someone needs a particular thing explained and I want to lower activation energy on asking the question. It’s the memo you hope no one ever needs to read.

  2. Altman says people like watching other people’s AI videos.

    1. Do they, though?

  3. Altman points out that everyone having great personal AI agents is way more interesting than all that, with new social dynamics.

    1. Indeed.

    2. The new social dynamics include ‘AI runs the social dynamics’ potentially along with everything else in short order.

  4. Altman’s goal is a new kind of computer with an AI-first interface very different from the last 50 years of computing. He wants to question basic assumptions like an operating system or opening a window, and he does notice the skulls along the ‘design a new type of computer’ road. Cowen notes that people really like typing into boxes.

    1. Should AI get integrated into computers far more? Well, yeah, of course.

    2. How much should this redesign the computer? I’m more skeptical here. I think we want to retain control, fixed commands that do fixed things, the ability to understand what is happening.

    3. In gaming, Sid Meier called this ‘letting the player have the fun.’ If you don’t have control or don’t understand what is happening and how mechanics work, then the computer has all the fun. That’s no good, the player wants the fun.

    4. Thus my focus would be, how do we have the AI enable the user to have the fun, as in understand what is happening and direct it and control it more when they want to? And also to enable the AI to automate the parts the user doesn’t want to bother about?

    5. I’d also worry a lot about predictability and consistently across users. You simultaneously want the AI to customize things to your preferences, but also to be able to let others share with you the one weird trick or explain how to do a thing.

  1. What would an ideal partnership with a university look like? Altman isn’t sure, maybe try 20 different experiments. Cowen worries that higher education institutions lack internal reputational strength or credibility to make any major changes and all that happens is privatized AI use, and Altman says he’s ok with it.

    1. It does seem like academia and universities in America are not live players, they lack the ability to respond to AI or other changes, and they are mostly going to collect what rents they can until they get run over.

    2. In some senses I agree This Is Fine, obviously it is a huge tragedy all the time and money being wasted but there is not much we can do about this and it will be increasingly viable to bypass the system, or to learn in spite of it.

  2. How will the value of a typical college degree change in 5-10 years? Cowen notes it’s gone down in the last 10, after previously going up. Altman says further decline, faster than before, but not to zero as fast as it should.

    1. Sounds right to me under an ‘economic normal’ scenario.

  3. So what does get returns other than learning AI? Altman says yes, wide benefits to learning to use AI well, including but not limited to things like new science or starting companies.

    1. I notice Altman didn’t name anything non-AI that goes up in value.

    2. I don’t think that’s because he missed a good answer. Ut oh.

  4. How do you teach normies to use AI five years from now, for their own job? Altman says basically people learn on their own.

    1. It’s great that they can learn on their own, but this definitely is not optimal.

    2. As in, you should be able to do a lot better by teaching people?

    3. There’s definitely a common theme of lack of curiosity, where people need pushes in the right directions. Perhaps AI itself can help more with this.

  5. Will we still read books? Altman notes books have survived a lot of things.

    1. Books are on rapid decline already though. Kids these days, AIUI, read lots of text, but basically don’t read books.

  6. Will we start creating our own movies? What else will change? Altman says how we use emails and calls and meetings and write documents will change a lot, family time or time in nature will change very little.

    1. There’s the ‘economic normal’ and non-transformational assumption here, that the outside world looks the same and it’s about how you personally interact with AIs. Altman and Cowen both sneak this in throughout.

    2. Time with family has changed a lot in the last 50-100 years. Phones, computers and television, even radio, the shift in need for various household activities, cultural changes, things like that. I expect more change here, even if in some sense it doesn’t change much, and even if those who are wisest in many ways let it change the least, again in these ‘normal’ worlds.

    3. All the document shuffling, yes, that will change a lot.

    4. Altman doesn’t take the bait on movies and I think he’s mostly right. I mostly don’t want customized movies, I want to draw from the same movies as everyone else, I want to consume someone’s particular vision, I want a fixed document.

    5. Then again, we’ve moved into a lot more consumption of ephemeral, customized media, especially short form video, mostly I think this is terrible, and (I believe Cowen agrees here) I think we should watch more movies instead, I would include television.

    6. I think there’s a divide. Interactive things like games and in the future VR, including games involving robots or LLM characters, are a different kind of experience that should often be heavily customizable. There’s room for personalized, unique story generation, and interactions, too.

  1. Will San Francisco, at least within the West, remain the AI center? Altman says this is the default, and he loves the Bay Area and thinks it is making a comeback.

  2. What about housing costs? Can AI make them cheaper? Altman thinks AI can’t help much with this.

    1. Other things might help. California’s going at least somewhat YIMBY.

    2. I do think AI can help with housing quite a lot, actually. AI can find the solutions to problems, including regulations, and it can greatly reduce ‘transaction costs’ in general and reduce the edge of local NIMBY forces, and otherwise make building cheaper and more tractable.

    3. AI can also potentially help a lot with political dysfunction, institutional design, and other related problems, as well as to improve public opinion.

    4. AI and robotics could greatly impact space needs.

    5. Or, of course, AI could transform the world more generally, including potentially killing everyone. Many things impact housing costs.

  3. What about food prices? Altman predicts down, at least within a decade.

    1. Medium term I’d predict down for sure at fixed quality. We can see labor shift back into agriculture and food, probably we get more highly mechanized agriculture, and also AI should optimize production in various ways.

    2. I’d also predict people who are wealthier due to AI invest more in food.

    3. I wouldn’t worry about energy here.

  4. What about healthcare? Cowen predicts we will spend more and live to 98, and the world will feel more expensive because rent won’t be cheaper. Altman disagrees, says we will spend less on healthcare, we should find cures and cheap treatments, including through pharmaceuticals and devices and also cheaper delivery of services, whereas what will go up in price are status goods.

    1. There’s two different sets of dynamics in healthcare I think?

    2. In the short run, transaction costs go down, people get better at fighting insurance companies, better at identifying and fighting for needed care. Demand probably goes up, total overall real spending goes up.

    3. Ideally we would also be eliminating unnecessary, useless or harmful treatments along the way, and thus spending would go down, since much of our medicine is useless, but alas I mostly don’t expect this.

    4. We also should see large real efficiency gains in provision, which helps.

    5. Longer term (again, in ‘normal’ worlds), we get new treatments, new drugs and devices, new delivery systems, new understanding, general improvement, including making many things cheaper.

    6. At that point, lots of questions come into play. We are wealthier with more to buy, so we spend more. We are wiser and know what doesn’t work and find less expensive solutions and gain efficiency, so we spend less. We are healthier so we spend less now but live longer which means we spend more.

    7. In the default AGI scenarios, we don’t only live to 98, we likely hit escape velocity and live indefinitely, and then it comes down to what that costs.

    8. My default in the ‘good AGI’ scenarios is that we spend more on healthcare in absolute terms, but less as a percentage of economic capacity.

  1. Cowen asks if we should reexamine patents and copyright? Altman has no idea.

    1. Our current systems are obviously not first best, already were not close.

    2. Copyright needs radical rethinking, and already did. Terms are way too long. The ‘AI outputs have no protections’ rule isn’t going to work. Full free fair use for AI training is no good, we need to compensate creators somehow.

    3. Patents are tougher but definitely need rethinking.

  2. Cowen is big on freedom of speech and worries people might want to rethink the First Amendment in light of AI.

    1. I don’t see signs of this? I do see signs of people abandoning support for free speech for unrelated reasons, which I agree is terrible. Free speech will ever and always be under attack.

    2. What I mostly have seen are attempts to argue that ‘free speech’ means various things in an AI context that are clearly not speech, and I think these should not hold and that if they did then I would worry about taking all of free speech down with you.

  3. They discuss the intention to expand free expression of ChatGPT, the famous ‘erotica tweet.’ Perhaps people don’t believe in freedom of expression after all? Cowen does have that take.

    1. People have never been comfortable with actual free speech, I think. Thus we get people saying things like ‘free speech is good but not [misinformation / hate speech / violence or gore / erotica / letting minors see it / etc].’

    2. I affirm that yes LLMs should mostly allow adults full freedom of expression.

    3. I do get the issue in which if you allow erotica then you’re doing erotica now, and ChatGPT would instantly become the center of erotica and porn, especially if the permissions expand to image and even video generation.

  4. Altman wants to change subpoena power with respect to AI, to allow your AI to have the same protections as a doctor or lawyer. He says America today is willing to trust AI on that level.

    1. It’s unclear here if Altman wants to be able to carve out protected conversations for when the AI is being a doctor or lawyer or similar, or if he wants this for all AI conversations. I think it is the latter one.

    2. You could in theory do the former, including without invoking it explicitly, by having a classifier ask (upon getting a subpoena) whether any given exchange should qualify as privileged.

    3. Another option is to ‘hire the AI lawyer’ or other specialist by paying a nominal fee, the way lawyers will sometimes say ‘pay me a dollar’ in order to nominally be your lawyer and thus create legal privilege.

    4. There could also be specialized models to act as these experts.

    5. But also careful what you wish for. Chances seem high that getting these protections would come with obligations AI companies do not want.

    6. The current rules for this are super weird in many places, and the result of various compromises of different interests and incentives and lobbies.

    7. What I do think would be good at a minimum is if ‘your AI touched this information’ did not invalidate confidentiality, whereas third party sharing of information often will do invalidate confidentiality.

    8. Google search is a good comparison point because it ‘feels private’ but your search for ‘how to bury a body’ very much will end up in your court proceeding. I can see a strong argument that your AI conversations should be protected but if so then why not your Google searches?

    9. Similarly, when facing a lawsuit, if you say your ChatGPT conversations are private, do you also think your emails should be private?

  1. Cowen asks about LLM psychosis. Altman says it’s a ‘very tiny thing’ but not a zero thing, which is why the restrictions put in place in response to it pissed users off, most people are okay so they just get annoyed.

    1. Users always get annoyed by restrictions and supervision, and the ones that are annoyed are often very loud.

    2. The actual outright LLM psychosis is rare but the number of people who actively want sycophancy and fawning and unhealthy interactions, and are mostly mad about not getting enough of that, are very common.

I’m going to go full transcript here again, because it seems important to track the thinking:

ALTMAN: Someone said to me once, “Never ever let yourself believe that propaganda doesn’t work on you. They just haven’t found the right thing for you yet.” Again, I have no doubt that we can’t address the clear cases of people near a psychotic break.

For all of the talk about AI safety, I would divide most AI thinkers into these two camps of “Okay, it’s the bad guy uses AI to cause a lot of harm,” or it’s, “the AI itself is misaligned, wakes up, whatever, intentionally takes over the world.”

There’s this other category, third category, that gets very little talk, that I think is much scarier and more interesting, which is the AI models accidentally take over the world. It’s not that they’re going to induce psychosis in you, but if you have the whole world talking to this one model, it’s not with any intentionality, but just as it learns from the world in this continually coevolving process, it just subtly convinces you of something. No intention, it just does. It learned that somehow. That’s not as theatrical as chatbot psychosis, obviously, but I do think about that a lot.

COWEN: Maybe I’m not good enough, but as a professor, I find people pretty hard to persuade, actually. I worry about this less than many of my AI-related friends do.

ALTMAN: I hope you’re right.

  1. On Altman’s statement:

    1. The initial quote is wise.

    2. The division into these three categories is a vast oversimplification, as all such things are. That doesn’t make the distinction not useful, but I worry about it being used in a way that ends up being dismissive.

    3. In particular, there is a common narrowing of ‘the AI itself is misaligned’ into ‘one day it wakes up and takes over the world’ and then people think ‘oh okay all we have to do is ensure that if one day one of them wakes up it doesn’t get to take over the world’ or something like that. The threat model within the category is a lot broader than that.

    4. There’s also ‘a bunch of different mostly-not-bad guys use the AI to pursue their particular interests, and the interactions and competitions and evolutions between them go badly or lead to loss of human control’ and there’s ‘we choose to put the AIs in charge of the world on purpose’ with or without AI having a hand in that decision, and so on and so forth.

    5. On the particular worry here of Altman’s, yes, I think that extended AI conversations are very good at convincing people of things, often in ways no one (including the AI) intended, and as AIs gain more context and adjust to it more, as they will, this will become a bigger and more common thing.

    6. People are heavily influenced by, and are products of, their environment, and of the minds they interact with on a regular basis.

  2. On Cowen’s statement:

    1. A professor is not especially well positioned to be persuasive, nor does a professor typically get that much time with engaged students one-on-one.

    2. When people talk about people being ‘not persuadable’ they typically talk about cases where people’s defenses are relatively high, in limited not-so-customized interactions in which the person is not especially engaged or following their curiosity or trusting, and where the interaction is divorced from their typical social context.

    3. We have very reliable persuasion techniques, in the sense that for the vast majority of human history most people in each area of the world believed in the local religion and local customs and were patriots of the local area and root for the local sports team and support the local political perspectives, and so on, and were persuaded to pass all that along to their own children.

    4. We have a reliable history of armies being able to break down and incorporate new people, of cults being able to do so for new recruits, for various politicians to often be very convincing and the best ones to win over large percentages of people they interact with in person, for famous religious figures to be able to do massive conversions, and so on.

    5. Marxists were able to persuade large percentages of the world, somehow.

    6. Children who attend school and especially go to college tend to exit with the views of those they attend with, even when it conflicts with their upbringing.

    7. If you are talking to an AI all the time, and it has access to your details and stuff, this is very much an integrated social context, so yes many are going over time to be highly persuadable.

    8. This is all assuming AI has to stick to Ordinary Human levels of persuasiveness, which it won’t have to.

    9. There are also other known techniques to persuade humans that we will not be getting into here, that need to be considered in such contexts.

    10. Remember the AI box experiments.

    11. I agree that if we’re talking about ‘the AI won’t in five minutes be able to convince you to hand over your bank account information’ that this will require capabilities we don’t know about, but that’s not the threshold.

  3. If you have a superintelligence ready to go, that is ‘safety-tested,’ that’s about to self-improve, and you get a prompt to type in, what do you type? Altman raises this question, says he doesn’t have an answer but he’s going to have someone ask the Dalai Lama.

    1. I also do not know the right answer.

    2. You’d better know that answer well in advance.

Discussion about this post

On Sam Altman’s Second Conversation with Tyler Cowen Read More »

on-altman’s-interview-with-theo-von

On Altman’s Interview With Theo Von

Sam Altman talked recently to Theo Von.

Theo is genuinely engaging and curious throughout. This made me want to consider listening to his podcast more. I’d love to hang. He seems like a great dude.

The problem is that his curiosity has been redirected away from the places it would matter most – the Altman strategy of acting as if the biggest concerns, risks and problems flat out don’t exist successfully tricks Theo into not noticing them at all, and there are plenty of other things for him to focus on, so he does exactly that.

Meanwhile, Altman gets away with more of this ‘gentle singularity’ lie without using that term, letting it graduate to a background assumption. Dwarkesh would never.

Quotes are all from Altman.

Sam Altman: But also [kids born a few years ago] will never know a world where products and services aren’t way smarter than them and super capable, they can just do whatever you need.

Thank you, sir. Now actually take that to heart and consider the implications. It goes way beyond ‘maybe college isn’t a great plan.’

Sam Altman: The kids will be fine. I’m worried about the parents.

Why do you think the kids will be fine? Because they’re used to it? So it’s fine?

This is just a new tool that exists in the tool chain.

A new tool that is smarter than you are and super capable? Your words, sir.

No one knows what happens next.

True that. Can you please take your own statements seriously?

How long until you can make an AI CEO for OpenAI? Probably not that long.

No, I think it’s awesome, I’m for sure going to figure out something else to do.

Again, please, I am begging you, take your own statements seriously.

There will be some jobs that totally go away. But mostly I think we will rely on the fact that people’s desire for more stuff for better experiences for you know a higher social status or whatever seems basically limitless, human creativity seems basically limitless and human desire to like be useful to each other and to connect.

And AI will be better at doing all of that. Yet Altman goes through all the past falsified predictions as if they apply here. He keeps going on and on as if the world he’s talking about is a bunch of humans with access to cool tools, except by his own construction those tools can function as OpenAI’s CEO and are smarter than people. It is all so absurd.

What people really want is the agency to co-create the future together.

Highly plausible this is important to people. I don’t see any plan for giving it to them? The solution here is redistribution of a large percentage of world compute, but even if you pull that off under ideal circumstances no, that does not do it.

I haven’t heard any [software engineer] say their job lacks meaning [due to AI]. And I’m hopeful at least for a long time, you know, 100 years, who knows? But I’m hopeful that’s what it’ll feel like with AI is even if we’re asking it to solve huge problems for us. Even if we tell it to go develop a cure for cancer there will still be things to do in that process that feel valuable to a human.

Well, sure, not at this capability level. Where is this hope coming from that it would continue for 100 years? Why does one predict the other? What will be the steps that humans will meaningfully do?

We are going to find a way in our own telling of the story to feel like the main characters.

I think the actual plan is for the AI to lie to us? And for us to lie to ourselves? We’ll set it up so we have this idea that we matter, that we are important, and that will be fine? I disagree that this would be fine.

Altman discusses the parallel to discovering that Earth is not the center of the solar system, and the solar system is not the center of the galaxy, and so on, little blue dot. Well sure, but that wasn’t all that load bearing, we’re still the center of our own universes, and if there’s no other life out there we’re the only place that matters. This is very different.

Theo asks what Altman’s fears are about AI. Altman responds with a case where he couldn’t do something and GPT-5 could do it. But then he went on with his day. His second answer is impact on user mental health with heavy usage, which is a real concern and I’m glad he’s scared about that.

And then… that’s it. That’s what scares you, Altman? There’s nothing else you want to share with the rest of us? Nothing about loss of control issues, nothing about existential risks, and so on? I sure as hell hope that he is lying. I do think he is?

When asked about a legal framework for AI, Altman asks for AI privilege, sees this as urgent, and there is absolutely nothing else he thinks is worth mentioning that requires the law to adjust.

The last few months have felt very fast.

Theo then introduces Yoshua Bengio into the conversation, bringing up deception and sycophancy and neurolese.

We think it’s going to be great. There’s clearly real risks. It kind of feels like you should be able to say something more than that, But in truth, I think all we know right now is that we have discovered, invented, whatever you want to call it, something extraordinary that is going to reshape the course of human history. Dear God, man. But if you don’t know, we don’t know.

Well, of course. I mean, I think no one can predict the future. Like human society is very complex. This is an amazing new technology. Maybe a less dramatic example than the atomic bomb is when they discovered the transistor a few years later.

Yes, we can all agree we don’t know. We get a lot of good attitude, the missing mood is present, but it doesn’t cash out in the missing concerns. ‘There’s clearly real risks’ but that in context seems to apply to things like jobs and meaning and distribution given all the context.

There’s no time in human history at the beginning of the century when the people ever knew what the end of the century was going to be like. Yeah. So maybe it’s I do think it goes faster and faster each century.

The first half of this seems false for quite a lot of times and places? Sure, you don’t know how the fortunes of war might go but for most of human history ‘100 years from now looks a lot like today’ was a very safe bet. Nothing ever happens (other than cycling wars and famines and plagues and so on) did very well. But yes, in 1800 or 1900 or 2000 you would have remarkably little idea.

It certainly feels like [there is a race between companies.]

Theo equates this race to Formula 1 and asks what the race is for. AGI? ASI? Altman says benchmarks are saturated and it’s all about what you get out of the models, but we are headed for some model.

Maybe it’s a system that is capable of doing its own AI research. Maybe it’s a system that is smarter than all of humans put together… some finish line we are going to cross… maybe you call that superintelligence. I don’t have a finish line in mind.

Yeah, those do seem like important things that represent effective ‘finish lines.’

I assume that what will happen, like with every other kind of technology, is we’ll realize there’s this one thing that the tool’s way better than us at. Now, we get to go solve some other problems.

NO NO NO NO NO! That is not what happens! The whole idea is this thing becomes better at solving all the problems, or at least a rapidly growing portion of all problems. He mentions this possibility shortly thereafter but says he doesn’t think ‘the simplistic thing works.’ The ‘simplistic thing’ will be us, the humans.

You say whatever you want. It happens, and you figure out amazing new things to build for the next generation and the next.

Please take this seriously, consider the implications of what you are saying and solve for the equilibrium or what happens right away, come on man. The world doesn’t sit around acting normal while you get to implement some cool idea for an app.

Theo asks, would regular humans vote to keep AI or stop AI? Altman says users would say go ahead and users would say stop. Theo predicts most people would say stop it. My understanding is Theo is right for the West, but not for the East.

Altman asks Theo what he is afraid of with AI, Theo seems worried about They Took Our Jobs and loss of economic survival and also meaning, that we will be left to play zero-sum games of extraction. With Theo staying in Altman’s frame, Altman can pivot back to humans liking to be creative and help each other and so on and pour on the hopium that we’ll all get to be creatives.

Altman says, you get less enjoyment from a ghost robotic kitchen setup, something is missing, you’d rather get the food from the dude who has been making it. To which I’d reply that most of this is that the authentic dude right now makes a better product, but that ten years from now the robot will make a better product than the authentic dude. And yeah, there will still be some value you get from patronizing the dude, but mostly what you want is the food and thus will the market speak, and then we’ve got Waymos with GLP-1 dart guns and burrito cannons for unknown reasons when what you actually get is a highly cheap and efficient delicious food supply chain that I plan on enjoying very much thank you.

We realized actually this is not helping me be my best. you know, like doing the equivalent of getting the like burrito cannon into my mouth on my phone at night, like that’s not making me long-term happy, right? And that’s not helping me like really accomplish my true goals in life. And I think if AI does that, people will reject it.

I mean I think a thing that efficiently gives you burritos does help you with your goals and people will love it, if it’s violently shooting burritos into your face unprompted at random times then no but yeah it’s not going to work like that.

However, if Chhat GBT really helps you to figure out what your true goals in life are and then accomplish those, you know, it says, “Hey, you’ve said you want to be a better father or a better, you know, you want to be in better shape or you, you know, want to like grow your business.

I refer Altman to the parable of the whispering earring, but also this idea that the AI will remain a tool that helps individual humans accomplish their normal goals in normal ways only smarter is a fairy tale. Altman is providing hopium via the implicit overall static structure of the world, then assuming your personal AI is aligned to your goals and well being, and then making additional generous assumptions, and then saying that the result might turn out well.

On the moratorium on all AI regulations that was stripped from the BBB:

There has to be some sort of regulation at some point. I think it’d be a mistake to let each state do this kind of crazy patchwork of stuff. I think like one countrywide approach would be much easier for us to be able to innovate and still have some guardrails, but there have to be guardrails.

The proposal was, for all practical purposes, to have no guardrails. Lawmakers will say ‘it would be better to have one federal regulation than fifty state regulations’ and then ban the fifty state regulations but have zero federal regulation.

The concerns [politicians come to us with] are like, what is this going to do to our kids? Are they going to stop learning? Is this going to spread fake information? Is this going to influence elections? But we’ve never had ‘you can’t say bad things about the president.’

That’s good to hear versus the alternative, better those real concerns than an attempt to put a finger on the scale, although of course these are not the important concerns.

We could [make it favor one candidate over another]. We totally could. I mean, we don’t, but we totally could. Yeah… a lot of people do test it and we need to be held to a very high standard here… we can tell.

As Altman points out, it would be easy to tell if they made the model biased. And I think doing it ‘cleanly’ is not so simple, as Musk has found out. Try to put your finger on the scale and you get a lot of side effects and it is all likely deeply embarrassing.

Maybe we build a big Dyson sphere on the solar system.

I’m noting that because I’m tired of people treating ‘maybe we build a Dyson sphere’ as a statement worthy of mockery and dismissal of a person’s perspective. Please note that Altman thinks this is very possibly the future.

You have to be both [excited and scared]. I don’t think anyone could honestly look at the trajectory humanity is on and not feel both excited and scared.

Being chased by a goose, asking scared of what. But yes.

I think people get blinded by ambition. I think people get blinded by competition. I think people get caught up like very well-meaning people can get caught up in very negative incentives. Negative for society as a whole. By the way, I include us in this.

I think people come in with good intentions. They clearly sometimes do bad stuff.

I think Palantir and Peter Thiel do a lot of great stuff… We’re very close friends…. His brain just works differently… I’m grateful he exists because he thinks the things no one else does.

I think we really need to prioritize the right to privacy.

I’m skipping over a lot of interactions that cover other topics.

Altman is a great guest, engaging, fun to talk to, shares a lot of interesting thoughts and real insights, except it is all in the service of painting a picture that excludes the biggest concerns. I don’t think the deflections I care about most (as in, flat out ignoring them hoping they will go away) are the top item on his agenda in such an interview, or in general, but such deflections are central to the overall strategy.

The problem is that those concerns are part of reality.

As in, something that, when you stop looking at it, doesn’t go away.

If you are interviewing Altman in the future, you want to come in with Theo’s curiosity and friendly attitude. You want to start by letting Altman describe all the things AI will be able to do. That part is great.

Except also do your homework, so you are ready when Altman gives answers that don’t make sense, and that don’t take into account what Altman says that AI will be able to do. That notices the negative space being not mentioned, and that points it out. Not as a gotcha or an accusation, but to not let him get away with ignoring it.

At minimum, you have to point out that the discussion is making one hell of a set of assumptions, ask Altman if he agrees that those assumptions are being made, and check if how confident he is those assumptions are true, and why, even if that isn’t going to be your focus. Get the crucial part on the record. If you ask in a friendly way I don’t think there is a reasonable way to dodge answering.

Discussion about this post

On Altman’s Interview With Theo Von Read More »

ai-#51:-altman’s-ambition

AI #51: Altman’s Ambition

Sam Altman is not playing around.

He wants to build new chip factories in the decidedly unsafe and unfriendly UAE. He wants to build up the world’s supply of energy so we can run those chips.

What does he say these projects will cost?

Oh, up to seven trillion dollars. Not a typo.

Even scaling back the misunderstandings, this is what ambition looks like.

It is not what safety looks like. It is not what OpenAI’s non-profit mission looks like. It is not what it looks like to have concerns about a hardware overhang, and use that as a reason why one must build AGI soon before someone else does. The entire justification for OpenAI’s strategy is invalidated by this move.

I have spun off reactions to Gemini Ultra to their own post.

  1. Introduction.

  2. Table of Contents.

  3. Language Models Offer Mundane Utility. Can’t go home? Declare victory.

  4. Language Models Don’t Offer Mundane Utility. Is AlphaGeometry even AI?

  5. The Third Gemini. Its own post, link goes there. Reactions are mixed.

  6. GPT-4 Real This Time. Do you remember when ChatGPT got memory?

  7. Deepfaketown and Botpocalypse Soon. Bot versus bot, potential for AI hacking.

  8. They Took Our Jobs. The question is, will they also take the replacement jobs?

  9. Get Involved. A new database of surprising AI actions.

  10. Introducing. Several new competitors.

  11. Altman’s Ambition. Does he actually seek seven trillion dollars?

  12. Yoto. You only train once. Good luck! I don’t know why. Perhaps you’ll die.

  13. In Other AI News. Andrej Karpathy leaves OpenAI, self-discover algorithm.

  14. Quiet Speculations. Does every country need their own AI model?

  15. The Quest for Sane Regulation. A standalone post on California’s SR 1047.

  16. Washington D.C. Still Does Not Get It. No, we are not confused about this.

  17. Many People are Saying. New Yorkers do not care for AI, want regulations.

  18. China Watch. Not going great over there, one might say.

  19. Roon Watch. If you can.

  20. How to Get Ahead in Advertising. Anthropic super bowl ad.

  21. The Week in Audio. Sam Altman at the World Government Summit.

  22. Rhetorical Innovation. Several excellent new posts, and a protest.

  23. Please Speak Directly Into this Microphone. AI killer drones now?

  24. Aligning a Smarter Than Human Intelligence is Difficult. Oh Goody.

  25. Other People Are Not As Worried About AI Killing Everyone. Timothy Lee.

  26. The Lighter Side. So, what you’re saying is…

Washington D.C. government exploring using AI for mundane utility.

Deliver your Pakistani presidential election victory speech while you are in prison.

Terrance Tao suggests a possible application for AlphaGeometry.

Help rescue your Fatorio save from incompatible mods written in Lua.

Shira Ovide says you should use it to summarize documents, find the exact right word, get a head start on writing something difficult, dull or unfamiliar, or make cool images you imagine, but not to use it to get info about an image, define words, identify synonyms, get personalized recommendations or to give you a final text. Her position is mostly that this second set of uses is unreliable. Which is true, and you do not want to exclusively or non-skeptically rely on the outputs, but so what? Still seems highly useful.

AlphaGeometry is not about AI? It seems that what AlphaGeometry is mostly doing is combining DD+AR, essentially labeling everything you can label and hoping the solution pops out. The linked post claims that doing this without AI is good enough in 21 of the 25 problems that it solved, although a commentor notes the paper seems to claim it was somewhat less than that. If it was indeed 21, and to some extent even if it wasn’t, then what we learned was less that AI can do math, and more that IMO problems are very often solvable by DD+AR.

That makes sense, IMO geometry is a compact space. One could still also ask, how often will it turn out that our problems have that look hard turn out to be solvable simple or brute force ways? And if AI then figures out what those ways are, or the use of AI allows us to figure it out, or the excuse of AI enables us to find it, what are the practical differences there?

The comments have some interesting discussions about whether IMO problems and experiences are good for training human mathematicians and are a good use of time, or not. My guess is that they are a very good use of time relative to salient alternatives, but also the samples involved are of course hopelessly confounded so there is no good way to run a study, on so many levels. The network effects likely are a big game too, as are the reputational and status effects.

ChatGPT gets a memory feature, which you can turn on and off to control what it remembers, or give it specific notes on purpose. Right now it is limited to a few users. You go to Settings > Personalization > Memory, or you find that there is no ‘Personalization’ section for you yet.

It works via a bio in which it saves snippets of persistent information.

Some of you are in for a nasty surprise, perhaps?

Yes, your periodic reminder that the ChatGPT system prompt appears to be 1700 tokens and full of things it is hard to be polite when describing.

Kevin Fischer makes clear that he believes that yes, the open source vs. closed source gap is large, they haven’t even caught up to GPT-3.5 yet.

Kevin Fischer: I’m noticing a lot of open source models performing well on benchmarks against OpenAI’s 3.5, sometimes beating them, including in Chatbot Arena

But! Even the Chatbot Arena upset is super misleading. GPT 3.5 is still WAY smarter than any current open source model. The benchmarks out there test the models as if they’re single entities, but that’s actually not the correct frame for these objects

GPT should be thought more of a semantic processor for issuing instructions, and when testing against that sort of frame, 3.5 is still way ahead of any open source model in its general intelligence. @OpenAI still has a significant lead here.

Floating Point: What do you see with Claude and Gemini?

Kevin Fischer: Haven’t experimented much with Gemini – Claude is very smart, but handicapped with Safetyist philosophy.

What does ‘smart’ mean in this context? It will vary depending on who you ask. Kevin’s opinion is definitely not universal. But I am guessing that Kevin is right that, while Arena is far better than standard benchmarks, it is still not properly taking raw model intelligence into account.

Remember to be nice to your GPT-4 model. Put in a smiley face, tell it to take a break. It is a small jump, but when you care every little bit helps. Similar to humans, perhaps, in that it is only sometimes worth the effort to motivate them a little better. How long until these things get done for you?

How worried should one be here?

Andrew: We’re so cooked lol.

Blessing: I’ve been saying from the start; AI doesn’t need to f0ol YOU, it only needs to fool the tens of thousands of people who haven’t spent the last year learning the signs to identify AI generated photos, and it’s doing a great job of that.

The photo in question isn’t merely ‘I can tell it is AI,’ it is ‘my brain never considered the hypothesis that it was not AI.’ That style is impossible to miss.

Regular people, it seems, largely did not see it. But also they did not have to, so they were not on alert, and no one found it important to correct them. And they haven’t yet had the practice. So my inclination is probably not too worried?

Meanwhile, you say ‘the end of the free, open, human internet’ and I say ‘hey free public chatbot.’

Alan Cole: We may be witnessing the beginning of the end for the open, free, and human internet. But, well, at least some of us are enjoying ourselves as Rome burns.

Alas, the actual account looks like it is a person and is mostly in what I presume is Arabic, so not that exciting.

Bots also are reported to be on the rise on Reddit.

Emmett Shear has a talk about this, saying we will need relational definitions of truth and authenticity. I am unsure, and would be careful to say trust and authenticity, rather than truth.

Zebleck: Just a personal anecdote and maybe a question, I’ve been seeing a lot of AI-generated textposts in the last few weeks posing as real humans, feels like its ramping up. Anyone else feeling this?

At this point the tone and smoothness of ChatGPT generated text is so obvious, it’s very uncanny when you find it in the wild since its trying to pose as a real human, especially when people responding don’t notice. Heres an example bot: u/deliveryunlucky6884

I guess this might actually move towards taking over most reddit soon enough. To be honest I find that very sad, Reddit has been hugely influential to me, with thousands of people imparting their human experiences onto me. Kind of destroys the purpose if it’s just AIs doing that, no?

A contrast not noticed as often as it should be:

Aella: Weird how people are worried that ai porn will kill men’s desire for real girlfriends, but are convinced that ai ___ porn will increase desire for __. I’m not saying either one is right, just seems a bit inconsistent.

Research consistently says that, for previous human levels of porn, the net effect has been to reduce incidence of anti-social sexual behaviors.

The direction of porn access in general on pro-social activities in general is a question I have not seen good data on? I did an Elicit search and there was nothing on point at all, pornography is ‘associated’ with more sexual behavior but that is not causal, and mostly it is people warning about behavior shifts they can label as problematic. File under questions that are rarely asked?

My prediction continues to be that AI girlfriends and other related offerings will not be net negative for actual dating any time soon.

Scam versus scam?

f4mi: this is insane the spambots here are now programmed to spam keywords posts for other spambots to answer to so so that these other competing spambots get cluttered with bogus requests and are slower to answer and therefore less effective at scamming people.

what the f?

this makes me wonder how many girls are posting online about wanting a glucose father so that this scam is profitable enough to keep going at this scale.

This could be scam versus scam violence, but my guess is it is not. This is more likely to be a classic honeypot strategy. If a bot responds, you can report it or block it. If the ‘you’ in question is Twitter, then you can ban it, or have it universally muted into the void, or you can try to toy with it and waste its time and maybe turn the tables, as desired. The sky is the limit.

Can the good guy with an AI stop the bad guy with an AI? Sometimes yes, sometimes no, same as the same guys without AIs. In the case where the bad AI is optimizing to target absurdly stupid people, I would presume that defenses would be relatively more effective.

A quiz on whether it is an AI or human doing a breakup.

What about AI gents hacking websites? A new paper says we are there.

Daniel Kang: As LLMs have improved in their capabilities, so have their dual-use capabilities. But many researchers think they serve as a glorified Google.

We show that LLM agents can autonomously hack websites, showing they can produce concrete harm.

Our LLM agents can perform complex hacks like blind SQL union attacks. These attacks can take up to 45+ actions to perform and require the LLM to take actions based on feedback.

We further show a strong scaling law, with only GPT-4 and GPT-3.5 successfully hacking websites (73% and 7%, respectively). No open-source model successfully hacks websites.

Our results raise questions about the widespread deployment of LLMs, particularly open-source LLMs. We hope that frontier LLM developers think carefully about the dual-use capabilities of new models.

The jump from GPT-3.5 to GPT-4 is huge there. The failure of the open source models to succeed is one more reminder that they have universally failed to exceed or perhaps even reach the 3.5 threshold.

In the world’s least newsworthy headline, Microsoft and OpenAI say USA’s rivals are using AI in hacking. I guess technically it is news, of course everyone who hacks is using AI to aid in their hacking, but I did not know those companies were saying it.

In more meaningful news, OpenAI has identified five actors trying to use OpenAI’s services to access various information and complete various coding and other tasks, except they had in mind ultimately dirty deeds, so their accounts have been terminated. I do not see how, in practice, they can prevent those actors from opening new accounts and doing it anyway? I also don’t see much harm here.

Noema’s David Autor, inventor of the ‘China Shock,’ speaks of AI as having the potential to revive the middle class. Like many who think about AI, he is imagining the AI-Fizzle scenario, where AI as it exists today gets only marginally better, with the world remaining fundamentally human with AI as a tool and not even that powerful a tool.

Within that framework his core concept is that AI is fundamentally a way of creating supply of certain kinds of expertise, and that this combined with demographic trends will be very good for the middle class. There will be high demand for what they can provide. As usual, economists assume that technological improvement will always create jobs to replace the ones taken away, which has been true in the past, and the question is what types and quality of jobs are created and destroyed.

Economists continue to be very good at thinking about the next few years in terms of mundane utility and the practical implications, while refusing on principle to consider the possibility that the future beyond that will be fundamentally different from the present. Meanwhile Nvidia hit another all-time high as I typed this.

Once again not understanding that this time might be different:

Paul Graham: Historically, letting technology eliminate their jobs has been a sacrifice people have made for their kids’ sakes. Not intentionally, for the most part, but their kids ended up with the new jobs created as a result. No one weaves now, and that’s fine.

Geoffrey Miller: The thing is, Artificial General Intelligence, by definition, will be able to do _any_ cognitive task that humans can do — meaning any jobs that humans can do — including jobs that haven’t been invented yet.

That’s the problem. All previous tech eliminate some jobs but created some new jobs. AGI won’t create any new jobs that can’t be done better by AGI than by humans. It’s guaranteed mass unemployment.

Once more with feeling: The reason why new jobs always resulted from old jobs was because humans were the strongest cognitive beings and optimization engines on the planet, so automating or improving some tasks opened up others. If we built AGI, that would cease to be the case. We would create new tasks to compete and find new ways to provide value, and then the AGI would do those as well.

Which, of course, could be amazing, if it means we are free to do things other than ‘jobs’ and ‘work’ and live in various forms of abundance while dealing with distributional impacts. The obsession with jobs can get quite out of hand. It is also, however, a sign that humans will by default stop being competitive or economically viable.

It is also strange to say that X makes a sacrifice in order to get Y, but unintentionally. I did not think that was how sacrifice works? If it was not intentional it is simply two consequences of an action.

Know cases where an AI acted in ways that surprised its creator? You can submit them here, Jess Clune is building a database.

Is working for the EU AI Office a good way to get involved? Help Max decide.

Gab.ai? I saw a pointer to it but what do we do with another generic box? Says it is uncensored and unbiased. I mean, okay, I guess, next time everyone else keeps refusing I’ll see what it can do?

Chat with RTX, an AI chatbot from Nvidia that runs locally on your PC. I saw no claims on how good it is. It can search all your files, which would be good except that most of my files are in the cloud. The name is of course terrible, Bloomberg’s Dave Lee says naming AI systems is hard because you want to sound both cutting edge and cool. I agree with him that Bard was a good name, I would have stuck with it.

Goose, Google’s new AI to help create new products and help with coding, based on 25 years of Google’s engineering expertise. Introduced internally, that is. You can’t have it.

And if Googlers have specific development questions while using Goose, they’re encouraged to turn to the company’s internal chatbot, named Duckie.

Love it. Google is big enough that the advantage of having a better LLM and better coding abilities than the rest of the word could plausibly exceed the value of offering those skills on the market. Let’s (not) go. Now, let’s talk about all those papers.

Sam Altman looking to raise money for a project projected to perhaps ultimately cost five to seven trillion (yes, trillion with a T) dollars to build power and chip capacity. This would dwarf the existing semiconductor industry, or even all corporate-issued debt, or the GDP of the countries he is in talks with to provide funding, and is a good fraction of the debts of the American government.

We should be careful not to take this 7 trillion dollar number too seriously. He is not attempting to raise that much capital right away. Which is good news for him, since that is not a thing that is possible to do.

Daniel Eth: I feel like the “possibly requiring up to” part of this is doing a lot of legwork. I obviously don’t know what this is about, but no, Sama isn’t actively raising $7T right now

Timothy Lee: There is no way this is a real number. $7 trillion is like two orders of magnitude larger than any private investment in any project in the history of the world.

Tack’s annual capex is around $35 billion, so we are talking about 200 times the spending of the largest fab company on a single project. Even if this somehow made sense from a demand perspective the world just doesn’t have the tangible resources to build a hundred tsmcs.

Even in terms of the full project, notice that there are two things here, the chips and the power.

If you are going to spend 7 trillion dollars, there are not that many things you can in theory spend it on.

Chips are not on that list. Electrical power potentially is on that list. It is a much, much bigger industry, with much bigger potential to usefully spend.

The power part of the plan I can get behind. The world could use massively more clean energy faster for so many reasons. I have not seen a cost breakdown, but the power almost has to be most of it? Which would be trillions for new power, and all of the realistic options available for that are green. Wouldn’t it be wild if Sam Altman used the fig leaf of AI to go solve climate change with fusion plants?

Scott Alexander breaks down the plan and request, noting that currently we do not have the compute or power necessary to train several generations ahead if the costs continue to scale the same way they have so far.

Tolga Bilge notes:

OpenAI’s website: Building AGI fast is safer because the takeoff will be slow since there’s still not too much compute around.

Sam Altman: Give me 7 trillion dollars for GPUs

This is consistent with: He’ll say/do at the time whatever permits him to build AGI as fast as possible.

Ronny Fernandez: It was actually Sam Altman who wrote the article, so it’s Sam Altman writing that, not just openAI.

Juan Gil: If Sam does this, then the “computer overhang” reasoning for pushing capabilities forward was bullshit, right?

Garrison writes a short post explaining this: Sam Altman’s Chip Ambitions Undercut OpenAI’s Safety Strategy.

The chip plan seems entirely inconsistent with both OpenAI’s claimed safety plans and theories, and with OpenAI’s non-profit mission. It looks like a very good way to make things riskier faster. You cannot both try to increase investment on hardware by orders of magnitude, and then say you need to push forward because of the risks of allowing there to be an overhang.

Or, well, you can, but we won’t believe you.

This is doubly true given where he plans to build the chips. The United States would be utterly insane to allow these new chip factories to get located in the UAE. At a minimum, we need to require ‘friend shoring’ here, and place any new capacity in safely friendly countries.

Also, frankly, this is not The Way in any sense and he has to know it:

Sam Altman: You can grind to help secure our collective future or you can write substacks about why we are going fail.

guerilla artfare: do you have any idea how many substacks are going to be written in response to this? DO YOU.

Hey, hey, I’m grinding here, no one pretend otherwise. Still unhappy that Tyler Cowen passed me up at the writing every day awards.

What exactly does he think someone is doing, when they are trying to figure out and explain to others how we are going to fail?

We are trying to ensure that we do not fail, that’s what. Or, if we were already going to succeed, to be convinced of this.

If I thought that accelerating AI development was the way to secure our collective future, I would be doing that. There is way more money in it. I would have little trouble getting hired or raising funds. It is fascinating and fun as hell, I have little doubt. I am constantly having ideas and getting frustrated that I do not see anyone trying them – even when I am happy no one is trying them, it is still frustrating.

Of course, English is strange, so you can interpret the statement the other way, the actually correct way: That some of you should do one thing and some of you should do the other. Division of labor is a thing, and we need both people building bridges and people trying to figure out the ways those bridges would fall down so we can modify the designs of the bridges, or if necessary or the economics don’t make sense to not build a particular bridge.

You Only Train Once:

Jason Wei: An incredible skill that I have witnessed, especially at OpenAI, is the ability to make “yolo runs” work.

The traditional advice in academic research is, “change one thing at a time.” This approach forces you to understand the effect of each component in your model, and therefore is a reliable way to make something work. I personally do this quite religiously. However, the downside is that it takes a long time, especially if you want to understand the interactive effects among components.

A “yolo run” directly implements an ambitious new model without extensively de-risking individual components. The researcher doing the yolo run relies primarily on intuition to set hyperparameter values, decide what parts of the model matter, and anticipate potential problems. These choices are non-obvious to everyone else on the team.

Yolo runs are hard to get right because many things have to go correctly for it to work, and even a single bad hyperparameter can cause your run to fail. It is probabilistically unlikely to guess most or all of them correctly.

Yet multiple times I have seen someone make a yolo run work on the first or second try, resulting in a SOTA model. Such yolo runs are very impactful, as they can leapfrog the team forward when everyone else is stuck.

I do not know how these researchers do it; my best guess is intuition built up from decades of running experiments, a deep understanding of what matters to make a language model successful, and maybe a little bit of divine benevolence. But what I do know is that the people who can do this are surely 10-100x AI researchers. They should be given as many GPUs as they want and be protected like unicorns.

When is it more efficient to do a Yoro, versus a standard approach, in AI or elsewhere? That depends on how likely it is to work given your ability to guess the new parameters, how much time and money it costs to run each iteration, and how much you can learn from what results you get from each approach. What are your scarce resources? To what extent is it the time of your top talent?

Yoro also allows you to do multiple things that rely on each other. If you have to hill climb on each change, that is not only slow, it can cut off promising approaches.

I have definitely pulled off Yoro in various capacities. My Aikido model of baseball was a Yoro. Many of my best Magic decks, including Mythic, were Yoro.

There is an obvious downside, as well. Training new state of the art models by changing tons of things according to intuition and seeing what happens does… not… seem… especially… safe?

Geoffrey Miller: Doing a lot of YOLO runs with advanced AI systems sounds like the exact opposite of being safe with advanced AI systems.

Good to know that @OpenAI has abandoned all pretense of caring about safety.

I guess the new principle is YOGEO – you only go extinct once.

Roon: the principles for safe AGI will also be discovered by big bets and cowboy attitude.

Roon could easily be right. I do think a lot of things are discovered by big bets and cowboy attitude. Trying out bold new safety ideas, in a responsible manner, could easily involve big (resource or financial) bets.

There is also such a thing as bankroll management.

If you place a big cowboy bet, and you are betting the company, then any gambler will tell you that you do not get to keep doing that, there is a rare time and a place for it, and you better be damn sure you are right or have no choice. But sometimes, when that perfect hand comes along, you bet big, and then you take the house.

If you place a big cowboy bet, and the cost of losing it is human extinction, then any gambler will tell you that this is not good bankroll management.

There are of course different kinds of Yoro runs.

Andrej Karpathy left OpenAI. We do not know why, other than that whatever the reason he is not inclined to tell us. Could be anything.

Andrej Karpathy: Hi everyone yes, I left OpenAI yesterday. First of all nothing “happened” and it’s not a result of any particular event, issue or drama (but please keep the conspiracy theories coming as they are highly entertaining :)). Actually, being at OpenAI over the last ~year has been really great – the team is really strong, the people are wonderful, and the roadmap is very exciting, and I think we all have a lot to look forward to. My immediate plan is to work on my personal projects and see what happens. Those of you who’ve followed me for a while may have a sense for what that might look like 😉 Cheers

A new technique called ‘self-discover’ is claimed to greatly improve performance of GPT-4 and PaLM 2 on many benchmarks. Note as David does that we can expect further such improvements in the future, so you cannot fully count on evaluations to tell you what a model can and cannot do, even in the best case.

Here is the abstract:

We introduce SELF-DISCOVER, a general framework for LLMs to self-discover the task-intrinsic reasoning structures to tackle complex reasoning problems that are challenging for typical prompting methods. Core to the framework is a self-discovery process where LLMs select multiple atomic reasoning modules such as critical thinking and step-by-step thinking, and compose them into an explicit reasoning structure for LLMs to follow during decoding.

SELF-DISCOVER substantially improves GPT-4 and PaLM 2’s performance on challenging reasoning benchmarks such as BigBench-Hard, grounded agent reasoning, and MATH, by as much as 32% compared to Chain of Thought (CoT). Furthermore, SELFDISCOVER outperforms inference-intensive methods such as CoT-Self-Consistency by more than 20%, while requiring 10-40x fewer inference compute. Finally, we show that the self-discovered reasoning structures are universally applicable across model families: from PaLM 2-L to GPT-4, and from GPT-4 to Llama2, and share commonalities with human reasoning patterns.

I hear you, Shakeel. I hear you.

Shakeel: I constantly talk myself out of buying stocks bc I assume obvious things like “AI will lead to high chip demand” are priced in… and then stuff like this happens and I kick myself to death.

Joe Weisenthal: Not often you see a company this big surge this much in one day. $ARM now a $123 billion co after surging 56% so far today.

I have indeed bought some of the obvious things, and that part of my portfolio is doing fine. But oh my could things have gone so much better if I’d gone for it.

Sebastian Ruder offers thoughts on the AI job market. Many good notes, most with what I would consider flipped reactions – he is worried that things are too practical rather than theoretical, too closed rather than open, publishing is getting harder to justify, and this may interfere with capabilities progress. Whereas I am excited to see people focus on mundane utility and competitive advantages, in ways that do not bring us closer to death.

More agents are all you need? This paper says yes.

Aran Komatsuzkai: More Agents Is All You Need Finds that, simply via a sampling-and-voting method, the performance of LLMs scales with the number of agents instantiated.

Abstract: We find that, simply via a sampling-and-voting method, the performance of large language models (LLMs) scales with the number of agents instantiated. Also, this method is orthogonal to existing complicated methods to further enhance LLMs, while the degree of enhancement is correlated to the task difficulty. We conduct comprehensive experiments on a wide range of LLM benchmarks to verify the presence of our finding, and to study the properties that can facilitate its occurrence. Our code is publicly available [here].

Simeon: The improvements from “More Agents Is All You Need”, especially on LLaMa2-13B, are pretty surprising to me. We’re still far from knowing the upper bound of capabilities of any LLM.

There has been a strange lack of enthusiasm about strategies of the form ‘use technique that queries the model lots of times to make the answer better.’ Until we have explored such spaces more there might be a lot of room for improvement of output quality, although it would come at a cost in efficiency.

The results here show a very large leap from zero agents to ten, such that we need to see the answers for one, or for three. The gains from there are smaller. I am suspicious of the gains from 30 to 40 being even as large as they are here, this is not a log scale.

They note that the harder the task the larger the efficiency gains here, up to a point where the problem gets too difficult and the gains taper off. Makes sense.

I do not think this shows that ‘more agents are all you need.’ It does show that you can get a substantial boost this way if you can spare the compute. I would have predicted the effect, agents have large issues with failure on individual steps and going in circles and making dumb mistakes and a consensus seems likely to help, so it almost has to be helpful, but I would have had no idea on the magnitude.

I also want to call out the ‘impact statement’ at the end, because it is so disconnected from the topic at hand.

This paper introduces a simple method designed to enhance the performance of Large Language Models (LLMs). While the proposed method aims to improve the efficacy of LLMs in various tasks, it is necessary to acknowledge the potential risks.

LLMs can sometimes produce outputs that, while plausible, may be factually incorrect or nonsensical. Such hallucinations can lead to the misguidance of decisionmaking processes and the propagation of biases. These concerns are particularly acute in the context of critical decision-making scenarios, where the accuracy and reliability of information are paramount.

The broader adoption of LLMs, without adequate safeguards against these risks, could exacerbate these issues. Therefore, it is crucial to continue developing mechanisms to mitigate the potential adverse effects of LLM hallucinations to ensure that the deployment of these powerful models is both responsible and beneficial.

They are attempting to develop the ability to scale the skills of AI agents. I am not saying their research is unethical to publish, but this statement does not scratch the surface of even the mundane risks of improving AI agent performance, let alone mention the existential dangers if applied to sufficiently advanced and capable models.

Transcript of talk on AI by Scott Aaronson, covering all his bases.

He asks how we would know if an AI could compose genuinely different music the way The Beatles did, noting that they carried along all of civilization so the training data is corrupted. Well, it is not corrupted if you only feed in data from before a given date, and then do recursive feedback without involving any living humans. That is severely limiting, to be sure, but it is the test we have. Or we could have it do something all of us haven’t done yet. That works too.

His brainstorming suggestion for ensuring a good future is that perhaps we could focus on minds that operate in ways that make them impossible to copy, via ‘instilling a new religion’ into them. The theory is that if an AI can be copied, then it does not matter, it is one’s uniqueness that makes you special. He ends this way:

Does this help with alignment?  I’m not sure.  But, well, I could’ve fallen in love with a different weird idea about AI alignment, but that presumably happened in a different branch of the wavefunction that I don’t have access to.  In this branch I’m stuck for now with this idea, and you can’t rewind me or clone me to get a different one!  So I’m sorry, but thanks for listening.

I do not think that would work for overdetermined reasons. It is still better thinking than most similar proposals.

New 90-page paper from Gavin Leech and others uses the frame of ‘ten hard problems’ from Eric Schmidt and James Manyika, that we must solve these ten problems if we want good outcomes to result from AI by 2050.

I like the overall idea of pointing out there are lots of places things can go haywire and fail, many of which are extremely hard, whereas even one failure could be fatal, or sufficient to turn the situation quite bleak.

Are these the right things to be concerned about? Did we pick a good ten?

Looking above, I would say that this focuses heavily on intra-human distributional questions. Who will have a job? Who will get benefits and ‘access’ and ‘a say’? What will happen to social infrastructure? These both highly relate to each other, and to me are missing the point, which is whether humans get the benefits and stay in control and even survive, generally, at all.

Similarly, assurance’s goals (of safety, security, robustness and reliability) are important, and taking responsibility generally is a necessary if you want something to happen and go well. But the goal is the outcome, not the mechanism. I care about assurance and responsibility in this context only instrumentally in order to get the outputs. This could still be a useful distinction, but it could also be distracting.

And of course, I would say that opportunities is not really so hard a problem, if you have capabilities. Quite the opposite.

The hard problems I see missing here are some of the ones I most worry about, that AI might greatly exceed human capabilities, and that competitive and capitalistic and evolutionary style dynamics among AIs could lead places we do not want even if each individual move is ‘safe.’ If we are worried about whether humans even matter in #10, this list does not feel like it is appreciating the practical implications properly.

This could be thought of as addressed partially in problem eight, but I think mostly it is not. I see these problems as being less foundational, more symptoms than roots.

From what I can tell, however, this is still a highly thoughtful, thorough work that moves the conversation forward. I like the question in section 4, asking whether the problems are wicked, inherently defying a solution. They say they are not wicked problems if defined properly and ‘realistically,’ I am not so sure, and am worried that the parts that are wicked were excluded in part because they are indeed wicked.

Anton predicts a wave of mundane utility provision over the next 1-3 years, as people get used to business automation on enormous scales, and figure out how to have sufficient fault tolerance, as the problems that matter seem tractable. I agree.

Nvidia’s CEO Huang featured in an article that goes for a trifecta of Good Advice. He says ‘every country needs sovereign AI,’ that young people should not study computer science because it is their job to create computing technologies that no one has to program, and then projects a $320 billion boost to the Middle East’s economy from AI by 2030 as if that number meant anything. Study computer science, kids. Does every country need its own AI? I mean it seems reasonable to fine-tune one to better represent your culture, I guess. Beyond that I don’t see much point.

Davidad: Each Country Must Make Its Own Widgets, Insists CEO of Global Widget-Factory Monopoly.

Tyler Cowen offers ‘a periodic reminder of your pending competitive inadequacy.’

Many people think “I will do […], AI will not anytime soon do [….] as well as I will.”  That may or may not be true.

But keep in mind many of us are locked into a competition for attention.  AI can beat you without competing against you in your task directly.  What AI produces simply might draw away lots of attention from what you hope to be producing.  Maybe looking MidJourney images, or chatting with GPT, will be more fun than reading your next column or book.  Maybe talking with your deceased cousin will grip you more than the marginal new podcast, and so on.

This competition can occur even in the physical world.  There will be many new, AI-generated and AI-supported projects, and they will bid for real resources.  How about “AI figures out cost-effective desalination and so many deserts are settled and built out”?  That will draw away resources from competing deployments, and your project will have to bid against that.

I hope it’s good.

Quite so. Even if the AI cannot beat you at your exact job in particular, that does not mean it will win in a competition for attention, or a competition for dollars spent.

I often see such very good predictions, especially from Tyler Cowen in particular, and wonder how one can get this right and then fail to extrapolate to the logical conclusions. Even if AI never goes Full Superintelligence (perhaps because we somehow realize you never go full superintelligence), and a few spheres remain uniquely human when evaluated by humans, have you solved for the equilibrium when the AI is better than us at all the important economic activities and at executing all positions of power, and those who do not hand them over get outcompeted? Have you actually thought about what such worlds look like, while keeping in mind that we are considering the best case scenarios if civilization chooses this line of play?

I also wonder about the economics of the desalination example. If the AI figures out how to make the desert bloom cheaply, wouldn’t standard economics say that this creates an economic boom and also lowers the cost of housing as people get to move into the new areas, and shouldn’t it tighten the labor market? Yes, it draws investment away, but not in a way that anyone should feel threatened. If those dynamics shift to where this is bad news for the value of your labor, you were already obsolete, no?

A very strange position to take:

Gary Marcus: The ultimate rinse and repeat: “A survey from Boston Consulting Group showed that while nearly 90% of business executives said generative AI was a top priority for their companies this year, nearly two-thirds said it would take at least two years for the technology to move beyond hype.”

Repeat hype cycle in two years. Delivery entirely optional.

If generative AI is a top priority for your company this year, that does not sound like all hype. Nor is it, as highly useful products have already shipped. I know because I use them. The actual WSJ article centers on companies not sure they want to pay $30/month per user for Microsoft Copilot. I am not going to buy it because I prefer other methods and do not use Microsoft’s office products, but for those doing so in an office this seems like it is very obviously worthwhile.

Earlier this week I took a look at California’s proposed SR 1047. I believe that while there are still technical details one could improve or question, and this type of regulation should be coming from Congress rather than California (and we should worry that if California passes this bill that they might then attempt to block congressional action), this is an unusually good and well-crafted bill.

I had a chance to speak with Dean Bell, who took the perspective that this bill was a no-good, very-bad idea that he described as an ‘effort to strange AI’ and ‘effectively outlaw all new open source AI models,’ claims I strongly believe are inaccurate and to which I respond in the second half of my post. We had a very good conversation, much better than the usual, and mostly identified our core disagreements. I would describe them as whether or not it is wise to attempt to regulate such matters at all any time soon, and how to view how laws are interpreted and enforced in practice versus viewing them based on how they would be interpreted and enforced in an alternative regime where we had far superior rule of law.

Lennart Haim points out that in order to govern training compute, we will need a better understanding of exactly how to measure training compute. Current regulatory efforts do not sufficiently reflect attention to detail on this and other fronts. That seems right. The good news is that when one talks orders of magnitude, there is only so much room to weasel.

Fred Lewsey explains the case for regulation targeting compute, chips and datacenters. I do think this the right approach, but worry about the attempt to declare an ‘expert consensus.’

Patrick McKenzie points to Dave Kasten pointing out repeatedly that most of those in Washington D.C. do not understand how any of this existential risk stuff works. That a lot of the work remains on the level of ‘explain how the hell any of this works’ up to and including things like refuting the stochastic parrot hypothesis.

In particular, that that national security apparatus, with notably rare individuals as exceptions, continues to be unable to comprehend any threat that is not either another nation or a ‘non-state actor.’ To this line of thinking only foreign humans can be threats. The ideas we consider do not parse in that lexicon.

Dave Kasten: Have a conversation 3 times, tweet about it rule:

People who work on AI policy outside of the DC area cannot _imagine_ how different the conversation is in DC.

Berkeley: “AI will kill us all..”

Inside DC: “Here is our process for industry to comment on AI use cases”

(“Industry” is how federal government folks refer to all of capitalism. On my good days, I think it is charming anachronism; on my bad days, I think it demonstrates an unhealthy power relationship)

I am not trying to convince you of either Berkeley or K Street’s view on this topic — I am merely trying to convince you that if you have the Berkeley mindset, you should be talking to folks in DC 100x as much as you are

If you told the average US policymaker that “AI will kill us all,” their default assumption is that you mean, “because a Certain Nation in Asia powers up and we fight WW3”, not “we all get paperclipped”.

Alyssa Vance: I live in DC and know many DC AI people and many of them are concerned about x-risk. I have a somewhat biased sample, obviously, but I don’t think it’s nearly this black-and-white. (And many people in Berkeley worry about mundane issues too)

Dave Kasten: Oh, I think there is a cohort of DC AI people who are smart, and I’m very sorry if this tweet comes across as saying _no one_ is concerned. My point was more about the default conversations in Many Rooms in DC right now; it is very true that there are counterexamples.

So the work continues. Presumably if you are reading deep into my posts you are aware that the work continues, but it is good to offer periodic reminders.

Meanwhile, the former head of the NSA is in the Washington Post saying his biggest worry is us failing to reauthorize Section 702 of the Foreign Intelligence Surveillance Act, or renew it while requiring that surveillance on US persons be authorized by a court first. So the biggest threat to America is that we might enforce the Constitution.

The people, of course, continue to strongly support the policies in question. This is also something that Washington D.C. does not get, that supporting these interventions would help win elections, yes this is another new one from AIPI, this one from New York somehow?

Pause AI: – 71% want to slow down AI

– 48% oppose open sourcing powerful AI (21% support)

– 53% want more focus on catastrophic future risks (17% on current harms)

– 53% support compute caps (12% oppose)

– 70% support legal liability (12% oppose)

Acceleration is deeply unpopular, and people do not trust the labs to self-regulate. Note the complete lack of a partisan split here:

Open source? No thanks, says public, I think this wording is mostly fair? Notice that this time the partisan split is that more Republicans know not to release the kraken.

On the question of whether to focus on today’s risks or future risks, people are not buying the ‘focus on today’ arguments, despite that seeming like it should appeal to regular people. I think this framing is slightly unfair, but look at the splits:

People are actually far stronger on liability than I am. Notice that this is 87% support for a policy that in practice bans quite a lot of AI use cases.

And here it is, the straight up question of whether New York should stick its nose in a place that in a sane civilization it would not belong, this is a federal job, but good luck getting them to do anything, so…

Again, regulation with a 52-14 split among Republicans is something that you would expect to become law over time.

I very much do worry about what happens if you have a different license requirement for your AI in each of 50 states, unless they are restricted to only apply to the very top companies – Microsoft and Google can handle it, but yes that starts to be an unreasonable burden for others.

Here’s another one that shows how people are way more opposed to AI than I am:

That second question is the only one in the whole survey where AI had majority support for anything. People really, really do not like AI.

Remember the people who will ‘beat’ us if we ever take a single safety precaution? Remind me what they have been up to lately?

Dimitri Dadiomov: China’s take down of Jack Ma and the whole tech sector – right before an epic rally in tech stocks and the emergence of AI and the Magnificent Seven, which Alibaba could’ve perhaps been one of – was so incredibly shortsighted and ill-timed. Total self-own.

Paul Graham: The Great Leap Forward of tech.

Yes, if we were to halt all AI work forever then eventually someone would surpass us, and that someone might be China.

We still have to be consistent. If X would kill AI, and China has already done things far harsher than X, then is AI killed there or not?

It has been quiet. Too quiet.

Daniel Eth (4: 16am February 10): While I don’t think OpenAI should be open about everything (there are legitimate safety concerns at play here) I do think they should strive to be more open about important matters that don’t present risks to safety. Specifically, they should inform the public on WHERE IS ROON.

Mr. Gunn: He’s chained in the gooncave, with shadows of AGI cast on the wall in front of him.

Then good news, everyone!

Sam Altman (February 10, 9: 14pm local time): i don’t really know that much about this rumored compute thing but i do know @trevorycai is absolutely crushing the game and would love to answer your detailed questions on it. meanwhile @tszzl are hosting a party so i gtg.

Sam Altman: also roon is my alt.

Roon: I’m the main.

Well, actually, no. Which I flat out knew, but I was hoping to have more fun with it.

In any case, he’s so back. Oh no? Oh, yeah!

Roon: the anthropic commercials are the hubris that brought doom to San Francisco.

anton: calling the top right now, 5 second @AnthropicAI superbowl ad. it’s over, sell sell sell.

maybe superbowl ads are just ea-coded.

I do think the logic on this was sound for crypto. Once you advertise at the Super Bowl you are saturating the market for suckers, which is what was driving the crypto prices at the time. And indeed, it does not seem implausible that AI stock market valuations are perhaps ‘a bit ahead of themselves’ given the dramatic rises recently. I still expect such investments to turn out well, I think there is a huge persistent mispricing going on, but that mispricing can persist while a different upward pressure temporarily peaks. Who knows.

I do think Super Bowl ads are somewhat EA-coded, because EA is about doing the thing that is effective, and this counts. Anton is anti-EA and presumably sees this as a negative. I see this association as mostly a positive.

I believe that Super Bowl ads are likely underpriced even at $7 million for 30 seconds. They provide a cultural touchstone, a time when half of America will actually watch your damn advertisement seeking to be entertained and part of the conversation.

I do not think that you should buy 4 copies of the same generic spot as one e-commerce business did, that is a waste of money, but buying one well-considered spot seems great. For 2025, I would be unsurprised and approving if people bought at least one ad talking about AI safety and existential risk.

Evan Hubinger discusses the sleeper agent paper. Very good explanations, more worrisome than I expected or realized.

Sam Altman at the World Government Summit. Big fan of the UAE, this one.

Comes out in favor of mundane utility. Talks early about how in education they moved to ban ChatGPT then walked it back, clear implication of don’t make the mistake of touching my stuff. But I see that as a hopeful tale, people (correctly) came around quickly once they had enough information.

He says the reason more people haven’t used ChatGPT is because it’s still super early, it’s like the first primitive cell phones. He says timeline requires patience to reach the iPhone 16, but in a few years it will be better, in a decade it will be remarkable. I think even without improvement, the main barrier to further adaptation is purely time.

Why should we be excited for GPT-5? Because it will be smarter, so it will be better at everything across the board. Well, yes.

When asked what regulation he would pass for the UAE, he says he would create a regulatory sandbox for experimentation. I notice I am confused. Why do we need a sandbox when you can do whatever you want anyway? How will you ‘give people the future’ now?

He then says we will need a global regulatory system like the IAEA for when people might deploy superintelligence, so he would host a conference about that to show leadership, as the UAE is well-positioned for that for reasons I do not understand. I do agree such an agency is a good idea.

Asked about regulation, he says we’re in the discussion stage and that is okay, but in the next few years we will need an action plan with real global buy-in with world leaders coming together. He is pushed on what to actually do, he says that is not for OpenAI to say.

The host says at 15: 50 or so ‘I want to ask something that the fearmongers and opportunists ask’ and then asks what Altman is most worried and optimistic about. Sam Altman says what keeps him up at night is easy, it is:

Sam Altman: All of the sci-fi stuff. I think sci-fi writers are a really smart bunch. In the decades that people have written about this they have been unbelievably creative ways to imagine how this can go wrong and I think most of them are, like, comical, but there’s some things in there that are easy to imagine where things really go wrong. And I’m not really interested in Killer Robots walking down the street direction of things going wrong I’m much more interested in the very subtle societal misalignments where we just have these systems out in society and through no particular ill intention things just go horribly wrong.

But what wakes me up in the morning is I believe that things are just going to go tremendously right. We got to work hard to mitigate all of the downside cases…. but the upside is remarkable. We can raise the standard of living so incredibly much… Imagine if everyone on Earth has the resources of a corporation of hundreds of thousands of people.

Certainly this is much better than talking about unemployment or misinformation. I very much appreciate the idea that things can go horribly wrong without anyone’s ill intent, and that is indeed similar to my baseline scenario. That said, it is not ‘lights out for all of us’ and there is no mention of existential risks, other than bringing up the silly ‘Killer Robots walking down the street’ in order to dismiss it.

So this is at best a mixed response to the most important question. Altman is very good at letting everyone see what they want to see, and adjusting his answers for the right setting. He is clearly doing both here.

He then says that current young people are coming of age at the best time in human history, just think of the potential. This raises the question of how he thinks about the generation after, whether there will even be one, and what experience they will have if they do get to exist.

The second half of the video is an interview with Yann LeCun. I did not listen. I do not have to and I suppose in theory you could make me but you’d have to pay. I hear reports he says that LLMs are ‘not as smart as housecats.’

This long piece in Jacobin (!) by Garrison Lovely, entitled ‘Can Humanity Survive AI?’ is excellent throughout. It takes the questions involved seriously. It does a great job of exploring the arguments and rhetoric involved given its audience is fully non-technical.

Philosophy Compass publishes Artificial Intelligence: Arguments for Catastrophic Risk. Nothing new here, but it seems academics often have to read things in the right places or they think the words do not count.

Seth Lazar writes about Frontier AI Ethics. Didn’t feel new to me.

Tyler Austin Harper narrows in on the point that many people in Silicon Valley not only are willing to risk but actively welcome the possibility of human extinction.

Tyler Austin Harper: This entire essay is worth reading, but this is a crucial point that normies really don’t understand about Silicon Valley culture and desperately need to: many tech bros think creating AI is about ushering into being humanity’s successor species, and that this is a good thing.

Notice the quote from Sutton here: the focus is not on humanity, but *intelligence*. This idea — that human extinction doesn’t matter so long as some successor being continues to bear the light of intelligence — is a deeply misanthropic claim with a long history.

Early discussions of human extinction in the 19th century often talked about human extinction as a moral catastrophe because HUMANITY has a basic dignity and creative spirit that would be lost from the cosmos in the event of our demise. That changes in the early 20th century.

There’s a rhetorical shift that catches speed in the early 20th century where the moral catastrophe of extinction is no longer seen as the demise of HUMANITY, but rather the loss of INTELLIGENT LIFE from the cosmos. A subtle rhetorical pivot, but an absolutely momentous one.

Suddenly, our species is no longer conceived of as having value in and of itself. We’re valuable only insofar as we are the temporary evolutionary stewards of abstract intelligence. It is INTELLIGENCE, not humanity, that is valuable and that must be saved from extinction.

It’s this pivot, away from valuing the human species toward valuing abstract intelligence, that makes up the backbone of the ideologies swirling around AI in Silicon Valley. AI is viewed as the next rightful evolutionary steward of intelligence. It’s a scary, misanthropic view.

And I’ll add, reasonable people can disagree about the risks posed by AI. But regardless of the risk, the prevalence of the belief that helping intelligence flourish is more important than helping humanity flourish is concerning ipso facto, independent of whether AI is dangerous.

It is concerning if you care about the survival of humanity. Connor Leahy here highlights some good quotes.

One way to look at it, I suppose?

Amanda Askell: The view that advanced AI poses no extinction risk to humans but that climate change does pose an extinction risk to humans is interesting in that it rejects expert opinion in two pretty unrelated fields.

PauseAI and No AGI do another protest, this one at OpenAI’s offices, against AGI and military AI.

The event was organized in part as a response to OpenAI deleting language from its usage policy last month that prohibited using AI for military purposes. Days after the usage policy was altered, it was reported that OpenAI took on the Pentagon as a client

“The goal for No AGI is to spread awareness that we really shouldn’t be building AGI in the first place,” Sam Kirchener, head of No AGI, told VentureBeat. “Instead we should be looking at things like whole brain emulation that keeps human thought at the forefront of intelligence.”

I coined that last line (‘Earth is nothing without its people’) on Twitter. No AGI is the proof that there is always someone who takes a stronger position than you do. Pause AI wants to build AI once we find out how to make it safe, whereas No AGI is full Team Dune, and wants to build it never.

Reddit covered the protest, everyone said it was pointless without realizing that the coverage is the point. A lot of ‘we are going to do AI weapons no matter what, why are you objecting to building AI weapons you idiots.’ Yes, well.

Offered without further comment:

Also offered:

Eliezer Yudkowsky: The founder of e/acc speaks. Presented without direct comment.

Based Beff Jezos (e/acc): Doomers: “YoU cAnNoT dErIvE wHaT oUgHt fRoM iS” 😵‍💫

Reality: you *literallycan derive what *oughtto be (what is probable) from the out-of-equilibrium thermodynamical equations, and it simply depends on the free energy dissipated by the trajectory of the system over time.

[he then shows the following two images]

I want to be fully fair to Jezos, who kind of walked this back slightly afterwards, but also in the end mostly or entirely didn’t, so here is the rest of the thread for you to judge for yourself:

BBJ: While I am purposefully misconstruing the two definitions here, there is an argument to be made by this very principle that the post-selection effect on culture yields a convergence of the two.

How do you define what is “ought”? Based on a system of values. How do you determine your values? Based on cultural priors. How do those cultural priors get distilled from experience? Through a memetic adaptive process where there is a selective pressure on the space of cultures.

Ultimately, the value systems that survive will be the ones that are aligned towards growth of its ideological hosts, i.e. according to memetic fitness. Memetic fitness is a byproduct of thermodynamic dissipative adaptation, similar to genetic evolution.

As I interpret this, Jezos is saying that we ought to do that which maximizes a thermodynamic function, and we should ignore any other consequences.

Goody-2 is a dangerously misaligned model. Yes, you also never get an answer, but that’s not the real risk here. By giving the best possible reason to refuse to answer any query, it is excellent at allowing a malicious actor to figure out the worst possible use of any information or question. Will no one stop this before something goes wrong?

Timothy Lee says we should have extreme epistemic humility about future AGIs, because we have no idea how they will work or how the resulting physical world would look or operate, and says this is a ‘problem with the doomer worldview.’

And I would reply that yes, perhaps we should think there is a rather large existential risk involved in making unknown things that are smarter and more capable than us, that behave in unknown ways, in a very different radically uncertain type of world? That what we value, and also our physical selves, are not that likely to survive that, without getting into any further details?

I see far more epistemic humility among those worried about risk, than those who say that we definitely do not have risk, or we should proceed to this future as quickly as possible. And that seems rather obvious?

His other point is that there is no point in George Washington hiring nuclear safety researchers. Which is strictly true, but:

  1. George Washington’s inability to usefully hire specifically ‘nuclear safety researchers’ is strongly related to his inability to realize that this is a future need at all.

  2. George Washington was directly involved in a debate over how to deal with the safe and fair distribution of weapons, settled on the Declaration of Independence, then tried the Articles of Confederation followed by the Constitution including among others the second amendment, and we have been dealing with the consequences ever since, with mixed results. It was designed to handle a radically uncertain future. Mistakes were made. More research, one might say, was needed, or at least would have been useful, and they did the best they could.

  3. George Washington was also directly involved in debates over the scope of governmental powers versus freedoms, state capacity, the conditions that justify surveillance, what constitutes proper authority and so on. All very important.

  4. The AI landscape would look radically different today without George Washington, and in a way very closely related to many things he knew mattered and for the reasons they mattered. The ideas of the founding fathers matter.

  5. George Washington was deeply involved in diplomacy, international treaties and international relations, all of which are highly relevant and could be usefully advanced.

  6. If you can’t hire nuclear safety engineers, you don’t build a nuclear power plant.

I have some good news and some bad news.

This has more insight than most people who think about such questions. You think that if the AIs (or robots) start doing X that you can instead do Y. But there is no reason they cannot also do Y.

Of course, if you want to watch movies all day for your own enjoyment, the fact that a robot can watch them faster is irrelevant. Consumption is different. But consumption does not keep one in business, or sticking around.

Are the bots going to make things worse?

In light of it all, you can’t be too careful these days.

Daniel Eth: Lotta people I know are telling their parents to watch out for AI scams impersonating their voice, but if you really want to train your parents to be more careful, you should periodically red team them by calling them from a random number and doing a fake self-impersonating scam.

Okay, so the stranger whose phone I borrowed for this seemed to think it was kinda weird and disagreed that it constituted an “emergency”, but at least now I know my parents aren’t likely to fall for these types of scams.

Either that or they’re just not that bothered by the prospect of my kidnapping 🤔

AI #51: Altman’s Ambition Read More »