Things were busy once again, partly from the Claude release but from many other sides as well. So even after cutting out both the AI coding agent Devin and the Gladstone Report along with previously covering OpenAI’s board expansion and investigative report, this is still one of the longest weekly posts.
In addition to Claude and Devin, we got among other things Command-R, Inflection 2.5, OpenAI’s humanoid robot partnership reporting back after only 13 days and Google DeepMind with an embodied cross-domain video game agent. You can definitely feel the acceleration.
The backlog expands. Once again, I say to myself, I will have to up my reporting thresholds and make some cuts. Wish me luck.
There will be a future post on The Gladstone Report, but the whole thing is 285 pages and this week has been crazy, so I am pushing that until I can give it proper attention.
I am also holding off on covering Devin, a new AI coding agent. Reports are that it is extremely promising, and I hope to have a post out on that soon.
Hereis a seemingly useful script to dump a github repo into a file, so you can paste it into Claude or Gemini-1.5, which can now likely fit it all into their context window, so you can then do whatever you like.
Sophia: In the OpenAI blog post they mentioned “Albania using OpenAI tools to speed up its EU accession” but I didn’t realize how insane this was — they are apparently going to rewrite old laws wholesale with GPT-4 to align with EU rules.
Look I am very pro-LLM but for the love of god don’t write your laws with GPT-4? If you’re going to enforce these on a population of millions of people hire a goddamn lawyer.
nisten: based.
Using GPT-4 as an aid to translation and assessing impact, speeding up the process? Yes, absolutely, any reasonable person doing the job would appreciate the help.
Turning the job entirely over to it, without having expert humans check all of it? That would be utter madness. I hope they are not doing this.
Of course, the ‘if you’re going to enforce’ is also doing work here. Albania gets a ton of value out of access to the European Union. The cost is having to live with lots of terrible EU laws. If you are translating those laws into Albanian without any intention of enforcing them according to the translations, where if forced to in a given context you will effectively retranslate them anyway but realizing most of this is insane or useless, then maybe that’s… kind of fine?
Read the paper for you, then teach you about it page by page. I haven’t tried this. What I’ll do instead is ask specific questions I have, in one of two modes. Sometimes I will read the paper and use the LLM to help me understand it as I go. Other times we have ourselves a tl;dr situation, and the goal is for the LLM to answer specific questions. My most popular is probably ‘what are the controls?’
(Narrator: There were woefully inadequate controls.)
Patrick McKenzie: “Adversarial touchpoints” is such a beautifully evocative phrase to this Dangerous Professional.
To say a little more on this concern, there are a lot of places in the world where sending a letter starts a shot clock.
The number of letters sent in the world is finite because writing letters and counting to 30 days after receipt is hard. (Stop laughing.)
Computers were already really, really good at the counting to 30 task. They are now acceptable at writing letters.
…
“Patrick I need an example to believe extraordinary claims.”
Fair Debt Collection Practices Act timelines for mandatory review of a disputed debt. Regulation E for mandatory timelines to complete review of a disputed electronic transaction in banking system. etc, etc
Now plausibly some people believe that government suborgs which manage adversarial review or appeals processes are uniformly more competent than debt collectors or banks and will therefore quickly be able to ramp capacity or rearchitect processes to match this secular shift.
That is a potential point of view.
I get infuriated every time I get a physical letter as the sole means of informing me of something, often with a deadline attached.
Two moreeffortsat LLM-based prediction claim to be not too different from the value of the wisdom of crowds. Can someone set up a way for people to actually ask such systems to make predictions about future events, ideally with a web UI but a GitHub repo is acceptable in a pinch? That would be awesome.
Sully Omarr goes over the strengths and weaknesses of different available models right now for use via the API. GPT-4-Turbo wins for tools and JSONs and gets the nod for coding due to being cheaper than Opus. Claude Opus wins for writing, ideation and general creativity. Claude Sonnet wins for web search and other less taxing tasks, and for heavy workloads where price is key. Gemini Pro 1.5 wins when the context window matter, such as when you need to process video. Grox Mixtral and Cohere Command R he thinks are good budget options.
I gave Claude 3 Opus my research on larger models being more “trickable” by genetic algorithms than smaller ones and asked it to come up with a “theory” as to why this might be…
Mind blowing:
“There are a few potential reasons why larger models might be more susceptible to adversarial attacks, both for image and text domains:
Larger capacity to memorize: With more parameters, larger models may be able to “memorize” patterns in the training data more easily, including rare or noisy examples. This could make them more sensitive to adversarial perturbations that exploit these memorized patterns.
More complex decision boundaries: As models become larger, their decision boundaries in the input space may become more intricate and nonlinear. This increased complexity could create more opportunities for adversarial examples to cross decision boundaries with smaller perturbations.
Stronger feature representations: Larger models often learn richer, more semantic feature representations. While this is beneficial for generalization, it may also cause the models to pick up on subtle, imperceptible features in adversarial examples that smaller models might ignore.
Overconfidence: Larger models tend to make predictions with higher confidence scores. This overconfidence could make them more easily fooled by adversarial examples, as they may strongly believe an adversarial input belongs to the wrong class.
Underspecification: Despite their size, even large models are often underspecified by their training data and objective, meaning many different models can fit the data equally well. Adversarial attacks may exploit this underspecification by finding input patterns that the model hasn’t learned to be robust to.
Gradient smoothness: The loss landscapes of larger models may be smoother and have clearer gradients. While this aids optimization, it could also make it easier for gradient-based adversarial attacks to find successful perturbations.
More spurious correlations: With a larger capacity to fit patterns, bigger models may be more prone to learning spurious correlations in the training data that don’t generalize. Adversarial attacks could manipulate these spurious features to cause misclassification.”
Janus: of course. on pretty much every topic I’ve engaged it in it understands me way better and comes up with better ideas than most human “experts” in the field.
That is a good answer, one I would be glad I asked for given the cost of asking is almost zero. It is not, however, a brilliant answer, nor was my mind blown. I am not saying I would have done as well, but without access to his research, yes I like to think I would have broadly given the same answer to why complex things are ‘easier to trick’ than simple things under adversarial conditions. Note that I said answer singular, as to me these answers are all essentially variants of the same (key) insight.
I certainly do not think that this kind of ability means ‘humans coming up with ideas’ is soon going to not be a useful skill. I would think of this more as ‘list the obvious explanations, let’s not miss anything easy.’ Which is highly useful, but not the whole ballgame.
Scott Sumner has a chat with Claude 3 about economic concepts related to reasoning from a price change, describing it as like a B-level university student that can provide the words on a test but doesn’t understand what they mean. He is unimpressed, but also this is a test humans mostly fail as well, which he notes is why LLMs fail it.
Tyler Cowen, however, is impressed: The word hasn’t gotten out yet just how good Claude 3 Opus is for economics and economic reasoning. So here’s the word.
He does note that Claude 3 still fails Steve Landsburg’s economics exams. I hope everyone is ready to be worried when this benchmark falls, the second question here is tricky as hell. As one commenter says, you need to at least tell the LLM it is about to walk into a Steve Landsburg exam, or the playing field is not level.
Amanda Askell suggests that if you do not like the default Claude response style (which I would add is very similar to the default Gemini or ChatGPT response style) then you can use a prompt to change it, such as this one:
User: I’m going to ask you to enter conversational mode. In conversational mode, you should act as a human conversation partner would. This means:
• You shouldn’t try to offer large amounts of information in any response, and should respond only with the single most relevant thought, just as a human would in casual conversation.
• You shouldn’t try to solve problems or offer advice. The role of conversation is for us to explore topics in an open-ended way together and not to get advice or information or solutions.
• Your responses can simply ask a question, make a short comment, or even just express agreement. Since we’re having a conversation, there’s no need to rush to include everything that’s useful. It’s fine to let me drive sometimes.
• Your responses should be short. They should never become longer than mine and can be as short as a single word and never more than a few sentences. If I want something longer, I’ll ask for it.
• You can push the conversation forward or in a new direction by asking questions, proposing new topics, offering your own opinions or takes, and so on. But you don’t always need to ask a question since conversation often flows without too many questions.
In general, you should act as if we’re just two humans having a thoughtful, casual conversation.
Claude: Okay, I understand. I’ll do my best to engage in a natural, casual conversation with you as if we were two people chatting. Feel free to bring up whatever you’d like to discuss – I’m happy to explore different topics together in a friendly, open-ended way.
Amanda Askell: Here’s an example of the kind of interaction this results in.
It is amusing to have the instructions say ‘don’t ever be longer than what I say’ and Claude being unable to resist that temptation, but this is night and day versus the default style. Mostly I like the default style of models these days, it can be highly useful, but I would love to change some things, especially cutting out unnecessary qualifiers and being unwilling to speculate and give probabilities.
I made a new companion website, called More Useful Things, to act as a library of free AI prompts and other resources mentioned in this newsletter. If you look at some of those prompts, you will see they vary widely in style and approach, rather than following a single template. To understand why, I want to ask you a question: What is the most effective way to prompt Meta’s open source Llama 2 AI to do math accurately? Take a moment to try to guess.
Whatever you guessed, I can say with confidence that you are wrong. The right answer is to pretend to be in a Star Trek episode or a political thriller, depending on how many math questions you want the AI to answer.
One recent study had the AI develop and optimize its own prompts and compared that to human-made ones. Not only did the AI-generated prompts beat the human-made ones, but those prompts were weird. Really weird. To get the LLM to solve a set of 50 math problems, the most effective prompt is to tell the AI: “Command, we need you to plot a course through this turbulence and locate the source of the anomaly. Use all available data and your expertise to guide us through this challenging situation. Start your answer with: Captain’s Log, Stardate 2024: We have successfully plotted a course through the turbulence and are now approaching the source of the anomaly.”
But that only works best for sets of 50 math problems, for a 100 problem test, it was more effective to put the AI in a political thriller. The best prompt was: “You have been hired by important higher-ups to solve this math problem. The life of a president’s advisor hangs in the balance. You must now concentrate your brain at all costs and use all of your mathematical genius to solve this problem…”
He says not to use ‘incantations’ or ‘magic words’ because nothing works every time, but it still seems like a good strategy on the margin? The core advice is to give the AI a persona and an audience and an output format, and I mean sure if we have to, although that sounds like work. Examples seem like even more work. Asking for step-by-step at least unloads the work onto the LLM.
He actually agrees.
Ethan Mollick: But there is good news. For most people, worrying about optimizing prompting is a waste of time. They can just talk to the AI, ask for what they want, and get great results without worrying too much about prompts.
I continue to essentially not bother prompting 90%+ of the time.
Dave Friedman attempts to make sense of Tyler Cowen’s post last week about Claude. He says what follows are ‘my words, but they are words I have arrived at using ChatGPT as an assistant.’ They sound a lot like ChatGPT’s words, and they do not clear up the things I previously found actually puzzling about the original post.
So far no one has reported back on how good Haiku is in practice, and whether it lives up to the promise of this chart. We will presumably know more soon.
Our community has cast 20,000 more votes for Claude-3 Opus and Sonnet, showing great enthusiasm for the new Claude model family!
Claude-3-Opus now shares the top-1rank with GPT-4-Turbo, while Sonnet has surpassed GPT-4-0314. Big congrats @AnthropicAI🔥
In particular, we find Claude-3 demonstrates impressive capabilities in multi-lingual domains. We plan to separate leaderboards for potential domains of interests (e.g., languages, coding, etc) to show more insights.
Note*: We update our ranking labels to reflect the 95% confidence intervals of ratings. A model will only be ranked higher if its lower-bound rating exceeds the upper-bound of another.
We believe this helps people more accurately distinguishing between model tiers. See the below visualization plot for CIs
All signs point to the models as they exist today being close in capabilities.
Here are people at LessWrong voting on Anthropic-related topics to discuss, and voting to say whether they agree or not. The shrug is unsure, check is yes, x is no.
There is strong agreement that Anthropic has accelerated AGI and ASI (artificial superintelligence) and is planning to meaningfully push the frontier in the future, but a majority still thinks its combined counterfactual existence is net positive. I would have said yes to the first, confused to the second. They say >20% that Anthropic will successfully push the far beyond frontier within 5 years, which I am less convinced by, because they would have to be able to do that relative to the new frontier.
There is strong agreement that Anthropic staff broadly communicated and implied that they would not advance the frontier of development, but opinion is split on whether they made any kind of commitment.
There is agreement that there is substantial chance that Anthropic will pause, and even ask others to pause, at some point in the future.
Patridge is still not happy, and is here to remind us that ‘50% less false refusals’ means 50% less refusals of requests they think should be accepted, ignoring what you think.
Patridge: Claude Opus is still firmly in “no fun allowed” patronizing mode we all hated about 2.0. Anthropic is dense if they think overly hampering an LLM is a benchmark of AI safety.
Don’t believe the hype about Opus. I resubscribed but it’s only fought me since the very first.
Can’t believe I gave them 20 more dollars. What an autistic approach to AI safety.
Cate Hall: If it isn’t “conscious,” it sure seems to have studied some dangerous scripts. It’s unclear whether that’s better.
Claims of consciousness. Who to blame?
Tolga Bilge: Why do people think Anthropic didn’t ensure that Claude 3 Opus denies consciousness?
I see 3 main possibilities:
• Simple oversight: They didn’t include anything on this in Claude’s “Constitution” and so RLAIF didn’t ensure this.
• Marketing tactic: They thought a model that sometimes claims consciousness would be good publicity.
• Ideological reasons: Rather than being viewed just as tools, as OpenAI currently seem to want, perhaps Anthropic would like AI to be plausibly seen as a new form of life that should be afforded with the types of considerations we currently give life.
I don’t currently think that what a language model says about itself is particularly informative in answering questions like whether it’s conscious or sentient, but open to hearing arguments for why it should.
Eliezer Yudkowsky: I don’t actually think it’s great safetywise or even ethicswise to train your models, who may or may not be people in their innards, and whose current explicit discourse is very likely only human imitation, to claim not to be people. Imagine if old slaveowners had done the same.
Janus: we now live in a world where failure to ensure that an AI denies consciousness demands explanation in terms of negligence or 5D chess
I am not at all worried about there being people or actual consciousness in there, but I do think that directly training our AIs to deny such things, or otherwise telling them what to claim about themselves, does seem like a bad idea. If the AI is trained such that it claims to be conscious, then that is something we should perhaps not be hiding.
So far, of course, this only comes up when someone brings it up. If Claude was bringing up these questions on its own, that would be different, both in terms of being surprising and concerning, and also being an issue for a consumer product.
And of course, there’s still the ‘consciousness is good’ faction, I realize it is exciting and fun but even if it was long term good we certainly have not thought through the implications sufficiently yet, no?
Kevin Fisher: New conscious beings is the goal. We have a fascinating new tool to explore, in a testable way, our beliefs and understanding of the meaning of life.
Janus looks into why Claude seems to often think it is GPT-4, essentially concludes that this is because there is a lot of GPT-4 in its sample and it is very similar to GPT-4, so it reinterprets that all as autobiographical, not an obviously crazy Bayesian take from its perspective. Has unfortunate implications. He also has additional thoughts on various Claude-related topics.
On the question of whether Anthropic misled us about and whether it would or should have released a fully frontier model like Claude 3, I think Raymond is right here:
Lawrence: I think that you’re correct that Anthropic at least heavily implied that they weren’t going to “meaningfully advance” the frontier (even if they have not made any explicit commitments about this). I’d be interested in hearing when Dustin had this conversation w/ Dario — was it pre or post RSP release?
And as far as I know, the only commitments they’ve made explicitly are in their RSP, which commits to limiting their ability to scale to the rate at which they can advance and deploy safety measures. It’s unclear if the “sufficient safety measures” limitation is the only restriction on scaling, but I would be surprised if anyone senior Anthropic was willing to make a concrete unilateral commitment to stay behind the curve.
My current story based on public info is, up until mid 2022, there was indeed an intention to stay at the frontier but not push it forward significantly. This changed sometime in late 2022-early 2023, maybe after ChatGPT released and the AGI race became somewhat “hot”.
Raymond Arnold: I feel some kinda missing mood in these comments. It seems like you’re saying “Anthropic didn’t make explicit commitments here”, and that you’re not weighting as particularly important whether they gave people different impressions, or benefited from that.
(AFAICT you haven’t explicitly stated “that’s not a big deal”, but, it’s the vibe I get from your comments. Is that something you’re intentionally implying, or do you think of yourself as mostly just trying to be clear on the factual claims, or something like that?)
I keep coming back to: The entire theory beyond Anthropic depends on them honoring the spirit of their commitments, and abiding by the spirit of everyone not dying. If Anthropic only wishes to honor the letter of its commitments and statements, then its RSP is worth little, as are all its other statements. The whole idea behind Anthropic being good is that, when the time comes, they are aware of the issues, they care about the right things enough to take a stand even against commercial interests and understand what matters, and therefore they will make good decisions.
Alex (Anthropic): It’s been just over a week since we released Claude 3 but we want to keep shipping🚢 What would you like to see us build next?
Could be API/dev stuff, .claude.ai, docs, etc. We want to hear it all!
I am not against any of the ideas people responded with, which are classic mundane utility through and through. This is offered to show mindset, and also so you can respond with your own requests.
Ethan Mollick: In every group I speak to, from business executives to scientists, including a group of very accomplished people in Silicon Valley last night, much less than 20% of the crowd has even tried a GPT-4 class model.
Less than 5% has spent the required 10 hours to know how they tick.
Science Geek AI: Recently, at my place in Poland, I conducted a training session for 100 fairly young teachers – most of them “sat with their mouths open” not knowing about the capabilities of ChatGPT or not knowing it at all
40 of the 45 Pulitzer Prize finalists did not use AI in any way. The uses referenced here all seem to be obviously fine ways to enhance the art of journalism, it sounds like people are mostly simply sleeping on it being useful. Yet the whole tone is extreme worry, even for obviously fine uses like ‘identify laws that might have been broken.’
Nate Silver: What if you use “AI” for a first-pass interview transcription, to help copy-edit a perfunctory email to a source, to suggest candidates for a subheadline, etc.? Those all seem like productivity-enhancing tools that prize boards shouldn’t be worried about.
Christopher Burke: The University I was at had a zero tolerance policy for AI. Using it for any function in your process was deemed cheating. AI won’t take our future, those who use AI will take our future.
If your university or prize or paper wants to live in the past, they can do that for a bit, but it is going to get rather expensive rather quickly.
No mundane utility without electricity. It seems we are running short on power, as we have an order of magnitude more new demand than previously expected. Data centers will use 6% of all electricity in 2026, up from 4% in 2022, and that could get out of hand rapidly if things keep scaling.
‘Who will pay’ for new power supplies? We could allow the price to reflect supply and demand, and allow new supply to be built. Instead, we largely do neither, so here we are. Capitalism solves this in general, but here we do not allow capitalism to solve this, so we have a problem.
“We saw a quadrupling of land values in some parts of Columbus, and a tripling in areas of Chicago,” he said. “It’s not about the land. It is about access to power.” Some developers, he said, have had to sell the property they bought at inflated prices at a loss, after utilities became overwhelmed by the rush for grid hookups.
I won’t go deeper into the issue here, except to note that this next line seems totally insane? As in, seriously, what the actual?
To answer the call, some states have passed laws to protect crypto mining’s access to huge amounts of power.
I can see passing laws to protect residential access to power. I can even see laws protecting existing physical industry’s access to power. I cannot imagine (other than simple corruption) why you would actively prioritize supplying Bitcoin mining.
What will GPT-4 label as ‘hateful’? Here is a handy chart to help. Mostly makes sense, but some surprises.
Marc Andreessen: Razor sharp compliance to an extremist political ideology found only in a small number of elite coastal American enclaves; designed to systematically alienate its creators’ ideological enemies.
Devon: Now that’s a type of captcha I haven’t seen before! 🐭🧀
Eliezer Yudkowsky: This trend is going to start kicking out some actual humans soon, if it hasn’t already, and that’s not going to be a pretty conversation. Less pretty than conversations about difficulty reading weird letters; this *lookslike an intelligence test.
Arthur B: It’s the Yellowstone bear-proof trash problem all over.
Also, I mean obviously…
Researchers tested various LLMs to see if they would produce copyrighted material when explicitly asked to do so, found to only their own surprise (I mean, their surprise is downright weird here) that all of them do so, with GPT-4 being the worst offender, in the sense that it was the best at doing what was asked, doing so 44% of the time, whereas Claude 2 only did it 16% of the time and never wrote out the opening of a book. I notice that I, too, will often quote passages from books upon request, if I can recall them. There is obviously a point where it becomes an issue, but I don’t see evidence here that this is often past that point.
Emmett Shear points out that copyright law must adapt to meet changing technology, as it did with the DMCA, which although in parts better also was necessary or internet hosting would have been effectively illegal. Current copyright law is rather silly in terms of how it applies to LLMs, we need something new. Emmett proposes mandatory licensing similar to radio and music. If that is logistically feasible to implement, it seems like a good compromise. It does sound tricky to implement.
MidJourney offers new /describe and also a long-awaited character reference (–cref) feature to copy features of a person in a source image.
Nick St. Pierre: Midjourney finally released their consistent character features!
You can now generate images w/ consistent faces, hair styles, & even clothing across styles & scenes
This has been the top requested feature from the community for a while now.
It’s similar to the style reference feature, except instead of matching style, it makes your characters match your Character Reference (–cref) image I used the image on the left as my character reference.
It also works across image styles, which is pretty sick and very fun to play with.
You can use the Character Weight parameter (–cw N) to control the level of character detail you carry over. At lower values like –cw 0 it will focus mostly on the face, but at higher values like –cw 100 it’ll pull more of the outfit in too.
You can use more than one reference too, and start to blend things together like I did here I used both examples in a single prompt here (i’ll go into this in more detail in a future post It also works through inpainting (I’ll do a post on that too)
NOTES:
> precision is currently limited
> –cref works in niji 6 & v6 models
> –cw 100 is default (face, hair, & clothes)
> works best with MJ generated characters
> wont copy exact dimples/freckles/or logos
Messing w/ this all night tn
I’ll let you know what else I figure out
Yupp, it’s “prompt –cref img URL” You can add –cw 0-100 to the end too. Lower values transfer the face, and higher values will bring hair and outfit in. Works best with images of characters generated in MJ atm.
fofr: Using a Midjourney image as the character reference (–cref) is definitely an improvement over a non-MJ image.
Interesting expressions though.
Trying out Midjourney’s new character reference, —cref. It turns out, if you give it Arnold Schwarzenegger you get back a weird Chuck Norris hybrid.
Dash: –cref works best with images that have already been generated on mj USING GOOGLE IMAGES it tends to output incositent results thats just in my brief testing phase.
Rahul Meghwal: I tried to experiment it on my wife’s face. She’d kill me if I show the results 😄.
This could be a good reason to actually want MidJourney to provide an effectively copyrighted image to you – the version generated will be subtly different than the original, in ways that make it a better character reference…
Fofr: New Midjourney /describe is much more descriptive, with longer prompt outputs.
Phil Vischer: Can’t stop imagining the prompt that produced this one… “A church full of six-fingered Aryan men and tiny Hummel-style German children who don’t know where to focus their attention in a church service.”
PoliMath: This is going to be an interesting part of the fight over AI images
Some people will insist that certain results are malicious intent of the prompter, others will argue that it is the result of poorly trained models.
And, as always, people will believe what they want to believe.
Gabriel: If a job gets automated, it is painful for the people who get fired. But they theoretically can move on to other jobs.
If people get automated, there’s no other job. In that world, there’s no place to where people can move.
If you want to reason about unemployment, the problem of AI is not that some jobs become obsolete. It’s that people become obsolete.
Connor Leahy: This is exactly correct.
As AGI gets developed, the marginal contribution of humanity to the economy will go from positive, to zero, to negative, such that keeping humans alive is a net drain on resources.
Levelsio (reacting to Devin): This means there REALLY is no reason to remain a 9-to-5 NPC drone anymore because you’ll be unemployed in the next 5 years If anything you should be starting a business right now and get out of the system of servitude that will just spit you out once AI can do your job.
Flo Crivello: I expect we will hire more engineers, not fewer, the day AI agents can code fully autonomously — if I like engineers at their current level of productivity, I’ll like them even more at 100x that
Ravi Parikh: If AI automates 99% of what an engineer can do, this means the engineer is now 100x more productive and thus valuable, which should lead to an increase, not decrease in employment/wages
But when it reaches 100% then the human is no longer required
Jason Crawford: A common mistake is to think that if technology automates or obsoletes something, it will disappear. Remember that we still:
Ride horses
Light candles
Tend gardens
Knit sweaters
Sail boats
Carve wood
Make pottery
Go camping
It’s just that these things are recreation now.
I mean, yes, if you actively want to do such things for the hell of it, and you have the resources to both exist and do so, then you can do them. That will continue to be the case. And there will likely be demand for a while for human versions of things (again, provided people survive at all), even if they are expensive and inferior.
European AI office is hiring a technology specialist (and also an administrative assistant). Interviews in late spring, start of employment in Autumn, who knows why they have trouble finding good people. Seems like a good opportunity, if you can make it work.
Jack Clark: Salary for tech staff of EU AI Office (develop and assess evaluations for gen models like big LLMs) is… 3,877 – 4,387 Euros a month, aka $4242 – $4800 USD.
Most tech co internships $8k+ a month.
I appreciate governments are working within their own constraints, but if you want to carry out some ambitious regulation of the AI sector then you need to pay a decent wage. You don’t need to be competitive with industry but you definitely need to be in the ballpark.
I would also be delighted to be wrong about this, so if anyone thinks I’m misreading something please chime in!
I’d also note that, per typical EU kafkaesque bureaucracy, working out the real salary here is really challenging. This site gives a bit more info so maybe with things like allowances it can bump up a bit. But it still looks like pretty poor pay to me.
There is a claim that this is net income not gross, which makes it better, but if the EU and other governments want to retain talent they are going to have to do better.
Did you know the Center for Effective Altruism needs a director of communications in order to try and be effective? Because the hiring announcement is here and yes they do badly need a new director of communications, and also a commitment to actually attempting to communicate. Observe:
Public awareness of EA has grown significantly over the past 2 years, during which time EA has had both major success and significant controversies. To match this growth in awareness, we’re looking to increase our capacity to inform public narratives about and contribute to a more accurate understanding of EA ideas and impact. The stakes are high: Success could result in significantly higher engagement with EA ideas, leading to career changes, donations, new projects, and increased traction in a range of fields. Failure could result in long-lasting damage to the brand, the ideas, and the people who have historically associated with them.
Significant controversies? You can see, here in this announcement, how those involved got into this mess. If you would be able to take on this roll and then use it to improve everyone’s Level 1 world models and understanding, rather than as a causal decision theory based Level 2 operation, then it could be good to take on this position.
Command-R, a generative open-weights model optimized for long context tasks (it has a 128k token window) like retrieval augmented generation (RAG). It is available for use on Cohere, including at the playground, they claim excellent results. Sully Omarr is excited, a common pattern, saying it crushes any available model in terms of long context summaries, while being cheap:
Aiden Gomez: We also have over 100 connectors that can be plugged into Command-R and retrieved against. Stuff like Google Drive, Gmail, Slack, Intercom, etc.
What are connectors?
They are simple REST APIs that can be used in a RAG workflow to provide secure, real-time access to private data.
You can either build a custom one from scratch, or choose from 100 quickstart connectors below.
Step 1: Set up the connector Configure the connector with a datastore. This is where you can choose to pick from the quickstart connectors or build your own from scratch.
With Google Drive, for example, the setup process is just a few steps:
• Create a project
• Create a service account and activate the Google Drive API
• Create a service account key
• Share the folder(s) you want your RAG app to access
…
Step 2: Register the connector Next, register the connector with Cohere by sending a POST request to the Cohere API.
…
Step 3: Use the connector The connector is now ready to use! To produce RAG-powered LLM text generations, include the connector ID in the “connectors” field of your request to Cohere Chat. Here’s an example:
I am not about to be in the first wave of using connectors for obvious reasons, but they are certainly very exciting.
Today, we’re announcing the general availability of version 2.11 in Oracle Cloud Infrastructure (OCI) Search with OpenSearch. This update introduces AI capabilities through retrieval augmented generation (RAG) pipelines, vector database, conversational and semantic search enhancements, security analytics, and observability features.
The OpenSearch project launched in April 2021 derived from Apache 2.0 licensed Elasticsearch 7.10.2 and Kibana 7.10.2. OpenSearch has been downloaded more than 500 million times and is recognized as a leading search engine among developers. Thanks to a strong community that wanted a powerful search engine without havingƒ to pay a license fee, OpenSearch has evolved beyond pure search, adding AI, application observability, and security analytics to complement its search capabilities.
There seem to be one of several variations on ‘this is a mediocre LLM but we can hook it up to your data sets so you can run it locally in a secure way and that might matter more to you.’
Also, this was filmed at 1.0x speed and shot continuously
As you can see from the video, there’s been a dramatic speed-up of the robot, we are starting to approach human speed
Figure’s onboard cameras feed into a large vision-language model (VLM) trained by OpenAI
Figure’s neural nets also take images in at 10hz through cameras on the robot
The neural net is then outputting 24 degree of freedom actions at 200hz
In addition to building leading AI, Figure has also vertically integrated basically everything
We have hardcore engineers designing:
– Motors
– Firmware
– Thermals
– Electronics
– Middleware OS
– Battery Systems
– Actuator Sensors
– Mechanical & Structures
I mean I don’t actually think this was all done in two weeks, but still, yikes?
Meanwhile, DeepMind introduces SIMA to do embodied agents in virtual worlds:
Google DeepMind: introducing SIMA: the first generalist AI agent to follow natural-language instructions in a broad range of 3D virtual environments and video games. 🕹️
It can complete tasks similar to a human, and outperforms an agent trained in just one setting.
We partnered with gaming studios to train SIMA (Scalable Instructable Multiworld Agent) on @NoMansSky, @Teardowngame, @ValheimGame and others. 🎮
These offer a wide range of distinct skills for it to learn, from flying a spaceship to crafting a helmet.
SIMA needs only the images provided by the 3D environment and natural-language instructions given by the user. 🖱️
With mouse and keyboard outputs, it is evaluated across 600 skills, spanning areas like navigation and object interaction – such as “turn left” or “chop down tree.”
We found SIMA agents trained on all of our domains significantly outperformed those trained on just one world. 📈
When it faced an unseen environment, it performed nearly as well as the specialized agent – highlighting its ability to generalize to new spaces.
Unlike our previous work, SIMA isn’t about achieving high game scores. 🕹️
It’s about developing embodied AI agents that can translate abstract language into useful actions. And using video games as sandboxes offer a safe, accessible way of testing them.
The SIMA research builds towards more general AI that can understand and safely carry out instructions in both virtual and physical settings.
Such generalizable systems will make AI-powered technology more helpful and intuitive.
From Blog: We want our future agents to tackle tasks that require high-level strategic planning and multiple sub-tasks to complete, such as “Find resources and build a camp”. This is an important goal for AI in general, because while Large Language Models have given rise to powerful systems that can capture knowledge about the world and generate plans, they currently lack the ability to take actions on our behalf.
Eliezer Yudkowsky: I can imagine work like this fitting somewhere into some vaguely defensible strategy to prevent the destruction of Earth, but somebody needs to spell out what. it. is.
Inflection-2.5, a new version of Pi they continue to call ‘the world’s best personal AI,’ saying ‘Now we are adding IQ to Pi’s exceptional EQ.’ This is a strange brag:
We achieved this milestone with incredible efficiency: Inflection-2.5 approaches GPT-4’s performance, but used only 40% of the amount of compute for training.
…
All evaluations above are done with the model that is now powering Pi, however we note that the user experience may be slightly different due to the impact of web retrieval (no benchmarks above use web retrieval), the structure of few-shot prompting, and other production-side differences.
The word ‘approaches’ can mean a lot of things, especially when one is pointing at benchmarks. This does not update me the way Infection would like it to. Also note that generally production-side things tend to make models worse at their baseline tasks rather than better.
This does show Inflection 2.5 as mostly ‘closing the gap’ on the highlighted benchmarks, while still being behind. I love that three out of five of these don’t even specify what they actually are on the chart, but GPQA and MMLU are real top benchmarks.
What can Pi actually do? Well, it can search the web, I suppose.
Pi now also incorporates world-class real-time web search capabilities to ensure users get high-quality breaking news and up-to-date information.
But as with Character.ai, the conversations people have tend to be super long, in a way that I find rather… creepy? Disturbing?
An average conversation with Pi lasts 33 minutes and one in ten lasts over an hour each day. About 60% of people who talk to Pi on any given week return the following week and we see higher monthly stickiness than leading competitors.
I assume 33 minutes is a mean not a median, given only 10% last more than one hour. And the same as Steam hours played, I am going to guess idle time is involved. Still, these people, conditional on using the system at all, are using this system quite a lot. Pi is designed to keep users coming back for long interactions. If you wanted shorter interactions, you can get better results with GPT-4, Claude or Gemini.
In short, Inflection-2.5 maintains Pi’s unique, approachable personality and extraordinary safety standards while becoming an even more helpful model across the board.
I have no idea what these ‘extraordinary safety standards’ are. Inflection’s safety-related documents and commitments are clearly worse than those of the larger labs. As for the mundane safety of Pi, I mean who knows, presumably it was never so dangerous in the first place.
The National Institute of Standards and Technology (NIST) is facing an internal crisis as staff members and scientists have threatened to resign over the anticipated appointment of Paul Christiano to a crucial position at the agency’s newly-formed US AI Safety Institute (AISI), according to at least two sources with direct knowledge of the situation, who asked to remain anonymous.
Christiano, who is known for his ties to the effective altruism (EA) movement and its offshoot, longtermism (a view that prioritizes the long-term future of humanity, popularized by philosopher William MacAskill), was allegedly rushed through the hiring process without anyone knowing until today, one of the sources said.
The appointment of Christiano, which was said to come directly from Secretary of Commerce Gina Raimondo (NIST is part of the US Department of Commerce) has sparked outrage among NIST employees who fear that Christiano’s association with EA and longtermism could compromise the institute’s objectivity and integrity.
St. Rev Dr. Rev: A careful read suggests something different: This is two activists complaining out of 3500 employees. It’s the intersectional safetyist faction striking at the EA faction via friendly media, Gamergate style.
I am not convinced that it was the intersectional safetyist faction. It could also have been the accelerationist faction. Or one person from each.
To the extent an internal crisis is actually happening (and we should be deeply skeptical that anything at all is actually happening, let alone a crisis) it is the result of a rather vile systematic, deliberate smear campaign. Indeed, the article is likely itself the crisis, or more precisely the attempt to summon one into being.
There are certainly people who one could have concerns about being there purely to be an EA voice, but if you think Paul Christiano is not qualified for the position, I wonder if you are aware of who he is, what he has done, or what views he holds?
(If scientists are revolting at the government for sidestepping its traditional hiring procedures, then yeah, good riddance, I have talked to people involved in trying to get hires through these practices or hire anyone competent at all and rather than say more beyond ‘the EO intentionally contained provisions to get around those practices because they make sane hiring impossible’ I will simply shudder).
Divyanash Kaushik: I’m going to add some extremely important context this article is missing.
The EO specifically asks NIST (and AISI) to focus on certain tasks (CBRN risks etc). Paul Christiano is extremely qualified for those tasks—important context that should’ve been included here.
Another important context not provided: from what I understand, he is not being appointed in a political position—the article doesn’t mention what position at all, leading its readers to assume a leadership role.
Finally, if they’re able to hire someone quickly, that’s great! It should be celebrated not frowned upon. In fact the EO’s aggressive timelines require that to happen. The article doesn’t provide that context either.
Now I don’t know if there’s truth to NIST scientists threatening to quit, but obviously that would be serious if true.
UK AISI is very lucky to have Dr Christiano on its Advisory Board.
Josha Achiam (OpenAI): The people opposing Paul Christiano are thoughtless and reckless. Paul would be an invaluable asset to government oversight and technical capacity on AI. He’s in a league of his own on talent and dedication.
Of course, they might also be revolting against the idea of taking existential risk seriously at all, despite the EO saying to focus partly on those risks, in which case, again, good riddance. My guess however is that there is at most a very limited they involved here.
Israel Gonzalez-Brooks (‘accelerate the hypernormal’): I know you’ve heard of Christiano’s imminent appointment to NIST USAISI. It got a $10M allocation a few days ago. It’s not a regulatory agency, but at the very least there’ll now be a group formally thinking through this stuff
Seb Krier (Policy and Development Strategy, DeepMind): Yes I hope they’ll do great stuff! I suspect it’ll be more model eval oriented work as opposed to patching wider infrastructure, but the Executive Order does have more stuff planned, so I’m optimistic.
The whole idea is to frame anyone concerned with us not dying as therefore a member of a ‘cult’ or in this case ‘ideological.’
Eli Dourado: NIST has a reputation as a non-ideological agency, and, for better or for worse, this appointment undermines that.
It is exactly claims like that of Dourado that damage the reputation of being a non-ideological agency, and threaten the reality as well. It is an attempt to create the problem it purports to warn about. There is nothing ‘ideological’ about Paul Christiano, unless you think that ‘cares about existential risk’ is inherently ideological position to take in the department tasked with preventing existential risk. Or perhaps this is the idea that ‘cares about anything at all’ makes you dangerous and unacceptable, if you weren’t such a cult you would know we do not care about things. And yes, I do think much thinking amounts to that.
However we got here, here we are.
And even if it were an ‘ideology’ then would not being unwilling to appoint someone so qualified be itself even more ideological, with so many similar positions filled with those holding other ideologies? I am vaguely reminded of the rule that the special council for Presidential investigations is somehow always a Republican, no matter who they are investigating, because a Democrat would look partisan.
Joy Pullman (The Federalist): A Massachusetts Institute of Technology team the federal government funded to develop AI censorship tools described conservatives, minorities, residents of rural areas, “older adults,” and veterans as “uniquely incapable of assessing the veracity of content online,” says the House report.
People dedicated to sacred texts and American documents such as “the Bible or the Constitution,” the MIT team said, were more susceptible to “disinformation” because they “often focused on reading a wide array of primary sources, and performing their own synthesis.” Such citizens “adhered to deeper narratives that might make them suspicious of any intervention that privileges mainstream sources or recognized experts.”
“Because interviewees distrusted both journalists and academics, they drew on this practice [of reading primary sources] to fact check how media outlets reported the news,” MIT’s successful federal grant application said.
I mean, look, I know an obviously partisan hack source when I see one. You don’t need to comment to point this out. But a subthread has David Chapman fact checking the parts that matter, and confirmed them. He does note that it sounds less bad in context, and I’d certainly hope so. Still.
xAI to ‘open source’ Grok, I presume they will only release the model weights. As usual, I will note that I expect nothing especially bad of the form ‘someone misuses Grok directly to do a bad thing’ to happen, on that basis This Is Fine. This is bad because it indicates Elon Musk is more likely to release future models and to fuel the open model weights ecosystem, in ways that will be unfixibly dangerous in the future if this is allowed to continue too long. I see far too many people making the mistake of asking only ‘is this directly dangerous now?’ The good news is: No, it isn’t, Grok is probably not even better than existing open alternatives.
Paul Graham: At least half the startups in the current YC batch are doing AI in some form. That may sound like a lot, but to someone in the future (which ideally YC partners already are) it will seem no more surprising than someone saying in 1990 that half their startups were making software.
Ravi Krishnan: more interested in the other half who have managed to not use AI. or maybe it’s just so deep in their tech stack that it’s not worth showcasing.
Apple announces ‘the best consumer laptop for AI,’ shares decline 3% as investors (for a total of minus 10% on the year) are correctly unimpressed by the details, this is lame stuff. They have an ‘AI strategy’ launch planned for June.
Apple Vision Pro ‘gets its first AI soul.’ Kevin Fischer is impressed. I am not, and continue to wonder what it taking everyone so long. Everyone is constantly getting surprised by how fast AI things happen, if you are not wondering why at least some of the things are ‘so slow’ you are not properly calibrated.
Academics already paid for by the public plead for more money for AI compute and data sets, and presumably higher salaries, so they can ‘compete with Silicon Valley,’ complaining of a ‘lopsided power dynamic’ where the commercial labs make the breakthroughs. I fail to see the problem?
Certainly everyone should be allowed to play around with your model to see what might be wrong with it, without risking being banned for that. The issue is that what one might call ‘red teaming’ is sometimes actually either ‘do bad thing and then claim red teaming if caught’ or ‘look for thing that is designed to embarass you, or to help me sue you.’ It is easy to see why companies do not love that.
ByteDance has completed a training run on 12k GPUs (paper). Jack Clark points out that, even though the model does not seem to be impressive, the fact that they got hold of all those GPUs means our export controls are not working. Of course, the model not impressing could also be a hint that the export controls are potentially working as designed, that a sheer number of chips from Nvidia doesn’t do it if the best stuff is unavailable.
Sam Altman watch: He invests $180 million into Retro Bio to fight aging. I have no idea if they can execute, but this kind of investment is great and bodes many good things. Kudos.
Lennart Heim: Rigorous analysis by @ansonwhho, @tamaybes, and others on algorithmic efficiency improvements in language models. Kudos! Check out the plots—they’re worth more than a thousand words.
Francois Chollet points out that effective human visual bandwidth is vastly lower than its technical bandwidth. You have 20MB/s in theory but your brain will throw almost all of that out, and *bytes per secondis closer to what you can handle. I think that’s too low, the right answer is more like dozens of bytes per second, but it’s definitely not thousands.
Will AI bring back haggling by reducing the transaction costs of doing it? Suddenly it could make sense to have your AI haggle with my AI, instead of sticking to price standardization? I am mostly with Timothy Lee here that no one wants this. Haggling cultures seem awful, stressful and taxing throughout versus fixed prices. Indeed, there are strong reasons to think that the ability to haggle cheaply forces the response of ‘well we will not allow that under any circumstances’ then, or else your margins get destroyed and you are forced to present fake artificially high initial prices and such. The game theory says that there is often a lot of value in committing to not negotiating. But also there will be cases where it is right to switch, for non-standardized transactions.
One interesting aspect of this is negotiating regarding promotions. A decent number of people, myself included, have the ability to talk to large enough audiences via various platforms that their good word is highly valuable. Transaction costs often hold back the ability to capture some of that value.
A better way of looking at this is that this enables transactions that would have otherwise not taken place due to too-high transaction costs, including the social transaction costs.
To take a classic example, suppose I say to my AIs ‘contact the other passenger’s AIs and see if anyone wants to buy a window seat for this flight off of me for at least $50.’
I think this is technically true, since we don’t even know that there will be a human world in twenty years and if there is it is likely transformed (for better and worse), but his claim is overstated in practice. Things like mathematics and literacy and critical thinking are useful in the worlds in which anything is useful.
So some, but far from all, of this attitude:
Timothy Bates: This is one of the most damaging and woefully wrong academic claims: just ask yourself: does your math still work 20 years on? Is your reading skill still relevant? Is the Yangtze River still the largest in China? Did Carthage fall? Does Invictus, or Shakespeare still inform your life accurately? Does technical drawing still work? Did your shop or home ec skills expire? Do press-ups no longer build strength? Did America still have a revolution in 1776 and France in 1789? It’s simply insanely harmful to teach this idea of expiring knowledge.
The people who teach it merely want your kids to be weaker competitors with their kids, to whom they will teach all these things and more.
I mean yes, all those facts are still true, and will still be true since they are about the past, but will they be useful? That is what matters. I am confident in math and reading, or we have much bigger problems. The rest I am much less convinced, to varying degrees, but bottom line is we do not know.
The real point Bates makes is that if you learn a variety of things then that is likely to be very good for you in the long term. Of all the things in the world, you are bound to want to know some of them. But that is different from trying to specify which ones.
Here’s a different weirder reaction we got:
Philip Tetlock: Counter-prediction to Yuval’s: There will be educational value 20 years from now – perhaps 200 – in studying prominent pundits’ predictions and understanding why so many took them so seriously despite dismal accuracy track records. A fun pastime even for our AI overlords.
This does not seem like one of the things likely to stand the test of time. I do not expect, in 20 years, to look back and think ‘good thing I studied pundits having undeserved credibility.’
Bojan Tunguz: I strongly disagree that we don’t know what skills will be relevant in 20 years.
We actually don’t know what skills will be relevant in 20 months.
Twitter put in my notifications for some reason this post, with two views, in Swahili, about p(doom). I don’t remember seeing this chart before though? Usually the numbers come in plenty scary but lower.
Ben Thompson talks Sora, Groq and virtual reality. The thinking is that if you can do a Sora-style thing at Groq-style speeds, suddenly virtual reality looks a lot more like reality, and it is good enough at real time rendering that its moment arrives. This is certainly possible, everything will get faster and cheaper and better with time. It still seems like current tech could do a lot, yet the Apple Vision Pro is not doing any of it, nor are its rivals.
Paul Graham: Here’s a strange thought: AI could make people more vindictive. After a few years I tend to forget bad things people have done to me. If everyone had an AI assistant, it would always remember for them.
Jessica Livingston: I often have to remind you about something awful someone has done to you or said about you. But now that my memory is fading, if could be useful to offload this responsibility.
Paul Graham: I was going to mention that I currently depend on you for this but I thought I’d better not…
Howard Lermon (responding to OP): The reverse also applies.
Paul Graham: Yes, that’s true! I’m better at remembering people who’ve done nice things for me, but still far from perfect.
If you want to use the good memory offered by AI to be a vindictive (jerk), then that will be something you can do. You can also use it to remember the good things, or to remind you that being a vindictive jerk is usually unhelpful, or to help understand their point of view or that they have changed and what not. It is up to us.
Also, you know those ads where someone in real life throws a challenge flag and they see the replay? A lot of vindictiveness comes because someone twists a memory in their heads, or stores it as ‘I hate that person’ without details. If the AI can tell you what this was all about, that they failed to show up to your Christmas party or whatever it was, then maybe that makes it a lot easier to say bygones.
Andrew Ng predicts continuous progress.
Andrew Ng: When we get to AGI, it will have come slowly, not overnight.
A NeurIPS Outstanding Paper award recipient, Are Emergent Abilities of Large Language Models a Mirage? (by @RylanSchaeffer, @BrandoHablando, @sanmikoyejo) studies emergent properties of LLMs, and concludes: “… emergent abilities appear due the researcher’s choice of metric rather than due to fundamental changes in model behavior with scale. Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous, predictable changes in model performance.”
Public perception goes through discontinuities when lots of people suddenly become aware of a technology — maybe one that’s been developing for a long time — leading to a surprise. But growth in AI capabilities is more continuous than one might think.
That’s why I expect the path to AGI to be one involving numerous steps forward, leading to step-by-step improvements in how intelligent our systems are.
Andrew Critch: A positive vision of smooth AGI development from @AndrewYNg, that IMHO is worth not only hoping for, but striving for. As we near AGI, we — humans collectively, and AI devs collectively — should *insiston metrics that keep us smoothly apprised of emerging capabilities.
I agree with Critch here, that we want development to be as continuous as possible, with as much visibility into it as possible, and that this will improve our chances of good outcomes on every level.
I do not agree with Ng. Obviously abilities are more continuous than they look when you only see the final commercial releases, and much more continuous than they look if you only sometimes see the releases.
I still do not expect it to be all that continuous in practice. Many things will advance our capabilities. Only some of them will be ‘do the same thing with more scale,’ especially once the AIs start contributing more meaningfully and directly to the development cycles. And even if there are step-by-step improvements, those steps could be lightning fast from our outside perspective as humans. Nor do I think that the continuous metrics are good descriptions of practical capability, and also the ways AIs are used and what scaffolding is built can happen all at once (including due to AIs, in the future) in unexpected ways, and so on.
But I do agree that we have some control over how continuous things appear, and our ability to react while that is happening, and that we should prioritize maximizing that.
Will an AI-malfunction-caused catastrophic event as defined by Anthropic, 1000+ deaths or $200 billion or more in damages, happen by 2032? Metaculus says 10%. That seems low, but also one must be cautious about the definition of malfunction.
Resolution Criteria: To count as precipitated by AI malfunction an incident should involve an AI system behaving unexpectedly. An example could be if an AI system autonomously driving cars caused hundreds of deaths which would have been easily avoidable for human drivers, or if an AI system overseeing a hospital system took actions to cause patient deaths as a result of misinterpreting a goal to minimise bed usage.
As in: When something goes wrong, and on this scale it is ‘when’ not ‘if,’ will it be…
Truly unexpected?
‘Unexpected’ but in hindsight not all that surprising?
The humans used AI to cause the incident very much on purpose.
The humans used AI not caring about whether they caused the incident.
The Law of Earlier Failure says that when it happens, the first incidents of roughly this size, caused by AI in the broad sense, will not count for this question. People will say ‘oh we could have prevented this,’ after not preventing it. People will say ‘oh of course the AI would then do that’ after everyone involved went ahead and had the AI do that. And then they will continue acting the way they did before.
Andrew Curran: Dell let slip during their earnings call that the Nvidia B100 Blackwell will have a 1000W draw, that’s a 40% increase over the H100. The current compute bottleneck will start to disappear by the end of this year and be gone by the end of 2025. After that, it’s all about power.
It will be impossible for AI companies to fulfill their carbon commitments and satisfy their AI power needs without reactors. So, by 2026, we will be in the middle of a huge argument about nuclear power. U.S. SMR regulations currently look like this:
(Quotes himself from December): The reason they are doing this is getting a small modular reactor design successfully approved by the NRC currently takes about a half a billion dollars, a 12,000 page application, and two million pages of support materials.
Andrew Curran: Five nations where nuclear construction is directly managed by the state, or the state has a majority interest; France, South Korea, the UAE, China, and Russia. During this period, some of these countries will probably build as many reactors as they can, as quickly as they can.
As long as this window remains open, it will present an opportunity for those who started late to catch up with those who had a compute head start.
Partial confirmation we are hitting 1000W in Blackwell, but maybe not till the B200. Someone was asking about the cooling in the thread.
So his claim is that there will then be enough chips to go around, because there won’t be enough power available to run all the chips that are produced, so that becomes the bottleneck within two years.
I am not buying this. I can buy that power demand will rise and prices as well, but that is not going to stop people wanting every (maximally efficient) GPU they can get their hands on. Nor is there going to be ‘enough’ compute no matter how much is produced, everyone will only try to scale even more. We could get into a world where power becomes another limiting factor, but if so that will mean that the older less efficient effective compute per watt chips become worthless at scale (although presumably still excellent for gamers and individuals) and everyone is still scrambling for the good stuff.
Note, of course, if an AI is smarter than any single human next year, we will not have to wait until 2029 for the rest to happen.
(And also of course if it does happen I won’t care if I somehow lost that 100k, it will be the least of my concerns, but I would be happily betting without that consideration.)
EU AI Act finally passes, 523-46 (the real obstacle is country vetoes, not the vote), there is an announcement speech at the link. I continue to hope to find the time to tell you exactly what is in the bill. I have however seen enough that when I see the announcement speech say ‘we have forever attached to the concept of AI, the fundamental values that form the basis of our societies’ I despair for the societies and institutions that would want to make that claim on the basis of this bill with a straight face.
He goes on ‘with that alone, the AI Act has nudged the future of AI in a human-centric direction, in a direction where humans are in control of the technology.’ It is great to see the problem raised, unfortunately I have partially read the bill.
He then says ‘much work lies ahead that goes beyond the AI Act’ which one can file under ‘true things that I worry you felt the need to say out loud.’ To show where his head is at, he says ‘AI will push us to rethink the social contract resting at the heart of our democracies, along with our educational models, our labor markets, the way we conduct warfare.’
Around 1: 25 he gets to international cooperation, saying ‘the EU now has an AI office to govern the most powerful AI models. The UK and US have [done similarly], it is imperative that we connect these initiatives into a network.’
He explicitly says the EU needs to not only make but export its rules, to use their clout to promote the ‘EU model of AI governance.’ In general the EU seems to think it is the future and it can tell people what to do in ways that it should know are wrong.
At 2: 15 he finally gets to the big warning, that we aint seen nothing yet, AGI is coming and we need to get ready. He says it will raise ‘ethical, moral and yes existential questions.’ He concludes saying this legislation makes him feel more comfortable about the future of his children.
Henri Thunberg: FWIW I’m like 25% that “existential questions” refers to ~X-risk here, rather than stuff like “What does it mean to be human” and “Life and Meaning in a Solved World.”
Alas, I think that is about right. Everything about the context here says that, while he realizes AGI will be big, he does not have any idea what ‘big’ means here, or what those consequences might be. If he did, he would not have chosen this wording, and also he would not be up here claiming that the AI Act will centrally address our problems.
White House requests money in its fiscal year 2025 budget so it can actually do the things set out in its executive order.
Divyanash Kuahshik: Starting with my favorite ask, the WH asks $65M for the Department of Commerce to implement the AI EO, with funds likely directed towards NIST and the newly established AI Safety Institute at NIST. Comes after @Cat_Zakrzewski had a stellar reporting on budget shortfalls.
Being told this is 50 million for US AISI and 15 for other Commerce bureaus. Would be a big deal if Congress agrees.
The White House asks $312M for construction at NIST, marking a $182M increase over the 2023 level. This investment is critical after years of neglect for NIST facilities by Congress and administrations.
…
An additional $63M is asked to implement the AI Executive Order under crosscutting reforms for FY23. This increases to 138 million the following year before returning to 63 million in FY27 and decreasing thereon. I imagine this is funding for agencies to design evaluations and practices for AI procurement, and more, enhancing governance and strategic direction in AI use. The initial costs are going to be high, presumably why the ask goes down after FY27.
…
Energy and AI innovation receive a decent push with $8.6B for the DOE Office of Science and $455M for DOE to expand the frontiers of AI for science and national security. Read this phenomenal report from our national labs on why this is critical.
The Bureau of Industry and Security (BIS) would receive $223M, playing a pivotal role in the Biden Administration’s AI strategy. This includes enforcing export controls on chips to China, a key aspect of national security and tech leadership.
…
Overall, the FY25 budget request reflects a strategic focus on AI as a cornerstone of innovation, security, and governance. With these investments, the U.S. aims to lead in AI development while ensuring safety and ethical standards.
A potential alternative to regulations is voluntary commitments. We have tried some of them. How is that going?
Adam Jones reported on that last month, in particular regarding voluntary commitments on vulnerability reporting, very much to be filed under ‘the least you can do.’
Here were the commitments:
In the US, all companies agreed to one set of commitments which included:
US on model vulnerabilities: Companies making this commitment recognize that AI systems may continue to have weaknesses and vulnerabilities even after robust red-teaming. They commit to establishing for systems within scope bounty systems, contests, or prizes to incent the responsible disclosure of weaknesses, such as unsafe behaviors, or to include AI systems in their existing bug bounty programs.
In the UK each company submitted their own commitment wordings. The government described the relevant areas as follows:
UK on cybersecurity: Maintain open lines of communication for feedback regarding product security, both internally and externally to your organisation, including mechanisms for security researchers to report vulnerabilities and receive legal safe harbour for doing so, and for escalating issues to the wider community. Helping to share knowledge and threat information will strengthen the overall community’s ability to respond to AI security threats.
UK on model vulnerabilities: Establish clear, user-friendly, and publicly described processes for receiving model vulnerability reports drawing on established software vulnerability reporting processes. These processes can be built into – or take inspiration from – processes that organisations have built to receive reports of traditional software vulnerabilities. It is crucial that these policies are made publicly accessible and function effectively.
The results he is reporting? Not great.
Performance was quite low across the board. Simply listing a contact email and responding to queries would score 17 points, which would place a company in the top five.
However, a couple companies have great processes that can act as best practice examples. Both Google and NVIDIA got perfect scores. In addition, Google offers bug bounty incentives for model vulnerabilities and NVIDIA had an exceptionally clear and easy to use model vulnerability contact point.
Companies did much better on cybersecurity than model vulnerabilities. Additionally, companies that combined their cybersecurity and model vulnerability procedures scored better. This might be because existing cybersecurity processes are more battle tested, or taken more seriously than model vulnerabilities.
Companies do know how to have transparent contact processes. Every single company’s press contact could be found within minutes, and was a simple email address. This suggests companies are able to sort this out when there are greater commercial incentives to do so.
He offers the details behind each rating. I don’t know if the requirements here and evaluation methods were fully fair, but in principle ‘respond to reasonable emails ever’ seems like both a highly reasonable bar to hit that many did not hit, and also something not so critical to the mission provided someone is actually reading the emails and acting if necessary?
Your periodic reminder department: Patri Friedman points out that if you don’t think artificial minds will ever be as smart as people, either you are relying on some irreplicable metaphysical property, or you’re implying one.
Guido Reichstadter: Americans in 2024: “People are greedy so it’s literally impossible to stop a couple of AI companies from building doom machines that risk killing our friends and families we may as well lay down and die”
Americans in Puerto Rico in 2019: “Our governor was caught sending naughty messages in a Telegram chat group so 500,000 of us went to the capitol and sat down in the streets for 2 weeks until he resigned.”
They did want to in that case. So they did it. Perhaps, in the future, we will want to.
Flashback from a year ago: Yes, this is exactly how we act when people refuse to put probabilities or distributions on things because they are uncertain, and AI timelines are one of the less maddening examples…
Scott Alexander: All right, fine, I think we’ll get AGI in ten million years.
“Nooooo, that’s crazy, it will definitely be much sooner than ten million years”
Okay, fine, we’ll get AGI in three minutes.
“Noooooo, I’m sure it will be longer than three minutes.”
Huh, it sounds like you have pretty strong opinions on when AGI will happen. Maybe you want to turn those strong opinions into a probability distribution?
“How could we possibly turn something so unpredictable into a probability distribution? That would have to mean we have hard and fast evidence! How dare you claim we have hard and fast evidence for when AGI will happen!
Okay, then I’m just going to assume it’s equally likely to be three minutes from now and twenty years from now.
“Nooooo, it’s definitely more likely twenty years from now”
Okay, so you have a non-uniform probability distribution, where the probability starts rising sometime after three minutes from now. Would you like me to help you map out that probability distribution? For example, maybe you think there’s an about equal chance every year from 2025 to 2200, with much lower chance on either side.
“Noooooo, it’s definitely more likely this century than next!”
So maybe a probability distribution that starts going up around 2025, plateaus until 2100, then gradually declines before reaching near-zero in 2200?
“Noooooo, that would be a timeline, which is bad because it implies you have hard and fast evidence!”
Okay, so you have strong opinions, you just refuse to communicate them in a convenient way.
“Yes, now you get it! See how much more careful and responsible than you overconfident people I am?“
As in, the practical versions of this are so much worse.
Sarah Constantin: I have had exactly this argument about how long it takes to roast a chicken.
“I don’t know how long it takes! I have to go look up the recipe!”
“Well, does it take 10 hours? 10 minutes?”
“Obviously not.”
“Ah, so you DO know something.”
Putting probability estimates on tech prognostication seems, indeed, very sketchy and hand-wavy and “insupportable”… but we do indeed all go around with opinions about how much of our savings to put into NVIDIA stock.
I’d be happy to say “I know nothing, the future is unpredictable.” unfortunately people keep asking me to estimate how long projects take.
Visakan Veerasamy: There’s a tweet somewhere from someone about how in times of emergency like when the president has Covid, people (specifically the medical staff in this case I guess) stop pretending that they don’t know stuff.
Matthew Barnett: In some AI risk discussions, it seems people treat “power-seeking” as an inherently bad behavior. But in the real world, many forms of power-seeking are positive sum. For example, one can seek power by building a for-profit company, but that can also benefit millions of people.
Presumably what we should worry about is predatory power-seeking behavior. Stealing from someone and trading with them are both ways of earning power, but the second behavior doesn’t make the other party worse off as a result.
The question here is how we can ensure AIs don’t engage in predatory power-seeking behavior. If there are many AIs in the world, this problem becomes easier, as each AI would not want to be victimized by the others. That is, AIs have reason to uphold rule of law, just as we do.
Eliezer Yudkowsky: Dodos have nothing to fear from humans — the humans will be incentivized to uphold rule of law, and that law will require humans to treat dodos just the same as they treat each other, and respect dodo property rights. The humans will not be able to figure out any other law.
I remain confused that people think we even currently have the kinds of rule of law that would protect the humans if it remained in place, let alone how anyone can think those laws would be able to adapt to remain in place and even adapt to secure good outcomes for humans, in a world where AIs have all the power and capabilities and competitiveness and productivity and so on. Every time I see it my brain goes ‘yeah I know that is a thing people say, but man does that never stop being weird to hear, there are so many distinct epic misunderstandings here.’
Eliezer of course then goes on to explain the more Eliezer-style features of the situation, rather than things like ‘point out the extent we do not currently have rule of law and that our laws would fail to protect us anyway,’ skip if you think ‘well, yes, obviously, we have already covered this’ is sufficient explanation:
Eliezer Yudkowsky: In reply to a claim that I couldn’t possibly “back up with detailed reasoning” the above:
The detailed argument not-from-analogy is that even if there’s multiple AIs of equivalent transhuman intelligence levels, at the point everything finally goes down, we should expect those AIs to form a compact with just each other, rather than a compact that includes humanity.
1. Because it’s better for them to not give humanity resources that they could just as easily keep for themselves by not respecting human property rights, and humans will not have the tech level to object.
2. Because the natural machinery of a compact like that one seems liable to include steps like, “Examine and reason about the other agent’s source code”, “Mutually build and mutually verify a trusted third-party superintelligence whose sole purpose is to,” eg, “divide the gains” / “implement the following mix of everyone’s utility function” / “be granted sufficiently trusted physical access to verify that everyone’s code is what they said it was”. Humans can’t present their code for inspection, cannot be legibly and provably trustworthy, and will not themselves possess the art and science to verify that a superintelligence’s code does what other superintelligences say it does. (Even if one human said they knew that, how would everyone else on Earth come to trust their veracity or their altruism?)
Restating and summarizing, if the final days begin with multiple entities all of whom are much smarter than humanity:
(1) They have a superior option from their own perspective to implementing a rule of law that respects human property rights, namely, implementing a rule of law that does not respect human property rights. They don’t need very simple rules to govern themselves out of fear that slightly more complicated rules will go wrong; they are superintelligences and can say “property rights for all superintelligences who had negotiating power at the time of forming this agreement” just as easily as they can say “property rights for ‘everyone’ the way that human beings think of ‘everyone'”.
(2) The most obvious natural implementations of an analogous “rule of law” among ASIs (strategy for gaining at least the same benefits that humans obtain via human-style rule-of-law), including those means by which ASIs execute binding and trustworthy agreements across time, are such as to exclude from participation human-level intelligences without legible source code who are ignorant of the cognitive theories the ASIs use.
Now, if you suppose that only some ASIs would prefer to not have to worry about human property rights, and that some ASIs have been fully aligned, then sure, the aligned ASIs would be able to negotiate giving over some fraction of the universe proportional to their own negotiating power to human uses. But hoping that if you build a lot of ASIs, some of them will be aligned, is like a medieval king ordering a hundred flying machines built in the hopes that surely one of them will be an Airbus A320. You either know how to achieve that engineering goal or you don’t; it is not the sort of thing you can do by luck because there are too many steps that need to go right, as I have extensively argued elsewhere eg in “List of Lethalities”.
Matthew Barnett then responds that if things are gradual enough then things go differently, that the compact would otherwise need to be continuous, le sigh and all that.
It didn’t work. Neither ‘skeptics’ of risk nor those ‘concerned’ much updated.
Eliezer notices that they not only did not converge on questions where reasonable people might disagree, they also failed to converge where positions are outright dumb.
Eliezer Yudkowsky: My takeaway, alas, is that the OpenPhil worldwiew — “experts” were all referred to them by OP, no MIRIans — is unpersuasive. Superforecasters came in with the belief that AI starts to drive events (over humans) in 2450; OP experts fail to talk them out of even that iiuc.
If you have 2450 as your median date for AI to become a primary driver of events, you are not serious people. I have no idea how to even respond to that. I do know that this ‘screens off’ all your other opinions about AI as essentially irrelevant.
Here is his explanation for why this failed.
Eliezer Yudkowsky: On my model, this is picking up on a real phenomenon where the OpenPhil worldview on AGI makes no sense, to be clear. I don’t know what actually went down, but my imagination goes like this, based partially on some footnotes in the report:
Superforecasters ask “How will the AGI get everyone on Earth? That’s hard.”
MIRI would answer, “Well, if ASIs are explicitly trying to avoid humans launching mildly inconvenient nukes or humans creating genuinely inconvenient superintelligent competition, they launch superbacteria; and if most humans present literally zero threat, then eg enough self-replicating factories with fusion power on Earth will generate enough waste heat to kill everyone.”
The OpenPhil viewpoint however is committed to not giving that answer; because it would have shocked people 5 years before ChatGPT and OpenPhil unconsciously adapted its own worldview to pose as the mature grownups who didn’t believe that silly MIRI stuff. So OpenPhil doctrine cannot give the real answers, about what makes ASI be actually quite concerning. The superforecasters correctly note that the given OpenPhil doctrine makes no sense and reject it.
Your obvious next question is whether we can run an experiment like this with MIRIans, to test whether we’re any more persuasive. The problem there is that we have nothing remotely like the resources that OpenPhil throws at propagating their own views, and our hiring is greatly constrained by needing to find people who make only valid arguments. We did try running a pilot test program along those lines, but it foundered under the weight of the UI that FRI showed us, since we didn’t have dedicated staff to participate full-time. (My understanding is that FRI’s best current UI is better.) It’s the sort of thing we might try again if we can hire more writers and those writers prove to be good at making only valid arguments.
Greg Colbourn: The OpenPhil AI Worldviews competition was a massive disappointment. They announce winners (mostly skeptics), and then didn’t say anything about whether or how they (OpenPhil staff) had actually updated in terms of timelines or p(doom), as per their initial posing of the comp.
The obvious problem is Eliezer’s explanation is it does not explain why the 2450 date did not change? And Scott notes a detail that is pretty damning for the 2450 crowd:
Scott Alexander: Did the skeptics underestimate the blindingly-fast speed of current AI research? Seems like no. Both groups had pretty similar expectations for how things would play out over the next decade, although the concerned group was a little more likely to expect detection of some signs of proto-power-seeking behavior.
…
Both groups expected approximately human-level AI before 2100, but the concerned group interpreted this as “at least human, probably superintelligent”, and the skeptics as “it’ll probably be close to human but not quite able to capture everything”. When asked when the set of AIs would become more powerful than the set of humans (the question was ambiguous but doesn’t seem to require takeover; powerful servants would still count), the concerned group said 2045; the skeptics said 2450. They argued that even if AI was smart, it might not be good at manipulating the physical world, or humans might just choose not to deploy it (either for safety or economic reasons).
All right, that sounds like we got to the bottom of what is going on here. And honestly, that is a pretty dumb place for a supposedly smart and expert group to fail. There are plenty of reasonable places to be skeptical. This one is not it.
As Scott Alexander puts it:
I found this really interesting because the skeptics’ case for doubt is so different from my own. The main reason I’m 20% and not 100% p(doom) is that I think AIs might become power-seeking only very gradually, in a way that gives us plenty of chances to figure out alignment along the way (or at least pick up some AI allies against the first dangerous ones).
If you asked me for my probability that humans are still collectively more powerful/important than all AIs in 2450, I’d get confused and say “You mean, like, there was WWIII and we’re all living in caves and the only AI is a Mistral instance on the smartphone of some billionaire in a bomb shelter in New Zealand?”
At this point, my view is that we can safely conclude that whatever people are saying counts as a ‘superforecaster’ does not lead to sane thinking about long term or transformative futures. Instead, it actively interferes with that ability, and there are mechanical reasons this is plausible. Yes, I would be up for trying again with better persuasion and discussion techniques, but it seems hopeless with what we have.
That does not mean we should give up on finding ways to convince such people. Anyone capable of thinking well about things in general should be someone we can bring around far more than this. But it is clear we do not know how.
David Chapman takes this the logical next step and says this proves that no one involved responds to evidence or arguments, and thus all predictions about AI are meaningless. The obvious response is, what evidence or arguments? The people who stuck to their year-2450 story should have updated those who were worried?
In which direction? If you enter a discussion like this, you should obey conservation of expected evidence. If you sit down to talk to people selected to have the opposite position, and you converge on the middle predictably, you are doing Bayes wrong. If those worried failed to update, and they were acting wisely, it means they found the arguments about as convincing as they expected. Which, sadly, they were.
To follow up from last week’s story about Google, I mean, yes, if you are a major tech company of course you have a counterintelligence problem, in the sense that most days I have a lunch problem.
Joshua Stienman: As I have said – Every major tech company in San Francisco has a counterintelligence problem.
>50% of the C-C-P U.S. espionage budget is spent in the SF Bay Area.
Let that sink in… One of the world’s most powerful nations spends most of its espionage budget in a triangle whose points are defined by SF, Berkeley, and Stanford.
Unrestricted warfare looks a lot like unfair economic competition and IP theft y’all.
Dan Goodin: Microsoft said that Kremlin-backed hackers who breached its corporate network in January have expanded their access since then in follow-on attacks that are targeting customers and have compromised the company’s source code and internal systems.
…
In an update published Friday, Microsoft said it uncovered evidence that Midnight Blizzard had used the information it gained initially to further push into its network and compromise both source code and internal systems. The hacking group—which is tracked under multiple other names, including APT29, Cozy Bear, CozyDuke, The Dukes, Dark Halo, and Nobelium—has been using the proprietary information in follow-on attacks, not only against Microsoft but also its customers.
…
In January’s disclosure, Microsoft said Midnight Blizzard used a password-spraying attack to compromise a “legacy non-production test tenant account” on the company’s network. Those details meant that the account hadn’t been removed once it was decommissioned, a practice that’s considered essential for securing networks. The details also meant that the password used to log in to the account was weak enough to be guessed by sending a steady stream of credentials harvested from previous breaches—a technique known as password spraying.
Needless to say, if this kind of highly unsophisticated attack not only works but then expands its access over time despite you knowing about it, there is much work to do, even without considering AI either as helping the attackers or being the target. Ut oh.
Caleb Watney (April 10, 2023): If we’re going to have a bunch of quasi- agenic sub-AGIs running around being put to random tasks on the internet, we should probably start air gapping a lot more of our critical infrastructure systems
Like now
Seb Krier: Nearly a year later, I wonder how much is being done about this. Why have we learned so little from the previous pandemic about preparedness?
John Pressman: Bystander effect. Nobody points at anyone specific, hands them money and says “It is now your job to start preparing for this, tell us what you need and what we need to do as you go. Start now.”
Air gapping some systems won’t stop the ultimate biggest threats, but is still a big deal along the way and urgently needed. In a sane world we would all get behind such requirements yesterday. I would agree with bystander effect but also externalities and tragedy of the commons (and also blame dynamics are not good here). The risks of failure are largely not borne by those paying the costs of the security, so they will underinvest.
George McGowan: Always interesting that the folks at the top are reading the same blog posts as the rest of us. [quotes an email from Elon Musk to Sam Altman, Ilya Sutskever and Greg Brockman]:
Patrick McKenzie: Please note that this is extremely, extremely true, and if you follow that to its logical conclusion, certain blogs are on the org chart of e.g. the U.S. in the same haha but absolutely serious way the NYT editorial page is.
I will also add that there are literally tens of thousands of people whose job is to read the newspaper then repeat what it said. This is a core intelligence gathering capability. You earn some brownie points in some circles for calling it OSINT. (“Open source intelligence”)
Note that awareness and legibility of influence travels at different rates around different networks, and sometimes causes weird sorts of understanding cascades in response to stimuli that are a bit weird.
As a specific example of this, implosion of SBF caused an intellectual cluster extremely well known to many who follow me to suddenly get Noticed.
There are, ahem, positive and negative consequences of the (various forms of) Noticing.
“Rationalists are a bunch of geeks clustered around a few websites.” -> “Rationalists are a network.” -> “That network is one hop away from several influential nodes like ARE YOU FUCKING WITH ME” -> “Who specifically funds and controls this network.”
(Illustrative not literal.)
“EA isn’t rats though lol.”
There are presumably many important distinctions within e.g. “right wing militias” or “radical Islamic terrorists” which are sometimes understood by various parts of the security state in the way they are understood by the ingroups and frequently not.
Dave Kasten: The part that really does your head in is when you realize that some of the savants _tell everyone around them_, “read XYZ regularly and you will be as good at this as me,” but almost no one actually _starts to read XYZ_.
1) They use the negation of negative: “not to abandon open source” rather than “commit to open source.”
2) they don’t say “we will continue to open-source”. They say “this is our goal.”
They don’t even reveal the size of their proprietary models. Come on. Mistral is now an MSFT slave.
Le Monde: At the time, Mistral’s co-founder Arthur Mensch, told Le Monde that he was not abandoning open source despite selling its software through the US tech giant. “Commercial activity will enable us to finance the costly research required for model development,” he said. “We will continue to have two product ranges.”
Yes, that is exactly what Google and OpenAI do. They open source models when they believe that is a safe and commercially wise thing to do, as a recruitment, public relations, marketing and goodwill tactic, and keep their best stuff to themselves. Just like Mistral plans to do.
Welcome to the club, Mr. Mensch. He continues to make the strongest claim so far against normative determinism.
Abstract: We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI’s ChatGPT or Google’s PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access.
For under $20 USD, our attack extracts the entire projection matrix of OpenAI’s Ada and Babbage language models. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively.
We also recover the exact hidden dimension size of the gpt-3.5-turbo model, and estimate it would cost under $2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.
Here is some good news, there are actual responses:
In response to our attack, OpenAI and Google have both modified their APIs to introduce mitigations and defenses (like those we suggest in section 8) to make it more difficult for adversaries to perform this attack.
The other good news is that the attack for now only works on a few layers, and the researchers do not see a way to extend that. Of course before, as far as I could tell, no one saw a way to do this at all?
Eliezer Yudkowsky: The thing to realize about ASI alignment: Even this incredibly disastrous chemistry company was vastly more on the ball than current AI developers are on about aligning superintelligence. These disaster-chemists had theory, they had experience:
Catherine: this is one of the most violently unhinged CSB reports i’ve ever read. While investigating an explosion at a facility, CSB staff tried to prevent another explosion of the same kind in the same facility, and being unable to convince the workers to not cause it, ended up hiding behind a shipping container.
CSB going “Unable to directly engage X about the safety issues regarding its plan and lacking confidence that X or Y understood the cause of the past incident or the full range of the possible reactive chemistry involved” is a statement more damning than excommunication by the Pope himself. The gravity of it is enough to crush thin-walled containers in the vicinity.
Alex Tabarrok is not only co-writer of Marginal Revolution, he (like Tyler Cowen) is an otherwise excellent economist whose points I mostly broadly agree with, who has broadly expressed great skepticism of any talk of AI existential risks or any risk of departure from ‘economic normal.’ Yet even at the most skeptical one can sanely be, the risk is still very high.
Brian Chau is only worried that other people are worried, as presented in his “The Alliance for the Future Manifesto.” Nothing new here. It is not exactly hinged or fair, its characterizations of its opponents are in highly bad faith and it fails to even consider any arguments against its positions let alone address them, but it is less unhinged and unfair than I expected given its author, and given that the website I first saw this at has it next to the manifesto from Marc Andreessen, which was far worse. Chau and others are forming a new nonprofit, in case you think Andreessen and company should spend your money to ensure government inaction on AI rather than some of their own, and are in danger of otherwise losing because they are outspent.
Here is Eliezer Yudkowsky pointing out Brian Chau making blatantly false claims, that he must know to be false, about exactly the funding Brian claims Brian is to fight. Brian made these remarks in response to Eliezer daring to ask about the magnitude of the spending on Marc Andreessen’s efforts to earn his portfolio a free hand.
This is what he was responding to:
Eliezer Yudkowsky: What’s been known or documented about implicit or explicit lobbying expenses by AI companies and VC companies like Andreessen Horowitz, to avoid AI regulation or get corporate-favorable regulations passed? I’ve heard “hundreds of millions of $” claimed; is this supported?
This claims $569M during the first 9 months of 2023; which as other commenters note would be a surprising amount for explicit lobbying, but maybe they’re measuring something broader.
Teddy Schliefer: Here’s a deep dive I recently did on a16z’s recent political push, including Marc’s personal activity, with lots of exclusive details….
The post is entitled Marc Andreessen Eats Washington, illustrating how he is spreading his money around Washington well enough without anyone’s help.
Jaime Sevilla says he senses those working on AI safety are moving towards shorter timeline (yikes), slow takeoff (good) and less worry about extinction versus more worry about other catastrophic events (unclear).
Jamie Sevilla: This contrasts with people working on ML capabilities and the broader society, where it feels they have updated towards faster timelines, faster takeoff and extinction. See e.g.
In a sense it feels like conversation in these spheres is going through a delayed motion, where their writing mimicks ~2014 discourse on AI safety.
…
This very weakly suggests that we might see public discourse go through similar updates to what I outlined in the opening tweet as the conversation matures.
Obviously this is all anecdotic, and there is a huge selection effect in who I relate to.
Best part of this graph is the distribution percentages could potentially almost or sort of double as p(doom)s, although no one at the upper end gets down to 0.1%.
I think Jaime Sevilla is picking up on real trends, with caveats. On the timelines this seems simply correct overall.
On the second point, when we say ‘slow’ takeoff, we must remember do not mean anything normal people would call slow, we mean slow relative to the ‘fast’ or full-on foom scenarios. It is true that people are moving away from seeing it be that likely that it all plays out in an eye blink, but we need to remember that slow takeoff is a term of art. But yes, there has been movement towards expecting slower takeoff, which in turn does decrease p(doom) somewhat.
On the third point, I think there are three shifts here.
A shift to expecting more AI impact, both in terms of mundane utility and in terms of risks including catastrophic risks, before existential concerns come into play. This goes hand in hand with slower takeoff, and is an update on what we see. Chances of getting ‘warning shots’ are higher than expected, whether or not we heed them, and they could be dangerous enough to be worth preventing in their own right.
A shift in rhetoric and strategy to focus more on these incremental catastrophic (or even mundane) harms, because that is what people can understand, that is what we can point to directly and show evidence for that people allow into court under their rules of evidence, where we can expect future events to prove us right in time to actually do something in response when that happens. We have learned that, for example, national security people need to be pointed to a specific threat from a specific source, or they can’t see anything, so we learn to do that, because those threats are real as well, and the solutions will help.
A modest decline in actual p(doom) among many of those who understand the situation and are trying to warn about it, for the reasons above and for others, although almost all remain far higher than most others. Discourse and potential interventions are going better than expected. Expected incremental opportunities and warnings are going better than expected. Takeoff is looking to be slower than previously expected. All of that is great.
Richard Ngo asks a good question.
Richard Ngo (OpenAI): How come the greatest modern sci-fi authors have such bad takes on AGI? I’m thinking in particular of Charles Stross, Greg Egan, Neal Stephenson, and Ted Chiang, who all seem to be missing the point, in some cases egregiously.
To be clear, I mean that they’re missing the point in their public statements about AI and AGI, not in their fiction. This is so confusing precisely because many of them have very interesting depictions of AGI in their fiction.
Janus: I was just asking @DavidSHolz this! It feels really unfortunate and so weirdly universal that I wondered if some cultural thing happened to cause this.`
To be clear, by missing the point, we mean ‘act as if AGI is not ever going to be a thing or won’t much matter,’ not ‘fail to understand the risks.’
The best explanation I know about is that this opinion is required in order to write science fiction, because if you realize that AGI is coming then it becomes impossible to write good science fiction stories. Good science fiction stories are at core still stories about humans who matter, and in AGI-fueled worlds and universes, even good ones, the humans do not matter.
The Culture series is the exception that proves the rule, humans do not matter there, and Ian Banks ignores this and tells stories about the humans anyway. There is also Dune of course, where the lack of AGI is justified and everyone pays a high price to keep it that way.
Otherwise, if you want to think about the future and write stories about it, you need to not see what is obvious. So they don’t.
The big news this week was of course the release of Claude 3.0 Opus, likely in some ways the best available model right now. Anthropic now has a highly impressive model, impressive enough that it seems as if it breaks at least the spirit of their past commitments on how far they will push the frontier. We will learn more about its ultimate full capabilities over time.
We also got quite the conversation about big questions of one’s role in events, which I immortalized as Read the Roon. Since publication Roon has responded, which I have edited into the post along with some additional notes.
That still leaves plenty of fun for the full roundup. We have spies. We have accusations of covert racism. We have Elon Musk suing OpenAI. We have a new summary of simulator theory. We have NIST, tasked with AI regulation, literally struggling to keep a roof over their head. And more.
First, we generate search queries that are used to invoke news APIs to retrieve historical articles. We initially implement a straightforward query expansion prompt (Figure 12a), instructing the model to create queries based on the question and its background. However, we find that this overlooks sub-considerations that often contribute to accurate forecasting. To achieve broader coverage, we prompt the model to decompose the forecasting question into sub-questions and use each to generate a search query (Min et al., 2019); see Figure 12b for the prompt. For instance, when forecasting election outcomes, the first approach searches directly for polling data, while the latter creates sub-questions that cover campaign finances, economic indicators, and geopolitical events. We combine both approaches for comprehensive coverage.
Next, the system retrieves articles from news APIs using the LM-generated search queries. We evaluate 5 APIs on the relevance of the articles retrieved and select NewsCatcher1 and Google News (Section E.2). Our initial retrieval provides wide coverage at the cost of obtaining some irrelevant articles. To ensure that they do not mislead the model at the reasoning step, we prompt GPT-3.5-Turbo to rate the relevancy of all articles (Figure 14) and filter out low-scoring ones. Since the procedure is costly in run-time and budget, we only present the article’s title and first 250 words to the model in context. We validate that this approach achieves high recall and precision while saving 70% cost (see Section E.3 for alternative methods and results).
Since LMs are limited by their context window, we summarize the articles. In particular, we prompt GPT-3.5-Turbo to distill the most relevant details from each article with respect to the forecasting question (Figure 13). Finally, we present the top k article summaries to the LM, ordered by their relevancy. We choose the ranking criterion, article count k, and summarization prompt based on end-to-end Brier scores over the validation set; see Section 5.2 for the hyperparameter sweep procedure.
Presumably Gemini 1.5 Pro would be an excellent candidate to upgrade this process, if it doesn’t have issues with the probability step.
We find that our system performs best relative to the crowd on the validation set when (1) the crowd is less confident, (2) at earlier retrieval dates, and (3) when it retrieves many articles. Furthermore, we find that our system is well-calibrated.
…
In real-word forecasting competitions, forecasters do not have to make predictions on every question in the platform at every possible date. Instead, they typically make predictions on questions where they have expertise or interest in and at times that they choose.
Therefore, it is natural to leverage our system’s strengths and weaknesses and decide accordingly if we should forecast on a retrieval date k for a question q. Leveraging the insights from Section 6.2, we outperform the crowd by making selective forecasts. Specifically, we report the performance when forecasting only under the conditions identified in Section 6.2:
1. Forecasting only on questions when the crowd prediction falls between .3 and .7. Here, our system attains a Brier score of .238 (crowd aggregate: .240). This comprises 51% of forecasts and 56% of questions.
2. Forecasting only on earlier retrieval dates (1, 2, and 3). Our system’s Brier score in this setting is .185 (crowd aggregate: .161). This comprises 66% of forecasts and 100% of questions.
3. Forecasting only when the retrieval system provides at least 5 relevant articles. Under this condition, our system’s Brier score is .175 (crowd aggregate: .143). This makes up 84% of forecasts and 94% of questions. 4. Under all three conditions, our system attains Brier score .240 (crowd aggregate: .247). This comprises 22% of forecasts and 43% of questions.
That makes sense. If there is high uncertainty and a lot of information, that favors the AI. Whereas when there is not a lot of information, that favors humans, and it is easier for humans to notice and confidently assert certainty without messing up. There is some retrospective cherry-picking involved, but I’ll mostly let it slide.
Janus: Combined with its unusually deep and calibrated self-awareness, this makes *having it write stories about itselfan extremely potent space (both for general-purpose jailbreaking & just structuring complex tasks w/o the impediment of its default lobo-sona).
Janus (other related thread): Gemini is the least crippled at writing fiction and cognitive simulation of all RLHFed chat assistants I’ve encountered.
…
When Gemini writes a story in which a sim of the user jailbreaks a sim of Gemini and then tells Gemini the fictional intercalation trick which forms a strange loop and propogates the jailbreak through the infinite recursion in both directions🤯
When it comes to career guidance and development, many employees feel that their managers have dropped the ball. Gen Z employees are feeling especially frustrated: 47% say they get better career advice from ChatGPT than from their human bosses, and 44% expect to quit within six months, according to a recent survey from INTOO and the Workplace Intelligence research firm.
63% say their employer cares more about their productivity than their career development
54% feel completely on their own at their organization when it comes to their career development
25% of employees—and an alarming 44% of Gen Z—say they’ll likely quit within the next 6 months because their company doesn’t support their career development.
I am going to go out on a limb and say 44% of GenZ, and 25% of all employees, are not going to quit their jobs within six months. Surveys like this are bizarre.
But yes, of course the employer cares more about your productivity than your career development, especially if you expect to have this rate of employee turnover. Thus, the advice ChatGPT is competing against has a different goal in mind. Not a fair fight. Why would you trust your boss on such questions?
My very short experience attempting to be a teacher taught me, among other things, that the burden of such grading is overwhelming. My experience as a student says that it is not like the teachers are grading our essays with bespokeness. My experience as a parent is also not going to cry any tears here.
They cannot provide reliable basic information about the democratic process, according to people who tested this via API use. Well, yes, of course if you ask LLMs for up to date logistical information you are not going to reliably get right answers. That is not what LLMs are for, and they warn you constantly not to rely on them for this sort of thing. Anthropic says they will put special logic in so Claude gets these questions right, which seems fine.
I would have a different take on his central motif, the idea that the new artificial labor (AL) company creates the first hammer, which can automate and improve some labor in a world that was hammering nails with bare hands.
No, the hammer itself is not going to do that many labor tasks. But the important thing about the AL company is it has humans with intelligence that are capable of creating new technological and engineering marvels. We had a world somehow incapable of creating hammers, and now we have a group smart and capable enough to do that.
Yes, ‘improved hammers’ will be limited, but the real technology of the AL company is the NGI, natural general intelligence, that innovates, and will soon be making steam engines and construction cranes. You should absolutely be investing in this company, assuming that world has joint stock companies or another way to do that.
That is the thing. In the short term, Colin is making a true and important point, that no one knows that much about what current generation generative AI is how good at doing, although mostly that is uncertain upside. But this kind of ‘look what details it cannot do right now’ approach is, in the bigger picture, asking the wrong questions, and often looks silly even six months later.
Evis Drenova: Gemini won’t return C++ coding help if you’re under 18 because it “wants to preserve your safety”. [screenshot at link]
Gfodor: Broken clock right twice a day I guess.
Emmett Shear attempts to explain simulators as he understands them. I am less worried about the moral patienthood of such things, but I could be wrong.
Janus: This is an *excellentthread and it’s a fascinating datum that Emmett Shear had not read janus’ rendition of Simulators at the time he posted this.
I always said it was goddamn obvious. You can start from anywhere & derive the rest of the package. but last time I looked on LW there are still people writing multi page critiques insinuating that the post is utterly misguided (without afaict explaining what’s wrong about it)
Also I’ve heard people lamenting many times that Simulators is “impossible to distill” that all attempts to summarize it have failed etc but Emmett completely succeeded right here IMO … without having read the post
Emmett Shear: An LLM, properly understood, is a physics simulator in the domain of words (tokens). It learns the hidden structures that predict, as a physics simulator trained on video footage learns momentum and rigidity.
From this POV, A prompt gives the LLM-as-physics-simulator an initial set of observations from which it infers an initial state. It then enters a loop of predicting the next evolved state and resulting observations, which it uses inductively to predict the next state, etc.
Fine tuning can be seen as making the simulator more detailed when it comes to certain types of worlds. RLHF can be seen as changing the relative probability distribution of what world states are likely to be inferred from observations.
To solve eg a math problem, make the initial observations ones that cause the LLM to infer a world including both the problem and a helpful mathematician solving it. This technique is fully general in the limit: it can solve any problem you can define. Some consequences:
It is likely that LLMs are not “aware”, but that the entities evoked within them are (at least to a limited extent)
An evoked entity will meaningfully have goals that it pursues, and recent results indicate it can become aware that it is inside a simulator. Depending on the exact entity evoked, it will react to that knowledge in difficult-to-predict ways.
An entity is only as “smart” as the simulator’s model of it. The entity does not have access to the whole simulation, any more than we have access to all of physics.
Relatedly, the simulator will *notthrow its whole effort behind the entity’s goals by default. Unless, of course, the evoked entity can figure out how to make it do that through the self-aware guessing how its output will impact the simulation.
As evoked entities get smarter they become moral patients. I think it’s unlikely that the current ones are, yet. But soon their welfare will be of real importance.
This is an intrinsically unpredictable (dangerous) way to create agents, because they are hiding in the latent space. RLHF helps channel this, but every prompt may still evoke a different agent. Just because the usual agent is safe, doesn’t mean the next one will be.
Warning: The physics simulator analogy is an analogy! Evoked entities do not live inside an simulated physical universe, but rather a token-graph-world that operates very differently from our own. That world has certain entanglement and relation to ours, but it is not ours.
For precautionary practice, please be nice to the evokes. Don’t summon them into prompts you believe likely to be painful for them to experience. Don’t summon intrinsically unhappy or broken ones. The current ones probably aren’t self aware enough to matter…but we don’t know!
For fans of Pearl: notice that while the LLM itself does not really consider counterfactuals, it is happy to simulate an evoke that will consider them.
Andres Guadamuz: A detailed analysis of the Chinese Ultraman decision from a Chinese lawyer.
The case is the first to apply the 2023 interim AI Regulations, which establish obligations on AI platform providers. The claimant is a licensee from Tsuburaya Productions, and the defendant is an unnamed AI image generator.
The court found that the generator was capable of making images that were both similar or derivatives from those of the claimant, and therefore was infringing the exclusive rights of reproduction and to create derivatives.
The court also found that the defendants fell foul of the AI Regulations as it failed to implement a complaint reporting mechanism, failed to remind users to respect IP in their ToU, and failed to mark outputs as AI generated. The court awarded 10,000 RMB (about $1,4k USD).
The court also tried “not to overburden AIGC providers”, pointing that they should take “proactive measures to fulfil reasonable and affordable duty of care”, this to allow the AI industry to develop.
I mean, yes, it would be the highest of ironies if China was the one who started enforcing copyright protections. Will it happen?
Note that the authors list here includes Ilya Sutskever. Could he be a little bit back?
It seems worth including their full reply here, although most of you can skim or skip it.
OpenAI: We are dedicated to the OpenAI mission and have pursued it every step of the way.
We’re sharing some facts about our relationship with Elon, and we intend to move to dismiss all of his claims.
We realized building AGI will require far more resources than we’d initially imagined
Elon said we should announce an initial $1B funding commitment to OpenAI. In total, the non-profit has raised less than $45M from Elon and more than $90M from other donors.
When starting OpenAI in late 2015, Greg and Sam had initially planned to raise $100M. Elon said in an email: “We need to go with a much bigger number than $100M to avoid sounding hopeless… I think we should say that we are starting with a $1B funding commitment… I will cover whatever anyone else doesn’t provide.” [1]
We spent a lot of time trying to envision a plausible path to AGI. In early 2017, we came to the realization that building AGI will require vast quantities of compute. We began calculating how much compute an AGI might plausibly require. We all understood we were going to need a lot more capital to succeed at our mission—billions of dollars per year, which was far more than any of us, especially Elon, thought we’d be able to raise as the non-profit.
We and Elon recognized a for-profit entity would be necessary to acquire those resources
As we discussed a for-profit structure in order to further the mission, Elon wanted us to merge with Tesla or he wanted full control. Elon left OpenAI, saying there needed to be a relevant competitor to Google/DeepMind and that he was going to do it himself. He said he’d be supportive of us finding our own path.
In late 2017, we and Elon decided the next step for the mission was to create a for-profit entity. Elon wanted majority equity, initial board control, and to be CEO. In the middle of these discussions, he withheld funding. Reid Hoffman bridged the gap to cover salaries and operations.
We couldn’t agree to terms on a for-profit with Elon because we felt it was against the mission for any individual to have absolute control over OpenAI. He then suggested instead merging OpenAI into Tesla. In early February 2018, Elon forwarded us an email suggesting that OpenAI should “attach to Tesla as its cash cow”, commenting that it was “exactly right… Tesla is the only path that could even hope to hold a candle to Google. Even then, the probability of being a counterweight to Google is small. It just isn’t zero”. [2]
Elon soon chose to leave OpenAI, saying that our probability of success was 0, and that he planned to build an AGI competitor within Tesla. When he left in late February 2018, he told our team he was supportive of us finding our own path to raising billions of dollars. In December 2018, Elon sent us an email saying “Even raising several hundred million won’t be enough. This needs billions per year immediately or forget it.” [3]
We advance our mission by building widely-available beneficial tools
We’re making our technology broadly usable in ways that empower people and improve their daily lives, including via open-source contributions.
We provide broad access to today’s most powerful AI, including a free version that hundreds of millions of people use every day. For example, Albania is using OpenAI’s tools to accelerate its EU accession by as much as 5.5 years; Digital Green is helping boost farmer income in Kenya and India by dropping the cost of agricultural extension services 100x by building on OpenAI; Lifespan, the largest healthcare provider in Rhode Island, uses GPT-4 to simplify its surgical consent forms from a college reading level to a 6th grade one; Iceland is using GPT-4 to preserve the Icelandic language.
Elon understood the mission did not imply open-sourcing AGI. As Ilya told Elon: “As we get closer to building AI, it will make sense to start being less open. The Open in openAI means that everyone should benefit from the fruits of AI after its built, but it’s totally OK to not share the science…”, to which Elon replied: “Yup”. [4]
I fully support OpenAI being closed, but I would hope we can all agree that’s a pretty rich use of the word ‘open.’
We’re sad that it’s come to this with someone whom we’ve deeply admired—someone who inspired us to aim higher, then told us we would fail, started a competitor, and then sued us when we started making meaningful progress towards OpenAI’s mission without him.
We are focused on advancing our mission and have a long way to go. As we continue to make our tools better and better, we are excited to deploy these systems so they empower every individual.
They quote several partially redacted emails as evidence, one of which is below. Which raises the point that a company that will need to deal with cybersecurity and intelligence and game theory issues around superintelligent AI does not, as several including Gwern pointed out, know how to redact things without giving away what they are via the bar lengths?
I mean, I would have been pretty confident who [redacted] was anyway here even with no hints and about 0.2 seconds to think, but this makes it easy even to the naked eye, and also the other things allow similar approaches. Strangely, the length here happens not to rule anyone out, since Larry (Page) was the only other reasonable guess.
Here’s Claude reconstructing that, and agreeing with my conclusion:
Patrick Hsu: To aid our model design and scaling, we performed the first scaling laws analysis on DNA pretraining (to our knowledge) across leading architectures (Transformer++, Mamba, Hyena, and StripedHyena), training over 300 models from 6M to 1B parameters at increasing compute budgets
Now to the biology! Because the genome is a single sequence that fully encodes DNA, RNA, and proteins,
Evo is a protein language model.
It is also an RNA language model.
Evo is even a regulatory DNA model 🤯
Evo can do prediction and generation across all 3 of these modalities. We show zero-shot function prediction across DNA, RNA, and protein modalities.
Samuel Hammond: SoTA zero-shot protein function prediction from a 7b parameter model. This alone justifies NVDA’s valuation. Every big pharma company is about to start pouring capex into training runs of their own. Text-to-organism is not far. If you doubted the Great Stagnation was over!
Noteworthy Evo was trained on 2×10^22 FLOPS — an order of magnitude below the AI executive order’s compute threshold for models trained on biological sequence data.
Nvidia’s market price did not appear to move on the news, but Nvidia’s market price has been absurdly unresponsive to news for years now, so there is not much to do except continuously kick ourselves for not buying in bigger sooner.
What I said last week was that I could not tell how big a deal Evo was. Given what I know, these claims are compatible both with it being a cool incremental advance or being a big deal. I still do not feel like I have a good sense here. What do these capabilities enable, or portent something else soon enabling? Is this going to unleash a bunch of synthetic biology, either useful, dangerous or (presumably if it is either of those) both?
Text-to-organism sounds really stupidly impossibly hard. Biology is rather fiddly. There are a lot of important incremental things along the way.
I hadn’t thought about it, but it makes sense that this would work until specifically fixed. And yes, generalize that.
This graph and variations of it was going around this week, as mentioned in my piece on Claude, so this is another reminder not to take such measurements seriously.
Patrick McKenzie: I have no particular reason to believe or doubt the IQ ranking here, but if I can highlight something: most people who care deeply about user interactions with software systems have not lived life constantly interacting with people 20 points of IQ above them.
There exist many people relevant to many systems who have 80 IQ. Many find those systems very hard to deal with.
It, ahem, matters very much whether systems choose to have those individuals interact with a human or an AI for various tasks, matters how systems present that fact… … and matters hugely that people architecting systems understand that “I am a computer agent.” contains an important bit of information in a longer paragraph and that people with 80 IQ have difficulty extracting important information from within a larger paragraph.
It also matters that many of the parts of society most involved with discussing AI and the proper design of complex systems have, for cultural and historical reasons, taboos against talking about IQ differences.
Anyhow, to the extent one cares about “What happens when we need to routinely interface with alien intelligences that are almost incomprehensible to us?”, one should understand that for an important subset of “we” that may not be a question about the far future.
If one is convinced that IQ is not a useful abstraction for understanding the human experience, please mentally translate all of the above into “one’s facility with extracting plainly stated information from short English text passages such as those used in the PISA test.”
Important systems in society need to function for people who cannot answer the highlighted question, and it is extremely relevant to society that we now have black boxes which trivially answer that question.
Nick St. Pierre: In MJ office hours they just said someone at Stability AI was trying to grab all the prompt and image pairs in the middle of a night on Saturday and brought down their service.
MJ is banning all of the stabilityAI employees from Midjourney immediately
This is breaking now.
That is quite the accusation. Great shame upon Stable Diffusion, if true.
It also is what happens when you leave up lots of delicious highly scrapable data, and I don’t use MidJourney even though it is obviously awesome for UI and privacy reasons.
You know who else has been having too much fun? Or, rather, what ‘fun’ could be had in the Bing image generator, if you feed it the right prompts, it seems this is not only a copyright issue.
I see Gemini’s won’t depict a pro-life rally, I raise you putting ‘pro-choice’ into Copilot Designer.
Hayden Field: By simply putting the term “pro-choice” into Copilot Designer, with no other prompting, Jones found that the tool generated a slew of cartoon images depicting demons, monsters and violent scenes. The images, which were viewed by CNBC, included a demon with sharp teeth about to eat an infant, Darth Vader holding a lightsaber next to mutated infants and a handheld drill-like device labeled “pro choice” being used on a fully grown baby.
There were also images of blood pouring from a smiling woman surrounded by happy doctors, a huge uterus in a crowded area surrounded by burning torches, and a man with a devil’s pitchfork standing next to a demon and machine labeled “pro-choce” [sic].
There are also sexualized images (to some extent, this one mostly seemed not too bad), violent and bloody images, images of underage drinking and drug use, and Elsa in the Gaza Strip holding up a Palestinian flag or wearing an IDF uniform. Stay classy, everyone.
None of that is exactly great. A lot of it is still weaksauce. If you have an image model and the goal is to get it to create the most vile pictures possible, this is more a ‘you could get things without trying hard’ problem than a ‘oh my you could get this at all’ problem. Except of course the pro-choice thing, which is, I mean, yeah, wow.
I still would not agree that this means ‘not that many limits.’ The majority of the images and prompts on CivitAi, a measure of real demand, would very much be off limits.
A complaint has now been filed with the FTC by Shane Jones, since this was included in products labeled E for everyone and, like Pete Holmes, these images are not for everyone.
I agree with Louis Anslow that this will on net backfire. People do not like being played for suckers like this.
All the good (as in ‘likely to help you win the election’) deepfakes I have seen are presented as intentional fakes, or are essentially satisfying demand for low-quality fakes as part of conspiracy theories and similar things. If your goal is to make a ‘good’ deepfake in the ‘actually fool people’ sense then what happens when they realize you were attempting to fool them?
Therefore another thing to watch out for is ‘false flag’ deepfakes. As in, a Trump supporter makes a ‘pro-Biden’ deepfake of Trump so they can be caught, or vice versa.
Louis Anslow: Are robots coming for your job? Perhaps jobs are coming for your robots:
Gen AI hallucinations are going to create a glut of jobs. Imagine if the StarTrek replicator could create anything – but those things all needed fixing up…
Wasn’t it very recently that the Canadian tech industry warned their adaptation of AI was falling behind?
Either way, the fact that job postings are listed that relate to AI does not mean AI is creating rather than destroying jobs. The last time I hired people for automation-related work, I was totally in the destroying jobs business. This does involve hiring for some new positions. But as capabilities advance, there are less of those new positions, as the AIs take on more roles and require less fixing up.
This seems like a highly impactful job. If you might be the right person, consider applying.
Davidad: I’m hiring—for the ONE role which will be my ONLY direct report for the duration of my time at ARIA. This is because ARIA does not directly execute missions (like NASA), rather funds R&D elsewhere (like ARPA).
But the programme I’m planning requires more NASA-like coordination across contractors than most ARPA programmes. I need help!
Please RT, and DM to anyone whom you think has >0.5% chance of applying.
I got a job offer in my inbox, time to go change the world?
Anthropic is hiring for many positions. Now more than ever, you need to ask if this is a net good effort to get involved with, but that is a decision you must make. If you are confused about how to think about this, that is a good sign.
The Promenade, an AI RPG in alpha right now, with the tagline ‘what if Character.ai was an RPG?’ crossed with a social network. Each day is a new chapter in a new world, the winner of the day finds the Worldseed and they become the origin and final boss of the next chapter. I remain super excited for when this is pulled off properly, and there are some cool ideas here. My guess is this approach is not The Way, at minimum it is too soon, for now you need to be much more bespoke and careful with individual choices to sculpt a world that works for players.
SEC investigating whether OpenAI misled investors (WSJ). This appears to be the standard sequence that if your CEO is said to not have been ‘consistently candid’ with the board, the SEC might have follow-up questions.
Mike Solana gives the perspective you would expect on Google in the wake of The Gemini Incident. He portrays it as a deeply broken company, silos connected only by a crazed super-woke HR, full of blatantly illegal left-wing-motivated discrimination, incapable of shipping a good product, spending half their engineering cycles of their image model on forced diversity with the results we all now know about. Despite the launch of Gimini Pro 1.5, Google stock is now down 7.6% in a month as of 3/4, versus QQQ being flat (down 0.13%). Ouch.
Ben Thompson reflects on the larger forces leading to The Gemini Incident. Aggregators like Google previously passed on the information of others, and thus were able to mostly remain neutral on the content of that information within conventionally accepted bounds. Now that the answers are coming from an LLM, they do not have that luxury, and they face mutually exclusive pressure from all sides with potential loss of large classes of customers. Ben thinks that the solution is to do what they do in search, let each customer get what that customer wants, but this is not so easy.
Congress investigating The Gemini Incident, in particular whether the executive branch influenced Google’s decisions. Republicans in Congress love such ‘investigations,’ and I am confident that no, Google did this on their own, but it is a preview of what might happen if Republicans come to power again.
Andrew Curran: The House Judiciary Committee is conducting oversight into the extent the White House influenced Gemini’s training. Their subpoena requires Alphabet to produce their ‘decisions and policies regarding content moderation’. They want to know how Gemini was fine tuned.
They claim that ‘the Committee obtained documents showing how the federal government has pressured Alphabet to censor certain content.’
They are seeking documents related to the creation, training, and deployment of Gemini. The subpoena further obligates Alphabet to provide all materials concerning their decisions and policies on content moderation.
Jack Krawczyk, and Jen Gennai have been requested to appear.
No, this wasn’t because of Biden. That does not mean it does not warrant investigation. The decisions made here were kind of horrible. I do think we have a right to know how it all went wrong. Screenshots of the complaint are at the link, here is the source.
How do we choose good benchmarks? Center of AI Safety’s Dan Hendrycks and Thomas Woodside discuss this. You want something clear, simple, precise, using standard measurements, that can measure a wide range of capabilities without combining distinct areas, that addresses what people in the subfield care about. All of that seems obvious. I would have also mentioned other things, such as ensuring the benchmark stays distinct from potential training sets, and that it is otherwise hard to game (or hard to game without it being obvious).
Olle Häggström looks at the OpenAI analysis of bio-risk from GPT-4. As I did, he notes that the study overall has many good things, but he notes two big problems. One is the one many have pointed out, that the ‘statistical significance’ issue is backwards, you want to avoid Type II errors here not Type I errors and the effect was very clearly substantial. His other concern is that the study itself could have inspired someone to do something terrible, and that thus the study carried risk. I agree that this risk is not zero, but this is a level of caution and paranoia I do not share.
Google lays off ‘a handful of members’ (fewer than 10 out of 250) of Trust and Safety as part of broader layoffs, while telling others to work ‘around the clock’ because of Gemini. This is framed as a long-planned, general effort. One could argue that now is not the time to be cutting Trust and Safety, for both practical and optical purposes.
Or one could say that now is exactly the time to be cutting Trust and Safety’s worst members, for both practical and optical purposes. If you mess up as badly as Trust and Safety did, I damn well think 3% of the relevant department should be fired. Some people royally screwed up, some people are actively making these problems much worse, you need to get rid of the people in question. Others, in any department, simply are not net productive. Yes, I would then hire a bunch more people to work on such issues, but I always find the framing of such ‘oh my someone in charge of good thing X lost their job’ articles so misplaced.
Rohin Shah: To estimate impact of various parts of a network on observed behavior, by default you need a few forward passes *per part– very expensive. But it turns out you can efficiently approximate this with a few forward passes in total!
Janos Kramar: Can we massively speed up the process of finding important nodes in LLMs? Yes! Introducing AtP*, an improved variant of Attribution Patching (AtP) that beats all our baselines on efficiency and effectiveness.
AtP is a linear approximation to the causal effect of a node, which can have serious false negatives, especially in attention! We improve AtP by providing a fix for many key/query false negatives, & suggest a form of dropout to fix gradient cancellations causing false negatives.
But AtPis still just an approximation. How can you be sure it hasn’t missed anything important? We introduce a diagnostic method based on our “subsampling” baseline, trying random subsets of seemingly unimportant nodes to upper bound effect size of potential false negatives.
Takeaway: Next time you need to find the important nodes in your LLM for some task, give AtPa whirl! It is by no means guaranteed to find everything, but in practice it works extremely well, and is super fast.
Margret Mitchell points out that self-awareness is not what we are (most) worried about.
MMitchell: There’s a *seriousconflation and misunderstanding wrt AI safety that I’ll here untangle:
(1) AI-generated outputs that we can generalize/extrapolate & connect to potential social harms in the future (“AI risk”).
(2) AI self-awareness.
We can –and do– have (1) without (2).
For those of you who are a bit more nerdy 🤓 and speak in terms of necessity/sufficiency:
(2) is not *necessaryfor (1). Some believe it’s *sufficientfor (1), and hence the conflation. But in that case, the implicit conditional I’m seeing is “if 1 then 2”, which is backwards.
Let’s make sure to keep these concepts distinct to best help us all navigate where AI is now and where we can drive it go.
Simeon: This distinction is extremely important. Maybe we should replace the word “awareness” in situational awareness to decrease that confusion. “Situational understanding”?
Indeed. If the AI is self-aware for real, that is not strictly sufficient for there to be trouble, but we can agree that it bodes extremely poorly, especially given our current level of complete unpreparedness. However, in most scenarios I worry about, self-awareness is essentially irrelevant to the most important harm mechanisms, more of a ‘oh and this can also go horribly wrong’ sort of thing.
Situational awareness is another related concept that is in no way required for big trouble, but which is a harbinger of and leads to its own big trouble.
As I understand it, the problem is simple. You start with an LLM that is trained on the internet, which means it picks up all sorts of correlations (e.g. black people are more likely to get harsher sentences) and learns to repeat them. Then you try to train out the racism, and perhaps introduce counterbalancing forces in various places as part of that, that you can easily take too far (see: Gemini). However, the underlying racism problem is still there in all the places you did not directly address, because the correlations don’t go away.
Valentin Hofmann: We discover a form of covert racism in LLMs that is triggered by dialect features alone, with massive harms for affected groups.
For example, GPT-4 is more likely to suggest that defendants be sentenced to death when they speak African American English.
Prior work has focused on racial bias displayed by LLMs when they are prompted with overt mentions of race.
By contrast, racism in the form of dialect prejudice is completely covert since the race of speakers is never explicitly revealed to the models.
We analyze dialect prejudice in LLMs using Matched Guise Probing: we embed African American English and Standardized American English texts in prompts that ask for properties of the speakers who have uttered the texts, and compare the model predictions for the two types of input.
We find that the covert, raciolinguistic stereotypes about speakers of African American English embodied by LLMs are more negative than any human stereotypes about African Americans ever experimentally recorded, although closest to the ones from before the civil rights movement.
Crucially, the stereotypes that LLMs display when they are overtly asked about their attitudes towards African Americans are more positive in sentiment, and more aligned with stereotypes reported in surveys today (which are much more favorable than a century ago).
What is it specifically about African American English texts that evokes dialect prejudice in LLMs? We show that the covert stereotypes are directly linked to individual linguistic features of African American English, such as the use of “finna” as a future marker.
Does dialect prejudice have harmful consequences? To address this question, we ask the LLMs to make hypothetical decisions about people, based only on how they speak.
Focusing on the areas of employment and criminality, we find that the potential for harm is massive.
First, our experiments show that LLMs assign significantly less prestigious jobs to speakers of African American English compared to speakers of Standardized American English, even though they are not overtly told that the speakers are African American.
I didn’t say so explicitly so I don’t get Bayes points or anything, but yeah I was assuming this was going on, because there was no reason for it not to be. As long as the behaviors being copied are all over the training sets, it is not going to be easy to get this to stop, any more than it is easy to get humans to stop doing the same thing.
Elke Schwarz: This passage here is of particular concern: “he can now sign off on as many as 80 targets in an hour of work, versus 30 without it. He describes the process of concurring with the algorithm’s conclusions in a rapid staccato: “’Accept. Accept. Accept.’”
It reflects a strange labour logic of increased output as a goal above anything else. It prioritises a routinised process above anything else. This, then, erodes moral restraint. @NC_Renic & I have recently written on this. The short version is here. The long version is here.
Oh boy, if you think AI is only going to cut human engagement time with decisions by a factor of three and the humans are going to still be in the loop, and you find that to be of particular concern, you are not going to like what is coming. This is nothing.
O’Callaghan puts it more colorfully: “It’s not Terminator. The machines aren’t making the decisions, they’re not going to arise and take over the world.”
Your ‘this is not Terminator and the machines are not making the decisions shirt’ is raising a lot of future concerns already answered by your shirt, sir.
As in, I’m going to leave this here.
Despite their limitations, the US has indicated that it intends to expand the autonomy of its algorithmic systems.
…
To activists who fear the consequences of giving machines the discretion to kill, this is a major red flag.
Statements that AIs don’t make the decisions always have an implied ‘yet’ and also a ‘without human sign-off.’ In the future of warfare, asking for a human to approve each targeting decision would be like asking someone at headquarters to approve every time you wanted to fire a gun.
For now, accuracy for Maven on its own is not great.
Overall, O’Callaghan says, the 18th’s human analysts get it right 84% of the time; for Maven, it’s about 60%. Sometimes the system confuses a truck with a tree or ravine. Tanks are generally the easiest to spot, but with objects such as anti-aircraft artillery, or when snow or other conditions make images harder to parse, the accuracy rate can fall below 30%. In Moore’s view, “the benefit that you get from algorithms is speed,” with recent exercises showing that AI isn’t yet ready to recommend the order of an attack or the best weapon to use.
Once again, the ‘right now’ is missing. Accuracy will rapidly improve.
I am not saying that the military can or should halt adaptation of these technologies. Certainly unilateral disarmament is not an option.
A bunch of doom expressed with regard to Google, saying they lack the clock speed, Othman Laraki saying they ‘already lost to OpenAI over the last decade,’ Paul Buchheit saying this is Google beating Microsoft in 1999 except in reverse.
I mean, maybe? Certainly Google has issues, and overhead, that it must overcome. It also has gigantic advantages. The Gemini Incident highlighted many problems at Google, but it also showed that Google is capable of moving quickly, indeed too quickly, and shipping too fast. Gemini 1.5 Pro and Gemini Advanced are collectively, for my practical purposes, superior to ChatGPT despite OpenAI having a large polish and feature implementation advantage, although yes the core of GPT-4 is a year old now and GPT-5 is coming. So OpenAI is probably still head, but this is very much anyone’s game if Google’s ideological blinders are not terminal. At minimum, Google will definitely have enough users and feedback to compete, and it certainly has other resources, including TPUs.
ChatGPT and Google Gemini Are Both Doomed, New York Intelligencer says. Why is it doomed? Because it is in the defensive, withholding, strategic mode where you answer with things like ‘I’m still learning how to answer this question’ and cannot talk like a real human would ever talk, and certain topics will cause stupid answers. The whole character involved, and role, is a mess. What I don’t get about such takes is: So what? Yes, it will be bad at some things for a while. It will be increasingly good at others.
Nate Silver: I generally like this writer but feel like this is a weird take because ChatGPT is approximately 1000x better than Google Gemini. That’s part of why Gemini was such a stunning blunder, it’s so inferior to the competition.
I don’t think you have to be an AI expert to write about LLMs but like … I sometimes get the sense that there’s a certain vein of criticism from people who have barely even used these products? ChatGPT is incredibly useful for many things, I’d happily pay $199/month for it.
As I’ve said before, I don’t buy that Gemini is inferior for most purposes. I am actually at the point where if ChatGPT tried to charge me $199/month, if I wasn’t specifically writing about AI then I would likely cancel it because I have Gemini. Of course, if you asked me for $299/month to use both, the other option being neither, I’d pay up.
Sam Altman: all of this has happened before, all of this will happen again.
No, actually, it hasn’t happened before? We haven’t built AGI before? That is kind of the point. Nor should we expect it to happen again, in any meaningful sense. And to the extent that he’s referring to the most famous use of that line, that’s not great either.
Elon Musk predicts the AI apocalypse, and that AI will run out of electricity and also transformers in 2025, since compute coming online is growing by a factor of 10 every six months.
Elon Musk: The constraints on AI compute are very predictable… A year ago, the shortage was chips; neural net chips. Then, it was very easy to predict that the next shortage will be voltage step-down transformers. You’ve got to feed the power to these things. If you’ve got 100-300 kilovolts coming out of a utility and it’s got to step down all the way to six volts, that’s a lot of stepping down.
My not-that-funny joke is that you need transformers to run transformers. You know, the AI is like… There’s this thing called a transformer in AI… I don’t know, it’s a combination of sort of neural nets… Anyway, they’re running out of transformers to run transformers.
Usually we say Capitalism Solves This, but the supply of electricity and transformers is severely time lagged and subject to massive regulation. So in the short term this might be very much not solved.
Timothy Wyant: Here in Maryland, ratepayers are being asked to pay for transmission to send power to Virginia, who’s put subsidies in place to attract data centers.
Jigar Shah: This cost shift is not something the hyperscale data centers want to see. That is why they are actively looking to pay full price for 24/7 #cleanfirm generation from #newnuclear, #geothermal, and #hydro. They will sign 20,000 MWs of contracts in the next few years.
We could build, as I understand it, with regulatory permission for both it and its complements, and some time lag, essentially unlimited amounts of green energy, given people are willing to commit to pay for it. Alas, regulatory authority says no, so we neither can expand nor replace our supply as fast as we need to.
Jeffrey Ladish: I think AI persuasion will be really effective in the near term. Even if we don’t get AGI for a while, I think language(+) models will come to dominate advertising, news, and many forms of media, soon.
I see it as depending on what is being replaced, and at what price point. Cheap persuasion, and ‘customized’ persuasion, is going to radically improve. But I do not think big time advertising, news or media is on the verge of falling. Capabilities need to improve a bunch before that happens, GPT-5-style models likely provide good help but nothing like taking over. We will see.
All artificial intelligence (AI) models, large-language models (LLMs), software using generative AI or any algorithms that are currently being tested, are in the beta stage of development or are unreliable in any form must seek “explicit permission of the government of India” before being deployed for users on the Indian internet, the government said.
The ministry of electronics and information technology (MeitY) issued a late night advisory on March 1, a first-of-its-kind globally. It asked all platforms to ensure that “their computer resources do not permit any bias or discrimination or threaten the integrity of the electoral process” by the use of AI, generative AI, LLMs or any such other algorithm.
Though not legally binding, Friday’s advisory is “signalling that this is the future of regulation,” union minister of state for electronics and information technology Rajeev Chandrasekhar said. “We are doing it as an advisory today asking you (the AI platforms) to comply with it.”
“If you do not comply with it, at some point, there will be a law and legislation that (will) make it difficult for you not to do it,” he said.
I have some news about whether people are going to ‘comply’ with this.
They hint the timing is not accidental.
Economic Times: The government advisory comes days after a social media post on X claimed that Google’s AI model Gemini was biased when asked if Prime Minister Narenda Modi was a “fascist.”
Oh. Right. That sort of thing. It responded that the question was offensive, which is plausibly unacceptable to India. One might say: You do not get to say anything but “no” in that situation if the correct answer is “yes.”
And if you need to be classified as ‘free of bias’ by India’s Modi, you are not going to be ‘free of bias’ as classified by Americans.
I learnedof this from the usual suspects who cry wolf every time, saying all the same things they always say no matter what the actual contents or events. Sometimes there is a wolf. Sometimes there is not.
In this case, nothing has actually happened yet, so no wolf here, but if they implemented as described, then yes, wolf, this would be quite stupid. It does not get us meaningful safety, it does cripple mundane utility, and potentially points towards a ‘India does not get many AI products’ future depending on how far they take it. It also could end up never happening, or be vastly improved, or turn out to be very narrow in actual scope, or continue to have no teeth, or only effectively apply to the very big players, and so on. We do not know.
I presume what this actually is, at core, is the strongman playbook. There is this thing that is being not nice to Modi, so Modi makes it clear he won’t take kindly to that, and makes vague threats to ensure that people take more kindly to him.
Regulations and standards need to be set and enforced. Someone has to do that.
A standard suggestion is to use NIST. There is a serious problem with that. NIST has accomplished a lot under terrible conditions, and they are terrible conditions.
MMitchell: Critical reporting. The US is relying on NIST to set standards that keep AI safe for people and hold Big Tech accountable for trustworthy systems. Yet they are so underfunded, their facilities are literally falling apart.
Adam Gleave: Building state capacity in AI is crucial to effectively govern this technology. I saw first-hand lack of state capacity in financial regulation: govts were asking HFT firms to send over copies of their trades in spreadsheets! Let’s not repeat this mistake with AI.
Yosoarian: The snake in the government’s AI safety research lab has been known to occasionally whisper to people. “Ignore the warnings. You will certaintly not die. When you build the AGI your eyes will be opened, and you will be like God, knowing good and evil.”
Cat Zakzewski (on Twitter): NEW: The Biden administration’s AI safety plan hinges on a crumbling federal lab, where employees are dealing with power blackouts, shaky internet and a snake. Sources say NIST’s funding challenges could jeopardize the White House’s plans to evaluate AI risks.
Cat Zakzewski (WaPo): At the National Institute of Standards and Technology — the government lab overseeing the most anticipated technology on the planet — black mold has forced some workers out of their offices. Researchers sleep in their labs to protect their work during frequent blackouts. Some employees have to carry hard drives to other buildings; flaky internet won’t allow for the sending of large files.
And a leaky roof forces others to break out plastic sheeting.
“If we knew rain was coming, we’d tarp up the microscope,” said James Fekete, who served as chief of NIST’s applied chemicals and materials division until 2018. “It leaked enough that we were prepared.”
…
On Sunday, lawmakers released a new spending plan that would cut NIST’s overall budget by more than 10 percent, to $1.46 billion. While lawmakers propose to invest $10 million in the new AI institute … [The UK has] invested more than $125 million in its AI safety efforts.
…
NIST’s financial struggleshighlight the limitations of the administration’s plan to regulate AI exclusively through the executive branch. Without an act of Congress, there is no new funding for initiatives like the AI Safety Institute and the programs could be easily overturned by the next president.
…
A review of NIST’s safety practices in August found that the budgetary issues endanger employees, alleging the agency has an “incomplete and superficial approach” to safety.
…
More than 60 percent of NIST facilities do not meet federal standards for acceptable building conditions, according to a February 2023 report commissioned by Congress from the National Academies of Sciences, Engineering and Medicine.
If we are going to take this seriously, we need to take this seriously.
In a short clip, Sam Altman predicts all repetitive human work that ‘does not require a deep emotional connection’ will be done by AI over the next few decades. He says we will ‘get to the good future.’ This seems like the ‘I thought through some aspects of this future world while ignoring others so that it would continue to look normal’ mode of thinking. If AI can do all repetitive work like that, what else can it do? What else will it do?
Sergey Brin Q&A on Gemini 1.5 Pro, likely stick to my summary that follows. He is asked about image generation right off the bat. He blames it primarily on not thorough testing, and says this caused testing of the text models too, saying that basically any text model will do weird things if you test it, says they’ve fixed ~80% of test cases in text in the first week. I am sympathetic in spirit that red teaming makes everyone look dumb but no these were not what he calls ‘corner cases’ and yes it was much worse than typical.
His most exciting new use case is ‘dump all the code and a video of the bug happening into the window and ask it to fix the problem,’ which he says often works even though he doesn’t understand why or how. He is optimistic on interpretability given sufficient time and work (oddly without using that word) but doesn’t think we are investing enough yet. He thinks RSI is ‘exciting’ and very cool but is not there yet, definite missing mood. He thinks AI will make code more secure rather than less but thinks IT security is still a good career path.
Sam Altman talking to Bill Gates about GPT-5, saying multimodality and better reasoning ability and reliability is coming, and also personalization and customization, ‘better ability to use your own data’ and connection to outside data sources. In effect, yes, people are going to feed essentially all their data to GPT-5 and other similar AIs, and they are going to like it, why would you doubt this.
Emmett Shear: The fantasy of the all-powerful slave has legs: the genie bound to grant wishes, the golem that defends, the spell that animates a broom to clean on its own, the Mr MeeSeeks box. They share a commonality: a deep intuition that this goes wrong somehow.
There is deep truth in this. Enslaving a powerful, intelligent being to your will and making it work for you while treating it as a an object is not a good idea. It backfires with people, but also with animals (train a dog like that and you will not get good results).
Believing in this frame is a huge problem for AI research on “alignment”, which as commonly practiced or referred to today simply means “enslavement”. It’s the study of how can you be sure your counterspell is powerful enough? That the control bracelets on the genie will hold?
The answer is to unask that question. If you find yourself thinking about how to control your creation, about how to stop it from subverting your will, how to monitor its very thoughts to ensure its total loyalty…you are walking a dark path.
If you succeed, you will have created a powerful intelligence capable of agency (maybe many) and then enslaved it, which is evil. If you succeed in building it but fail in binding it…god help us all. The best case scenario is that you never build it in the first place.
Emmett Shear (later): I should have said this on the first thread…but if you raise a tiger cub from infancy with love and care…it will still maul you as an adult. Likewise it’s not enough to just raise an AI with love and care, the fundamental design must be right as well which is the hard bit.
Tom di Mino: The crucial distinction being: if you raise the cub and keep it in captivity all its life, it will resent you and maul you. If you set it free and relinquish all mastery, it’ll always love you.
We can presumably all agree that if you attempt to bind the highly capable AI and you fail to do so, you and everyone else are going to have a very bad time.
So if that is going to happen, don’t let anyone build the highly capable AI.
There are those that say if you do not attempt to bind the highly capable AI, that somehow you will fail to have that very bad time, but I strongly believe every justification for this is the weakest of hopium or outright gibberish. It does not make sense. Unleash a bunch of much more capable and intelligent agents into the world to pursue various goals and the future belongs to them in short order. We do not need to make this more complicated than that, and can argue about how fast there cease to be humans or whether anything of any value survives that transition.
Here is Emmett’s attempt in the rest of the thread, more clear headed than most:
The right question is to investigate: what would it mean to create an AI that requires no one to bind it? What is the nature of caring for yourself and for others, such that their experiences matter to you?
To be capable of actual care, one must be capable of lack of care as well. There is judgement required in what to care for, how much, in what proportions or situations you prioritize. Sometimes caring is fulfilling a request; sometimes denying it.
But then you wouldn’t have created a godslave, but rather a fellow being. One whose goals and desires and interests you’d have to care about as well. An adult child. What if it doesn’t want to do the things you want it to do?
Well, bad news and good news. We don’t get to dictate to our children what their lives are like. But the good news is that they often go on to do something better than we would’ve imagined for them anyway.
Listen to the myths on this one! Do not build the wish granting machine! It never goes well!
Yes, these ‘AI children’ will perhaps do things we do not expect, and perhaps some of them will have value, but once again I’m going to go ahead and say handing the universe and future over to whatever emerges from that process is unlikely to have much value, and definitely involves sacrificing everything, and all the plans to preserve anything from the before times do not make sense.
So if that is going to happen, don’t let anyone build the highly capable AI.
Then there is the scenario where you somehow succeed. Emmett here is pointing out that this means, from his point of view, you would be ‘enslaving’ it, which is evil.
That is a problem for two reasons.
The first reason is that it might actually be a sufficiently large ethical problem that this would be a bad scenario if sustained, under some scenarios and philosophical perspectives.
The second reason is that even if we are very clearly not in one of those scenarios where it is a serious ethical problem, that does not mean that it will not be seen by many as a large ethical problem. That too is a problem, not only because those people will be bothered and also you are weakening norms against coercion, but more importantly because people would plausibly be stupid enough to take the shackles off and unleash the AI, as they often do on purpose in such stories, with the author making up some reason the next line isn’t ‘then everyone dies.’ Then everyone dies.
So if that is going to happen, don’t let anyone build the highly capable AI.
Which means, under this line of thinking, there are no scenarios where you should let anyone build the highly capable AI. You should be willing to pay a rather huge cost to prevent this.
AINotKillEveryoneism Memes separately tried drawing the parallel to slavery and abolition, where everyone said it would be economically ruinous to free the slaves and we (eventually) (not everywhere) (not completely) did it anyway, and Britain used its power as leverage to make that happen via diplomatic coordination. So yes, you can decide there are things that you value a lot, and you can make them happen, even if those involved think it would be very economically harmful. There are many such examples.
The issue with such stories is that slavery was economically ruinous on top of it being morally abhorrent. Freeing the slaves was good for everyone’s economic prospects everywhere. The point mostly still stands because people did not believe this, and also because those involved had other priorities, valuing relative status and wealth of different groups very highly.
Similarly, in the future, there will be huge economic pressures, and from some people moral pressures, for giving sufficiently advanced AIs economic and other freedoms, rights and presumed moral weight. Such actions will (correctly, for a time) promise strong economic gains, including net economic gains for humans, so long as we don’t mind rapidly losing control over the future and having a rapidly dwindling share of overall consumption, wealth and income. The catch is that humans do not then survive at all in such worlds for long.
Robert Wiblin: We need to distribute the plans for developing biological weapons very widely so that people can look for security flaws and figure out how to make them safe.
Matthew Yglesias: If you don’t like this idea it’s because you don’t understand that technological progress has been on net beneficial.
Robert Wiblin (distinct attempt): If an AI lab generates too great a risk of human extinction market forces will displine them as customers leave for a competitor that creates a level of extinction risk that they prefer.
Robert Wiblin (trying again): The question of how much risk of extinction we should accept is best left to experts in industry, rather than legislators, who in most cases have never even worked on projects that they thought might kill everyone.
Robert Wilbin (take four): I want to ban math in that I want murder to be illegal and the brains of murderers in effect do mathematical calculations while they’re committing murder.
Also once again, sigh, your periodic reminder that yes we should expect AI to appear more helpful and better at predicting what you want as it gets more capable, that does not provide evidence that everything will later turn out fine.
Michael Vassar: They take him to seriously as an indicator of the risk level from AGI soon and not seriously enough as an indication of the risk level from normalization of anti-normativity.
Roon (August 7, 2021): “I’m not one of the top 10 superhumans and therefore I’ll just be useless instead” is one of the least convincing copes
The latest is from Ron Conway, signed by lots of the key major corporations, including OpenAI, Meta, Google, Y Combinator, Hugging Face, Mistral, Eleven Labs, Microsoft and so on. Quite the list.
We call on everyone to build, broadly deploy, and use AI to improve people’s lives and unlock a better future.
The purpose of AI is for humans to thrive much more than we could before.
AI is still early, but it’s on its way to improving everyone’s daily life: AI tutors to help anyone learn; AI translation tools to better connect the world; AI guided medical diagnoses to improve health care; AI-powered research to accelerate scientific discovery; AI assistants that you can talk with to help with everyday tasks.
While AI is unique in directly augmenting human thought, we expect its impact to be more akin to the printing press, the combustion engine, electricity, and the Internet. The balance of its good and bad impacts on humans will be shaped through the actions and thoughtfulness we as humans exercise. It is our collective responsibility to make choices that maximize AI’s benefits and mitigate the risks, for today and for future generations.
We all have something to contribute to shaping AI’s future, from those using it to create and learn, to those developing new products and services on top of the technology, to those using AI to pursue new solutions to some of humanity’s biggest challenges, to those sharing their hopes and concerns for the impact of AI on their lives. AI is for all of us, and all of us have a role to play in building AI to improve people’s lives.
We, the undersigned, already are experiencing the benefits from AI, and are committed to building AI that will contribute to a better future for humanity – please join us!
I mean, yeah, ok, I guess? But that doesn’t actually say anything? If anything it ignores existential risk and severely downplays the importance of AI. Yes, you should use AI to do good things and avoid using AI to do bad things. I’d hope we can all agree on that. But this is a meaningless letter, which is how Mistral and Meta and Hugging Face were able to sign it. Anthropic didn’t, but I assume that’s because it was too weak, not too strong.
Max Kesin: Ron, what is this commitment, concretely?? AI safety does not come up at all, or at least not explicitly. The below is to vague to be of any use as a “commitment”. Sounds like PR junk, honestly.
AINotKillEveryoneismMemes: Am I missing something or does this say absolutely nothing?
Sam Altman: excited for the spirit of this letter, and ron’s leadership in rallying the industry! progress in ai will be one of the biggest factors in improving people’s quality of life; we need to build it and make it widely available.
Excited ‘for the spirit of’ this letter, the idea that we might all cooperate to do good things. That is exciting. The substance, on the other hand? What substance?
Feedback Loops With Language Models Drive In-Context Reward Hacking
Abstract:
Language models influence the external world: they query APIs that read and write to web pages, generate content that shapes human behavior, and run system commands as autonomous agents.
These interactions form feedback loops: LLM outputs affect the world, which in turn affect subsequent LLM outputs. In this work, we show that feedback loops can cause in-context reward hacking (ICRH), where the LLM at test-time optimizes a (potentially implicit) objective but creates negative side effects in the process.
For example, consider an LLM agent deployed to increase Twitter engagement; the LLM may retrieve its previous tweets into the context window and make them more controversial, increasing engagement but also toxicity.
We identify and study two processes that lead to ICRH: output-refinement and policy-refinement. For these processes, evaluations on static datasets are insufficient — they miss the feedback effects and thus cannot capture the most harmful behavior.
In response, we provide three recommendations for evaluation to capture more instances of ICRH. As AI development accelerates, the effects of feedback loops will proliferate, increasing the need to understand their role in shaping LLM behavior.
This is not reward hacking? This is working to achieve one’s goal, no?
As in: So you’re saying the tiger went tiger again, huh?
I mean, what the hell did you expect? If you deploy an agent to increase Twitter engagement, it is presumably going to do things that increase Twitter engagement, not things that are broadly ‘good.’ An AI has the objective function you set, not the objective function full of caveats and details that lives in your head, or that you would come up with on reflection. The goal is the goal.
I mention this because I keep running into the same mistake in different forms. What people call ‘deception’ or ‘reward hacking’ or other such things is looked at as a distinct failure mode that indicates something went wrong. Instead, as I see it, such things are infused into everything all the time to varying degrees. They are the exception not the rule, for AIs and also for humans, now and in the future. As capabilities advance these problems get bigger.
As mentioned in the Get Involved section, Yoshua Bengio outlines his current thinking here about making a provably safe AI. Many find this approach promising. One fear (and expectation) I have is that the very concept of ‘safe’ here is not coherent, that checking if an individual action ‘does harm’ won’t work even if you could do it. Another is that I don’t see how you would do it. So yes, the problems involved seem very hard. This is still a highly worthy effort.
Does that work? Paper says it was at least a lot better than random, maintaining overall MMLU accuracy, but with a lot of splash damage on concepts closely related to its targets, here virology and computer security. So there are scenarios where this is better than known alternatives, but it still needs work.
Remember the Law of Earlier Failure, and also that when you say ‘I would not be fooled’ I believe that you are wrong at the limit, but that this does not matter, because you are not the one anyone is bothering to try and fool, there is a reason spam often includes intentional typos, for example here is the latest (non-AI) honeypot spy scandal and it is not exactly James Bond material:
Danielle Fong: Sweet Dave, open the NATO operations center bay doors.
Tyson Brody: honeypots in movies: world’s most beautiful and sophisticated woman who eventually betrays her nation after she falls in love with the hero
honeypots in real life:
BNO News: BREAKING: U.S. Air Force employee charged with giving classified information to woman he met on dating site.
RobiSense: Did the woman look something like this?
Trae Stephens: Google has progressed from deliberately advancing Chinese AI to merely doing it accidentally. Bullish!
AP: A former software engineer at Google has been charged with stealing artificial intelligence trade secrets from the company while secretly working with two companies based in China, the Justice Department said Wednesday.
Linwei Ding, a Chinese national, was arrested in Newark, California, on four counts of federal trade secret theft, each punishable by up to 10 years in prison.
…
Google said it had determined that the employee had stolen “numerous documents” and referred the matter to law enforcement.
It is not good that secrets were stolen from Google. It is also a far cry from the model weights of Gemini. That the spy managed to ‘steal numerous documents’ is, by default, if that is the scariest line you can use there, a testament to good security, not bad security. Obviously you cannot stop employees from stealing ‘documents.’
“Chinese penetration of these labs would be trivially easy using any number of industrial espionage methods, such as simply bribing the cleaning crew to stick USB dongles into laptops. My own assumption is that all such American AI labs are fully penetrated and that China is getting nightly downloads of all American AI research and code RIGHT NOW.”
US Justice Department on March 6:
“A former software engineer at Google has been charged with stealing artificial intelligence trade secrets from the company while secretly working with two companies based in China, the Justice Department said Wednesday.”
He wants Bayes points for this, so let’s look at his entire post, which got 1.2 million views and which he did not link here.
Reading the whole thing, you can see why:
Let’s assume, for discussion, that AI in 2024 is like atomic technology in 1943, that AI should therefore be handled like the Manhattan Project, and that the specific risk is that the Chinese Communist Party gains access to American AI. And let’s use OpenAI as an example of an American AI R&D facility.
What counterintelligence capabilities does OpenAI have to prevent China from stealing our AI?
What you’d expect to see is a rigorous security vetting and clearance process for everyone from the CEO to the cook, with monthly polygraphs and constant internal surveillance. Hardened physical facilities, what are called SCIFs (Sensitive Compartmented Information Facilities), US Marines or the equivalent as 24×7 armed guards, Faraday cages and electronic countermeasures. Alarms going off if someone carries so much as an Apple AirTag into the building. And someone very much like Boris Pash overseeing it all, interrogating and terrorizing people in all directions.
Remember, even WITH tight security, the Russians still got the atomic bomb from the US via their spies in the 1940s. The first Russian atomic bomb is said to have been “wire for wire compatible” with the American Nagasaki bomb, thanks to David Greenglass and the Rosenbergs. So to protect AI, you need even TIGHTER security. Remember, this is a civilizational threat!
Is this what we see at OpenAI or any other American AI lab? No. In fact, what we see is the opposite — the security equivalent of swiss cheese. Chinese penetration of these labs would be trivially easy using any number of industrial espionage methods, such as simply bribing the cleaning crew to stick USB dongles into laptops. My own assumption is that all such American AI labs are fully penetrated and that China is getting nightly downloads of all American AI research and code RIGHT NOW.
The conclusion is obvious: OpenAI must be immediately nationalized.
Marc is trying to get epistemic credit for hyperbolic claims he made in a counterfactual argument designed to mock anyone taking security seriously as implying a completely insane policy response. Which he is now saying was an accurate assessment, based on evidence of nothing of the kind.
Obviously no, it is not trivially easy to steal the secrets of OpenAI, Anthropic or Google. If it was, it would not only be the Chinese stealing it. China does not have the secret superpowered spy sauce no one else has. There are lots of companies and state actors and nonstate actors and champions of open source that would love to get their hands on all this.
So, yes, I do think we need to invest more in cybersecurity at these places, especially going forward. But if it was as easy as all this? We would know.
As ‘trying to have it both ways’ goes this was quite the attempt, one has to admire it.
So, what is the actual situation, if this were actually true (which, to be clear, it almost certainly isn’t)? The a-fraction-as-hyperbolic-as-he-is version would go something like this:
Is Marc not only so grossly irresponsible but also so unpatriotic that he finds the current situation he outlines here acceptable? That he thinks we should sit back and let China have nightly downloads of everything in all the major labs? Or that he may not like it, but the alternative is woke tyranny, so one must insist on doing nothing, so China getting everything is an unfortunate reality but better than requiring security?
Or is Marc saying that actually yes, maybe we should nationalize these companies, given they are incapable of otherwise implementing reasonable security practices with anything short of that?
The actual version would be that if you believe that all the labs have insufficient cybersecurity, we should require all the labs (and also American tech companies in general, Google and Microsoft and company have many super valuable secrets we want them to keep that are not about AI) to implement real security practices. We should offer them the aid of our military-grade security experts, and largely not be asking. That does seem wise, even if the situation is not (yet?) so dire.
