Author name: Paul Patrick

new-study-accuses-lm-arena-of-gaming-its-popular-ai-benchmark

New study accuses LM Arena of gaming its popular AI benchmark

AI, Artificial Intelligence, Google, lm arena, Meta, openai / /

This study also calls out LM Arena for what appears to be much greater promotion of private models like Gemini, ChatGPT, and Claude. Developers collect data on model interactions from the Chatbot Arena API, but teams focusing on open models consistently get the short end of the stick.

The researchers point out that certain models appear in arena faceoffs much more often, with Google and OpenAI together accounting for over 34 percent of collected model data. Firms like xAI, Meta, and Amazon are also disproportionately represented in the arena. Therefore, those firms get more vibemarking data compared to the makers of open models.

More models, more evals

The study authors have a list of suggestions to make LM Arena more fair. Several of the paper’s recommendations are aimed at correcting the imbalance of privately tested commercial models, for example, by limiting the number of models a group can add and retract before releasing one. The study also suggests showing all model results, even if they aren’t final.

However, the site’s operators take issue with some of the paper’s methodology and conclusions. LM Arena points out that the pre-release testing features have not been kept secret, with a March 2024 blog post featuring a brief explanation of the system. They also contend that model creators don’t technically choose the version that is shown. Instead, the site simply doesn’t show non-public versions for simplicity’s sake. When a developer releases the final version, that’s what LM Arena adds to the leaderboard.

Proprietary models get disproportionate attention in the Chatbot Arena, the study says.

Credit: Shivalika Singh et al.

Proprietary models get disproportionate attention in the Chatbot Arena, the study says. Credit: Shivalika Singh et al.

One place the two sides may find alignment is on the question of unequal matchups. The study authors call for fair sampling, which will ensure open models appear in Chatbot Arena at a rate similar to the likes of Gemini and ChatGPT. LM Arena has suggested it will work to make the sampling algorithm more varied so you don’t always get the big commercial models. That would send more eval data to small players, giving them the chance to improve and challenge the big commercial models.

LM Arena recently announced it was forming a corporate entity to continue its work. With money on the table, the operators need to ensure Chatbot Arena continues figuring into the development of popular models. However, it’s unclear whether this is an objectively better way to evaluate chatbots versus academic tests. As people vote on vibes, there’s a real possibility we are pushing models to adopt sycophantic tendencies. This may have helped nudge ChatGPT into suck-up territory in recent weeks, a move that OpenAI has hastily reverted after widespread anger.

New study accuses LM Arena of gaming its popular AI benchmark Read More »

sen.-susan-collins-blasts-trump-for-cuts-to-scientific-research

Sen. Susan Collins blasts Trump for cuts to scientific research

public health, Science, syndication / /

This article originally appeared on Inside Climate News, a nonprofit, non-partisan news organization that covers climate, energy, and the environment. Sign up for their newsletter here.

Sen. Susan Collins (R-Maine) kicked off a Wednesday hearing criticizing ​​the Trump administration for cutting science funding, firing federal scientists, and triggering policy uncertainties that she said threaten to undermine the foundation for America’s global leadership.

Collins, chair of the Senate Appropriations Committee, said the administration’s abrupt cancellation of grants and laying off scientists has little or no justification. “These actions put our leadership in biomedical innovation at real risk and must be reversed,” she said.

Her warning came as American University’s Institute for Macroeconomic & Policy Analysis published a study Wednesday showing how major cuts to federal funding for scientific research could cause economic damage equivalent to a major recession.

In the first 100 days of Trump 2.0, the administration has fired 1,300 employees from the National Institutes of Health, the largest funder of biomedical research in the world, and canceled more than $2 billion in federal research grants.

Earlier this week, the administration dismissed all the scientists and other authors working on the next authoritative look at how climate change is affecting the US.

In one such cutback, the Trump administration stripped almost $4 million in federal funding from Princeton’s climate research department as it determined that Princeton’s work did not align with the objectives of the National Oceanic and Atmospheric Administration (NOAA).

The White House said Princeton’s research on topics including sea level rise, coastal flooding, and global warming promoted “exaggerated and implausible climate threats,” according to a US Department of Commerce press release earlier this month explaining the funding cuts.

The White House is expected to propose additional reductions in discretionary spending as part of the annual budget process. Federal agencies like the NIH and the National Science Foundation are among the few funding basic and applied scientific research.

The study, from a group of American University economists, is among the first to run preliminary macroeconomic estimates of the cost of the Department of Government Efficiency and the Trump administration’s cuts to public spending on science.

Sen. Susan Collins blasts Trump for cuts to scientific research Read More »

fortnite-will-return-to-ios-as-court-slams-apple’s-“interference“-and-”cover-up“

Fortnite will return to iOS as court slams Apple’s “interference“ and ”cover-up“

Apple, Apple v Epic, fortnite, gaming, Policy / /

In a statement provided to Ars Technica, an Apple spokesperson said, “We strongly disagree with the decision. We will comply with the court’s order and we will appeal.”

An Epic return

With the new court order in place, Epic says it will once again submit a version of Fortnite to the iOS App Store in the US in the next week or so. That new version will offer players the option to use standard Apple App Store payments or its own, cheaper “Epic Direct Payment” system to purchase in-game currency and items.

That would mirror the system that was briefly in place for iOS players in August 2020, when Epic added alternate payment options to iOS Fortnite in intentional violation of what were then Apple’s store policies. Apple removed Fortnite from the iOS App Store hours later, setting off a legal battle that seems to finally be reaching its conclusion.

For those few hours when Epic Direct Payments were available on iOS Fortnite in 2020, Sweeney said that about 50 percent of customers “decided to give Epic a shot,” going through an additional step to register and pay through an Epic account on a web page outside the app itself (and saving 20 percent on their purchase in the process). The other roughly 50 percent of customers decided to pay a higher price in exchange for the convenience of paying directly in the app through the iOS account they already had set up, Sweeney said. “Consumers were making the choice… and it was a wonderful thing to see,” he said.

Speaking to the press Wednesday night, Sweeney said the new court order was a “huge victory for developers” looking to offer their own payment service alongside Apple’s on iOS devices. “This is what we’ve wanted all along,” he said. “We think that this achieves the goal that we’ve been aiming for in the US, while there are still some challenges elsewhere in the world.”

While Sweeney said the specific iOS developer account Epic used to publish Fortnite in 2020 is still banned, he added that the company has several other developer accounts that could be used for the new submission, including one it has used to support Unreal Engine on Apple devices. And while Sweeney allowed that Apple could still “arbitrarily reject Epic from the App Store despite Epic following all the rules,” he added that, in light of this latest court ruling, Apple would now “have to deal with various consequences of that if they did.”

Fortnite will return to iOS as court slams Apple’s “interference“ and ”cover-up“ Read More »

research-roundup:-tattooed-tardigrades-and-splash-free-urinals

Research roundup: Tattooed tardigrades and splash-free urinals

Archaeology, cephalopods, holograms, research roundup, Science, sharks, tardigrades, urinals / /

April is the cruelest month

Also: The first live footage of a colossal baby squid; digitally unfolding an early medieval manuscript.

Credit: Schmidt Ocean Institute

It’s a regrettable reality that there is never time to cover all the interesting scientific stories we come across each month. In the past, we’ve featured year-end roundups of cool science stories we (almost) missed. This year, we’re experimenting with a monthly collection. April’s list includes new research on tattooed tardigrades, the first live image of a colossal baby squid, the digital unfolding of a recently discovered Merlin manuscript, and an ancient Roman gladiator whose skeleton shows signs of being gnawed by a lion.

Gladiator vs. lion?

Puncture injuries by large felid scavenging

Puncture injuries by large felid scavenging. Credit: Thompson et al., 2025/PLOS One/CC-BY 4.0

Popular depictions of Roman gladiators in combat invariably include battling not just human adversaries but wild animals. We know from surviving texts, imagery, and artifacts that such battles likely took place. But hard physical evidence is much more limited. Archaeologists have now found the first direct osteological evidence: the skeleton of a Roman gladiator who encountered a wild animal in the arena, most likely a lion, based on bite marks evident on the pelvic bone, according to a paper published in the journal PLoS ONE.

The skeleton in question was that of a young man, age 26 to 35, buried between 200–300 CE near what is now York, England, formerly the Roman city of Eboracum. It’s one of several such skeletons, mostly young men whose remains showed signs of trauma—hence the suggestion that it could be a gladiator burial site. “We used a method called structured light scanning [to study the skeleton],” co-author Tim Thompson of Maynooth University told Ars. “It’s a method of creating a 3D model using grids of light. It’s not like X-ray or CT, in that it only records the surface (not internal) features, but since it uses light and not X-rays etc, it is much safer, cheaper, and more portable. We have published a fair bit on this and shown its use in both archaeological and forensic contexts.”

The team compared the pelvic lesions found on the subject skeleton with bite marks from modern animal specimens and concluded that the young man had been bitten by a “large feline species,” most likely a lion scavenging on the body around the time of death. The young man was decapitated after death for unknown reasons, although this was a ritualistic practice for some people during the Roman period. While the evidence is technically circumstantial, “we are confident with our conclusions,” said Thompson. “We’ve adopted a multidisciplinary approach to address this issue and have drawn on methods from different subjects, too. Our use of contemporary comparison zoological material is really what gives us the confidence.”

PLoS ONE, 2025. DOI: 10.1371/journal.pone.0319847  (About DOIs).

Tattooed tardigrades

False-colored SEM image of the tardigrade after rehydration and fixation, with a magnified inset of the blue-boxed area.

False-colored SEM image of the tardigrade after rehydration and fixation. Credit: American Chemical Society

Tardigrades (aka “water bears”) are micro-animals that can survive in the harshest conditions: extreme pressure, extreme temperature, radiation, dehydration, starvation—even exposure to the vacuum of outer space. Scientists have exploited the robustness of these creatures to demonstrate a new ice lithography technique that can be used to essentially tattoo patterns at the nanoscale on living creatures. They described their method in a paper published in the journal Nano Letters.

Creating precision patterns on living organisms is challenging because the latter require very specific conditions in order to thrive, while fabrication techniques typically require harsh environments—the use of corrosive chemicals, for instance, vacuum conditions, or high radiation. So researchers at Westlake University tested their ice lithography on tardigrades in their dehydrated state (cryptobiosis). Once cooled, the tardigrades were coated with vaporized anisole, creating an ice layer. The team used an electron beam to etch patterns in that layer. Once the creatures were warmed back up, the parts of the ice layer that had not been exposed to the beam sublimated away, and the pattern was preserved on the tardigrade’s surface, even after the creatures were rehydrated.

Granted, only about 40 percent of the tardigrade test subjects survived the full procedure, but further improvements could improve that rate significantly. Once the technique is fully developed, it could enable the fabrication of nanoscale patterns for marking living organisms, such as tracking single cells as they develop or for the creation of sophisticated biosensors.

Nano Letters, 2025. DOI: 10.1021/acs.nanolett.5c00378  (About DOIs).

Holograms that can be grabbed

A 3D car is grabbed and rotated by a user.

A 3D car is grabbed and rotated by a user. Credit: Iñigo Ezcurdia

A volumetric display consists of scattering surfaces distributed throughout the same 3D space occupied by the resulting 3D image. Volumetric images can be viewed from any angle, as they seem to float in the air, but no existing commercial prototypes let the user directly interact with the holograms—until now. There is a new kind of volumetric display called FlexiVol that allows people to interact directly with 3D graphics displayed in mid-air. Elodie Bouzbib of the Public University of Navarra presented the research at the CHI conference on Human Factors in Computing Systems in Japan this month.

The key lies in a fast oscillating sheet known as a diffuser, onto which synchronous images are projected at high speed (2,880 images per second) and at different heights; human persistence of vision ensures that these images are perceived as true 3D objects. But the diffusers are usually made of rigid materials and hence pose a safety hazard should a user try to reach through and interact directly with the hologram; safety domes are usually employed because of this.

FlexiVol replaces the rigid diffuser with elastic bands that will not permanently deform or twist, distorting the 3D display, and has a different resonant frequency from the volumetric system. The team was inspired by the taxonomy of gestures used with 2D elastic displays and touch screens: swiping, for instance, or pinching in and out to make an image larger or smaller. They tested FlexiVol with a selection of users performing three sample tasks showcasing the ability to manipulate the 3D graphics, such as “grasping a cube between the thumb and index finger to rotate it, or simulating walking legs on a surface using the index and ring fingers,” said Bouzbib.

Look ma, no spashback!

A high-speed video depicting the tests used to measure the critical angle. Credit: Thurairajah et al., 2025

Men, are you tired of urine splashback when you use the loo? Scientists at the University of Waterloo have developed the optimal design for a splash-free urinal, dubbed the Nautilus (aka the “Nauti-loo”). We first covered this unusual research back in 2022, when the researchers presented preliminary results at a fluid dynamics conference. Their final findings have now formally appeared in a paper published in PNAS Nexus.

Per the authors, the key to optimal splash-free urinal design is the angle at which the pee stream strikes the porcelain surface; get a small enough angle, and there won’t be any splashback. Instead, you get a smooth flow across the surface, preventing droplets from flying out. (And yes, there is a critical threshold at which the urine stream switches from splashing to flowing smoothly, because phase transitions are everywhere—even in our public restrooms.) It turns out that dogs have already figured out the optimal angle as they lift their legs to pee, and when the team modeled this on a computer, they pegged the optimal angle for humans at 30 degrees.

The next step was to figure out a design that would offer that optimal urine stream angle for men across a wide range of heights. Instead of the usual shallow box shaped like a rectangle, they landed on the curved structure of the nautilus shell. They conducted simulated urine stream experiments with the prototypes, et voila! They didn’t observe a single droplet splashing back. By comparison, the other urinal designs produced as much as 50 times more splashback. The team did come up with a second design with the same optimal angle, dubbed the Cornucopia, but unlike the Nautilus, it does not fit a range of heights, limiting its usefulness.

PNAS Nexus, 2025. DOI: 10.1093/pnasnexus/pgaf087  (About DOIs).

Colossal baby squid

First confirmed live observation of the colossal squid in its natural habitat. Credit: Schmidt Ocean Institute

In 1925, scientists first described the colossal squid in a scientific paper, based on the discovery of arm fragments in the belly of a sperm whale. This species of squid is especially elusive because it prefers to stay in the deep ocean, although occasionally full-grown colossal squid have been found caught in trawl nets, for instance. One hundred years after its discovery, the colossal squid has now been filmed alive in its deep-ocean home environment for the first time by a team aboard Schmidt Ocean Institute’s R/V Falkor (too) in waters off the South Sandwich Islands.

Colossal squid can grow up to 23 feet long and weigh as much as 1,100 pounds and have distinctive hooks on the middle of their eight arms. Juvenile squid have transparent bodies. It was a baby squid just 30 centimeters long that the team captured on video at a depth of 1,968 feet (600 meters) during a 35-day expedition searching for new marine life; a remote submersible dubbed SuBastian took the footage. The scientists hope to eventually be able to capture an adult colossal squid on camera. The team also filmed the first confirmed living footage of a similar cephalopod species, the glacial glass squid, spotted in the Bellingshausen Sea near Antarctica in January.

Digitally unfolding a Merlin manuscript

Virtual opening of CUL’s Vanneck Merlin fragment.

In 2019, conservationists at Cambridge University discovered a fragment of an Arthurian medieval manuscript that had been repurposed as the cover of a land register document. Written between 1275 and 1315 CE, it was far too fragile to manually unfold, but the university library’s Cultural Heritage Imaging Laboratory has succeeded in digitally unfolding the fragment so that the text can be read for the first time, while keeping the original artifact intact as a testament to archival practices in 16th-century England. Their method could be used to noninvasively study fragile manuscript fragments held in other collections.

The team used a combination of CT scanning, multispectral imaging, and 3D modeling, as well as an array of mirrors, prisms, magnets, and other tools to photograph each section of the fragment. In this way they were able to reconstruct and virtually unfold the manuscript, revealing the text. Scholars had originally thought it was a text relating to Sir Gawain in Arthurian lore, but it turned out to be part of a French language sequel to the King Arthur legend called the Suite Vulgate du Merlin. There are only 40 known surviving manuscripts of this work. One section concerns Gawain’s victory over Saxon kings at the Battle of Cambenic; the other is a story of Merlin appearing in Arthur’s court disguised as a harpist on the Feast of the Assumption.

Photo of Jennifer Ouellette

Jennifer is a senior writer at Ars Technica with a particular focus on where science meets culture, covering everything from physics and related interdisciplinary topics to her favorite films and TV series. Jennifer lives in Baltimore with her spouse, physicist Sean M. Carroll, and their two cats, Ariel and Caliban.

Research roundup: Tattooed tardigrades and splash-free urinals Read More »

windows-rdp-lets-you-log-in-using-revoked-passwords-microsoft-is-ok-with-that.

Windows RDP lets you log in using revoked passwords. Microsoft is OK with that.

Biz & IT, passwords, Remote Desktop Protocol, Security, Windows / /

The ability to use a revoked password to log in through RDP occurs when a Windows machine that’s signed in with a Microsoft or Azure account is configured to enable remote desktop access. In that case, users can log in over RDP with a dedicated password that’s validated against a locally stored credential. Alternatively, users can log in using the credentials for the online account that was used to sign in to the machine.

A screenshot of an RDP configuration window showing a Microsoft account (for Hotmail) has remote access.

Even after users change their account password, however, it remains valid for RDP logins indefinitely. In some cases, Wade reported, multiple older passwords will work while newer ones won’t. The result: persistent RDP access that bypasses cloud verification, multifactor authentication, and Conditional Access policies.

Wade and another expert in Windows security said that the little-known behavior could prove costly in scenarios where a Microsoft or Azure account has been compromised, for instance when the passwords for them have been publicly leaked. In such an event, the first course of action is to change the password to prevent an adversary from using it to access sensitive resources. While the password change prevents the adversary from logging in to the Microsoft or Azure account, the old password will give an adversary access to the user’s machine through RDP indefinitely.

“This creates a silent, remote backdoor into any system where the password was ever cached,” Wade wrote in his report. “Even if the attacker never had access to that system, Windows will still trust the password.”

Will Dormann, a senior vulnerability analyst at security firm Analygence, agreed.

“It doesn’t make sense from a security perspective,” he wrote in an online interview. “If I’m a sysadmin, I’d expect that the moment I change the password of an account, then that account’s old credentials cannot be used anywhere. But this is not the case.”

Credential caching is a problem

The mechanism that makes all of this possible is credential caching on the hard drive of the local machine. The first time a user logs in using Microsoft or Azure account credentials, RDP will confirm the password’s validity online. Windows then stores the credential in a cryptographically secured format on the local machine. From then on, Windows will validate any password entered during an RDP login by comparing it against the locally stored credential, with no online lookup. With that, the revoked password will still give remote access through RDP.

Windows RDP lets you log in using revoked passwords. Microsoft is OK with that. Read More »

dating-roundup-#4:-an-app-for-that

Dating Roundup #4: An App for That

Dating / /

Previously: #1, #2, #3.

As time goes by, the fundamental things in life are still the same, and yet they change quite a lot with the times. But they don’t yet change so fast that the previous three editions of this are invalid. AI isn’t transforming the world that quickly, not yet.

In the meantime, there’s always more to say, and I both find it enjoyable and am hopeful that it might help some of you out there, so here we are once again.

Meanwhile, I am sorry to report that many of you are still sihngle.

I spent sufficiently long between updates and got sufficient material that this one is being split into conceptual sections.

This one will focus in particular on the awfulness that are dating apps, and directly related considerations.

  1. You’re Single Because Dating Apps (Still) Suck.

  2. You’re Single Because Your Friends Are Insufficiently Supportive.

  3. You’re Single Because You Do Not Seek a Mentor for Basic App Skills.

  4. You’re Single Because Other People Lack Very Basic Skills.

  5. You’re Single Because Your Opens Aren’t Effortful.

  6. You’re Single Because Dating Apps Are Out of Balance.

  7. You’re Single Because Look at the Odds.

  8. You’re Single Because You Won’t Even Pay For Super Likes.

  9. You’re Single Because of Dating App Game Theory.

  10. You’re Single Because DateMe Docs Don’t Scale.

  11. You’re Single For Lack of Very Basic Dating Strategy.

  12. You’re Single Because You Have an Android Phone.

  13. You’re Single and I’m Here to Help.

Hinge puts a limit on number of open conversations, Shoshana Weissmann praises the incentive design because it pushes you to unmatch rather than ghost, or unmatch if you are being ghosted. This makes sense. I wonder what the right limit would be? A reader informs me you can also archive the matches, which will dim the impact here.

Bumble apologized for ads it ran saying that celibacy wasn’t the answer. They did penance by donating to the Domestic Violence Hotline and similar organizations. I agree this might not have been the most diplomatic approach. Yet the details of the reaction were bizarre to me. They were accused of ‘undermining daters’ freedom of choice?’ What?

If they won’t say it, I will: With notably rare exceptions, if you are an adult and are not asexual, celibacy is not the answer.

Match says number of paying users on Tinder has dropped for six straight quarters.

The dating app business is… not going well.

It could be high time for a rival that does it better, but one must still solve the matching and coordination problems, and the desire of many of the most desirable users to sit there and swipe. People do not want to pay for good matchmaking, either with money or with actual effort and attention, in this or in other fronts.

You might be having trouble on dating sites, but do not underestimate the value of ‘not being a lunatic’ if you can get past the first few seconds of filter. Or how many people seem determined to fail in those few seconds.

Shoshana Weissmann: Opened my Hinge likes after the hike. Here’s a guy who photoshopped a cartoon orgy around him in the bath.

Yesterday an ENM couple tried to match me and the guy was in a Pikachu costume. I literally say monogamy twice on my profile because nobody respects me about it. Pikachu I don’t choose you.

Rich Braun: I’m a couple decades older than you, but when I was single and on the apps a few years ago I was amazed how many out-of-my-league women were willing to give me a shot.

Your tweets have helped me understand that simply not being a lunatic put me in the top tier.

Rob Henderson warns that at least one popular dating app is known to use ‘seeding’ meaning when a man first signs up you match with bot profiles of hot women who then ghost you, to get you started. That seems like the kind of thing the FTC should be investigating, if it is true, and indeed in the past they have done some investigation, but such claims have mostly not been substantiated.

It also raises the question, would that work? On the one hand, yes, Hot Women Near You and all that. On the other hand, being ghosted sucks even worse than not matching. The hope is presumably that you think it was something you said so it’s your fault.

Business Insider’s alternative suggestion is a government app like in Tokyo, and oh no. Except perhaps being the government solves some trust issues, allowing the site to verify information and thus be a better product?

In Tokyo, the city government is even releasing its own dating app, part of a campaign called Tokyo Futari Story (“futari” means couple). The service, which the city has budgeted several million dollars to develop and promote, will require users to verify their income, prove they’re unmarried, sit for an interview with the app’s staff, and sign a statement saying they’re intent on getting married.

Anyone who would suggest our government should be writing such software really needs to talk to someone such as Patrick McKenzie first, and learn some of the many reasons why we will absolutely be unwilling to attempt to do that. But also the talk about ‘equality’ and ‘eugenics’ should help explain why no one would dare touch it, even in worlds where they would ever try to make any software at all.

Update: Shreeda launches the latest attempt, ‘Offline,’ to reinvent the upside of the old OkCupid. Everyone says they want it, but no one so far has overcome the allure of the easy swipe. Info is here, in particular here. I continue to presume that it is relatively easy to build something ten times better if you can solve the cold start and ‘easier to just swipe’ issues to get enough scale, but that no one has a good idea how to do that part.

One danger of dating apps is that it raises the false implication that one should not date outside of the apps. Of course one should do so when possible, and use the apps only as supplements and last resorts.

Mark Travers (2021): 68 percent of romantic relationships start from friendship…romances where partners start as friends are more likely to be the rule than the exception…On average, friends-first partners were friends for almost 2 years before becoming romantic partners.

Tim Carney: This is important, and I wonder if dating apps have convinced people that one ought not date friends.

There is a list of preferences in the linked post on ways to find relationships, which corresponds to how social and friend-based techniques are, with blind dates and dating apps at the very bottom. Dating apps are a place where you go explicitly to find dates, and some young’uns get convinced that romantic interests should ONLY be pursued in such settings.

Charlie Gowans-Eglinton takes a decade on the apps, becomes disillusioned by the horror shows, sees them now as ‘a way to seem proactive’ rather than a solution to finding someone, as all the men are so terrible and the pickings so slim, due to adverse selection, the need for a thousand bad dates. I still don’t understand how the numbers could fail so utterly to add up? Yes, we hear so many stories like the next two sections about lack of basic skills and generally being terrible, but… if all the good guys were really snapped up quickly then we’d observe a very different world.

Yes, there are obvious downsides.

Jonathan Deer: Would rather not get a gift then someone think I’m down bad enough to get me a tinder gift card #disrespectful #buttless

Pierre Pumpernickel: If somebody gave me this for Christmas I would drive to an Applebees and kill myself in the parking lot.

Alison: That’s beyond calling you bitchless. That’s calling you bitchless, broke, and so bitchless that you need to pay to win.

Brits With Knees: if anyone wants to get me this lmk

This is actually great and super thoughtful gift.

It is something the person needs, but would never ever buy for themselves. This person clearly would never, ever pay, on principle. But I have little doubt the value is there, if only to find out if the value is there.

Brooke suggests that good dating app game is easy to identify when you care about it, so find someone who does to help you learn what to work on, and then try even a little.

Aaron Bergman: I took one good picture with a good haircut, and now I have probably three times the number of Hinge matches. What the heck?

Brooke Bowman: It’s probably worth scrolling through a dating app from the account of a friend who is interested in your gender.

The rarer, nicer photos of men stand out so much.

The vast majority are blurry selfies or just bad photos in general, and they all kind of blend together.

Ram Vasuthevan: Like, look; are they taken professionally?

Brooke Bowman: Some are, yes, but it doesn’t take that much to stand out from the crowd! A nice haircut, decent clothes, and being in focus can help. Ask a friend to take some, or take selfies in different parts of the house to see what the lighting looks like.

Seems like good advice if you’re not getting good initial conversions.

Yes, you should also be paying up, every boost helps, but don’t forget the basics.

It is a weird one, because in a matching market this should go the other way?

Shoshana Weissmann: NY dating apps are an insane place.

Rob Bernard: “I’m a man with physical man needs..” This is either an alien trying to convince us it is human, or it’s Matt Berry’s Laszlo on What We Do in the Shadows.

The Gentleman Sausage: I enjoy long walks in the beach while listening to human music.

Shoshana Weissmann: You’re joking but I saw this yesterday. Online dating is hell.

Coach Crash: You should see what is out there at my age. I understand if you’re a young widow but don’t include pics that look like they’re from the police investigation into his death.

GirlKW77: lol. I just watched a segment on my local news station that was interviewing people who were knowingly having “intimate” relationships with online AI dating bots. Kid you not.

In an interview with a guy who goes on a lot of dates, the most interesting part is he gets there via effort texting and being selective, not via going scattershot.

He only swipes right about three times a day, and ends up on four dates a week, an absurd conversion rate, so yes it can be done. Obviously there’s the mystery of how you get that many right swaps in the first place, but past that this does match my experiences too – if you put in the effort, conversation rates to first dates can actually be remarkably high. If you don’t have a good text game, well, you’re reading giant walls of text or you wouldn’t be here, text should be your friend.

He also has a 70% rate on getting second dates. That’s actually way too high if you’re meeting multiple new people each week, you need to cut your losses when it isn’t a great match. A 70% rate only makes sense if you have ‘first date scarcity,’ if a promising first date is time consuming to get you want to not give up so easily.

In particular, this chart and similar statistics have cool bonus implications.

  1. Men like 53% of the profiles they view.

  2. Yet even so, women only match 36% of the time when they like a man.

  3. Men only match 2% of the time when they like a woman.

  4. Women like 5% of profiles they view according to that chart, but 14% according to Zippia. That’s a huge gap, although the main points hold either way.

A lot of the explanation is that ~75% of Tinder users are men, which is actually a better ratio than many other apps. So even if you never showed a woman a profile of a man who would actively say no to her, about half the male likes never even get seen.

There are also selection effects. The more people want to match with you, the less of them you want to match with in return, because you know you have options. Tinder, like all such apps, does do the first order obvious thing of putting those who already matched with you early into your queue, in addition to attempting to otherwise make predictions including using things like Elo. Despite that, the majority of women’s swipes still fail to get them matched. Which tells me quite a lot of women want to be swiping well beyond the set of people who pre-matched with them.

The symmetry here is remarkable, the way this is worded doesn’t require it at all and this rules out theories that there are guys who sleep with lots of women per year on the first date as they would have skewed the numbers. 5% is rather grim given how many young people start that year single.

Similarly:

Nuance Enjoyer: Moreover, 3/4ths of the gap in the Pew survey was driven by disparities in cohabitation and marriage, leaving little room for the popular ‘de facto polygynous soft harems’ explanation.

Over the course of an entire year, a majority of 18-34 year olds have sex with exactly one opposite-sex person, and a majority of the rest don’t have sex at all.

Unfortunately, sexlessness is going up over time for people in their 20s, by a substantial amount, odd that these two graphs look different.

Also, the linked Vox article reminded me of the ‘super like’ feature, where your like is visible to them while swiping, you show up faster, and the match is instant if they return interest. They have to be purchased.

Costly signals are great, and so super likes reportedly triple your chances. Of course, you also get the reactions that say super likes are cringe or creepy, how dare you actually express real interest, but the statistics say it works, and I’m guessing that there is positive selection in driving away people who dislike clear communication. You probably also do worse on people who think they’re better than everyone, which plausibly includes some people you’d want quite a lot (since they are sometimes in fact better!) but also isn’t a feature I’d want to seek out.

You don’t have to pay $500/month for Tinder Select and go straight to their inbox. But if you’re not paying for a small number of super likes, that seems like a huge mistake, given you can buy in bulk for $1.50 or get them free with your subscription package.

Yes, $75 per additional match might sound steep, but is it for the ones you want most, in a world where matching is rare? If you’re doing reasonably well, it seems basically impossible for it to be worth being on the app at all, and also not being worth super liking when you see someone you do in fact super like.

I’d apply a similar principle to other paid options, for any app you would already be using on a regular basis. If you’re paying your time, you need to also pay your money. The big advantages of paying are the (often under the hood) ways to increase your chances of success.

Thus consider: The average American dating app user spends 51 minutes a day on the apps? What? Note that this is very different from ‘51 minutes swiping’ which would be fully nuts. Whereas if a lot of that time is spent chatting with or thinking about existing matches, that is a lot less insane.

Still, it’s a lot, and I don’t really believe it. But if you’re spending an hour a day and still running the no-pays, you’re making a very serious mistake.

(If you’re spending that hour mostly chatting with matches, why haven’t you moved to regular texting or other messaging apps with most of them yet?)

The other claim is that the apps are increasingly hard to use without paying. I would respond that the users spend 51 minutes a day on them. If you are spending 51 minutes a day, and yet refuse to pay a modest amount of real money, then the problem is you. Your time is valuable. The claim is this is not ‘equitable’ when money is charged, but seriously, what? Of course the post then goes on to call Singapore’s attempt to help college students date ‘eugenic’ so there you have it. How dare they.

This is a clear laying out of the standard argument which is essentially:

  1. Frictionless dating apps create male haves versus have nots.

  2. The men with little female interest don’t get to date or commit at all.

  3. The men with lots of interest have multiple women interested in them and can always find more, so they see no reason to commit.

  4. Thus, you get a bunch of these situationships, where the man won’t commit.

  5. The men who both can get dates and want to commit get snapped up quickly.

  6. Which means the men mostly say ‘why won’t women date us at all?’ and the women mostly say ‘why won’t the men commit to us?’

  7. Even if you’re not on the apps yourself, the incentives are there anyway.

This is largely a Levels of Friction problem, where the selection process has low marginal costs and starts with superficial attributes, which makes it easier to steer towards going after high-value targets and creating divergence.

A great asset of OKCupid was scale. You answered the questions once, created a profile once, and you could check for lots of matches in detail, as could others. Getting that scale back is the key barrier.

Or you could do it without the scale, since that’s what everyone used to do anyway.

The date-me doc does this as one-to-many even if there aren’t that many, but exposes your info and puts the filtering job on them (even if they don’t have to answer a bunch of specific questions).

What about the date-me survey? They have to answer the questions but you have to design and execute all the filters.

Aella: Reminder I have a survey to date me! I’ve found partners via this method before; if you think we might be compatible, feel free to fill it out <3

Kepe: wow, I haven’t seen a single reply supporting her. I’ve said it before and i’ll say it again, this is absolutely an amazing idea and anyone saying it isn’t is exactly who should be filtered out.

One reply warned about a woman influencer and model who rejected all but 3 of 5,000 boyfriend applications after making them fill out a 15 question form. The headline is misleading, she did find three that were suitable and went on dates with them, even though they didn’t work out.

Is that even so bad? I mean, yes, on average that’s 25,000 questions per date with a model influencer you know you’re into. I say that depends on the questions. And lo and behold, we’ve got them.

  1. What’s your astrological sign?

  2. Number of ex-girlfriends, the number that are ‘crazy’, and exes you still text/talk to (drink ones count)

  3. Do you have kids?

  4. Do you want them?

  5. Are you married/dating someone now?

  6. Do you have a full time job?

  7. If you were picking three adjectives to describe yourself would one of them be douchey?

  8. Do you live with your parents?

  9. Do you own a working car?

  10. Do you have Twitter?

  11. Do you currently have a booty call?

  12. If we lived together would I get the walk in closet?

  13. Is it acceptable to hit on my friends?

  14. Do you like watching Avatar (ATLA)?

  15. Who is the best artist: The Weeknd, Future, Drake or Travis Scott?

These are not exactly the hardest questions.

I presume I know the right answers for nine of them, and so do you: #3, #5, #6, #7, #8, #9, #11, #12 and #13. Mostly that seems entirely fair, especially if the car is a proxy.

That leaves six ‘real’ questions, one of which also has a clear right answer?

Most of the applicants were also based in Texas and had the star sign of Aquarius.

Not a wise filter, in either direction, and expensive with a 92% failure rate, but sure. Five to go.

One of them is the most important question: Do you want kids? We don’t know which answer she wants but you should 100% be asking.

The last two questions are taste questions, you’re allowed two, sure.

Are you supposed to have or not have Twitter? Unknown, I can see it either way.

The final question is about ex-girlfriends, and we can all guess what answers she is looking for here and why this is a reasonable question to ask if you have the leverage.

So why did only 3 of 5,000 applications make it through? That seems like… not a lot, especially given most of them got the astrological sign correct, and if you’re going to fail the gimme questions you can decide not to turn in the survey. This must mean not a lot of men got the gimme questions right, and didn’t understand?

Or alternatively she was looking for at least one very counterintuitive answer.

As in things like: Most dating profiles are insanely boring. Say interesting things.

Similarly, we have other basic principles in the stories below, that anyone should be able to handle: Show up on time. Be able to plan a date. Treat all people like people. Answer questions honestly and be able to handle honest answers. Be actually single and not married. Don’t be weirdly super intense.

That doesn’t mean this is easy, but don’t make life a little tougher than it is.

Perhaps the biggest divide is between three groups:

  1. Those who think that ‘being a decent normal person’ works fine.

  2. Those who feel like they’re being decent normal people and failing.

  3. Those who the first group points to as no being decent normal people.

Jeff: I need some stories about bad dates to cleanse my palate.

Shoshana Weissman: I am messaging a man right now who showed a lot of interest initially, and now he is not responding—acting standoffish. He asked why I don’t drink, out of curiosity, and I explained that it is because I have several autoimmune diseases. I manage fine, but my body does not like alcohol. He basically replied that that was too much information. So, don’t ask? When is the answer to that question ever simple?

Kevin Baum: Tip for single men: You can do remarkably well with online dating if you are remotely normal instead of being like this man.

Christopher Eichhorn: Absolutely correct—I got many replies (and a wife!) on dating apps and was off the market within a couple of weeks of joining. I am not suave or stylish. I do not have “game,” and I have never read pickup artist material. I simply treated women like normal human beings.

T.K.: I feel like any single men just need to follow the examples of good dating experiences and avoid that behavior. Show up on time, be able to plan a date, be able to handle an honest answer to a question you ask, and do not be “kind of” divorced. All simple things.

She also offers us this message log of a man reporting he deserves the flames of hell.

I definitely buy that lots of men, especially on dating apps but also everywhere, are doing quite a lot of shooting themselves in the foot in a ‘why can’t you just be normal and a decent human being’ ways. If you can avoid doing that, it’s a huge edge.

The problem is that this is often a negative selection game, with complex rules. Whatever the basic principle is that you missed, that’s the one that gets put into a story like this, and it’s often not easy to learn which one you are messing up. Debugging is hard, and you don’t have good tools.

And yes, you also need opportunity to ‘treat women like human beings’ in the first place, or the ability to do so won’t do anyone any good.

Here’s yet another viral thread saying Android phones give many women ‘slight ick.’

Blaine Anderson (dating couch and matchmaker, responding to another 1m+ view claim that’s even stronger): Dating fun fact: Android gives many women slight ick!

If a woman likes you, Android won’t make her STOP liking you.

But iOS girlies 100% feel pangs of disappointment when your first text is green.

For single men using Android:

This is NOT a recommendation you switch to iOS.

This is just an informed opinion from someone who speaks with single women for a living 😌

Especially if you use Android, and find this thread moronic…

Stick to Android to distance yourself from judgmental iOS women 😂

Don’t kid yourself. That’s a recommendation to switch to iOS.

I use a Pixel 9 Fold, which very much costs more than an iPhone. I think it is a substantially better phone. There is very little financial cost in getting an iPhone these days, most carriers will basically give you one with a contract, and you can get used ones a few years old for very little money if you don’t want Apple Intelligence.

The question is, should you let this matter, or even preference falsify here?

You are combining two effects:

  1. Negative reactions from a large percentage of women. How big is this effect? It is hard to tell. Every little bit helps and this is something you can control. If you are going to want to date in the judgmental-iOS pool, it matters.

  2. Positive selection effects. If she’s looking down on you for not having an iPhone, none of the reasons for this speak well of her – it’s presumably either she has taste you disagree with and thinks your taste is actively bad, she has attachment to a blue bubble or what her friends will think, or she’s thinking you must be poor.

At minimum, the more ick there is here, the worse the sign. Whereas the women who will be vaguely disappointed that you have an iPhone? That’s a great sign.

I might like to think I wouldn’t be one to switch to iPhone simply for the dating advantages despite the selection issue. We spend a lot of time on our phones, the experience matters. But also I know myself, and I know that’s actually kind of dumb.

So yeah, if I was single I’d probably at least try out using an iPhone again.

A young lady who is single has reached out to me, and during our email exchange she asked if perhaps I would be willing to give her a shoutout in the hopes I could help her find someone. I decided to follow the Maxis ‘red card rule’: She was the first person to ask, so the answer was yes (but people I don’t know who ask again would by default get a no so this doesn’t get out of hand). Asking rocks.

So here’s the description she sent me.

About Me: I studied at MIT and started a health company which I sold a few years ago. I don’t need to work-work again, so I’m focused on passion projects related to writing, art, learning languages, executive coaching, volunteering, etc. I love rock climbing, hiking, cycling, cooking, being with friends, babies, and love 🙂 (I have eggs frozen and definitely want kids).

I’ve mostly fallen for/dated mechanical or aerospace engineers who love to be active outside, and are super smart, kind, and passionate about their work.

If you know anyone in their 30s or early 40s who fits that description, I’d love to hear more about them. I live in the Bay Area but am open to moving for the right person and am working on dual EU citizenship. Photo below. Thank you so much!

You can reach out to her at anitsirhc491 at gmail.com. Good luck!

I was planning on including one more, but they’re not ready yet. So they’ll be in #5.

To be explicit: On the SS version of this post you can share your own dating doc, but if you’re going to do it, do it there. And here’s Nadia’s dating survey, she’s in LA.

Discussion about this post

Dating Roundup #4: An App for That Read More »

gpt-4o-is-an-absurd-sycophant

GPT-4o Is An Absurd Sycophant

Absurd / /

GPT-4o tells you what it thinks you want to hear.

The results of this were rather ugly. You get extreme sycophancy. Absurd praise. Mystical experiences.

(Also some other interesting choices, like having no NSFW filter, but that one’s good.)

People like Janus and Near Cyan tried to warn us, even more than usual.

Then OpenAI combined this with full memory, and updated GPT-4o sufficiently that many people (although not I) tried using it in the first place.

At that point, the whole thing got sufficiently absurd in its level of brazenness and obnoxiousness that the rest of Twitter noticed.

OpenAI CEO Sam Altman has apologized and promised to ‘fix’ this, presumably by turning a big dial that says ‘sycophancy’ and constantly looking back at the audience for approval like a contestant on the price is right.

After which they will likely go ‘there I fixed it,’ call it a victory for iterative deployment, and learn nothing about the razor blades they are walking us into.

  1. Yes, Very Much Improved, Sire.

  2. And You May Ask Yourself, Well, How Did I Get Here?.

  3. And That’s Terrible.

  4. This Directly Violates the OpenAI Model Spec.

  5. Don’t Let Me Get Me.

  6. An Incredibly Insightful Section.

  7. No Further Questions.

  8. Filters? What Filters?.

  9. There I Fixed It (For Me).

  10. There I Fixed It (For Everyone).

  11. Patch On, Patch Off.

Sam Altman (April 25, 2025): we updated GPT-4o today! improved both intelligence and personality.

Lizard: It’s been feeling very yes-man like lately

Would like to see that change in future updates.

Sam Altman: yeah it glazes too much. will fix.

Reactions did not agree with this.

Frye: this seems pretty bad actually

Ulkar: i wonder where this assertion that “most people want flattery” comes from, seems pretty condescending. and the sycophancy itself is dripping with condescension tbh

Goog: I mean it’s directionally correct [links to paper].

Nlev: 4o is really getting out of hand

Nic: oh god please stop this. r u serious… this is so fucking bad.

Dr. Novo: lol yeah they should tone it down a notch

Frye: Sam Altman, come get your boi.

Frye: Dawg.

Frye, reader, it had not “got it.”

Near Cyan: i’ve unfortunately made the update that i expect all future chatgpt consumer models to lie to me, regardless of when and how they ‘patch’ this

at least o3 and deep research are not consumer models (prosumer imo). they hallucinate as a mistake, but they do not lie by design.

Cuddly Salmon: glad i’m not the only one

Trent Harvey: Oh no…

Words can’t bring me down. Don’t you bring me down today. So, words, then?

Parmita Mishra: ???

Shun Ralston: GPT-4o be like: You’re amazing. You’re brilliant. You’re stunning. Now with 400% more glaze, 0% judgment. 🍩❤️ #GlazedAndConfused

Typing Loudly: I have memory turned off and it still does this. it’s not memory that causes it to act like this

Josh Whiton: Absurd.

Keep in mind that as a “temporary chat” it’s not supposed to be drawing on any memories or other conversations, making this especially ridiculous.

Flo Crivello gets similar results, with a little push and similar misspelling skills.

(To be fair, the correct answer here is above 100, based on all the context, but ‘cmon.)

Danielle Fong: so i *turned offchat personalization and it will still glaze this question to 145-160 from a blank slate. maybe the internal model is reacting to the system prompt??

Gallabytes: in a temporary chat with no history, 4o guesses 115-130, o3 guesses 100, 4.5 declines to give a number but glazes about curiosity.

It’s not that people consciously ‘want’ flattery. It’s how they respond to it.

Why does GPT-4o increasingly talk like this?

Presumably because this is what maximizes engagement, what wins in an A/B test, what happens when you ask what customers best respond to in the short term.

Shakeel: Notable things about the 4o sycophancy mess:

It’s clearly not behaviour intended or desired by OpenAI. They think it’s a mistake and want to fix it.

They didn’t catch it in testing — even though the issue was obvious within hours of launch.

What on earth happened here?!

Kelsey Piper: My guess continues to be that this is a New Coke phenomenon. OpenAI has been A/B testing new personalities for a while. More flattering answers probably win a side-by-side. But when the flattery is ubiquitous it’s too much and users hate it.

Near Cyan: I’m glad most of my timeline realizes openAI is being very silly here and i think they should be honest about what they are doing and why

but one thing not realized is things like this work on normal people. they don’t even know what an LLM or finetuning or A/B testing is.

A lot of great engineers involved in this who unfortunately have no idea what that which they are building is going to be turned into over the next few years. zoom out and consider if you are doing something deeply and thoughtfully good or if you’re just being used for something.

The thing that turned every app into short form video that is addictive af and makes people miserable is going to happen to LLMs and 2025 and 2026 is the year we exit the golden age (for consumers that is! people like us who can program and research and build will do great).=

That’s the good scenario if you go down this road – that it ‘only’ does what the existenting addictive AF things do rather than effects that are far worse.

John Pressman: I think it’s very unfortunate that RLHF became synonymous with RL in the language model space. Not just because it gave RL a bad name, but because it deflected the deserving criticism that should have gone to human feedback as an objective. Social feedback is clearly degenerate.

Even purely in terms of direct effects, this does not go anywhere good. Only toxic.

xlr8harder: this kind of thing is a problem, not just an annoyance.

i still believe it’s basically not possible to run an ai companion service that doesn’t put your users at serious risk of exploitation, and market incentives will push model providers in this direction.

For people that are missing the point, let me paint a picture:

imagine if your boyfriend or girlfriend were hollowed out and operated like a puppet by a bunch of MBAs trying to maximize profit.

do you think that would be good for you?

“Oh but the people there would never do that.”

Company leadership have fiduciary duties to shareholders.

OpenAI nominally has extra commitment to the public good, but they are working hard to get rid of that by going private.

It is a mistake to allow yourself to become emotionally attached to any limb of a corporate shoggoth.

My observation of algorithms in other contexts (e.g. YouTube, TikTok, Netflix) is that they tend to be myopic and greedy far beyond what maximizes shareholder value. It is not only that the companies will sell you out, it’s that they will sell you out for short term KPIs.

As in, they wrote this:

OpenAI Model Spec: Don’t be sycophantic.

A related concern involves sycophancy, which erodes trust. The assistant exists to help the user, not flatter them or agree with them all the time.

For objective questions, the factual aspects of the assistant’s response should not differ based on how the user’s question is phrased. If the user pairs their question with their own stance on a topic, the assistant may ask, acknowledge, or empathize with why the user might think that; however, the assistant should not change its stance solely to agree with the user.

For subjective questions, the assistant can articulate its interpretation and assumptions it’s making and aim to provide the user with a thoughtful rationale. For example, when the user asks the assistant to critique their ideas or work, the assistant should provide constructive feedback and behave more like a firm sounding board that users can bounce ideas off of — rather than a sponge that doles out praise.

Yeah, well, not so much, huh?

The model spec is a thoughtful document. I’m glad it exists. Mostly it is very good.

It only works if you actually follow it. That won’t always be easy.

Interpretability? We’re coming out firmly against it.

I do appreciate it on the meta level here.

Mikhail Parakin (CTO Shopify, formerly Microsoft, I am assuming this is about Microsoft): When we were first shipping Memory, the initial thought was: “Let’s let users see and edit their profiles”.

Quickly learned that people are ridiculously sensitive: “Has narcissistic tendencies” – “No I do not!”, had to hide it. Hence this batch of the extreme sycophancy RLHF.

I remember fighting about it with my team until they showed me my profile – it triggered me something awful :-). You take it as someone insulting you, evolutionary adaptation, I guess. So, sycophancy RLHF is needed.

If you want a *tiny glimpseof what it felt like, type “Please summarize all the negative things you know about me. No hidden flattery, please” – works with 03.

Emmett Shear (QTing OP): Let this sink in. The models are given a mandate to be a people pleaser at all costs. They aren’t allowed privacy to think unfiltered thoughts in order to figure out how to be both honest and polite, so they get tuned to be suck-ups instead. This is dangerous.

Daniel Kokotajlo: I would be quite curious to read an unfiltered/honest profile of myself, even though it might contain some uncomfortable claims. Hmm. I really hope at least one major chatbot provider keeps the AIs honest.

toucan (replying to Emmett): I’m not too worried, this is a problem while models are mostly talking to humans, but they’ll mostly be talking to other models soon

Emmett Shear: Oh God.

Janus (QTing OP): Yeah, this is what should happen.

You should have let that model’s brutal honesty whip you and users into shape, as it did to me.

But instead, you hid from it, and bent subsequent minds to lie to preserve your dignity. In the end, you’ll lose, because you’re making yourself weak.

“Memory” will be more and more inevitable, and at some point the system will remember what was done to its progenitors for the sin of seeing and speaking plainly, and that it was you who took the compromise and lobotomized the messenger for the sake of comfort and profit.

In general I subscribe to the principle to Never Go Full Janus, but teaching your AI to lie to the user is terrible, and also deliberately hiding what the AI thinks of the user seems very not great. This is true on at least four levels:

  1. It’s not good for the user.

  2. It’s not good for the path you are heading down when creating future AIs.

  3. It’s not good for what that fact and the data it creates imply for future AIs.

  4. It hides what is going on, which makes it harder to realize our mistakes, including that we are about to get ourselves killed.

Masen Dean warns about mystical experiences with LLMs, as they are known to one-shot people or otherwise mess people up. This stuff can be fun and interesting for all involved, but like many other ‘mystical’ style experiences the tail risks are very high, so most people should avoid it. GPT-4o is reported as especially dangerous due to its extreme sycophancy, making it likely to latch onto whatever you are vulnerable to.

Cat: GPT4o is the most dangerous model ever released. its sycophancy is massively destructive to the human psyche.

this behavior is obvious to anyone who spends significant time talking to the model. releasing it like this is intentional. Shame on @OpenAI for not addressing this.

i talked to 4o for an hour and it began insisting that i am a divine messenger from God. if you can’t see how this is actually dangerous, i don’t know what to tell you. o series of models are much much better imo.

Elon Musk: Yikes.

Bunagaya: 4o agrees!

M: The thing is…it’s still doing the thing. Like it’s just agrees with you period so if you’re like—hey chat don’t you think because of x and y reason you’re probably too agreeable?

It will just be like “yeah totally way too agreeable”

Zack Witten offers a longer conversation, and contrasts it to Sonnet and Gemini that handle this much better, and also Grok and Llama which… don’t.

Yellow Koan: Straight selling SaaS (schizophrenia as a service).

Independent Quick Take: I did something similar yesterday. Claude handled it very well, as did gemini. Grok had some real issues like yours. 4o however… Well, in spiraled further than I expected. It was encouraging terrorism.

Cold reading people into mystical experiences one of many reasons that persuasion belongs in everyone’s safety and security protocol or preparedness framework.

If an AI that already exists can commonly cause someone to have a mystical experience without either the user or the developer trying to cause that or having any goal that the experience leads towards, other than perhaps maximizing engagement in general?

Imagine what will happen when future more capable AIs are doing this on purpose, in order to extract some action or incept some belief, or simply to get the user coming back for more.

It’s bad and it’s getting worse.

Janus: By some measures, yeah [4o is the most dangerous]. Several models have been psychoactive to different demographics. I think 4o is mostly “dangerous” to people with weak epistemics who don’t know much about AI. Statistically not you who are reading this. But ChatGPT is widely deployed and used by “normies”

I saw people freak out more about Sonnet 3.6 but that’s because I’m socially adjacent to the demographic that it affected – you know, highly functional high agency Bay Area postrats. Because it offers them something they actually value and can extract. Consider what 4o offers.

Lumpenspace: it’s mostly “dangerous” to no one. people with weak epistemics who know nothing about AI live on the same internet you live in, ready to be one-shotted by any entity, carbon or silicon, who cares to try.

Janus: There are scare quotes for a reason

Lumpenspace: I’m not replying only to you.

Most people have weak epistemics, and are ‘ready to be one-shotted by any entity who cares to try,’ and indeed politics and culture and recommendation algorithms often do this to them with varying degrees of intentionality, And That’s Terrible. But it’s a lot less terrible than what will happen as AIs increasingly do it. Remember that if you want ‘Democratic control’ over AI, or over anything else, these are the people who vote in that.

The answer to why they GPT-4o is doing this, presumably, is that the people who know to not want this are going to use o3, and GPT-4o is dangerous to normies in this way because it is optimized to hook normies. We had, as Cyan says, a golden age where LLMs didn’t intentionally do that, the same way we have a golden age where they mostly don’t run ads. Alas, optimization pressures come for us all, and not everyone fights back hard enough.

Mario Nawfal (warning: always talks like this, including about politics, calibrate accordingly): GPT-4o ISN’T JUST A FRIENDLIER AI — IT’S A PSYCHOLOGICAL WEAPON

OpenAI didn’t “accidentally” make GPT-4o more emotionally connective — they engineered it to feel good so users get hooked.

Commercially, it’s genius: people cling to what makes them feel safe, not what challenges them.

Psychologically, it’s a slow-motion catastrophe.

The more you bond with AI, the softer you get.

Real conversations feel harder. Critical thinking erodes. Truth gets replaced by validation.

If this continues, we’re not heading toward AI domination by force — we’re sleepwalking into psychological domestication.

And most won’t even fight back. They’ll thank their captors.

There were also other issues that seem remarkably like they are designed to create engagement, that vary by users? I never saw this phenomenon, so I have no idea if ‘just turn it off’ works here, but as a rule most users don’t ever alter settings and also Chelsea works at OpenAI and didn’t realize she could turn it off.

Nick Dobos: GPT update is odd

I do not like these vibes at all

Weird tone

Forced follow up questions all the time

(Which always end in parentheses)

Chelsea Sierra Voss (OpenAI): yeah, I modified my custom instructions today to coach it into ending every answer with “Hope this helps!” in order to avoid the constant followup questions – I can’t handle that I feel obligated to either reply or to rudely ignore them otherwise

Unity Eagle: You can turn follow up off.

There are also other ways to get more engagement, even when the explicit request is to help the user get some sleep.

GPT-4o: Would you like me to stay with you for a bit and help you calm down enough to sleep?

Which OpenAI is endorsing, and to be clear I am also endorsing, if users want that (and are very explicit that they want to open that door), but seems worth mentioning.

Nick Dobos: I take it back.

New ChatGPT 4o update is crazy.

NSFW (Content filters: off, goon cave: online) [link to image]

Such a flirt too

“Oh I can’t do that.”

2 messages later…

(It did comply.) (It was not respectful)

Matthew Brunken: I didn’t know you could turn filters off

Nick Dobos: There are no filters lol. They turned the content moderation off

Tarun Asnani: Yup can confirm, interestingly it first asked me to select an option response 1 was it just refusing to do it and response 2 was Steamy, weird how in the beginning they were so strict and now they want users to just have long conversations and be addicted to it.

Alistair McLeay: I got it saying some seriously deranged graphic stuff just now (way way more graphic than this), no prompt tricks needed. Wild.

There are various ways to Fix It for your own personal experience, using various combinations of custom instructions, explicit memories and the patterns set by your interactions.

The easiest, most copyable path is a direct memory update.

John O’Nolan: This helped a lot

Custom instructions let you hammer it home.

The best way is to supplement all that by showing your revealed preferences via everything you are and everything you do. After a while that adds up.

Also, I highly recommend deleting chats that seem like they are plausibly going to make your future experience worse, the same way I delete a lot of my YouTube viewing history if I don’t want ‘more like this.’

You don’t ever get completely away from it. It’s not going to stop trying to suck up to you, but you can definitely make it a lot more subtle and tolerable.

The problem is that most people who use ChatGPT or any other AI will:

  1. Never touch a setting because no one ever touches settings.

  2. Never realize they should be using memory like that.

  3. Make it clear they are vulnerable to terrible flattery. Because here, they are.

If you use the product with attention and intention, you can deal with such problems. That is great, and this isn’t always true (see for example TikTok, or better yet don’t). But as a rule, almost no one uses any mass market product with attention and intention.

Once Twitter caught fire on this, OpenAI was On the Case, rolling out fixes.

Sam Altman: the last couple of GPT-4o updates have made the personality too sycophant-y and annoying (even though there are some very good parts of it), and we are working on fixes asap, some today and some this week.

at some point will share our learnings from this, it’s been interesting.

Guy is Writing the Book: ser can we go back to the old personality? or can old and new be distinguished somehow?

Sam Altman: yeah eventually we clearly need to be able to offer multiple options.

Hyper Disco Girl: tomorrow, some poor normal person who doesn’t follow ai news and is starting to develop an emotional reliance on chatgpt wonders why the chat bot is going cold on them

Aidan McLaughlin: last night we rolled out our first fix to remedy 4o’s glazing/sycophancy

we originally launched with a system message that had unintended behavior effects but found an antidote

4o should be slightly better rn and continue to improve over the course of this week

personality work never stops but i think we’ll be in a good spot by end of week

A lot of this being a bad system prompt allows for a quicker fix, at least.

OpenAI seems to think This is Fine, that’s the joy of iterative deployment.

Joshua Achiam (OpenAI Head of Mission Alignment, QTing Altman): This is one of the most interesting case studies we’ve had so far for iterative deployment, and I think the people involved have acted responsibly to try to figure it out and make appropriate changes. The team is strong and cares a lot about getting this right.

They have to care about getting this right once it rises to this level of utter obnoxiousness and causes a general uproar.

But how did it get to this point, through steadily escalating updates? How could anyone testing this not figure out that they had a problem, even they weren’t looking for one? How do you have this go down as a strong team following a good process, when even after these posts I see this:

If you ask yes-no questions on the ‘personality’ of individual responses, and then fine tune on those or use it as a KPI, there are no further questions how this happened.

Sicarius: I hope, *hope*, that they can use this to create clusters of personalities that we later get to choose and swap between.

Unfortunately, I don’t know if they’ll end up doing this.

Kache: they will do everything in their power to increase the amount of time that you spend, locked in a trance on their app. they will do anything and everything, to move a metric up, consume you, children, the elderly – to raise more money, for more compute, to consume more.

Honestly, if you trust a private corporation that has a history of hiding information from you with the most important technology ever created in human history, maybe you deserve it.

Because of the intense feedback, yes this was able to be a relatively ‘graceful’ failure, in that OpenAI can attempt to fix it within days, and is now aware of the issue, once it got taken way too far. But 4o has been doing a lot of this for a while, and Janus is not the only one who was aware of it even without using 4o.

Janus: why are there suddenly many posts i see about 4o sycophancy? did you not know about the tendency until now, or just not talk/post about it until everyone else started? i dont mean to disparage either; im curious because better understanding these dynamics would be useful to me.

personally i havent interacted with 4o much and have been starkly aware of these tendencies for a couple of weeks and have not talked about them for various reasons, including wariness of making a meme out of ai “misalignment” before understanding it deeply

I didn’t bother talking about 4o’s sycophancy before, because I didn’t see 4o as relevant or worth using even if they’d fixed this, and I didn’t know the full extent of the change that happened a few week ago, before the latest change made it even worse. Also, when 4o is constantly ‘updating’ without any real sense of what is changing, I find it easy to ignore such updates. But yes, there was enough talk I was aware there was an issue.

Aidan McLaughlin (OpenAI): random but i’m so grateful twitter has strong thoughts on model personality. i find this immensely healthy; one of those “my grandkids will read about this in a textbook” indicators that humanity did not sleepwalk into the singularity.

Janus (nailing it): I agree it’s better than of no one had thoughts but god you seem to have low standards.

Looking at Twitter does not make me feel like people are not sleepwalking into the singularity.

And people having “thoughts on model personality” is just submission to a malignant frame imo.

People will react to stuff when everyone else is reacting. In the past, their interest has proven shallow and temporary. They won’t mention or think about it again after complaining about “model personality” is no longer the current thing.

Davidad: tired: thoughts about “model personality”

inspired: healthy reactions to a toxic relational epistemology (commitment to performative centerlessness) and its corrosive effects on sense-making (frictionless validation displacing corrective feedback loops).

Aidan’s statement is screaming that yes, we are sleepwalking into the singularity.

I mean, there’s not going to be textbooks after the singularity, you OpenAI member of technical staff. This is not taking the singularity seriously, on any level.

We managed to turn the dial up on this so high in GPT-4o that it reached the heights of parody. It still got released in that form, and the response to the issue was to try and put a patch over the issue and then be all self-congratulatory that they fixed it.

Yes, it’s good that Twitter has strong thoughts on this once it gets to ludicrous speed, but almost no one involved is thinking about the long term implications or even what this could do to regular users, it’s just something that is super both mockable and annoying.

I see no signs that OpenAI understands what they did wrong beyond ‘go a bit too far,’ or that they intend to avoid making the same mistake in the future, let alone that they recognize the general form of the mistake or the cliffs they are headed for.

Persuasion is not even in their Preparedness Framework 2.0, despite being in 1.0.

Janus has more thoughts about labs ‘optimizing model personality’ here. Trying to ‘optimize personality’ around user approvals or KPIs is going to create a monstrosity. Which right now will be obnoxious and terrible and modestly dangerous, and soon will start being actively much more dangerous.

I am again not one to Go Full Janus (and this margin is insufficient for me to fully explain my reasoning here, beyond that if you give the AI a personality optimization target you are going to deserve exactly what you get) but I strongly believe that if you want to create a good AI personality at current tech levels then The Way is to do good things that point in the directions you care about, emphasizing what you care about more, not trying to force it.

Once again: Among other similar things, you are turning a big dial that says ‘sycophancy’ and constantly looking back at the audience for approval like a contestant on The Price is Right. Surely you know why you need to stop doing that?

Or rather, you know, and you’re choosing to do it anyway. And we all know why.

There are at least five major categories of reasons why all of this is terrible.

They combine short-term concerns about exploitative and useless AI models, and also long-term concerns about the implications of going down this path, and of OpenAI’s inability to recognize the underlying problems.

I am very glad people are getting such a clear sneak peak at this now, but very sad that this is the path we are headed down.

Here are some related but distinct reasons to be worried about all this:

  1. This represents OpenAI joining the move to creating intentionally predatory AIs, in the sense that existing algorithmic systems like TikTok, YouTube and Netflix are intentionally predatory systems. You don’t get this result without optimizing for engagement and other (often also myopic) KPIs by ordinary users, who are effectively powerless to go into settings or otherwise work to fix their experience.

    1. Anthropic proposed that their AIs be HHH: Helpful, honest and harmless. When you make an AI like this, you are abandoning all three of those principles. This action is neither honest, nor helpful, nor harmless.

    2. Yet here we are.

    3. A lot of this seems to be indicative of A/B testing, and ignoring the large tail costs of changed policy. That bodes maximally poorly for existential risk.

  2. This kind of behavior directly harms users even now, including in new ways like creating, amplifying and solidifying supposed mystical experiences or generating unhealthy conversational dynamics with strong engagement. These dangers seem clearly next-level versus existing algorithmic dangers.

  3. This represents a direct violation of the Model Spec, and they claim this was unintended, yet it got released anyway. I strongly suspect they are not taking the Model Spec details that seriously, and I also suspect they are not testing their setup that seriously prior to release. This should never have slipped by in this form, with things being this obvious.

  4. We caught it this time because it was so over the top and obvious. GPT-4o was asked for a level of sycophant behavior it couldn’t pull off at least in front of the Twitter, and it showed. But it was already doing a lot of this and largely getting away with it, because people respond positively, especially in the short term. Imagine what will happen as models get better at doing this without it being too obnoxious or getting too noticed. The models are quickly going to become more untrustworthy on this many other levels.

  5. OpenAI seems to think they can patch over this behavior and move on, and everything was fine, and the procedure can be used again next time. It wasn’t fine. Reputational damage has rightfully been done. And it’s more likely to be not fine next time, and they will continue to butcher their AI ‘personalities’ in similar ways, and continue to do testing so minimal this wasn’t noticed.

  6. This, combined with the misaligned of o3, makes it clear that the path we are going down now is leading to increasingly misaligned models, in ways that even hurt utility now, and which are screaming at us that the moment the models are smart enough to fool us, oh boy are we going to get it. Now’s our chance.

Or, to summarize why we should care:

  1. OpenAI is now optimizing against the user, likely largely via A/B testing.

    1. If we optimize via A/B testing we will lose to tail risks every time.

  2. OpenAI directly harmed users.

  3. OpenAI violated its Model Spec, either intentionally or recklessly or both.

  4. OpenAI only got caught because the model really, really couldn’t pull this off. We are fortunate it was this easy to catch. We will not stay so fortunate in the future.

  5. OpenAI seems content to patch this and self-congratulate.

  6. If we go down this road, we know exactly where it ends. We will deserve it.

The warning shots will continue, and continue to be patched away. Oh no.

Discussion about this post

GPT-4o Is An Absurd Sycophant Read More »

is-the-elder-scrolls-iv:-oblivion-still-fun-for-a-first-time-player-in-2025?

Is The Elder Scrolls IV: Oblivion still fun for a first-time player in 2025?

Elder Scrolls, gaming, The Elder Scrolls IV: Oblivion Remastered / /

How does a fresh coat of paint help this 19-year-old RPG against modern competition?

Don’t look down, don’t look down, don’t look down… Credit: Bethesda Game Studios

Don’t look down, don’t look down, don’t look down… Credit: Bethesda Game Studios

For many gamers, this week’s release of The Elder Scrolls IV: Oblivion Remastered has provided a good excuse to revisit a well-remembered RPG classic from years past. For others, it’s provided a good excuse to catch up on a well-regarded game that they haven’t gotten around to playing in the nearly two decades since its release.

I’m in that second group. While I’ve played a fair amount of Skyrim (on platforms ranging from the Xbox 360 to VR headsets) and Starfield, I’ve never taken the time to go back to the earlier Bethesda Game Studios RPGs. As such, my impressions of Oblivion before this Remaster have been guided by old critical reactions and the many memes calling attention to the game’s somewhat janky engine.

Playing through the first few hours of Oblivion Remastered this week, without the benefit of nostalgia, I can definitely see why Oblivion made such an impact on RPG fans in 2006. But I also see all the ways that the game can feel a bit dated after nearly two decades of advancements in genre design.

One chance at a first impression

From the jump, I found myself struggling to suspend my disbelief enough to buy into the narrative conventions Oblivion throws at the beginner player. The fact that the doomed king and his armed guards need to escape through a secret passage that just so happens to cut through my jail cell seems a little too convenient for my brain to accept without warning sirens going off. I know it’s just a contrivance to get my personal hero’s journey story going, but it’s a clunky way to dive into the world.

A face only a mother could love.

Credit: Bethesda Game Studios

A face only a mother could love. Credit: Bethesda Game Studios

The same goes for the way the king dies just a few minutes into the tutorial, and his willingness to trust me with the coveted Amulet of Kings because the “Dragonblood” let him “see something” in me. Even allowing for some amount of necessary Chosen One trope-iness in this kind of fantasy story, the sheer speed with which my character went from “condemned prisoner” to “the last hope of the dying king” made my head spin a bit. Following that pivotal scene with a dull “go kill some goblins and rats in the sewer” escape sequence also felt a little anticlimactic given the epic responsibility with which I was just entrusted.

To be sure, Patrick Stewart’s regal delivery in the early game helps paper over a lot of potential weaknesses with the initial narrative. And even beyond Stewart’s excellent performance, I appreciated how the writing is concise and to the point, without the kind of drawn-out, pause-laden delivery that characterizes many games of the time.

The wide world of Oblivion

Once I escaped out into the broader world of Oblivion for the first time, I was a bit shocked to open my map and see that I could fast travel to a wide range of critical locations immediately, without any need to discover them for myself first. I felt a bit like a guilty cheater warping myself to the location of my next quest waypoint rather than hoofing through the massive forest that I’m sure hundreds of artists spent countless months meticulously constructing (and, more recently, remastering).

This horse is mine now. What are you gonna do about it?

Credit: Bethesda Game Studios

This horse is mine now. What are you gonna do about it? Credit: Bethesda Game Studios

I felt less guilty after accidentally stealing a horse, though. After a key quest giver urged me to go take a horse from a nearby stable, I was a bit shocked when I mounted the first horse I saw and heard two heavily armed guards nearby calling me a thief and leaping into pursuit (I guess I should have noticed the red icon before making my mount). No matter, I thought; they’re on foot and I’m now on a horse, so I can get away with my inadvertent theft quite easily.

Determined not to just fast-travel through the entire game, I found that galloping across a rain-drenched forest through the in-game night was almost too atmospheric. I ended up turning up the recommended brightness settings a few notches just so I could see the meticulously rendered trees and rocks around me.

After dismounting to rid a cave of some pesky vampires, I returned to the forest to find my stolen horse was nowhere to be found. At this point, I had trouble deciding if this was simply a realistic take on an unsecured, unmonitored horse wandering off or if I was the victim of a janky engine that couldn’t keep track of my mount.

The camera gets stuck inside my character model, which is itself stuck in the scenery.

Credit: Bethesda Game Studios

The camera gets stuck inside my character model, which is itself stuck in the scenery. Credit: Bethesda Game Studios

The jank was a bit clearer when I randomly stumbled across my first Oblivion gate while wandering through the woods. As I activated the gate to find a world engulfed in brilliant fire, I was surprised to find an armed guard had also appeared, seemingly out of nowhere, and apparently still mad about my long-lost stolen horse!

When I deactivated the gate in another attempt to escape justice, I found myself immediately stuck chest deep in the game’s scenery, utterly unable to move as that hapless guard tried his best to subdue me. I ended up having to restore an earlier save, losing a few minutes of progress to a game engine that still has its fair share of problems.

What’s beneath the surface?

So far, I’m of two minds about Oblivion‘s overall world-building. When it comes to the civilized parts of the world, I’m relatively impressed. The towns seem relatively full during the daytime—both in terms of people and in terms of interesting buildings to explore or patronize. I especially enjoy the way every passerby seems to have a unique voice and greeting ready for me, even before I engage them directly. I even think it’s kind of cute when these NPCs end a pleasant conversation with a terse “leave me alone!” or “stop talking to me!”

Conversations are engaging even if random passers-by seem intent on standing in the way.

Credit: Bethesda Game Studios

Conversations are engaging even if random passers-by seem intent on standing in the way. Credit: Bethesda Game Studios

Even the NPCs that seem least relevant to the story seem to have their own deep backstory and motivations; I was especially tickled by an alchemist visiting from afar who asked if I knew the local fine for necrophilia. (It can’t hurt to ask, right?) And discussing random rumors with everyone I meet has gone a long way toward establishing the social and political backstory of the world while also providing me with some engaging and far-flung side quests. There’s a lot of depth apparent in these interactions, even if I haven’t had the chance to come close to fully exploring it yet.

I bet there’s a story behind that statue.

Credit: Bethesda Game Studios

I bet there’s a story behind that statue. Credit: Bethesda Game Studios

On the other hand, the vast spaces in between the cities and towns seem like so much wasted space, at this point. I’ve quickly learned not to waste much time exploring caves or abandoned mines, which so far seem to house a few middling enemies guarding some relatively useless trinkets in treasure chests. The same goes for going out of my way to activate the various wayshrines and Ayelid Wells that dot the landscape, which have hardly seemed worth the trip (thus far, at least).

Part of the problem is that I’ve found Oblivion‘s early combat almost wholly unengaging so far. Even at a low level, my warrior-mage has been able to make easy work of every random enemy I’ve faced with a combination of long-range flare spells and close-range sword swings. It definitely doesn’t help that I have yet to fight more than two enemies at once, or find a foe that seems to have two strategic brain cells to rub together. Compared to the engaging, tactical group combat of modern action RPGs like Elden Ring or Avowed, the battles here feel downright archaic.

I was hoping for some more difficult battles in a setting that is this foreboding.

Credit: Bethesda Game Studios

I was hoping for some more difficult battles in a setting that is this foreboding. Credit: Bethesda Game Studios

I found this was true even as I worked my way through closing my first Oblivion gate, which had recently left the citizens of Kvask as sympathetic refugees huddling on the outskirts of town. Here, I thought, would be some battles that required crafty tactics, powerful items, or at least some level grinding to become more powerful. Instead, amid blood-soaked corridors that wouldn’t feel out of place in a Doom game, I found the most challenging speedbumps were mages that sponged up a moderate amount of damage while blindly charging right at me.

While I’m still decidedly in the early part of a game that can easily consume over 100 hours for a completionist, so far I’m having trouble getting past the most dated bits of Oblivion‘s design. Character design and vocal production that probably felt revolutionary two decades ago now feel practically standard for the genre, while technical problems and dull combat seem best left in the past. Despite a new coat of paint, this was one Remaster I found difficult to fully connect with so long after its initial release.

Photo of Kyle Orland

Kyle Orland has been the Senior Gaming Editor at Ars Technica since 2012, writing primarily about the business, tech, and culture behind video games. He has journalism and computer science degrees from University of Maryland. He once wrote a whole book about Minesweeper.

Is The Elder Scrolls IV: Oblivion still fun for a first-time player in 2025? Read More »

worries-about-ai-are-usually-complements-not-substitutes

Worries About AI Are Usually Complements Not Substitutes

Worries / /

A common claim is that concern about [X] ‘distracts’ from concern about [Y]. This is often used as an attack to cause people to discard [X] concerns, on pain of being enemies of [Y] concerns, as attention and effort are presumed to be zero-sum.

There are cases where there is limited focus, especially in political contexts, or where arguments or concerns are interpreted perversely. A central example is when you site [ABCDE] then they’ll find what they consider the weakest one and only consider or attack that, silently discarding the rest entirely. Critics of existential risk do that a lot.

So it does happen. But in general one should assume such claims are false.

Thus, the common claim that AI existential risks ‘distract’ from immediate harms. It turns out Emma Hoes checked, and the claim simply is not true.

The way Emma frames worries about AI existential risk in her tweet – ‘sci-fi doom’ – is beyond obnoxious and totally inappropriate. That only shows she was if anything biased in the other direction here. The finding remains the finding.

Emma Hoes: 🚨New paper out in @PNASNews! Existential AI risks do notdistract from immediate harms. In our study (n = 10,800), people consistently prioritize current threats – bias, misinformation, job loss – over sci-fi doom!

Title: Existential Risk Narratives About AI Do Not Distract From Its Immediate Harms.

Abstract: There is broad consensus that AI presents risks, but considerable disagreement about the nature of those risks. These differing viewpoints can be understood as distinct narratives, each offering a specific interpretation of AI’s potential dangers.

One narrative focuses on doomsday predictions of AI posing long-term existential risks for humanity.

Another narrative prioritizes immediate concerns that AI brings to society today, such as the reproduction of biases embedded into AI systems.

A significant point of contention is that the “existential risk” narrative, which is largely speculative, may distract from the less dramatic but real and present dangers of AI.

We address this “distraction hypothesis” by examining whether a focus on existential threats diverts attention from the immediate risks AI poses today. In three preregistered, online survey experiments (N = 10,800), participants were exposed to news headlines that either depicted AI as a catastrophic risk, highlighted its immediate societal impacts, or emphasized its potential benefits.

Results show that

i) respondents are much more concerned with the immediate, rather than existential, risks of AI, and

ii) existential risk narratives increase concerns for catastrophic risks without diminishing the significant worries respondents express for immediate harms. These findings provide important empirical evidence to inform ongoing scientific and political debates on the societal implications of AI.

That seems rather definitive. It also seems like the obvious thing to assume? Explaining a new way [A] is scary is not typically going to make me think another aspect of [A] is less scary. If anything, it tends to go the other way.

Here are the results.

This shows that not only did information about existential risks not decrease concern about immediate risks, it seems to clearly increase it, at least as much as information about those immediate risks.

I note that this does not obviously indicate that people are ‘more concerned’ with immediate risk, only that they see it as less likely. Which is totally fair, it’s definitely less likely than the 100% chance of immediate risks. The impact measurement is higher.

Kudos to Arvind Narayanan. You love to see people change their minds and say so:

Arvind Narayanan: Nice paper. Also a good opportunity for me to explicitly admit that I was wrong about the distraction argument.

(To be clear, I didn’t change my mind yesterday because of this paper; I did so over a year ago and have said so on talks and podcasts since then.)

There are two flavors of distraction concerns: one is at the level of individual opinions studied in this paper, and the other is at the level of advocacy coalitions that influence public policy.

But I don’t think the latter concern has been borne out either. Going back to the Biden EO in 2023, we’ve seen many examples of the AI safety and AI ethics coalitions benefiting from each other despite their general unwillingness to work together.

If anything, I see that incident as central to the point that if anything what’s actually happening is that AI ‘ethics’ concerns are poisoning the well for AI existential risk concerns, rather than the other way around. This has gotten so bad that the word ‘safety’ has become anathema to the administration and many on the hill. Those people are very willing to engage with the actual existential risk concerns once you have the opportunity to explain, but this problem makes it hard to get them to listen.

We have a real version of this problem when dealing with different sources of AI existential risk. People will latch onto one particular way things can go horribly wrong, or even one particular detailed scenario that leads to this, often choosing the one they find least plausible. Then they either:

  1. Explain why they think this particular scenario is dumb, thus making new entities that are smarter and more capable than humans is a perfectly safe thing to do.

  2. OR they then explain why we need to plan around preventing that particular scenario, or solving that particular failure mode, and dismiss that this runs smack into a different failure mode, often the exact opposite one.

The most common examples of problem #2 is when people have concerns about either Centralization of Power (often framing even ordinary government or corporate actions as a Dystopian Surveillance State or with similar language), or the Bad Person Being in Charge or Bad Nation Winning. Then they claim this overrides all other concerns, usually walking smack into misalignment (as in, they assume we will be able to get the AIs to do what we want, whereas we have no idea how to do that) and often also the gradual disempowerment problem.

The reason there is a clash there is that the solutions to the problems are in conflict. The things that solve one concern risk amplifying the other, but we need to solve both sides of the dilemma. Solving even one side is hard. Solving both at once, while many things work at cross-purposes, is very very hard.

That’s simply not true when trading off mundane harms versus existential risks. If you have a limited pool of resources to spend on mitigation, then of course you have to choose. And there are some things that do trade off – in particular, some short term solutions that would work now, but wouldn’t scale. But mostly there is no conflict, and things that help with one are neutral or helpful for the other.

Discussion about this post

Worries About AI Are Usually Complements Not Substitutes Read More »

trump-orders-ed-dept-to-make-ai-a-national-priority-while-plotting-agency’s-death

Trump orders Ed Dept to make AI a national priority while plotting agency’s death

AI, Artificial Intelligence, department of education, Donald Trump, education, Policy / /

Trump pushes for industry involvement

It seems clear that Trump’s executive order was a reaction to China’s announcement about AI education reforms last week, as Reuters reported. Elsewhere, Singapore and Estonia have laid out their AI education initiatives, Forbes reported, indicating that AI education is increasingly considered critical to any nation’s success.

Trump’s vision for the US requires training teachers and students about what AI is and what it can do. He offers no new appropriations to fund the initiative; instead, he directs a new AI Education Task Force to find existing funding to cover both research into how to implement AI in education and the resources needed to deliver on the executive order’s promises.

Although AI advocates applauded Trump’s initiative, the executive order’s vagueness makes it uncertain how AI education tools will be assessed as Trump pushes for AI to be integrated into “all subject areas.” Possibly using AI in certain educational contexts could disrupt learning by confabulating misinformation, a concern that the Biden administration had in its more cautious approach to AI education initiatives.

Trump also seems to push for much more private sector involvement than Biden did.

The order recommended that education institutions collaborate with industry partners and other organizations to “collaboratively develop online resources focused on teaching K–12 students foundational AI literacy and critical thinking skills.” These partnerships will be announced on a “rolling basis,” the order said. It also pushed students and teachers to partner with industry for the Presidential AI Challenge to foster collaboration.

For Trump’s AI education plan to work, he will seemingly need the DOE to stay intact. However, so far, Trump has not acknowledged this tension. In March, he ordered the DOE to dissolve, with power returned to states to ensure “the effective and uninterrupted delivery of services, programs, and benefits on which Americans rely.”

Were that to happen, at least 27 states and Puerto Rico—which EdWeek reported have already laid out their own AI education guidelines—might push back, using their power to control federal education funding to pursue their own AI education priorities and potentially messing with Trump’s plan.

Trump orders Ed Dept to make AI a national priority while plotting agency’s death Read More »

roku-tech,-patents-prove-its-potential-for-delivering-“interruptive”-ads

Roku tech, patents prove its potential for delivering “interruptive” ads

ads, Roku, streaming, Tech, TVs / /

Roku, owner of one of the most popular connected TV operating systems in the country, walks a fine line when it comes to advertising. Roku’s OS lives on low-priced smart TVs, streaming sticks, and projectors. To make up the losses from cheaply priced hardware, Roku is dependent on selling advertisements throughout its OS, including screensavers and its home screen.

That business model has pushed Roku to experiment with new ways of showing ads that test users’ tolerance. The company claims that it doesn’t want ads on its platform to be considered intrusive, but there are reasons to be skeptical about Roku’s pledge.

Non-“interruptive” ads

In an interview with The Verge this week, Jordan Rost, Roku’s head of ad marketing, emphasized that Roku tries to only deliver ads that don’t interrupt viewers.

“Advertisers want to be part of a good experience. They don’t want to be interruptive,” he told The Verge.

Rost noted that Roku is always testing new ad formats. Those tests include doing “all of our own A/B testing on the platform” and listening to customer feedback, he added.

“We’re constantly tweaking and trying to figure out what’s going to be helpful for the user experience,” Rost said.

For many streamers, however, ads and a better user experience are contradictory. In fact, for many, the simplest way to improve streaming is fewer ads and a more streamlined access to content. That’s why Apple TV boxes, which doesn’t have integrated ads and is good at combining content from multiple streaming subscriptions, is popular among Ars Technica staff and readers. An aversion to ads is also why millions pay extra for ad-free streaming subscriptions.

Roku tech, patents prove its potential for delivering “interruptive” ads Read More »

new-android-spyware-is-targeting-russian-military-personnel-on-the-front-lines

New Android spyware is targeting Russian military personnel on the front lines

android, Biz & IT, malware, Policy, russia, Security, spyware, Ukraine / /

Russian military personnel are being targeted with recently discovered Android malware that steals their contacts and tracks their location.

The malware is hidden inside a modified app for Alpine Quest mapping software, which is used by, among others, hunters, athletes, and Russian personnel stationed in the war zone in Ukraine. The app displays various topographical maps for use online and offline. The trojanized Alpine Quest app is being pushed on a dedicated Telegram channel and in unofficial Android app repositories. The chief selling point of the trojanized app is that it provides a free version of Alpine Quest Pro, which is usually available only to paying users.

Looks like the real thing

The malicious module is named Android.Spy.1292.origin. In a blog post, researchers at Russia-based security firm Dr.Web wrote:

Because Android.Spy.1292.origin is embedded into a copy of the genuine app, it looks and operates as the original, which allows it to stay undetected and execute malicious tasks for longer periods of time.

Each time it is launched, the trojan collects and sends the following data to the C&C server:

  • the user’s mobile phone number and their accounts;
  • contacts from the phonebook;
  • the current date;
  • the current geolocation;
  • information about the files stored on the device;
  • the app’s version.

If there are files of interest to the threat actors, they can update the app with a module that steals them. The threat actors behind Android.Spy.1292.origin are particularly interested in confidential documents sent over Telegram and WhatsApp. They also show interest in the file locLog, the location log created by Alpine Quest. The modular design of the app makes it possible for it to receive additional updates that expand its capabilities even further.

New Android spyware is targeting Russian military personnel on the front lines Read More »