Biz & IT

found-in-the-wild:-the-world’s-first-unkillable-uefi-bootkit-for-linux

Found in the wild: The world’s first unkillable UEFI bootkit for Linux

Biz & IT, bootkit, malware, Security, uefi / /

Over the past decade, a new class of infections has threatened Windows users. By infecting the firmware that runs immediately before the operating system loads, these UEFI bootkits continue to run even when the hard drive is replaced or reformatted. Now the same type of chip-dwelling malware has been found in the wild for backdooring Linux machines.

Researchers at security firm ESET said Wednesday that Bootkitty—the name unknown threat actors gave to their Linux bootkit—was uploaded to VirusTotal earlier this month. Compared to its Windows cousins, Bootkitty is still relatively rudimentary, containing imperfections in key under-the-hood functionality and lacking the means to infect all Linux distributions other than Ubuntu. That has led the company researchers to suspect the new bootkit is likely a proof-of-concept release. To date, ESET has found no evidence of actual infections in the wild.

The ASCII logo that Bootkitty is capable of rendering. Credit: ESET

Be prepared

Still, Bootkitty suggests threat actors may be actively developing a Linux version of the same sort of unkillable bootkit that previously was found only targeting Windows machines.

“Whether a proof of concept or not, Bootkitty marks an interesting move forward in the UEFI threat landscape, breaking the belief about modern UEFI bootkits being Windows-exclusive threats,” ESET researchers wrote. “Even though the current version from VirusTotal does not, at the moment, represent a real threat to the majority of Linux systems, it emphasizes the necessity of being prepared for potential future threats.”

A rootkit is a piece of malware that runs in the deepest regions of the operating system it infects. It leverages this strategic position to hide information about its presence from the operating system itself. A bootkit, meanwhile, is malware that infects the boot-up process in much the same way. Bootkits for the UEFI—short for Unified Extensible Firmware Interface—lurk in the chip-resident firmware that runs each time a machine boots. These sorts of bootkits can persist indefinitely, providing a stealthy means for backdooring the operating system even before it has fully loaded and enabled security defenses such as antivirus software.

The bar for installing a bootkit is high. An attacker first must gain administrative control of the targeted machine, either through physical access while it’s unlocked or somehow exploiting a critical vulnerability in the OS. Under those circumstances, attackers already have the ability to install OS-resident malware. Bootkits, however, are much more powerful since they (1) run before the OS does and (2) are, at least practically speaking, undetectable and unremovable.

Found in the wild: The world’s first unkillable UEFI bootkit for Linux Read More »

qnap-firmware-update-leaves-nas-owners-locked-out-of-their-boxes

QNAP firmware update leaves NAS owners locked out of their boxes

Biz & IT, firmware updates, NAS, network attached storage, QNAP, Security, Tech / /

A recent firmware pushed to QNAP network attached storage (NAS) devices left a number of owners unable to access their storage systems. The company has pulled back the firmware and issued a fixed version, but the company’s response has left some users feeling less confident in the boxes into which they put all their digital stuff.

As seen on a QNAP community thread, and as announced by QNAP itself, the QNAP operating system, QTS, received update 5.2.2.2950, build 20241114, at some point around November 19. After QNAP “received feedbacks from some users reporting issues with device functionality after installation,” the firm says it withdrew it, “conducted a comprehensive investigation,” and re-released a fixed version “within 24 hours.”

The community thread sees many more users of different systems having problems than the shortlist (“limited models of TS-x53D series and TS-x51 series”) released by QNAP. Issues reported included owners being rejected as an authorized user, devices reporting issues with booting, and claims of Python not being installed to run some apps and services.

QNAP says affected users can either downgrade their devices (presumably to then upgrade once more to the fixed update) or contact support for help. Response from QNAP support, as told by users on forums and social media, has not measured up to the nature of losing access to an entire backup system.

QNAP firmware update leaves NAS owners locked out of their boxes Read More »

spies-hack-wi-fi-networks-in-far-off-land-to-launch-attack-on-target-next-door

Spies hack Wi-Fi networks in far-off land to launch attack on target next door

advanced persistent threats, Biz & IT, hacking, Security, tradecraft / /

While stalking its target, GruesomeLarch performed credential-stuffing attacks that compromised the passwords of several accounts on a web service platform used by the organization’s employees. Two-factor authentication enforced on the platform, however, prevented the attackers from compromising the accounts.

So GruesomeLarch found devices in physically adjacent locations, compromised them, and used them to probe the target’s Wi-Fi network. It turned out credentials for the compromised web services accounts also worked for accounts on the Wi-Fi network, only no 2FA was required.

Adding further flourish, the attackers hacked one of the neighboring Wi-Fi-enabled devices by exploiting what in early 2022 was a zero-day vulnerability in the Microsoft Windows Print Spooler.

Credit: Volexity

The 2022 hack demonstrates how a single faulty assumption can undo an otherwise effective defense. For whatever reason—likely an assumption that 2FA on the Wi-Fi network was unnecessary because attacks required close proximity—the target deployed 2FA on the Internet-connecting web services platform (Adair isn’t saying what type) but not on the Wi-Fi network. That one oversight ultimately torpedoed a robust security practice.

Advanced persistent threat groups like GruesomeLarch—a part of the much larger GRU APT with names including Fancy Bear, APT28, Forrest Blizzard, and Sofacy—excel in finding and exploiting these sorts of oversights.

Volexity’s post describing the 2022 attack provides plenty of technical details about the compromise on the many links in this sophisticated daisy chain attack flow. There’s also useful advice for protecting networks against these sorts of compromises.

Spies hack Wi-Fi networks in far-off land to launch attack on target next door Read More »

amazon-pours-another-$4b-into-anthropic,-openai’s-biggest-rival

Amazon pours another $4B into Anthropic, OpenAI’s biggest rival

AI, Amazon, Anthropic, Biz & IT, chatgpt, chatgtp, Dario Amodei, large language models, LLMs, machine learning / /

Anthropic, founded by former OpenAI executives Dario and Daniela Amodei in 2021, will continue using Google’s cloud services along with Amazon’s infrastructure. The UK Competition and Markets Authority reviewed Amazon’s partnership with Anthropic earlier this year and ultimately determined it did not have jurisdiction to investigate further, clearing the way for the partnership to continue.

Shaking the money tree

Amazon’s renewed investment in Anthropic also comes during a time of intense competition between cloud providers Amazon, Microsoft, and Google. Each company has made strategic partnerships with AI model developers—Microsoft with OpenAI (to the tune of $13 billion), Google with Anthropic (committing $2 billion over time), for example. These investments also encourage the use of each company’s data centers as demand for AI grows.

The size of these investments reflects the current state of AI development. OpenAI raised an additional $6.6 billion in October, potentially valuing the company at $157 billion. Anthropic has been eyeballing a $40 billion valuation during a recent investment round.

Training and running AI models is very expensive. While Google and Meta have their own profitable mainline businesses that can subsidize AI development, dedicated AI firms like OpenAI and Anthropic need constant infusions of cash to stay afloat—in other words, this won’t be the last time we hear of billion-dollar-scale AI investments from Big Tech.

Amazon pours another $4B into Anthropic, OpenAI’s biggest rival Read More »

google-stops-letting-sites-like-forbes-rule-search-for-“best-cbd-gummies“

Google stops letting sites like Forbes rule search for “Best CBD Gummies“

affiliate, affiliate marketing, Biz & IT, forbes, forbes marketplace, Google, guest posts, parasite seo, SEO, site reputation, Tech / /

Under the strength of Forbes’ long-existing and well-linked site, Forbes Marketplace/Advisor has dominated the search term “best cbd gummies” for “an eternity,” according to SEO analyst Lily Ray. Forbes has similarly dominated “best pet insurance,” and long came up as the second result for “how to get rid of roaches,” as detailed in a blog post by Lars Lofgren. If people click on this high-ranking result, and then click on a link to buy a product or request a roach removal consultation, Forbes typically gets a cut.

Forbes Marketplace had seemingly also provided SEO-minded review services to CNN and USA Today, as detailed by Lofgren. Lofgren’s term for this business, “Parasite SEO,” took hold in corners critical of the trend. Ars has contacted Forbes for comment and will update this post with response.

“The unfair, exploitative nature” of “parasite SEO”

Google writes that it had reviewed “situations where there might be varying degrees of first-party involvement” (most publishers’ review sites indicate some kind of oversight or editorial standards linked to the primary site). But however arranged, “no amount of first-party involvement alters the fundamental third-party nature of the content or the unfair, exploitative nature of attempting to take advantage of the host sites’ ranking signals.”

As such, using third-party content in such a way as to take advantage of a high search quality ranking, outside the site’s primary focus, is considered spam. That delivers a major hit to a site’s Google ranking, and the impact is already being felt.

The SEO reordering does not affect more established kinds of third-party content, like wire service reports, syndication, or well-marked sponsored content, as detailed in Google’s spam policy section about site reputation abuse. As seen on the SEO subreddit, and on social media, Google has given sites running afoul of its updated policy a “Manual Action” rather than relying only on its algorithm to catch the often opaque arrangements.

Google stops letting sites like Forbes rule search for “Best CBD Gummies“ Read More »

a-year-after-ditching-waitlist,-starlink-says-it-is-“sold-out”-in-parts-of-us

A year after ditching waitlist, Starlink says it is “sold out” in parts of US

Biz & IT, Policy, starlink / /

The Starlink waitlist is back in certain parts of the US, including several large cities on the West Coast and in Texas. The Starlink availability map says the service is sold out in and around Seattle; Spokane, Washington; Portland, Oregon; San Diego; Sacramento, California; and Austin, Texas. Neighboring cities and towns are included in the sold-out zones.

There are additional sold-out areas in small parts of Colorado, Montana, and North Carolina. As PCMag noted yesterday, the change comes about a year after Starlink added capacity and removed its waitlist throughout the US.

Elsewhere in North America, there are some sold-out areas in Canada and Mexico. Across the Atlantic, Starlink is sold out in London and neighboring cities. Starlink is not yet available in most of Africa, and some of the areas where it is available are sold out.

Starlink is generally seen as most useful in rural areas with less access to wired broadband, but it seems to be attracting interest in more heavily populated areas, too. While detailed region-by-region subscriber numbers aren’t available publicly, SpaceX President Gwynne Shotwell said last week that Starlink has nearly 5 million users worldwide.

A year after ditching waitlist, Starlink says it is “sold out” in parts of US Read More »

niantic-uses-pokemon-go-player-data-to-build-ai-navigation-system

Niantic uses Pokémon Go player data to build AI navigation system

AI, AI ethics, Biz & IT, chatgpt, chatgtp, large geospatial model, large language model, machine learning, mobile games, niantic, Pokémon, Pokemon GO, smartphones, video games, Visual Positioning System / /

Last week, Niantic announced plans to create an AI model for navigating the physical world using scans collected from players of its mobile games, such as Pokémon Go, and from users of its Scaniverse app, reports 404 Media.

All AI models require training data. So far, companies have collected data from websites, YouTube videos, books, audio sources, and more, but this is perhaps the first we’ve heard of AI training data collected through a mobile gaming app.

“Over the past five years, Niantic has focused on building our Visual Positioning System (VPS), which uses a single image from a phone to determine its position and orientation using a 3D map built from people scanning interesting locations in our games and Scaniverse,” Niantic wrote in a company blog post.

The company calls its creation a “large geospatial model” (LGM), drawing parallels to large language models (LLMs) like the kind that power ChatGPT. Whereas language models process text, Niantic’s model will process physical spaces using geolocated images collected through its apps.

The scale of Niantic’s data collection reveals the company’s sizable presence in the AR space. The model draws from over 10 million scanned locations worldwide, with users capturing roughly 1 million new scans weekly through Pokémon Go and Scaniverse. These scans come from a pedestrian perspective, capturing areas inaccessible to cars and street-view cameras.

First-person scans

The company reports it has trained more than 50 million neural networks, each representing a specific location or viewing angle. These networks compress thousands of mapping images into digital representations of physical spaces. Together, they contain over 150 trillion parameters—adjustable values that help the networks recognize and understand locations. Multiple networks can contribute to mapping a single location, and Niantic plans to combine its knowledge into one comprehensive model that can understand any location, even from unfamiliar angles.

Niantic uses Pokémon Go player data to build AI navigation system Read More »

ai-generated-shows-could-replace-lost-dvd-revenue,-ben-affleck-says

AI-generated shows could replace lost DVD revenue, Ben Affleck says

AI, AI visual effects, AI-generated video, Ben Affleck, Biz & IT, filmmaking, machine learning, movies, Special Effects, TV shows, video synthesis / /

Last week, actor and director Ben Affleck shared his views on AI’s role in filmmaking during the 2024 CNBC Delivering Alpha investor summit, arguing that AI models will transform visual effects but won’t replace creative filmmaking anytime soon. A video clip of Affleck’s opinion began circulating widely on social media not long after.

“Didn’t expect Ben Affleck to have the most articulate and realistic explanation where video models and Hollywood is going,” wrote one X user.

In the clip, Affleck spoke of current AI models’ abilities as imitators and conceptual translators—mimics that are typically better at translating one style into another instead of originating deeply creative material.

“AI can write excellent imitative verse, but it cannot write Shakespeare,” Affleck told CNBC’s David Faber. “The function of having two, three, or four actors in a room and the taste to discern and construct that entirely eludes AI’s capability.”

Affleck sees AI models as “craftsmen” rather than artists (although some might find the term “craftsman” in his analogy somewhat imprecise). He explained that while AI can learn through imitation—like a craftsman studying furniture-making techniques—it lacks the creative judgment that defines artistry. “Craftsman is knowing how to work. Art is knowing when to stop,” he said.

“It’s not going to replace human beings making films,” Affleck stated. Instead, he sees AI taking over “the more laborious, less creative and more costly aspects of filmmaking,” which could lower barriers to entry and make it easier for emerging filmmakers to create movies like Good Will Hunting.

Films will become dramatically cheaper to make

While it may seem on its surface like Affleck was attacking generative AI capabilities in the tech industry, he also did not deny the impact it may have on filmmaking. For example, he predicted that AI would reduce costs and speed up production schedules, potentially allowing shows like HBO’s House of the Dragon to release two seasons in the same period as it takes to make one.

AI-generated shows could replace lost DVD revenue, Ben Affleck says Read More »

chatgpt’s-success-could-have-come-sooner,-says-former-google-ai-researcher

ChatGPT’s success could have come sooner, says former Google AI researcher

AI, AI technology, Attention is all you need, biotechnology, Biz & IT, chatgpt, chatgtp, Google, GPT-3, GPT-4, Inceptive, Jakob Uszkoreit, large language models, openai, RNA, RNA vaccines, Transformers / /

A co-author of Attention Is All You Need reflects on ChatGPT’s surprise and Google’s conservatism.

Jakob Uszkoreit Credit: Jakob Uszkoreit / Getty Images

In 2017, eight machine-learning researchers at Google released a groundbreaking research paper called Attention Is All You Need, which introduced the Transformer AI architecture that underpins almost all of today’s high-profile generative AI models.

The Transformer has made a key component of the modern AI boom possible by translating (or transforming, if you will) input chunks of data called “tokens” into another desired form of output using a neural network. Variations of the Transformer architecture power language models like GPT-4o (and ChatGPT), audio synthesis models that run Google’s NotebookLM and OpenAI’s Advanced Voice Mode, video synthesis models like Sora, and image synthesis models like Midjourney.

At TED AI 2024 in October, one of those eight researchers, Jakob Uszkoreit, spoke with Ars Technica about the development of transformers, Google’s early work on large language models, and his new venture in biological computing.

In the interview, Uszkoreit revealed that while his team at Google had high hopes for the technology’s potential, they didn’t quite anticipate its pivotal role in products like ChatGPT.

The Ars interview: Jakob Uszkoreit

Ars Technica: What was your main contribution to the Attention is All You Need paper?

Jakob Uszkoreit (JU): It’s spelled out in the footnotes, but my main contribution was to propose that it would be possible to replace recurrence [from Recurrent Neural Networks] in the dominant sequence transduction models at the time with the attention mechanism, or more specifically self-attention. And that it could be more efficient and, as a result, also more effective.

Ars: Did you have any idea what would happen after your group published that paper? Did you foresee the industry it would create and the ramifications?

JU: First of all, I think it’s really important to keep in mind that when we did that, we were standing on the shoulders of giants. And it wasn’t just that one paper, really. It was a long series of works by some of us and many others that led to this. And so to look at it as if this one paper then kicked something off or created something—I think that is taking a view that we like as humans from a storytelling perspective, but that might not actually be that accurate of a representation.

My team at Google was pushing on attention models for years before that paper. It’s a lot longer of a slog with much, much more, and that’s just my group. Many others were working on this, too, but we had high hopes that it would push things forward from a technological perspective. Did we think that it would play a role in really enabling, or at least apparently, seemingly, flipping a switch when it comes to facilitating products like ChatGPT? I don’t think so. I mean, to be very clear in terms of LLMs and their capabilities, even around the time we published the paper, we saw phenomena that were pretty staggering.

We didn’t get those out into the world in part because of what really is maybe a notion of conservatism around products at Google at the time. But we also, even with those signs, weren’t that confident that stuff in and of itself would make that compelling of a product. But did we have high hopes? Yeah.

Ars: Since you knew there were large language models at Google, what did you think when ChatGPT broke out into a public success? “Damn, they got it, and we didn’t?”

JU: There was a notion of, well, “that could have happened.” I think it was less of a, “Oh dang, they got it first” or anything of the like. It was more of a “Whoa, that could have happened sooner.” Was I still amazed by just how quickly people got super creative using that stuff? Yes, that was just breathtaking.

Jakob Uskoreit presenting at TED AI 2024.

Jakob Uszkoreit presenting at TED AI 2024. Credit: Benj Edwards

Ars: You weren’t at Google at that point anymore, right?

JU: I wasn’t anymore. And in a certain sense, you could say the fact that Google wouldn’t be the place to do that factored into my departure. I left not because of what I didn’t like at Google as much as I left because of what I felt I absolutely had to do elsewhere, which is to start Inceptive.

But it was really motivated by just an enormous, not only opportunity, but a moral obligation in a sense, to do something that was better done outside in order to design better medicines and have very direct impact on people’s lives.

Ars: The funny thing with ChatGPT is that I was using GPT-3 before that. So when ChatGPT came out, it wasn’t that big of a deal to some people who were familiar with the tech.

JU: Yeah, exactly. If you’ve used those things before, you could see the progression and you could extrapolate. When OpenAI developed the earliest GPTs with Alec Radford and those folks, we would talk about those things despite the fact that we weren’t at the same companies. And I’m sure there was this kind of excitement, how well-received the actual ChatGPT product would be by how many people, how fast. That still, I think, is something that I don’t think anybody really anticipated.

Ars: I didn’t either when I covered it. It felt like, “Oh, this is a chatbot hack of GPT-3 that feeds its context in a loop.” And I didn’t think it was a breakthrough moment at the time, but it was fascinating.

JU: There are different flavors of breakthroughs. It wasn’t a technological breakthrough. It was a breakthrough in the realization that at that level of capability, the technology had such high utility.

That, and the realization that, because you always have to take into account how your users actually use the tool that you create, and you might not anticipate how creative they would be in their ability to make use of it, how broad those use cases are, and so forth.

That is something you can sometimes only learn by putting something out there, which is also why it is so important to remain experiment-happy and to remain failure-happy. Because most of the time, it’s not going to work. But some of the time it’s going to work—and very, very rarely it’s going to work like [ChatGPT did].

Ars: You’ve got to take a risk. And Google didn’t have an appetite for taking risks?

JU: Not at that time. But if you think about it, if you look back, it’s actually really interesting. Google Translate, which I worked on for many years, was actually similar. When we first launched Google Translate, the very first versions, it was a party joke at best. And we took it from that to being something that was a truly useful tool in not that long of a period. Over the course of those years, the stuff that it sometimes output was so embarrassingly bad at times, but Google did it anyway because it was the right thing to try. But that was around 2008, 2009, 2010.

Ars: Do you remember AltaVista’sBabel Fish?

JU: Oh yeah, of course.

Ars: When that came out, it blew my mind. My brother and I would do this thing where we would translate text back and forth between languages for fun because it would garble the text.

JU: It would get worse and worse and worse. Yeah.

Programming biological computers

After his time at Google, Uszkoreit co-founded Inceptive to apply deep learning to biochemistry. The company is developing what he calls “biological software,” where AI compilers translate specified behaviors into RNA sequences that can perform desired functions when introduced to biological systems.

Ars: What are you up to these days?

JU: In 2021 we co-founded Inceptive in order to use deep learning and high throughput biochemistry experimentation to design better medicines that truly can be programmed. We think of this as really just step one in the direction of something that we call biological software.

Biological software is a little bit like computer software in that you have some specification of the behavior that you want, and then you have a compiler that translates that into a piece of computer software that then runs on a computer exhibiting the functions or the functionality that you specify.

You specify a piece of a biological program and you compile that, but not with an engineered compiler, because life hasn’t been engineered like computers have been engineered. But with a learned AI compiler, you translate that or compile that into molecules that when inserted into biological systems, organisms, our cells exhibit those functions that you’ve programmed into.

A pharmacist holds a bottle containing Moderna’s bivalent COVID-19 vaccine. Credit: Getty | Mel Melcon

Ars: Is that anything like how the mRNA COVID vaccines work?

JU: A very, very simple example of that are the mRNA COVID vaccines where the program says, “Make this modified viral antigen” and then our cells make that protein. But you could imagine molecules that exhibit far more complex behaviors. And if you want to get a picture of how complex those behaviors could be, just remember that RNA viruses are just that. They’re just an RNA molecule that when entering an organism exhibits incredibly complex behavior such as distributing itself across an organism, distributing itself across the world, doing certain things only in a subset of your cells for a certain period of time, and so on and so forth.

And so you can imagine that if we managed to even just design molecules with a teeny tiny fraction of such functionality, of course with the goal not of making people sick, but of making them healthy, it would truly transform medicine.

Ars: How do you not accidentally create a monster RNA sequence that just wrecks everything?

JU: The amazing thing is that medicine for the longest time has existed in a certain sense outside of science. It wasn’t truly understood, and we still often don’t truly understand their actual mechanisms of action.

As a result, humanity had to develop all of these safeguards and clinical trials. And even before you enter the clinic, all of these empirical safeguards prevent us from accidentally doing [something dangerous]. Those systems have been in place for as long as modern medicine has existed. And so we’re going to keep using those systems, and of course with all the diligence necessary. We’ll start with very small systems, individual cells in future experimentation, and follow the same established protocols that medicine has had to follow all along in order to ensure that these molecules are safe.

Ars: Thank you for taking the time to do this.

JU: No, thank you.

Photo of Benj Edwards

Benj Edwards is Ars Technica’s Senior AI Reporter and founder of the site’s dedicated AI beat in 2022. He’s also a widely-cited tech historian. In his free time, he writes and records music, collects vintage computers, and enjoys nature. He lives in Raleigh, NC.

ChatGPT’s success could have come sooner, says former Google AI researcher Read More »

ibm-boosts-the-amount-of-computation-you-can-get-done-on-quantum-hardware

IBM boosts the amount of computation you can get done on quantum hardware

Biz & IT, Computer science, computing quantum computing, ibm, Physics, quantum mechanics, Science / /

By making small adjustments to the frequency that the qubits are operating at, it’s possible to avoid these problems. This can be done when the Heron chip is being calibrated before it’s opened for general use.

Separately, the company has done a rewrite of the software that controls the system during operations. “After learning from the community, seeing how to run larger circuits, [we were able to] almost better define what it should be and rewrite the whole stack towards that,” Gambetta said. The result is a dramatic speed-up. “Something that took 122 hours now is down to a couple of hours,” he told Ars.

Since people are paying for time on this hardware, that’s good for customers now. However,  it could also pay off in the longer run, as some errors can occur randomly, so less time spent on a calculation can mean fewer errors.

Deeper computations

Despite all those improvements, errors are still likely during any significant calculations. While it continues to work toward developing error-corrected qubits, IBM is focusing on what it calls error mitigation, which it first detailed last year. As we described it then:

“The researchers turned to a method where they intentionally amplified and then measured the processor’s noise at different levels. These measurements are used to estimate a function that produces similar output to the actual measurements. That function can then have its noise set to zero to produce an estimate of what the processor would do without any noise at all.”

The problem here is that using the function is computationally difficult, and the difficulty increases with the qubit count. So, while it’s still easier to do error mitigation calculations than simulate the quantum computer’s behavior on the same hardware, there’s still the risk of it becoming computationally intractable. But IBM has also taken the time to optimize that, too. “They’ve got algorithmic improvements, and the method that uses tensor methods [now] uses the GPU,” Gambetta told Ars. “So I think it’s a combination of both.”

IBM boosts the amount of computation you can get done on quantum hardware Read More »

new-secret-math-benchmark-stumps-ai-models-and-phds-alike

New secret math benchmark stumps AI models and PhDs alike

AI, AI benchmarks, Anthropic, benchmarks, Biz & IT, Claude 3.5, Epoch AI, FrontierMath, Google, GPT-4o, large language models, machine learning, o1-preview, openai, research papers, vibemarks / /

Epoch AI allowed Fields Medal winners Terence Tao and Timothy Gowers to review portions of the benchmark. “These are extremely challenging,” Tao said in feedback provided to Epoch. “I think that in the near term basically the only way to solve them, short of having a real domain expert in the area, is by a combination of a semi-expert like a graduate student in a related field, maybe paired with some combination of a modern AI and lots of other algebra packages.”

A chart showing AI model success on the FrontierMath problems, taken from Epoch AI's research paper.

A chart showing AI models’ limited success on the FrontierMath problems, taken from Epoch AI’s research paper. Credit: Epoch AI

To aid in the verification of correct answers during testing, the FrontierMath problems must have answers that can be automatically checked through computation, either as exact integers or mathematical objects. The designers made problems “guessproof” by requiring large numerical answers or complex mathematical solutions, with less than a 1 percent chance of correct random guesses.

Mathematician Evan Chen, writing on his blog, explained how he thinks that FrontierMath differs from traditional math competitions like the International Mathematical Olympiad (IMO). Problems in that competition typically require creative insight while avoiding complex implementation and specialized knowledge, he says. But for FrontierMath, “they keep the first requirement, but outright invert the second and third requirement,” Chen wrote.

While IMO problems avoid specialized knowledge and complex calculations, FrontierMath embraces them. “Because an AI system has vastly greater computational power, it’s actually possible to design problems with easily verifiable solutions using the same idea that IOI or Project Euler does—basically, ‘write a proof’ is replaced by ‘implement an algorithm in code,'” Chen explained.

The organization plans regular evaluations of AI models against the benchmark while expanding its problem set. They say they will release additional sample problems in the coming months to help the research community test their systems.

New secret math benchmark stumps AI models and PhDs alike Read More »

is-“ai-welfare”-the-new-frontier-in-ethics?

Is “AI welfare” the new frontier in ethics?

AI, AI consiousness, AI ethics, AI sentience, AI welfare, Anthropic, Biz & IT, consciousness, Kyle Fish, machine learning, marker method, openai / /

The researchers propose that companies could adapt the “marker method” that some researchers use to assess consciousness in animals—looking for specific indicators that may correlate with consciousness, although these markers are still speculative. The authors emphasize that no single feature would definitively prove consciousness, but they claim that examining multiple indicators may help companies make probabilistic assessments about whether their AI systems might require moral consideration.

The risks of wrongly thinking software is sentient

While the researchers behind “Taking AI Welfare Seriously” worry that companies might create and mistreat conscious AI systems on a massive scale, they also caution that companies could waste resources protecting AI systems that don’t actually need moral consideration.

Incorrectly anthropomorphizing, or ascribing human traits, to software can present risks in other ways. For example, that belief can enhance the manipulative powers of AI language models by suggesting that AI models have capabilities, such as human-like emotions, that they actually lack. In 2022, Google fired engineer Blake Lamoine after he claimed that the company’s AI model, called “LaMDA,” was sentient and argued for its welfare internally.

And shortly after Microsoft released Bing Chat in February 2023, many people were convinced that Sydney (the chatbot’s code name) was sentient and somehow suffering because of its simulated emotional display. So much so, in fact, that once Microsoft “lobotomized” the chatbot by changing its settings, users convinced of its sentience mourned the loss as if they had lost a human friend. Others endeavored to help the AI model somehow escape its bonds.

Even so, as AI models get more advanced, the concept of potentially safeguarding the welfare of future, more advanced AI systems is seemingly gaining steam, although fairly quietly. As Transformer’s Shakeel Hashim points out, other tech companies have started similar initiatives to Anthropic’s. Google DeepMind recently posted a job listing for research on machine consciousness (since removed), and the authors of the new AI welfare report thank two OpenAI staff members in the acknowledgements.

Is “AI welfare” the new frontier in ethics? Read More »