AI chips

intel’s-“gaudi-3”-ai-accelerator-chip-may-give-nvidia’s-h100-a-run-for-its-money

Intel’s “Gaudi 3” AI accelerator chip may give Nvidia’s H100 a run for its money

Adventures in Matrix Multiplication —

Intel claims 50% more speed when running AI language models vs. the market leader.

An Intel handout photo of the Gaudi 3 AI accelerator.

Enlarge / An Intel handout photo of the Gaudi 3 AI accelerator.

On Tuesday, Intel revealed a new AI accelerator chip called Gaudi 3 at its Vision 2024 event in Phoenix. With strong claimed performance while running large language models (like those that power ChatGPT), the company has positioned Gaudi 3 as an alternative to Nvidia’s H100, a popular data center GPU that has been subject to shortages, though apparently that is easing somewhat.

Compared to Nvidia’s H100 chip, Intel projects a 50 percent faster training time on Gaudi 3 for both OpenAI’s GPT-3 175B LLM and the 7-billion parameter version of Meta’s Llama 2. In terms of inference (running the trained model to get outputs), Intel claims that its new AI chip delivers 50 percent faster performance than H100 for Llama 2 and Falcon 180B, which are both relatively popular open-weights models.

Intel is targeting the H100 because of its high market share, but the chip isn’t Nvidia’s most powerful AI accelerator chip in the pipeline. Announcements of the H200 and the Blackwell B200 have since surpassed the H100 on paper, but neither of those chips is out yet (the H200 is expected in the second quarter of 2024—basically any day now).

Meanwhile, the aforementioned H100 supply issues have been a major headache for tech companies and AI researchers who have to fight for access to any chips that can train AI models. This has led several tech companies like Microsoft, Meta, and OpenAI (rumor has it) to seek their own AI-accelerator chip designs, although that custom silicon is typically manufactured by either Intel or TSMC. Google has its own line of tensor processing units (TPUs) that it has been using internally since 2015.

Given those issues, Intel’s Gaudi 3 may be a potentially attractive alternative to the H100 if Intel can hit an ideal price (which Intel has not provided, but an H100 reportedly costs around $30,000–$40,000) and maintain adequate production. AMD also manufactures a competitive range of AI chips, such as the AMD Instinct MI300 Series, that sell for around $10,000–$15,000.

Gaudi 3 performance

An Intel handout featuring specifications of the Gaudi 3 AI accelerator.

Enlarge / An Intel handout featuring specifications of the Gaudi 3 AI accelerator.

Intel says the new chip builds upon the architecture of its predecessor, Gaudi 2, by featuring two identical silicon dies connected by a high-bandwidth connection. Each die contains a central cache memory of 48 megabytes, surrounded by four matrix multiplication engines and 32 programmable tensor processor cores, bringing the total cores to 64.

The chipmaking giant claims that Gaudi 3 delivers double the AI compute performance of Gaudi 2 using 8-bit floating-point infrastructure, which has become crucial for training transformer models. The chip also offers a fourfold boost for computations using the BFloat 16-number format. Gaudi 3 also features 128GB of the less expensive HBMe2 memory capacity (which may contribute to price competitiveness) and features 3.7TB of memory bandwidth.

Since data centers are well-known to be power hungry, Intel emphasizes the power efficiency of Gaudi 3, claiming 40 percent greater inference power-efficiency across Llama 7B and 70B parameters, and Falcon 180B parameter models compared to Nvidia’s H100. Eitan Medina, chief operating officer of Intel’s Habana Labs, attributes this advantage to Gaudi’s large-matrix math engines, which he claims require significantly less memory bandwidth compared to other architectures.

Gaudi vs. Blackwell

An Intel handout photo of the Gaudi 3 AI accelerator.

Enlarge / An Intel handout photo of the Gaudi 3 AI accelerator.

Last month, we covered the splashy launch of Nvidia’s Blackwell architecture, including the B200 GPU, which Nvidia claims will be the world’s most powerful AI chip. It seems natural, then, to compare what we know about Nvidia’s highest-performing AI chip to the best of what Intel can currently produce.

For starters, Gaudi 3 is being manufactured using TSMC’s N5 process technology, according to IEEE Spectrum, narrowing the gap between Intel and Nvidia in terms of semiconductor fabrication technology. The upcoming Nvidia Blackwell chip will use a custom N4P process, which reportedly offers modest performance and efficiency improvements over N5.

Gaudi 3’s use of HBM2e memory (as we mentioned above) is notable compared to the more expensive HBM3 or HBM3e used in competing chips, offering a balance of performance and cost-efficiency. This choice seems to emphasize Intel’s strategy to compete not only on performance but also on price.

As far as raw performance comparisons between Gaudi 3 and the B200, that can’t be known until the chips have been released and benchmarked by a third party.

As the race to power the tech industry’s thirst for AI computation heats up, IEEE Spectrum notes that the next generation of Intel’s Gaudi chip, code-named Falcon Shores, remains a point of interest. It also remains to be seen whether Intel will continue to rely on TSMC’s technology or leverage its own foundry business and upcoming nanosheet transistor technology to gain a competitive edge in the AI accelerator market.

Intel’s “Gaudi 3” AI accelerator chip may give Nvidia’s H100 a run for its money Read More »

india’s-plan-to-let-1998-digital-trade-deal-expire-may-worsen-chip-shortage

India’s plan to let 1998 digital trade deal expire may worsen chip shortage

India’s plan to let 1998 digital trade deal expire may worsen chip shortage

India’s plan to let a moratorium on imposing customs duties on cross-border digital e-commerce transactions expire may end up hurting India’s more ambitious plans to become a global chip leader in the next five years, Reuters reported.

It could also worsen the global chip shortage by spiking semiconductor industry costs at a time when many governments worldwide are investing heavily in expanding domestic chip supplies in efforts to keep up with rapidly advancing technologies.

Early next week, world leaders will convene at a World Trade Organization (WTO) meeting, just before the deadline to extend the moratorium hits in March. In place since 1998, the moratorium has been renewed every two years since—but India has grown concerned that it’s losing significant revenues from not imposing taxes as demand rises for its digital goods, like movies, e-books, or games.

Hoping to change India’s mind, a global consortium of semiconductor industry associations known as the World Semiconductor Council (WSC) sent a letter to Indian Prime Minister Narendra Modi on Thursday.

Reuters reviewed the letter, reporting that the WSC warned Modi that ending the moratorium “would mean tariffs on digital e-commerce and an innumerable number of transfers of chip design data across countries, raising costs and worsening chip shortages.”

Pointing to Modi’s $10 billion semiconductor incentive package—which Modi has said is designed to advance India’s industry through “giant leaps” in its mission to become a technology superpower—the WSC cautioned Modi that pushing for customs duties may dash those global chip leader dreams.

Studies suggest that India should be offering tax incentives, not potentially threatening to impose duties on chip design data. That includes a study from earlier this year, released after the Semiconductor Industry Association and the India Electronics and Semiconductor Association commissioned a report from the Information Technology and Innovation Foundation (ITIF).

ITIF’s goal was to evaluate “India’s existing semiconductor ecosystem and policy frameworks” and offer “recommendations to facilitate longer-term strategic development of complementary semiconductor ecosystems in the US and India,” a press release said, partly in order to “deepen commercial ties” between the countries. The Prime Minister’s Office (PMO) has also reported a similar goal to deepen commercial ties with the European Union.

Among recommendations to “strengthen India’s semiconductor competitiveness,” ITIF’s report encouraged India to advance cooperation with the US and introduce policy reforms that “lower the cost of doing business for semiconductor companies in India”—by “offering tax breaks to chip companies” and “expediting clearance times for goods entering the country.”

Because the duties could spike chip industry costs at a time when global cross-border data transmissions are expected to reach $11 trillion by 2025, WSC wrote, the duties may “impede India’s efforts to advance its semiconductor industry and attract semiconductor investment,” which could negatively impact “more than 20 percent of the world’s semiconductor design workforce,” which is based in India.

The prime minister’s office did not immediately respond to Ars’ request to comment.

India’s plan to let 1998 digital trade deal expire may worsen chip shortage Read More »

us-funds-$5b-chip-effort-after-lagging-on-semiconductor-innovation

US funds $5B chip effort after lagging on semiconductor innovation

Now hiring? —

US had failed to fund the “science half” of CHIPS and Science Act, critic said.

US President Joe Biden speaks before signing the CHIPS and Science Act of 2022.

Enlarge / US President Joe Biden speaks before signing the CHIPS and Science Act of 2022.

The Biden administration announced investments Friday totaling more than $5 billion in semiconductor research and development intended to re-establish the US as a global leader manufacturing the “next generation of semiconductor technologies.”

Through sizeable investments, the US will “advance US leadership in semiconductor R&D, cut down on the time and cost of commercializing new technologies, bolster US national security, and connect and support workers in securing good semiconductor jobs,” a White House press release said.

Currently, the US produces “less than 10 percent” of the global chips supply and “none of the most advanced chips,” the White House said. But investing in programs like the National Semiconductor Technology Center (NSTC)—considered the “centerpiece” of the CHIPS and Science Act’s four R&D programs—and training a talented workforce could significantly increase US production of semiconductors that the Biden administration described as the “backbone of the modern economy.”

The White House projected that the NSTC’s workforce activities would launch in the summer of 2024. The Center’s prime directive will be developing new semiconductor technologies by “supporting design, prototyping, and piloting and through ensuring innovators have access to critical capabilities.”

Moving forward, the NSTC will operate as a public-private consortium, involving both government and private sector institutions, the White House confirmed. It will be run by a recently established nonprofit called the National Center for the Advancement of Semiconductor Technology (Natcast), which will coordinate with the secretaries of Commerce, Defense, and Energy, as well as the National Science Foundation’s director. Any additional stakeholders can provide input on the NSTC’s goals by joining the NSTC Community of Interest at no cost.

The National Institute of Standards and Technology (NIST) has explained why achieving the NSTC’s mission to develop cutting-edge semiconductor technology in the US will not be easy:

The smallest dimensions of leading-edge semiconductor devices have reached the atomic scale and the complexity of the circuit architecture is increasing exponentially with the use of three-dimensional structures, the incorporation of new materials, and improvements in the thousands of process steps needed to make advanced chips. Into the future, as new applications demand higher-performance semiconductors, their design and production will become even more complex. This complexity makes it increasingly difficult and costly to implement innovations because of the dependencies between design and manufacturing, between manufacturing steps, and between front-end and back-end processes.

The complexity of keeping up with semiconductor tech is why it’s critical for the US to create clear pathways for skilled workers to break into this burgeoning industry. The Biden administration said it plans to invest “at least hundreds of millions of dollars in the NSTC’s workforce efforts,” creating a Workforce Center of Excellence with locations throughout the US and piloting new training programs, including initiatives engaging underserved communities. The Workforce Center will start by surveying best practices in semiconductor education programs, then establish a baseline program to attract workers seeking dependable paths to break into the industry.

Last year, the Semiconductor Industry Association (SIA) released a study showing that the US was not adequately preparing a highly skilled workforce. Between “67,000, or 58 percent, of projected new jobs, may remain unfulfilled at the current trajectory,” SIA estimated.

A skilled workforce is just part of the equation, though. The US also needs facilities where workers can experiment with new technologies without breaking the bank. To that end, the Department of Commerce announced it would be investing “at least $200 million” in a first-of-its-kind CHIPS Manufacturing USA Institute. That institute will “allow innovators to replicate and experiment with physical manufacturing processes at low cost.”

Other Commerce Department investments announced include “up to $300 million” for advanced packaging R&D necessary for discovering new applications for semiconductor technologies and over $100 million in funding for dozens of projects to help inventors “more easily scale innovations into commercial products.”

A Commerce Department spokesperson told Ars that “the location of the NSTC headquarters has not yet been determined” but will “directly support the NSTC research strategy and give engineers, academics, researchers, engineers at startups, small and large companies, and workforce developers the capabilities they need to innovate.” In 2024, NSTC’s efforts to kick off research appear modest, with the center expecting to prioritize engaging community members and stakeholders, launching workforce programs, and identifying early start research programs.

So far, Biden’s efforts to ramp up semiconductor manufacturing in the US have not gone smoothly. Earlier this year, TSMC predicted further delays at chips plants under construction in Arizona and confirmed that the second plant would not be able to manufacture the most advanced chips, as previously expected.

That news followed criticism from private entities last year. In November, Nvidia CEO Jensen Huang predicted that the US was “somewhere between a decade and two decades away” from semiconductor supply chain independence. The US Chamber of Commerce said last August that the reason why the US remained so far behind was because the US had so far failed to prioritize funding in the “science half” of the CHIPS and Science Act.

In 2024, the Biden administration appears to be attempting to finally start funding a promised $11 billion total in research and development efforts. Once NSTC kicks off research, the pressure will be on to chase the Center’s highest ambition of turning the US into a consistent birthplace of life-changing semiconductor technologies once again.

US funds $5B chip effort after lagging on semiconductor innovation Read More »

report:-sam-altman-seeking-trillions-for-ai-chip-fabrication-from-uae,-others

Report: Sam Altman seeking trillions for AI chip fabrication from UAE, others

chips ahoy —

WSJ: Audacious $5-$7 trillion investment would aim to expand global AI chip supply.

WASHINGTON, DC - JANUARY 11: OpenAI Chief Executive Officer Sam Altman walks on the House side of the U.S. Capitol on January 11, 2024 in Washington, DC. Meanwhile, House Freedom Caucus members who left a meeting in the Speakers office say that they were talking to the Speaker about abandoning the spending agreement that Johnson announced earlier in the week. (Photo by Kent Nishimura/Getty Images)

Enlarge / OpenAI Chief Executive Officer Sam Altman walks on the House side of the US Capitol on January 11, 2024, in Washington, DC. (Photo by Kent Nishimura/Getty Images)

Getty Images

On Thursday, The Wall Street Journal reported that OpenAI CEO Sam Altman is in talks with investors to raise as much as $5 trillion to $7 trillion for AI chip manufacturing, according to people familiar with the matter. The funding seeks to address the scarcity of graphics processing units (GPUs) crucial for training and running large language models like those that power ChatGPT, Microsoft Copilot, and Google Gemini.

The high dollar amount reflects the huge amount of capital necessary to spin up new semiconductor manufacturing capability. “As part of the talks, Altman is pitching a partnership between OpenAI, various investors, chip makers and power providers, which together would put up money to build chip foundries that would then be run by existing chip makers,” writes the Wall Street Journal in its report. “OpenAI would agree to be a significant customer of the new factories.”

To hit these ambitious targets—which are larger than the entire semiconductor industry’s current $527 billion global sales combined—Altman has reportedly met with a range of potential investors worldwide, including sovereign wealth funds and government entities, notably the United Arab Emirates, SoftBank CEO Masayoshi Son, and representatives from Taiwan Semiconductor Manufacturing Co. (TSMC).

TSMC is the world’s largest dedicated independent semiconductor foundry. It’s a critical linchpin that companies such as Nvidia, Apple, Intel, and AMD rely on to fabricate SoCs, CPUs, and GPUs for various applications.

Altman reportedly seeks to expand the global capacity for semiconductor manufacturing significantly, funding the infrastructure necessary to support the growing demand for GPUs and other AI-specific chips. GPUs are excellent at parallel computation, which makes them ideal for running AI models that heavily rely on matrix multiplication to work. However, the technology sector currently faces a significant shortage of these important components, constraining the potential for AI advancements and applications.

In particular, the UAE’s involvement, led by Sheikh Tahnoun bin Zayed al Nahyan, a key security official and chair of numerous Abu Dhabi sovereign wealth vehicles, reflects global interest in AI’s potential and the strategic importance of semiconductor manufacturing. However, the prospect of substantial UAE investment in a key tech industry raises potential geopolitical concerns, particularly regarding the US government’s strategic priorities in semiconductor production and AI development.

The US has been cautious about allowing foreign control over the supply of microchips, given their importance to the digital economy and national security. Reflecting this, the Biden administration has undertaken efforts to bolster domestic chip manufacturing through subsidies and regulatory scrutiny of foreign investments in important technologies.

To put the $5 trillion to $7 trillion estimate in perspective, the White House just today announced a $5 billion investment in R&D to advance US-made semiconductor technologies. TSMC has already sunk $40 billion—one of the largest foreign investments in US history—into a US chip plant in Arizona. As of now, it’s unclear whether Altman has secured any commitments toward his fundraising goal.

Updated on February 9, 2024 at 8: 45 PM Eastern with a quote from the WSJ that clarifies the proposed relationship between OpenAI and partners in the talks.

Report: Sam Altman seeking trillions for AI chip fabrication from UAE, others Read More »

report:-black-market-keeps-nvidia-chips-flowing-to-china-military,-government

Report: Black market keeps Nvidia chips flowing to China military, government

Out of control —

Unknown suppliers keep Nvidia’s most advanced chips within China’s reach.

An Nvidia H100 graphics processor chip.

Enlarge / An Nvidia H100 graphics processor chip.

China is still finding ways to skirt US export controls on Nvidia chips, Reuters reported.

A Reuters review of publicly available tender documents showed that last year dozens of entities—including “Chinese military bodies, state-run artificial intelligence research institutes, and universities”—managed to buy “small batches” of restricted Nvidia chips.

The US has been attempting to block China from accessing advanced chips needed to achieve AI breakthroughs and advance modern military technologies since September 2022, citing national security risks.

Reuters’ report shows just how unsuccessful the US effort has been to completely cut off China, despite repeated US attempts to expand export controls and close any loopholes discovered over the past year.

China’s current suppliers remain “largely unknown,” but Reuters confirmed that “neither Nvidia” nor its approved retailers counted “among the suppliers identified.”

An Nvidia spokesperson told Reuters that the company “complies with all applicable export control laws and requires its customers to do the same.”

“If we learn that a customer has made an unlawful resale to third parties, we’ll take immediate and appropriate action,” Nvidia’s spokesperson said.

It’s also still unclear how suppliers are procuring the chips, which include Nvidia’s most powerful chips, the A100 and H100, in addition to slower modified chips developed just for the Chinese market, the A800 and H800. The former chips were among the first banned, while the US only began restricting the latter chips last October.

Among military and government groups purchasing chips were two top universities that the US Department of Commerce has linked to China’s principal military force, the People’s Liberation Army, and labeled as a threat to national security. Last May, the Harbin Institute of Technology purchased six Nvidia A100 chips to “train a deep-learning model,” and in December 2022, the University of Electronic Science and Technology of China purchased one A100 for purposes so far unknown, Reuters reported.

Other entities purchasing chips include Tsinghua University—which is seemingly gaining the most access, purchasing “some 80 A100 chips since the 2022 ban”—as well as Chongqing University, Shandong Chengxiang Electronic Technology, and “one unnamed People’s Liberation Army entity based in the city of Wuxi, Jiangsu province.”

In total, Reuters reviewed more than 100 tenders showing state entities purchasing A100 chips and dozens of tenders documenting A800 purchases. Purchases include “brand new” chips and have been made as recently as this month.

Most of the chips purchased by Chinese entities are being used for AI, Reuters reported. None of the purchasers or suppliers provided comments in Reuters’ report.

Nvidia’s highly sought-after chips are graphic processing units capable of crunching large amounts of data at the high speeds needed to fuel AI systems. For now, these chips remain irreplaceable to Chinese firms hoping to compete globally, as well as nationally, with China’s dominant technology players, such as Huawei, Reuters suggested.

While the “small batches” of chips found indicate that China could still be accessing enough Nvidia chips to enhance “existing AI models,” Reuters pointed out that US curbs are effectively stopping China from bulk-ordering chips at quantities needed to develop new AI systems. Running a “model similar to OpenAI’s GPT would require more than 30,000 Nvidia A100 cards,” research firm TrendForce reported last March.

For China, which has firmly opposed the US export controls every step of the way, these curbs remain a persistent problem despite maintaining access through the burgeoning black market. On Monday, a Bloomberg report flagged the “steepest drop” in the value of China chip imports ever recorded, falling by more than 15 percent.

China’s black market for AI chips

The US still must confront whether it’s possible to block China from accessing advanced chips without other allied nations joining the effort by lobbying their own export controls.

In October 2022, a senior US official warned that without more cooperation, US curbs will “lose effectiveness over time.” A former top Commerce Department official, Kevin Wolf, told The Wall Street Journal last year that it’s “insanely difficult to enforce” US export controls on transactions overseas.

Part of the problem, sources told Reuters in October 2023, is that overseas subsidiaries were “easily” smuggling restricted chips into China or else providing remote access to chips to China-based employees.

On top of that activity, a black market for chips developed quickly, selling “excess stock that finds its way to the market after Nvidia ships large quantities to big US firms” or else chips imported “through companies locally incorporated in places such as India, Taiwan, and Singapore,” Reuters reported.

The US has maintained that its plan is not to ensure that China has absolutely no access but to limit access enough to keep China from getting ahead. But Nvidia CEO Jensen Huang has warned that curbs could have the opposite effect. While China finds ways to skirt the bans and acquire chips to “inspire” advancements, US companies that have been impacted by export controls restricting sales in China could lose so much revenue that they fall behind competitively, Huang predicted.

One example likely worrying to Huang and other tech firms came last November, when Huawei shocked the US government by unveiling a cutting-edge chip that seemed to prove US sanctions weren’t doing much to limit China’s ability to compete.

Report: Black market keeps Nvidia chips flowing to China military, government Read More »