DistantNews
Support us
AI's race hits the memory wall
🇳🇱 Netherlands /Technology

AI's race hits the memory wall

From NRC Handelsblad · () Dutch

Translated from Dutch, summarized and contextualized by DistantNews.

At a glance

News Sources not specified Context piece
  • Investors are eager to fund new AI infrastructure, with chip designer Nvidia planning to mobilize $500 billion for data centers.
  • The AI revolution faces a bottleneck due to a shortage of specialized high-bandwidth memory (HBM) chips, slowing down mass adoption.
  • This scarcity drives up prices for AI chips and other memory components, impacting consumer electronics like laptops and gaming consoles.

The artificial intelligence boom continues to attract massive investment, with chip designer Nvidia announcing plans to secure $500 billion in capital for additional data centers through partnerships with asset managers. The AI industry anticipates a future powered by artificial intelligence, yet the progress, despite the hype, is slower than expected.

While AI models are becoming increasingly sophisticated, widespread adoption hinges on the availability of faster and more numerous memory chips. The current "memory wall", a bottleneck caused by the scarcity and insufficient bandwidth of specialized semiconductors, impedes the full potential of AI.

The memory wall is palpable. Compare it to a phone at a pop festival, where only limited bandwidth is available and web pages load slowly.

— unnamed authorDescribing the user experience of AI models facing memory limitations.

This memory crunch is already noticeable in the performance of AI services. Similar to a phone struggling with limited bandwidth at a crowded music festival, AI models can experience delays, with responses appearing slowly. This lag is particularly apparent in the daily use of AI models, as they constantly transfer large amounts of data. Training new models is less affected, but the user experience suffers significantly.

The bandwidth of memory chips cannot keep up with the processing speed of the super-fast computing chips, such as those from Nvidia.

— unnamed authorExplaining the technical bottleneck in AI development.

The core issue lies in the inability of memory chip bandwidth to keep pace with the processing speed of advanced chips like those from Nvidia. Generative AI applications, such as ChatGPT, require substantial data transfer, exacerbating this imbalance. The processors must repeatedly access "weights", the knowledge base of the AI model, stored in memory, leading to delays.

Leading suppliers of high-bandwidth memory (HBM) chips, including South Korea's SK Hynix and Samsung, along with U.S. firm Micron, are capitalizing on the AI demand by charging premium prices. This focus on lucrative HBM chips has led them to reduce production of standard memory chips, causing a general scarcity and price increase for DRAM. Consequently, tech companies like Microsoft and Apple have raised prices for their products, and manufacturers like HP, Asus, and Acer are reportedly considering using Chinese-made memory chips to mitigate the shortage.

The three largest suppliers of these chips are the South Korean SK Hynix and Samsung, and the American Micron.

— unnamed authorIdentifying the key manufacturers of high-bandwidth memory chips.
DistantNews Editorial

Originally published by NRC Handelsblad in Dutch. Translated, summarized, and contextualized by our editorial team with added local perspective. Read our editorial standards.

More Stories

From Our Blog