How Hot is the HBM Market?

According to the latest analysis report from Yole Group, due to the demand for artificial intelligence servers exceeding that of other applications, the share of HBM in the total DRAM shipment is expected to rise from about 2% in 2023 to 6% in 2029. Due to the HBM price being much higher than DDR5, in terms of revenue, its share is expected to climb from $14 billion in 2024 to $38 billion in 2029—this follows a year-on-year increase of more than 150% from approximately $5.5 billion in 2023.

Yole Group states that memory suppliers have increased HBM wafer production, with estimated production increasing from 44,000 wafers per month (WPM) in 2022 to 74,000 WPM in 2023, and potentially reaching 151,000 WPM in 2024.

In this booming HBM market, although there are only three players, the intensity of competition is beyond the imagination of many. To put it simply, the haves have a lot and the have-nots have little. The leading player, Hynix, is far ahead in technology and market, taking the most profits. The second-ranked Samsung is actively launching an offensive, taking a share of the profits. As for the third-ranked Micron, due to a misjudgment in the technological path, it has a smaller market share and is currently in a state of catching up, and it is difficult to contribute substantial profits in the short term.

Advertisement

The latest news also indirectly proves this gap. In February, SK Hynix confirmed that its HBM sales in the past few months have set a new record, driving profitability in the fourth quarter, and predicting an upcoming industry recovery. SK Hynix Vice President Kim Ki-tae pointed out that as generative AI services become increasingly diverse and continue to develop, the demand for HBM as an AI storage solution has also seen explosive growth.

More importantly, he mentioned that all of Hynix's HBM for this year has already been sold out. Although 2024 has just begun, the company has started preparing for 2025 in order to maintain its leading position in the market.

As 2024 has just started, SK Hynix is already considering the HBM market for 2025. The pressure on neighboring Samsung and Micron is also huge. While catching up with technology and the market, they are thinking about whether they can bypass the HBM technology and seize the AI market from another direction.

At this time, CXL (Compute Express Link) has once again come into the sight of memory manufacturers.

A New Direction for AI Storage?It is reported that, in order to enhance its competitiveness in the field of AI storage chips, Samsung plans to showcase its technology and vision for CXL DRAM, known as the next-generation technology of HBM, at the MemCon 2024 Global Semiconductor Conference to be held in Silicon Valley next month.

During the two-day event on March 26th and 27th, Samsung Executive Vice President Han Jin-man will deliver the opening speech. In addition, the company's Executive Vice President will share Samsung's CXL technology and vision in a keynote speech titled "Leading HBM and CXL Innovation in the AI Era, Achieving High Memory Bandwidth and High-Capacity Implementation."

What is CXL, and why does Samsung place such importance on it as a substitute for HBM?

Let's first briefly introduce the origin and development of this technology. CXL (Compute Express Link) technology is an open standard for high-speed, high-capacity central processing unit (CPU) to device and CPU to memory connections, specifically designed for high-performance data center computers. It is based on the serial PCI Express (PCIe) physical and electrical interface, including a PCIe-based block input/output protocol (CXL.io) and new high-speed cache consistency protocols for accessing system memory (CXL.cache) and device memory (CXL.mem). The serial communication and pooling functions enable CXL memory to overcome the performance and slot packaging limitations of common DIMM memory when implementing high storage capacity. It was initially launched jointly by Intel, AMD, and other companies and has received extensive support from companies including Google, Microsoft, and others.

The background of CXL technology can be traced back to PCIe (Peripheral Component Interconnect Express) technology, which is a standard interface technology for connecting internal components of a computer. PCIe devices can initiate a DMA to access memory as long as they know the target physical address. Before CXL, there were already various protocols such as OpenCAPI led by IBM, CCIX supported by ARM, GenZ supported by AMD, and Nvlink proposed by Nvidia. Although PCIe has undergone many improvements, it is difficult to meet the high bandwidth and low latency communication requirements between modern computer processors and accelerators. Thus, CXL technology was born.

On March 11, 2019, the CXL 1.0 specification based on PCIe 5.0 was released, allowing the host CPU to use cache consistency protocols to access shared memory on accelerator devices. The improved CXL 1.1 specification was released in June 2019.

On November 10, 2020, the CXL 2.0 specification was released. The new version added support for CXL switches, allowing multiple CXL 1.x and 2.0 devices to be connected to a CXL 2.0 host processor, while aggregating each device to multiple host processors, using distributed shared memory and disaggregated storage configurations. It also implemented device integrity and data encryption. However, compared to CXL 1.x, the bandwidth did not increase because CXL 2.0 still uses the PCIe 5.0 PHY.

On August 2, 2022, the CXL 3.0 specification was released, based on the PCIe 6.0 physical interface and double-bandwidth PAM-4 encoding; new features include structural functions with multi-level switching and multiple device types per port, as well as enhanced point-to-point DMA and memory sharing consistency.

On November 14, 2023, the CXL 3.1 specification was released. The new specification made additional structural improvements to horizontal scaling CXL, new trusted execution environment enhancements, and improvements to memory expanders.From 2019 to 2023, CXL experienced rapid development, with its applications involving two major aspects: the server side and the storage products and solutions. In the past two years, many manufacturers have actually released CXL-related components, products, and complete solutions. However, there is a practical problem that has restricted the development of CXL: due to the delay by processor manufacturers Intel and AMD, the server side has been slow to actually support the CXL protocol, preventing the formation of a complete CXL application environment.

It was not until the end of 2022 and the beginning of 2023 that a turning point finally arrived. With the release of AMD's fourth-generation EPYC (codenamed Genoa) and Intel's fourth-generation Xeon Scalable (codenamed Sapphire Rapids), the new processor platforms finally brought CXL to the server side, filling the missing link in the CXL application environment.

After several years of development, the CXL ecosystem is now quite complete and rich. At the component level, chip suppliers and designers such as Astera Labs, Cadence, Marvell, Microchip, Rambus, Synopsys, Montage Technology, Mobiveil, SmartDV, and Xconn have released corresponding CXL controllers, retimer, and switch products.

At the system level, companies such as Samsung, SK Hynix, Micron, Astera Labs, and Taiwan's SMART Modular Technologies have introduced CXL products that extend storage types. In addition, Elastics.cloud, IntelliProp, UnifabriX, SK Hynix, Samsung/H3 Platform, and Panmnesia have released or demonstrated CXL solutions for storage pooling.

Among them, Samsung, as a director of the CXL Consortium, has spared no effort in promoting CXL.

On May 11, 2021, Samsung announced the launch of the industry's first memory module supporting the new Compute Express Link (CXL) interconnect standard (based on 128 GB DDR5). It stated that this CXL-based module, integrated with Samsung's DDR5 technology, will enable server systems to significantly expand memory capacity and bandwidth, thereby accelerating artificial intelligence (AI) and high-performance computing (HPC) workloads in data centers.

Unlike traditional DDR memory with limited memory channels, Samsung's CXL-supported DDR5 module can expand memory capacity to the TB level while significantly reducing system latency caused by memory caching. In addition to CXL hardware innovation, Samsung has also integrated various controller and software technologies, such as memory mapping, interface conversion, and error management, enabling CPUs or GPUs to recognize CXL-based memory and use it as main memory.

On May 10, 2022, Samsung released a 512 GB version of the CXL memory module. The new CXL DRAM, built with a dedicated integrated circuit (ASIC) CXL controller, is the first product equipped with 512GB DDR5 DRAM. Compared with the previous 128GB version, the memory capacity has increased fourfold, and system latency has been reduced by one-fifth.

Subsequently, Samsung also released an updated version of its open-source Scalable Memory Development Kit (SMDK). This toolkit is a comprehensive software package that allows CXL memory expanders to work seamlessly in heterogeneous memory systems, enabling system developers to integrate CXL memory into various IT systems running artificial intelligence, big data, and cloud applications without modifying the existing application environment.On May 12, 2023, Samsung announced the development of the industry's first 128 GB DRAM supporting CXL 2.0. The new CXL DRAM supports the PCIe 5.0 interface (with an x8 channel) and provides a bandwidth of up to 35 GB per second. It stated that, through close cooperation with Intel, this milestone progress was finally achieved on the Intel Xeon platform.

It is worth noting that the CXL 2.0 DRAM module launched by Samsung is equipped with a controller chip from China's Montage Technology. Montage Technology had released the world's first CXL memory expansion controller chip (MXC) in May 2022. In August 2023, Montage Technology became the first CXL memory expansion chip manufacturer to enter the CXL compliance supplier list (CXL Integrators List) globally.

On December 26, 2023, Samsung announced the optimization of CXL memory for the open-source software provider Red Hat's Enterprise Linux 9.3 (i.e., RHEL 9.3) and verified memory identification and read/write operations in the Red Hat KVM and Podman environments. It stated that this would enable data center customers to use Samsung's CXL memory without additional adjustments to their existing hardware.

Samsung's continuous investment in CXL has made it a leader among current CXL memory manufacturers.

Favored by AI

Samsung's emphasis on CXL is not only due to the desire to take the initiative on the next generation of advanced standards but also because of the advantages of CXL in artificial intelligence.

In recent years, with the rapid growth of data throughput, the limits of existing computing systems have been overwhelmed. The data throughput of artificial intelligence is growing tenfold every year, and the memory capacity of existing computing systems is insufficient to handle the rapidly increasing data volume. HBM has solved the bandwidth problem, but it cannot solve the problem of capacity expansion at the same time.

Currently, a central processing unit (CPU) can accommodate up to 16 DRAMs (up to 8 TB), which is far less than the capacity required for processing the massive data storage used in artificial intelligence and machine learning. As the era of artificial intelligence approaches, the demand for memory platforms that support fast interfaces and easy scalability is becoming increasingly apparent, and the new DRAM module based on CXL may be one of the most promising memory solutions in the future era of artificial intelligence.

Compared with traditional interfaces, the biggest advantage of CXL's DRAM module is its scalability. Other advantages are also significant.

Firstly, it has a strong memory expansion capability. Similar to external storage devices such as solid-state drives (SSDs), when a CXL memory expander is installed in the position where an SSD is inserted, it can expand the DRAM capacity. In other words, by improving the interface, the DRAM capacity of IT systems can be expanded without modifying or completely changing the existing server structure.Then comes the simplification of data processing. A major advantage of memory expanders is efficient data processing. By expanding to a higher bandwidth, it allows different devices to share memory and make more efficient use of their resources. They can use the memory of accelerators as if it were main memory by sharing a common memory area. Devices without their own internal memory can also utilize the main memory and use it as their own.

Finally, there is the acceleration of computing speed. A key feature of the CXL memory expander is to minimize the latency issues (or delays) caused by increased data throughput. The memory expander uses both the accelerator and the CPU to improve the system's computing speed, supporting smoother and faster data processing.

The various advantages of CXL have made it a new favorite with the advent of the artificial intelligence era. Although it is not as dazzling as HBM, its prospects are not much inferior to the latter.

According to a forecast by market research company Yole Group on October 12, by 2028, the global CXL market is expected to reach $15 billion (about 20.1 trillion won). Although currently less than 10% of CPUs are compatible with the CXL standard, it is expected that by 2027, all CPUs worldwide will be compatible with the CXL interface.

The core of the CXL market is DRAM. Yole Group predicts that by 2028, $12 billion (i.e., 80% of the total revenue of the CXL market) will come from DRAM. In today's artificial intelligence era characterized by data explosion, existing computing standards such as PCIe limit the simple installation of DRAM modules and hinder physical scalability. CXL solves these challenges and is expected to drive a new round of development in the DRAM market in the future.

Interestingly, SK Hynix and Micron have also seen the potential of CXL and have become members of the CXL Consortium, but their progress in this technology still has a certain gap compared to Samsung.

On August 1, 2022, SK Hynix developed the first CXL (Compute Express Link) memory sample based on DDR5 DRAM. The sample has a form factor of EDSFF (Enterprise and Data Center Standard Form Factor) E3.S, supports PCIe 5.0 x8 Lane, uses DDR5 standard DRAM, and is equipped with a CXL controller. In October of the same year, Hynix also launched the industry's first CXL-based computing memory solution (CMS) at the OCP Global Summit.

Compared with Samsung, Hynix only demonstrated its first CXL 2.0 product in September 2023, about four months behind Samsung. However, Hynix itself is very concerned. The president of SK Hynix emphasized the importance of the CXL market in October 2023, saying, "We are heavily investing in the development of emerging storage technologies based on CXL, which can play the role of the second and third HBM."Although Micron officially entered the CXL market later, its development in this technology should not be underestimated. In August 2023, Micron announced at the FMS 2023 conference the launch of the first sample of CXL 2.0 memory expansion modules for servers. The module is equipped with 128 GB and 256 GB DRAM and is connected to the CPU via a PCIe 5.0 x8 interface. In terms of supporting new standards, Micron was about two months earlier than Hynix.

In addition to the three major memory manufacturers, the importance of CXL IP is becoming more and more prominent. According to HTF MI Research, it is expected that the CXL IP market will reach $892.3 million by 2029, with a compound annual growth rate of 37.6%. Companies currently providing CXL IP include Cadence, Synopsys, Rambus (PLDA), etc.; facing a broad prospect, domestic manufacturers have also begun to enter this field. It is reported that Wen Dao Yi Xin has launched CXL2.0 IP supporting 20TB-level memory expansion.

So far, many industry insiders are still unfamiliar with the concept of the CXL interface. Although memory manufacturers have started to actively launch samples and mass production due to the demand for AI, specific applications are still relatively few, far less than the popularity of HBM.

In conclusion, as of early 2024, there are still many obstacles to overcome for the implementation of the CXL standard. The CXL 2.0 DRAM developed by Samsung has limited scalability compared to existing standards. Challenges such as developing CXL switching equipment compatible with GPUs, CPUs, and DRAM, designing CXL DRAM modules, and rapidly developing supporting software are all urgent priorities.

Building an ecosystem is also equally important. Renowned companies in the CXL field include South Korean startup Panmesia, which cooperates with global semiconductor companies with CXL 3.0 design assets and integrated solutions, and China's Lanqi Technology, which has CXL DRAM controller design technology. Industry insiders pointed out: "To achieve the ideal scalability in CXL 3.0, it is not enough to just stop at partial technology development; there must be a comprehensive technology covering the entire ecosystem."

The good news is that, so far, the application environment for CXL memory expansion modules has been relatively well-developed. Major memory manufacturers have all launched their own CXL memory expansion module products. Although current servers can only support the CXL 1.1 standard, they can basically connect and run. When the next generation, which supports the CXL 2.0 server, comes online, it will be able to better utilize its advantages, thereby achieving the popularization of this standard.

At the beginning of 2024, we can describe it in one sentence: everything is ready, and CXL is only waiting for the application of this east wind.

Of course, competition in CXL is still inevitable. Some South Korean industry insiders have said, "Although the public doesn't know, Samsung Electronics and SK Hynix are 'going all out' to ensure a leading position in CXL technology, just as they did with HBM."Having wrapped up with HBM, memory manufacturers are now eyeing CXL, signaling the imminent start of a new memory war.