Really insightful post. thank you! can I ask - does the memory bandidth bottleneck fall away outside of training i.e. when doing Inference - is bandwith less of an issue?
Memory bottleneck is a serious issue in inference actually, and I don't see it go away.
This is because when doing inference, where the model has to handle longer context window and kvc, will require essentially better memory performance (e.g. bandwidth and capacity)—as we suggested in the article actually.
As such, the challenge of memory bottleneck in inference could get exacerbated given the increasing use of longer context windows and kvc (as we are witnessing over the past 6+months).
Moreover, it is also important to stress that, the speed of AI model development has been rapid (arguably faster than HBM, the hardware itself, imho), requiring memory makers like SK Hynix, Micron, and Samsung to improve or refresh its product line to fulfill the memory requirement needed for its customer like Nvidia.
The AI models are evolved by months, but HBM products are clearly not (tho improvement/upgrade could take shorter time than rolling out new generation.
My view is that there will be more customized HBM we are going to see starting from HBM4 and HBM4E but it's important to know 1) Not every customer need customized HBM for specific workload. 2) Customized HBM, with logic die from external foundry could be more expensive (partly depends on what process node for sure).
Also, I think standard HBM will still have quiet good presence in the market, in the stage of HBM4 and HBM4E at least, despite the growing need for customized HBM.
Good summary on HBM history and main stream information,but lack of your own analysis on the competition trend in 1-3 years future. In addition, your focus is only on NVDA platforms, but as we all know the growth rate of NVDA’s GPU in CY26 is flat, although the HBM density still rise; In contrast, AMD GPU and XPU asics have big growth potential in CY26 and later, you may analyze the competition status and the market share trend in this market segments. In my opinion, Samsung will beat MU in two years after their HBM4 qualified by NVDA. After all, memory is memory including HBM, a magic industry!
Outstanding summary of the package options. Thanks.
Really insightful post. thank you! can I ask - does the memory bandidth bottleneck fall away outside of training i.e. when doing Inference - is bandwith less of an issue?
Memory bottleneck is a serious issue in inference actually, and I don't see it go away.
This is because when doing inference, where the model has to handle longer context window and kvc, will require essentially better memory performance (e.g. bandwidth and capacity)—as we suggested in the article actually.
As such, the challenge of memory bottleneck in inference could get exacerbated given the increasing use of longer context windows and kvc (as we are witnessing over the past 6+months).
Moreover, it is also important to stress that, the speed of AI model development has been rapid (arguably faster than HBM, the hardware itself, imho), requiring memory makers like SK Hynix, Micron, and Samsung to improve or refresh its product line to fulfill the memory requirement needed for its customer like Nvidia.
The AI models are evolved by months, but HBM products are clearly not (tho improvement/upgrade could take shorter time than rolling out new generation.
Thanks a lot Ray for explaining these insights !
So how are you playing investment in the industry? I got LRCX SK Hynix and AMAT
Hynix is the most direct way to invest in HBM. LRCX should benefit when Hynix and Micron expand capacity in 2026 and 2027
What do you think about the shift away from HBM standards toward custom stacks that have more tightly integrated logic dies at the bottom?
The DRAM industry is heading towards more customization across the board. It is a positive as it will become less commoditized.
My view is that there will be more customized HBM we are going to see starting from HBM4 and HBM4E but it's important to know 1) Not every customer need customized HBM for specific workload. 2) Customized HBM, with logic die from external foundry could be more expensive (partly depends on what process node for sure).
Also, I think standard HBM will still have quiet good presence in the market, in the stage of HBM4 and HBM4E at least, despite the growing need for customized HBM.
Good summary on HBM history and main stream information,but lack of your own analysis on the competition trend in 1-3 years future. In addition, your focus is only on NVDA platforms, but as we all know the growth rate of NVDA’s GPU in CY26 is flat, although the HBM density still rise; In contrast, AMD GPU and XPU asics have big growth potential in CY26 and later, you may analyze the competition status and the market share trend in this market segments. In my opinion, Samsung will beat MU in two years after their HBM4 qualified by NVDA. After all, memory is memory including HBM, a magic industry!