AI is entering a new phase. The industry is moving beyond simple prompt-response interactions toward agentic systems that sustain multi-step reasoning over long time horizons. As a result, the bottleneck is shifting from raw compute performance to how efficiently data is delivered to and moved across the system.

This puts the spotlight on a long-standing constraint: the memory wall. Compute performance has surged ahead of memory bandwidth, making it increasingly difficult to keep processors fed with data. Overcoming this gap requires tightly integrated advances in memory and advanced packaging.

Moving into the Era of Agentic Workflows

Agentic AI models operate in multi-step, iterative workflows. They break tasks into subtasks, call tools, and maintain context across extended interactions. This requires retaining significantly more information in what’s known as the KV (key–value) cache.

A token, a basic unit of text in the prompt, is represented by “key” (K) and “value” (V) structures that are later used to compute its relationship to the rest of the sequence. Rather than recomputing this information at every step, KV cache stores and reuses intermediate representations.

This greatly improves compute efficiency but shifts the burden to memory. As context length and concurrency grow, KV cache expands rapidly, demanding both high capacity and high bandwidth simultaneously.

Agentic AI Calls for Tiered Memory with DRAM as Backbone     

No single memory technology can meet these requirements. As a result, modern AI systems rely on a tiered memory hierarchy, where data is distributed across memory layers to balance bandwidth, capacity, and cost constraints.

At the highest tier, SRAM delivers the fastest access and best energy efficiency, but its low density and high cost limit it to megabyte-scale on‑chip caches. By contrast, AI workloads require gigabyte‑to‑terabyte scale, far beyond what SRAM can provide.

On the other hand, NAND offers higher storage density, but at the cost of speed and endurance. Holding more bits per cell means tighter voltage margins, higher raw error rates, slower access, and accelerated cell wear. Thus, while NAND can store “cold”, non-active data, it’s far too slow for real-time computation.

This leaves DRAM (both DDR and HBM) as the backbone of active AI memory. The most latency-sensitive data, including the hot KV cache and active working set, is streamed into HBM for maximum bandwidth. Because HBM is capacity-constrained and costly, DRAM acts as its essential extension, staging larger portions of model weights and context that are continuously fed into HBM.

Performance‑critical data must flow through DRAM and HBM, making them indispensable to AI workloads. Given DRAM’s lower bit density relative to NAND, delivering equivalent capacity in a system requires orders of magnitude more DRAM wafers. AI scaling hence disproportionately amplifies DRAM wafer demand.

To keep pace, DRAM is scaling across multiple dimensions: increasing cell density, advancing peripheral circuitry, and adopting bonded architectures. Each step forward raises nanoscale and integration complexity, placing greater demands on innovations in materials engineering and process control.

Advanced Packaging Becomes New Scaling Axis

Even with an optimized memory hierarchy, data movement dominates energy consumption. This makes system‑level integration now as important as—if not more than—device‑level scaling.

In response, the industry has embraced a new 3.5D integration paradigm that combines:

• 2.5D (lateral integration): chiplets placed side‑by‑side
• 3D (vertical stacking): HBM and stacked memory

Leading-edge memory-compute architectures exemplify this approach. HBM, for example, vertically stacks DRAM dies using through-silicon vias (TSVs), microbumps, and emerging hybrid bonding. Multiple HBM stacks are then integrated alongside compute chiplets on a shared interposer using redistribution layers (RDLs).

As a result, HBM bandwidth sets the pace of the system. If data cannot reach the processor fast enough, utilization stalls and token throughput drops. Achieving this performance depends on dense interconnect networks that minimize data movement while enabling higher parallelism and lower latency.

In turn, this architectural shift is reshaping manufacturing. Larger substrates, more complex interconnect structures, and emerging materials such as glass introduce tighter constraints on thickness, uniformity, and warpage. Meeting these challenges requires advanced process control to sustain system-level scaling—particularly eBeam-based metrology and inspection.

Enabling Key Inflections in DRAM and Advanced Packaging

The future of energy-efficient AI will be defined as much by DRAM memory and advanced packaging as by raw compute. These are not just the fastest-growing segments in wafer fab equipment, but also key areas where Applied leads. Both DRAM and advanced packaging face critical inflection points that demand continued advances in materials and process technologies.

In our DRAM and Advanced Packaging Master Class, Applied experts explored the inflections fueling the next era of AI. Access the materials here.