US engineers develop 3D chip that offers order-of-magnitude speed gains, accelerates AI

Engineers in the United States have developed a novel multilayer computer chip with a unique architecture that could help usher in a new era of AI hardware and domestic semiconductor innovation.The team highlighted that in hardware tests and simulations, the new 3D chip outperforms 2D chips by roughly an order of magnitude.Unlike today’s largely flat, 2D chips, the new prototype’s key ultra-thin components rise like stories in a tall building, with vertical wiring acting like numerous high-speed elevators that enable fast, massive data movement.Its record-setting density of vertical connections and carefully interwoven mix of memory and computing units help the chip bypass the bottlenecks that have long slowed improvement in flat designs. This opens the door to a new era of chip production This opens the door to a new era of chip production and innovation, said Subhasish Mitra, the William E.

Ayer Professor in Electrical Engineering and professor of computer science at Stanford University, and principal investigator of a new paper describing the chip presented at the 71st Annual IEEE International Electron Devices Meeting (IEDM). Breakthroughs like this are how we get to the 1,000-fold hardware performance improvements future AI systems will demand. While academic labs have previously built experimental 3D chips, this is the first time such a chip has shown clear performance gains and been manufactured in a commercial foundry.The team also pointed out that on conventional 2D chips, components sit on a single, flat surface with limited, spread-out memory, so data must travel across a few long, crowded routes.

Because the computing elements run much faster than the data can move – and because the chip can’t store enough memory close by – the system ends up constantly waiting on information. Engineers call this bottleneck the “memory wall,” the point at which processing speed outpaces the chip’s ability to deliver data, according to a press release.

Integrating memory and computation vertically By integrating memory and computation vertically, we can move a lot more information much quicker, just as the elevator banks in a high-rise let many residents travel between floors at once, said Tathagata Srimani, assistant professor of electrical and computer engineering at Carnegie Mellon University, the paper’s senior author, who began the work as a postdoctoral fellow advised by Mitra. Engineers at Stanford University, Carnegie Mellon University, University of Pennsylvania, and the Massachusetts Institute of Technology collaborated with SkyWater Technology to develop this 3D chip.

Early hardware tests show that the prototype already outperforms comparable 2D chips by roughly a factor of four. Simulations of taller, future versions – with more stacked layers of memory and compute – point to even greater gains. Designs with additional tiers show up to a twelve-fold improvement on real AI workloads, including those derived from Meta’s open-source LLaMA model, as per the release.

The research team also claims that the design opens a realistic path to 100- to 1,000-fold improvements in energy-delay product (EDP), a key metric that balances speed and energy efficiency. By drastically shortening data movement and adding many more vertical pathways, the chip can achieve both higher throughput and lower energy per operation, a combination long viewed as out of reach for conventional, flat architectures.

The memory wall and the miniaturization wall form a deadly combination, said Robert M. Radway, assistant professor of electrical and systems engineering at the University of Pennsylvania and a co-author of the study. We attacked it head-on by tightly integrating memory and logic and then building upward at extremely high density. It’s like the Manhattan of computing – we can fit more people in less space.