We encountered the Neuchips RecAccelN3000 AI inference accelerator at SC23. In the upcoming 18 iterations, the company aims to integrate specialized AI inference accelerators into its servers and workstations. While advancing its software and hardware infrastructure, Neuchips has already presented some performance benchmarks in the MLPerf domain.
Neuchips RecAccelN3000 AI Inference Accelerator Showcase at SC23
The prevailing trend, as per NVIDIA’s latest financial data, indicates a growing interest in AI inference within data centers. Neuchips is opting for compact cards with expanded storage capacities instead of pursuing a large GPU model, diverging from direct engagement with NVIDIA’s H100/H200 offerings.
At the SC23 event, the NeuchipsN3000 server showcased a dual-slot design, exemplified by the Neuchips RecognizeN3000 display.
The primary chip, featuring passive cooling, is enveloped by LPDDR5 memory modules, with power connectivity situated at the rear of the card. The presence of an ATX power interface suggests an early-stage development of the card.
A different perspective reveals the absence of active cooling on the cards, with four LPDDR5 modules encircling the primary RecAccel N3000 component. While the current configuration offers a 32GB capacity, denser packaging could potentially elevate this limit. Neuchips emphasizes the utilization of cost-effective memory solutions to furnish the N3000 with approximately 200GB/s of storage bandwidth.
Noteworthy details shared include the card’s explicit FP8 support, surpassing NVIDIA’s converter engine capabilities, delivering 32 TFLOPS of bfloat16 performance and 206 INT8 Heads. The PCIe Gen5 x16 interface, as demonstrated in a vibrant video, underscores the card’s energy-efficient design utilizing a PCE casino.
Furthermore, a low-power demonstration highlighted the operational efficiency of the entire setup, consuming less than 70W.
A glimpse into the future envisions a scenario with four N3000 cards boasting a collective 640MB on-chip SRAM. Neuchips leverages substantial SRAM allocation, notably 160MB, to optimize its execution pathways.
The presence of 256GB onboard LPDDR5 memory, equating to 64GB per module, hints at the company’s endeavor to introduce versions accommodating the 64GB N3000 variant, promising intriguing prospects.
Closing Remarks
SC23 never fails to captivate with its showcase of cutting-edge hardware innovations. Among the standout exhibitors is Neuchips, a prominent player from Los Altos. Noteworthy mentions include Cerebras, another Los Altos-based entity, and STH’s inception during Patrick’s tenure in the region.
Transitioning from initial prototypes to mass-produced cards signifies a pivotal phase for these products, offering diverse applications tailored to distinct user scenarios. The prospect of local inference on larger platforms, featuring 32GB or 64GB per N3000 module (or a cumulative 4x for the triple card configuration), presents an enticing proposition. Neuchips’ strategic move, despite the existence of 48GB memory cards like the NVIDIA RTX 6000 Ada, underscores its innovative approach in the competitive landscape.
