There are multiple pathways to achieving a large language model with over 1 trillion parameters, enabling diverse functionalities that enterprises can leverage for AI training and inference infrastructure across numerous applications.
One strategy involves a grand-scale initiative akin to what OpenAI has accomplished with GPT-4 and potentially with GPT-5, as well as what Google has pursued with PaLM and potentially with Gemini. This method entails constructing a single colossal model with trillions of parameters and exposing it to an extensive knowledge corpus during training to perform a multitude of tasks simultaneously. Through strategic optimization, it’s feasible to guide such an immense model to activate specific pathways within the Language Learning Model (LLM) to address distinct types of queries or generate specific outputs. The concept of “pathways” in Google’s Pathways Language Model (PaLM) and its controversial successor, Gemini, stems from this approach.
Training such massive models like GPT-4, GPT-5, PaLM, or Gemini demands significant computational resources, ranging from thousands to tens of thousands of GPUs or alternative AI accelerators, and can span up to three months to digest the vast dataset and establish the requisite relationships for prompt responses. While inference activation can be selective, the training phase is comprehensive, resource-intensive, and expensive.
Conversely, an alternative strategy, termed a composition of experts, championed by AI startup SambaNova Systems, involves amalgamating dozens to hundreds of pretrained models specialized in distinct tasks. These models operate in parallel, collectively presenting themselves as a gargantuan model with over a trillion parameters. The workflow entails delegating tasks to selected models, amalgamating the results, and cross-validating them through iterative refinement processes.
This composition of experts methodology, although not novel, represents a fusion of experts technique predating deep learning and generative AI. It mirrors various ensemble techniques utilized in classical Bayesian machine learning, HPC simulation, and modeling applications over decades.
SambaNova’s CEO, Rodrigo Liang, emphasized the necessity to shift away from constructing ever-expanding models with trillions of parameters and tokens towards a more sustainable approach. SambaNova’s model envisions employing approximately 150 distinct pretrained models tailored for enterprise tasks, integrated through a software router to simulate a unified LLM interface.
The current iteration of SambaNova’s composition of experts model, embodied in the Samba-1 collective comprising 54 models with a cumulative parameter count of 1.3 trillion, underscores a curated selection of experts tailored to meet enterprise demands. This model diversity, underpinned by open-source community contributions, facilitates high-quality checkpoints and enhances transparency for customers utilizing the system.
Moving forward, SambaNova aims to expand its expert ensemble to encompass a total of 100 models before reaching the envisioned 150-model threshold. The company’s commitment to open-source models aligns with its ethos, although enterprises retain the flexibility to license additional models and datasets for private training on SambaNova’s infrastructure or cloud-based GPU platforms.
The innovative aspect of SambaNova’s approach lies in its emphasis on distributed expertise, allowing for diverse viewpoints to cross-validate responses and mitigate biases inherent in monolithic models. The proprietary routing software, dubbed “The Conductor,” orchestrates model activation and aggregation based on prompts, optimizing computational efficiency and latency management.
As SambaNova progresses towards its goal of onboarding 100 Fortune 500 enterprises this year, the viability of its composition of experts stack presents a cost-effective alternative to monolithic models, potentially reshaping the AI landscape. With a burgeoning customer base, escalating revenues, and a compelling value proposition, SambaNova anticipates a transformative growth trajectory in the AI solutions market.
