A group of engineers and executives from Databricks, a company specializing in data science and artificial intelligence, convened in virtual meeting rooms via Zoom to review the outcome of their endeavor in developing a significant artificial intelligence language model. The team had invested months of effort and approximately $10 million in training DBRX, a robust language model akin to OpenAI’s ChatGPT. However, the true extent of their creation’s capabilities remained unknown until the results of the final assessments were disclosed.
Jonathan Frankle, the lead neural network engineer at Databricks overseeing the DBRX project, excitedly informed the team, “We have exceeded all expectations.” This announcement elicited cheers, applause, and jubilation from the team members. Despite her usual preference for avoiding tea, Frankle was sipping iced coffee after a strenuous all-nighter to compile the results.
There was a possibility of DBRX being released for public use under an open-source license, enabling individuals to utilize and enhance its functionalities. Frankle presented data illustrating DBRX’s superiority over existing open-source models across various benchmarks measuring its proficiency in answering queries, comprehending text, solving intricate puzzles, and generating high-quality code.
DBRX outperformed Meta’s Llama 2 and Mistral’s Mixtral, two prominent open-source AI models. The CEO of Databricks, Ali Ghodsi, expressed his excitement when the results were unveiled, pondering if they had achieved Elon Musk’s benchmark. Frankle confirmed that they had even surpassed Musk’s xAI’s Grok AI model, humorously adding, “I’ll consider it a triumph if we receive a tweet from him.”
To everyone’s amazement, DBRX demonstrated remarkable proximity to GPT-4, OpenAI’s renowned language model powering ChatGPT, known for its cutting-edge machine intelligence. Frankle proudly declared, “We have established a new standard for open-source large language models.”
Establishing Boundaries
Databricks is revolutionizing the landscape of relational AI by considering the release of DBRX as an open-source project, challenging the dominance of major players in the AI industry like OpenAI and Google, who guard the code for their GPT-4 and Gemini models closely. In contrast, competitors such as Meta have opted to share their models with the public, asserting that it fosters innovation by providing access to a broader audience comprising researchers, entrepreneurs, startups, and established enterprises.
Furthermore, Databricks aims to maintain transparency regarding the development of its open-source model, a practice Meta has not adopted in detailing the creation of its Llama 2 model. The company allowed WIRED to observe the decision-making process of Databricks engineers during the initial phases of the multimillion-dollar training program for DBRX, intending to publish a comprehensive blog post outlining the model’s creation process. This transparency sheds light on the complexities and challenges involved in crafting cutting-edge AI models while hinting at potential cost reductions due to recent advancements in the field. This suggests that the pace of AI innovation is poised to accelerate, bolstered by the availability of open-source models like DBRX.
Ali Farhadi, CEO of the Allen Institute for AI, emphasizes the critical need for enhanced transparency in AI model development. In an industry increasingly veering towards secrecy for competitive advantage, Farhadi stresses the importance of transparency, particularly concerning the potential risks associated with advanced AI models. He applauds initiatives towards openness, foreseeing a substantial shift towards open models within the market.
Databricks has a strategic motive for its openness. Despite tech giants like Google embracing AI advancements in the past year, Ghodsi notes that many large corporations in diverse sectors have yet to leverage this technology within their data infrastructure. Databricks aims to assist industries such as finance and healthcare, which exhibit a strong demand for ChatGPT-like tools but harbor reservations about entrusting sensitive data to cloud platforms.
“We define it as data intelligence—the capacity to comprehend your data,” Ghodsi explains. Databricks offers tailored DBRX solutions for clients, enabling businesses to harness the model’s capabilities for their specific needs. Ghodsi underscores the viability of deploying a DBRX-scale solution for major enterprises, citing it as a lucrative business opportunity for Databricks. Following the acquisition of MosaicML, a startup specializing in efficient AI model development, Databricks has integrated key personnel involved in DBRX’s construction, including Frankle, marking a significant milestone in their AI journey.
Technical Insights
Like its counterparts, DBRX operates as a colossal artificial neural network, a mathematical framework inspired by biological neurons, fed with extensive text data. Built on the transformer architecture introduced by Google in 2017, DBRX and similar models have undergone training on vast text datasets sourced from diverse platforms, a process spanning several months. The scalability of the model and the dataset it learns from are pivotal factors enhancing its capabilities, coherence, and apparent intelligence in generating outputs.
The pursuit of larger-scale models remains a focal point for AI pioneers like OpenAI. Sam Altman, CEO of OpenAI, has sought substantial funding for AI-specialized chip development, underlining the significance of scale in model creation. Frankle highlights the myriad decisions involved in constructing an advanced neural network, drawing insights from research papers and community knowledge to optimize training methodologies. Managing a network of interconnected computers during training poses significant challenges, requiring expertise in network infrastructure.
The quality and preprocessing of data significantly influence the final model output, elucidating Databricks’ decision to withhold informal data disclosures. Naveen Rao, a Databricks vice president and former CEO of MosaicML, emphasizes the pivotal role of data quality, preparation, and filtering in shaping model performance. These models are a direct reflection of the data they are trained on, underscoring the paramount importance of data quality in ensuring model efficacy.
Continual advancements in AI model design aim to enhance performance and efficiency. The “mixture of experts” approach, which enables selective model components to respond based on content relevance, has revolutionized training and operational efficiency. With approximately 136 billion parameters, DBRX surpasses Llama 2, Mixtral, and Grok in parameter count, yet selectively activates around 36 billion parameters on average for query processing. Databricks’ hardware optimizations have bolstered training efficiency by 30% to 50%, facilitating quicker responses to inquiries while minimizing energy consumption.
Unveiling Potential
The decision-making process behind training a large AI model is a blend of emotional and technical considerations. Two weeks before the model’s anticipated release, the Databricks team faced a pivotal decision amidst a multimillion-dollar challenge.
After intensive training using 3,072 potent Nvidia H100s GPUs over two months, DBRX showcased impressive benchmark results, with additional supercomputer idle time remaining. Team members deliberated various strategies on Slack to maximize the remaining computing resources. Options ranged from creating smaller model versions for hobbyists to focusing on targeted data curation to enhance specific model capabilities through curriculum learning. Ultimately, the team, guided by Frankle, opted for a data-centric approach, a decision that proved immensely fruitful in the subsequent weeks.
Frankle, initially uncertain about DBRX’s coding capabilities, was pleasantly surprised by the model’s superior performance on standard coding benchmarks. The success of DBRX underscored the team’s meticulous approach and validated their strategic decisions. The team’s commitment to exploring the model’s evolution post-training reflects their dedication to advancing AI research and understanding the model’s developmental trajectory.
Risk Evaluation
The release of the final DBRX version as an open-source model marks a significant milestone in AI accessibility and innovation. While concerns persist regarding the potential risks associated with powerful AI models, Databricks has rigorously tested DBRX’s safety protocols and remains committed to ongoing assessments.
Stella Biderman, from EleutherAI, emphasizes that open models do not inherently pose elevated risks compared to closed models, challenging prevailing apprehensions surrounding AI openness. Advocates for open-source AI projects contend that transparency fosters a deeper understanding of AI model risks and benefits, advocating for regulatory frameworks that accommodate open AI initiatives.
Databricks remains optimistic about DBRX’s dual role in advancing AI research and empowering developers to explore new frontiers in AI innovation. Frankle envisions DBRX as a tool for researchers to delve into AI intricacies, offering valuable insights for model refinement and development. The team’s future research endeavors aim to unravel the model’s post-training evolution, shedding light on the dynamic nature of advanced AI models and the scientific opportunities they present.
