AI is expanding. The drive to innovate, enhance, segment, and interconnect the utilization of Large Language Models (LLMs) as integral components in Artificial Intelligence (AI)-focused enterprise software applications continues to make headlines.
The CEO of Google DeepMind, Demis Hassabis, has introduced the latest iteration of Google’s Gemini LLM, now at version 1.5 and accessible in a new format. Formerly known as Bard, this LLM has been characterized as a significant leap forward in technological progress, with its Pro version release offered as a developer preview.
Comprehensive Context Comprehension
Hassabis has highlighted Gemini 1.5 for its capability to provide ‘comprehensive context comprehension’ – a term denoting an AI model’s capacity to track vector relationships across extensive passages of text and various data sources, including images, videos, sound files, and more, as the shift towards multi-modal AI integration progresses.
Given the inherent performance decline with the increase in data volume within a piece of information, technology for comprehensive context comprehension is designed to recognize the necessity of establishing connections between data points that are distantly situated and not necessarily at the beginning or end of the information.
Initially introduced last December, this version of the Gemini series is positioned as a ‘research release’ exclusively targeting software application developers and Google cloud clients. In a manner reminiscent of the developer preview system utilized by Redmond in the Microsoft Developer Network (MSDN), the company seems to be forging closer ties with the programming community, possibly prioritizing it over some of the open-source methodologies observed elsewhere. Whether Google aims to garner support, enhance AI safety protocols, or simply exercise greater control and direction is subject to debate.
Mixture-of-Experts
Hassabis has also elucidated how Gemini 1.5 incorporates a novel Mixture-of-Experts (MoE) architecture. This approach entails segmenting neural network architecture logic into smaller specialized ‘expert’ networks, reflecting the current trend towards developing AI with more focused component model structures that excel in specific areas compared to their larger counterparts.
Given the vast amount of information present in any ‘training corpora’ (knowledge or work repository), allowing AI models to concentrate on specific areas enables them to better comprehend their surroundings. Analogous to a diverse group of specialists gathered in a room, where some possess expertise in food science and gastronomy while others specialize in rocket science, MoE models are engineered to selectively activate relevant expert pathways within a neural network architecture only when pertinent.
“By adapting to the input type, MoE models learn to activate the most pertinent expert pathways in their neural network selectively. This specialization significantly boosts the model’s efficiency. Google has been at the forefront of adopting and pioneering the MoE technique for deep learning through research,” elucidated Hassabis in a Google AI blog post. “Our latest advancements in model architecture empower Gemini 1.5 to expedite the learning of complex tasks while maintaining quality, enhancing efficiency in training and deployment. These optimizations facilitate our teams in iterating, training, and delivering more advanced versions of Gemini at an accelerated pace… with further enhancements in progress.”
Token Offerings
In this next-generation rendition of Gemini, Google has augmented its AI models’ processing capacity to consistently handle up to 1 million tokens. As previously discussed, tokens are a fundamental AI technique utilized to segment, define, and categorize words, word components (including letters), or parts of words to enable AI models to attribute relationships and values to information fragments. Tokens can encompass images, videos, audio, or code, with an AI model’s potential knowledge increasing with its token-handling capability.
According to Google engineer Chaim Gartenberg, “The complete 1 million token context window demands substantial computational resources and ongoing optimizations to enhance latency, a challenge we are actively addressing as we scale it.”
Is it secure? Hassabis affirms that this latest technological update adheres to Google’s AI Principles and stringent safety protocols.
“We subject our models to rigorous ethics and safety evaluations. Subsequently, we integrate these research insights into our governance processes, model development, and assessments to continually enhance our AI systems,” he stated. “Since the introduction of [Gemini] 1.0 Ultra in December, our teams have been refining the model to ensure its safety for a broader release. We have also conducted innovative safety risk research and devised red-teaming methodologies to assess various potential risks.”
Gemini Family
Google offers a range of Gemini options, starting from the Nano product tailored for mobile devices, progressing to Gemini Pro for developers, and culminating in the premium Gemini Ultra version. The differentiation in pricing and service capabilities that Google currently provides may influence the product’s future channeling strategy, aligning with the trend for AI to diversify and specialize, as evidenced here.
As the incorporation of Small Language Models (SLMs) or ‘private AI’ alongside other variants becomes imperative in the LLM landscape, the necessity to blend, broaden, and extend AI utilization with precise tokenization control and Mixture-of-Experts (MoE) architectures is increasingly apparent.
While the name Gemini may not directly signify Google’s AI as a ‘twin’ to human existence (as most sources attribute it to the consolidation of Google Brain & Google DeepMind teams), the astrological connotation adds an intriguing layer to its branding.
