Meta recently upgraded its foundational model, Code Llama, to support 70B, establishing it as a viable alternative to closed AI code models.
The latest iteration, Code Llama 70B, is lauded as the most comprehensive and high-performing model to date, capable of handling a larger volume of queries compared to its predecessors. This expanded capacity enables developers to input more prompts during programming sessions, thereby enhancing its accuracy.
Evolved from Llama 2, Code Llama 70B aids developers in generating code snippets from prompts and debugging human-written code. It was trained on an extensive 1TB dataset containing code and related information. Currently hosted on the Hugging Face code repository, the model is offered in three distinct versions. Similar to the original Llama 2 model, it remains freely accessible for research purposes. The inference code for Code Llama models can be accessed on GitHub.
In addition to the core model, there are two specialized variants, namely Code Llama – Python and Code Llama – Instruct, tailored for specific coding languages. Code Llama 70B Python has been enriched with an additional 100 billion tokens of Python code, enhancing its fluency and accuracy in generating Python code. On the other hand, Code Llama 70B Instruct is an enhanced version of Code Llama 2 designed to understand natural language instructions and generate code accordingly. Its advanced capabilities not only enhance the quality but also the efficiency of code generation.
Code Llama 70B Instruct excels in various operations such as data manipulation, sorting, searching, filtering, and implementing algorithms like factorial, Fibonacci, and binary search. It achieved a commendable score of 67.8 on HumanEval, a benchmark dataset evaluating the logic and correctness of code generation models. This score competes with closed models like GPT-4 (68.2) and Gemini Pro (69.4), surpassing previous benchmarks set by open models like CodeGen-16B-Mono (29.3) and StarCoder (40.1).
Moreover, Code Llama 70B Instruct supports AI coding assistants with chat capabilities, providing an interactive experience beyond traditional code completion. While most coding assistants offer inline code suggestions based on comments and naming conventions, chat-based AI assistants offer best practices and deployment scripts to enhance developers’ coding experience.
Foundational models like StarCoder, GPT-4, and CodeGen have significantly influenced AI tools like Code Llama. StarCoder, a Large Language Model for Code, has excelled in popular programming benchmarks, showcasing superior input processing capabilities. GPT-4, a multimodal large language model by OpenAI, is proficient in multiple languages and dialects, empowering tools like GitHub Copilot’s assistant, “Copilot X.”
CodeGen, a family of large language models specializing in program synthesis, has been trained sequentially on datasets like The Pile, BigQuery, and BigPython.
The introduction of Code Llama 70B now provides enterprises with the opportunity to deploy a robust code generation model within their private environments, ensuring control and safeguarding their intellectual property.
