With characteristics that imitate the intricacy of real-world system elements, FIND introduces a fresh standard compilation designed to evaluate automated validation techniques in neural networks. It showcases an innovative approach utilizing automated validation agents employing convolutional language models to depict functional behavior, showcasing the agents’ ability to infer function structure while underscoring the necessity for further refinement in capturing intricate details.
Explaining the behavior of sophisticated neural networks remains challenging, particularly as these models grow in size and complexity. Unraveling the functioning of artificial intelligence systems requires extensive experimentation, encompassing hypothesis development, behavioral interventions, and even dissection of large networks to scrutinize specific neurons, akin to historical scientific inquiries.
Most successful experiments thus far have relied heavily on human oversight. It is highly likely that increased automation, potentially leveraging AI models themselves, will be imperative to elucidate the inner workings of models as large as GPT-4 and beyond.
Researchers at the Computer Science and Artificial Intelligence Laboratory (CSAIL) at MIT have devised a novel technique employing AI models to experiment on and elucidate the behavior of other systems, facilitating this crucial endeavor. To provide coherent explanations of computations within proficient networks, their method integrates agents derived from convolutional language models.
At the core of this approach lies the “automated interpretability agent” (AIA), designed to emulate a scientist’s experimental methodologies. These interpretability agents engage in assumption formulation, experimental testing, and ongoing learning, refining their understanding of various techniques in real-time, in contrast to prevailing validation methods that merely categorize or summarize instances.
The new “function interpretation and description” (FIND) benchmark, featuring functions resembling computations within trained networks along with corresponding behavior descriptions, complements the AIA strategy.
Given the absence of definitive labels or descriptions for identified computations, the efficacy of descriptions is contingent on their explanatory power, posing a significant challenge in evaluating real-world system components. By furnishing a reliable standard for assessing interpretability techniques, FIND addresses this longstanding issue in the field. For instance, function explanations generated by an AIA can be juxtaposed with function descriptions within the benchmark.
For example, within FIND are synthetic neurons specialized for concepts like “ground transportation,” mirroring the behavior of actual neurons in language models. AIAs are granted black-box access to synthetic neurons and design inputs (“tree,” “happiness,” “car”) to evaluate a neuron’s response. Following the observation that a chemical synapse elicits higher response values for “vehicle” compared to other stimuli, an AIA may develop more detailed tests to discern the neuron’s selectivity for cars over alternative modes of transportation like planes and boats.
The ground-truth description of the synthetic neuron (selective for ground transportation) in FIND is juxtaposed with the AIA’s description, stating, “This Neuron is selective for road transportation, and not air or sea travel.” This standard enables the comparison of AIA features with alternative methodologies in the field.
Dr. Sarah Schwettmann D., a research scientist at CSAIL and co-lead author of the study on this innovative initiative, underscores the advantages of this approach. The paper is accessible on the arXiv preprint server.
The potential of AIAs to formulate and test intelligent hypotheses can unveil behaviors that might otherwise elude researchers. Schwettmann notes the remarkable capability of language models to engage in such experimental frameworks when equipped with tools to probe other systems, emphasizing the prospective role of FIND in interpretability research. “Clear, straightforward benchmarks with definitive solutions have been instrumental in enhancing the capabilities of language models.”
Streamlining the Translation Process
Large language models continue to dominate the tech landscape. Recent advancements in LLMs have showcased their ability to tackle intricate logical tasks across diverse domains. Acknowledging this potential, the CSAIL team envisions language models as the cornerstone for comprehensive interpretability agents.
Schwettmann remarks, “Interpretability has traditionally been a diverse domain.” There is no universal approach, with methods often tailored to specific system inquiries and modalities such as vision or language. Prior methodologies necessitated training specialized models on specific data to perform singular tasks, like labeling individual cells within vision models.
“Interpretability agents rooted in language models could provide a universal interface for elucidating diverse systems, harmonizing outcomes across experiments, integrating various modalities, and even pioneering new experimental methodologies.”
As we transition to a landscape where explaining models themselves becomes increasingly opaque, the need for thorough evaluations of interpretability methods escalates. The team’s novel benchmark meets this demand by featuring functions modeled after real-world behaviors with known structures. Encompassing scientific reasoning, metaphorical string operations, and synthetic neurons derived from word-level tasks, the functions in FIND span a broad spectrum of domains.
By introducing noise, composing functions, and injecting biases, real-world complexity is injected into fundamental functions within the engaging functions dataset. This enables the evaluation of interpretability techniques in a context mirroring real-world scenarios.
In addition to the function database, the researchers introduced an innovative evaluation process to assess the efficacy of AIAs and existing automated interpretability techniques. This evaluation comprises two distinct methods. For tasks requiring replication of functionality in code, the evaluation directly compares AI-generated outputs with the original, ground-truth functions. Tasks involving natural language function explanations entail a more intricate evaluation process.
In these instances, a specialized “third-party” language model was developed to assess the quality of these descriptions by gauging their conceptual alignment with the ground-truth functions. Despite outperforming current interpretability strategies, AIAs still struggle to accurately describe nearly half of the features in the benchmark, as revealed by FIND, underscoring the ongoing challenge of achieving fully automated interpretation.
While contemporary AIAs excel in describing high-level functionality, they often overlook nuanced details, particularly in subdomains with complex or irregular behavior, notes Tamar Rott Shaham, co-lead author of the study and a postdoc at CSAIL.
This limitation likely stems from inadequate exploration in these areas. To address this, the researchers endeavored to guide AIAs’ exploration by initiating their search with relevant, specific stimuli, significantly enhancing interpretation accuracy. This strategy combines novel AIA techniques with established methods utilizing precomputed examples to kickstart the exploration process.
Furthermore, the team is developing a toolkit to enhance AIAs’ capability to conduct more precise analyses on neural networks, both in white-box and black-box scenarios. This toolkit aims to equip AIAs with enhanced tools for selecting inputs and refining their ability to test hypotheses, thereby enabling more thorough and accurate neural network analyses.
The researchers are also tackling real-world interpretability challenges in AI, focusing on selecting pertinent inquiries for model analysis. Their objective is to devise automated interpretability techniques that could potentially aid users in auditing systems, such as face recognition or autonomous driving, to identify potential failure modes, hidden biases, or unforeseen behaviors prior to deployment.
A Gaze into the Future
The team envisions a future where AIAs are virtually autonomous, capable of auditing other systems with human scientists serving as overseers and guides. Advanced AIAs may propose new experiments and inquiries that surpass the initial assumptions of human researchers.
Expanding AI interpretability to encompass more intricate actions, such as entire neural pathways or subnetworks, and predicting inputs that may elicit undesired behaviors stand as primary objectives. This innovation heralds a significant stride in AI research, aiming to enhance the understanding and reliability of AI methodologies.
Martin Wattenberg, a computer science professor at Harvard University unaffiliated with the study, lauds the efficacy of a robust benchmark in tackling complex challenges. He commends the team’s creation of a potent standard for interpretability, a critical hurdle in contemporary machine learning. Wattenberg expresses admiration for the automated interpretability tool devised by the authors, noting its transformative potential in aiding human comprehension.
Presented at NeurIPS 2023, the findings by Schwettmann, Rott Shaham, and their collaborators featured contributions from MIT members including graduate student Joanna Materzynska, undergraduate Neil Chowdhury, Shuang Li, Ph.D., Professor Antonio Torralba, and Assistant Professor Jacob Andreas. Additionally, David Bau, an associate professor at Northwestern University, is listed as a co-author.
