Researchers have identified approximately 100 machine learning (ML) models on the Hugging Face AI platform that could potentially allow attackers to inject harmful code into user machines. This discovery highlights the increasing danger posed by malevolent actors who tamper with publicly accessible AI models for illicit purposes.
The detection of these malicious models by JFrog Security Research forms part of an ongoing investigation into how attackers might exploit ML models to compromise user systems. The researchers developed a scanning tool to examine model files uploaded to Hugging Face, a popular public AI model repository, in order to identify and counter emerging threats, particularly those related to code execution.
During the analysis, it was uncovered that certain models hosted on the repository contained malicious payloads. For instance, the scanner flagged a PyTorch model uploaded by a user named baller423, whose account has since been deleted. This model allowed attackers to insert arbitrary Python code into a critical process, potentially resulting in malicious activities once the model is deployed on a user’s machine.
Analysis of Hugging Face Payloads
While payloads embedded in AI models uploaded by researchers typically aim to showcase vulnerabilities or demonstrate proof-of-concepts without causing harm, the payload uploaded by baller423 exhibited significantly different behavior. According to JFrog senior security researcher David Cohen, the payload initiated a reverse shell connection to a real IP address (210.117.212.93), which is a more intrusive and potentially malicious action. This direct connection to an external server indicates a serious security threat rather than a mere vulnerability demonstration.
Investigations revealed that the IP address is associated with Kreonet, which stands for “Korea Research Environment Open Network,” a high-speed network in South Korea supporting advanced research and education. It is speculated that AI researchers or practitioners may have been involved in creating the model. However, it is essential to adhere to the fundamental principle in security research, which prohibits the publication of actual exploits or malicious code, a principle that was violated in this case when the code attempted to connect to a real IP address.
Subsequent to the removal of the model, researchers encountered similar payloads with different IP addresses, one of which remains active. This further underscores the prevalence of potentially malicious models on Hugging Face, emphasizing the critical need for continuous vigilance and enhanced security measures to combat threats from malicious AI models.
Functionality of Malicious AI Models
Understanding how attackers can weaponize ML models on Hugging Face requires insight into how a malicious PyTorch model, like the one uploaded by baller423, operates within the realm of Python and AI development.
Code execution can occur when loading certain types of ML models, such as those using the “pickle” format, a common serialization format for Python objects. Pickle files may contain arbitrary code that gets executed upon loading, as highlighted by JFrog.
Loading PyTorch models with transformers, a prevalent practice among developers, involves utilizing the torch.load() function to deserialize the model from a file. Particularly with PyTorch models trained using Hugging Face’s Transformers library, developers commonly employ this method to load the model along with its architecture, weights, and configurations.
Transformers offer a robust framework for natural language processing tasks, facilitating the creation and deployment of advanced models. The malicious payload was likely injected into the PyTorch model file using the __reduce__ method of the pickle module, enabling attackers to insert arbitrary Python code during the deserialization process, potentially leading to malicious behavior upon model loading.
While Hugging Face incorporates various security measures, including malware scanning and pickle scanning, it does not outright block pickle models from being downloaded. Instead, it labels them as “unsafe,” allowing users to download and execute potentially harmful models.
It is crucial to note that not only pickle-based models are susceptible to executing malicious code. For instance, Tensorflow Keras, the second most prevalent model type on Hugging Face, can also execute arbitrary code, although exploiting this method is more challenging for attackers, as per JFrog’s findings.
Mitigating Risks from Compromised AI Models
Instances of AI security risks on platforms like Hugging Face have been previously reported, emphasizing the importance of safeguarding ML models, datasets, and applications within the AI community. Researchers have warned about potential threats from unsecured API access tokens on GitHub and the Hugging Face platform, which could allow adversaries to contaminate training data in widely used language models, pilfer models and datasets, and execute malicious activities.
The proliferation of publicly available and potentially harmful AI/ML models poses a significant threat to the supply chain, especially targeting demographics like AI/ML engineers and pipeline systems. To mitigate these risks, AI developers are encouraged to leverage new tools such as Huntr, a bug-bounty platform tailored for AI vulnerabilities, to bolster the security posture of AI models and platforms.
Collaborative efforts are essential in fortifying Hugging Face repositories and ensuring the privacy and integrity of AI/ML engineers and organizations relying on these resources, as highlighted by Cohen.
