Organizations utilizing Ray, the open-source framework designed for scaling artificial intelligence and machine learning workloads, face potential security risks due to a set of unresolved vulnerabilities within the technology, as highlighted by researchers recently.
Implications of Vulnerabilities
These vulnerabilities provide malicious actors with avenues to gain access to the operating system across all nodes within a Ray cluster, facilitate remote code execution, and potentially escalate privileges. Such weaknesses pose a significant threat to organizations that expose their Ray instances either to the Internet or even within a local network setting.
The vulnerabilities were initially identified by researchers at Bishop Fox, who promptly reported them to Anyscale in August. Additionally, security experts from Protect AI also privately disclosed two of these vulnerabilities to Anyscale earlier.
Despite the notifications, Anyscale has yet to address these vulnerabilities, according to Berenice Flores Garcia, a senior security consultant at Bishop Fox. Garcia mentioned that Anyscale’s stance on the matter is that the vulnerabilities are inconsequential since Ray is not meant for deployment outside of a strictly controlled network environment, as indicated in their documentation.
Anyscale has not provided a response to inquiries from Dark Reading regarding this issue.
Ray serves as a technology enabling organizations to distribute the execution of intricate, infrastructure-intensive AI and machine learning workloads. Notably, several prominent entities such as OpenAI, Spotify, Uber, Netflix, and Instacart leverage Ray for developing scalable AI and machine learning applications. Amazon’s AWS has also integrated Ray into various cloud services, positioning it as a tool to expedite the scaling of AI and ML applications.
Vulnerabilities Overview
The vulnerabilities disclosed by Bishop Fox concern improper authentication and input validation within Ray Dashboard, Ray Client, and potentially other components. These vulnerabilities impact Ray versions 2.6.3 and 2.8.0, allowing attackers to retrieve data, scripts, or files stored within a Ray cluster. In cloud environments like AWS, the vulnerabilities could potentially lead to the extraction of highly privileged IAM credentials, facilitating privilege escalation, as outlined in Bishop Fox’s report.
The three vulnerabilities identified by Bishop Fox are CVE-2023-48023, a remote code execution vulnerability associated with missing authentication for a critical function; CVE-2023-48022, a server-side request forgery vulnerability in the Ray Dashboard API enabling RCE; and CVE-2023-6021, an insecure input validation error that permits a remote attacker to execute malicious code on the affected system.
Bishop Fox’s report provides insights into how threat actors could exploit these vulnerabilities to execute arbitrary code.
These vulnerabilities are easily exploitable, requiring minimal technical expertise from attackers. Remote access to vulnerable component ports — typically ports 8265 and 10001 — from the Internet or a local network, along with basic Python knowledge, is adequate for exploitation, according to Garcia. Detecting the vulnerable components becomes straightforward if the Ray Dashboard UI is exposed, serving as the entry point for exploiting the identified vulnerabilities.
Emphasis on Network Security
While Anyscale has not responded to Dark Reading, their documentation emphasizes the necessity for organizations to deploy Ray clusters within a controlled network environment. The documentation underscores the importance of running Ray in a secure network environment, trusting the code being executed, ensuring isolated network traffic between Ray components, and implementing stringent network controls and authentication mechanisms for accessing additional services.
Anyscale underscores the responsibility of Ray developers to construct their applications with a clear understanding that Ray executes provided code faithfully, without differentiation between various types of tasks.
