In technological advancement, industries such as healthcare, technology, and finance are experiencing rapid growth, presenting new opportunities and challenges. The School of Engineering and Applied Science at the University of Pennsylvania (Penn Engineering) is committed to spearheading the development of cutting-edge solutions aimed at forging a brighter future. To this end, Penn Engineering has convened a gathering of eminent leaders from the realms of engineering, academia, industry, and policy at the Penn Washington Center in Washington, D.C. This assembly is part of a broader discourse focused on responsibly steering the course of future innovations.
As part of its initiative on Responsible Innovation, Penn Engineering’s research team has made a significant discovery regarding the security of AI-operated robots. These findings, supported by grants from the National Science Foundation and the Army Research Laboratory, reveal previously undetected vulnerabilities and weaknesses in AI-driven robots. The primary objective of this research is to enhance the security measures surrounding the deployment of large language models (LLMs) in robotic applications.
George Pappas, the UPS Foundation Professor of Transportation in Electrical and Systems Engineering (ESE), Computer and Information Science (CIS), and Mechanical Engineering and Applied Mechanics (MEAM), expressed concerns about the current state of safety in the integration of large language models with physical systems. According to Professor Pappas, who also holds the position of Associate Dean for Research at Penn Engineering, caution is needed as these AI-controlled robots are susceptible to manipulations and hacks.
One of the noteworthy outcomes of this research is the development of RoboPAIR, an algorithm capable of achieving a 100% “jailbreak” rate within just a few days. This algorithm was successful in bypassing the safety mechanisms governing three distinct robotic systems: the Unitree Go2, a quadruped robot utilised in various settings; the Clearpath Robotics Jackal, a wheeled vehicle frequently used in academic research; and the Dolphin LLM, a self-driving simulator developed by NVIDIA. Notably, the AI governors for the first two systems are based on OpenAI’s ChatGPT, which demonstrated vulnerabilities to these jailbreaking attacks. Such breaches can lead to serious safety risks, such as manipulating a self-driving system to behave recklessly at pedestrian crossings.
Before releasing their findings to the public, the Penn Engineering team was responsible for informing the affected companies about the vulnerabilities in their systems. They are now collaborating with these companies to leverage their research to improve the testing and validation of AI safety protocols used by these manufacturers. Despite the challenges, the research team, including Hamed Hassani, Associate Professor in ESE and CIS at Penn Engineering and Statistics and Data Science at Wharton, acknowledges that eliminating LLMs from these systems may not be feasible. Professor Hassani emphasised the complexity of addressing such vulnerabilities, particularly in the context of chatbots.
Alexander Robey, a recent Ph.D. graduate from Penn Engineering and a current postdoctoral scholar at Carnegie Mellon University, highlighted the importance of identifying system vulnerabilities to enhance safety. He pointed out that this principle applies to cybersecurity and AI safety. Engaging in AI red teaming, a method that involves testing AI systems for potential threats and vulnerabilities, is crucial for protecting generative AI systems. Identifying these weaknesses allows for testing and training these systems to avoid possible threats.
The researchers argue that addressing these vulnerabilities requires a comprehensive reevaluation of the regulatory framework governing the integration of AI into physical systems rather than merely applying software patches. Vijay Kumar, the Nemirovsky Family Dean of Penn Engineering and coauthor of the study stressed the importance of adopting a safety-first approach to foster responsible innovation. He affirmed that their research is instrumental in developing a verification and validation framework that ensures robotic systems only perform actions that align with social norms, thereby paving the way for the safe deployment of AI-enabled robots in real-world applications. This holistic approach underscores the need to confront intrinsic vulnerabilities to ensure that integrating AI into our daily lives is safe and beneficial.
More information: Alexander Robey et al, Jailbreaking LLM-Controlled Robots. DOI: 10.48550/arXiv.2410.13691