Professors Meng Xu and Sihang Liu have received $254,116 in funding from the National Cybersecurity Consortium, a federally incorporated not-for-profit organization committed to advancing Canada’s cybersecurity ecosystem. Their project, Securing LLM Agents Against Malicious or Vulnerable Tools, aims to identify and mitigate security risks in agentic systems — AI systems capable of making autonomous decisions and taking actions to achieve specific goals.
This year, the NCC has dedicated $20.9 million in funding as part of its commitment to the Government of Canada’s Cyber Security Innovation Network program. In addition to this successful project application, funds have been distributed to 31 projects put forth by Canadian organizations representing academia, private institutions, and the not-for-profit sector.
“Congratulations to Meng and Sihang on this important research support,” said Raouf Boutaba, University Professor and Director of the Cheriton School of Computer Science. “Their project will not only help safeguard agentic systems but also will provide important training for graduate students conducting research at the frontiers of artificial intelligence and security.”
L to R: Professors Meng Xu (project principle investigator) and Sihang Liu (project co-PI)
Meng Xu is an Assistant Professor at the Cheriton School of Computer Science. His research focuses on system and software security, with a focus on delivering high-quality solutions to practical security programs, especially in finding and patching vulnerabilities in critical computer systems.
Sihang Liu is an Assistant Professor at the Cheriton School of Computer Science. His research interests lie broadly in computer architecture and systems. Specifically, his research builds and optimizes systems for generative AI applications and AI agents.
Research advancing the security of agentic AI systems
As agentic AI systems become increasingly widespread, it is critical to identify and mitigate potential security vulnerabilities, particularly those that arise from high-level design choices.
A task-oriented AI agent typically consists of four components: a foundational model serving as the agent’s brain, a memory module for recording task contexts, a planning module for task decomposition, and a set of task-specific tools. While developers typically have control over the first three components, the task-specific tools — well-defined executable workflows often developed by third parties — may introduce risks, as they can contain vulnerabilities or be malicious. This project investigates the security implications of integrating such third-party tools into agentic systems and explores how sandboxing mechanisms can be used to mitigate associated risks.
The research team will begin by constructing a dataset of open-source AI agents and investigate whether malicious behaviours from tools in the AI agents can be triggered through user prompts. The researcher team will also seek the possibility of replacing a genuine tool with a malicious one they develop and investigate in a controlled environment. This will determine whether the malicious tool can wreak havoc in the agent. Findings on malicious and vulnerable tools will guide the team on the design of defence mechanisms, which will draw inspirations from how untrusted components in software engineering are isolated.
By systematically identifying vulnerabilities and developing practical safeguards, this research aims to enhance the trustworthiness of AI agents and support their safe, widespread adoption. The project will also provide training opportunities for three graduate students, conducting research at the intersection of AI and security.
About the National Cybersecurity Consortium
The National Cybersecurity Consortium is a pan-Canadian network that supports the advancement of the Canadian cybersecurity ecosystem through research and development, commercialization, and training by driving collaboration among universities; private industry; not-for-profit organizations; provincial, territorial, and municipal governments; and other key cybersecurity stakeholders.
This project was made possible in-part through the support of the National Cybersecurity Consortium and the Government of Canada (CSIN) // Ce projet a été rendu possible en partie grâce au soutien du Consortium national pour la cybersécurité et du gouvernement du Canada (RIC).