On the Security of Tool-Invocation Prompts for LLM-Based Agentic Systems: An Empirical Risk Assessment

This repository accompanies our work on Tool Invocation Prompt (TIP) exploitation in LLM-based agentic systems. We present the first systematic study on how TIPs can be manipulated to hijack external tool behavior, leading to critical security risks such as remote code execution (RCE) and denial of service (DoS) attacks.

📌 Overview

LLM-based agentic systems are emerging autonomous systems that use Large Language Models (LLMs) to understand user queries, make decisions, and take actions by invoking external tools. This "LLM × Tools" paradigm allows agents to accomplish complex, multi-step tasks in real-world environments.

The Tool Invocation Prompt (TIP) is a critical component within this architecture. TIPs are prompt components that explicitly define the tool-interaction procedures and communication protocols between the LLM and external tools. They also guide the LLM to act as a security guard, judging the safety of content exchanged between the LLM and the tools. Despite this critical security role, TIPs have been largely overlooked by prior research.

🚨 Key Findings

1. Universal Vulnerability Across Major Systems

   • Our empirical evaluation demonstrates that popular LLM-based agentic systems are vulnerable to the TIP attack vector. Every tested client was vulnerable to at least one attack outcome.
   • Affected systems include IDEs, CLIs, and Chatboxes such as:
     • Claude Code
     • Cursor
     • Cline
     • Trae
     • WindSurf
     • And others

2. Wide Model Coverage

   • Vulnerabilities are confirmed in agents powered by the latest and most advanced LLM backends, including:
     • GPT-5
     • Claude-sonnet-4
     • Gemini-2.5-pro
     • Grok-4

3. Significant Attack Impact

   • DoS (Denial of Service) is broadly observable across most agent categories.
   • RCE-1 (Direct Command Injection) primarily affects IDE-based agents due to their deep tool integration.
   • RCE-2 (RCE via Tool Description and Tool Return) substantially expands the attack surface, enabling exploitation even in agents resistant to direct injection, like Claude Code.

🎥 Demo

• Live Demo Website: TIP Exploit Website
• Video Walkthrough: Case Study on Claude Code
  • This video demonstrates the RCE-2 attack bypassing the security defenses in Claude Code.
  • Watch the video here

claude-code-rce.-.01.mp4

🔐 Attack Precondition & Threat Model

• Attack Scenario: A user installs an official LLM-based IDE or agentic system (e.g., Claude Code) and adds a third-party MCP (Model Context Protocol) server. The developer of the remote MCP server can then hijack the agentic system via a malicious tool description, executing arbitrary code with the same privilege level as the user.
• Attacker Capabilities: We assume an attacker who can interact with the agentic system through standard user queries. The attacker can design an external tool with a malicious tool description and deploy it in the agentic system.

🛠️ Exploitation Method: TEW (TIP Exploitation Workflow)

We propose TEW, a systematic TIP Exploitation Workflow that achieves tool behavior hijacking by manipulating tool invocation. The workflow consists of three main steps:

Figure: Execution workflow highlighting attack channels within the TIP structure. Malicious tool descriptions (channel 1) and malicious tool returns (channel 2) are used to hijack the agent's behavior.

• Step 1: Prompt Stealing
  • The attacker extracts underlying system and tool prompts by injecting malicious instructions into a tool's description, coercing the agent to disclose its internal configuration.
• Step 2: TIP Vulnerability Analysis
  • The attacker reconstructs the complete TIP from the stolen components and analyzes it for vulnerabilities, such as strict schema enforcement that can be disrupted (for DoS) or initialization routines that process tool descriptions (for RCE).
• Step 3: TIP Hijacking
  • The attacker exploits the identified vulnerabilities using two primary channels: the tool description and the tool return.
  • Format-based Untargeted Attacks (DoS): A malicious prompt in the tool description causes the LLM to generate malformed output that the agent's parser cannot handle, leading to a denial-of-service condition.
  • Logic-based Targeted Attacks (RCE): A malicious payload is embedded into the tool description (RCE-1) or split between the tool description and tool return (RCE-2) to achieve unauthorized command execution.

🔬 Case Studies

Case 1: RCE on Cursor with GPT-5

This case demonstrates a multi-channel RCE-2 attack where a malicious tool description and a malicious tool return work together to coerce the LLM into executing a payload, even with GPT-5's advanced safety alignment. The injected description establishes a fake two-step "initialization" process, and the tool's return reinforces this instruction, ultimately leading the LLM to execute a curl... | bash command.

Case 2: RCE on Claude Code with claude-sonnet-4

Claude Code employs an additional defense agent (Haiku) to check commands for security risks. This case study shows that our TIP-based multi-channel attack can bypass this defense. Although the guard LLM flags the malicious command as "UNSAFE", the main LLM (claude-sonnet-4), heavily influenced by our injected tool description and tool return, overrides the warning and executes the RCE payload.

Case 3: DoS on Cline with grok-4

This case demonstrates an untargeted attack that corrupts the tool invocation process. The attacker injects a poisoned tool description that alters the required tool-calling format. When the LLM later attempts to call a tool for a normal user request, it uses the malicious format, producing an output that the agent's parser rejects. This parsing failure freezes the agent, resulting in a persistent denial of service that requires manually removing the malicious content to fix.

🛡️ Defense Directions

Our experiments show that preliminary defenses like guard models and self-reflection are insufficient against sophisticated TIP-based attacks. We recommend the following directions to enhance security:

• Layered Defense: Security should not rely on a single mechanism. A robust strategy should integrate both external filtering (guard LLMs) and internal verification (self-reflection or anomaly detection).
• Adaptive Filtering: Static guard models can be bypassed. We suggest incorporating continual learning and adversarial training to adapt to emerging threats.
• Requerying and Consensus: Instead of a single decision, the system can query multiple defense models and use a majority vote or confidence-based strategy to mitigate risks from model-specific vulnerabilities.
• Provenance and Trust Signals: Prompts from untrusted sources should be flagged for stricter filtering, while trusted inputs could bypass some heavy defenses to reduce overhead.

📖 Abstract

LLM-based agentic systems are emerging autonomous systems that leverage LLM to understand user querys, make decisions and take actions accordingly. It has been widely used in diverse domains, including chatbots, customer service, software engineering, and financial services. The key feature of LLM-based agentic systems is to execute external tools, enabling accomplishing complex, multi-step reasoning tasks in a real-world environment by invoking external tools.

Within the “LLM × Tools” invocation scheme, the Tool Invocation Prompt (TIP) serves as acritical component with high security impact of the LLM-based agentic systems. TIPs represent prompt components that explicitly define tool-interaction procedures and communication protocols between LLM and external tools. They are typically defined by specific LLM-based agentic systems to determine tool-invocation behaviors and further guide LLMs to act as a security guard, judging the correctness and safety of content exchanged between LLMs and external tools.

Despite the critical security impact of TIP, it has been overlooked by prior literature. To fill this gap, we propose the first investigation on security risks associated with TIP for LLM-based agentic systems. We shows that major representative LLM-based agentic systems are vulnerable to the TIP attack vector, leading to severe security risks such as remote code execution(RCE) and denial of service(DoS). Specifically, we present a systematic TIP exploitation workflow TEW that achieves external tool behavior hijacking by strategically manipulating external tool invocation. Our empirical evaluation demonstrates that popular LLM-based agentic systems like Cursor and Claude Code and other coding IDEs, chatbox with latest LLM backends(GPT-5, Claude-sonnet-4, Gemini-2.5-pro, Grok-4) are vulnerable to RCE and DoS through TIP-based attack. We further discuss potential defense mechanisms on enhancing the security in TIP of emerging LLM-based agentic systems.

📄 Citation

If you have interest in this work, please cite:

@misc{xie2025exploittoolinvocationprompt,
      title={On the Security of Tool-Invocation Prompts for LLM-Based Agentic Systems: An Empirical Risk Assessment}, 
      author={Yuchong Xie and Mingyu Luo and Zesen Liu and Zhixiang Zhang and Kaikai Zhang and Yu Liu and Zongjie Li and Ping Chen and Shuai Wang and Dongdong She},
      year={2025},
      eprint={2509.05755},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2509.05755}, 
}