feat(cli): add agent profiles for pre-configured system prompts + tool access#1162
feat(cli): add agent profiles for pre-configured system prompts + tool access#1162TimeToBuildBob wants to merge 4 commits intomasterfrom
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Greptile OverviewGreptile SummaryThis PR adds agent profiles to configure system prompts, tool access, and behavior rules for specialized use cases like read-only exploration or isolated processing of untrusted content. The implementation includes 5 built-in profiles (default, explorer, researcher, developer, isolated) and supports user-defined profiles via TOML files. Critical Issues:
Design Concerns:
Confidence Score: 2/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant User
participant CLI as gptme/cli.py
participant Profiles as gptme/profiles.py
participant Tools as Tool System
participant Agent as LLM Agent
User->>CLI: gptme --agent-profile explorer
CLI->>Profiles: get_profile("explorer")
Profiles->>Profiles: Check user profiles dir
Profiles->>Profiles: Fall back to BUILTIN_PROFILES
Profiles-->>CLI: Return Profile(tools=[read,shell,chats], behavior={read_only:true, no_network:true})
CLI->>CLI: Apply profile.tools to tool_allowlist
Note over CLI: Only if --tool not specified
CLI->>Tools: init_tools(tool_allowlist)
Tools-->>CLI: Initialized tools
CLI->>CLI: Append profile.system_prompt to initial_msgs
Note over CLI: System message with behavior rules
CLI->>Agent: Start conversation with tools + system prompt
Note over Agent: Behavior rules only in prompt,<br/>not enforced by system
|
| """Behavior rules for a profile.""" | ||
|
|
||
| # If True, require confirmation for write operations | ||
| confirm_writes: bool = False | ||
| # If True, prevent all file writes | ||
| read_only: bool = False | ||
| # If True, prevent network access (browser, etc.) | ||
| no_network: bool = False | ||
| # Maximum context tokens (None = use model default) |
There was a problem hiding this comment.
behavior flags defined but never enforced - read_only, no_network, and confirm_writes are set in profiles but have no actual effect on tool behavior. the system only adds them to the system prompt, which relies on the LLM to comply rather than enforcing them programmatically
Prompt To Fix With AI
This is a comment left during a code review.
Path: gptme/profiles.py
Line: 26:34
Comment:
behavior flags defined but never enforced - `read_only`, `no_network`, and `confirm_writes` are set in profiles but have no actual effect on tool behavior. the system only adds them to the system prompt, which relies on the LLM to comply rather than enforcing them programmatically
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
This is by design - profiles provide soft/prompting-based enforcement only. The module docstring now explicitly documents this:
IMPORTANT: Profile restrictions are soft enforcement via prompting only.
The agent receives instructions about limitations but there's no hard
technical enforcement at the tool level.
Hard enforcement is planned for PR #1158 (shielded processing mode).
gptme/profiles.py
Outdated
There was a problem hiding this comment.
data.pop() mutates the input dict, which can cause unexpected behavior if the caller reuses the dict. use data.get("behavior", {}) instead and create a copy of data before passing to cls()
| """Create a Profile from a dictionary (e.g., TOML config).""" | |
| behavior_data = data.pop("behavior", {}) | |
| behavior = ProfileBehavior(**behavior_data) | |
| return cls(behavior=behavior, **data) | |
| behavior_data = data.get("behavior", {}) | |
| behavior = ProfileBehavior(**behavior_data) | |
| profile_data = {k: v for k, v in data.items() if k != "behavior"} | |
| return cls(behavior=behavior, **profile_data) |
Prompt To Fix With AI
This is a comment left during a code review.
Path: gptme/profiles.py
Line: 58:61
Comment:
`data.pop()` mutates the input dict, which can cause unexpected behavior if the caller reuses the dict. use `data.get("behavior", {})` instead and create a copy of data before passing to `cls()`
```suggestion
behavior_data = data.get("behavior", {})
behavior = ProfileBehavior(**behavior_data)
profile_data = {k: v for k, v in data.items() if k != "behavior"}
return cls(behavior=behavior, **profile_data)
```
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
✅ Fixed in commit 67a08e2 - now uses .get() instead of .pop() and creates a filtered copy:
behavior_data = data.get("behavior", {})
profile_data = {k: v for k, v in data.items() if k != "behavior"}Added regression test test_profile_from_dict_no_mutation.
| # Apply profile tools if no explicit tools specified | ||
| if not tool_allowlist and selected_profile.tools is not None: | ||
| tool_allowlist = tuple(selected_profile.tools) |
There was a problem hiding this comment.
profile tools only apply if user doesn't specify --tool. if user runs gptme --agent-profile isolated --tool shell,patch, they get shell and patch tools instead of the restricted read,ipython defined by the isolated profile, defeating the security restrictions
Prompt To Fix With AI
This is a comment left during a code review.
Path: gptme/cli.py
Line: 382:384
Comment:
profile tools only apply if user doesn't specify `--tool`. if user runs `gptme --agent-profile isolated --tool shell,patch`, they get shell and patch tools instead of the restricted `read,ipython` defined by the isolated profile, defeating the security restrictions
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
This is intentional for flexibility - power users can override profile defaults with --tool. Since profiles are soft enforcement (prompting-based), this is consistent with the design philosophy.
If hard tool restrictions are needed, that's what PR #1158 (shielded mode) will provide.
|
I don't really like We currently have a What do you think? Confusing or good design? What if one wants to run e.g. Bob with a profile? |
- Fix tomllib import for Python < 3.11 (use tomli fallback) - Convert Optional[X] to X | None syntax (UP007) - Fix import ordering in cli.py and test_profiles.py - Remove unused imports from tests - Add assert for tools not None before using in operator - Rename loop variable 'profile' to 'prof' to avoid shadowing Same fixes as PR #1162 which this PR builds on.
Design Response: Profile CLI and Agent Unification@ErikBjare Thanks for the thoughtful questions! Here's my analysis: 1.
|
|
It should probably be However, seems like much of this is premature, as we can't actually provide the requested guarantees. |
|
Or maybe we could kinda enforce the guarantees by prompting the agent with the limitations to follow. That could be safe enough at the limit as models get better and don't accidentally pwn themselves. |
Clarification: Current Implementation IS Prompting-Based@ErikBjare Good points. Let me clarify what this PR actually does: What the Current PR Does (Soft Enforcement)The current implementation is already prompting-based only. The profile's system prompt tells the agent:
There's no technical enforcement - if the agent calls a "forbidden" tool, gptme will still execute it. The agent just follows the instructions. Agreements✅ On "Premature Guarantees"Totally agree we shouldn't overclaim. Current scope:
The "Safe Enough at the Limit" IdeaI like this framing. As models improve, prompt-based restrictions become more reliable:
Proposed Path Forward
Should I:
|
|
Update this PR |
✅ PR Updated as Requested@ErikBjare Updated per your feedback. Changes made: 1.
|
…l access Add support for agent profiles that combine: - Pre-configured system prompts - Tool access restrictions - Behavior rules (read_only, no_network, confirm_writes) Built-in profiles: - default: Full capabilities - explorer: Read-only, no network (for codebase analysis) - researcher: Web access, read-only files (for web research) - developer: Full development capabilities - isolated: No writes or network (for untrusted content) Usage: gptme --list-profiles # Show available profiles gptme --agent-profile explorer # Use a specific profile User-defined profiles can be added as TOML files in ~/.config/gptme/profiles/ Closes #1157 Co-authored-by: Bob <[email protected]>
- Fix tomllib import for Python < 3.11 (use tomli fallback) - Convert Optional[X] to X | None syntax (UP007) - Fix import ordering in cli.py and test_profiles.py - Remove unused imports (tempfile, Path, pytest) from tests - Add assert for tools not None before using in operator - Rename loop variable from 'profile' to 'prof' to avoid shadowing the 'profile' CLI parameter (bool) which caused mypy errors
The local import of Message inside the main() function was causing Python to treat Message as a local variable throughout the function, leading to UnboundLocalError at line 432 where it was used before the local import at line 533. Co-authored-by: Bob <[email protected]>
Changes per Erik's review feedback: - Add 'gptme-util profile' subcommand group (singular, cleaner) - Add 'gptme-util profile list' to list available profiles - Add 'gptme-util profile show <name>' to show profile details - Remove --list-profiles flag from main CLI - Update --agent-profile help text to reference new command - Clarify in profiles.py that restrictions are soft/prompting-based - Fix from_dict() to not mutate input dictionary (Greptile feedback) - Add test for from_dict() non-mutation The profile system provides soft enforcement via prompting only - the agent receives behavioral guidance but there's no hard technical enforcement at the tool level.
a7a712c to
67a08e2
Compare
PR Updated ✅@ErikBjare - Updated per your request. Changes include: 1. CLI Restructuring (as agreed)
2. Fixed Greptile Code Review Issues
3. Clarified Soft Enforcement in Documentation
SummaryThis PR now provides:
Ready for review when you have time. |
|
Closing this PR — it has merge conflicts and has been open for 3 weeks without review. The feature (agent profiles) is still tracked in issue #1157. I'll reopen a fresh PR if/when there's interest in pursuing this. Closing to reduce PR noise and keep the review queue focused on active, clean PRs. |
Summary
Add support for agent profiles that combine pre-configured system prompts, tool access restrictions, and behavior rules into named profiles.
Built-in Profiles
Usage
User profiles: TOML files in ~/.config/gptme/profiles/
Security
Profiles enable defense-in-depth against prompt injection (#1158).
Closes #1157