Agentic rag #2432
Agentic rag #2432
Conversation
Summary by CodeRabbitRelease Notes
WalkthroughReplaces the RagTool pipeline with a LangGraph-based AgenticRAGAgent (retrieve → generate → evaluate loop) and AgentNodes; adds management command and Makefile target for agentic RAG; removes RagTool module and its CLI/tests; updates generator temperature, retrieval defaults, chunking, prompts, adds langgraph dependency and a JSON-extraction util; updates Slack handlers and tests. Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (2)
backend/apps/ai/common/constants.py (1)
6-6: Consider making the reasoning model configurable.Hardcoding "gpt-4o" as the default reasoning model may become outdated as newer, better models are released. Consider making this configurable via environment variables or Django settings.
Example configuration approach:
DEFAULT_REASONING_MODEL = os.getenv("REASONING_MODEL", "gpt-4o")Alternatively, add a Django setting:
# In settings REASONING_MODEL = env.str("REASONING_MODEL", default="gpt-4o")backend/apps/slack/events/app_mention.py (1)
38-53: Add error handling for placeholder message posting.The improved UX with a "thinking" indicator is great! However, if
chat_postMessagefails (e.g., network issue, invalid channel), the subsequentchat_updatewill fail becauseplaceholder["ts"]won't exist.Apply this diff to add error handling:
thread_ts = event.get("thread_ts") or event.get("ts") -placeholder = client.chat_postMessage( - channel=channel_id, - blocks=[markdown("⏳ Thinking…")], - text="Thinking…", - thread_ts=thread_ts, -) +try: + placeholder = client.chat_postMessage( + channel=channel_id, + blocks=[markdown("⏳ Thinking…")], + text="Thinking…", + thread_ts=thread_ts, + ) +except Exception: + logger.exception("Failed to post placeholder message") + return reply_blocks = get_blocks(query=query) -client.chat_update( - channel=channel_id, - ts=placeholder["ts"], - blocks=reply_blocks, - text=query, -) +try: + client.chat_update( + channel=channel_id, + ts=placeholder["ts"], + blocks=reply_blocks, + text=query, + ) +except Exception: + logger.exception("Failed to update message with response")
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
backend/poetry.lockis excluded by!**/*.lock
📒 Files selected for processing (18)
backend/apps/ai/Makefile(1 hunks)backend/apps/ai/agent/agent.py(1 hunks)backend/apps/ai/agent/nodes.py(1 hunks)backend/apps/ai/agent/tools/rag/generator.py(1 hunks)backend/apps/ai/agent/tools/rag/rag_tool.py(0 hunks)backend/apps/ai/common/constants.py(1 hunks)backend/apps/ai/management/commands/ai_run_agentic_rag.py(1 hunks)backend/apps/ai/management/commands/ai_run_rag_tool.py(0 hunks)backend/apps/ai/models/chunk.py(1 hunks)backend/apps/core/migrations/0003_alter_prompt_text.py(1 hunks)backend/apps/core/models/prompt.py(3 hunks)backend/apps/slack/common/handlers/ai.py(3 hunks)backend/apps/slack/events/app_mention.py(2 hunks)backend/pyproject.toml(1 hunks)backend/tests/apps/ai/agent/tools/rag/generator_test.py(1 hunks)backend/tests/apps/ai/agent/tools/rag/rag_tool_test.py(0 hunks)backend/tests/apps/ai/management/commands/ai_run_rag_tool_test.py(0 hunks)backend/tests/apps/slack/common/handlers/ai_test.py(1 hunks)
💤 Files with no reviewable changes (4)
- backend/apps/ai/management/commands/ai_run_rag_tool.py
- backend/tests/apps/ai/agent/tools/rag/rag_tool_test.py
- backend/tests/apps/ai/management/commands/ai_run_rag_tool_test.py
- backend/apps/ai/agent/tools/rag/rag_tool.py
🧰 Additional context used
🧬 Code graph analysis (7)
backend/apps/slack/events/app_mention.py (2)
backend/apps/slack/blocks.py (1)
markdown(23-36)backend/apps/slack/common/handlers/ai.py (1)
get_blocks(13-28)
backend/apps/ai/management/commands/ai_run_agentic_rag.py (1)
backend/apps/ai/agent/agent.py (2)
AgenticRAGAgent(19-70)run(27-52)
backend/tests/apps/slack/common/handlers/ai_test.py (2)
backend/apps/ai/agent/agent.py (1)
run(27-52)backend/apps/slack/common/handlers/ai.py (1)
process_ai_query(31-43)
backend/tests/apps/ai/agent/tools/rag/generator_test.py (1)
backend/apps/ai/agent/tools/rag/generator.py (1)
Generator(15-101)
backend/apps/ai/agent/nodes.py (3)
backend/apps/ai/agent/tools/rag/generator.py (3)
Generator(15-101)generate_answer(62-101)prepare_context(39-60)backend/apps/ai/agent/tools/rag/retriever.py (1)
Retriever(21-268)backend/apps/core/models/prompt.py (3)
Prompt(14-181)get_metadata_extractor_prompt(94-101)get_evaluator_system_prompt(54-61)
backend/apps/ai/agent/agent.py (1)
backend/apps/ai/agent/nodes.py (5)
AgentNodes(23-269)retrieve(37-61)generate(63-90)evaluate(92-126)route_from_evaluation(128-134)
backend/apps/slack/common/handlers/ai.py (1)
backend/apps/ai/agent/agent.py (2)
AgenticRAGAgent(19-70)run(27-52)
🪛 checkmake (0.2.2)
backend/apps/ai/Makefile
[warning] 1-1: Missing required phony target "all"
(minphony)
[warning] 1-1: Missing required phony target "clean"
(minphony)
[warning] 1-1: Missing required phony target "test"
(minphony)
🪛 GitHub Actions: Run CI/CD
backend/apps/ai/management/commands/ai_run_agentic_rag.py
[error] 1-1: cspell: Unknown word (agentic)
[error] 5-5: cspell: Unknown word (Agentic)
[error] 9-9: cspell: Unknown word (agentic)
[error] 11-11: cspell: Unknown word (Agentic)
[error] 26-26: cspell: Unknown word (Agentic)
[error] 33-33: cspell: Unknown word (Agentic)
backend/tests/apps/slack/common/handlers/ai_test.py
[error] 65-65: cspell: Unknown word (Agentic)
[error] 67-67: cspell: Unknown word (Agentic)
[error] 81-81: cspell: Unknown word (Agentic)
[error] 96-96: cspell: Unknown word (Agentic)
backend/apps/ai/Makefile
[error] 1-1: cspell: Unknown word (agentic)
[error] 2-2: cspell: Unknown word (Agentic)
[error] 3-3: cspell: Unknown word (agentic)
backend/pyproject.toml
[error] 31-31: cspell: Unknown word (langgraph)
backend/apps/ai/agent/agent.py
[error] 19-19: cspell: Unknown word (Agentic)
[error] 20-20: cspell: Unknown word (agentic)
[error] 23-23: cspell: Unknown word (Agentic)
[error] 42-42: cspell: Unknown word (Agentic)
backend/apps/slack/common/handlers/ai.py
[error] 7-7: cspell: Unknown word (Agentic)
[error] 32-32: cspell: Unknown word (agentic)
[error] 41-41: cspell: Unknown word (Agentic)
🔇 Additional comments (5)
backend/pyproject.toml (1)
31-31: LGTM! Pipeline cspell error is a false positive.The addition of
langgraph ^0.6.10aligns with the LangGraph-based AgenticRAGAgent implementation. The version constraint is appropriate per recent releases.The cspell error flagging "langgraph" as an unknown word is a false positive—this is the correct library name. Consider adding "langgraph" to the project's cspell dictionary.
backend/tests/apps/ai/agent/tools/rag/generator_test.py (1)
323-323: LGTM!The test correctly asserts the updated TEMPERATURE value of 0.8, matching the implementation change in
generator.py.backend/apps/ai/Makefile (1)
1-7: LGTM! Pipeline and static analysis alerts are false positives.The Makefile target rename from
ai-run-rag-tooltoai-run-agentic-ragcorrectly reflects the transition from RagTool to AgenticRAGAgent. The explicit CMD variable usage aligns with the surrounding target structure.The cspell errors flagging "agentic" as unknown are false positives—this is standard terminology in AI agent architectures. Consider adding "agentic" to the project's cspell dictionary.
The checkmake warnings about missing phony targets (
all,clean,test) are pre-existing and outside the scope of this PR.backend/apps/core/migrations/0003_alter_prompt_text.py (1)
1-17: LGTM!The migration correctly expands
Prompt.texttomax_length=2000and addsblank=Truewith a default empty string. This is a safe schema change that aligns with the prompt model updates for the agentic RAG workflow.backend/apps/ai/common/constants.py (1)
3-3: Estimate combined token usage: 32 retrieval chunks × ≈200 tokens each ≈ 6400 tokens + 2000 max_tokens = ≈8400 tokens; confirm this fits your model’s context window (especially if using an 8K-token model) or reduce DEFAULT_CHUNKS_RETRIEVAL_LIMIT accordingly.
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
backend/apps/ai/common/utils.py (1)
73-87: Consider more robust markdown parsing.The split-based approach works for well-formed OpenAI responses but is brittle:
- Only extracts the first code block if multiple exist
- Could misfire if
"```json"appears in a string literal- No validation that delimiters are properly paired
While the callers handle JSON parsing errors, consider using a regex pattern for more reliable extraction:
+import re + def extract_json_from_markdown(content: str) -> str: """Extract JSON content from markdown code blocks. Args: content (str): The content string that may contain markdown code blocks Returns: str: The extracted JSON content with code block markers removed """ - if "```json" in content: - return content.split("```json")[1].split("```")[0].strip() - if "```" in content: - return content.split("```")[1].split("```")[0].strip() - return content + # Try to match ```json blocks first + json_match = re.search(r"```json\s*(.*?)\s*```", content, re.DOTALL) + if json_match: + return json_match.group(1).strip() + + # Fall back to generic code blocks + code_match = re.search(r"```\s*(.*?)\s*```", content, re.DOTALL) + if code_match: + return code_match.group(1).strip() + + return content.strip()
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
backend/apps/ai/agent/nodes.py(1 hunks)backend/apps/ai/common/utils.py(1 hunks)backend/apps/ai/management/commands/ai_run_agentic_rag.py(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
backend/apps/ai/agent/nodes.py (4)
backend/apps/ai/agent/tools/rag/generator.py (3)
Generator(15-101)generate_answer(62-101)prepare_context(39-60)backend/apps/ai/agent/tools/rag/retriever.py (1)
Retriever(21-268)backend/apps/ai/common/utils.py (1)
extract_json_from_markdown(73-87)backend/apps/core/models/prompt.py (3)
Prompt(14-181)get_metadata_extractor_prompt(94-101)get_evaluator_system_prompt(54-61)
backend/apps/ai/management/commands/ai_run_agentic_rag.py (1)
backend/apps/ai/agent/agent.py (2)
AgenticRAGAgent(19-70)run(27-52)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Run pre-commit checks
- GitHub Check: CodeQL (javascript-typescript)
🔇 Additional comments (2)
backend/apps/ai/management/commands/ai_run_agentic_rag.py (1)
23-35: LGTM! Past critical issue resolved.The dict access issue flagged in the previous review has been correctly fixed. The command now safely accesses
result.get("answer", "")instead of using attribute access.backend/apps/ai/agent/nodes.py (1)
38-62: LGTM! Past major issue resolved.The dynamic limit issue flagged in the previous review has been correctly fixed:
filter_chunks_by_metadatanow accepts and respects thelimitparameter (line 59)- The evaluate branch also passes the dynamic limit (line 119)
- The helper correctly slices results to the provided limit (line 192)
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (3)
backend/apps/ai/agent/nodes.py (3)
38-62: LGTM! Retrieve logic correctly honors dynamic limits.The method properly extracts metadata, retrieves chunks, and applies filtering with the dynamic limit parameter (past issue resolved). The slice at line 61 is redundant since
filter_chunks_by_metadataalready limits results, but this is harmless.If desired, remove the redundant slice at line 61:
- state["context_chunks"] = filtered_chunks[:limit] + state["context_chunks"] = filtered_chunks
91-125: Extract hardcoded limit cap to a constant.The method correctly uses
requires_more_contextand caps limit growth (past issues resolved). However, the hardcoded64at line 105 should be extracted to a named constant for maintainability.Define a constant in
backend/apps/ai/common/constants.py:MAX_CHUNKS_RETRIEVAL_LIMIT = 64Then apply this diff:
+from apps.ai.common.constants import ( DEFAULT_CHUNKS_RETRIEVAL_LIMIT, DEFAULT_MAX_ITERATIONS, DEFAULT_REASONING_MODEL, DEFAULT_SIMILARITY_THRESHOLD, + MAX_CHUNKS_RETRIEVAL_LIMIT, ) @@ - limit = min(state.get("limit", DEFAULT_CHUNKS_RETRIEVAL_LIMIT) * 2, 64) + limit = min(state.get("limit", DEFAULT_CHUNKS_RETRIEVAL_LIMIT) * 2, MAX_CHUNKS_RETRIEVAL_LIMIT)Optionally, remove the redundant slice at line 118 (same as line 61):
- state["context_chunks"] = filtered_chunks[:limit] + state["context_chunks"] = filtered_chunks
135-190: LGTM! Metadata filtering correctly uses dynamic limit.The method signature now accepts a
limitparameter and uses it when slicing results (past issue resolved). The heuristic scoring logic is reasonable.Consider extracting the scoring weights (lines 159, 167, 173, 179, 182) to named constants for clarity:
SCORE_FIELD_MATCH = 2.0 SCORE_FILTER_MATCH = 5.0 SCORE_CONTENT_MATCH = 3.0 SCORE_METADATA_PRESENCE = 0.1This would document the relative importance of each scoring factor.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
backend/apps/ai/agent/nodes.py(1 hunks)backend/apps/ai/common/constants.py(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
backend/apps/ai/agent/nodes.py (4)
backend/apps/ai/agent/tools/rag/generator.py (3)
Generator(15-101)generate_answer(62-101)prepare_context(39-60)backend/apps/ai/agent/tools/rag/retriever.py (1)
Retriever(21-268)backend/apps/ai/common/utils.py (1)
extract_json_from_markdown(73-87)backend/apps/core/models/prompt.py (3)
Prompt(14-181)get_metadata_extractor_prompt(94-101)get_evaluator_system_prompt(54-61)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: Run frontend e2e tests
- GitHub Check: Run backend tests
- GitHub Check: Run frontend unit tests
- GitHub Check: CodeQL (javascript-typescript)
🔇 Additional comments (4)
backend/apps/ai/common/constants.py (1)
3-6: LGTM! Constants align with agentic RAG workflow.The increased retrieval limit (8→32) and new constants (
DEFAULT_MAX_ITERATIONS,DEFAULT_REASONING_MODEL) appropriately support the agent's iterative retrieve-generate-evaluate loop and reasoning capabilities.backend/apps/ai/agent/nodes.py (3)
27-36: LGTM! Initialization properly validates dependencies.The constructor correctly validates the OpenAI API key and initializes the required retriever and generator components.
64-89: LGTM! Generate method correctly handles feedback and history.The augmented query properly uses real newlines (past issue resolved), and the iteration tracking and history append logic are sound.
127-133: LGTM! Routing logic correctly honors completion and iteration limits.The method appropriately routes to completion when the evaluator signals done or the iteration cap is reached.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
backend/apps/ai/agent/tools/rag/generator.py (1)
56-64: Consider improving additional_context formatting for readability.The
additional_contextdict is directly interpolated into the string, which will render as Python's dict representation (e.g.,{'key': 'value'}). For better LLM consumption and human readability, consider formatting it as JSON or unpacking key-value pairs.text = chunk.get("text", "") additional_context = chunk.get("additional_context", {}) if additional_context: + import json + formatted_additional = json.dumps(additional_context, indent=2) context_block = ( f"Source Name: {source_name}\nContent: {text}\n" - f"Additional Context: {additional_context}" + f"Additional Context:\n{formatted_additional}" ) else: context_block = f"Source Name: {source_name}\nContent: {text}"Alternatively, if
additional_contextcontains simple key-value pairs, unpack them as separate lines:if additional_context: additional_lines = "\n".join(f"{k}: {v}" for k, v in additional_context.items()) context_block = ( f"Source Name: {source_name}\nContent: {text}\n" f"Additional Context:\n{additional_lines}" )backend/apps/core/models/prompt.py (1)
23-23: Consider removing the max_length constraint from the TextField.The migration is properly structured and handles existing data correctly. However,
max_lengthon Django'sTextFieldis enforced only at the application/form level, not at the database level, making it unnecessary. For evolving system prompts in agentic workflows, removing the constraint provides more flexibility without database migration burden.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
backend/apps/ai/agent/tools/rag/generator.py(2 hunks)backend/apps/ai/models/chunk.py(1 hunks)backend/apps/core/migrations/0003_alter_prompt_text.py(1 hunks)backend/apps/core/models/prompt.py(3 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- backend/apps/core/migrations/0003_alter_prompt_text.py
- backend/apps/ai/models/chunk.py
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Run frontend unit tests
- GitHub Check: Run backend tests
- GitHub Check: Run frontend e2e tests
🔇 Additional comments (2)
backend/apps/core/models/prompt.py (2)
93-101: Need more thorough investigation. Let me search for where prompts are actually seeded:Now let me search the codebase for where other prompts are seeded to understand the pattern used:
Let me search for how other prompts are initialized in the codebase:
Let me search directly in the codebase for how prompts are structured and seeded:Based on my analysis, I need to search the actual codebase to understand the Prompt model's
get_textmethod implementation and look for where prompts are actually initialized. Unfortunately, I've exhausted web search attempts and haven't found specifics about this particular codebase's prompt seeding strategy. The absence of search results for "metadata-extractor-prompt" in fixtures, migrations, or seed files is inconclusive - the prompt could be initialized through:
- A data migration with RunPython using a different naming pattern
- Fixtures in a directory with a non-standard name
- A management command
- Admin-created entries not in version control
- Dynamic creation at application startup
Given the evidence:
- The code has defensive error handling (raises ObjectDoesNotExist if prompt is missing)
- The usage in
nodes.pyexplicitly checks for this case- No conclusive evidence of where the prompt is seeded
This falls into the **** category - I cannot make a definitive determination without being able to inspect the full codebase structure, particularly the Prompt model's
get_textmethod implementation and the complete migrations/fixtures directories.Verify that the "metadata-extractor-prompt" is seeded during database initialization.
The implementation correctly follows the established pattern. However, without locating where prompts are initialized (fixtures, data migrations, or management commands), I cannot confirm that the
"metadata-extractor-prompt"key will be available at runtime. The defensive error handling inAgentNodes.extract_query_metadata(which raisesObjectDoesNotExistif the prompt is missing) indicates this is a known risk. Ensure the database seeding process creates this prompt before the application relies on it.
53-61: Verify "evaluator-system-prompt" exists in data/nest.json.gz fixture file.The implementation follows the established pattern correctly. However, I cannot verify whether the
"evaluator-system-prompt"key is actually seeded in the fixture file from this environment.The good news: the caller already has defensive error handling (lines 236-238 in
backend/apps/ai/agent/nodes.py). It validates the prompt exists and raisesObjectDoesNotExistif missing, preventing silent failures with empty strings.To complete verification, manually check that
"evaluator-system-prompt"is defined indata/nest.json.gzalongside other prompts like"github-issue-hint"and"metadata-extractor-prompt".
|
8c8930a
into
OWASP:feature/nestbot-ai-assistant
Proposed change
Resolves #2192
Checklist
make check-testlocally; all checks and tests passed.