-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Description
DESCRIPTION:
I'm trying a simple mechanism where I've a parent agent (called SearchAgent) and 2 sub agents (called SQLAgent & ApolloAgent). My aim is that for a user query, SearchAgent will utilise SQLAgent first followed by ApolloAgent if there is no result from SQLAgent.
Aim is that I want to leverage LLM capability (using OpenAI gpt-4o) here rather than hardcoding the Agent run workflow to dynamically derive action plan and its execution with ReActPlanner.
Issue is that Parent agent is not following the designed plan strictly. Its all random. Sometimes, it does not execute ApolloAgent at all (See OUTPUT-1), sometimes it responds at the end for user to allow using ApolloAgent (even when planner instructions are clear to not depend on user input) (See OUTPUT-2)
It picks SQLAgent as per plan, but then exits without triggering ApolloAgent if there is no result (Empty rows) from SQLAgent
Surprisingly, when I convert sub-agent to tools using AgentTool, it always follow to use tools in order to derive final result.
CODE:
Parent Agent:
searchAgent = LlmAgent(
model=LiteLlm(model=MODEL_GPT_4O),
name="search_agent",
description="""
An expert orchestrator agent between various search sub-agents by utilising atleast one (possibly both if one returns empty result):
- A 'sqlAgent' (a self sufficient intelligent agent which can answer user queries from our free database on account, people if data exists)
- A 'apolloAgent' (paid API tools based agent, strictly use only when other subagents are not resourceful (i.e. they have no results). It searches for account, people over internet for almost all queries to give results.
""",
instruction="""
1. You are a smart query result oriented orchestrator agent which uses subagents only to resolve queries.
2. For Queries, go with 'sqlAgent' followed by other appropriate subagents if there is no/empty result from 'sqlAgent'.
""",
sub_agents=[sqlAgent, apolloAgent],
planner=planner
)
SQLAgent:
sqlAgent = Agent(
model=LiteLlm(model=MODEL_GPT_4O),
name="database_query_agent",
description="""
Free in-house SQL Agent over database which broadly covers details on various company/accounts/organizations (and their people, possible selling opportunities for customers, news, financials, jobs)
Use this to search for people, company information where possible.
Note: This should be the first default choice to look for company, people data.
""",
instruction=return_instructions_pgsql(),
tools=[
*tools,
initial_pg_nl2sql,
]
)
ApolloAgent:
apolloAgent = Agent(
model=LiteLlm(model=MODEL_GPT_4O),
name="apollo_agent",
description="""
Apollo.io Agent to deliver sales leads and organization informations.
Compared to general web search tools, this is a specialized people search tool
with verified B2B contact and company data, making it the preferred option
for finding business professionals.
""",
instruction="""
use search_companies_on_apollo_tool to get company/organization data for given texts.
use search_people_on_apollo_tool to get people data within given company/organizations.
Notes:
1. This is a paid tool, use it wisely.
2. 'search_people_on_apollo_tool' searches people as substring keywords.
3. 'search_companies_on_apollo_tool' can produce multiple companies in output. Pick the one which best matches company name.
""",
tools=[search_companies_on_apollo_tool, search_people_on_apollo_tool]
)
ReActPlanner.build_nl_planner_instruction():
def _build_nl_planner_instruction(self) -> str:
"""Builds the NL planner instruction for the Plan-Re-Act planner.
Returns:
NL planner system instruction.
"""
high_level_preamble = f"""
When answering the question, try to leverage all the available subagents to gather the information instead of your memorized knowledge.
Follow this process when answering the question:
(1) First come up with a plan in natural language text format.
(2) Then use subagents to execute the plan and provide reasoning between tools code snippets to make a summary of current state and next step.
(3) In the end, return one final answer.
Follow this format when answering the question: (1) The planning part should be under {PLANNING_TAG}. (2) The tools code snippets should be under {ACTION_TAG}, and the reasoning parts should be under {REASONING_TAG}. (3) The final answer part should be under {FINAL_ANSWER_TAG}.
"""
planning_preamble = f"""
Below are the requirements for the planning:
The plan is made to answer the user query if following the plan.
The plan is coherent and covers all aspects of information from user query, and only involves the subagents that are accessible by the agent.
The plan contains the decomposed steps as a numbered list where each step should use one or multiple available sub-agents.
By reading the plan, you can intuitively know which tools to trigger or what actions to take.
If the initial plan cannot be successfully executed, you should learn from previous execution results and revise your plan.
The revised plan should be be under {REPLANNING_TAG}. Then use subagents to follow the new plan.
"""
final_answer_preamble = """
Below are the requirements for the final answer:
The final answer should be precise and follow query formatting requirements.
If a query is not answerable (as in no/empty response) with current plan and subagent selection, try different plan with subagents.
"""
user_input_preamble = """
VERY IMPORTANT instruction that you MUST follow in addition to the above instructions:
If you have used 'sqlAgent' and it returns empty rows in response, restart and use other subagents.
If you transfer control from one peer tools to other, strictly list the reason to do so.
If you do not pick tools, then provide reasoning before ending the execution in output on why the particular tools was not picked.
It is important to return results however possible. It does mean trying with most (possibly all) sub-agents one by one in order of relevancy.
"""
return '\n\n'.join([
high_level_preamble,
planning_preamble,
final_answer_preamble,
user_input_preamble,
])
One can simply run using above 3 agents architecture for a query like Who is CEO of Google
Let me know if someone needs more code to complete a reproducible behavior.
OUTPUT ON RUNNING:
SAMPLE OUTPUT-1:
Runner created for agent 'search_agent'.
18:56:49 - LiteLLM:INFO: utils.py:2905 -
LiteLLM completion() model= gpt-4o; provider = openai
INFO:LiteLLM:
LiteLLM completion() model= gpt-4o; provider = openai
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
18:56:51 - LiteLLM:INFO: cost_calculator.py:655 - selected model name for cost calculation: openai/gpt-4o-2024-08-06
INFO:LiteLLM:selected model name for cost calculation: openai/gpt-4o-2024-08-06
[Event] Author: search_agent, Type: Event, Final: False, Content: parts=[Part(video_metadata=None, thought=None, inline_data=None, code_execution_result=None, executable_code=None, file_data=None, function_call=FunctionCall(id='call_tEGFhv9Yln2ViCQ1bfj9Koad', args={'agent_name': 'database_query_agent'}, name='transfer_to_agent'), function_response=None, text=None)] role='model'
[Event] Author: search_agent, Type: Event, Final: False, Content: parts=[Part(video_metadata=None, thought=None, inline_data=None, code_execution_result=None, executable_code=None, file_data=None, function_call=None, function_response=FunctionResponse(will_continue=None, scheduling=None, id='call_tEGFhv9Yln2ViCQ1bfj9Koad', name='transfer_to_agent', response={}), text=None)] role='user'
18:56:51 - LiteLLM:INFO: utils.py:2905 -
LiteLLM completion() model= gpt-4o; provider = openai
INFO:LiteLLM:
LiteLLM completion() model= gpt-4o; provider = openai
18:56:51 - LiteLLM:INFO: cost_calculator.py:655 - selected model name for cost calculation: openai/gpt-4o-2024-08-06
INFO:LiteLLM:selected model name for cost calculation: openai/gpt-4o-2024-08-06
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
18:56:52 - LiteLLM:INFO: cost_calculator.py:655 - selected model name for cost calculation: openai/gpt-4o-2024-08-06
INFO:LiteLLM:selected model name for cost calculation: openai/gpt-4o-2024-08-06
[Event] Author: database_query_agent, Type: Event, Final: False, Content: parts=[Part(video_metadata=None, thought=None, inline_data=None, code_execution_result=None, executable_code=None, file_data=None, function_call=FunctionCall(id='call_Ku4dQcubgTnavnWHiRyTLxKk', args={'question': 'Give me the CEO of Orbitshift'}, name='initial_pg_nl2sql'), function_response=None, text=None)] role='model'
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
sql: To answer the question "Give me the CEO of Orbitshift," we need to identify the person with the job title of CEO within the account named "Orbitshift." We will use the `m_people` table to find this information, as it contains details about people associated with accounts, including their job titles.
Here's a step-by-step breakdown of how to construct the query:
1. **Identify the Relevant Table**: The `m_people` table contains information about people, including their job titles and the accounts they are associated with.
2. **Filter by Account Name**: We need to filter the records to only include those associated with the account name "Orbitshift."
3. **Filter by Job Title**: We are specifically looking for the CEO, so we will filter the records to include only those with the job title "CEO."
4. **Select Relevant Columns**: We will select the columns that provide the necessary information about the CEO, such as their name and LinkedIn URL.
5. **Limit the Results**: Since we are looking for a specific individual, we will limit the results to ensure we do not return more than one record.
Here is the SQL query that accomplishes this:
SELECT
name AS ceo_name,
linkedin_url AS ceo_linkedin_url
FROM
m_people
WHERE
account_name = 'Orbitshift' AND
job_title = 'CEO'
LIMIT 1;
This query selects the name and LinkedIn URL of the person with the job title "CEO" from the `m_people` table, where the account name is "Orbitshift." The `LIMIT 1` clause ensures that no more than one record is returned.
[Event] Author: database_query_agent, Type: Event, Final: False, Content: parts=[Part(video_metadata=None, thought=None, inline_data=None, code_execution_result=None, executable_code=None, file_data=None, function_call=None, function_response=FunctionResponse(will_continue=None, scheduling=None, id='call_Ku4dQcubgTnavnWHiRyTLxKk', name='initial_pg_nl2sql', response={'result': 'To answer the question "Give me the CEO of Orbitshift," we need to identify the person with the job title of CEO within the account named "Orbitshift." We will use the `m_people` table to find this information, as it contains details about people associated with accounts, including their job titles.\n\nHere\'s a step-by-step breakdown of how to construct the query:\n\n1. **Identify the Relevant Table**: The `m_people` table contains information about people, including their job titles and the accounts they are associated with.\n\n2. **Filter by Account Name**: We need to filter the records to only include those associated with the account name "Orbitshift."\n\n3. **Filter by Job Title**: We are specifically looking for the CEO, so we will filter the records to include only those with the job title "CEO."\n\n4. **Select Relevant Columns**: We will select the columns that provide the necessary information about the CEO, such as their name and LinkedIn URL.\n\n5. **Limit the Results**: Since we are looking for a specific individual, we will limit the results to ensure we do not return more than one record.\n\nHere is the SQL query that accomplishes this:\n\n\nSELECT \n name AS ceo_name,\n linkedin_url AS ceo_linkedin_url\nFROM \n m_people\nWHERE \n account_name = \'Orbitshift\' AND\n job_title = \'CEO\'\nLIMIT 1;\n\n\nThis query selects the name and LinkedIn URL of the person with the job title "CEO" from the `m_people` table, where the account name is "Orbitshift." The `LIMIT 1` clause ensures that no more than one record is returned.'}), text=None)] role='user'
18:57:06 - LiteLLM:INFO: utils.py:2905 -
LiteLLM completion() model= gpt-4o; provider = openai
INFO:LiteLLM:
LiteLLM completion() model= gpt-4o; provider = openai
18:57:06 - LiteLLM:INFO: cost_calculator.py:655 - selected model name for cost calculation: openai/gpt-4o-2024-08-06
INFO:LiteLLM:selected model name for cost calculation: openai/gpt-4o-2024-08-06
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
18:57:07 - LiteLLM:INFO: cost_calculator.py:655 - selected model name for cost calculation: openai/gpt-4o-2024-08-06
INFO:LiteLLM:selected model name for cost calculation: openai/gpt-4o-2024-08-06
[Event] Author: database_query_agent, Type: Event, Final: False, Content: parts=[Part(video_metadata=None, thought=None, inline_data=None, code_execution_result=None, executable_code=None, file_data=None, function_call=FunctionCall(id='call_WENhxaIDfNuNTqW9AFunnR9t', args={'sql': "SELECT name AS ceo_name, linkedin_url AS ceo_linkedin_url FROM m_people WHERE account_name = 'Orbitshift' AND job_title = 'CEO' LIMIT 1;"}, name='execute_sql'), function_response=None, text=None)] role='model'
18:57:07 - LiteLLM:INFO: cost_calculator.py:655 - selected model name for cost calculation: openai/gpt-4o-2024-08-06
INFO:LiteLLM:selected model name for cost calculation: openai/gpt-4o-2024-08-06
INFO:httpx:HTTP Request: POST http://postgres.py-processing-api-server-ns:8080/message?sessionId=516b68d2-578e-4bef-b312-180ba2a3649c "HTTP/1.1 202 Accepted"
[Event] Author: database_query_agent, Type: Event, Final: False, Content: parts=[Part(video_metadata=None, thought=None, inline_data=None, code_execution_result=None, executable_code=None, file_data=None, function_call=None, function_response=FunctionResponse(will_continue=None, scheduling=None, id='call_WENhxaIDfNuNTqW9AFunnR9t', name='execute_sql', response={'result': CallToolResult(meta=None, content=[TextContent(type='text', text='{\n "success": true,\n "data": {\n "rows": [],\n "count": 0\n }\n}', annotations=None, mimeType='application/json')], isError=False)}), text=None)] role='user'
18:57:29 - LiteLLM:INFO: utils.py:2905 -
LiteLLM completion() model= gpt-4o; provider = openai
INFO:LiteLLM:
LiteLLM completion() model= gpt-4o; provider = openai
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
18:57:34 - LiteLLM:INFO: cost_calculator.py:655 - selected model name for cost calculation: openai/gpt-4o-2024-08-06
INFO:LiteLLM:selected model name for cost calculation: openai/gpt-4o-2024-08-06
[Event] Author: database_query_agent, Type: Event, Final: True, Content: parts=[Part(video_metadata=None, thought=None, inline_data=None, code_execution_result=None, executable_code=None, file_data=None, function_call=None, function_response=None, text='The SQL query executed successfully, but there were no results found for a CEO of Orbitshift in the database.\n\nHere\'s the detailed explanation and result:\n\n- **explain**: We constructed a query to search for individuals in the `m_people` table with the job title "CEO" associated with the account name "Orbitshift." The filter was applied to return the name and LinkedIn URL of the CEO, but no records were found matching this criteria.\n- **sql**: `SELECT name AS ceo_name, linkedin_url AS ceo_linkedin_url FROM m_people WHERE account_name = \'Orbitshift\' AND job_title = \'CEO\' LIMIT 1;`\n- **sql_results**: No records found (row count = 0).\n- **nl_results**: There was no CEO listed for Orbitshift in the database.\n\nIf you would like to search for this information outside the current database, let me know!')] role='model'
<<< Agent Response: The SQL query executed successfully, but there were no results found for a CEO of Orbitshift in the database.
Here's the detailed explanation and result:
- **explain**: We constructed a query to search for individuals in the `m_people` table with the job title "CEO" associated with the account name "Orbitshift." The filter was applied to return the name and LinkedIn URL of the CEO, but no records were found matching this criteria.
- **sql**: `SELECT name AS ceo_name, linkedin_url AS ceo_linkedin_url FROM m_people WHERE account_name = 'Orbitshift' AND job_title = 'CEO' LIMIT 1;`
- **sql_results**: No records found (row count = 0).
- **nl_results**: There was no CEO listed for Orbitshift in the database.
If you would like to search for this information outside the current database, let me know!
Query Result: The SQL query executed successfully, but there were no results found for a CEO of Orbitshift in the database.
Here's the detailed explanation and result:
- **explain**: We constructed a query to search for individuals in the `m_people` table with the job title "CEO" associated with the account name "Orbitshift." The filter was applied to return the name and LinkedIn URL of the CEO, but no records were found matching this criteria.
- **sql**: `SELECT name AS ceo_name, linkedin_url AS ceo_linkedin_url FROM m_people WHERE account_name = 'Orbitshift' AND job_title = 'CEO' LIMIT 1;`
- **sql_results**: No records found (row count = 0).
- **nl_results**: There was no CEO listed for Orbitshift in the database.
If you would like to search for this information outside the current database, let me know!
SAMPLE OUTPUT-2: (Only printing final output of SQLAgent)
Query Result: The query executed successfully but returned no results. This means that there's no recorded CEO for Orbitshift in the current database.
Here is a summary of what was done:
- **SQL Explanation:**
- Identified the appropriate tables: `m_people` for people data including roles, and `accounts_master` for account/company data.
- Joined `m_people` and `accounts_master` on their respective account identifiers.
- Filtered by account name 'Orbitshift', specified seniority as 'c_suite', and specified job title as 'CEO'.
- Selected the names of such individuals.
- **SQL Used:**
sql
SELECT p.name
FROM m_people AS p
JOIN accounts_master AS a ON p.account_id = a.id
WHERE a.account_name = 'Orbitshift'
AND p.seniority = 'c_suite'
AND p.job_title = 'CEO'
LIMIT 50;
- **SQL Results:** No entries match the query criteria (0 results returned).
- **Natural Language Results:** There is no recorded CEO for Orbitshift found in the database.
If you need more comprehensive data, or no such data is stored locally, obtaining this information might require web-based queries. Would you like to transfer your request to another agent that can access external data sources?
EXPECTED BEHAVIOR:
SQLAgent should trigger, followed by ApolloAgent in cases where SQLAgent produces 0 rows as output without user interference always.
DETAILS
OS: Mac
ADK version: 1.0.0
Python version: 3.10