AI Agent Design & Concepts - Class Notes Compilation
What are AI Agents?
- Goal-driven systems: The goal is set by a human (e.g., marketing manager).
- Autonomous: Once the goal is set and optionally approved, the agent executes the task on its own.
- Decision-making: Must be rational and data-informed (e.g., not recommending thrillers to someone
who only watches sci-fi).
- Interaction with environment: Collects data, builds context, and takes action toward achieving the
goal.
- Uses LLMs for understanding and reasoning, but also relies on external tools.
Benefits of AI Agents
- Improved productivity: Handles repetitive tasks.
- Human focus on strategy: Humans can focus on high-level work.
- Reduced costs: Automates processing.
- Better customer experience: Through personalization and conversational interfaces.
- Enables new use cases: Like content generation and natural language interfaces.
Challenges
- Data privacy concerns
- Large data volumes
- Bias, unfairness, hallucinations
- Technical complexity: Requires knowledge and experience.
- High resource requirements: Compute power, storage, stable internet.
Agentic Design Patterns
1. Planning + Execution: Agent builds a plan, then executes.
2. Tool Use: Agent can search the web, run code, query databases using registered tools.
3. ReAct (Reason + Act): Agent reasons through a task step-by-step, taking actions as needed.
4. Memory-Based Agents: Retain information across steps or sessions.
5. Multi-Agent Systems: Different agents handle different tasks (e.g., researcher + summarizer).
6. RAG (Retrieval-Augmented Generation):
- Retrieves relevant documents.
- Adds context to LLM prompt.
- LLM then answers using retrieved information.
7. Multi-Agent Collaboration:
- Break down large, complex jobs into smaller tasks.
- Assign tasks to specialized agents (e.g., researcher, summarizer, coder, tester, critic).
- Agents communicate and coordinate to complete the overall goal.
- Especially useful for very large projects that require division of roles and autonomy.
Tools and Function Calling
- LLMs aren't always good at tasks like math
- Agents can define Python functions as "tools"
- eval() in Python: evaluates string expressions safely
- Agents use a JSON schema to describe how a tool works (its inputs and outputs)
- You add tools via the tools attribute when initializing the agent
Retrieval-Augmented Generation (RAG)
- LLMs lack specific/niche data
- RAG system does:
1. Get question from user
2. Retrieve documents from a vector store
3. Include them in LLM prompt as context
4. Generate answer based on context
Reflection & Improvement
- Agent critiques its own code/output using an LLM
- LLM provides feedback
- Agent incorporates the critique, iterates, and improves
- Example: Python prime-checker code critiqued and updated
Using Data with Agents
- Use document loaders for different file types (CSV, PDF, Word, etc.)
- Use vector stores for unstructured info
- Use SQL agents to query structured databases with natural language
- Combine structured (SQL) + unstructured (PDF) data for hybrid answers
Final Insights
- AI agents = digital interns that plan, reason, and act independently
- They rely on context, tools, and retrieval to succeed
- You can mix LLMs based on strengths (e.g., GPT-3.5 for reasoning, GPT-4 for critique)
- Its okay to feel overwhelmed start small, build up
- Large jobs can be broken down into smaller tasks and distributed across agents
- Each agent in a team can handle a specific role: testing, critique, generation, validation, etc.
References & Follow-up
- Class project folder includes: Gen AI Confluence LangChain
- Request slides from the professor at the end of the course
- Course ends this week (Summer Camp format)
*Bhau, you survived the lecture, took fire notes, and now youre ready to actually build this stuff.
Respect.*
