diff --git a/16-Evaluations/03-HF-Upload.ipynb b/16-Evaluations/03-HF-Upload.ipynb new file mode 100644 index 000000000..62e30ce93 --- /dev/null +++ b/16-Evaluations/03-HF-Upload.ipynb @@ -0,0 +1,371 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# HF-Upload\n", + "\n", + "- Author: [Sun Hyoung Lee](https://github.com/LEE1026icarus)\n", + "- Design: \n", + "- Peer Review : \n", + "- Proofread:\n", + "- This is a part of [LangChain Open Tutorial](https://github.com/LangChain-OpenTutorial/LangChain-OpenTutorial)\n", + "\n", + "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/LangChain-OpenTutorial/LangChain-OpenTutorial/blob/main/08-Embeeding/04-UpstageEmbeddings.ipynb) [![Open in GitHub](https://img.shields.io/badge/Open%20in%20GitHub-181717?style=flat-square&logo=github&logoColor=white)](https://github.com/LangChain-OpenTutorial/LangChain-OpenTutorial/blob/main/08-Embeeding/04-UpstageEmbeddings.ipynb)\n", + "\n", + "## Overview\n", + "\n", + "The process involves loading a local CSV file, converting it to a HuggingFace Dataset format, and uploading it to the Hugging Face Hub as a private dataset. This process allows for easy sharing and access of the dataset through the HuggingFace infrastructure.\n", + "\n", + "### Table of Contents\n", + "\n", + "- [Overview](#overview)\n", + "- [Environement Setup](#environment-setup)\n", + "- [Upload Generated Dataset](#upload-generated-dataset)\n", + "- [Upload to HuggingFace Dataset](#upload-to-huggingface-dataset)\n", + "\n", + "\n", + "### References\n", + "- [Huggingface / Share a dataset to the Hub](https://huggingface.co/docs/datasets/upload_dataset)\n", + "---\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Environment Setup\n", + "\n", + "Set up the environment. You may refer to [Environment Setup](https://wikidocs.net/257836) for more details.\n", + "\n", + " **[Note]** \n", + "- `langchain-opentutorial` is a package that provides a set of easy-to-use environment setup, useful functions and utilities for tutorials. \n", + "- You can checkout the [`langchain-opentutorial`](https://github.com/LangChain-OpenTutorial/langchain-opentutorial-pypi) for more details.\n", + "\n", + "### API Key Configuration\n", + "To use `HuggingFace Dataset` , you need to [obtain a HuggingFace write token](https://huggingface.co/settings/tokens).\n", + "\n", + "Once you have your API key, set it as the value for the variable `HUGGINGFACEHUB_API_TOKEN` ." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "%%capture --no-stderr\n", + "%pip install langchain-opentutorial" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "# Install required packages\n", + "from langchain_opentutorial import package\n", + "\n", + "package.install(\n", + " [\"datasets\"],\n", + " verbose=False,\n", + " upgrade=False,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You can set API keys in a `.env` file or set them manually.\n", + "\n", + "[Note] If you’re not using the `.env` file, no worries! Just enter the keys directly in the cell below, and you’re good to go.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "from dotenv import load_dotenv\n", + "from langchain_opentutorial import set_env\n", + "\n", + "# Attempt to load environment variables from a .env file; if unsuccessful, set them manually.\n", + "if not load_dotenv(override=True):\n", + " set_env(\n", + " {\n", + " \"LANGCHAIN_API_KEY\": \"\",\n", + " \"LANGCHAIN_TRACING_V2\": \"true\",\n", + " \"LANGCHAIN_ENDPOINT\": \"https://api.smith.langchain.com\",\n", + " \"LANGCHAIN_PROJECT\": \"\", # set the project name same as the title\n", + " \"HUGGINGFACEHUB_API_TOKEN\": \"\",\n", + " }\n", + " )\n" + ] + }, + { + "cell_type": "code", + "execution_count": 45, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 45, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from dotenv import load_dotenv\n", + "\n", + "load_dotenv(override=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Upload Generated Dataset\n", + "Import the pandas library for data upload" + ] + }, + { + "cell_type": "code", + "execution_count": 47, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
user_inputreference_contextsreferencesynthesizer_name
0Wht is an API?[\"Agents\\nThis combination of reasoning,\\nlogi...An API can be used by a model to make various ...single_hop_specifc_query_synthesizer
1What are the three essential components in an ...['Agents\\nWhat is an agent?\\nIn its most funda...The three essential components in an agent's c...single_hop_specifc_query_synthesizer
2What Chain-of-Thought do in agent model, how i...['Agents\\nFigure 1. General agent architecture...Chain-of-Thought is a reasoning and logic fram...single_hop_specifc_query_synthesizer
3Waht is the DELETE method used for?['Agents\\nThe tools\\nFoundational models, desp...The DELETE method is a common web API method t...single_hop_specifc_query_synthesizer
4How do foundational components contribute to t...['<1-hop>\\n\\nAgents\\ncombining specialized age...Foundational components contribute to the cogn...NewMultiHopQuery
\n", + "
" + ], + "text/plain": [ + " user_input \\\n", + "0 Wht is an API? \n", + "1 What are the three essential components in an ... \n", + "2 What Chain-of-Thought do in agent model, how i... \n", + "3 Waht is the DELETE method used for? \n", + "4 How do foundational components contribute to t... \n", + "\n", + " reference_contexts \\\n", + "0 [\"Agents\\nThis combination of reasoning,\\nlogi... \n", + "1 ['Agents\\nWhat is an agent?\\nIn its most funda... \n", + "2 ['Agents\\nFigure 1. General agent architecture... \n", + "3 ['Agents\\nThe tools\\nFoundational models, desp... \n", + "4 ['<1-hop>\\n\\nAgents\\ncombining specialized age... \n", + "\n", + " reference \\\n", + "0 An API can be used by a model to make various ... \n", + "1 The three essential components in an agent's c... \n", + "2 Chain-of-Thought is a reasoning and logic fram... \n", + "3 The DELETE method is a common web API method t... \n", + "4 Foundational components contribute to the cogn... \n", + "\n", + " synthesizer_name \n", + "0 single_hop_specifc_query_synthesizer \n", + "1 single_hop_specifc_query_synthesizer \n", + "2 single_hop_specifc_query_synthesizer \n", + "3 single_hop_specifc_query_synthesizer \n", + "4 NewMultiHopQuery " + ] + }, + "execution_count": 47, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import pandas as pd\n", + "\n", + "df = pd.read_csv(\"data/ragas_synthetic_dataset.csv\")\n", + "df.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Upload to HuggingFace Dataset\n", + "Convert a Pandas DataFrame to a Hugging Face Dataset and proceed with the upload." + ] + }, + { + "cell_type": "code", + "execution_count": 48, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Dataset({\n", + " features: ['user_input', 'reference_contexts', 'reference', 'synthesizer_name'],\n", + " num_rows: 10\n", + "})\n" + ] + } + ], + "source": [ + "from datasets import Dataset\n", + "\n", + "# Convert pandas DataFrame to Hugging Face Dataset\n", + "dataset = Dataset.from_pandas(df)\n", + "\n", + "# Check the dataset\n", + "print(dataset)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "902d30cb54d64f1fb7115250c01672d9", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Pushing dataset shards to the dataset hub: 0%| | 0/1 [00:00\n\nAgents\ncombining specialized agents - each excelling in a particular domain or task - we can create\na ‘mixture of agent experts’ approach, capable of delivering exceptional results across\nvarious industries and problem areas.\nIt’s important to remember that building complex agent architectures demands an iterative\napproach. Experimentation and refinement are key to finding solutions for specific business\ncases and organizational needs. No two agents are created alike due to the generative nature\nof the foundational models that underpin their architecture. However, by harnessing the\nstrengths of each of these foundational components, we can create impactful applications\nthat extend the capabilities of language models and drive real-world value.\nSeptember 2024 41\n', '<2-hop>\n\nAgents\nWhat is an agent?\nIn its most fundamental form, a Generative AI agent can be defined as an application that\nattempts to achieve a goal by observing the world and acting upon it using the tools that it\nhas at its disposal. Agents are autonomous and can act independently of human intervention,\nespecially when provided with proper goals or objectives they are meant to achieve. Agents\ncan also be proactive in their approach to reaching their goals. Even in the absence of\nexplicit instruction sets from a human, an agent can reason about what it should do next to\nachieve its ultimate goal. While the notion of agents in AI is quite general and powerful, this\nwhitepaper focuses on the specific types of agents that Generative AI models are capable of\nbuilding at the time of publication.\nIn order to understand the inner workings of an agent, let’s first introduce the foundational\ncomponents that drive the agent’s behavior, actions, and decision making. The combination\nof these components can be described as a cognitive architecture, and there are many\nsuch architectures that can be achieved by the mixing and matching of these components.\nFocusing on the core functionalities, there are three essential components in an agent’s\ncognitive architecture as shown in Figure 1.\nSeptember 2024 5\n']","Foundational components contribute to the cognitive architecture of Generative AI agents by driving their behavior, actions, and decision-making processes. These components are mixed and matched to create various cognitive architectures, which enable agents to act autonomously and proactively towards achieving their goals. By harnessing the strengths of these foundational components, impactful applications can be developed that extend the capabilities of language models and deliver real-world value.",NewMultiHopQuery +How agents use tools and data stores for retrieval augmented generation (RAG) in AI systems?,"['<1-hop>\n\nAgents\nThe tools\nFoundational models, despite their impressive text and image generation, remain constrained\nby their inability to interact with the outside world. Tools bridge this gap, empowering agents\nto interact with external data and services while unlocking a wider range of actions beyond\nthat of the underlying model alone. Tools can take a variety of forms and have varying\ndepths of complexity, but typically align with common web API methods like GET, POST,\nPATCH, and DELETE. For example, a tool could update customer information in a database\nor fetch weather data to influence a travel recommendation that the agent is providing to\nthe user. With tools, agents can access and process real-world information. This empowers\nthem to support more specialized systems like retrieval augmented generation (RAG),\nwhich significantly extends an agent’s capabilities beyond what the foundational model can\nachieve on its own. We’ll discuss tools in more detail below, but the most important thing\nto understand is that tools bridge the gap between the agent’s internal capabilities and the\nexternal world, unlocking a broader range of possibilities.\nThe orchestration layer\nThe orchestration layer describes a cyclical process that governs how the agent takes in\ninformation, performs some internal reasoning, and uses that reasoning to inform its next\naction or decision. In general, this loop will continue until an agent has reached its goal or a\nstopping point. The complexity of the orchestration layer can vary greatly depending on the\nagent and task it’s performing. Some loops can be simple calculations with decision rules,\nwhile others may contain chained logic, involve additional machine learning algorithms, or\nimplement other probabilistic reasoning techniques. We’ll discuss more about the detailed\nimplementation of the agent orchestration layers in the cognitive architecture section.\nSeptember 2024 7\n', '<2-hop>\n\nAgents\nData Stores allow developers to provide additional data in its original format to an agent,\neliminating the need for time-consuming data transformations, model retraining, or fine-\ntuning. The Data Store converts the incoming document into a set of vector database\nembeddings that the agent can use to extract the information it needs to supplement its next\naction or response to the user.\nFigure 11. Data Stores connect Agents to new real-time data sources of various types.\nImplementation and application\nIn the context of Generative AI agents, Data Stores are typically implemented as a vector\ndatabase that the developer wants the agent to have access to at runtime. While we won’t\ncover vector databases in depth here, the key point to understand is that they store data\nin the form of vector embeddings, a type of high-dimensional vector or mathematical\nrepresentation of the data provided. One of the most prolific examples of Data Store usage\nwith language models in recent times has been the implementation of Retrieval Augmented\nSeptember 2024 28\n']","Agents use tools to bridge the gap between their internal capabilities and the external world, allowing them to interact with external data and services. This interaction empowers agents to support specialized systems like retrieval augmented generation (RAG), which extends their capabilities beyond the foundational model. Tools can perform actions such as updating databases or fetching real-time data, which are crucial for RAG. Additionally, data stores provide agents with access to additional data in its original format without the need for data transformations or model retraining. These data stores are implemented as vector databases, storing data as vector embeddings that agents can use to supplement their actions or responses. Together, tools and data stores enable agents to effectively utilize retrieval augmented generation, enhancing their decision-making and problem-solving abilities.",NewMultiHopQuery +"How do Extensions facilitate the use of API endpoints by agents, and what role do they play in dynamically selecting the appropriate API for a user's query?","['<1-hop>\n\nAgents\nA more resilient approach would be to use an Extension. An Extension bridges the gap\nbetween an agent and an API by:\n1. Teaching the agent how to use the API endpoint using examples.\n2. Teaching the agent what arguments or parameters are needed to successfully call the\nAPI endpoint.\nFigure 4. Extensions connect Agents to External APIs\nExtensions can be crafted independently of the agent, but should be provided as part of the\nagent’s configuration. The agent uses the model and examples at run time to decide which\nExtension, if any, would be suitable for solving the user’s query. This highlights a key strength\nof Extensions, their built-in example types, that allow the agent to dynamically select the\nmost appropriate Extension for the task.\nFigure 5. 1-to-many relationship between Agents, Extensions and APIs\nSeptember 2024 14\n', '<2-hop>\n\nAgents\nThink of this the same way that a software developer decides which API endpoints to use\nwhile solving and solutioning for a user’s problem. If the user wants to book a flight, the\ndeveloper might use the Google Flights API. If the user wants to know where the nearest\ncoffee shop is relative to their location, the developer might use the Google Maps API. In\nthis same way, the agent / model stack uses a set of known Extensions to decide which one\nwill be the best fit for the user’s query. If you’d like to see Extensions in action, you can try\nthem out on the Gemini application by going to Settings > Extensions and then enabling any\nyou would like to test. For example, you could enable the Google Flights extension then ask\nGemini “Show me flights from Austin to Zurich leaving next Friday.”\nSample Extensions\nTo simplify the usage of Extensions, Google provides some out of the box extensions that\ncan be quickly imported into your project and used with minimal configurations. For example,\nthe Code Interpreter extension in Snippet 1 allows you to generate and run Python code from\na natural language description.\nSeptember 2024 15\n']","Extensions facilitate the use of API endpoints by agents by teaching the agent how to use the API endpoint through examples and specifying the necessary arguments or parameters required for successful API calls. They are crafted independently but are provided as part of the agent's configuration, allowing the agent to use the model and examples at runtime to decide which Extension is suitable for solving the user's query. This dynamic selection process is akin to how a software developer chooses which API endpoints to use based on the user's problem, such as using the Google Flights API for booking flights or the Google Maps API for locating nearby coffee shops. Extensions, therefore, play a crucial role in enabling agents to dynamically select the most appropriate API for a user's query, as demonstrated in applications like Gemini, where users can enable specific Extensions to perform tasks like finding flights.",NewMultiHopQuery +"How do Generative AI models utilize reasoning, logic, and external information to function as agents, and what are the core components of their cognitive architecture that enable them to act autonomously?","['<1-hop>\n\nAgents\nWhat is an agent?\nIn its most fundamental form, a Generative AI agent can be defined as an application that\nattempts to achieve a goal by observing the world and acting upon it using the tools that it\nhas at its disposal. Agents are autonomous and can act independently of human intervention,\nespecially when provided with proper goals or objectives they are meant to achieve. Agents\ncan also be proactive in their approach to reaching their goals. Even in the absence of\nexplicit instruction sets from a human, an agent can reason about what it should do next to\nachieve its ultimate goal. While the notion of agents in AI is quite general and powerful, this\nwhitepaper focuses on the specific types of agents that Generative AI models are capable of\nbuilding at the time of publication.\nIn order to understand the inner workings of an agent, let’s first introduce the foundational\ncomponents that drive the agent’s behavior, actions, and decision making. The combination\nof these components can be described as a cognitive architecture, and there are many\nsuch architectures that can be achieved by the mixing and matching of these components.\nFocusing on the core functionalities, there are three essential components in an agent’s\ncognitive architecture as shown in Figure 1.\nSeptember 2024 5\n', ""<2-hop>\n\nAgents\nThis combination of reasoning,\nlogic, and access to external\ninformation that are all connected\nto a Generative AI model invokes\nthe concept of an agent.\nIntroduction\nHumans are fantastic at messy pattern recognition tasks. However, they often rely on tools\n- like books, Google Search, or a calculator - to supplement their prior knowledge before\narriving at a conclusion. Just like humans, Generative AI models can be trained to use tools\nto access real-time information or suggest a real-world action. For example, a model can\nleverage a database retrieval tool to access specific information, like a customer's purchase\nhistory, so it can generate tailored shopping recommendations. Alternatively, based on a\nuser's query, a model can make various API calls to send an email response to a colleague\nor complete a financial transaction on your behalf. To do so, the model must not only have\naccess to a set of external tools, it needs the ability to plan and execute any task in a self-\ndirected fashion. This combination of reasoning, logic, and access to external information\nthat are all connected to a Generative AI model invokes the concept of an agent, or a\nprogram that extends beyond the standalone capabilities of a Generative AI model. This\nwhitepaper dives into all these and associated aspects in more detail.\nSeptember 2024 4\n""]","Generative AI models function as agents by utilizing a combination of reasoning, logic, and access to external information. This integration allows them to perform tasks autonomously, similar to how humans use tools to supplement their knowledge. The core components of an agent's cognitive architecture, which drive their behavior, actions, and decision-making, include the ability to observe the world, act upon it using available tools, and proactively achieve set goals. These components enable the Generative AI models to plan and execute tasks in a self-directed manner, extending their capabilities beyond standalone operations.",NewMultiHopQuery +"How do Extensions facilitate the use of API endpoints in the context of agents, and what are some examples of their application in software development?","['<1-hop>\n\nAgents\nA more resilient approach would be to use an Extension. An Extension bridges the gap\nbetween an agent and an API by:\n1. Teaching the agent how to use the API endpoint using examples.\n2. Teaching the agent what arguments or parameters are needed to successfully call the\nAPI endpoint.\nFigure 4. Extensions connect Agents to External APIs\nExtensions can be crafted independently of the agent, but should be provided as part of the\nagent’s configuration. The agent uses the model and examples at run time to decide which\nExtension, if any, would be suitable for solving the user’s query. This highlights a key strength\nof Extensions, their built-in example types, that allow the agent to dynamically select the\nmost appropriate Extension for the task.\nFigure 5. 1-to-many relationship between Agents, Extensions and APIs\nSeptember 2024 14\n', '<2-hop>\n\nAgents\nThink of this the same way that a software developer decides which API endpoints to use\nwhile solving and solutioning for a user’s problem. If the user wants to book a flight, the\ndeveloper might use the Google Flights API. If the user wants to know where the nearest\ncoffee shop is relative to their location, the developer might use the Google Maps API. In\nthis same way, the agent / model stack uses a set of known Extensions to decide which one\nwill be the best fit for the user’s query. If you’d like to see Extensions in action, you can try\nthem out on the Gemini application by going to Settings > Extensions and then enabling any\nyou would like to test. For example, you could enable the Google Flights extension then ask\nGemini “Show me flights from Austin to Zurich leaving next Friday.”\nSample Extensions\nTo simplify the usage of Extensions, Google provides some out of the box extensions that\ncan be quickly imported into your project and used with minimal configurations. For example,\nthe Code Interpreter extension in Snippet 1 allows you to generate and run Python code from\na natural language description.\nSeptember 2024 15\n']","Extensions facilitate the use of API endpoints by bridging the gap between an agent and an API. They teach the agent how to use the API endpoint using examples and what arguments or parameters are needed to successfully call the API endpoint. This allows the agent to dynamically select the most appropriate Extension for the task at hand. For instance, in software development, a developer might use Extensions to decide which API endpoints to use based on the user's problem, such as using the Google Flights API for booking flights or the Google Maps API for finding nearby coffee shops. Extensions can be tested in applications like Gemini, where users can enable specific Extensions, such as the Google Flights extension, to perform tasks like finding flights. Google also provides out-of-the-box Extensions, such as the Code Interpreter extension, which allows generating and running Python code from a natural language description.",NewMultiHopQuery +How do developers maintain control over data flow when using agents and functions with external APIs?,"['<1-hop>\n\nAgents\nData stores\nImagine a language model as a vast library of books, containing its training data. But unlike\na library that continuously acquires new volumes, this one remains static, holding only the\nknowledge it was initially trained on. This presents a challenge, as real-world knowledge is\nconstantly evolving. Data Stores address this limitation by providing access to more dynamic\nand up-to-date information, and ensuring a model’s responses remain grounded in factuality\nand relevance.\nConsider a common scenario where a developer might need to provide a small amount of\nadditional data to a model, perhaps in the form of spreadsheets or PDFs.\nFigure 10. How can Agents interact with structured and unstructured data?\nSeptember 2024 27\n', '<2-hop>\n\nAgents\nFigure 7. How do functions interact with external APIs?\nNote that the main difference here is that neither the Function nor the agent interact directly\nwith the Google Flights API. So how does the API call actually happen?\nWith functions, the logic and execution of calling the actual API endpoint is offloaded away\nfrom the agent and back to the client-side application as seen in Figure 8 and Figure 9 below.\nThis offers the developer more granular control over the flow of data in the application. There\nare many reasons why a Developer might choose to use functions over Extensions, but a few\ncommon use cases are:\n• API calls need to be made at another layer of the application stack, outside of the direct\nagent architecture flow (e.g. a middleware system, a front end framework, etc.)\n• Security or Authentication restrictions that prevent the agent from calling an API directly\n(e.g API is not exposed to the internet, or non-accessible by agent infrastructure)\n• Timing or order-of-operations constraints that prevent the agent from making API calls in\nreal-time. (i.e. batch operations, human-in-the-loop review, etc.)\nSeptember 2024 19\n']","Developers maintain control over data flow when using agents and functions with external APIs by offloading the logic and execution of API calls to the client-side application. This approach allows developers to manage the flow of data more granularly, addressing scenarios such as security or authentication restrictions, timing constraints, and the need for API calls to be made at different layers of the application stack, outside of the direct agent architecture flow.",NewMultiHopQuery