LangChain-OpenTutorial · teddylee777 · Jan 2, 2025 · Dec 30, 2024 · Dec 31, 2024 · Dec 31, 2024
diff --git a/01-BASIC/03-LangSmithTrackingSetup.ipynb b/01-BASIC/03-LangSmithTrackingSetup.ipynb
@@ -0,0 +1,237 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# LangSmith Tracking Setup\n",
+    "\n",
+    "- Author: [JeongGi Park](https://github.com/jeongkpa)\n",
+    "- Design: []()\n",
+    "- Peer Review:[Wooseok Jeong](https://github.com/jeong-wooseok), [r14minji](https://github.com/r14minji)\n",
+    "- This is a part of [LangChain Open Tutorial](https://github.com/LangChain-OpenTutorial/LangChain-OpenTutorial)\n",
+    "\n",
+    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain-academy/blob/main/module-4/sub-graph.ipynb) [![Open in LangChain Academy](https://cdn.prod.website-files.com/65b8cd72835ceeacd4449a53/66e9eba12c7b7688aa3dbb5e_LCA-badge-green.svg)](https://academy.langchain.com/courses/take/intro-to-langgraph/lessons/58239937-lesson-2-sub-graphs)\n",
+    "\n",
+    "## Overview\n",
+    "\n",
+    "This tutorial covers how to set up and use LangSmith, a powerful platform for developing, monitoring, and testing LLM applications. \n",
+    "LangSmith provides comprehensive tracking capabilities that are essential for understanding and optimizing your LLM applications.\n",
+    "\n",
+    "LangSmith tracking helps you monitor:\n",
+    "\n",
+    "Token usage and associated costs\n",
+    "- Execution time and performance metrics\n",
+    "- Error rates and unexpected behaviors\n",
+    "- Agent interactions and chain operations\n",
+    "\n",
+    "In this tutorial, we'll walk through the process of setting up LangSmith tracking and integrating it with your LangChain applications.\n",
+    "\n",
+    "### Table of Contents\n",
+    "\n",
+    "- [Overview](#overview)\n",
+    "- [Setting up a LangSmith trace](#setting-up-a-langsmith-trace)\n",
+    "- [Using LangSmith tracking](#using-langsmith-tracking)\n",
+    "- [Enable tracking in your Jupyter notebook or code](#enable-tracking-in-your-jupyter-notebook-or-code)\n",
+    "\n",
+    "### References\n",
+    "\n",
+    "- [OpenAI API Pricing](https://openai.com/api/pricing/)\n",
+    "- [Token Usage Guide](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them)\n",
+    "- [LangChain Python API Reference](https://python.langchain.com/api_reference/community/callbacks/langchain_community.callbacks.manager.get_openai_callback.html)\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setting up a LangSmith trace\n",
+    "\n",
+    "LangSmith is a platform for developing, monitoring, and testing LLM applications. \n",
+    "If you're starting a project or learning LangChain, LangSmith is a must-have to get set up and running."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Project-Level Tracking\n",
+    "At the project level, you can check execution counts, error rates, token usage, and billing information.\n",
+    "\n",
+    "![project-level-tracking](./assets/03-langsmith-tracking-setup-01.png)\n",
+    "\n",
+    "When you click on a project, all executed Runs appear.\n",
+    "\n",
+    "![project-level-tracking-detail](./assets/03-langsmith-tracking-setup-02.png)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Detailed Step-by-Step Tracking for a Single Execution\n",
+    "\n",
+    "![detailed-step-by-step-tracking](./assets/03-langsmith-tracking-setup-03.png)\n",
+    "\n",
+    "\n",
+    "After a single execution, it records not only the search results of retrieved documents but also detailed logs of GPT's input and output content. \n",
+    "Therefore, it helps you determine whether to change the search algorithm or modify prompts after reviewing the searched content.\n",
+    "\n",
+    "\n",
+    "Moreover, at the top, it shows the time taken for a single Run (about 30 seconds) and tokens used (5,104), and when you hover over the tokens, it displays the billing amount."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using LangSmith tracking\n",
+    "\n",
+    "Using traces is very simple."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Get a LangSmith API Key\n",
+    "\n",
+    "\n",
+    "1. Go to https://smith.langchain.com/ and sign up.\n",
+    "2. After signing up, you will need to verify your email.\n",
+    "3. Click the left cog (Setting) - centre \"Personal\" - \"Create API Key\" to get an API key.\n",
+    "\n",
+    "![get-api-key](./assets/03-langsmith-tracking-setup-04.png)\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In Description, enter a description that makes sense to you and click the Create API Key button to create it.\n",
+    "\n",
+    "![create-api-key](./assets/03-langsmith-tracking-setup-05.png\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Copy the generated key and proceed to the next step.\n",
+    "\n",
+    "(Caution!) Copy the generated key somewhere safe so that it doesn't leak.\n",
+    "\n",
+    "![copy-api-key](./assets/03-langsmith-tracking-setup-06.png)\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Setting the LangSmith key in .env\n",
+    "\n",
+    "\n",
+    "First, enter the key you received from LangSmith and your project information in the .env file.\n",
+    "\n",
+    "- LANGCHAIN_TRACING_V2: Set to \"true\" to start tracking.\n",
+    "- LANGCHAIN_ENDPOINT: https://api.smith.langchain.com Do not change.\n",
+    "- LANGCHAIN_API_KEY: Enter the key issued in the previous step.\n",
+    "- LANGCHAIN_PROJECT: If you enter a project name, all runs will be traced to that project group.\n",
+    "\n",
+    "![setting-api-key](./assets/03-langsmith-tracking-setup-07.png)\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Enable tracking in your Jupyter notebook or code\n",
+    "\n",
+    "Enabling tracking is very simple. All you need to do is set an environment variable.\n",
+    "\n",
+    "Copy the contents of .env_sample and load it into your .env with the key you set for Bonnie."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "%pip install python-dotenv"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "True"
+      ]
+     },
+     "execution_count": 1,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from dotenv import load_dotenv\n",
+    "\n",
+    "load_dotenv(override=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As long as your traces are enabled and your API key and project name are set correctly, this should be sufficient.\n",
+    "\n",
+    "However, if you want to change the project name or change the tracking, you can do so with the code below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
+    "os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.smith.langchain.com\"\n",
+    "os.environ[\"LANGCHAIN_PROJECT\"] = \"<LangChain Peoject Name>\"\n",
+    "os.environ[\"LANGCHAIN_API_KEY\"] = \"<LangChain API KEY>\""
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "langchain-opentutorial-cyV7o7ra-py3.11",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/01-BASIC/assets/03-langsmith-tracking-setup-01.png b/01-BASIC/assets/03-langsmith-tracking-setup-01.png
diff --git a/01-BASIC/assets/03-langsmith-tracking-setup-02.png b/01-BASIC/assets/03-langsmith-tracking-setup-02.png
diff --git a/01-BASIC/assets/03-langsmith-tracking-setup-03.png b/01-BASIC/assets/03-langsmith-tracking-setup-03.png
diff --git a/01-BASIC/assets/03-langsmith-tracking-setup-04.png b/01-BASIC/assets/03-langsmith-tracking-setup-04.png
diff --git a/01-BASIC/assets/03-langsmith-tracking-setup-05.png b/01-BASIC/assets/03-langsmith-tracking-setup-05.png
diff --git a/01-BASIC/assets/03-langsmith-tracking-setup-06.png b/01-BASIC/assets/03-langsmith-tracking-setup-06.png
diff --git a/01-BASIC/assets/03-langsmith-tracking-setup-07.png b/01-BASIC/assets/03-langsmith-tracking-setup-07.png
diff --git a/01-BASIC/assets/03-langsmith-tracking-setup-08.png b/01-BASIC/assets/03-langsmith-tracking-setup-08.png