LangChain-OpenTutorial · teddylee777 · Feb 1, 2025 · Jan 22, 2025 · Jan 25, 2025
diff --git a/10-Retriever/10-Kiwi-BM25-Retriever.ipynb b/10-Retriever/10-Kiwi-BM25-Retriever.ipynb
@@ -5,7 +5,6 @@
    "metadata": {},
    "source": [
     "# Kiwi BM25 Retriever\n",
-    "\n",
     "- Author: [JeongGi Park](https://github.com/jeongkpa)\n",
     "- Design: []()\n",
     "- Peer Review: \n",
@@ -14,27 +13,24 @@
     "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/LangChain-OpenTutorial/LangChain-OpenTutorial/blob/main/01-Basic/07-LCEL-Interface.ipynb) [![Open in GitHub](https://img.shields.io/badge/Open%20in%20GitHub-181717?style=flat-square&logo=github&logoColor=white)](https://github.com/LangChain-OpenTutorial/LangChain-OpenTutorial/blob/main/01-Basic/07-LCEL-Interface.ipynb)\n",
     "\n",
     "## Overview\n",
-    "This document explores the use of `kiwipiepy` for Korean morphological analysis and demonstrates its integration within the `LangChain` framework. \n",
-    "It highlights methods to tokenize text, compare retrieval models like `BM25` and `FAISS`, and analyze relationships between queries and documents using metrics such as cosine similarity. \n",
-    "Additionally, it emphasizes the role of these techniques in enhancing workflows like text analysis and information retrieval.\n",
-    "\n",
-    "Since this tutorial covers Korean morphological analysis, the output primarily contains Korean text, reflecting the language structure being analyzed\n",
-    "For international users, we provide English translations alongside Korean examples in this tutorial.\n",
+    "This tutorial explores the use of `kiwipiepy` for Korean morphological analysis and demonstrates its integration within the `LangChain` framework. \n",
+    "It highlights Korean text tokenization, and the comparison of different retrievers with various setups.\n",
     "\n",
+    "Since this tutorial covers Korean morphological analysis, the output primarily contains Korean text, reflecting the language structure being analyzed.\n",
+    "For international users, we provide English translations alongside Korean examples.\n",
     "\n",
     "### Table of Contents\n",
-    "\n",
     "- [Overview](#overview)\n",
     "- [Environment Setup](#environment-setup)\n",
-    "- [Korean Word Retriever Tuning](#Korean-Word-Retriever-Tuning)\n",
-    "- [Testing with Various Sentences](#Testing-with-Various-Sentences)\n",
-    "- [Experiment: Compare Search Results Using Different Retrievers](#Experiment-Compare-Search-Results-Using-Different-Retrievers)\n",
+    "- [Korean Tokenization](#korean-tokenization)\n",
+    "- [Testing with Various Sentences](#testing-with-various-sentences)\n",
+    "- [Comparing Search Results Using Different Retrievers](#comparing-search-results-using-different-retrievers)\n",
     "- [Conclusion](#conclusion)\n",
     "\n",
     "### References\n",
     "- [kiwipiepy](https://github.com/bab2min/kiwipiepy)\n",
-    "- [fiass](https://python.langchain.com/docs/integrations/vectorstores/faiss/)\n",
-    "- [openai-embeddings](https://python.langchain.com/docs/integrations/text_embedding/openai/)\n",
+    "- [FAISS](https://python.langchain.com/docs/integrations/vectorstores/faiss/)\n",
+    "- [OpenAIEmbeddings](https://python.langchain.com/docs/integrations/text_embedding/openai/)\n",
     "\n",
     "---"
    ]
@@ -150,26 +146,16 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Why Korean Tokenization?\n",
+    "## Korean Tokenization\n",
     "\n",
-    "- In Korean, words are morphologically rich. For instance, “안녕하세요” is tokenized into:\n",
+    "Korean words are morphologically rich. A single word is often split into multiple morphemes (root, affix, suffix, etc.).\n",
+    "\n",
+    "For instance, “안녕하세요” is tokenized into:\n",
     "  - Token(form='안녕', tag='NNG')\n",
     "  - Token(form='하', tag='XSA')\n",
     "  - Token(form='세요', tag='EF')\n",
     "\n",
-    "- Compared to English tokenization at the word level (e.g., “Hello” remains one word), Korean often splits into multiple morphemes (어근, 접사, 어미 등).\n",
-    "- Kiwi provides detailed POS tagging such as NNG(일반 명사), XSA(형용사 파생 접사), EF(종결 어미) to reflect these language-specific nuances.\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Korean Word Retriever Tuning\n",
-    "\n",
-    "Install the Korean morphological analyzer library, `kiwipiepy`.\n",
-    "\n",
-    "[Project Link for kiwipiepy](https://github.com/bab2min/kiwipiepy)"
+    "We utilize `kiwipiepy`, which is a Python module for **Kiwi**, an open-source Korean morphological analyzer, to tokenize Korean text."
    ]
   },
   {
@@ -187,7 +173,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Perform Tokenization"
+    "With this, we can easily perform tokenization."
    ]
   },
   {
@@ -216,7 +202,7 @@
    ],
    "source": [
     "kiwi.tokenize(\"안녕하세요? 형태소 분석기 키위입니다\")\n",
-    "# \"안녕하세요? 형태소 분석기 키위입니다.\" it means \"Hi, this is Kiwi, the morphological analyser.\""
+    "# Translation: Hi, this is Kiwi, a morphological analyzer."
    ]
   },
   {
@@ -225,10 +211,7 @@
    "source": [
     "## Testing with Various Sentences\n",
     "\n",
-    "BM25 is a traditional ranking function based on term frequency and inverse document frequency. It works well when exact keyword matches are important.\n",
-    "\n",
-    "\n",
-    "FAISS uses vector embeddings to capture semantic similarity. By combining BM25 with FAISS in an ensemble, we can leverage the lexical match benefits from BM25 and the semantic understanding from FAISS."
+    "To test different retrieval methods, we define a list of documents composed of similar yet distinguishable contents."
    ]
   },
   {
@@ -247,47 +230,33 @@
     "docs = [\n",
     "    Document(\n",
     "        page_content=\"금융보험은 장기적인 자산 관리와 위험 대비를 목적으로 고안된 금융 상품입니다.\"\n",
+    "        # Translation: Financial insurance is a financial product designed for long term asset management and risk coverage.\n",
     "    ),\n",
     "    Document(\n",
     "        page_content=\"금융저축보험은 규칙적인 저축을 통해 목돈을 마련할 수 있으며, 생명보험 기능도 겸비하고 있습니다.\"\n",
+    "        # Translation: Financial savings insurance allows individuals to accumulate a lump sum through regular savings, and also offers life insurance benefits.\n",
     "    ),\n",
     "    Document(\n",
     "        page_content=\"저축금융보험은 저축과 금융을 통해 목돈 마련에 도움을 주는 보험입니다. 또한, 사망 보장 기능도 제공합니다.\"\n",
+    "        # Translation: Savings financial insurance helps individuals gather a lump sum through savings and finance, and also provides death benefit coverage.\n",
     "    ),\n",
     "    Document(\n",
     "        page_content=\"금융저축산물보험은 장기적인 저축 목적과 더불어, 축산물 제공 기능을 갖추고 있는 특별 금융 상품입니다.\"\n",
+    "        # Translation: Financial savings livestock insurance is a special financial product designed for long term savings, which also includes provisions for livestock products.\n",
     "    ),\n",
     "    Document(\n",
     "        page_content=\"금융단폭격보험은 저축은 커녕 위험 대비에 초점을 맞춘 상품입니다. 높은 위험을 감수하고자 하는 고객에게 적합합니다.\"\n",
+    "        # Translation: Financial 'carpet bombing' insurance focuses on risk coverage rather than savings. It is suitable for customers willing to take on high risk.\n",
     "    ),\n",
     "    Document(\n",
     "        page_content=\"금보험은 저축성과를 극대화합니다. 특히 노후 대비 저축에 유리하게 구성되어 있습니다.\"\n",
+    "        # Translation: Gold insurance maximizes returns on savings. It is especially advantageous for retirement savings.\n",
     "    ),\n",
     "    Document(\n",
     "        page_content=\"금융보씨 험한말 좀 하지마시고, 저축이나 좀 하시던가요. 뭐가 그리 급하신지 모르겠네요.\"\n",
+    "        # Translation: Hey, Mr. 'Financial Bo,' please refrain from harsh words and consider saving money. I'm not sure why you're in such a hurry.\n",
     "    ),\n",
-    "]\n",
-    "\n",
-    "# 금융보험은 장기적인 자산 관리와 위험 대비를 목적으로 고안된 금융 상품입니다.\n",
-    "# Financial insurance is a financial product designed for long term asset management and risk coverage.\n",
-    "\n",
-    "# 금융저축보험은 규칙적인 저축을 통해 목돈을 마련할 수 있으며, 생명보험 기능도 겸비하고 있습니다.\n",
-    "# Financial savings insurance allows individuals to accumulate a lump sum through regular savings, and also offers life insurance benefits.\n",
-    "\n",
-    "# 저축금융보험은 저축과 금융을 통해 목돈 마련에 도움을 주는 보험입니다. 또한, 사망 보장 기능도 제공합니다.\n",
-    "# Savings financial insurance helps individuals gather a lump sum through savings and finance, and also provides death benefit coverage.\n",
-    "\n",
-    "# 금융저축산물보험은 장기적인 저축 목적과 더불어, 축산물 제공 기능을 갖추고 있는 특별 금융 상품입니다.\n",
-    "# Financial savings livestock insurance is a special financial product designed for long term savings, which also includes provisions for livestock products.\n",
-    "\n",
-    "# 금융단폭격보험은 저축은 커녕 위험 대비에 초점을 맞춘 상품입니다. 높은 위험을 감수하고자 하는 고객에게 적합합니다.\n",
-    "# Financial 'carpet bombing' insurance focuses on risk coverage rather than savings. It is suitable for customers willing to take on high risk.\n",
-    "\n",
-    "# 금보험은 저축성과를 극대화합니다. 특히 노후 대비 저축에 유리하게 구성되어 있습니다.\n",
-    "# Gold insurance maximizes returns on savings. It is especially advantageous for retirement savings.\n",
-    "\n",
-    "# 금융보씨 험한말 좀 하지마시고, 저축이나 좀 하시던가요. 뭐가 그리 급하신지 모르겠네요.\n",
-    "# Hey, Mr. 'Financial Bo,' please refrain from harsh words and consider saving money. I'm not sure why you're in such a hurry.\n"
+    "]"
    ]
   },
   {
@@ -321,42 +290,42 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Create a tokenization function\n",
-    "\n",
+    "# Define a tokenization function\n",
     "def kiwi_tokenize(text):\n",
-    "    return [token.form for token in kiwi.tokenize(text)]\n"
+    "    return [token.form for token in kiwi.tokenize(text)]"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Experiment: Compare Search Results Using Different Retrievers\n",
+    "## Comparing Search Results Using Different Retrievers\n",
+    "\n",
     "\n",
     "In this section, we compare how different retrieval methods rank documents when given the same query. We are using:\n",
     "\n",
-    "* `BM25`: A traditional ranking function based on term frequency (TF) and inverse document frequency (IDF).\n",
-    "* `Kiwi BM25`: `BM25` with an added benefit of kiwipiepy tokenization, enabling more accurate splitting of Korean words into morphemes (especially important for Korean queries).\n",
-    "* `FAISS`: A vector-based retriever using embeddings (in this case, `OpenAIEmbeddings`). It captures semantic similarity, so it’s less reliant on exact keyword matches and more on meaning.\n",
-    "* `Ensemble`: A combination of BM25 (or `Kiwi BM25`) and `FAISS`, weighted to leverage both the lexical matching strengths of `BM25` and the semantic understanding of FAISS.\n",
+    "- **BM25**: A traditional ranking function based on term frequency (TF) and inverse document frequency (IDF).\n",
+    "- **Kiwi BM25**: BM25 with an added benefit of kiwipiepy tokenization, enabling more accurate splitting of Korean words into morphemes (especially important for Korean queries).\n",
+    "- **FAISS**: A vector-based retriever using embeddings (in this case, `OpenAIEmbeddings`). It captures semantic similarity, so it’s less reliant on exact keyword matches and more on meaning.\n",
+    "- **Ensemble**: A combination of BM25 (or Kiwi BM25) and FAISS, weighted to leverage both the lexical matching strengths of BM25 and the semantic understanding of FAISS.\n",
     "\n",
     "### Key points of Comparison\n",
     "\n",
     "**Exact Keyword Matching vs. Semantic Matching**\n",
     "\n",
-    "* `BM25` (and `Kiwi BM25`) excel in finding documents that share exact terms or closely related morphological variants.\n",
-    "* `FAISS` retrieves documents that may not have exact lexical overlap but are semantically similar (e.g., synonyms or paraphrases).\n",
+    "- **BM25** (and **Kiwi BM25**) excel in finding documents that share exact terms or closely related morphological variants.\n",
+    "- **FAISS** retrieves documents that may not have exact lexical overlap but are semantically similar (e.g., synonyms or paraphrases).\n",
     "\n",
-    "**Impact of Korean Morphological Analysis**\n",
+    "**Impact of Korean morphological analysis**\n",
     "\n",
-    "* Korean often merges stems and endings into single words (“안녕하세요” → “안녕 + 하 + 세요”). `Kiwi BM25` handles this by splitting the query and documents more precisely.\n",
-    "* This can yield more relevant results when dealing with conjugated verbs, particles, or compound nouns.\n",
+    "- Korean often merges stems and endings into single words (“안녕하세요” → “안녕 + 하 + 세요”). **Kiwi BM25** handles this by splitting the query and documents more precisely.\n",
+    "- This can yield more relevant results when dealing with conjugated verbs, particles, or compound nouns.\n",
     "\n",
     "**Ensemble Approaches**\n",
     "\n",
-    "* By combining lexical (`BM25`) and semantic (`FAISS`) retrievers, we can produce a more balanced set of results.\n",
-    "* The weighting (e.g., 70:30 or 30:70) can be tuned to emphasize one aspect over the other.\n",
-    "* Using MMR (Maximal Marginal Relevance) ensures diversity in the retrieved results, reducing redundancy."
+    "- By combining lexical (BM25) and semantic (FAISS) retrievers, we can produce a more balanced set of results.\n",
+    "- The weighting (e.g., 70:30 or 30:70) can be tuned to emphasize one aspect over the other.\n",
+    "- Using MMR (Maximal Marginal Relevance) ensures diversity in the retrieved results, reducing redundancy."
    ]
   },
   {
@@ -424,7 +393,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Function to print search results from multiple retrievers\n",
+    "# Define a function to print search results from multiple retrievers\n",
     "def print_search_results(retrievers, query):\n",
     "    \"\"\"\n",
     "    Prints the top search result from each retriever for a given query.\n",
@@ -444,41 +413,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Display Search Results\n",
-    "\n",
-    "금융보험은 장기적인 자산 관리와 위험 대비를 목적으로 고안된 금융 상품입니다.\n",
-    "\n",
-    "-> Financial insurance is a financial product designed for long term asset management and risk coverage\n",
-    "\n",
-    "\n",
-    "금융저축보험은 규칙적인 저축을 통해 목돈을 마련할 수 있으며, 생명보험 기능도 겸비하고 있습니다.\n",
-    "\n",
-    "-> Financial savings insurance allows individuals to accumulate a lump sum through regular savings, and also offers life insurance benefits\n",
-    "\n",
-    "\n",
-    "저축금융보험은 저축과 금융을 통해 목돈 마련에 도움을 주는 보험입니다. 또한, 사망 보장 기능도 제공합니다.\n",
-    "\n",
-    "-> Savings financial insurance helps individuals gather a lump sum through savings and finance, and also provides death benefit coverage\n",
-    "\n",
-    "\n",
-    "금융저축산물보험은 장기적인 저축 목적과 더불어, 축산물 제공 기능을 갖추고 있는 특별 금융 상품입니다.\n",
-    "\n",
-    "-> Financial savings livestock insurance is a special financial product designed for long term savings, which also includes provisions for livestock products\n",
-    "\n",
-    "\n",
-    "금융단폭격보험은 저축은 커녕 위험 대비에 초점을 맞춘 상품입니다. 높은 위험을 감수하고자 하는 고객에게 적합합니다.\n",
-    "\n",
-    "-> Financial 'carpet bombing' insurance focuses on risk coverage rather than savings. It is suitable for customers willing to take on high risk\n",
-    "\n",
-    "\n",
-    "금보험은 저축성과를 극대화합니다. 특히 노후 대비 저축에 유리하게 구성되어 있습니다.\n",
-    "\n",
-    "-> Gold insurance maximizes returns on savings. It is especially advantageous for retirement savings\n",
-    "\n",
-    "\n",
-    "금융보씨 험한말 좀 하지마시고, 저축이나 좀 하시던가요. 뭐가 그리 급하신지 모르겠네요.\n",
-    "\n",
-    "-> Hey, Mr. 'Financial Bo,' please refrain from harsh words and consider saving money. I'm not sure why you're in such a hurry.\n"
+    "### Displaying Search Results\n",
+    "Let's display the search results for a variety of queries, and see how different retrievers perform."
    ]
   },
   {
@@ -637,13 +573,11 @@
    "source": [
     "## Conclusion\n",
     "\n",
-    "By running the code and observing the top documents returned for each query, you’ll see how each retriever type has its strengths:\n",
-    "\n",
-    "`BM25` / `Kiwi BM25`: Great for precise keyword matching, beneficial for Korean morphological nuances.\n",
-    "\n",
-    "`FAISS`: Finds semantically related documents even if the wording differs.\n",
+    "By running the code and observing the top documents returned for each query, you can see how each retriever type has its strengths:\n",
     "\n",
-    "`Ensemble`: Balances both worlds, often achieving better overall coverage for a wide range of queries.\n"
+    "- `BM25` / `Kiwi BM25`: Great for precise keyword matching, beneficial for Korean morphological nuances.\n",
+    "- `FAISS`: Finds semantically related documents even if the wording differs.\n",
+    "- `Ensemble`: Balances both worlds, often achieving better overall coverage for a wide range of queries.\n"
    ]
   }
  ],