Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[E-4] 13-LangChain-Expression-Language / 11-Fallbacks #496

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 22, 2025
Merged

[E-4] 13-LangChain-Expression-Language / 11-Fallbacks #496

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 34 additions & 34 deletions 13-LangChain-Expression-Language/11-Fallbacks.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"\n",
"## Overview\n",
"\n",
"This tutorial covers how to implement fallback mechanisms in LangChain applications to handle various types of failures and errors gracefully.\n",
"This tutorial covers how to implement fallback mechanisms in LangChain applications to gracefully handle various types of failures and errors.\n",
"\n",
"`Fallbacks` are crucial for building robust LLM applications that can handle API errors, rate limits, and other potential failures without disrupting the user experience.\n",
"\n",
Expand All @@ -26,15 +26,15 @@
"\n",
"- [Overview](#overview)\n",
"- [Environment Setup](#environment-setup)\n",
"- [What is Fallbacks?](#what-is-fallbacks)\n",
"- [How to Handle LLM API Errors](#how-to-handle-llm-api-errors)\n",
"- [What are Fallbacks?](#what-are-fallbacks)\n",
"- [Handling LLM API Errors](#handling-llm-api-errors)\n",
"- [Introduction to Rate Limit Testing](#introduction-to-rate-limit-testing)\n",
"- [Why Handle Rate Limit Errors?](#why-handle-rate-limit-errors)\n",
"- [Benefits of Mock Testing](#benefits-of-mock-testing)\n",
"- [Setting up LLM Fallback Configuration](#setting-up-llm-fallback-configuration)\n",
"- [Testing API Rate Limits with Fallback Models](#testing-api-rate-limits-with-fallback-models)\n",
"- [If you specify an error that needs to be handled](#if-you-specify-an-error-that-needs-to-be-handled)\n",
"- [Specifying multiple models in fallback sequentially](#specifying-multiple-models-in-fallback-sequentially)\n",
"- [Specifying Exceptions to Trigger Fallbacks](#specifying-exceptions-to-trigger-fallbacks)\n",
"- [Specifying Multiple Fallback Models Sequentially](#specifying-multiple-fallback-models-sequentially)\n",
"- [Using Different Prompt Templates for Each Model](#using-different-prompt-templates-for-each-model)\n",
"- [Automatic Model Switching Based on Context Length](#automatic-model-switching-based-on-context-length)\n",
"\n",
Expand All @@ -48,14 +48,14 @@
" - Implementation of simple fallback chains\n",
"\n",
"2. **API Error Management**\n",
" - Handling rate limit errors effectively\n",
" - Effectively handling rate limit errors\n",
" - Managing API downtime scenarios\n",
" - Implementing retry strategies\n",
" - Simulating errors through mock testing\n",
"\n",
"3. **Advanced Fallback Patterns**\n",
" - Configuring multiple fallback models\n",
" - Custom exception handling setup\n",
" - Setting up custom exception handling\n",
" - Sequential fallback execution\n",
" - Context-aware model switching\n",
" - Model-specific prompt templating\n",
Expand Down Expand Up @@ -200,25 +200,25 @@
"id": "43c96ba4",
"metadata": {},
"source": [
"## What is Fallbacks?\n",
"## What are Fallbacks?\n",
"\n",
"In LLM applications, there are various errors or failures such as LLM API issues, degradation in model output quality, and other integration-related issues. The `fallback` feature can be utilized to gracefully handle and isolate these problems.\n",
"In LLM applications, various errors or failures can occur, such as LLM API issues, degradation in model output quality, and other integration-related problems. The `fallback` feature gracefully handle and isolate these issues.\n",
"\n",
"Importantly, fallbacks can be applied not only at the LLM level but also at the entire executable level."
"Note that they can be applied at both the LLM calls and the level of an entire executable chain."
]
},
{
"cell_type": "markdown",
"id": "d66ad995",
"metadata": {},
"source": [
"## How to Handle LLM API Errors\n",
"## Handling LLM API Errors\n",
"\n",
"Handling LLM API errors is one of the most common use cases for using `fallbacks`.\n",
"Handling LLM API errors is one of the most common use cases for `fallbacks`.\n",
"\n",
"Requests to the LLM API can fail for various reasons. The API might be down, you might have reached a rate limit, or there could be several other issues. Using `fallbacks` can help protect against these types of problems.\n",
"API requests can fail due to various reasons. The API might be down, you might have reached usage rate limits, or other issues. By implementing `fallbacks`, you can protect your application against these types of problems.\n",
"\n",
"**Important**: By default, many LLM wrappers capture errors and retry. When using `fallbacks`, it is advisable to disable this default behavior. Otherwise, the first wrapper will keep retrying and not fail."
"**Important**: By default, many LLM wrappers capture errors and retry. When using `fallbacks`, it is advisable to disable this default behavior; otherwise, the first wrapper will keep retrying and prevent the fallback from triggering."
]
},
{
Expand All @@ -228,15 +228,15 @@
"source": [
"## Introduction to Rate Limit Testing\n",
"\n",
"First, let's perform a mock test for the `RateLimitError` that can occur with OpenAI. A `RateLimitError` is **an error that occurs when you exceed the API usage limits** of the OpenAI API.\n",
"First, let's perform a mock test for the `RateLimitError` that can occur with OpenAI. A `RateLimitError` is **an error that occurs when you exceed the OpenAI API usage limits** of the OpenAI API.\n",
"\n",
"## Why Handle Rate Limit Errors?\n",
"\n",
"When this error occurs, API requests are restricted for a certain period, so applications need to handle this situation appropriately. Through mock testing, we can verify how the application behaves when a `RateLimitError` occurs and check the error handling logic.\n",
"`RateLimitError` restricts API requests for a certain period, so applications need to handle them appropriately. Mock testing verifies application behaves and error-handling logic during `RateLimitError`s.\n",
"\n",
"## Benefits of Mock Testing\n",
"\n",
"This allows us to prevent potential issues that could arise in production environments and ensure stable service delivery."
"Mock testing helps prevent potential production issues and ensures stable service delivery."
]
},
{
Expand Down Expand Up @@ -266,9 +266,9 @@
"source": [
"## Setting up LLM Fallback Configuration\n",
"\n",
"Create a `ChatOpenAI` object and assign it to the `openai_llm` variable, setting the `max_retries` parameter to 0 to **prevent retry attempts** that might occur due to API call limits or restrictions.\n",
"Create a `ChatOpenAI` object and assign it to `openai_llm`, setting `max_retries=0` to **prevent retry attempts** that might occur due to API call limits or restrictions.\n",
"\n",
"Using the `with_fallbacks` method, configure `anthropic_llm` as the `fallback` LLM and assign this configuration to the `llm` variable.\n"
"Use `with_fallbacks` to configure `anthropic_llm` as the fallback LLM and assign this configuration to `llm`.\n"
]
},
{
Expand Down Expand Up @@ -299,13 +299,13 @@
"source": [
"## Testing API Rate Limits with Fallback Models\n",
"\n",
"In this example, we'll simulate OpenAI API rate limits and test how the system behaves when encountering API cost limitation errors.\n",
"In this example, we'll simulate OpenAI API rate limits and test system behavior during API cost limitation errors.\n",
"\n",
"You'll see that when the OpenAI GPT model encounters an error, the fallback model (Anthropic) successfully takes over and performs the inference instead.\n",
"When the OpenAI GPT model encounters an error, the Anthropic fallback model successfully takes over and performs the inference instead.\n",
"\n",
"When a fallback model is configured using `with_fallbacks()` and successfully executes, the `RateLimitError` won't be raised, ensuring continuous operation of your application.\n",
"When a fallback model, configured with `with_fallbacks()`, executes successfully, the `RateLimitError` is not raised, ensuring continuous operation of your application.\n",
"\n",
"> 💡 This demonstrates how LangChain's fallback mechanism provides resilience against API limitations and ensures your application continues to function even when the primary model is unavailable."
">💡 This demonstrates LangChain's fallback mechanism, which provides resilience against API limitations and ensures continued application function even when the primary model is unavailable."
]
},
{
Expand Down Expand Up @@ -360,9 +360,9 @@
"id": "866996f2",
"metadata": {},
"source": [
"A model set to `llm.with_fallbacks()` will also behave the same as a regular runnable model.\n",
"A model configured with `llm.with_fallbacks()` behaves like a regular Runnable model.\n",
"\n",
"The code below also doesn't throw an ‘error’ because the fallbacks model did a good job."
"The code below also does not throw an **error** because the fallback model performed successfully."
]
},
{
Expand Down Expand Up @@ -407,13 +407,13 @@
"id": "58a91a9b",
"metadata": {},
"source": [
"## If you specify an error that needs to be handled\n",
"## Specifying Exceptions to Trigger Fallbacks\n",
"\n",
"When working with fallbacks, you can precisely define when the `fallback` should be triggered. This allows for more granular control over the fallback mechanism's behavior.\n",
"You can precisely define when a `fallback` should trigger, allowing for more granular control over the fallback mechanism's behavior.\n",
"\n",
"For example, you can specify certain exception classes or error codes that will trigger the fallback logic. This approach helps you to **reduce unnecessary fallback calls and improve error handling efficiency.**\n",
"For example, you can specify certain exception classes or error codes to trigger the fallback logic, **reducing unnecessary calls and improving efficiency in error handling.**\n",
"\n",
"In the example below, you'll see an \"error occurred\" message printed. This happens because we've configured the `exceptions_to_handle` parameter to only trigger the fallback when a `KeyboardInterrupt` exception occurs. As a result, the `fallback` won't be triggered for any other exceptions.\n"
"The example below prints an \\\"error\\\" message because `exceptions_to_handle` is configured to trigger the fallback only for `KeyboardInterrupt`. The `fallback` will not trigger for other exceptions.\n"
]
},
{
Expand Down Expand Up @@ -452,9 +452,9 @@
"id": "f2eafd75",
"metadata": {},
"source": [
"## Specifying multiple models in fallback sequentially\n",
"## Specifying Multiple Fallback Models Sequentially\n",
"\n",
"You can specify multiple models in the `fallback` model, not just one. When multiple models are specified, they will be tried sequentially.\n"
"You can specify multiple fallback models, not just one. They will be tried sequentially if multiple models are specified."
]
},
{
Expand All @@ -479,7 +479,7 @@
"id": "6185c984",
"metadata": {},
"source": [
"Create two chains, one that causes an error and one that works normally.\n"
"Create two chains: one that causes an error and one that works normally.\n"
]
},
{
Expand Down Expand Up @@ -537,7 +537,7 @@
"metadata": {},
"source": [
"## Using Different Prompt Templates for Each Model\n",
"You can use different prompt templates tailored to each model's characteristics. For example, GPT-4 can handle complex instructions while GPT-3.5 can work with simpler ones."
"You can use different prompt templates tailored to each model's characteristics. For example, GPT-4 handles complex instructions, while GPT-3.5 works with simpler ones."
]
},
{
Expand Down Expand Up @@ -647,7 +647,7 @@
"metadata": {},
"source": [
"## Automatic Model Switching Based on Context Length\n",
"When handling long contexts, you can automatically switch to models with larger context windows if token limits are exceeded."
"For long contexts, automatically switch to models with larger context windows if token limits are exceeded."
]
},
{
Expand Down
Loading