Hi @PLNech

I am developing my own API using FastAPI and ran into the same "problem" as I am trying to add a global timeout to all my requests.

I am still new to fastapi but from what I understand I believe the "fastapi" way to do so would be to use a middleware as they are designed to be ran at every request by nature. As I searched on how to do so I found this
gitter community thread and thought it could maybe help you.

I am going to implement both your solution and the middleware based one and see which one I prefer and works best. Also note that there seems to be a problem with starlette 0.13.3 and higher so keep that in mind.

Also if you found a workaround by now I am more than interested.

Hope it helped you a bit

Hi @ZionStage, thanks for your message! I haven't found a workaround for now. Looking forward to continuing this conversation with you as we move forward on this topic :)

@PLNech have you tried changing the timeout settings for gunicorn? By default it times out after 60 sec I believe but you can overwrite the settings.

https://docs.gunicorn.org/en/latest/settings.html#timeout
#551

@ZionStage: thanks for sharing your implementation, this looks promising! I'll make some room in our backlog to give it a try in our next sprint and will let you know how it goes :)

@thomas-maschler: thanks for the advice. Unfortunately I've tried using Gunicorn's timeout, but it triggers a full restart of the app, disrupting other users of the service (e.g. by unloading their models from memory). What I'm trying to achieve is rather to enforce a timeout on individual requests, without affecting any other work handled by this worker.

Thanks for the discussion here everyone!

Yes, indeed I think the solution would be with a middleware.

About the failing tests from @ZionStage, I understand there are no guarantees about sub-second precisions in async/await (I think Python in general). Either way, it would probably be impossible to expect absolute sub-second precision from something on the network. I would test only with integers to be sure.

But anyway, I think that's pretty much the right approach. ✔️

Assuming the original need was handled, this will be automatically closed now. But feel free to add more comments or create new issues or PRs.

This is good to return an error message to the user in case of timeout, but is there a way to actually kill the request at the same time so it doesn't keep using resources?

Bumping this for @MasterScrat's question. Wondering the same thing

Another bump for @MasterScrat's question

@lionel-ovaert When raising the error once the time limit has been reached should stop any undergoing processes linked to the request, doesn't it ?

Even if the router function contains async code it doesn't get interrupted/cancelled with this middleware solution.
The following example keeps printing Running... endlessly even though TimeoutException is triggered and underlying Task created by asyncio.wait_for(...) gets cancelled.

app = FastAPI()

@app.get("/long_running")
async def long_running():
    try:
        while True:
            print("Running...")
            await asyncio.sleep(1)
    except asyncio.CancelledError:  # This never happens :(
        print("Cancelled.")

@app.middleware("http")
async def timeout_middleware(request: Request, call_next):
    try:
        return await asyncio.wait_for(call_next(request), timeout=3)
    except asyncio.TimeoutError:
        return JSONResponse({'detail': 'Request processing time exceeded limit'}, 504)

@tiangolo shouldn't we hit except asyncio.CancelledError in this case? 🙏

@LMalikov I got the same error. It looks like you need at least two middleware dectorators in the main.py but it's super wierd. For example,

@app.middleware("http")
async def add_process_time_header(request: Request, call_next):
    start_time = time.time()
    response = await call_next(request)
    process_time = time.time() - start_time
    # response.headers["X-Process-Time"] = str(process_time)
    print("adfadsfasdf")
    # response.headers["Test middleware"] = str(random.randint(1, 1000))
    return response


REQUEST_TIMEOUT_ERROR = 1.0 # seconds to wait for
from fastapi.responses import JSONResponse
from starlette.status import HTTP_504_GATEWAY_TIMEOUT

#Adding a middleware returning a 504 error if the request processing time is above a certain threshold
@app.middleware("http")
async def timeout_middleware(request: Request, call_next):
    try:
        start_time = time.time()
        return await asyncio.wait_for(call_next(request), timeout=REQUEST_TIMEOUT_ERROR)

    except asyncio.TimeoutError:
        process_time = time.time() - start_time
        res = timeout_fallback(process_time)
        return res

def timeout_fallback(process_time):
    response = JSONResponse({'detail': 'Request processing time excedeed limit',
                             'processing_time': process_time},
                            status_code=HTTP_504_GATEWAY_TIMEOUT)
    return response

Does anyone

@LMalikov I got the same error. It looks like you need at least two middleware dectorators in the main.py but it's super wierd. For example,

@app.middleware("http")
async def add_process_time_header(request: Request, call_next):
    start_time = time.time()
    response = await call_next(request)
    process_time = time.time() - start_time
    # response.headers["X-Process-Time"] = str(process_time)
    print("adfadsfasdf")
    # response.headers["Test middleware"] = str(random.randint(1, 1000))
    return response


REQUEST_TIMEOUT_ERROR = 1.0 # seconds to wait for
from fastapi.responses import JSONResponse
from starlette.status import HTTP_504_GATEWAY_TIMEOUT

#Adding a middleware returning a 504 error if the request processing time is above a certain threshold
@app.middleware("http")
async def timeout_middleware(request: Request, call_next):
    try:
        start_time = time.time()
        return await asyncio.wait_for(call_next(request), timeout=REQUEST_TIMEOUT_ERROR)

    except asyncio.TimeoutError:
        process_time = time.time() - start_time
        res = timeout_fallback(process_time)
        return res

def timeout_fallback(process_time):
    response = JSONResponse({'detail': 'Request processing time excedeed limit',
                             'processing_time': process_time},
                            status_code=HTTP_504_GATEWAY_TIMEOUT)
    return response

Does anyone know why? It's super weird. Basically, you need to have two @app.middleware("http"). Otherwise, the timeout exception won't work.

Basically, my problem is like this https://stackoverflow.com/questions/74132015/asyncio-wait-for-doesnt-time-out-as-expected

Same problem as whats noted in the stackoverflow link above. The aysncio timeout is not respected.

I think this can be fixed using python 3.11 and asyncio.timeout_at instead asyncio.wait_for.
In the meantime (I'm using python 3.9) I'll try something else. I'll tell you if it works.

Thanks @Naish21 I really like your solution!
Note that it only works with seconds as integer (so no timeout below 1s).
To use this solution with a floating number of second, one can replace

signal.alarm(max_execution_time)

with

signal.setitimer(signal.ITIMER_REAL, max_execution_time)

setitimer can work with floating number, so it is possible to define a timeout of 300ms for example (@abort_after(0.3))

I am afraid this solution will not play well in a concurrent environment since there is only one timer per process, whereas there will be many co-routines running concurrently within the same process.

So far, I have not seen any satisfactory solution for this problem. And the underlying problem seems to be that we might use functions on the router that are not friendly with asyncio.

Hello everyone, I created this solution to stop running process if timeout expires, can you guys please provide suggestions if this is the right solution or not

import uvicorn
import asyncio
from fastapi import FastAPI, Request, HTTPException

from fastapi import FastAPI, APIRouter, Response, Request, HTTPException
from fastapi.routing import APIRoute
from typing import Callable

REQUEST_TIMEOUT = 5
app = FastAPI()

class CustomAPIRoute(APIRoute):
    def get_route_handler(self) -> Callable:
        original_route_handler = super().get_route_handler()

        async def custom_route_handler(request: Request) -> Response:
            try:
                return await asyncio.wait_for(original_route_handler(request), timeout=REQUEST_TIMEOUT)
            except asyncio.TimeoutError:
                raise HTTPException(status_code=504, detail='timeout error !!!')
       
        return custom_route_handler

async def cpu_bound_task(text):
    while 1:
        print("hello")
        await asyncio.sleep(1)

    return text


router = APIRouter(route_class=CustomAPIRoute)
@router.get('/')
async def main():
    await cpu_bound_task(text='Hello world')
    print("this should not executed if timeout")

    return {'response': 111}

app.include_router(router)

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=9999)

i get the reference from this

We still have no answer on how to release the process after the timeout is sent, right? This is a MAJOR resource leak.

Even in pure async cases, no sync blocking calls, the request seems to keep running in the background forever even after the timeout response is returned to the user

@LMalikov wrote a great example:

Even if the router function contains async code it doesn't get interrupted/cancelled with this middleware solution. The following example keeps printing Running... endlessly even though TimeoutException is triggered and underlying Task created by asyncio.wait_for(...) gets cancelled.

app = FastAPI()

@app.get("/long_running")
async def long_running():
    try:
        while True:
            print("Running...")
            await asyncio.sleep(1)
    except asyncio.CancelledError:  # This never happens :(
        print("Cancelled.")

@app.middleware("http")
async def timeout_middleware(request: Request, call_next):
    try:
        return await asyncio.wait_for(call_next(request), timeout=3)
    except asyncio.TimeoutError:
        return JSONResponse({'detail': 'Request processing time exceeded limit'}, 504)

@tiangolo shouldn't we hit except asyncio.CancelledError in this case? 🙏

If you call /long_running multiple times, you end up with many infinite loops running in the background without stopping.

Is there a solution to this?

I understand that signal.alarm doesn't solve the problem if we expect to have concurrency in the process, as it kills requests of other users.

For people looking for a proper solution using an ASGI middleware (don't use BaseHTTPMiddleware or @app.middleware("http"), they have side effects), here it is:

class TimeoutMiddleware:
    def __init__(self, app: ASGIApp, timeout: float) -> None:
        self.app = app
        self.timeout = timeout

    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
        if scope["type"] != "http":  # pragma: no cover
            await self.app(scope, receive, send)
            return

        response_started = False

        async def send_wrapper(message: Message) -> None:
            nonlocal response_started
            if message["type"] == "http.response.start":
                response_started = True
            await send(message)

        try:
            with anyio.fail_after(self.timeout):
                await self.app(scope, receive, send_wrapper)
        except TimeoutError as err:
            if not response_started:
                response = Response(status_code=504)
                await response(scope, receive, send)

This will work for async routes, and will correctly cancel any awaiting route.