Description

Now that we have an AdaptiveSemaphore, we need a way of keeping track of successful and failed requests, and use this information to influence our inference concurrency.

The adaptive concurrency controller handles this by exposing methods to track successes and failures on top of being able to be used as a semaphore. The controller adjusts the semaphore based on a computed error rate, changing the concurrency after a certain number of requests and certain amount of time.

Depending on error rate, the controller will reduce or increase concurrency up to some limits, and waits to make further changes depending on how requests do.

Related issues

Towards OPE-1307

Before submitting

• This PR only changes documentation. (You can ignore the following checks in that case)
• Did you read the contributor guideline Pull Request guidelines?
• Did you link the issue(s) related to this PR in the section above?
• Did you add / update tests where needed?

Reviewers

At least one review from a member of oumi-ai/oumi-staff is required.