Thanks to visit codestin.com
Credit goes to github.com

Skip to content

docs: update README to include information about Windows support #213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 13, 2024
Merged

docs: update README to include information about Windows support #213

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7fa9222
docs: update README to include information about Windows support
Mar 13, 2024
4092776
Update wording
Mar 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 9 additions & 8 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The library contains
* Implementations of the `ModelRunner` interface. `ModelRunner` encapsulates the logic for invoking different types of LLMs, exposing a `predict` method to simplify interactions with LLMs within the eval algorithm code. We have built-in support for Amazon SageMaker Endpoints and JumpStart models. The user can extend the interface for their own model classes by implementing the `predict` method.

## Installation
`fmeval` is developed under python3.10. To install the package from PIP you can simply do:
`fmeval` is developed under python3.10. To install the package, simply run:

```
pip install fmeval
Expand Down Expand Up @@ -62,7 +62,9 @@ eval algorithms.*

## Troubleshooting

1. If you you run into the error `error: can't find Rust compiler` while installing on a Mac, please try running the steps below.
1. Users running `fmeval` on a Windows machine may encounter the error `OSError: [Errno 0] AssignProcessToJobObject() failed` when `fmeval` internally calls `ray.init()`. This OS error is a known Ray issue, and is detailed [here](https://github.com/ray-project/ray/issues/21994). Multiple users have reported that installing Python from the [official Python website](https://www.python.org/downloads/windows/) rather than the Microsoft store fixes this issue. You can view more details on limitations of running Ray on Windows on [Ray's webpage](https://docs.ray.io/en/latest/ray-overview/installation.html#windows-support).

2. If you run into the error `error: can't find Rust compiler` while installing `fmeval` on a Mac, please try running the steps below.

```
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Expand All @@ -73,16 +75,15 @@ rm -rf $HOME/.rustup/toolchains/stable-aarch64-apple-darwin
mv $HOME/.rustup/toolchains/1.72.1-aarch64-apple-darwin $HOME/.rustup/toolchains/stable-aarch64-apple-darwin
```

2. If you run into OOM errors, especially while running evaluations that use LLMs as evaluators like toxicity and
summarization accuracy, it might be happening because your machine does not have enough memory to load the evaluator
models on to all the cores available to it by default to maximize parallelization. To reduce parallelization, users can
set the environment variable `PARALLELIZATION_FACTOR` to a value that works on their machine. This reduces the number of
cores on to which these models are loaded on to, and avoids running into OOM errors.
3. If you run into out of memory (OOM) errors, especially while running evaluations that use LLMs as evaluators like toxicity and
summarization accuracy, it is likely that your machine does not have enough memory to load the evaluator
models. By default, `femval` loads multiple copies of the model into memory to maximize parallelization, where the exact number depends on the number of cores on the machine. To reduce the number of models that get loaded in parallel, you can
set the environment variable `PARALLELIZATION_FACTOR` to a value that suits your machine.

## Development

### Setup and the use of `devtool`
Once you have created a virtual environment with python3.10, run the following command to setup the development environment:
Once you have created a virtual environment with python3.10, run the following command to set up the development environment:
```
./devtool install_deps_dev
./devtool install_deps
Expand Down