conda create -n evals python=3.12 && conda activate evals

This allows rerunning the fetch code without re-fetching identical prompts. Modify the @cached from 1 month as needed. Note that when you shut down the container, the cache dies, so keep the container open across fetch runs. Check docker ps -a to restore.

make redis

python bin/fetch_all.py