This is necessary because when the results are more than 1, we need to keep the order that they were retrieved at for some metrics to work. Example:

qrel = {
    'q1': {
        'd1': 0,
        'd2': 1,
        'd3': 1,
    },
}

run = {
    'q1': {
        'd1': 1,
        'd2': 1,
        'd3': 1,
    }
}
from beir.retrieval.evaluation import EvaluateRetrieval

evaluator = EvaluateRetrieval()
ndcg, _map, recall, precision = evaluator.evaluate(qrel, run, [1,2,3])
print(recall)

Old Version:
{'Recall@1': 0.5, 'Recall@2': 1.0, 'Recall@3': 1.0}
New version:
{'Recall@1': 0.0, 'Recall@2': 0.5, 'Recall@3': 1.0}

The new version is correct because for Recall@1 the d1 result should have been considered and not d3.