This repository is a develpoment build on Mind2Web's candidate generation module.

We incorporate LLM's ability to predict the anticipated action based on task description and previous action. The pair_model is trained with these aciton_guidance(anticipated action predicted by LLM) and HTML element to score the relevancy. Then ensemble strategies are implemented in inference, option of two modes: 1. Mix_rank , 2. Mix_score .

Detailed explanation of the access to the Mind2Web training dataset can be found here The action_guidance generated for train dataset is stored as pair.json and for evaluation dataset split test_website_0.json, test_task_1.json, test_domain_0.json is stored as pair_eval.json.

To finetune the candidate generation module, you need to prepare the trained DeBERTa-v3-base model which can be found on Huggingface Model Hub.

The pretrained pair_model can be downloaded here

Baseline of ~ Recall@50 ~ 85% is improved to ~ Recall@50 ~ 87%

python candidate_generation/evaluate.py\
    --model_path {MODEL_PATH_OR_NAME OF DeBERTa-v3-base}\
    --pair_model_path {MODEL_PATH OF FINETUNED MODEL}\
    --data_path {DATA_PATH}\
    --split_file {SPLIT_TO_EVALUATE}\
    --output_dir {OUTPUT_DIR}\
    --mix_type  mix_rank\
    --alpha  0.5

• model_path: path to the model or model name on Huggingface Model Hub.
• pair_model_path: path to the fine-tuned model with action guidance.
• data_path: path to the dataset directory, e.g., ${BASE_DIR}/Mind2Web.
• split_file: path to the split file, e.g., data/test_website/*.json.
• output_dir: path to the output directory. You will see two files: results_*.json with the evaluation metrics, and scores_*.pkl with the prediction scores which can be used for the action prediction module.
• mix_type: Two types available, mix_rank, mix_score
• alpha: alpha value from 0 to 1. alpha * base_line model, (1-alpha) * pair_model

To fine-tune the model,

1. Add your OPEN_API_KEY to candidate_generation/conf/config.yaml
2. Add path to save/load pair_dict_path(Action Guidance). Sample file is located at /src/candidate_generation/pair_summarize.json

run:

python candidate_generation/train.py model=deberta-v3-base

• model: Model config to load.

It uses config file in candidate_generation/conf/config.yaml. The checkpoints will be saved under the workdir configured by hydra.