This is the official code for the "BESPOKE: Benchmark for Search-Augmented Large Language Model Personalization via Diagnostic Feedback"

This repository does not include datasets. All data is provided in this repository.

If you plan to run the evaluators, place the provided data under the dataset/ directory so the scripts can reference it.

Query Table (queries/query_table.parquet)

Gold Information (gold_information/gold_information.parquet)

Personalized Rubric (personalized_rubric/personalized_rubric.parquet)

Meta Test (meta_test/meta_test_set.parquet)

Evaluation Shot (evaluation_shot/evaluation_shot.parquet)

Chat/Search History (chat_history/chat_history.parquet, search_history/search_history.parquet)

Query table row (queries/query_table.parquet):

{
  "user": "user1",
  "query-id": "1",
  "query": "Why is marketing so popular these days?",
  "gold_information_need": "..."
}

Set your API key (or pass it directly in code):

export OPENAI_API_KEY=YOUR_KEY

Required files:

• dataset/evaluation_shot/{user}/query-id_{N}.json
• dataset/personalized_rubric/{user}/query-id_{N}.txt

Run via the provided script (fill in placeholders inside evaluate/run_personalizaion_eval.py):

python evaluate/run_personalizaion_eval.py

Set these fields inside the script:

• user (e.g., "user2")
• query_id (e.g., 6)
• response_text (the response text to evaluate)
• eval_shot_root (e.g., dataset/evaluation_shot)
• rubric_root (e.g., dataset/personalized_rubric)
• model (default: gpt-5)

Output (printed as JSON):

• need_alignment_score, content_depth_score, tone_score, explanation_style_score plus corresponding feedback strings

Programmatic usage (alternative):

from pathlib import Path
from evaluate.evaluator import PersonalizationEvaluator

evaluator = PersonalizationEvaluator(
    eval_shot_root=Path("dataset/evaluation_shot"),
    rubric_root=Path("dataset/personalized_rubric"),
    model="gpt-5",
)
result = evaluator.evaluate(user="user2", query_id=6, response_text="... your response ...")

Required file:

• dataset/gold_information/{user}/query-id_{N}.json (format example)

{
  "gold_information": ["Claim A", "Claim B"]
}

Run via the provided script (fill in placeholders inside evaluate/run_recall_eval.py):

python evaluate/run_recall_eval.py

Set these fields inside the script:

• user (e.g., "user2")
• query_id (e.g., 6)
• response_text (the response text to evaluate)
• gold_information_root (e.g., dataset/gold_information)
• model (default: gpt-5)

Output (printed as JSON):

• matched_gold_information: list of gold claims found in the response
• recall: matched fraction (0.0–1.0)

Notes:

• Paths in the scripts can be absolute or relative to the repo root.
• The scripts are templates; fill in the placeholders before running.

CLI to generate a personalized answer for a single query. Use --enforce_search to enable web search/browsing for grounding.

Optional environment variables:

• OpenAI: OPENAI_API_KEY
• Perplexity: PERPLEXITY_API_KEY
• Gemini: GEMINI_API_KEY

Main arguments:

• --user_id (required), --query (required)
• --user_context Inline user preferences/context text
• --model_type openai|perplexity|gemini (default: openai)
• --model_name (default: gpt-4o-mini)
• --enforce_search Enable web search/browsing
• --output_path Path to save JSON result
• --print_json Print full JSON result

Output:

• Default: prints the final answer text only
• With --print_json: prints a JSON containing response, response_urls, model, etc.

If you use this dataset, please consider citing it:

@misc{kim2025bespokebenchmarksearchaugmentedlarge,
      title={BESPOKE: Benchmark for Search-Augmented Large Language Model Personalization via Diagnostic Feedback}, 
      author={Hyunseo Kim and Sangam Lee and Kwangwook Seo and Dongha Lee},
      year={2025},
      eprint={2509.21106},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.21106}, 
}