What does the PR do?

Removes the cached all_special_tokens_extended property from the OpenAI frontend tokenizer helper. This attribute was removed in transformers v5 (the PreTrainedTokenizer* classes were replaced by backend classes such as TokenizersBackend), so accessing it raises AttributeError: TokenizersBackend has no attribute all_special_tokens_extended and crashes the OpenAI frontend on startup.

The attribute is equivalent to all_special_tokens, is not used anywhere in the frontend, and removing it restores forward compatibility with transformers v5 (the vLLM test container currently ships transformers 5.6.0). This mirrors the equivalent upstream change in vLLM (vllm-project/vllm#29686).

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

• Upstream equivalent: vllm-project/vllm#29686

Where should the reviewer start?

python/openai/openai_frontend/engine/utils/tokenizer.py — the get_cached_tokenizer() function (removed the all_special_tokens_extended read and its cached @property).

Test plan:

• CI Pipeline ID: 53600268

Caveats:

HuggingFace documents _special_tokens_map / _extra_special_tokens as the v5 replacements for the extended special-token maps, but those are not needed here since the value was only an alias of all_special_tokens and is unused by the frontend.

Background

The vLLM test container was upgraded to transformers >= 5.0.0 (currently 5.6.0). transformers v5 removed special_tokens_map_extended and all_special_tokens_extended (huggingface/transformers#40936, v5 tokenization migration guide). This frontend tokenizer helper was adapted from older vLLM code and still referenced the removed attribute.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)