Thanks to visit codestin.com
Credit goes to github.com

Skip to content

More robust processor from pretrained#42668

Open
yonigozlan wants to merge 8 commits into
huggingface:mainfrom
yonigozlan:more-robust-processor-from-pretrained
Open

More robust processor from pretrained#42668
yonigozlan wants to merge 8 commits into
huggingface:mainfrom
yonigozlan:more-robust-processor-from-pretrained

Conversation

@yonigozlan

@yonigozlan yonigozlan commented Dec 5, 2025

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #42645

Depends on #42667

Add robustness to processor handling of subprocessors loading and saving by checking the signature of init in the parent classes as well.

@github-actions

github-actions Bot commented Dec 5, 2025

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, lasr, phi4_multimodal, pix2struct

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@yonigozlan yonigozlan requested review from ArthurZucker, Cyrilvallez and zucchini-nlp and removed request for Cyrilvallez and zucchini-nlp December 11, 2025 17:57

@ArthurZucker ArthurZucker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry LGTM later review, needs rebasing !

@MilkClouds

Copy link
Copy Markdown
Contributor

Thanks for the MRO walk, which fixes the inheritance case I reported.

One small follow-up: @zucchini-nlp suggested in the issue thread allowing an explicit attributes override with auto-detection as fallback. Doesn't seem to be in this PR. Would it be worth adding on top of the MRO walk?

@classmethod
def get_attributes(cls):
    if "attributes" in cls.__dict__ and cls.attributes is not None:
        return list(cls.attributes)
    # ... existing MRO-based detection ...

Two concrete reasons:

1. The original case in this issue is still broken with the MRO walk alone.

My actual code uses a custom (non-HF) tokenizer:

class ActionTokenizerMixin:
    def __init__(self, *args, action_tokenizer: Optional[BaseActionTokenizer] = None, **kwargs):
        ...

BaseActionTokenizer is not a PreTrainedTokenizerBase. After the MRO walk, action_tokenizer would be auto-detected as a sub-processor (it contains "tokenizer"), then check_argument_for_proper_class would reject it at processing_utils.py#L720-L723:

if not isinstance(argument, proper_class):
    raise TypeError(
        f"Received a {type(argument).__name__} for argument {argument_name}, "
        f"but a {class_name} was expected."
    )

An explicit attributes = [...] override is the only way to opt this parameter out of auto-detection without losing the MRO behavior for everything else.

2. Substring-based detection already needs core-level carve-outs.

audio_tokenizer matches the "tokenizer" substring but isn't actually a tokenizer (it's a DacModel per MODALITY_TO_BASE_CLASS_MAPPING), so it's currently excluded from the auto-detect flow at six places in processing_utils.py:

It's the only such case today, but any future component whose name happens to overlap a modality substring (such as the non-HF custom tokenizer in this issue) will hit the same shape of problem. An explicit attributes override pushes that decision to the model file instead of patching processing_utils.py again.

Happy to open a separate PR for this on top of yours.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Processor's attribute is mis-detected for child class from #41633

5 participants