What does this PR do?

This PR adds support for Param-2-17B-MoE-A2.4B, a large-scale Mixture-of-Experts (MoE) causal language model.

Param-2-17B-MoE-A2.4B uses a Hybrid Dense + MoE architecture with 17B total parameters while activating only 2.4B parameters per token, enabling high model capacity with efficient inference cost.

The model is pretrained from scratch with strong multilingual capabilities and particular emphasis on linguistic diversity and Indian language representation. It is released as a pretrained base model intended for downstream fine-tuning.

This PR introduces the following components:

• configuration_param2moe.py

  • Implements Param2MoEConfig
  • Defines model hyperparameters including transformer architecture, MoE routing configuration, number of experts, hidden sizes, and other initialization parameters.

• modeling_param2moe.py

  • Implements:
    • Param2MoEModel
    • Param2MoEForCausalLM
  • Includes hybrid transformer blocks with MoE routing and expert dispatch logic.
  • Supports causal language modeling and text generation workflows.

• __init__.py

  • Registers the model configuration and modeling classes to enable loading via:
    • AutoConfig
    • AutoModel
    • AutoModelForCausalLM

This integration enables seamless loading, inference, and downstream fine-tuning using standard Transformers APIs.

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Text models:
@ArthurZucker
@Cyrilvallez