Initial modular implementation covering DeepSeek-V4-Flash/Pro and their
-Base siblings (all share the same architecture). New pieces vs V3.2:

* Sliding-window attention with a per-layer KV Compressor (learned gated
  pooling) and an Indexer selecting top-k compressed positions for
  long-range attention. No MLA.
* Hyper-Connections replace the residual stream (always on).
* Mixtral-style top-k MoE routing, no expert groups. First num_hash_layers
  layers route via a frozen tid2eid lookup keyed by input token ids.
* Per-head learnable attention sink; grouped low-rank output projection.

MTP weights in the checkpoint are ignored on load (added elsewhere).
Eager-only attention for now — SDPA/flash backends do not yet support
the sink term.

* RoPE: drop custom embedding; use LlamaRotaryEmbedding. qk_rope_head_dim
  is honoured via rope_parameters['partial_rotary_factor'] which routes
  through the shared partial-aware init path. Main vs compressed rope
  bases built via a small helper (_build_rotary) at the Model level.
* RoPE apply: use apply_rotary_pos_emb_interleave from V3 for q/k rope
  slice (V4 reference uses interleaved-pair rotation via complex mul).
* Attention sink: port eager_attention_forward from GPT-OSS verbatim
  (renamed 'attn_sink' -> 'sinks' to match checkpoint/HF naming).
* SwiGLU clamp: match GPT-OSS clamp semantics on routed experts; shared
  expert is unclipped. Inlined into a forward override on DeepseekV4Experts
  to stay compatible with @use_experts_implementation.
* Compressor/Indexer statefulness: both are stateless now. State lives on
  a new DeepseekV4Cache(DynamicCache) — per-layer compressor_state and
  indexer_state dicts (buffer_kv, buffer_gate, pooled_kv). Window K/V
  continues through DynamicCache's DynamicSlidingWindowLayer.
* Remove _project_q / _project_kv helpers; fold into forward.
* Remove _score_fn; use ACT2FN via a tiny _resolve_activation wrapper
  that also understands 'sqrtsoftplus' (not in the global registry).
* HyperConnection: single module with a forward that wraps an inner
  callable and does pre-reduce -> inner -> post-expand. attn_hc and
  mlp_hc are now invoked through __call__.
* MLP: packed gate_up_proj; shared expert uses it too.
* Hash + top-k routers: unconditional norm_topk_prob normalisation
  (V4 ships with norm_topk_prob=True; dropped the conditional).
* hc_head + final RMSNorm live on DeepseekV4Model, not ForCausalLM.
  Matches the standard transformers contract: Model returns
  [B, S, hidden], ForCausalLM only owns lm_head.

Tests (4) pass. ruff + check_config_attributes clean.

* Config inherits DeepseekV3Config; V3 MLA/group fields set to None
  and allow-listed in check_config_attributes; skip V3 __post_init__
  so V4's head_dim=512 is preserved.
* RMSNorm + RotaryEmbedding inherit V3 classes directly (no rebuild).
  Main + compress rotary built inline in Model by swapping
  rope_parameters on a copy.copy(config).
* Drop _SqrtSoftplus / _resolve_activation. Routers use ACT2FN where
  possible; sqrtsoftplus fallback is an inline F.softplus(x).sqrt().
* Drop _build_rotary helper.
* Cache: DeepseekV4SlidingLayer stores K=V once (no double-update);
  DeepseekV4Cache installs those layers + compressor/indexer state.
* DeepseekV4GroupedLinear is an nn.Linear subclass for the grouped
  low-rank output projection — quantizers keyed on .weight still see
  a valid (out, in) shape; forward does per-group bmm.
* Remove module-level DeepseekV4Experts.forward monkey-patch; proper
  @use_experts_implementation class with clamp inline in forward.
* Shared expert inherits Qwen2MoeMLP (packed gate/up not used there —
  V3/Qwen2MoE convention).
* DeepseekV4TopKRouter / HashRouter: standalone, same weight+bias
  layout as V3, V4 scoring/renorm inline. Hash router's forward
  computes logits inline (no super() chain into V3 to survive
  modular conversion).
* HyperConnection: single module, forward(hidden_states, inner,
  layernorm, **kwargs); decoder layer calls attn_hc(...) and
  mlp_hc(...) directly — no _attn_inner / _mlp_inner callbacks.
* hc_head + final RMSNorm live on DeepseekV4Model.
* DeepseekV4ForCausalLM only defines __init__; forward is inherited
  from MixtralForCausalLM unchanged.

HyperConnection restructure:
* DeepseekV4HyperConnection now owns 'inner' (attn/mlp) and 'norm'
  (the per-site RMSNorm). Decoder-layer forward collapses to
      hidden_states = self.attn_hc(hidden_states, **kwargs)
      return self.mlp_hc(hidden_states, **kwargs)
  No more passing submodules as call arguments.
* DeepseekV4SparseMoeBlock.forward accepts **_ so the shared kwargs
  flow works for both attn and mlp sites.
* Hash router falls back to top-k over the learned gate weight when
  input_ids isn't threaded (inputs_embeds inference path).

Conversion mapping:
* New 'deepseek_v4' entry in src/transformers/conversion_mapping.py
  with four WeightRenaming rules mapping the standard decoder-layer
  names (self_attn, input_layernorm, mlp, post_attention_layernorm)
  onto the new HC-owned module tree ({attn,mlp}_hc.{inner,norm}).

Config + RoPE:
* rope_scaling removed (it's a property alias of rope_parameters on
  PreTrainedConfig; declaring it as a field made both mutate each
  other and broke to_dict roundtrips).
* partial_rotary_factor is a config field and is set to
  qk_rope_head_dim / head_dim when absent; this is the HF-standard
  mechanism for sizing cos/sin to the rope-only portion of each head.
* DeepseekV4RotaryEmbedding overrides compute_default_rope_parameters
  to honour partial_rotary_factor on the default rope path as well.
* compress_rope_parameters derived from rope_parameters at __post_init__
  with rope_theta swapped.
* compress_ratios: accept either num_hidden_layers or +MTP length,
  truncate to num_hidden_layers.

Stateless Compressor/Indexer:
* Both read/write state on the cache via getattr(..., state_key, None)
  so plain DynamicCache instances (generation default) work without
  crashing; stateful optimisation only kicks in with DeepseekV4Cache.

Tests:
* test_modeling_deepseek_v4.py now inherits CausalLMModelTest +
  CausalLMModelTester. Model-specific config attrs are declared on
  the tester class; get_config() threads them into DeepseekV4Config.
* Pipeline tests skipped (V4 has no ForSequenceClassification /
  ForTokenClassification / ForQuestionAnswering heads).
* 78 of 127 non-skipped tests pass; remaining failures are specific
  edge cases (gradient-checkpointing, torch.compile, rope scaling
  variants) to chase in follow-ups.

* Compressor and Indexer are now pure math. All state accounting
  (per-layer pre-pool buffers, running pooled cache) is managed via
  two free helpers, _accumulate_windows and _update_pool, that live on
  the cache instance (DeepseekV4Cache or, defensively, any DynamicCache).
  Single cache update per call, mirroring past_key_values.update(k, v)
  semantics.
* Compressor.forward always returns a tensor (empty shape [B, 1, 0, D]
  when no window has closed yet) — no more None code paths.
* Indexer no longer owns a nested Compressor; it pools inline through
  the same helpers with a distinct state_key. Only the Compressor owns
  the Indexer, never the other way around.
* Cache-type polymorphism: the helpers work on plain DynamicCache too
  (generation installs one by default), so V4 works with any
  Cache subclass without requiring our custom class.

* Inline the HC 'collapse' step in the decoder layer — it's a one-liner.
  Keep _hyper_connection_weights (shared mix-logit machinery) and
  _hyper_connection_expand (post·out + comb·streams) as helpers.
* Add ASCII diagrams to _hyper_connection_weights and
  DeepseekV4DecoderLayer explaining the HC pipeline vs the classic
  residual decoder layer.
* Add a block comment in DeepseekV4Attention.forward explaining *why*
  the output's rope slice is un-rotated (V shares with K in V4, so
  attention outputs carry position-entangled content on the rope dims;
  conjugate rotation at the query position pulls it back into a
  position-independent frame before the output projection).

HC parameters are cast to fp32 at use time for Sinkhorn stability.

Tests: 109/125 pass, 16 known failures (TP on MPS [env], torch.compile
paths, a numerical-match test sensitive to the attention sink under
padding).

Co-authored-by: Quentin Gallouédec <[email protected]>

Co-authored-by: Arthur <[email protected]>

…s into add-deepseek-v4