What does this PR do?

This PR optimizes Flash Attention variable-length paths by precomputing the maximum sequence length once at the model/encoder level after cu_seqlens is built, then passing it down through blocks into each attention layer.

Why

Previously, several vision/audio attention implementations recomputed:

(cu_seqlens[1:] - cu_seqlens[:-1]).max()

inside every attention layer on every forward pass.
For Flash Attention varlen kernels, max_seqlen_q and max_seqlen_k are ultimately consumed as Python integer arguments. In eager mode, passing a 0-dimensional CUDA tensor can therefore move the scalar conversion to the Flash Attention / custom op boundary, which may introduce repeated device-to-host synchronization. For an N-layer encoder, this repeats the same max_seqlen computation and scalar conversion path in every layer.

What changed

Compute max_seqlen once in the model/encoder forward path when Flash Attention is requested.
Compute max_window_seqlen once for sliding-window variants.
Pass the precomputed values down through blocks into attention modules.
Keep the attention-level fallback without .item() for standalone attention calls, preserving previous behavior when max_seqlen is not provided.
Guard the computation behind is_flash_attention_requested, so non-Flash-Attention paths do not pay this cost.

Related

This is related to #44962 and the discussion in #44973. Instead of adding scalar conversion inside each attention layer, this PR lifts the computation to the model/encoder level where cu_seqlens is already available, reducing repeated per-layer work and avoiding repeated per-layer scalar conversions in the normal model forward path.

Experiment

Before

After

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

• I confirm that this is not a pure code agent PR.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline and the
  Pull Request checks?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes according to the guidelines?
• Did you write any new necessary tests?

Who can review?

cc @vasqu