Automated integration-failure triage for the daily CI window 2026-09-06 → 2026-09-12.
This issue was generated by AI-assisted automation. The grouping, summaries, and recommended follow-up can be incomplete or misleading; verify the failures before acting.
Serge dispatched one task per failure group below — each opens or updates its own PR on a serge/fix/itf-<fingerprint> branch. This table is refreshed in place as Serge runs: a group links its #<pr> when opened, shows 🚫 no fix when Serge found no safe change, ⚠️ task failed on error, or (pending) while still running (a late PR links on the next nightly run).
Dispatched failure groups
| Model |
Error |
Occurrences |
PR |
inkling (regressed by PR #47827) |
mixed — tensor values differ (4) |
4 |
#48667 |
generation |
cuda_runtime — other (12) |
12 |
🚫 no fix |
qwen3_vl_moe |
other — other (6) |
6 |
⚠️ task failed |
blt |
import_or_config — other (4) |
4 |
⚠️ task failed |
fastspeech2_conformer |
output_mismatch — tensor values differ (2) |
2 |
🚫 no fix |
Not dispatched — environment / dependency
These groups were triaged but NOT handed to Serge: their failure mode is a property of the runner or the environment, so no minimal source patch can fix them. They need a human (runner capacity, a dependency pin).
2 models ran out of device memory (7 failures) — needs runner capacity, not a patch, so none of these were dispatched: bamba (4), llama4 (3).
9 group(s) skipped — stopped failing before this run (musicgen_melody, chameleon, flex_olmo, git +5 more). Not dispatched and not awaiting a human.
Outcome recap
Why each group that opened no PR ended without one, and what it cost — surfaced here from the Serge dashboard. Failing tests links straight to each test's dashboard view, since these are the groups a human has to pick up.
| Group |
Reason |
LLM |
Tokens (in / out) |
Failing tests |
generation |
[not_reproduced] GPU reproduce: the targeted tests did NOT fail at the base commit (not_reproduced) — the failure is stale, already fixed, flaky, or environment-specific. Skipping this group; no investigation, no PR. |
moonshotai/Kimi-K2.7-Code |
— / — |
test_validate_assistant · test_matches_immediate_check_at_max_length · test_matches_immediate_check_on_a_batch · +3 more |
qwen3_vl_moe |
The repository's normalizer (`bash -lc — normalizer: 7 failed: auto_mappings, copies, modular_conversion, inits, docstrings, pipeline_typing, add_dates |
moonshotai/Kimi-K2.7-Code |
1,169,820 / 30,936 |
test_small_model_integration_test · test_small_model_integration_test_batch_different_resolutions · test_small_model_integration_test_batch_wo_image · +3 more |
blt |
The repository's normalizer (`bash -lc — normalizer: 7 failed: auto_mappings, copies, modular_conversion, inits, docstrings, pipeline_typing, add_dates |
moonshotai/Kimi-K2.7-Code |
2,147,022 / 11,781 |
test_model_logits · test_model_logits_bf16 |
fastspeech2_conformer |
The test_training_integration test fails because its random labels are generated from the global PyTorch RNG after set_seed(42). The RNG sequence for torch.rand / torch.normal on CUDA is not stable across PyTorch/driver versions, so the current run produces duration_labels that sum to 209 while the hard-coded expectation assumes a spectrogram of length 219. This is a fixture/input bug, not value drift: the relative difference reaches ~196× and 100% of the 10×10 checked elements mismatch. |
moonshotai/Kimi-K2.7-Code |
1,245,501 / 11,403 |
test_training_integration |
`qwen3_vl_moe` — normalizer output (tail)
--- omitted 5057 leading chars ---
/huggingface__transformers__task__main__f3104ce27e984019a7efe2aebfaf48b2/src/transformers/utils/__init__.py", line 77, in <module>
from .hub import (
File "/tmp/serge-clones/worktrees/huggingface__transformers__task__main__f3104ce27e984019a7efe2aebfaf48b2/src/transformers/utils/hub.py", line 43, in <module>
from huggingface_hub.utils import (
ImportError: cannot import name 'httpx' from 'huggingface_hub.utils' (/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/__init__.py)
FAILED (0.82s)
Doctest list
$ python utils/check_doctest_list.py --fix_and_overwrite
OK (0.05s)
Model dates
$ python utils/add_dates.py
Traceback (most recent call last):
File "/tmp/serge-clones/worktrees/huggingface__transformers__task__main__f3104ce27e984019a7efe2aebfaf48b2/utils/add_dates.py", line 11, in <module>
from transformers import logging
File "/tmp/serge-clones/worktrees/huggingface__transformers__task__main__f3104ce27e984019a7efe2aebfaf48b2/src/transformers/__init__.py", line 30, in <module>
from . import dependency_versions_check
File "/tmp/serge-clones/worktrees/huggingface__transformers__task__main__f3104ce27e984019a7efe2aebfaf48b2/src/transformers/dependency_versions_check.py", line 16, in <module>
from .utils.versions import require_version, require_version_core
File "/tmp/serge-clones/worktrees/huggingface__transformers__task__main__f3104ce27e984019a7efe2aebfaf48b2/src/transformers/utils/__init__.py", line 77, in <module>
from .hub import (
File "/tmp/serge-clones/worktrees/huggingface__transformers__task__main__f3104ce27e984019a7efe2aebfaf48b2/src/transformers/utils/hub.py", line 43, in <module>
from huggingface_hub.utils import (
ImportError: cannot import name 'httpx' from 'huggingface_hub.utils' (/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/__init__.py)
FAILED (0.90s)
Dependency versions table
$ python setup.py deps_table_update
running deps_table_update
OK (0.34s)
7 failed: auto_mappings, copies, modular_conversion, inits, docstrings, pipeline_typing, add_dates
Using Python 3.10.21 environment at: /usr/local
Resolved 1 package in 1.50s
Building transformers @ file:///tmp/serge-clones/worktrees/huggingface__transformers__task__main__f3104ce27e984019a7efe2aebfaf48b2
Built transformers @ file:///tmp/serge-clones/worktrees/huggingface__transformers__task__main__f3104ce27e984019a7efe2aebfaf48b2
Prepared 1 package in 3.45s
Uninstalled 1 package in 1ms
warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
If the cache and target directories are on different filesystems, hardlinking may not be supported.
If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
Installed 1 package in 5ms
~ transformers==5.18.0.dev0 (from file:///tmp/serge-clones/worktrees/huggingface__transformers__task__main__f3104ce27e984019a7efe2aebfaf48b2)
make: *** [Makefile:63: fix-repo] Error 1
`blt` — normalizer output (tail)
--- omitted 5056 leading chars ---
ggingface__transformers__task__main__e417cf41bef6483abdd9bb5651b503b3/src/transformers/utils/__init__.py", line 77, in <module>
from .hub import (
File "/tmp/serge-clones/worktrees/huggingface__transformers__task__main__e417cf41bef6483abdd9bb5651b503b3/src/transformers/utils/hub.py", line 43, in <module>
from huggingface_hub.utils import (
ImportError: cannot import name 'httpx' from 'huggingface_hub.utils' (/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/__init__.py)
FAILED (0.88s)
Doctest list
$ python utils/check_doctest_list.py --fix_and_overwrite
OK (0.05s)
Model dates
$ python utils/add_dates.py
Traceback (most recent call last):
File "/tmp/serge-clones/worktrees/huggingface__transformers__task__main__e417cf41bef6483abdd9bb5651b503b3/utils/add_dates.py", line 11, in <module>
from transformers import logging
File "/tmp/serge-clones/worktrees/huggingface__transformers__task__main__e417cf41bef6483abdd9bb5651b503b3/src/transformers/__init__.py", line 30, in <module>
from . import dependency_versions_check
File "/tmp/serge-clones/worktrees/huggingface__transformers__task__main__e417cf41bef6483abdd9bb5651b503b3/src/transformers/dependency_versions_check.py", line 16, in <module>
from .utils.versions import require_version, require_version_core
File "/tmp/serge-clones/worktrees/huggingface__transformers__task__main__e417cf41bef6483abdd9bb5651b503b3/src/transformers/utils/__init__.py", line 77, in <module>
from .hub import (
File "/tmp/serge-clones/worktrees/huggingface__transformers__task__main__e417cf41bef6483abdd9bb5651b503b3/src/transformers/utils/hub.py", line 43, in <module>
from huggingface_hub.utils import (
ImportError: cannot import name 'httpx' from 'huggingface_hub.utils' (/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/__init__.py)
FAILED (0.86s)
Dependency versions table
$ python setup.py deps_table_update
running deps_table_update
OK (0.37s)
7 failed: auto_mappings, copies, modular_conversion, inits, docstrings, pipeline_typing, add_dates
Using Python 3.10.21 environment at: /usr/local
Resolved 1 package in 1.48s
Building transformers @ file:///tmp/serge-clones/worktrees/huggingface__transformers__task__main__e417cf41bef6483abdd9bb5651b503b3
Built transformers @ file:///tmp/serge-clones/worktrees/huggingface__transformers__task__main__e417cf41bef6483abdd9bb5651b503b3
Prepared 1 package in 3.45s
Uninstalled 1 package in 0.99ms
warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
If the cache and target directories are on different filesystems, hardlinking may not be supported.
If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
Installed 1 package in 5ms
~ transformers==5.18.0.dev0 (from file:///tmp/serge-clones/worktrees/huggingface__transformers__task__main__e417cf41bef6483abdd9bb5651b503b3)
make: *** [Makefile:63: fix-repo] Error 1
Generated 2026-09-12T22:26:21+00:00.
Automated integration-failure triage for the daily CI window
2026-09-06 → 2026-09-12.This issue was generated by AI-assisted automation. The grouping, summaries, and recommended follow-up can be incomplete or misleading; verify the failures before acting.
Serge dispatched one task per failure group below — each opens or updates its own PR on a
serge/fix/itf-<fingerprint>branch. This table is refreshed in place as Serge runs: a group links its#<pr>when opened, shows🚫 no fixwhen Serge found no safe change,⚠️ task failedon error, or(pending)while still running (a late PR links on the next nightly run).Dispatched failure groups
inkling(regressed by PR #47827)generationqwen3_vl_moebltfastspeech2_conformerNot dispatched — environment / dependency
These groups were triaged but NOT handed to Serge: their failure mode is a property of the runner or the environment, so no minimal source patch can fix them. They need a human (runner capacity, a dependency pin).
2 models ran out of device memory (7 failures) — needs runner capacity, not a patch, so none of these were dispatched:
bamba(4),llama4(3).9 group(s) skipped — stopped failing before this run (
musicgen_melody,chameleon,flex_olmo,git+5 more). Not dispatched and not awaiting a human.Outcome recap
Why each group that opened no PR ended without one, and what it cost — surfaced here from the Serge dashboard. Failing tests links straight to each test's dashboard view, since these are the groups a human has to pick up.
generationnot_reproduced) — the failure is stale, already fixed, flaky, or environment-specific. Skipping this group; no investigation, no PR.moonshotai/Kimi-K2.7-Codeqwen3_vl_moemoonshotai/Kimi-K2.7-Codebltmoonshotai/Kimi-K2.7-Codefastspeech2_conformertest_training_integrationtest fails because its random labels are generated from the global PyTorch RNG afterset_seed(42). The RNG sequence fortorch.rand/torch.normalon CUDA is not stable across PyTorch/driver versions, so the current run producesduration_labelsthat sum to 209 while the hard-coded expectation assumes a spectrogram of length 219. This is a fixture/input bug, not value drift: the relative difference reaches ~196× and 100% of the 10×10 checked elements mismatch.moonshotai/Kimi-K2.7-Code`qwen3_vl_moe` — normalizer output (tail)
`blt` — normalizer output (tail)
Generated 2026-09-12T22:26:21+00:00.