Codestin Search App

zucchini-nlp · 2026-04-24T12:26:43Z

What does this PR do?

The moonshot team is not very active and haven't yet replied, so I kept all the model_type naming to match the remote code. That allows us to load a model with local code. There are still a few problems such as tokenizer needing conversion and jinja chat template being non-consistent though

They have saved a tiktoken tokeninzer and tbh I am not sure if we can convert it, and nudge them to update. So as a workaround we convert from tiktoken at load-time

Tested with below script but uses a smaller/dummy model. Logits and eager generation are equivalent with official implementation.

import torch
from transformers import AutoProcessor, AutoTokenizer, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained('moonshotai/Kimi-K2.6', trust_remote_code=False)
model = AutoModelForImageTextToText.from_pretrained(
    'moonshotai/Kimi-K2.6',
    dtype=torch.bfloat16,
    device_map="cuda:0",
    trust_remote_code=False,
    attn_implementation="eager",
)


messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "image.png"},
            # {"type": "video", "video_url":{"url"  "video.mp4"}}, # note that videos need different layout due to jinja template
            {"type": "text", "text": "What is shown in this image?"},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(device=model.device, dtype=torch.bfloat16)

generated_ids = model.generate(**inputs, max_new_tokens=64)
generated_text = processor.batch_decode(generated_ids[:, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)[0]
print(generated_text)

zucchini-nlp · 2026-04-24T16:51:28Z

Oke, I think the model part looks same as the custom code, though I didn't test it. Also we need to check what is the exact chat template format and how to add placeholder tokens per modality, again with the real checkpoint

Kimi team might reply on Monday, coming back then

vasqu

Some initial comments from my side, I wanted to focus on things that we could simplify - mainly the RoPE I think.

I know it's WIP in a way so don't mind comments you already want to address yourself

zucchini-nlp · 2026-05-11T17:01:29Z

+        if image_processor_type is None:
+            try:
+                if not isinstance(config, PreTrainedConfig):
+                    config = AutoConfig.from_pretrained(
+                        pretrained_model_name_or_path, trust_remote_code=trust_remote_code, **kwargs
+                    )
+
+                image_processor_type = getattr(config, "image_processor_type", None)
+                if hasattr(config, "auto_map") and "AutoImageProcessor" in config.auto_map:
+                    image_processor_auto_map = config.auto_map["AutoImageProcessor"]
+            except ValueError:
+                # Config loading failed (unrecognized model_type, invalid config, etc.)
+                # Continue to fallback logic below (AutoTokenizer, AutoImageProcessor, etc.)
+                pass

        # Derive base_class_name from image_processor_type
        is_legacy_fast = False
        base_class_name = None
        if image_processor_type is not None:
            is_legacy_fast = image_processor_type.endswith("Fast")
            base_class_name = image_processor_type[:-4] if is_legacy_fast else image_processor_type

        backend = _resolve_backend(backend_kwarg, use_fast, base_class_name)

        image_processor_class = None
        if base_class_name is not None:
            image_processor_class = _load_backend_class(base_class_name, backend, is_legacy_fast)

        # Handle remote code
        has_remote_code = image_processor_auto_map is not None
        has_local_code = image_processor_class is not None or type(config) in IMAGE_PROCESSOR_MAPPING
        explicit_local_code = has_local_code and not (
            image_processor_class or _load_class_with_fallback(IMAGE_PROCESSOR_MAPPING[type(config)], backend)
        ).__module__.startswith("transformers.")
        if has_remote_code:
            class_ref = _resolve_auto_map_class_ref(image_processor_auto_map, backend)
            upstream_repo = class_ref.split("--")[0] if "--" in class_ref else None
            trust_remote_code = resolve_trust_remote_code(


copied from processing_auto.pt btw. The idea is to infer local class name if possible from config, even if we found a remote code

The remote code will still be loaded with higher prio if users pass trust_remote_code, so nothing should break

vasqu · 2026-07-03T10:19:18Z

+@requires(
+    backends=[
+        "torch",
+    ]
+)


Hmm why is this needed?

because it requires torch :)

The repo consistency isn't happy and tries to import it with no-torch env, then it raises error

oke, maybe this folder is supposed to be non-torch. I see FP conversions are in root/integrations/

vasqu · 2026-07-03T11:04:00Z

run-slow: kimi_k25

github-actions · 2026-07-03T11:05:14Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/kimi_k25"]
quantizations: []

IlyasMoutawwakil · 2026-07-03T11:35:18Z

+        outputs = []
+        running_length = 0
+        for t, h, w in grid_thw.tolist():
+            # Get the current sequence
+            seq = hidden_states[running_length : running_length + t * h * w]
+            # Reshape along self.merge_kernel_size and concat to the last dimension
+            new_height, new_width = h // kernel_height, w // kernel_width
+            reshaped_seq = seq.view(t, new_height, kernel_height, new_width, kernel_width, hidden_dim)
+            reshaped_seq = reshaped_seq.transpose(2, 3).mean(dim=0)  # temporal pooling
+            padded_seq = reshaped_seq.reshape(new_height * new_width, kernel_height * kernel_width, -1)
+            outputs.append(padded_seq)
+            running_length += t * h * w
+
+        return torch.cat(outputs, dim=0)


can you please use the compile/export friendly vision_utils here 🙏

there is no utility to permute and pooling each image, no?

Co-authored-by: Anton Vlasjuk <[email protected]>

vasqu · 2026-07-03T13:46:12Z

run-slow: kimi_k25

zucchini-nlp · 2026-07-03T13:54:40Z

run-slow: kimi_k25

zucchini-nlp · 2026-07-03T13:59:38Z

run-slow: kimi_k25

github-actions · 2026-07-03T14:00:58Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/kimi_k25"]
quantizations: []

…nto kimi

github-actions · 2026-07-03T14:22:37Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	487da00f	workflow commit (merge commit)
PR	0a1fcd4d	branch commit (from PR)
main	5d063058	base commit (on `main`)

Model CI Report

❌ 4 new failed tests from this PR 😭

kimi_k25:
tests/models/kimi_k25/test_modeling_kimi_k25.py::Kimi_K25ModelTest::test_generate_compilation_all_outputs (✅ ⟹ ❌)
tests/models/kimi_k25/test_modeling_kimi_k25.py::Kimi_K25ModelTest::test_generate_compile_model_forward_fullgraph (✅ ⟹ ❌)
tests/models/kimi_k25/test_modeling_kimi_k25.py::Kimi_K25ModelTest::test_generate_from_inputs_embeds_with_static_cache (✅ ⟹ ❌)
tests/models/kimi_k25/test_modeling_kimi_k25.py::Kimi_K25ModelTest::test_generate_with_static_cache (✅ ⟹ ❌)

This reverts commit 5952534.

vasqu · 2026-07-03T14:39:06Z

Looks like it was not reverted?

yeah, weird, it didn't get reverted

github-actions · 2026-07-03T15:20:41Z

CI recap

Dashboard: View test results in Grafana
Latest run: 28667688910:1
Result: failure | Jobs: 15 | Tests: 171,114 | Failures: 1 | Duration: 25h 54m

github-actions · 2026-07-03T15:28:38Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, kimi_k25

zucchini-nlp added 5 commits April 24, 2026 14:25

add kimi

7eea624

modular run conversion

89d373b

add processing for image for now

44c2063

video processor, maybe like this

1f940fd

i think this is defi gemma4 rope

f19aa2d

zucchini-nlp force-pushed the kimi branch from 2184255 to f19aa2d Compare April 24, 2026 15:34

add model tester

aacd808

zucchini-nlp force-pushed the kimi branch from 36c642d to aacd808 Compare April 24, 2026 16:49

zucchini-nlp added 10 commits April 30, 2026 12:52

convert weights

eb3d46b

fix processing

4a81176

looks fine with dummy weights

0249fa1

chunk/permute rope dynamically

2bc9732

fix repo

fe72152

fix some tests

1e2d306

.

86029a5

rename to match remote code config

a4bc583

rename tests as well

0c3267d

oops

8694a52

vasqu reviewed May 7, 2026

View reviewed changes

zucchini-nlp added 4 commits May 11, 2026 14:31

update

ef766b6

some comments

c17f932

more comments about model code

7a248c2

fast tokenizer loaded, thasnk to moonshot

533a7aa

zucchini-nlp commented May 11, 2026

View reviewed changes

zucchini-nlp added 2 commits May 12, 2026 12:26

.

43d9b76

move my ugly code from cluster to GH, needs mocing to proper conversion

1267030

skip export tests

d848e09

vasqu reviewed Jul 3, 2026

View reviewed changes

vasqu added 2 commits July 3, 2026 12:19

fix modular nit

cd241c2

aya

b7bd24c

zucchini-nlp commented Jul 3, 2026

View reviewed changes

Comment thread src/transformers/models/kimi_k25/modular_kimi_k25.py Outdated

erm, oke, mvoe it to another folder following fpquant

7959646

IlyasMoutawwakil reviewed Jul 3, 2026

View reviewed changes

IlyasMoutawwakil self-requested a review July 3, 2026 11:59

use the correct ckpt for tests

8a5c933

huggingface deleted a comment from github-actions Bot Jul 3, 2026

Merge remote-tracking branch 'origin/main' into kimi

29d43b0

vasqu reviewed Jul 3, 2026

View reviewed changes

Comment thread docs/source/en/model_doc/kimi_k25.md Outdated

Update docs/source/en/model_doc/kimi_k25.md

f9f0d12

Co-authored-by: Anton Vlasjuk <[email protected]>

Merge branch 'main' into kimi

0a1fcd4

IlyasMoutawwakil added 2 commits July 3, 2026 16:09

fix kimi k26 export

5952534

Merge branch 'kimi' of https://github.com/zucchini-nlp/transformers i…

cb30f09

…nto kimi

Revert "fix kimi k26 export"

762aacf

This reverts commit 5952534.

vasqu reviewed Jul 3, 2026

View reviewed changes

copy from main

ba7f97c

skip

448c296

vasqu merged commit 0983570 into huggingface:main Jul 3, 2026
99 of 100 checks passed

Uh oh!

Conversation

zucchini-nlp commented Apr 24, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

zucchini-nlp commented Apr 24, 2026

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vasqu commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vasqu commented Jul 3, 2026

Uh oh!

zucchini-nlp commented Jul 3, 2026

Uh oh!

zucchini-nlp commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

CI Results

Commit Info

Model CI Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 3, 2026

CI recap

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

zucchini-nlp commented Apr 24, 2026 •

edited by github-actions Bot

Loading