Codestin Search App

Kwai-Keye · 2025-09-22T03:51:07Z

Model Upgrade: Keye-VL-1.5-8B

Overview

This PR introduces an upgraded version of the visual-language model, transitioning from the previous keye-preview to keye-vl-1.5-8B. The update includes architectural refinements, documentation improvements, code optimizations, and style enhancements.

Key Changes

1. Model Architecture

Fine-tuned the model structure for improved performance and efficiency.
Enhanced visual-language alignment capabilities.

2. Documentation

Updated and optimized relevant documentation for better clarity and usability.
Added detailed descriptions of new features and modifications.

3. Code Optimization

Streamlined code workflow for enhanced maintainability.
Removed redundant parameters and code lines to improve efficiency.
Performed code style adaptations to ensure consistency with project standards.

Impact

Improved model performance and accuracy.
Enhanced code readability and maintainability.
Reduced computational overhead by eliminating unnecessary parameters.

Usage

Refer to the updated documentation for detailed instructions on using the new model version.

Notes

This upgrade maintains backward compatibility with existing pipelines.
Users are encouraged to review the updated docs for optimal utilization of new features.

Kwai-Keye · 2025-09-22T05:15:54Z

@zucchini-nlp Please help take a look at this PR, thanks.

zucchini-nlp · 2025-09-22T08:20:37Z

Nice, just wondering if it is same or similar to #39292 in terms of architecture? If it is, then imo it makes more sense to push the prev PR and allow 1.5 8B to re-use modules from it. WDYT?

Kwai-Keye · 2025-09-22T08:50:48Z

@zucchini-nlp
I closed #39292 just now, we won't be submitting #39292 version.
But how do we resolve the failures in the screenshot below? There's a bizarre issue in page where importing the torch library fails.

zucchini-nlp · 2025-09-22T08:59:27Z

To resolve the torch issue, you need to import from .image_processing_keye_vl_1_5 import KeyeVL1_5ImageProcessor and from transformers.models.qwen2.tokenization_qwen2_fast import Qwen2TokenizerFast in processing file with safe guards. Both require is_vision_available() and the second one also requires is_torchvision_available()

The other CI issues seem related to np version used, probably you need to find the equivalent of np.long in the version used in CI

zucchini-nlp · 2025-09-22T08:59:38Z

I will review the PR shortly

Kwai-Keye · 2025-09-22T09:04:45Z

@zucchini-nlp
Thanks.

zucchini-nlp

Thanks for the PR, I think we need to clean up a bit since most part of the model are identical to Qwen-VL series. Also, regarding the video processing, let's create a separate video processing class similar to Qwen2-VL

I reviewed most of it, will come back soon to finish reviewing

zucchini-nlp · 2025-09-22T09:25:27Z

@@ -0,0 +1,252 @@
+<!--Copyright 2025 The Kuai Keye Team and The HuggingFace Inc. team. All rights reserved.


just to make sure, is Kuai correct or should it be Kwai?

zucchini-nlp · 2025-09-22T09:26:23Z

+# Keye-VL-Preview
+[Keye-VL-Preview](https://huggingface.co/papers/2507.01949) is an 8-billion-parameter multimodal foundation model, excels in short-video understanding while maintaining robust general-purpose vision-language abilities through a comprehensive pre- and post-training process, including reinforcement learning and alignment.
+
+The abstract from the paper is the following:
+
+*While Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities on static images, they often fall short in comprehending dynamic, information-dense short-form videos, a dominant medium in today’s digital landscape. To bridge this gap, we introduce Kwai Keye-VL, an 8-billion-parameter multimodal foundation model engineered for leading-edge performance in short-video understanding while maintaining robust general-purpose vision-language abilities. The development of Keye-VL rests on two core pillars: a massive, high-quality dataset exceeding 600 billion tokens with a strong emphasis on video, and an innovative training recipe. This recipe features a four-stage pre-training process for solid vision-language alignment, followed by a meticulous two-phase post-training process. The first post-training stage enhances foundational capabilities like instruction following, while the second phase focuses on stimulating advanced reasoning. In this second phase, a key innovation is our five-mode “cold-start” data mixture, which includes “thinking”, “non-thinking”, “auto-think”, “think with image”, and high-quality video data. This mixture teaches the model to decide when and how to reason. Subsequent reinforcement learning (RL) and alignment steps further enhance these reasoning capabilities and correct abnormal model behaviors, such as repetitive outputs. To validate our approach, we conduct extensive evaluations, showing that Keye-VL achieves state-of-the-art results on public video benchmarks and remains highly competitive on general image-based tasks (Figure 1). Furthermore, we develop and release the KC-MMBench, a new benchmark tailored for real-world short-video scenarios, where Keye-VL shows a significant advantage. Comprehensive human evaluations also confirm that our model provides a superior user experience compared to other leading models of a similar scale. This paper details the architecture, data construction strategy, and training methodology of Keye-VL, offering valuable insights for building the next generation of MLLMs for the video era.*


preview is not available anymore, can delete

zucchini-nlp · 2025-09-22T09:27:08Z

    "qwen2_5_vl",
    "videollava",
    "vipllava",
+    "keyevl1_5",


to delete as well, instead we should change the checkpoint keys before release. This list is reserved for models we cannot change because they are released in the past

zucchini-nlp · 2025-09-22T09:33:01Z

+        image_token_id: int = None,
+        video_token_id: int = None,
+        attention_bias: bool = False,


The special image/video tokens also go in KeyeVL1_5Config. Then we can copy the text config from any existing LM's and reduce LOC :)

zucchini-nlp · 2025-09-22T10:22:22Z

+        spatial_merge_size = config.vision_config.spatial_merge_size
+        self.merge_kernel_size = (spatial_merge_size, spatial_merge_size)
+
+        self.hidden_size = self.vision_config.hidden_size * self.merge_kernel_size[0] * self.merge_kernel_size[1]


isn't used in forward, we can keep as hidden_size = my-value-here

zucchini-nlp · 2025-09-22T10:23:02Z

+        m1, m2 = self.merge_kernel_size
+        h_kernel, w_kernel = self.merge_kernel_size


zucchini-nlp · 2025-09-22T10:23:53Z

    "InformerConfig",
    "JukeboxPriorConfig",
    "JukeboxTokenizer",
+    "KeyeImageProcessor",


has to be fixed, we can't skip image processor docs

zucchini-nlp · 2025-09-22T10:24:19Z

+        # text_config={
+        #     "vocab_size": 99,
+        #     "hidden_size": 32,
+        #     "intermediate_size": 37,
+        #     "num_hidden_layers": 4,
+        #     "num_attention_heads": 4,
+        #     "num_key_value_heads": 2,
+        #     "hidden_act": "silu",
+        #     "max_position_embeddings": 512,


zucchini-nlp · 2025-09-22T10:24:53Z

@@ -0,0 +1,533 @@
+# Copyright 2025 The Keye Team and The HuggingFace Inc. team. All rights reserved.


we need test for processing, image processing and video processing classes as well

zucchini-nlp · 2025-09-22T10:40:34Z

+    _checkpoint_conversion_mapping = {
+        "^visual": "model.visual",
+        "^mlp_AR": "model.mm_projector",
+        r"^model(?!\.(language_model|visual|mm_projector))": "model.language_model",
+    }


to remove and set to {}

zucchini-nlp · 2025-09-22T10:41:40Z

+            device = pixel_values.device
+            pixel_values = pixel_values.type(self.visual.dtype)
+            pixel_values = pixel_values.unsqueeze(0)
+            assert torch.all(image_grid_thw[:, 0] == 1)
+            image_grid_thw = image_grid_thw.to(device)
+
+            total_patches = image_grid_thw.prod(dim=1)
+            width = torch.repeat_interleave(image_grid_thw[:, 2], total_patches)
+            cu_seqlens = total_patches.cumsum(0)
+
+            arange = torch.arange(cu_seqlens[-1], dtype=torch.long, device=device)
+            image_position_ids = arange - torch.repeat_interleave(cu_seqlens.to(device) - total_patches, total_patches)
+
+            width_position_ids = torch.remainder(image_position_ids, width)
+            height_position_ids = torch.div(image_position_ids, width, rounding_mode="floor")
+            cu_seqlens = F.pad(cu_seqlens, (1, 0), value=0).to(dtype=torch.int32, device=device)
+            width_position_ids = width_position_ids.to(device)
+            height_position_ids = height_position_ids.to(device)
+


we need to call self.get_image_features()

zucchini-nlp · 2025-09-22T10:41:55Z

+            device = pixel_values_videos.device
+            pixel_values_videos = pixel_values_videos.type(self.visual.dtype)
+            pixel_values_videos = pixel_values_videos.unsqueeze(0)
+            video_grid_thw = split_thw(video_grid_thw.squeeze(0)).to(device)
+
+            assert torch.all(video_grid_thw[:, 0] == 1)
+
+            total_patches = video_grid_thw.prod(dim=1)
+            width = torch.repeat_interleave(video_grid_thw[:, 2], total_patches)
+            cu_seqlens = total_patches.cumsum(0)
+            arange = torch.arange(cu_seqlens[-1], dtype=torch.long, device=device)
+            video_position_ids = arange - torch.repeat_interleave(cu_seqlens.to(device) - total_patches, total_patches)


same here, we need to call self.get_video_features()

zucchini-nlp · 2025-09-22T10:42:35Z

+            if input_ids is None:
+                image_mask = inputs_embeds == self.get_input_embeddings()(
+                    torch.tensor(self.config.image_token_id, dtype=torch.long, device=inputs_embeds.device)
+                )
+                image_mask = image_mask.all(-1)
+            else:
+                image_mask = input_ids == self.config.image_token_id
+


this is obtained from self.get_placeholder_mask similar to other VLMs (e,g. Qwen-VL)

zucchini-nlp · 2025-09-22T10:42:49Z

+            if input_ids is None:
+                video_mask = inputs_embeds == self.get_input_embeddings()(
+                    torch.tensor(self.config.video_token_id, dtype=torch.long, device=inputs_embeds.device)
+                )
+                video_mask = video_mask.all(-1)
+            else:
+                video_mask = input_ids == self.config.video_token_id
+
+            n_video_tokens = video_mask.sum().item()
+


same here, from self.get_placeholder_mask similar to other VLMs (e,g. Qwen-VL)

zucchini-nlp · 2025-09-22T10:46:48Z

+        assert torch.all(image_grid_thw[:, 0] == 1)
+        image_grid_thw = image_grid_thw.to(device)


we can delete these two lines

zucchini-nlp · 2025-09-22T10:48:15Z

+        total_patches = video_grid_thw.prod(dim=1)
+        width = torch.repeat_interleave(video_grid_thw[:, 2], total_patches)
+        cu_seqlens = total_patches.cumsum(0)
+        arange = torch.arange(cu_seqlens[-1], dtype=torch.long, device=device)
+        video_position_ids = arange - torch.repeat_interleave(cu_seqlens.to(device) - total_patches, total_patches)
+
+        width_position_ids = torch.remainder(video_position_ids, width)
+        height_position_ids = torch.div(video_position_ids, width, rounding_mode="floor")
+        cu_seqlens = F.pad(cu_seqlens, (1, 0), value=0).to(dtype=torch.int32, device=device)
+        width_position_ids = width_position_ids.to(device)
+        height_position_ids = height_position_ids.to(device)
+


these lines are identical for video and image. I think we can move it under VisionModel.forward so that users can pass directly the image pixels and let the model handle the rest

zucchini-nlp · 2025-09-22T10:50:52Z

+            Examples:
+                Temporal (Time): 3 patches, representing different segments of the video in time.
+                Height: 2 patches, dividing each frame vertically.
+                Width: 2 patches, dividing each frame horizontally.
+                We also have some important parameters:
+                fps (Frames Per Second): The video's frame rate, set to 1. This means one frame is processed each second.
+                tokens_per_second: This is a crucial parameter. It dictates how many "time-steps" or "temporal tokens" are conceptually packed into a one-second interval of the video. In this case, we have 25 tokens per second. So each second of the video will be represented with 25 separate time points. It essentially defines the temporal granularity.
+                temporal_patch_size: The number of frames that compose one temporal patch. Here, it's 2 frames.
+                interval: The step size for the temporal position IDs, calculated as tokens_per_second * temporal_patch_size / fps. In this case, 25 * 2 / 1 = 50. This means that each temporal patch will be have a difference of 50 in the temporal position IDs.
+                input_ids: [V V V V V V V V V V V V T T T T T], here V is for vision.
+                vision temporal position_ids: [0, 0, 0, 0, 2, 2, 2, 2, 4, 4, 4, 4]
+                vision height position_ids: [0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5]
+                vision width position_ids: [0, 1, 0, 1, 2, 3, 2, 3, 4, 5, 4, 5]
+                text temporal position_ids: [101, 102, 103, 104, 105]
+                text height position_ids: [101, 102, 103, 104, 105]
+                text width position_ids: [101, 102, 103, 104, 105]


can you comment what is the diff from qwen2-vl and update the docs?

zucchini-nlp · 2025-09-24T08:59:37Z

Ping me again when it is ready for review, I will unsubscribe myself from notifications

Kwai-Keye · 2025-10-20T12:11:52Z

@zucchini-nlp
Hi, Could you please take a look now? We've revised most of the code issues you pointed out, but some didn't quite fit our model, so we didn't make those changes.

github-actions · 2025-10-20T12:44:05Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, keye_vl_1_5

Kwai-Keye · 2025-10-21T06:19:55Z

@zucchini-nlp

zucchini-nlp · 2025-10-21T09:19:46Z

@Kwai-Keye oke, today or tomorrow I will try to take a look. I'll also be on vacation starting Thursday until November, so the review process will be a bit slower

In the meantime, can you take a look at failing tests?

lingzhixin and others added 28 commits August 25, 2025 00:03

update tests

8af7615

c

7855703

update modular

6a21816

add KeyeVisionModel

c9748d6

add KeyeVisionModel

99066f0

add KeyeVisionModel

55e33c9

update test ignore

edcf5e3

doc

8d5df6f

doc

04906c4

format

8ce68b6

ignore keyevision

33e85a9

Merge branch 'main' into dev2

c49f889

add __all__ processor

dd33837

add __all__ videoprocessor

802e429

add image process doc

fe1a747

del max_window_layers

33f5b95

del max_window_layers2

a1a2c73

del max_window_layers

eac88fd

del max_window_layers

0ae7046

del max_window_layers

540b28e

del max_window_layers

ead226e

del max_window_layers

02f9678

fix test

b56be32

fix test

67b8dea

request for merging keye-vl-8b-1.5 code

cb4932d

redo logger

21f8c50

new ling

f1009fc

new line

cbb7a87

zucchini-nlp reviewed Sep 22, 2025

View reviewed changes

remove comments

07a9f73

zucchini-nlp reviewed Sep 22, 2025

View reviewed changes

Kwai-Keye added 6 commits September 22, 2025 19:00

remove m1, m2

bbbca92

remove self.hidden_size

326a5e0

remove keye-vl-8b-preview docs

c890251

remove map parameters

3a210dc

modified

6e69558

modified

e61f426

Kwai-Keye added 4 commits September 25, 2025 20:13

add video processor

51eafc1

code refine

df19747

code refine

e6ce1f6

Merge branch 'main' into keye-vl-8b-1.5

f25d270

Merge branch 'main' into keye-vl-8b-1.5

5683fb7

Kwai-Keye requested a review from zucchini-nlp October 21, 2025 09:13

evalstate mentioned this pull request Apr 29, 2026

Cumulative feature and defect updates from recent Transformers PRs evalstate/transformers#42

Open

		@@ -0,0 +1,252 @@
		<!--Copyright 2025 The Kuai Keye Team and The HuggingFace Inc. team. All rights reserved.

		m1, m2 = self.merge_kernel_size
		h_kernel, w_kernel = self.merge_kernel_size

		@@ -0,0 +1,533 @@
		# Copyright 2025 The Keye Team and The HuggingFace Inc. team. All rights reserved.

		assert torch.all(image_grid_thw[:, 0] == 1)
		image_grid_thw = image_grid_thw.to(device)

Uh oh!

Conversation

Kwai-Keye commented Sep 22, 2025

Model Upgrade: Keye-VL-1.5-8B

Overview

Key Changes

1. Model Architecture

2. Documentation

3. Code Optimization

Impact

Usage

Notes

Uh oh!

Kwai-Keye commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zucchini-nlp commented Sep 22, 2025

Uh oh!

Kwai-Keye commented Sep 22, 2025

Uh oh!

zucchini-nlp commented Sep 22, 2025

Uh oh!

zucchini-nlp commented Sep 22, 2025

Uh oh!

Kwai-Keye commented Sep 22, 2025

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kwai-Keye commented Sep 22, 2025 •

edited

Loading