@yuanyao-nv Please review and verify that the changes are desired and accurate. Thanks!

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 57.46%. Comparing base (5ab1e42) to head (00668ee).
Report is 6 commits behind head on main.

✅ All tests successful. No failed tests found.

@@            Coverage Diff             @@
##             main    #6640      +/-   ##
==========================================
+ Coverage   57.44%   57.46%   +0.02%     
==========================================
  Files         507      507              
  Lines       31612    31584      -28     
  Branches     3058     3046      -12     
==========================================
- Hits        18158    18151       -7     
+ Misses      12610    12607       -3     
+ Partials      844      826      -18     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Thanks for the clarification.

The int4 packing description clarifies packing 2 int4 into 1 byte, but it doesn't clarify whether such bytes are then packed into int32 (4 each except for the last one) or each byte encoded as a single int32 or whether both are allowed?

I guess the same question applies to float8 as well, but the previous description seems to suggest that a single float8 is represented as a single int32 ...

I understand it as each byte encoded as a single int32. Please feel free to suggest wording. @yuanyao-nv do you have comments or further clarifications?

Thanks for the clarification.

The int4 packing description clarifies packing 2 int4 into 1 byte, but it doesn't clarify whether such bytes are then packed into int32 (4 each except for the last one) or each byte encoded as a single int32 or whether both are allowed?

I guess the same question applies to float8 as well, but the previous description seems to suggest that a single float8 is represented as a single int32 ...

I agree it's not clear from the proto docstring whether bytes are then packed into int32. But looking at the helper function implementations (eg numpy_helper.py::_to_array()) each int32_data only holds one byte's worth of data.

Thanks for the clarification.
The int4 packing description clarifies packing 2 int4 into 1 byte, but it doesn't clarify whether such bytes are then packed into int32 (4 each except for the last one) or each byte encoded as a single int32 or whether both are allowed?
I guess the same question applies to float8 as well, but the previous description seems to suggest that a single float8 is represented as a single int32 ...

I agree it's not clear from the proto docstring whether bytes are then packed into int32. But looking at the helper function implementations (eg numpy_helper.py::_to_array()) each int32_data only holds one byte's worth of data.

Wondering about its efficiency. Looking at this description of protobuf encoding, I assume that this ends up getting represented as a LEN encoding of varints, is that right? This means that a byte-value <= 127 is encoded using 1 byte, but a value >= 128 will use 2 bytes. Assuming both are equally likely, we could end up using 1.5 bytes for each byte (2 INT4s) on average. But if we pack 4 bytes (each with two INT4s) into an INT32, then that int32 value will likely need 5 bytes (most of the time) or 4 bytes in the varint form.

Ok, so there is a difference (may be 5 bytes vs 6 bytes to represent 8 INT4s). Not huge, but 15% difference could be significant for GB-size tensors. So, maybe we should allow both?

(What would likely be even better is packing 7 INT4s into an int32, which can always be represented using 4 bytes ... but that would impact the packing/unpacking logic more significantly than the above.)

But, looking into it more, I guess raw_data would be the best anyway for representing the packed INT4? If so, maybe we don't have to worry about this, but remind users that raw_data preferable for this case

From the protobuf docs, https://protobuf.dev/programming-guides/encoding/#packed yes it seems reasonable to say that raw_data is most reasonable, because that gives us 4bytes/8 int4s right? So that should be the most efficient?

I agree with Rama's assessment as well, that raw should be more efficient since no extra byte is needed.
And it seems like the only way ONNX exposes control over the use of int32_data vs raw_data is by setting the raw=True flag in make_tensor()?

I agree with Rama's assessment as well, that raw should be more efficient since no extra byte is needed. And it seems like the only way ONNX exposes control over the use of int32_data vs raw_data is by setting the raw=True flag in make_tensor()?

That's reasonable. There's only one toggle in the helper apis. I think simplicity helps here.

Please feel free to suggest wording. I tried to rephrase but wasn't happy with what I got. I was trying to convey that the all values should be reinterpreted as int32 without further packing.

I used chat gpt's help in rephrasing it

@yuanyao-nv take a final look? Thanks!

LGTM. Thanks!