Buffer protocol proposal #3261

mrkn · 2020-06-26T09:06:21Z

[Feature #14722]

buffer.c

include/ruby/buffer.h

kou · 2020-07-02T07:41:03Z

include/ruby/buffer.h

+    /* The original object that have the memory exported via this buffer.
+     * The consumer of this buffer has the responsibility to call rb_gc_mark
+     * for preventing this obj collected by GC.  */
+    VALUE obj;


Is this needed?
The consumer calls rb_obj_get_buffer(obj, ...). So it must know the obj.

Some committers suggest to let RubyVM manage the reference of obj until its buffer will be released.
This obj may be useful for such a purpose.

include/ruby/buffer.h

dsisnero · 2020-08-25T20:43:39Z

In python the memoryview constructor is used to build a memoryvieew object that references the original object. The original object implements the buffer protocol.

s = b"abcdefgh"
view = memoryview(s)
view[1]
98
limited = view[1:3]
limited
<memory at 0x7fca18b8d460>
bytes(view[1:3])
b'bc'

I think changing the name to memory view for the underlying buffer is confusing.
https://julien.danjou.info/high-performance-in-python-with-zero-copy-and-the-buffer-protocol/

mrkn · 2020-08-25T22:15:38Z

@dsisnero Our goal isn’t completely following Python.

mrkn · 2020-09-01T07:15:45Z

memory_view.c

+    bool alignment = false;
+
+    const char *p = format;
+    if (*p == '|') {  // alginment specifier


I employed | for specifying element aligning mode.

mrkn · 2020-09-01T07:27:09Z

@akr Could you please review this?

memory_view.c

test/ruby/test_memory_view.rb

akr · 2020-09-10T17:57:38Z

memory_view.c

+        *members = buf;
+        *n_members = len;
+    }
+


It seems no padding after the last component.
I guess that the size of "lc" would be 5.
But it should be 8 to align next item.

memory_view.c

akr · 2020-09-10T18:06:51Z

memory_view.c

+    uint8_t *ptr = view->data;
+
+    if (view->ndim == 1) {
+        return ptr + indices[0] * view->item_size;


view->strides is ignored.
Is it intentional?

No, view->strides shouldn't be ignored. I'll fix it.

akr · 2020-09-10T18:12:47Z

memory_view.c

+    if (view->strides == NULL) {
+        // row-major contiguous array
+        for (i = 0; i < view->ndim; ++i) {
+            ptr += indices[i] * view->shape[i] * view->item_size;


I guess view->shape[i] should be view->shape[i+1] * ... * view->shape[ndim-1].
Not tested.

Exactly. I should write a test of this function.

akr · 2020-09-10T18:30:40Z

ext/-test-/memory_view/memory_view.c

+
+#undef DEF_ALIGNMENT_CONST
+
+    exported_objects = rb_hash_new();


I think that string content-based comparison is not appropriate here.
It may needs compare_by_identity.

akr · 2020-09-10T18:39:21Z

ext/-test-/memory_view/memory_view.c

+
+    if (view.shape) {
+        VALUE shape = rb_ary_new_capa(view.ndim);
+        rb_hash_aset(hash, sym_shape, shape);


I feel this is incomplete.
shape is always an empty array.
strides and sub_offsets has same issue.

akr · 2020-09-10T18:48:27Z

include/ruby/memory_view.h

+    /* 1 for readonly memory, 0 for writable memory. */
+    int readonly;
+
+    /* A string to describe the format of an element, or NULL for unsigned byte.


It seems "item" and "element" are same meaning.
How about using one word for one meaning?

mrkn · 2020-09-25T10:59:10Z

The CI failure isn’t due to this pull request.

mrkn · 2020-09-25T11:39:11Z

Oh, I rewrote the merge commit message but I don't know why it was rolled back..
The message I wrote was:

Memory view interface (#3261)

The memory view interface is a C-API set to exchange a raw memory area,
such as a numeric array and a bitmap image, between extension libraries.
The extension libraries can share also the metadata of the memory area
that consists of the shape, the element format, and so on.
Using these kinds of metadata, the extension libraries can share even
a multidimensional array appropriately.
This feature is designed by referring to Python's buffer protocol.

[Feature #13767]
[Feature #14722]

* Add buffer protocol * Modify for some review comments * Per-object buffer availability * Rename to MemoryView from Buffer and make compilable * Support integral repeat count in memory view format * Support 'x' for padding bytes * Add rb_memory_view_parse_item_format * Check type in rb_memory_view_register * Update dependencies in common.mk * Add test of MemoryView * Add test of rb_memory_view_init_as_byte_array * Add native size format test * Add MemoryView test utilities * Add test of rb_memory_view_fill_contiguous_strides * Skip spaces in format string * Support endianness specifiers * Update documentation * Support alignment * Use RUBY_ALIGNOF * Fix format parser to follow the pack format * Support the _ modifier * Parse count specifiers in get_format_size function. * Use STRUCT_ALIGNOF * Fix test * Fix test * Fix total size for the case with tail padding * Fix rb_memory_view_get_item_pointer * Fix rb_memory_view_parse_item_format again

mrkn marked this pull request as draft June 26, 2020 09:06

mrkn force-pushed the buffer_protocol branch 2 times, most recently from c1dc40a to 7d5e5e6 Compare June 29, 2020 12:35

kou reviewed Jul 2, 2020

View reviewed changes

mrkn force-pushed the buffer_protocol branch 2 times, most recently from 88d54af to 8b8ecfb Compare July 19, 2020 14:46

shyouhei reviewed Jul 20, 2020

View reviewed changes

include/ruby/buffer.h Outdated Show resolved Hide resolved

mrkn force-pushed the buffer_protocol branch 3 times, most recently from 829c6f2 to 07fd1a9 Compare August 24, 2020 02:25

mrkn force-pushed the buffer_protocol branch 5 times, most recently from 325f8d5 to 604949f Compare September 1, 2020 07:07

mrkn commented Sep 1, 2020

View reviewed changes

mrkn requested a review from akr September 1, 2020 07:26

mrkn marked this pull request as ready for review September 1, 2020 07:37

mrkn mentioned this pull request Sep 10, 2020

WIP: Add buffer protocol ruby/fiddle#18

Closed

mrkn changed the title ~~WIP: Buffer protocol proposal~~ Buffer protocol proposal Sep 10, 2020

akr reviewed Sep 11, 2020

View reviewed changes

kou mentioned this pull request Sep 13, 2020

Implement buffer protocol for Ruby ruby/fiddle#17

Closed

mrkn force-pushed the buffer_protocol branch 5 times, most recently from e8a67fe to 657b021 Compare September 20, 2020 09:24

mrkn added 23 commits September 25, 2020 17:00

Support 'x' for padding bytes

c97f9b8

Add rb_memory_view_parse_item_format

12dd500

Check type in rb_memory_view_register

11426bc

Update dependencies in common.mk

ccbb0c9

Add test of MemoryView

a3b750e

Add test of rb_memory_view_init_as_byte_array

aa6aaf1

Add native size format test

08c54b7

Add MemoryView test utilities

0166ec6

Add test of rb_memory_view_fill_contiguous_strides

51ee869

Skip spaces in format string

934edd7

Support endianness specifiers

1b842a1

Update documentation

39a2a3f

Support alignment

01c3e0f

Use RUBY_ALIGNOF

71800fc

Fix format parser to follow the pack format

bdb94a0

Support the _ modifier

72fee2b

Parse count specifiers in get_format_size function.

3d9bdce

Use STRUCT_ALIGNOF

bff9105

Fix test

cfad9fe

Fix test

be8a1ce

Fix total size for the case with tail padding

13de9f7

Fix rb_memory_view_get_item_pointer

184fbef

Fix rb_memory_view_parse_item_format again

d264358

mrkn force-pushed the buffer_protocol branch from 2f0d906 to d264358 Compare September 25, 2020 08:26

mrkn merged commit 890bc2c into ruby:master Sep 25, 2020

mrkn deleted the buffer_protocol branch September 25, 2020 11:32

junaruga mentioned this pull request Jul 4, 2021

Installation of fiddle failing on Ruby 2.7.3 ruby/fiddle#84

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Buffer protocol proposal #3261

Buffer protocol proposal #3261

mrkn commented Jun 26, 2020 •

edited

Loading

kou Jul 2, 2020

mrkn Jul 22, 2020

dsisnero commented Aug 25, 2020

mrkn commented Aug 25, 2020

mrkn Sep 1, 2020

mrkn commented Sep 1, 2020

akr Sep 10, 2020

akr Sep 10, 2020

mrkn Sep 24, 2020

akr Sep 10, 2020

mrkn Sep 24, 2020

akr Sep 10, 2020

akr Sep 10, 2020

akr Sep 10, 2020

mrkn commented Sep 25, 2020

mrkn commented Sep 25, 2020 •

edited

Loading

Buffer protocol proposal #3261

Buffer protocol proposal #3261

Conversation

mrkn commented Jun 26, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsisnero commented Aug 25, 2020

mrkn commented Aug 25, 2020

Choose a reason for hiding this comment

mrkn commented Sep 1, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrkn commented Sep 25, 2020

mrkn commented Sep 25, 2020 • edited Loading

mrkn commented Jun 26, 2020 •

edited

Loading

mrkn commented Sep 25, 2020 •

edited

Loading