Summary

I am building a high-performance image processing pipeline using Node.js and C++ (libvips). The goal is to stream huge raw pixel data (generated by CanvasKit/Skia, potentially >100GB) from Node.js to a C++ Addon via a Pipe/FIFO, and then encode it to PNG/DeepZoom.

We cannot use Node.js Buffers to pass data to sharp because of the 4GB Buffer limit in V8. Therefore, we are using a named pipe (FIFO) to transfer data to a C++ addon using libvips.

The Problem

Currently, there seems to be no easy way to load raw pixel data (without a header) from a VSource (pipe/stream).

1. Missing rawload_source: There is vips_rawload for files, but there seems to be no corresponding rawload_source for streams where we can explicitly specify width, height, bands, and format.

2. PAM/PPM detection issues: As a workaround, we tried prepending a PAM (NetPBM) Header to the stream in Node.js and then calling VImage::new_from_source(source, "", ...). However, it seems new_from_source struggles to detect the PAM format from a non-seekable pipe source in some cases, or requires specific access options that are tricky to get right.

Proposed Solution

Option A (Preferred): Implement rawload_source It would be amazing if we could explicitly tell libvips the metadata of the incoming stream, avoiding the need to inject fake headers in the producer.

// Hypothetical API
VImage image = VImage::new_from_source(source, "rawload_source",
    VImage::option()
        ->set("width", 10000)
        ->set("height", 10000)
        ->set("bands", 4)
        ->set("format", VIPS_FORMAT_UCHAR)
        ->set("access", VIPS_ACCESS_SEQUENTIAL)
);

Option B: Improve ppmload_source detection Ensure that VImage::new_from_source can reliably detect a PAM/PPM header even if the source is a non-seekable pipe (FIFO).

Use Case / Reproduction Context

• Producer: Node.js Worker (CanvasKit) writing RGBA chunks to a FIFO.

• Consumer: C++ N-API Addon reading from the FIFO.

• Constraint: Node.js cannot hold the full image in memory (OOM > 4GB).

Current Workaround (Manual C++ Loop): We currently have to use libpng directly in C++, manually reading width * 4 bytes from the FD and writing rows, which bypasses libvips's powerful pipeline features (like SIMD flatten/embed).

Example C++ Code (What we are trying to do)

// descriptor is a file descriptor for a FIFO pipe
VSource source = VSource::new_from_descriptor(descriptor);

// We want to do this (load raw stream):
VImage image = VImage::new_from_source(source, "rawload_source", 
    VImage::option()
        ->set("width", width)
        ->set("height", height)
        ->set("bands", 4)
); 
// Error: class "rawload_source" not found

// Or this (load PAM stream):
// The stream starts with "P7\nWIDTH 1000..."
VImage image = VImage::new_from_source(source, "", 
    VImage::option()->set("access", VIPS_ACCESS_SEQUENTIAL)
);
// Sometimes Error: source is not in a known format (on pipes)

Additional Note: I have actually experimented with the source code, and it seems feasible to implement support for PAM headers. I noticed that the current codebase already supports PPM. However, my specific use case requires transparency (alpha channel), which is why full PAM support is necessary (as standard PPM typically excludes alpha).

Thanks for the great library!