Crate iter_merge 

A high-performance iterator merging library.

This crate provides MergeIter and a builder API to merge items from many iterators according to a comparator. By default, it performs a min-merge by Ord, breaking ties by insertion order. It’s no_std, with Vec-requiring functions behind the alloc feature.

Quick start

use iter_merge::merge;

let a = vec![1, 3, 5];
let b = vec![2, 4, 6];
let merged = merge([a, b]).into_vec();
assert_eq!(merged, vec![1, 2, 3, 4, 5, 6]);

Note that only the next item in each iterator is considered. If the input iterators are not sorted, the result won’t be sorted either:

use iter_merge::merge;

let merged = merge([vec![2, 1, 5], vec![4, 3, 6]]).into_vec();
assert_eq!(merged, vec![2, 1, 4, 3, 5, 6]);

Array storage

use core::pin::pin;

use iter_merge::{ArrayStorage, MergeIter};

// First create a storage with some fixed capacity
let mut storage = ArrayStorage::<2, _>::from_iter([[1, 3, 5]]);
// You can modify it by adding iterators of the same type
storage.push([2, 4, 6]);

// In order to construct a MergeIter you need to pin that storage.
// You won't be able to modify it once you've pinned it.
let storage = pin!(storage);
let mut merge = storage.build();
assert!(merge.eq([1, 2, 3, 4, 5, 6]));

Custom comparator

Use the builder to specify custom ordering (min/max by comparison function, by key, or by Ord). Implement a custom comparator for even more control.

use iter_merge::{MergeIter, VecStorage};

// Merge by descending absolute value
let res = VecStorage::from_iter([vec![-3_i32, -1], vec![2, -2]])
    .into_builder()
    .max_by_key(|&x| x.abs())
    .build()
    .into_vec();
assert_eq!(res, vec![-3, 2, -2, -1]);

Peeking and conditional consumption

MergeIter provides the same methods as iter::Peekable:

use iter_merge::merge;

let mut it = merge([vec![1, 1, 2], vec![1, 3]]);
assert_eq!(it.peek(), Some(&1));
// consume all 1s
while let Some(1) = it.next_if_eq(&1) {}
assert_eq!(it.next(), Some(2));

Performance

It’s 1.45-1.65x faster than itertools::kmerge in my benchmarks and scales as O(item_count + log₂(iterator_count))

Benchmark details

Benchmarks were run on a fresh Ubuntu install on dedicated Intel E-1270v3 (4 cores; 3.5GHz) server with maximal optimizations (opt-level=3, lto=true, codegen-units=1, target-cpu=native)

Iterators were over random u64's, and iterator itself had a size of 32 bytes long (for reference Vec::into_iter has the same size). Expect bigger performance improvements compared with itertools::kmerge when merging larger iterators and/or iterators over large values, since the key difference between these libraries is the use of pointer indirection in our heap, so we don't need to move the pair of (peeked_item, iterator), only pointers to them.

For detailed performance numbers in various scenarios, run the included benchmarks in the benches/ directory.

Crate Features

• alloc: Enables heap-allocated storage with VecStorage and methods like MergeIter::into_vec

Re-exports

pub use merge_iter::MergeIter;

pub use storage::ArrayStorage;

pub use storage::VecStorage;

Modules

comparators

Defines comparators for MergeIter

internal

Internal implementation details of this library.

merge_iter

Implementation of MergeIter

storage

Storage backends

Functions

merge

Constructs a new MergeIter with default parameters:

merge_by

Constructs a new MergeIter with default parameters:

merge_by_key

Constructs a new MergeIter with default parameters: