deprecate Iter::new, migrate its usage in moonbitlang/core
#3050
Conversation
| arr.clear() | ||
| inspect(iter.to_array(), content="[]") | ||
| } | ||
|
|
There was a problem hiding this comment.
this test pushes elements into the array during iteration, this is undefined behavior, and the actual behavior changes in the new implementation. In the worst case modifying data structure while iterating can crash the program, so we can provide no guarantee is this case
| Node(children, _) as t if curr_index < children.length() => { | ||
| let child = children.unsafe_get(curr_index) | ||
| parents.push((t, curr_index + 1)) | ||
| curr_tree = child |
There was a problem hiding this comment.
this is a bug in the previous implementation. After this PR, tests that previously use internal iterator now use external iterator under the hood, triggering the bug
Pull Request Test Coverage Report for Build 2136Warning: This coverage report may be inaccurate.This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.
Details
💛 - Coveralls |
2450f5e to
07b1d8b
Compare
07b1d8b to
bf837ed
Compare
There was a problem hiding this comment.
Pull request overview
This PR deprecates Iter::new and Iter2::new as part of the migration from internal iterators to external iterators. The migration converts all usages of Iter::new in the core library to use external iterator types (Iterator and Iterator2) which are then converted to internal iterators via .iter() and .iter2() methods.
- Adds deprecation markers to
Iter::newandIter2::newwith helpful migration instructions - Migrates all internal iterator implementations in moonbitlang/core to use external iterators
- Updates tests to handle the breaking behavior change where iterators can only be traversed once
Reviewed changes
Copilot reviewed 40 out of 40 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| builtin/iter.mbt | Adds deprecation marker to Iter::new |
| builtin/iter2.mbt | Adds deprecation marker to Iter2::new |
| builtin/iterator.mbt | Implements Iterator::iter(), Iterator::iter2(), and Iterator2::iter2() using direct Iter() constructor |
| builtin/array.mbt | Migrates array iterators to use external iterators |
| builtin/arrayview.mbt | Migrates arrayview iterators to use external iterators |
| builtin/bytes.mbt | Migrates bytes iterators to use external iterators |
| builtin/bytesview.mbt | Migrates bytesview iterators to use external iterators |
| builtin/fixedarray.mbt | Migrates fixedarray iterators and adds iterator2() method |
| builtin/string.mbt | Migrates string iterators to use external iterators |
| builtin/stringview.mbt | Migrates stringview iterators to use external iterators |
| builtin/string_methods.mbt | Migrates StringView::split() to use external iterators |
| builtin/linked_hash_map.mbt | Migrates linked hash map iterators to use external iterators |
| deque/deque.mbt | Migrates deque iterators to use external iterators |
| float/methods.mbt | Migrates Float::upto() and Float::until() to use external iterators |
| hashmap/utils.mbt | Migrates hashmap iterators including keys() and values() to use external iterators |
| hashset/hashset.mbt | Migrates hashset iterators to use external iterators |
| immut/array/array.mbt | Migrates immutable array iterators to use external iterators |
| immut/array/tree.mbt | Removes old internal iterator, keeps external iterator implementation |
| immut/hashmap/HAMT.mbt | Migrates immutable hashmap iter2() to use external iterators |
| immut/priority_queue/priority_queue.mbt | Migrates immutable priority queue iterators to use external iterators |
| immut/sorted_map/utils.mbt | Migrates immutable sorted map iterators to use external iterators |
| immut/sorted_set/generic.mbt | Migrates immutable sorted set iterators to use external iterators |
| list/list.mbt | Migrates list iterators to use external iterators |
| priority_queue/priority_queue.mbt | Migrates priority queue iterators to use external iterators |
| queue/queue.mbt | Migrates queue iterators to use external iterators |
| set/linked_hash_set.mbt | Migrates linked hash set iterators to use external iterators |
| sorted_map/map.mbt | Migrates sorted map iterators including keys(), values(), and range() to use external iterators |
| sorted_set/set.mbt | Migrates sorted set iterators including range() to use external iterators |
| quickcheck/arbitrary.mbt | Updates arbitrary implementations to create Iterator first, then convert to Iter |
| bigint/bigint_js.mbt | Changes BigInt::limbs() to return Array[UInt] instead of Iter[UInt] |
| bigint/bigint_nonjs.mbt | Changes BigInt::limbs() to return Array[UInt] instead of Iter[UInt] |
| bigint/bigint_wbtest.mbt | Updates tests to compare arrays directly instead of calling .to_array() |
| builtin/iter_test.mbt | Adds deprecation warning suppression to tests using Iter::new |
| builtin/iter2_test.mbt | Adds deprecation warning suppression to tests using Iter2::new |
| builtin/array_test.mbt | Removes test showing old internal iterator behavior, updates nth test to use closures |
| deque/deque_test.mbt | Updates test to avoid reusing iterators |
| immut/hashmap/HAMT_test.mbt | Updates tests to use closures for creating fresh iterators, adds deprecation warning suppression |
| string/view_test.mbt | Updates tests to call .iter() multiple times instead of reusing stored iterator |
| builtin/pkg.generated.mbti | Updates generated type signatures for deprecations and new methods |
| quickcheck/pkg.generated.mbti | Adds Arbitrary implementation for Iterator[X] |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| } else if cmp_key_low < 0 { | ||
| // `value < low`, the right subtree and the value itself | ||
| // should not be visited | ||
| continue left |
There was a problem hiding this comment.
The logic here appears incorrect. When cmp_key_low < 0 (meaning key < low), the code continues with the left subtree. However, if key < low, then all keys in the left subtree will also be < low since this is a binary search tree. The correct action should be to continue with the right subtree instead. The comment also confirms this: it says "the right subtree and the value itself should not be visited" but the code visits the left subtree.
| if cmp_key_high > 0 { | ||
| // `value > high`, the left subtree and the value itself | ||
| // should not be visited | ||
| continue right |
There was a problem hiding this comment.
The logic here appears incorrect. When cmp_key_high > 0 (meaning value > high), the code continues with the right subtree. However, if value > high, then all values in the right subtree will also be > high since this is a binary search tree. The correct action should be to continue with the left subtree instead. The comment also confirms this: it says "the left subtree and the value itself should not be visited" but the code visits the right subtree.
| } else if cmp_key_low < 0 { | ||
| // `value < low`, the right subtree and the value itself | ||
| // should not be visited | ||
| continue left |
There was a problem hiding this comment.
The logic here appears incorrect. When cmp_key_low < 0 (meaning value < low), the code continues with the left subtree. However, if value < low, then all values in the left subtree will also be < low since this is a binary search tree. The correct action should be to continue with the right subtree instead. The comment also confirms this: it says "the right subtree and the value itself should not be visited" but the code visits the left subtree.
| if cmp_key_high > 0 { | ||
| // `value > high`, the left subtree and the value itself | ||
| // should not be visited | ||
| continue right |
There was a problem hiding this comment.
The logic here appears incorrect. When cmp_key_high > 0 (meaning key > high), the code continues with the right subtree. However, if key > high, then all keys in the right subtree will also be > high since this is a binary search tree. The correct action should be to continue with the left subtree instead. The comment also confirms this: it says "the left subtree and the value itself should not be visited" but the code visits the right subtree.
This PR deprecates
Iter::new&Iter2::new, and migrates their usage inmoonbitlang/core. This is the next step in the process of migrating from internal iterator to external iterator, starting from #2851.Background
We have decided to migrate from internal iterator to external iterator for various reasons:
for .. inbody can performasyncoperationszipandequalOne challenge here is incremental migration: we don't want to break user code hard during the transition from internal iterator to external iterator.
The migration plan
Our migration plan from internal iterator to external iterator is as follows:
IteratorandIterator2(add experimental external iterator type #2851, addIterator2wrapper for two-variablefor .. inloop #2859)for .. inloop use external iterator by default, fallback to internal iterator only when no.iterator()method is foundIter::newandIter2::new. This is the only internal iterator API that external iterator cannot implement. Fortunately, end users do not invokeIter::newdirectly in general. People who do usesIter::new, such as author of a data structure package, should create an external iteratorIterator[_]first, and convert the external iterator to internal iterator viaIterator::iter.Iter::newis completely removed,Iterwill become an alias ofIterator, andIterator::iterwill become the identity function. This change should be non-breaking, as all methods onIterare supported byIterator. During this stage,Iterator::itermay get deprecated to cleanup redundant APIsWhat users should do to adapt this PR
Iter::newdirectly, this PR should have no effectIter::new, such as authors of data structure packages, should replaceIter::newwithIterator::iter(xxx), wherexxxis an external iterator with the same semantic as the previous internal one created viaIter::new.iter()methods used byfor .. inloops, a corresponding external iterator version called.iterator()should be addedCreating external iterators is harder than creating internal iterators, especially when the data structure is complex. If you are not sure how to create an external iterator for your own data structure, changes in this PR is a good reference:
ArrayView::iteratorinbuiltin/arrayview.mbt,Map::iteratorinbuiltin/linked_hash_map.mbt, andStringView::iteratorinbuiltin/stringview.mbt@sorted_map.SortedMap::iteratorinsorted_map/map.mbt. You can also find a more complex example in@sorted_map.SortedMap::range, which has an addition range limitation on iterated elements@immut/array.Tree::iteratorinimmut/array/tree.mbtOne breaking behavior change
Although this PR should be non-breaking in terms of type signature, it introduces a breaking behavior change that is hard to avoid. Previously, the
Iter[_]type, and various.iter()methods that create internal iterators, are immutable. The iterator can be iterated from the beginning multiple times. Iterating a iterator multiple times will result in duplicated computation, and side effects in the iterator will get triggered multiple times. So although possible, the pattern of traversing an iterator multiple times is discouraged for both performance and robustness reasons.After this PR,
.iter()methods inmoonbitlang/coreare implemented via the external iterator typeIterator[_], which is mutable and can be traversed only once. Subsequent traversal will continue from where the last traversal ends, typically the end of the whole iterator. So the behavior of code that traverses a iterator multiple times will change.