fix test sharding #156768
fix test sharding #156768
Conversation
Windows build_testsbringup: true from Windows build_tests
There was a problem hiding this comment.
Changed my mind, did you check the failures here? https://ci.chromium.org/ui/p/flutter/builders/luci.flutter.staging/Windows%20build_tests_8_8
From this: |
|
Let T be the number of tests. Let S be the number of shards. It looks like for some quantities of T and S, too many tests are assigned to each shard. The formula is tests per shard = ceil(T / S). In So shards 1-7 are responsible for tests 1-25 (27 clamped down to 25). They still are. Shard 8 is responsible the next 4 tests starting with 29, which doesn't make sense. I think we need a smarter distribution formula. Some shards need to be assigned 3 tests, some need to be assigned 4. |
|
code: Line 545 in 6ee200f |
54a239d to
6e812a5
Compare
bringup: true from Windows build_tests
dev/bots/utils.dart
Outdated
| // While there exists a closed formula figuring out the range of tests this | ||
| // shard is resposible for, modeling this as a simulation of distributing | ||
| // items equally into buckets is more intuitive. | ||
| // | ||
| // A bucket represents how many tests a shard should be allocated. | ||
| final List<int> buckets = List<int>.filled(total, 0); | ||
| // First, allocate an even amount of items to each bucket. | ||
| for (int i = 0; i < buckets.length; i++) { | ||
| buckets[i] = (tests.length / total).floor(); | ||
| } | ||
| // For the N leftover items, put one into each of the first N buckets. | ||
| final int remainingItems = tests.length % buckets.length; | ||
| for (int i = 0; i < remainingItems; i++) { | ||
| buckets[i] += 1; | ||
| } | ||
|
|
||
| // Lastly, compute the indices of the items in buckets[index]. | ||
| final int numberOfItemsInPreviousBuckets = index == 0 ? 0 : buckets.sublist(0, index - 1).sum; | ||
| final int start = (numberOfItemsInPreviousBuckets + 1) - 1; | ||
| final int end = (start + buckets[index - 1]) - 1; |
There was a problem hiding this comment.
The formula for directly computing start and end would look something like this:
// The idea here is that we distribute the tests perfectly equally amongst the shards.
// That is, each shard gets (tests.length / total).floor() tests.
// However, this leaves N = (tests.length % total) tests left over, so we will
// put one more test into the first N shards. We call these shards "busy" shards.
final int busyShards = tests.length % total;
final int testsPerBusyShard = (tests.length / total).ceil();
final int testsPerOtherShard = (tests.length / total).floor();
final int zeroIndexed = index - 1;
final int start = zeroIndexed <= busyShards
? zeroIndexed * testsPerBusyShard
: (busyShards) * testsPerBusyShard +
(zeroIndexed - busyShards) * testsPerOtherShard;
final int end = start +
(zeroIndexed <= busyShards ? testsPerBusyShard : testsPerOtherShard) -
1;
}|
RE the question of "how did we not run into problem this earlier," I plotted the value of The column number corresponds to how many shards there are. The row number corresponds to how many tests there are. Essentially, as more shards are added, the more likely it becomes that the current logic for subsharding will assign the last shard a range that exceeds the number of tests that exists. (this isn't meant to be legible) |
This comment was marked as spam.
This comment was marked as spam.
0768027 to
e2ea927
Compare
d7c6c73 to
bfa79c4
Compare
|
@christopherfujino This is ready for review again. There was an off-by-one error in the returned interval. |
dev/bots/utils.dart
Outdated
| final int testsPerShard = (tests.length / total).ceil(); | ||
| final int start = (index - 1) * testsPerShard; | ||
| final int end = math.min(index * testsPerShard, tests.length); | ||
| // While there exists a closed formula figuring out the range of tests this |
There was a problem hiding this comment.
Can we unit test this code, given it had a bug and how hard it was to fix it?
There was a problem hiding this comment.
I would recommend pulling this into its own function, to make testing easier--when doing that, I would recommend passing the index as index - 1, so that the function can just pretend its 0-indexed.
There was a problem hiding this comment.
Can we unit test this code, given it had a bug and how hard it was to fix it?
Done
I would recommend passing the index as
index - 1, so that the function can just pretend its 0-indexed.
I would like any parameter representing the subshard index to represent it exactly. For example, if the current subshard is 1_9, then I would be (slightly) surprised if a variable named subshardIndex (or similar) was set to 0.
|
The logic here seems right to me. |
dev/bots/utils.dart
Outdated
|
|
||
| // Lastly, compute the indices of the items in buckets[index]. | ||
| final int numberOfItemsInPreviousBuckets = index == 0 ? 0 : buckets.sublist(0, index - 1).sum; | ||
| final int start = (numberOfItemsInPreviousBuckets + 1) - 1; |
There was a problem hiding this comment.
Can we simplify this to final int start = numberOfItemsInPreviousBuckets;?
Which is more intuitive is maybe subjective, but to me this is just more confusing.
There was a problem hiding this comment.
Did so in refactor: remove one-indexing adjustment
bfa79c4 to
40659ea
Compare
Also cleans up flutter#156762 <details> <summary> Pre-launch checklist </summary> </details>
Also cleans up #156762
Pre-launch checklist
///).