After discussion and further investigation at a local Best Buy, the inconsistencies he found come down to the fact that Chrome on Windows criminally underperforms on this specific benchmark compared to Edge and similarly spec'd Chromebook or Mac devices.

My proposal for moving forward to address all concerns:

Do not use any sort of auto-detection like core(throttling): add option to slowdown to a device class #6162 in consistent environments like PSI. This was never the intention of the benchmark and I agree with all the concerns @exterkamp has raised here. Separate tuning of our exact throttling multiplier for PSI to better account for the longer task lengths we see there is still advised.
Maintain the policy of core(throttling): add option to slowdown to a device class #6162 that 750+ is all fast and will not be adjusted.
ULTRADUMB™ was never intended to distinguish between a 10-30% perf differential and these fluctuations on devices all within the "fast" device class are not a concern.
Investigate if the significant underperformance of Chrome in all JS benchmarks on Windows is at all reflective of reduced JS performance. (AI @patrickhulce)
Develop a slimmed down version of the Regexp benchmark from Octane 2.0, and use this in combination with or instead of ULTRADUMB™. Regexp is the only observable JS benchmark tested that actually maintains its differentiability on Windows. The only reason that it was not used to begin with is that it is quite large at 125KB. (AI @patrickhulce)

I have added a few notable device stats to the benchmarks datasheet that illustrates the difference between Chrome on Windows, Edge on Windows, and Chrome on *nix.

Improve BenchmarkIndex to land device targeting #9085

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions