Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Improve base time of assert_linear_performance #11369

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Aug 15, 2024
Merged

Improve base time of assert_linear_performance #11369

merged 2 commits into from
Aug 15, 2024

Conversation

tompng
Copy link
Member

@tompng tompng commented Aug 13, 2024

Base time of linear performance assertion was using this calculation.

tbase = 10 ** Math.log10(t).ceil

There is a 10x gap. When rehearsal time exceeds some threshold, tbase becomes 10x large.
It can potentially make the assertion loose and also flaky.

T = 0.0024 # n=100000 fails
T = 0.0025 # n=400000 passes
def test_linear_performance_quadratic
  assert_linear_performance([10000, 50000, 100000, 150000, 200000, 400000], rehearsal: 10) do |n|
    t = T * (n / 10000.0) ** 2
    t2 = Process.clock_gettime(:CLOCK_THREAD_CPUTIME_ID) + t
    1 while Process.clock_gettime(:CLOCK_THREAD_CPUTIME_ID) < t2
  end
end

Note that test in ruby/rexml uses seq = [10000, 50000, 100000, 150000, 200000]

I think this pull request makes the assertion more strict and also less flaky.

10 ** Math.log10(tmax * ([(tmax / tmin), 2].max ** 2)).ceil
# ↓ remove ceil that makes 10x gap
10 ** (Math.log10(tmax * ([(tmax / tmin), 2].max ** 2)) + 0.5)
# ↓
3.16 * tmax * ([(tmax / tmin), 2].max ** 2)
# ↓
12.65 * tmax * [(tmax / tmin) ** 2 / 4, 1].max
# ↓ round safe_factor to 10
10 * tmax * [(tmax / tmin) ** 2 / 4, 1].max
# safe_factor * tmax * variance_factor
# variance_factor == 1 when variance of rehearsal time is small

kou
kou approved these changes Aug 14, 2024
View reviewed changes
Copy link
Member

@kou kou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

tool/lib/core_assertions.rb
@@ -825,7 +825,7 @@ def assert_linear_performance(seq, rehearsal: nil, pre: ->(n) {n})
end
times.compact!
tmin, tmax = times.minmax
tbase = 10 ** Math.log10(tmax * ([(tmax / tmin), 2].max ** 2)).ceil
tbase = 10 * tmax * [(tmax / tmin) ** 2 / 4, 1].max
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a comment why we use this calculation?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done 👍

@kou
Copy link
Member

kou commented Aug 15, 2024

Could you rebase on master before you merge this?
Could you also update https://github.com/ruby/test-unit-ruby-core ?

@nobu Do you want to review this before we merge this?

tompng added 2 commits August 15, 2024 15:10
@tompng
Improve base time of assert_linear_performance
d2140d7 
Remove `.ceil` from base time calculation that makes 10x gap.
This will make the assertion more strict and also less flaky.
@tompng
Add a description comment for assert_linear_performance tbase calcula…
d76671f 
…tion
@tompng tompng force-pushed the linear_performance_strict_less_flaky branch from 7845c59 to d76671f Compare August 15, 2024 06:10
@tompng
Copy link
Member Author

tompng commented Aug 15, 2024

Rebased
I'll open a pull request to https://github.com/ruby/test-unit-ruby-core later.
(and I don't have a permission to merge)

@kou kou enabled auto-merge (squash) August 15, 2024 06:19
@kou
Copy link
Member

kou commented Aug 15, 2024

Oh, sorry. I've set auto-merge.

@kou kou merged commit 8a7e6f6 into ruby:master Aug 15, 2024
106 checks passed
@tompng tompng deleted the linear_performance_strict_less_flaky branch August 15, 2024 12:23
@tompng tompng mentioned this pull request Aug 15, 2024
Merged
matzbot pushed a commit that referenced this pull request Aug 31, 2024
@nagachika
merge revision(s) 8a7e6f6:
f095a08 
	Improve base time of assert_linear_performance (#11369)

	Remove `.ceil` from base time calculation that makes 10x gap.
	This will make the assertion more strict and also less flaky.
hsbt pushed a commit to hsbt/ruby that referenced this pull request Sep 10, 2024
@tompng @hsbt
Improve base time of assert_linear_performance (ruby#11369)
11f5e6c 
Remove `.ceil` from base time calculation that makes 10x gap.
This will make the assertion more strict and also less flaky.
hsbt pushed a commit that referenced this pull request Sep 10, 2024
@tompng @hsbt
Improve base time of assert_linear_performance (#11369)
b33c331 
Remove `.ceil` from base time calculation that makes 10x gap.
This will make the assertion more strict and also less flaky.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nobu nobu nobu approved these changes

@kou kou kou approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@tompng @kou @nobu