Update tools.md for missing color meaning issue #1491 #1624
Conversation
## Issue > Null hypothesis test discrepancy google#1491 ## Solution Proposed > documentation needs to be enhanced ## Changes made > Added the lines to the tools.md file which explains the following - 1. Clarify the meaning of the colors in the output. 2. Defining the 'success' and 'failure' explicitly. 3. Clarify the interpretation of the p-value and the null hypothesis.
Changes are based on my understanding from the code review from the PR.
When comparing benchmarks, `compare.py` uses statistical tests to determine whether there is a statistically-significant difference between the measurements being compared. The result of said statistical test is additionally communicated through color coding:
Restructured the text and made a few tweaks for better logical reading.
There was a problem hiding this comment.
Thanks for your guidance, Roman. The starting paragraph felt so interesting while reading, and the changes you suggested made this more well versed.
It was the first time that I came across about benchmarking, learned so much in the process.
If anything, still felt inconsistent or incomplete, I would love to make those changes.
|
@dmah42 someone more fluent in english will need to proofread the wording, |
seems good to me. feel free to merge it whenever you're happy with the content. |
Added an output summary from a benchmark comparison with statistics provided for a multi-threaded process. Furthermore, added the breakdown of each row.
There was a problem hiding this comment.
@varshneydevansh thank you!
I suppose this could use some more editorial work,
but this is certainly much better than what we have now :)
|
Thank you so much @LebedevRI for guiding me. I recently encountered benchmarking for the first time and began learning about it. Then I realized that this is something where, alongside learning, I can even contribute. I learned a lot from this process, not just about benchmarking, but how to organize thoughts in a way that makes it easier to understand for others. Regards. |
Issue >
Null hypothesis test discrepancy #1491
Solution Proposed >
documentation needs to be enhanced
Changes made >
Added the lines to the tools.md file which explains the following -