-
Notifications
You must be signed in to change notification settings - Fork 96
fix: use g17 format for doubles #434
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
It looks like the tests are currently failing due to some handling of the floating points:
This is a behavior change. Hopefully, someone does not depend on exact floating points, but I have concerns about this causing issues for existing users. I was hoping to have @bednar and @davidby-influx chime in on the change. Thanks! |
My suggestion is to change the tests to round the Something like: expected = 2.33333 got = ComputedValue() if Math.Round(got, 5) != expected This may not fix everything (because midpoint rounding can cause problems, but it will fix a lot of the cases. |
Thanks for the looking at the PR! The tests are primarily comparing two strings of line protocol. I think in those cases if we are good with updates to the floating points and it sounds like we are, I can update the expected. I'll let this this sit over night and see if Jakub chimes in. Thanks again, I appreciate the review. |
It looks good for me 👍 |
How will this work when we add a filter to compare
} |
Thank you both - the failing tests are updated, formatting corrected, and readme rebased. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
fixes: #408
Proposed Changes
Updates the format used for doubles when building a Point to use the G17 format. This ensures 17 bits of precision and that values are successful in round-trips. This is the recommendation from the dot.net docs as well.
Checklist
dotnet test
completes successfully