Double-to-Decimal conversions are not as faithful as they could be. #42775

airbreather · 2020-09-26T16:10:54Z

Description

Converting a System.Double value to a System.Decimal value does not seem to follow consistent rules. Packing more significant digits into the source value seems to result in a converted value that's more precise, but only to a point.

I probably would have let that go if not for the fact that System.Double.ToString() is (now) perfectly capable of taking a value and finding the smallest decimal string that represents that value, so I'm now looking at a situation where I'm encouraged to format a value to a string and then parse it right back out.

Considering the below code snippet, given how things work with inputs like 0.1, 0.2, 0.3, etc., I would expect all lines to write the same value, but they do not. Of course, I wouldn't attempt to run this code on anything older than .NET Core 3.0, because the IEEE-754 formatting / parsing improvements are a significant quality-of-life improvement when investigating this.

It would also make some sense if the valToDecimal line were to write a value with more precise digits, something closer to 174.28491752098176448271260597, since that's also a faithful representation of the double value.

ConsoleApp0.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>

</Project>

Program.cs

using static System.Console;

double val = 174.28491752098176;
decimal valAsDecimalOriginally = 174.28491752098176m;
decimal valToDecimal = (decimal)val;
double valToDecimalToDouble = (double)valToDecimal;
decimal valThroughString = decimal.Parse($"{val}");

WriteLine("val:                    " + val);
WriteLine("valAsDecimalOriginally: " + valAsDecimalOriginally);
WriteLine("valToDecimal:           " + valToDecimal);
WriteLine("valToDecimalToDouble:   " + valToDecimalToDouble);
WriteLine("valThroughString:       " + valThroughString);

`dotnet run` Output

val:                    174.28491752098176
valAsDecimalOriginally: 174.28491752098176
valToDecimal:           174.284917520982
valToDecimalToDouble:   174.284917520982
valThroughString:       174.28491752098176

Configuration

> dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.100-rc.1.20452.10
 Commit:    473d1b592e

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19041
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.100-rc.1.20452.10\

Host (useful for support):
  Version: 5.0.0-rc.1.20451.14
  Commit:  38017c3935

.NET SDKs installed:
  3.1.402 [C:\Program Files\dotnet\sdk]
  5.0.100-rc.1.20452.10 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.22 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.22 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.0-rc.1.20451.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.22 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.0-rc.1.20451.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.8 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.0-rc.1.20452.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

Regression?

Unknown.

Other information

Context: in NetTopologySuite, we're porting an algorithm from JTS that involves trying to infer the precision of a set of coordinate values. The fastest way I found to do this (in cases where the precision is actually realistic) is by converting the System.Double values to System.Decimal and extracting the scale byte from it, with a fallback that converts through System.String if needed.

I was expecting this fallback to only get hit for "haha, gotcha!" cases, where either double --> decimal --> double can't work (NaN / too large / too small) or would be lossy because the value is exceptionally close to zero. However, it seems to get hit for the overwhelming majority of values that I randomly generate.

Round-tripping through a string (which, I guess, I could stackalloc) is an OK workaround, and it also reveals some insight into the issue: whenever I format a randomly generated System.Double value (ABS(value) between 1 and 180), chop off the last two characters of its string representation, and parse it back to System.Double, the double --> decimal --> double pipeline has been perfectly faithful. I've only tried a few hundred million of these, though, so there might still be some sleepers in there.

Here's how I'm testing my hundred million:

using System;
using System.Threading.Tasks;

Parallel.For(0, 100_000_000, i =>
{
    var rng = new Random(i);
    double d = (rng.NextDouble() * 180) + 1;
    if (rng.Next(2) == 0)
    {
        d = -d;
    }

    d = double.Parse($"{d}"[..^2]);
    if (d != (double)(decimal)d)
    {
        Console.Error.WriteLine($"Error with {d}");
    }
});

The text was updated successfully, but these errors were encountered:

Dotnet-GitSync-Bot · 2020-09-26T16:11:07Z

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

airbreather · 2020-09-26T16:28:02Z

Addendum:

The fastest way I found to do this

Of course, a faster way that I could use to do this would be to directly reimplement parts of a System.Double formatting routine that can identify when to stop slapping more decimal digits onto the output stream... that seems like overkill.

ghost · 2020-09-26T17:07:43Z

Tagging subscribers to this area: @tannergooding, @pgovind, @jeffhandley
See info in area-owners.md if you want to be subscribed.

tannergooding · 2020-09-26T18:01:45Z

I do agree the current behavior seems confusing. However, its worth noting that ToString doesn't return the "nearest" decimal representation but rather it returns the "shortest roundtrippable string". That is the "shortest" string where double.Parse(x.ToString()) results in a value that is bitwise equal to x (excluding NaN).

Additionally, 174.28491752098176 is actually 174.28491752098176448271260596811771392822265625 and if we do a faithful conversion then the decimal result will be 174.28491752098176448271260597m, not 174.2849175209817m. Likewise (decimal)0.3 would result in 0.2999999999999999888977697537m, not 0.3 as the actual double value produced by parsing 0.3 is 0.299999999999999988897769753748434595763683319091796875.

It looks like the issue here is that https://source.dot.net/#System.Private.CoreLib/Decimal.DecCalc.cs,1726 assumes double only has 15-digits of precision. This is a flawed assumption as double may require as many as 17 decimal digits to correctly roundtrip a string, but may actually have up to 767 significant digits of precision.

I think there are 3 possible paths that could be taken here:

Leave everything as is
- This is a non-breaking change and maintains the status quo
- This does not resolve the issue you are seeing
Update double->decimal and float->decimal to convert to a decimal which is equivalent to the "shortest" roundtrippable string
- This is a breaking change and requires a bar check
- This may result in confusing results for some users as 0.3 will be 0.3m but (0.1 + 0.3) will be 0.30000000000000004m (it also produces 0.3m today)
- The computation here is potentially expensive and may regress perf for some scenarios
Update double->decimal and float->decimal to accurately convert the binary float-value to the closest equivalent decimal floating-point value
- This has essentially all the same drawbacks as the previous
- This may not resolve the issue you are seeing
- This is the most "technically correct" fix, but likely also the most breaking in terms of results returned

airbreather · 2020-09-26T20:09:01Z

However, its worth noting that ToString doesn't return the "nearest" decimal representation but rather it returns the "shortest roundtrippable string". That is the "shortest" string where double.Parse(x.ToString()) results in a value that is bitwise equal to x (excluding NaN).

Indeed -- sorry, I should have been more precise in those first few sentences. I'll fix them now.

Additionally, 174.28491752098176 is actually 174.28491752098176448271260596811771392822265625 and if we do a faithful conversion then the decimal result will be 174.28491752098176448271260597m, not 174.2849175209817m. Likewise (decimal)0.3 would result in 0.2999999999999999888977697537m, not 0.3 as the actual double value produced by parsing 0.3 is 0.299999999999999988897769753748434595763683319091796875.

Indeed -- from my experimentation before reporting this, I got the impression that the intended behavior of the System.Double --> System.Decimal conversion was to do essentially what it's doing right now: the System.Double value that's the closest legal representation of a number whose "true" decimal form (not System.Decimal) has just a few digits should yield a System.Decimal value that also has just a few digits, even if there are other legal System.Decimal values that are closer to the value that's stored in the System.Double (phew, that was a complicated sentence to type).

This is very similar to how ToString() converts to a decimal (not System.Decimal) representation today, and it avoids confusion in very common situations like "(decimal)0.3".

Stopping at 15 digits like this seems inconsistent with any of the answers that I can give for "what should it do?". Other than, of course, "it should do what it's always done", which is of course valid, but if that's the resolution, then I'd ask for at least some kind of mention in the documentation for the conversion operator to say how it deals with these issues.

Update double->decimal and float->decimal to convert to a decimal which is equivalent to the "shortest" roundtrippable string

This is a breaking change and requires a bar check

This may result in confusing results for some users as 0.3 will be 0.3m but (0.1 + 0.3) will be 0.30000000000000004m (it also produces 0.3m today)

The computation here is potentially expensive and may regress perf for some scenarios

I see the concern about confusion, but it's also confusing that (0.1 + 0.2) prints out something different than (decimal)(0.1 + 0.2). Printing out a System.Double is supposed to print it out in decimal (not System.Decimal) form anyway, so it's weird that some System.Double-to-"decimal" conversions (ToString) keep the precision and others (convert to System.Decimal) do not.

Plus, truncating to 15 places still doesn't make the issue of compounding rounding error go away, it just means you have to compound more rounding error before it happens.

Update double->decimal and float->decimal to accurately convert the binary float-value to the closest equivalent decimal floating-point value

This has essentially all the same drawbacks as the previous

This may not resolve the issue you are seeing

This is the most "technically correct" fix, but likely also the most breaking in terms of results returned

I considered proposing something like this, but IMO the downsides of (decimal)0.3 giving a value with dozens of decimal places seem to outweigh the benefits of being "technically correct".

I honestly wouldn't be too sad about this resolution, though, because at least it would be consistent and sane: after all, converting System.Single --> System.Double doesn't do anything smart like "oh, this is the closest single-precision value to 0.3, so I'll return the closest double-precision value to 0.3". It just converts exactly what's there.

I'd have to find another way to achieve my goal, of course, but that's just how it goes sometimes.

dotnet-policy-service · 2025-04-01T21:12:29Z

Due to lack of recent activity, this issue has been marked as a candidate for backlog cleanup. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will undo this process.

This process is part of our issue cleanup automation.

dotnet-policy-service · 2025-04-15T23:28:24Z

This issue will now be closed since it had been marked no-recent-activity but received no further activity in the past 14 days. It is still possible to reopen or comment on the issue, but please note that the issue will be locked if it remains inactive for another 30 days.

Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Sep 26, 2020

jkotas added the area-System.Numerics label Sep 26, 2020

tannergooding removed the untriaged New issue has not been triaged by the area owner label Sep 30, 2020

tannergooding added this to the Future milestone Sep 30, 2020

dotnet-policy-service bot added backlog-cleanup-candidate An inactive issue that has been marked for automated closure. no-recent-activity labels Apr 1, 2025

dotnet-policy-service bot removed this from the Future milestone Apr 15, 2025

dotnet-policy-service bot closed this as completed Apr 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Double-to-Decimal conversions are not as faithful as they could be. #42775

Double-to-Decimal conversions are not as faithful as they could be. #42775

airbreather commented Sep 26, 2020 •

edited

Loading

Dotnet-GitSync-Bot commented Sep 26, 2020

airbreather commented Sep 26, 2020

ghost commented Sep 26, 2020

tannergooding commented Sep 26, 2020 •

edited

Loading

airbreather commented Sep 26, 2020

dotnet-policy-service bot commented Apr 1, 2025

dotnet-policy-service bot commented Apr 15, 2025

Double-to-Decimal conversions are not as faithful as they could be. #42775

Double-to-Decimal conversions are not as faithful as they could be. #42775

Comments

airbreather commented Sep 26, 2020 • edited Loading

Description

ConsoleApp0.csproj

Program.cs

dotnet run Output

Configuration

Regression?

Other information

Dotnet-GitSync-Bot commented Sep 26, 2020

airbreather commented Sep 26, 2020

ghost commented Sep 26, 2020

tannergooding commented Sep 26, 2020 • edited Loading

airbreather commented Sep 26, 2020

dotnet-policy-service bot commented Apr 1, 2025

dotnet-policy-service bot commented Apr 15, 2025

airbreather commented Sep 26, 2020 •

edited

Loading

`dotnet run` Output

tannergooding commented Sep 26, 2020 •

edited

Loading