Fixes serialization of big decimals to json (#6530) #6531

mkljakubowski · 2016-09-06T12:54:22Z

Possible fix for #6530

Fixes serialization of BigDecimals to json in some particular case which was failing

mkljakubowski · 2016-09-06T14:00:37Z

framework/src/play-json/src/main/scala/play/api/libs/json/jackson/JacksonJson.scala

        val stripped = v.bigDecimal.stripTrailingZeros
        val raw = if (shouldWritePlain) stripped.toPlainString else stripped.toString

-        if (raw contains ".") json.writeTree(new DecimalNode(new JBigDec(raw)))


this was checking if number contains a . but 2e128 has no dot and still needs to be printed as BigDecimal

I'm suggesting we remove the conditional and write as:

json.writeTree(new DecimalNode(new JBigDec(raw)))

Because of the formatting logic above, raw should be written as a plain string for a number within the range such as 50.

Unfortunetly this breaks the tests as well. Whole issue is around jackson not beaing able to print 50 as bigdecimal and that's why we have this if to print it as bigInt.

Oops. I guess I didn't notice that Jackson had that special case.

gmethvin · 2016-09-06T20:23:18Z

Thanks. Maybe now we should just always use BigDecimal? It would make the logic a lot simpler and I'm not sure if we get much benefit from using BigInteger in some cases, as long as we know the number is being printed properly.

mkljakubowski · 2016-09-07T07:53:34Z

Yes, this code seems a bit overcomplicated. Let me check if using BigDecimal always will not cause any huge regression.

mkljakubowski · 2016-09-07T08:50:50Z

Ok, simplified code still works, but reintroduces #3784 which is still questionable. Imo makes no difference. 50 == 5e+1

Updating jackson didn't help.

gmethvin · 2016-09-07T20:14:14Z

framework/src/play-json/src/main/scala/play/api/libs/json/jackson/JacksonJson.scala

-        val stripped = v.bigDecimal.stripTrailingZeros
-        val raw = if (shouldWritePlain) stripped.toPlainString else stripped.toString
-
-        if (raw contains ".") json.writeTree(new DecimalNode(new JBigDec(raw)))


We still want to write a DecimalNode though, to preserve the formatting.

gmethvin · 2016-09-07T21:18:23Z

I don't think it makes a difference either but I'd rather avoid people complaining about the formatting change.

gmethvin · 2016-09-08T09:48:55Z

framework/project/Dependencies.scala

    "com.fasterxml.jackson.datatype" % "jackson-datatype-jdk8",
    "com.fasterxml.jackson.datatype" % "jackson-datatype-jsr310"
-  ).map(_ % "2.7.6")
+  ).map(_ % "2.8.2")


In master this should be okay but I think minor versions of Jackson can have breaking changes, so I'd rather not upgrade in 2.5.x/2.4.x. I'd suggest we do this change in another PR so we can easily backport this one.

Reverted. Do you want me move this PR to 2.5.x branch? It's were the bug is after all.

actually we backport things by cherry-pick the commit from master. and if that's not possible we manually adjust it.

gmethvin · 2016-09-08T18:37:32Z

framework/src/play-json/src/test/scala/play/api/libs/json/JsonSpec.scala

    "Deserialize integer JsNumber as Jackson number node" in {
      val jsNum = JsNumber(new java.math.BigDecimal("50"))
-      fromJson[JsonNode](jsNum).map(_.toString) must_== JsSuccess("50")
+      fromJson[JsonNode](jsNum).map(_.toString) must_== JsSuccess("5E+1")


Can you update the code so this passes again? It's fine to only use BigDecimal but I don't want to screw up the formatting behavior.

Yes, I'm on it. Will be fixed today.
I also started looking in jackson for the solution but with no luck so far.

mkljakubowski · 2016-09-09T11:02:38Z

So the issue is I think in Jackson. This is the only piece that seems to fail:
JacksonJson.jsValueToJsonNode(JsNumber(new java.math.BigDecimal("50"))).toString == "5e+1"
But if you read JsNumber instead of JsonNode everything is fine.

gmethvin · 2016-09-09T11:24:35Z

@mkljakubowski The reason for reading as JsonNode was to see how it was printed in the JSON. It seems like when Jackson parses back the number it doesn't preserve the formatting.

Can you just put back all of the old code besides the special case for integers that was causing the bug? I think that will fix it.

mkljakubowski · 2016-09-09T11:29:47Z

@gmethvin reverted old code
So in this case, until jackson is fixed we need to have the condition to check how the value should be printed (bigDec or bigInt).

gmethvin · 2016-09-09T12:02:18Z

framework/src/play-json/src/main/scala/play/api/libs/json/jackson/JacksonJson.scala

        val raw = if (shouldWritePlain) stripped.toPlainString else stripped.toString

-        if (raw contains ".") json.writeTree(new DecimalNode(new JBigDec(raw)))
+        if (raw.exists(ch => ".eE".contains(ch))) json.writeTree(new DecimalNode(new JBigDec(raw)))


Given that this is common code path, maybe the most optimal way to write it is:

if (raw.indexOf('E') < 0 && raw.indexOf('.') < 0) { json.writeTree(new BigIntegerNode(new JBigInt(raw))) } json.writeTree(new DecimalNode(new JBigDec(raw)))

…yframework#6531)

mkljakubowski force-pushed the js-big-decimal-serialization branch from f65ef0b to 3b6ef2f Compare September 6, 2016 12:56

mkljakubowski changed the title ~~Fixes serialization of big decimals to json (#6420)~~ Fixes serialization of big decimals to json (#6530) Sep 6, 2016

mkljakubowski force-pushed the js-big-decimal-serialization branch 2 times, most recently from 1477283 to b0be989 Compare September 6, 2016 13:02

mkljakubowski reviewed Sep 6, 2016
View reviewed changes

mkljakubowski force-pushed the js-big-decimal-serialization branch from b0be989 to beb751e Compare September 7, 2016 08:48

gmethvin reviewed Sep 7, 2016
View reviewed changes

mkljakubowski force-pushed the js-big-decimal-serialization branch from beb751e to fa815ae Compare September 8, 2016 08:33

gmethvin reviewed Sep 8, 2016
View reviewed changes

mkljakubowski force-pushed the js-big-decimal-serialization branch from fa815ae to ee653bb Compare September 8, 2016 09:52

gmethvin reviewed Sep 8, 2016
View reviewed changes

mkljakubowski force-pushed the js-big-decimal-serialization branch 2 times, most recently from a6e4e10 to 022d6f0 Compare September 9, 2016 11:19

mkljakubowski force-pushed the js-big-decimal-serialization branch from 022d6f0 to d08f674 Compare September 9, 2016 11:27

gmethvin reviewed Sep 9, 2016
View reviewed changes

Fixes serialization of big decimals to json (playframework#6530)

c2b82e2

mkljakubowski force-pushed the js-big-decimal-serialization branch from d08f674 to c2b82e2 Compare September 9, 2016 12:26

gmethvin merged commit 2a42231 into playframework:master Sep 11, 2016

gmethvin pushed a commit that referenced this pull request Sep 11, 2016

Fixes serialization of big decimals to json (#6530) (#6531)

8f54c88

gmethvin added this to the 2.5.7 milestone Sep 12, 2016

gmethvin mentioned this pull request Sep 12, 2016

JSON serialization broken for BigDecimal #6530

Closed

wsargent pushed a commit to wsargent/playframework that referenced this pull request Oct 25, 2016

Fixes serialization of big decimals to json (playframework#6530) (pla…

176934e

…yframework#6531)

Uh oh!

Fixes serialization of big decimals to json (#6530) #6531

Fixes serialization of big decimals to json (#6530) #6531

Uh oh!

Conversation

mkljakubowski commented Sep 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmethvin Sep 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmethvin commented Sep 6, 2016

Uh oh!

mkljakubowski commented Sep 7, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mkljakubowski commented Sep 7, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmethvin commented Sep 7, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmethvin Sep 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkljakubowski commented Sep 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gmethvin commented Sep 9, 2016

Uh oh!

mkljakubowski commented Sep 9, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mkljakubowski commented Sep 6, 2016 •

edited

Loading

gmethvin Sep 9, 2016 •

edited

Loading

mkljakubowski commented Sep 7, 2016 •

edited

Loading

gmethvin Sep 8, 2016 •

edited

Loading

mkljakubowski commented Sep 9, 2016 •

edited

Loading