Complete rewrite of Parser and Tokenizer #58

Arkatufus · 2018-07-10T17:38:29Z

Decouple the Tokenizer and Parser:
- No cross talk or dependency between Tokenizer and Parser for easier debugging.
- Parser works purely on token list generated by Tokenizer.
Complete include parsing support:
- Support required, file, url, and classpath keyword. Include types are passed to the HoconIncludeCallback delegate.
- Support parsing hocon configuration that contains a pure array (should be used for array include only).
- Include can be declared floating in an object (merged with the fields on the same level), assigned to a field key, and declared inside an array.
- require, file, url, and classpath starting declaration needs to be typed exactly as "required(", "file(", "url(" and "classpath(" with no space between the keyword and the open parenthesis.
- Change ExternalIncludes example to show how to use the new include implementation.
Complete self-referencing substitution support.
Support for the += array assignment operator.
Make Hocon parser to be completely free from Akka related implementation:
- Configuration related classes are moved to Hocon.Configuration.
- Create similar functionalities to Config class in HoconRoot class.
- CONFIGURATION compiler directive is removed by bumping Hocon.Configuration netstandard version to 2.0 and including System.Configuration.ConfigurationManager package.
Add ability to normalize a parsed Hocon to minimize the memory footprint.
Add PrettyPrint feature to HoconRoot.
Add Infinity, -Infinity and NaN support for float and double value.
Add hexadecimal and octal value support for long, int, and byte value.
Breaking changes:
- String types are now stored as separate types, meaning:
  - Unquoted null string are resolved as null value
  - Quoted "null" strings are resolved as a string "null"
  - Paths and keys are fully parsed, a quoted key needs to be quoted to be valid, eg:
    - "a.\"some quoted, key\"" is not the same as "a.some quoted, key" because commas are invalid inside an unquoted string.
    - "\"a.key.with.dots\"" will actually use the whole quoted string as a field key, while "a.key.with.dots" will be resolved as a path with 4 keys.

… reporting

- Decouple the Tokenizer and Parser: - No cross talk or dependency between Tokenizer and Parser for easier debugging. - Parser works purely on token list generated by Tokenizer. - Complete include parsing support: - Support required, file, url, and classpath keyword. Include types are passed to the HoconIncludeCallback delegate. - Support parsing hocon configuration that contains a pure array (should be used for array include only). - Include can be declared floating in an object (merged with the fields on the same level), assigned to a field key, and declared inside an array. - require, file, url, and classpath starting declaration needs to be typed exactly as "required(", "file(", "url(" and "classpath(" with no space between the keyword and the open parenthesis. - Change ExternalIncludes example to show how to use the new include implementation. - Make Hocon parser to be completely free from Akka related implementation: - Configuration related classes are moved to Hocon.Configuration. - Create similar functionalities to Config class in HoconRoot class. - CONFIGURATION compiler directive is removed by bumping Hocon.Configuration netstandard version to 2.0 and including System.Configuration.ConfigurationManager package. - Make Hocon parser async capable. - Async parser allows for long running process such as retrieving include configuration from a url possible. - Add ability to normalize a parsed Hocon to minimize the memory footprint. - Add PrettyPrint feature to HoconRoot. - Breaking changes: - String types are now stored as separate types, meaning: - Unquoted null string are resolved as null value - Quoted "null" strings are resolved as a string "null" - Paths and keys are fully parsed, a quoted key needs to be quoted to be valid, eg: - "a.\"some quoted, key" is not the same as "a.some quoted, key" because commas are invalid inside an unquoted string. - "\"a.key.with.dots\" will actually use the whole quoted string as a field key, while "a.key.with.dots" will be resolved as a path with 4 keys.

Horusiath

I haven't managed to get a deep insight - so more issues may arrise. Right now - congratz, it's a huge amount of work 👍 Nonetheless, problems I see are mostly related to performance. I can recommend reading this one: http://xoofx.com/blog/2016/06/13/implementing-a-markdown-processor-for-dotnet/ (more interesting this article is 2 years old, with new .NET APIs we can do much faster things now).

If we decide to make this library compatible with Spans and new super duper bits of .NET highly optimized libraries, the only parts that are actually allocating here would be a final HOCON objects. I think, this would be both a great learning experience and a powerfull selling point for HOCON standard.

Horusiath · 2018-07-21T17:50:38Z

src/HOCON.Tests/ArrayAndObjectConcatenation.cs

        {
            var hocon = @"a=[1,2] [3,4]";
-            Assert.True(new[] { 1, 2, 3, 4 }.SequenceEqual(ConfigurationFactory.ParseString(hocon).GetIntList("a")));
+            Assert.True(new[] { 1, 2, 3, 4 }.SequenceEqual(HoconParser.Parse(hocon).GetIntList("a")));


HoconParser.Parse? What has happened with ConfigurationFactory? Reminder: we have existing API, that we should keep for backward compatiblity.

ConfigurationFactory static class is moved to Hocon.Configuration namespace. I guess I can keep the namespace as Hocon for the "Hocon.Configuration" project?

On the same note, should I keep the old name "Parser" instead of the new "HoconParser"?

Hocon.Configuration namespace is moved to Hocon, HoconParser is refactored back to Parser

Horusiath · 2018-07-21T17:53:36Z

src/HOCON.Tests/Commas.cs

+                {
+                    while (config_1.MoveNext())
+                    {
+                        config_2.MoveNext();


Why not sequence equal anymore?

Because for this test, we're comparing 2 KeyValuePair from 2 different objects. The values of each KVP are HoconField, which does not have any IEquatable implementation. I'll add IEquatable implementation on the next commit.

IEquatable added, you can use object1.AsEnumerable().SequenceEqual(object2.AsEnumerable()) now.

Horusiath · 2018-07-21T17:57:56Z

src/HOCON/HoconParser.cs

+namespace Hocon
+{
+    public delegate string HoconIncludeCallback(HoconCallbackType callbackType, string value);
+    public delegate Task<string> HoconIncludeCallbackAsync(HoconCallbackType callbackType, string value);


What use cases do we expect for async callbacks? From what I see below this is the only purpose for that entire API to be async. Event then, I guess in grand most of the cases that will be no a resource like file/url etc, which would make a better use of ValueTask.

The main purpose is to have it non-blocking for "include url()" because it potentially has to wait a long time and large Hocon config file cases because the current Tokenizer implementation does not support string Stream. Since I'm trying to make Hocon as standalone and as generic as possible, I thought making everything async is the best move.

My big concern here is that how much this feature justifies all of this? Tbh. I never knew that you can include URL or file in you HOCON files and I'm using it for the last 4 years. On the other side parsing and text processing can be made very lightweight if we won't try to make it all async. My gut feeling is that 2 heaviest parts of Parse...Async methods are throwing exceptions and actually calling await/returning a Task.

From personal experience: few weeks ago I've wrote a CSV parser using the latests bits of .NET low-level API. After benchmarking it on 1.5GB files it turned out that over 80% of memory and CPU was used for async state machine related to async loading of files and 8% was used by garbage collector to recycle those tasks (the rest of parsing pipeline was not allocating anything).

Well, I can always remove all the async await, its pretty easy to do since none of my code are actually asynchronous. It was there for completeness, since I wanted to support the required url() spec.
The whole point of this exercise is trying to support the full spec and make Hocon a real standalone configuration library outside of Akka (ie. just like NewtonSoft Json), so I'm just covering all the bases.

Parsing HOCON should be a boot up process, it seems farfetched that there would be so many calls to the the parser that it justifies a complete async API IMO.

If the purpose is to make it more maintainable, aim to make it smaller/less code instead of adding things with low end user value.

my 2 cents

Refactored it so that only the include callback is async, would this be a better compromise?

Horusiath · 2018-07-21T17:59:28Z

src/HOCON/HoconParser.cs

+    /// This class contains methods used to parse HOCON (Human-Optimized Config Object Notation)
+    /// configuration strings.
+    /// </summary>
+    public class HoconParser


By default please keep classes sealed. Inheritance has no sense (and it's counter performance) is target class was not designed for it to begin with.

Horusiath · 2018-07-21T18:01:12Z

src/HOCON/Impl/Utils.cs

+                TaskContinuationOptions.None,
+                TaskScheduler.Default);
+
+        public static TResult RunSync<TResult>(Func<Task<TResult>> func)


How do these functions differ from func().GetAwaiter().GetResult()?

To be honest, I don't know. I lifted the code from
https://github.com/aspnet/AspNetIdentity/blob/master/src/Microsoft.AspNet.Identity.Core/AsyncHelper.cs
The biggest concern is that func().GetAwaiter().GetResult() can cause a deadlock related to context. I kind of figured that Microsoft knows their own code better than I do, so I tried to code as defensively as possible.

Horusiath · 2018-07-21T18:44:59Z

src/HOCON/Impl/HoconPath.cs

+
+namespace Hocon
+{
+    public class HoconPath:List<string>, IEquatable<HoconPath>


Horusiath · 2018-07-21T18:46:44Z

src/HOCON/Impl/HoconPath.cs

+
+            unchecked
+            {
+                return this.Aggregate(seed, (current, item) => (current * modifier) + item.GetHashCode());


From what I've seen, HoconPath may be used as key in dictionaries - and using aggregates instead of loops is super expensive way for computing hash code.

Changed to a loop. Internally, I only use the last key in the path as a key in the HoconObject field Dictionary, but other people may decide to use them as keys in dictionaries, HoconPath should be fixed at that time, though I didn't put the effort to make it into an immutable list, will this be a problem in the future?

Horusiath · 2018-07-21T18:49:57Z

src/HOCON/Impl/HoconTokenizer.cs

+                return false;

-            return _text[_index];
+            return _text.Substring(Index = pattern.Length, pattern.Length) == pattern;


I don't know which version of .NET we want to target, but I'm pretty sure we can do this check without allocating.

It was a dead code, pruned.

Horusiath · 2018-07-21T18:51:38Z

src/HOCON/Impl/HoconTokenizer.cs

+            if (Index + length > _text.Length)
+                return null;
+
+            string s = _text.Substring(Index, length);


Again, no need to allocate. The current state of HOCON parser can allocate 1000x more memory than the string which it parses. I think, if we are going to rewrite it we also should think about that.

Done, I've removed as much Substring as possible.

Horusiath · 2018-07-21T18:53:38Z

src/HOCON/Impl/HoconTokenizer.cs

+        private bool PullNewLine(HoconTokenizerResult tokens)
        {
-            int start = Index;
+            if (!Matches(Utils.NewLine))


If you do a check for both cases (\n and \r\n), you should not need that replacement of Unix style endings at the intial HoconParser.Parse.

Its already handled, \n and \r\n shouldn't matter anymore.

… SequenceEqual.

…lved.

…GetResult()`

…f to improve performance. Side effect: Improved `include` parsing.

…ll be processed during value query

…Parser` class name back to `Parser`

# Conflicts: # src/HOCON/Impl/HoconParserException.cs

Aaronontheweb · 2018-08-27T23:42:11Z

@Arkatufus looks like build server barfed on some new language features in your latest set of commits

Horusiath · 2018-08-28T06:00:56Z

@Aaronontheweb which C# version is our build server using? We should go to latest by default (now 7.3).

Aaronontheweb · 2018-08-28T12:34:53Z

@Horusiath since we're building against the .NET Core SDK, I think you can just set the language setting in the .CSPROJ file. Build server has 7.1 installed on it by default I think, but the CSPROJ settings should override.

@Arkatufus

This review is from a few months back and it looks like @Arkatufus made an effort to work with the new System.Memory types, as requested, but ultimately decided it wasn't worth the trouble. Still, the rest of the review is good and much appreciated.

Aaronontheweb · 2018-10-02T14:04:36Z

@Arkatufus, these changes look fantastic and I appreciate the amount of time and trouble it take to implement them. Also really appreciate the amount of time and effort spent by @Horusiath and @rogeralsing reviewing the code in detail over the summer.

I'm going to merge this in now so we can start applying other changes (such as porting akkadotnet/akka.net#3600 into this library) in separate PRs on top of this one, and I think we might be ready to do a new release of the library soon. Nice work.

Arkatufus added 10 commits June 26, 2018 12:47

Parsing improvements, spec conformation, and better parsing exception…

fb454d5

… reporting

Fix for inconsistent list/array getter behavior

0dff69e

Simplify HoconToken, remove SourceIndex and Length properties.

27b6d95

Simplify Config by moving methods to HoconRoot and inhering HoconRoot

22e0af1

Add infinity, NaN, hex, and octal support

50ee0fe

Add self-referencing substitution feature support

4b2fe3f

Fix, FsCheck was accidentally added.

0c57ae5

Add support for += array operator

8fe7f0d

Empty commit to trigger post_receive hooks

36f8cc9

Horusiath previously requested changes Jul 21, 2018

View reviewed changes

Arkatufus added 19 commits July 22, 2018 07:14

Add IEquatable<T> implementation to HoconField so HoconObject can use…

a254880

… SequenceEqual.

Seal non-inherited classes

31844a6

Remove end of line hack, it was already fixed.

268052e

Change string.Join to Stringbuilder

3028b3d

Unwrap Aggregate() to simple loop

84a390e

Remove as much allocation as it possibly can from the Tokenizer.

d0d5292

Unroll recursive IsValueCyclic test method

9f55bd0

Remove async support

fe2decf

Refactor async so that it is called only when require is being reso…

f6ebb4b

…lved.

Remove AsyncHelper in favor of `ConfigureAwait(false).GetAwaiter().…

2a90edf

…GetResult()`

Remove LINQ from Tokenizer to improve GC performance.

cdc7731

Change Tokenizer to use switch..case instead of using cascading i…

ed4a458

…f to improve performance. Side effect: Improved `include` parsing.

Simplify tokenizer, no need to parse object type when tokenizing, it'…

e81bb3d

…ll be processed during value query

Remove any Try...Catch block from all TryGetX methods

68b7cd4

Move Hocon.Configuration namespace back to Hocon. Refactor `Hocon…

be37acc

…Parser` class name back to `Parser`

Use Memory<T> aggresively

dc786cb

Minor improvements, fix literal equality bug

c473f0c

Add LineNumber and LinePosition properties to HoconParserException

58149d2

Add LineNumber and LinePosition properties to HoconParserException

4696f13

Arkatufus added 2 commits August 26, 2018 05:34

Clean up exception types

792aaaa

Merge branch 'Use_System_dot_Memory' into Complete_rewrite

48ab4b0

# Conflicts: # src/HOCON/Impl/HoconParserException.cs

Arkatufus added 2 commits September 11, 2018 02:02

Revert System.Memory<T>

b465ff2

Remove async when it is not required.

64a0cef

Aaronontheweb merged commit fd3fda2 into akkadotnet:dev Oct 2, 2018

Complete rewrite of Parser and Tokenizer #58

Complete rewrite of Parser and Tokenizer #58

Uh oh!

Conversation

Arkatufus commented Jul 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Horusiath left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Arkatufus Jul 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Arkatufus Jul 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aaronontheweb commented Aug 27, 2018

Uh oh!

Horusiath commented Aug 28, 2018

Uh oh!

Aaronontheweb commented Aug 28, 2018

Uh oh!

Aaronontheweb commented Oct 2, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

Arkatufus commented Jul 10, 2018 •

edited

Loading

Horusiath left a comment •

edited

Loading

Arkatufus Jul 21, 2018 •

edited

Loading

Arkatufus Jul 22, 2018 •

edited

Loading