Implement ignore_errors for DictParser.Reader #386
Conversation
The format.revmap method was always returning the 'name' it was checking the reversed map against and not only when it was actually in the keys of the reversed map.
Adding the 'ignore_errors' attribute to the DictReader init.
Adding the error / warning methods call to the DictReader is_valid_attribute handling.
Adding error handling to the DictReaders Section and Property creation to allow recovering an invalid document.
jgrewe
left a comment
There was a problem hiding this comment.
lgtm
One minor thing maybe for future tests, can we have the file creation strings (json, or yml) outside the sourcecode? it breaks the flow, methinks.
|
I also don't particularly enjoy the way the file creation strings are formatted in these tests, but in most cases it makes it easier to see at one glance what is being tested and how. I am not dogmatic about it though, if there are any suggestions for a different setup, let me know! :) |
odml/tools/dict_parser.py
Outdated
| except Exception as exc: | ||
| msg = "Section not created (%s)\n %s" % (sec_attrs, str(exc)) | ||
| self.error(msg) | ||
| # If recovered in ignore_error mode, return empty list |
There was a problem hiding this comment.
Not an issue but I guess the list will only be empty if it's the first section right?
Could this do the same as the parse_properties() method and continue with the rest of the sections instead of returning?
There was a problem hiding this comment.
Yes, good catch; that's what it should do.
Could put them in a global fake constant in the file, right at the top, at indent level 0. Maybe something like: file_content = """
Document:
id: 82408bdb-1d9d-4fa9-b4dd-ad78831c797c
sections:
- id: d4f3120a-c02f-4102-a9fe-2e8b77d1d0d2
name: sec
odml-version: '1.1'
""".strip()That way you preserve the locality somewhat (it's in the same file), the content is indented as it will appear in the file without breaking flow, and the |
|
I would like that and it also gives you the option to call the variable e.g. something like: |
In the DictReader, do not stop and return after a Section.create failed with an Exception, but continue with the next Section in the list.
Moving file content in test_parser_json and test_parser_yaml to increase readability.
jgrewe
left a comment
There was a problem hiding this comment.
👍
I think I really like this solution
The XMLParser.Reader comes with an option to ignore errors when opening a file while notifying the user with all encountered odml format problems, enabling users to fix problematic odml files using the odml library instead of having to directly manipulate the file content.
This option has now been added to the DictParser.Reader as well, also allowing to open and fix problematic odml JSON and YAML files.
The PR
odml.loadfunction.format.revmapwhere the reverse mapping of an odml attribute would always return the case that the attribute is part of the format, even if it was not.Don't be afraid, most of the changes in this PR again increase the test coverage.