WIP: support for binary schema deserialization #5001
Conversation
std::function makes code harder to debug because it requires stepping through a separate destructor and call operator. It's use unnecessary in the Parser since the functions taking functors are private and are only used within idl_parser.cpp. Therefore the definitions can stay in idl_parser.cpp as well. Only care must be taken that the definitions appear before use but that's already true and all compilers will complain equally if it get's violated. This change might also improve performance since it might allow inlining where it wasn't possible before but I haven't measured that.
Support binary schemas in flatc and flatbuffers::Parser. This can simplify loading JSON data in some cases since there's no issue with include files.
|
Yes, I think the big issue is that binary schemas were never intended to have the full set of information that the parser stores, as they initially were only needed for reflection. We could add more data to the reflection data to have the full coverage the JSON and language code-gens need.. I am just afraid it is going to clutter the schema a lot. For that reason, for most bools that are derived from attributes, I'd say store those as attributes and only turn them back into bools upon reading them back into the parser state. I know that uses way more space, but is probably acceptable for attributes that are used infrequently. |
aardappel
left a comment
There was a problem hiding this comment.
Generally very nice functionality! Can you document somewhere how this can be used? This could be either in CppUsage.md in the reflection section, and/or somewhere in Schemas.md
|
|
||
| Offset<reflection::Field> Serialize(FlatBufferBuilder *builder, uint16_t id, | ||
| const Parser &parser) const; | ||
| bool Deserialize(const reflection::Field* field, Parser& parser); |
There was a problem hiding this comment.
Please use the existing code style for *.
include/flatbuffers/idl.h
Outdated
| size_t parent_fieldn, | ||
| const StructDef *parent_struct_def); | ||
| // clang-format off | ||
| #if defined(FLATBUFFERS_CPP98_STL) |
| " --raw-binary Allow binaries without file_indentifier to be read.\n" | ||
| " This may crash flatc given a mismatched schema.\n" | ||
| " --size-prefixed Input binaries are size prefixed buffers.\n" | ||
| " --schema-size-prefixed\n" |
There was a problem hiding this comment.
why not reuse the existing flag for this?
There was a problem hiding this comment.
The reason is that with flatc --schema --binary --size-prefixed a size prefixed schema is created. Now if you want to read or create a plain buffer with a size prefixed schema or a size prefixed buffer with a plain schema you need flags to tell which is which. I imagine that the size-prefixed schema case itself is rare so you rarely ever need to give this flag.
| if (schema->services()) { | ||
| for (auto it = schema->services()->begin(); it != schema->services()->end(); | ||
| ++it) { | ||
| std::string qualified_name = it->name()->str(); |
There was a problem hiding this comment.
use auto where possible.
|
I don't I'll be able to finish this PR — at least not any time soon. I can leave it open if anybody else wants to pick it up. |
|
There is now a second PR trying to do the same: #5077 |
f381fd7 to
31f8799
Compare
f116b80 to
f12cca8
Compare
|
This pull request is stale because it has been open 6 months with no activity. Please comment or this will be closed in 14 days. |
This adds support for using a binary schema created with
flatc --binary --schemaas in input for JSON serialization and deserialization. This is useful since a binary schema can be stored as a single object without having to deal with include paths. Ultimately, it might be useful to be able to use a binary schema to generate code as well but I don't know if it's really worth it. This is still a work in progress at the moment as I've only added one test which doesn't even pass. I've already found one problem, though: by default builtin attributes are not serialized which means that flexbuffer[ubyte]members don't work unless--bfbs-builtinsis also specified. I think the same is true fornested_flatbufferbut I'm not that far yet. @aardappel Should we extend thereflection::Fieldwith aflexbuffer:boolandnested_flatbuffer:string?