Enable flatbuffer to initialize Parser from bfbs (#4283) #5077

tira-misu · 2018-12-03T06:15:02Z

Now its possible to generate json data from bfbs data type and flatbuffers data
and visa versa.

add deserialize functionality in parser from bfbs
add small usage sample

Thank you for submitting a PR!

Please make sure you include the names of the affected language(s) in your PR title.
This helps us get the correct maintainers to look at your issue.

If you make changes to any of the code generators, be sure to run
cd tests && sh generate_code.sh (or equivalent .bat) and include the generated
code changes in the PR. This allows us to better see the effect of the PR.

If your PR includes C++ code, please adhere to the Google C++ Style Guide,
and don't forget we try to support older compilers (e.g. VS2010, GCC 4.6.3),
so only some C++11 support is available.

Include other details as appropriate.

Thanks!

Now its possible to generate json data from bfbs data type and flatbuffers data and visa versa. * add deserialize functionality in parser from bfbs * add small usage sample

googlebot · 2018-12-03T06:15:06Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here (e.g. I signed it!) and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

tira-misu · 2018-12-03T06:21:14Z

I signed it!

googlebot · 2018-12-03T06:21:15Z

CLAs look good, thanks!

aardappel · 2018-12-03T17:55:29Z

include/flatbuffers/idl.h

  Offset<reflection::Object> Serialize(FlatBufferBuilder *builder,
                                       const Parser &parser) const;

+  void Deserialize(Parser *parser, const flatbuffers::Vector<flatbuffers::Offset<reflection::Field>> *flatFields);


Please use snake_case, not camelCase, here and elsewhere.

Also, lines <80 chars, { on same line.. please read Google C++ Style Guide and apply to your code.

Code is now Google C++ Style Guide conform.

aardappel · 2018-12-03T18:01:29Z

samples/sample_bfbs.cpp

+
+using namespace MyGame::Sample;
+
+// This is an example of parsing text straight into a buffer and then


I appreciate that you've made a sample, but really this would be better off in the tests, since these samples are not run as part of testing. Also, would be better to use the existing .bfbs file rather than generating it anew.

Using existing bfbs in example now.
Used/fixed existing test of @fbenkstein. (And found/fixed some issues in deserialization)

aardappel · 2018-12-03T18:02:12Z

src/idl_parser.cpp

+std::string Parser::GetNameFromFullQualifiedName(std::string fullQualifiedName)
+{
+  Namespace *ns = new Namespace();
+


no empty lines needed, unless function is really long.

aardappel · 2018-12-03T18:03:44Z

Thanks! This looks like a pretty good implementation, but how is it different from #5001 by @fbenkstein ? Could you two compare implementations and make sure we end up with the best of both somehow?

tira-misu · 2018-12-04T05:53:10Z

I will pick up @fbenkstein implementation and merge it into mine. It seams that he has no time for it. I wasn't aware that someone had already taken on the subject.

Attribute flexbuffer has be included in bfbs. Only with this attribute test will run. By initialization a parser by a bfbs the attribute has to be known for this filed. monsterdata_test.golden has a flexbuffer field so parse would fail.

aardappel · 2018-12-06T22:23:15Z

src/flatc.cpp

    "  --raw-binary       Allow binaries without file_indentifier to be read.\n"
    "                     This may crash flatc given a mismatched schema.\n"
    "  --size-prefixed    Input binaries are size prefixed buffers.\n"
+    "  --schema-size-prefixed\n"


I'd recommend not having this option. Having 2 kinds of binary schema files doesn't help anyone.

Problem is, you already have two kinds of schema files, depending whether you call flatc --binary --schema with the --size-prefixed option or not. In #5001 I added the flag so you can serialize and deserialize with or without size prefix independently of whether your schema is size prefixed.

Compromise: removing this option and allow both file formats. Binary will be verified against both verifier and right one will be choosen. What's your opinion?

Using the verifier to decide doesn't sound great, better would be to use reflection::SchemaBufferHasIdentifier, and when that fails, try again assuming the first 4 bytes are a size.

I guess to me --size-prefixed is a niche feature, so --schema-size-prefixed is a niche of niche, and would prefer to not polute the options with it if we can reasonably either demand only regular FlatBuffers are used for binary schemas, or silently detect both with the above option.

aardappel · 2018-12-06T22:25:11Z

src/idl_parser.cpp

    const auto repack = IsInteger(e.type.base_type);
    switch (e.type.base_type) {
-    // clang-format off
+      // clang-format off


If you're going to indent the comment, then also the code following it.

Damn IDE. It has changed some points of its own. I hoped that I reverted all, but somehow i missed something. I will look through all changes again an fix it.

fbenkstein · 2018-12-11T07:18:49Z

Yes. Makes sense.

…

On Mon 10. Dec 2018 at 21:46 Wouter van Oortmerssen < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In src/flatc.cpp <#5077 (comment)>: > @@ -106,6 +125,8 @@ std::string FlatCompiler::GetUsageString(const char *program_name) const { " --raw-binary Allow binaries without file_indentifier to be read.\n" " This may crash flatc given a mismatched schema.\n" " --size-prefixed Input binaries are size prefixed buffers.\n" + " --schema-size-prefixed\n" Using the verifier to decide doesn't sound great, better would be to use reflection::SchemaBufferHasIdentifier, and when that fails, try again assuming the first 4 bytes are a size. I guess to me --size-prefixed is a niche feature, so --schema-size-prefixed is a niche of niche, and would prefer to not polute the options with it if we can reasonably either demand only regular FlatBuffers are used for binary schemas, or silently detect both with the above option. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5077 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHXSIyCdWrjQ5TmSF7cP436UOjd_geSdks5u3sgogaJpZM4Y98Nt> .

aardappel · 2018-12-13T19:59:22Z

Thanks, looking good now!

…e#5077) * Enable flatbuffer to initialize Parser from bfbs (google#4283) Now its possible to generate json data from bfbs data type and flatbuffers data and visa versa. * add deserialize functionality in parser from bfbs * add small usage sample * Fix build break * Merge branch 'pr/1' into fix-issue4283 * Fix buildbreak * Build monster_test.bfbs with --bfbs-builtins Attribute flexbuffer has be included in bfbs. Only with this attribute test will run. By initialization a parser by a bfbs the attribute has to be known for this filed. monsterdata_test.golden has a flexbuffer field so parse would fail. * Fix generate_code.sh * Revert automatic indent changes by IDE * Auto detect size prefixed binary schema files * Use identifier (bfbs) to detect schema files

Enable flatbuffer to initialize Parser from bfbs (google#4283)

c896f23

Now its possible to generate json data from bfbs data type and flatbuffers data and visa versa. * add deserialize functionality in parser from bfbs * add small usage sample

Fix build break

69b060e

aardappel reviewed Dec 3, 2018

View reviewed changes

aardappel mentioned this pull request Dec 3, 2018

WIP: support for binary schema deserialization #5001

Closed

tira-misu added 4 commits December 6, 2018 18:07

Merge branch 'pr/1' into fix-issue4283

ce032fe

Fix buildbreak

6b54163

Build monster_test.bfbs with --bfbs-builtins

afd3ccd

Attribute flexbuffer has be included in bfbs. Only with this attribute test will run. By initialization a parser by a bfbs the attribute has to be known for this filed. monsterdata_test.golden has a flexbuffer field so parse would fail.

Fix generate_code.sh

d7b5eec

aardappel reviewed Dec 6, 2018

View reviewed changes

tira-misu added 2 commits December 10, 2018 11:42

Revert automatic indent changes by IDE

01e9266

Auto detect size prefixed binary schema files

a5051a4

Use identifier (bfbs) to detect schema files

b818412

aardappel merged commit dba962e into google:master Dec 13, 2018

tira-misu deleted the fix-issue4283 branch December 14, 2018 06:07

sjoblom65 mentioned this pull request Feb 11, 2019

Add --gen-bfbs-embed to flatc so the binary schema can part of the code / library #5162

Closed


		using namespace MyGame::Sample;

		// This is an example of parsing text straight into a buffer and then

Enable flatbuffer to initialize Parser from bfbs (#4283) #5077

Enable flatbuffer to initialize Parser from bfbs (#4283) #5077

Uh oh!

Conversation

tira-misu commented Dec 3, 2018

Uh oh!

googlebot commented Dec 3, 2018

What to do if you already signed the CLA

Individual signers

Corporate signers

Uh oh!

tira-misu commented Dec 3, 2018

Uh oh!

googlebot commented Dec 3, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aardappel commented Dec 3, 2018

Uh oh!

tira-misu commented Dec 4, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fbenkstein commented Dec 11, 2018 via email

Uh oh!

aardappel commented Dec 13, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants