Implement servus::Serializable::getSchema() #65

dnachbaur · 2016-10-20T08:58:10Z

emit JSON schema from FBS in zerobufCxx.py
create enum class for enums instead of 'old' enums; breaks existing enum names
fix wrong JSON value for empty arrays (was 'null', is '[]' now)

rdumusc

Are there unit tests for the new ZeroBuf::getSchema() functionality?

rdumusc · 2016-10-20T09:05:08Z

bin/zerobufCxx.py

-                       "::" + impl_function +
-                       "\n{" +
-                       NEXTLINE + self.body)
+            file.write('\n{0} {1}{2}\n{{{3}{4}'.format(self.ret_val,


is there not an extra '{' between \n and {3} ?

Yes, because .format() replaces all {} placeholders

Ahhhh I finally got it after testing for myself... so the single { is ESCAPED as {{, otherwise .format() complains about a missing matching }...

rdumusc · 2016-10-20T09:07:48Z

bin/zerobufCxx.py

-            return 'set{0}( {1}( ::zerobuf::fromJSON< {2} >( ::zerobuf::getJSONField( json, "{3}" ))));'.\
+        # convert enums to their name as a string
+        if self.value_type.is_enum_type:
+            return 'set{0}( {1}( from_string( ::zerobuf::fromJSON< std::string >( ::zerobuf::getJSONField( json, "{3}" )))));'.\


{2} not used, remove?

rdumusc · 2016-10-20T09:08:50Z

bin/zerobufCxx.py

-            return '::zerobuf::toJSON( {0}( get{1}( )), ::zerobuf::getJSONField( json, "{2}" ));'.\
+        # convert enums back from their name
+        if self.value_type.is_enum_type:
+            return '::zerobuf::toJSON( std::string( to_string( get{1}( ))), ::zerobuf::getJSONField( json, "{2}" ));'.\


same here, {0} not used

rdumusc · 2016-10-20T09:17:26Z

bin/zerobufCxx.py

+            strs.append('case {0}::{1}: return std::string( "{1}" );'.format(self.name, enumValue))
+        return Function('std::string',
+                        'to_string( const {0}& val )'.format(self.name),
+                        'switch( val )\n    {{\n'


replace '\n ' occurences with NEXTLINE

rdumusc · 2016-10-20T09:29:48Z

bin/zerobufCxx.py

                if len(attrib) == 2 and is_zerobuf_type:
                    member = DynamicZeroBufMember(name, value_type, dynamic_type_index)
+                    table = next(x for x in fbsFile.tables if x.name == cxxtype)
+                    properties[name] = table.json_schema


I'm a bit confused as to what these variables represent. Is 'property' the equivalent of a c++ pointer or a reference on properties[name]? can't you do "property = table.json_schema" instead of "properties[name] = table.json_schema"?

It's python, so it's most probably a reference. I tried the same assignment as you described, and as it didn't work, I fell back to the code I put.

property = table.json_schema can't work. In python all variables are pointers (and argument passing could be called pointer by value)

rdumusc · 2016-10-20T09:33:39Z

bin/zerobufCxx.py

+                    if is_byte_type:
+                        property['type'] = 'string'
+                        property['media'] = {}
+                        property['media']['binaryEncoding'] = 'base64'


these three lines are a copy-paste of the other "if is_byte_type" above, maybe can be combined somehow?

hernando · 2016-10-20T09:06:43Z

bin/zerobufCxx.py

+
+    if fbs in ['float', 'double']:
+        return 'number'
+    return 'integer'


Funny naming... that means that for json an integer is not a number.

hernando · 2016-10-20T09:11:07Z

bin/zerobufCxx.py

-                       NEXTLINE + self.body)
+            file.write('\n{0} {1}{2}\n{{{3}{4}'.format(self.ret_val,
+                (classname + '::') if classname else '', impl_function,
+                NEXTLINE, self.body))


I would write self.body in a separate file.write call, not so much for performance, but for readability.
I would even replace NEXTLINE with a function call startNextLine(file) to reduce the number of string concatenations (In some places the look ugly).

hernando · 2016-10-20T09:11:14Z

bin/zerobufCxx.py

-                       NEXTLINE + self.body + "\n")
+            file.write('\n{0}{1}{2}{3}\n'
+                .format((classname + '::') if classname else '', impl_function,
+                        NEXTLINE, self.body))


hernando · 2016-10-20T09:16:19Z

bin/zerobufCxx.py

+        # convert enums to their name as a string
+        if self.value_type.is_enum_type:
+            return 'set{0}( {1}( from_string( ::zerobuf::fromJSON< std::string >( ::zerobuf::getJSONField( json, "{3}" )))));'.\
+                format(self.cxxName, self.value_type.type, self.value_type.get_data_type(), self.cxxname)


The format string doesn't have any {2}, but you are passing four parameters, review.

hernando · 2016-10-20T09:16:49Z

bin/zerobufCxx.py

+        # convert enums back from their name
+        if self.value_type.is_enum_type:
+            return '::zerobuf::toJSON( std::string( to_string( get{1}( ))), ::zerobuf::getJSONField( json, "{2}" ));'.\
+                format(self.value_type.get_data_type(), self.cxxName, self.cxxname)


Format string has no {0}

hernando · 2016-10-20T09:39:25Z

bin/zerobufCxx.py

    def parse_members(self, fbsFile):
        dynamic_type_index = 0
+        self.json_schema['properties'] = {}
+        properties = self.json_schema['properties']


properties = {}
self.json_schema['properties'] = properties
?

hernando · 2016-10-20T09:39:43Z

bin/zerobufCxx.py

            value_type = ValueType(cxxtype, cxxtype_size, is_zerobuf_type, is_enum_type, is_byte_type)

+            properties[name] = {}
+            property = properties[name]


same as above

hernando · 2016-10-20T09:44:17Z

bin/zerobufCxx.py

            if self.is_dynamic(attrib, fbsFile):
                if len(attrib) == 2 and is_zerobuf_type:
                    member = DynamicZeroBufMember(name, value_type, dynamic_type_index)
+                    table = next(x for x in fbsFile.tables if x.name == cxxtype)


I find this code a bit hard to understand (not the code itself, but why you do this). Can you add some comments to explain it.

hernando · 2016-10-20T09:47:05Z

bin/zerobufCxx.py

                dynamic_type_index += 1
                self.dynamic_members.append(member)
            else:
                if len(attrib) == 2 or len(attrib) == 3:


The ifs in this function are quite cryptic, but this is the one I understand less. Can you add comments to know what are the cases that you are dealing with in each branch?

hernando · 2016-10-20T09:58:18Z

bin/zerobufCxx.py

        dynamic_type_index = 0
+        self.json_schema['properties'] = {}
+        properties = self.json_schema['properties']
        for attrib in self.attributes:


I'd like to suggest a big change in this for loop, feel free to do it, but I think it will improve readability. Actually it's two different proposals, if you don't want to do it any of them at least you should add more comments because the code it's quite convoluted.
Here I understand that you are doing two independent things in the same loop and it gets big. For readability I think it's better to separate the two processings. The easiest way to do it is to have two separate loops (despite the if sentences are going to be the same), one for the original code and another for the new stuff about properties. For maintainability the loop code should be refactored, and the only way I can imagine is using something similar to a visitor factor. In python you do it without to much hassle. Create a function for doing the loop and the if parts, this function takes as an argument a list of objects with callbacks to deal with each specific case. Declare two classes, one with the callbacks for the original code and another for the properties. With proper function names you won't need to write any single comment.

rdumusc · 2016-10-20T11:28:58Z

bin/zerobufCxx.py

-                        '\n    default: throw std::runtime_error( "{1}" );\n    }}'
-                        .format(NEXTLINE.join(strs), 'Unknown value for enum {0}'.format(self.name)), split=True)
+                        'switch( val ){0}{{\n'
+                        '    {1}'


there is still one NEXTLINE hiding here

rdumusc · 2016-10-20T11:48:04Z

bin/zerobufCxx.py

-                        property['media'] = {}
-                        property['media']['binaryEncoding'] = 'base64'
+                    if is_byte_type: # byte arrays are base64 strings
+                        property = _add_base64_string(property)


So if I understand correctly the assignment and function return statement are redundant, since the function already modifies the property object passed to it

Function arguments are passed by copy, hence I found this 'pattern' to mimic a in-place modification :)

I just discovered that it is more subtle than that, you can't really reason in C++ terms of copy or reference with the python 'variables', they are more like 'labels' pointing to objects which are either mutable, or immutable. In this case 'property' is mutable (it's like a list) and it is modified by the function - the assignment and return value have no effect (it just reassigns 'property' to 'property'). Some references:
http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html#other-languages-have-variables
http://stackoverflow.com/questions/986006/how-do-i-pass-a-variable-by-reference/8140747#8140747

@rdumusc is right, property = _add_base64_string(property) doesn't make sense in Python. Functions arguments are not passed by copy, what is copied is the pointer to the object. That means you can't change the variable at the caller scope (but remind the variable is just a pointer), but you can change the object it points to.

rdumusc · 2016-10-21T08:29:32Z

The code is good now, but what about the unit tests?

dnachbaur · 2016-10-21T08:33:47Z

Good point.

dnachbaur · 2016-10-21T08:35:12Z

On the other hand, the code coverage has not been affected. But I guess you want me to test the generated JSON schema.

rdumusc · 2016-10-21T08:42:18Z

Yes indeed, I meant testing the new JSON schema part. This is really the sort of code which is well suited to unit testing.

rdumusc · 2016-10-21T10:07:31Z

...almost there, the last thing missing are tests for the new enum-to-string conversion functions:

    std::string to_string( const TestEnum& val );
    TestEnum from_string( const std::string& val );
    std::ostream& operator << ( std::ostream& os, const TestEnum& val );

* emit JSON schema from FBS in zerobufCxx.py * create enum class for enums instead of 'old' enums; breaks existing enum names * fix wrong JSON value for empty arrays (was 'null', is '[]' now)

dnachbaur · 2016-10-21T11:31:11Z

retest this please

dnachbaur assigned eile, hernando and rdumusc Oct 20, 2016

rdumusc reviewed Oct 20, 2016

View reviewed changes

hernando requested changes Oct 20, 2016

View reviewed changes

rdumusc reviewed Oct 20, 2016

View reviewed changes

hernando approved these changes Oct 21, 2016

View reviewed changes

Implement servus::Serializable::getSchema()

049a952

* emit JSON schema from FBS in zerobufCxx.py * create enum class for enums instead of 'old' enums; breaks existing enum names * fix wrong JSON value for empty arrays (was 'null', is '[]' now)

dnachbaur merged commit e095397 into HBPVIS:master Oct 21, 2016

Implement servus::Serializable::getSchema() #65

Implement servus::Serializable::getSchema() #65

Uh oh!

Conversation

dnachbaur commented Oct 20, 2016

Uh oh!

rdumusc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hernando Oct 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdumusc commented Oct 21, 2016

Uh oh!

dnachbaur commented Oct 21, 2016

Uh oh!

dnachbaur commented Oct 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rdumusc commented Oct 21, 2016

Uh oh!

rdumusc commented Oct 21, 2016

Uh oh!

dnachbaur commented Oct 21, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

hernando Oct 20, 2016 •

edited

Loading

dnachbaur commented Oct 21, 2016 •

edited

Loading