Codestin Search App

DvirDukhan · 2020-11-03T21:22:34Z

!!! WIP !!! DO NOT REVIEW OR MERGE UNTIL READY

This PR scope:

codecov · 2020-11-04T22:36:56Z

Codecov Report

Merging #488 (978d25d) into master (64001de) will decrease coverage by 0.42%.
The diff coverage is 59.45%.

@@            Coverage Diff             @@
##           master     #488      +/-   ##
==========================================
- Coverage   76.16%   75.74%   -0.43%     
==========================================
  Files          25       35      +10     
  Lines        5433     5701     +268     
==========================================
+ Hits         4138     4318     +180     
- Misses       1295     1383      +88

Impacted Files	Coverage Δ
src/model.c	`78.48% <ø> (+8.10%)`	⬆️
src/script.c	`79.73% <ø> (+14.08%)`	⬆️
src/serialization/AOF/rai_aof_rewrite.c	`0.00% <0.00%> (ø)`
src/tensor.c	`83.97% <ø> (+0.40%)`	⬆️
.../serialization/RDB/decoder/previous/v0/decode_v0.c	`61.14% <61.14%> (ø)`
...c/serialization/RDB/decoder/current/v1/decode_v1.c	`64.41% <64.41%> (ø)`
src/redis_ai_types/script_type.c	`65.00% <65.00%> (ø)`
src/serialization/RDB/decoder/decode_previous.c	`66.66% <66.66%> (ø)`
src/serialization/RDB/encoder/rai_rdb_encode.c	`66.66% <66.66%> (ø)`
src/redis_ai_types/model_type.c	`70.00% <70.00%> (ø)`
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 64001de...978d25d. Read the comment docs.

lantiga · 2020-12-03T10:25:44Z

Hi @DvirDukhan can you do a format pass of the code?

alonre24

I added few minor comments and a few questions

alonre24 · 2020-12-06T13:48:37Z

+                                                            .dtype = dtype,
+                                                            .shape = shape,
+                                                            .strides = strides,
+                                                            .byte_offset = 0},


Why the byte_offset field is initialized to 0 and not to the loaded byte_offset?

from docs:

/*! \brief The offset in bytes to the beginning pointer to data */ uint64_t byte_offset;

We could do this indeed. Right now every tensor that has been set from the client will have byte offset of 0, but let's load it from the serialization, it's a good idea.

alonre24 · 2020-12-06T14:49:31Z

+
+void *RAI_RDBLoadModel(RedisModuleIO *io);
+
+void *RAI_RDBLoadScript(RedisModuleIO *io);


new line at the end?

alonre24 · 2020-12-06T14:51:00Z

+
+#include "../serialization_include.h"
+
+int ModelType_Register(RedisModuleCtx *ctx);


alonre24 · 2020-12-06T15:23:42Z

+                                                       RAI_TensorDim(tensor, i));
+    }
+
+    RedisModule_EmitAOF(aof, "AI.TENSORSET", "scvcb", key, dtypestr, dims, ndims, "BLOB", data,


What does the "scvcb" stands for?

see robj **moduleCreateArgvFromUserFormat(const char *cmdname, const char *fmt, int *argcp, int *flags, va_list ap) in Redis repo

One input of the function RedisModule_EmitAOF is fmt which is the string scvcb, after this input you have var args input, and the RedisModule_EmitAOF function calls moduleCreateArgvFromUserFormatto know how to handle the inputs, according to the chars infmt`

alonre24 · 2020-12-06T15:26:06Z

+
+    const char *backendstr = RAI_BackendName(model->backend);
+
+    RedisModule_EmitAOF(aof, "AI.MODELSET", "slccclclcvcvcv", key, backendstr, model->devicestr,


Here also, I couldn't understand what is "slccclclcvcvcv" (looks very random...)

alonre24 · 2020-12-06T15:26:48Z

+
+void RAI_AOFRewriteScript(RedisModuleIO *aof, RedisModuleString *key, void *value) {
+    RAI_Script *script = (RAI_Script *)value;
+    RedisModule_EmitAOF(aof, "AI.SCRIPTSET", "scccc", key, script->devicestr, script->tag, "SOURCE",


Same here, not clear what is "scccc"

alonre24 · 2020-12-06T15:26:58Z

+    RAI_Script *script = (RAI_Script *)value;
+    RedisModule_EmitAOF(aof, "AI.SCRIPTSET", "scccc", key, script->devicestr, script->tag, "SOURCE",
+                        script->scriptdef);
+}


lantiga

I agree on reading / writing the byte_offset.
One note: I'm not sure the definition of the data types should be under the serialization directory. Let's take the data type out of that and into a dedicated directory.
Also, coverage is giving several warnings, we should make sure we are testing those functions for the most part.

lantiga · 2020-12-09T13:05:03Z

+                                                            .dtype = dtype,
+                                                            .shape = shape,
+                                                            .strides = strides,
+                                                            .byte_offset = 0},


We could do this indeed. Right now every tensor that has been set from the client will have byte offset of 0, but let's load it from the serialization, it's a good idea.

lantiga

LGTM. Just as a heads up (I'm sure you are aware of this), with the binary strings PR we have changed tag to be a binary string, so older RDB's won't load.

lantiga

LGTM. How do we test encode/decode from previous versions in CI?

DvirDukhan · 2020-12-30T19:22:14Z

@lantiga You are right
I will add the required tests for v0

alonre24 · 2021-01-04T09:16:16Z

+    dtype.lanes = RedisModule_LoadUnsigned(io);
+
+    size_t ndims = RedisModule_LoadUnsigned(io);
+    if (RedisModule_IsIOError(io))


Why do you decide to check for IOError specifically in lines 24 and 41?

alonre24 · 2021-01-04T09:18:16Z

+        .backends_inter_op_parallelism = getBackendsInterOpParallelism(),
+    };
+
+    len;


What is the meaning of this declaration (line 112)?

alonre24 · 2021-01-04T09:20:29Z

+        RedisModule_FreeString(NULL, tag);
+
+    return NULL;
+}


alonre24 · 2021-01-04T09:24:21Z

+    def test_v0_torch_model(self):
+        key_name = "pt_minimal{1}"
+        con = self.env.getConnection()
+        model_rdb = b'\x07\x81\x00\x8f\xff0\xe0\xc4,\x00\x02\x02\x05\x04CPU\x00\x05\x0ePT_MINIMAL_V0\x00\x02\x00\x02\x00\x02\x00\x02\x00\x05\xc3C\x0eEH\x0ePK\x03\x04\x00\x00\x08\x08\x00\x00\x86\xb0zO\x00\xe0\x02\x00\x1a\x12\x00\x10\x00pt-minimal/versionFB\x0c\x00Z\xe0\x02\x00\n1\nPK\x07\x08S\xfcQg\x02 ;@\x03\x00P Q\x00\x14 Q\x00\x08\xe0\x08Q\x02\x1c\x004\xe0\x03Q\x13code/__torch__.pyFB0\xe0\x04[\xe0\x1b\x00\x1f5LK\n\x830\x10\xdd{\x8a\xb7T\xb0\x82\xdb\x80\xbd\x81\xbb\xeeJ\t\xa3\x19\xab\x90fd\x12[z\xfb\x1f\x06\xad\xab\xf7\x7f\xb2\xda7k\\$\xd8\xc8\t\x1d\xdab\xf4\x14#\xfao/n\xf3\\\x1eP\x99\x02\xb0v\x1f%\xa5\x17\xa7\xbc\xb06\x97\xef\x8f\xec&\xa5%,\xe1\t\x83A\xc4g\xc7\xf1\x84I\xf4C\xea\xca\xc8~2\x1fy\x99D\xc7\xd9\xda\xe6\xfc\xads\x0f \x83\x1b\x87(z\xc8\xe1\x94\x15.\xd7?5{\xa2\x9c6\r\xd8_\r\x1ar\xae\xa4\x1aC\r\xf2\xebL][\x15?A\x0b\x04a\xc1#I\x8e!\x0b\x00\xc8 \x03\xe1\x11\x0b\x02&\x00\x1e\xe1\x14\x0b\x0c.debug_pklFB\x1a\xe1\x12\x15\x1f5\x8eA\n\xc20\x10EcU\x90\x82+/0\xcb\x8a%\x07p\xe5V\x06t\xdb\x9d\xa4mB"m\xd3\x1f\xa4\x11q\xe7\xca\x1e\xc7S\xa8\xd72)\xe4m\x06\xde\x87?\xff\x99d\xb8)\x88\xc7\x10I\x90\x8cf\x86\xe1\x1f\xbc\xf0]\x9c\xbd\x05\xcf\xc1i[IzU\x8e\x0e\x95U\xbd\xbb\xb4\xdcI]\xa7!\xac\xb9\x00\xa1\xed\x9d\xd9\x1f:\x1bx#r`9\x94\xdb\xfd\x14\x06,w7\xdb\x01\x83\x1d\x94\xa9I\x8a\xb5o\r\x15\xaaS-kh\x15\xff0s\\\x8df\x81G<\xf9\xb7\x1f\x19\x07|et\xbf\xe8\x9cY\xd2a\x08\x04\xa7\x94\x1a\x02\x97!\x04\x00\xb2 \x03A\x08\xe2\rf\x02\x18\x00#\xe1\x05\x08\x07nstants.`\xfa\x00\x1f\xe0\x12\xfa`\x00\x03\x80\x02).Au\x03m/\tW a\x00\x00@\x03\xe0\x11l\x02\x13\x00;\xe0\x03l\x03data\x80g\x007\xe0\x17g\xe0\x0f\x00\x02\x80\x02c\xe2\x00\xc2\x10\nMyModule\nq\x00)\x81}(X#U\x0f\x00trainingq\x01\x88ubq\x02`\xac\x04z\xb8\x18\x811 \x1b@\x03\x02PK\x01C5#0\x83\x82\xe3\x03J\x00\x12 \x17\xe0\x05\x00\xe3\t\x90\x80?\xe3\x01p\xe2\x03~\x00\x1c\xe0\x04<\x00R \r\xe0\x02?\xe3\x08~\xe0\x07I\xe1\x03\xbf\x00& ;\xe0\x01\x00\x01^\x01\xe0\x15I\xe2\x01\xbc\x80S\xe0\x01\xdd\xe1\x03\xa6\x00\x18 \x17\xe0\x01\x00D?\xe0\x04\x9d\xe2\x02\x07\xe0\x07E\xe1\x03?\x00\x13\xe0\x01B \x00\x00\xd4!K\xe0\x02E\xc1\xe0\x04PK\x06\x06, \x1e@\x00\x02\x1e\x03-@\x06`\x00\x00\x05`\x05\xe0\x01\x07\x00e \xd8@\x00\x01\x81\x03@\x05A\x9c\x01\x06\x07 \x06\x01\x00\xe6BV \x00@\x1e\x03PK\x05\x06 \n ;\x00\x05`/@+\x01\x00\x00\x00\t\x00MQ\xab\x8e\xfdc\x97>'


Are you sure that you want the model_rdb to be hard coded instead of reading it from a file?

DvirDukhan self-assigned this Nov 3, 2020

DvirDukhan changed the title ~~Serialization refactor and handle short reads~~ Serialization - refactor and handle short reads Nov 3, 2020

DvirDukhan force-pushed the serialization_and_short_reads branch 3 times, most recently from 7f7e517 to aa2c3c7 Compare November 4, 2020 22:05

DvirDukhan added 5 commits December 3, 2020 12:01

wip

fd9393d

decoupled tensor logic from redis type

c031466

finished refactor

2388f3f

wip

a027107

added IO error handilng in v0 and v1 RDB decode

b4c3d33

DvirDukhan force-pushed the serialization_and_short_reads branch from eda88f9 to b4c3d33 Compare December 3, 2020 10:15

DvirDukhan requested review from alonre24 and lantiga December 3, 2020 10:21

DvirDukhan marked this pull request as ready for review December 3, 2020 10:21

DvirDukhan added 2 commits December 3, 2020 12:29

make format

aaa43b1

Merge branch 'master' into serialization_and_short_reads

7856446

alonre24 reviewed Dec 6, 2020

View reviewed changes

DvirDukhan added 3 commits December 7, 2020 11:06

Merge branch 'master' into serialization_and_short_reads

543cfea

fixed PR comments

8945d03

format

5b5392b

lantiga suggested changes Dec 9, 2020

View reviewed changes

DvirDukhan added 7 commits December 21, 2020 18:44

Merge branch 'master' into serialization_and_short_reads

56ec03c

wip

78d67a1

wip

6d0c6ff

fixed PR comments

cfc4167

byte_offset

58c6c4a

free string instead of free

f6153d2

Merge branch 'master' into serialization_and_short_reads

a46d098

Merge branch 'master' into serialization_and_short_reads

1131b7c

lantiga previously approved these changes Dec 30, 2020

View reviewed changes

fixed v0 decode

0d30280

DvirDukhan dismissed lantiga’s stale review via 0d30280 December 30, 2020 14:28

make format

9a5d19b

lantiga previously approved these changes Dec 30, 2020

View reviewed changes

changed RDB decode according to memcheck PR refernce

667867f

DvirDukhan dismissed lantiga’s stale review via 667867f December 30, 2020 19:36

DvirDukhan added 3 commits December 30, 2020 21:37

make format

156ef66

serialization tests

20730fe

Merge branch 'master' into serialization_and_short_reads

818c304

alonre24 reviewed Jan 4, 2021

View reviewed changes

fixed PR comments

978d25d

alonre24 approved these changes Jan 4, 2021

View reviewed changes

DvirDukhan merged commit 6a389c0 into master Jan 4, 2021

DvirDukhan deleted the serialization_and_short_reads branch January 4, 2021 10:31


		void RAI_RDBLoadModel(RedisModuleIO io);

		void RAI_RDBLoadScript(RedisModuleIO io); No newline at end of file


		#include "../serialization_include.h"

		int ModelType_Register(RedisModuleCtx *ctx); No newline at end of file


		const char *backendstr = RAI_BackendName(model->backend);

		RedisModule_EmitAOF(aof, "AI.MODELSET", "slccclclcvcvcv", key, backendstr, model->devicestr,

Uh oh!

Conversation

DvirDukhan commented Nov 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Nov 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

lantiga commented Dec 3, 2020

Uh oh!

alonre24 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lantiga left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lantiga left a comment

Choose a reason for hiding this comment

Uh oh!

lantiga left a comment

Choose a reason for hiding this comment

Uh oh!

DvirDukhan commented Dec 30, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

DvirDukhan commented Nov 3, 2020 •

edited

Loading

codecov Bot commented Nov 4, 2020 •

edited

Loading