Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Improve entry attribute assignment speed and serialization speed#534

Merged
hunterhector merged 36 commits intoasyml:masterfrom
hunterhector:slow
Sep 22, 2021
Merged

Improve entry attribute assignment speed and serialization speed#534
hunterhector merged 36 commits intoasyml:masterfrom
hunterhector:slow

Conversation

@hunterhector
Copy link
Member

@hunterhector hunterhector commented Sep 21, 2021

This PR fixes #533.

Description of changes

  1. Added parameters to allow serialization in multiple formats, implement a new pickle-based format.
  2. Remove complex checks in entry attributes, implement pointer translation in serialization/deserialization.

Possible influences of this PR.

  1. Some generation results could be different from before, but the default generation is still jsonpickle, which means it should not be different most of the time.

Test Conducted

  1. Created new tests for serialization-deserialization.
  2. Added a new expected serialization test.
  3. Remove old tests that check for pointers in the Entry state, now check whether the entry state is the correct/expected type.

@hunterhector
Add binary deserialization approach
3465455
@hunterhector
fix pylint
6de0ede
@hunterhector
Fix mypy.
22dccb9
@hunterhector
Fix mypy.
81a0b00
@hunterhector
Fix index update
8df3a9b
@hunterhector
Fix pylint
573bc80
@hunterhector
fix pylint
e89e5ed
@hunterhector
fix mypy
77ba42a
@hunterhector
fix value check
c524aa2
@hunterhector
fix json write
0acbf41
@hunterhector
Add configs to dbpedia tests.
48ff5c4
@hunterhector
Fix black.
4e4c937
@hunterhector
Pylint.
46b0c21
@hunterhector
mypy
175d345
@hunterhector
fix typo
2def196
@hunterhector
Merge branch 'master' of https://github.com/asyml/forte into slow
3d29e1d
@hunterhector
Merge branch 'master' of https://github.com/asyml/forte into slow
660df12
@hunterhector
Performance fixes on spans and attribute checks
9db1715
@hunterhector
get around span
c63f046
@hunterhector
Remove a few Span references.
533777d
@hunterhector
Black formatting
39acbc9
@hunterhector
Force ontology object dictionary keys to be integers.
2f81c83
@hunterhector
Remove attribute check during get and set of Entry
e67a378
@hunterhector
Merge branch 'master' of https://github.com/asyml/forte into slow
1d31b94
@hunterhector
Fix mypy and shared attributes
6bba103
@hunterhector
fix linting
8b5d3ab
@hunterhector
Fix serialization problems.
bb14c1b
@hunterhector
remove unused attribute checks
aa69093
@hunterhector
Fix testing.
904b792
@hunterhector
formatting.
b10e53d
@codecov
Copy link

codecov bot commented Sep 22, 2021 •
edited
Loading

Codecov Report

Merging #534 (c3ae595) into master (d7cbcfa) will increase coverage by 0.06%.
The diff coverage is 92.95%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #534      +/-   ##
==========================================
+ Coverage   79.35%   79.42%   +0.06%     
==========================================
  Files         220      220              
  Lines       15718    15781      +63     
==========================================
+ Hits        12473    12534      +61     
- Misses       3245     3247       +2
Impacted Files Coverage Δ
forte/data/index.py 79.12% <0.00%> (ø)
forte/data/span.py 79.16% <0.00%> (+2.24%) ⬆️
forte/processors/nlp/ner_predictor.py 17.12% <0.00%> (ø)
...orte/data/ontology/ontology_code_generator_test.py 100.00% <ø> (ø)
...e/data/readers/race_multi_choice_qa_reader_test.py 97.87% <ø> (ø)
tests/forte/data/selector_test.py 98.38% <ø> (ø)
tests/forte/pipeline_test.py 96.88% <ø> (-0.90%) ⬇️
...rte/datasets/wikipedia/dbpedia/dbpedia_datasets.py 81.35% <75.00%> (-0.08%) ⬇️
forte/data/multi_pack.py 78.11% <77.77%> (+5.98%) ⬆️
forte/data/ontology/core.py 82.22% <89.65%> (+3.48%) ⬆️
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d7cbcfa...c3ae595. Read the comment docs.

@hunterhector hunterhector merged commit 0ca9602 into asyml:master Sep 22, 2021
@hunterhector hunterhector deleted the slow branch September 22, 2021 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Serialization and entry attribute assignment are slow

1 participant

@hunterhector