Thanks to visit codestin.com
Credit goes to github.com

BigTable: Adding a row generator on a table. #4679

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

tseaver merged 14 commits into googleapis:master from zakons:feature/row_iterator

Jan 26, 2018

Contributor

aneepct commented Dec 29, 2017 •

edited by tseaver

Loading

Adding a row generator on a table. This will allow iteration over the rows in a table instead of reading the rows into an internal dictionary first. As soon as a row has been validated, it is available in the iterator, by adding a row generator on a table. This will allow iteration over the rows in a table instead of reading the rows into an internal dictionary first. As soon as a row has been validated, it is available in the iterator, by using yield in the generator. The read_rows() method on table will still return PartialRowsData, and consume_all() will now use the row generator to process the ReadRowsResponses to populate the rows or in an internal dictionary. The yield_rows() method on table provides a row iterator and does not store any of the rows internally. The rows are available as soon as they are read from the channel and validated.

Closes #4586.


          Adding a row generator on a table. This will allow iteration over the…

719720e

… rows in a table instead of reading the rows into an internal dictionary first. As soon as a row has been validated, it is available in the iterator, by using yield in the generator.

aneepct requested a review from lukesneeringer as a code owner

December 29, 2017 06:35

googlebot added the cla: yes label

Aneep Tandel and others added 2 commits

December 29, 2017 12:12


          Fix merged code

1f8f565


          Fix merges / unit tests

89ed0c5

googlebot commented Jan 2, 2018

So there's good news and bad news.

👍 The good news is that everyone that needs to sign a CLA (the pull request submitter and all commit authors) have done so. Everything is all good there.

😕 The bad news is that it appears that one or more commits were authored by someone other than the pull request submitter. We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that here in the pull request.

Note to project maintainer: This is a terminal state, meaning the cla/google commit status will not change from this State. It's up to you to confirm consent of the commit author(s) and merge this pull request when appropriate.

googlebot added cla: no and removed cla: yes labels

dhermes reviewed

View reviewed changes

bigtable/google/cloud/bigtable/row_data.py Outdated

@@ @@ -194,8 +194,66 @@ class PartialRowsData(object): @@
                   def __init__(self, response_iterator):
                       self._response_iterator = response_iterator
+                      self.generator = YieldRowsData(response_iterator)

This comment was marked as spam.

Sign in to view

bigtable/google/cloud/bigtable/row_data.py Outdated

+                  def __eq__(self, other):
+                      if not isinstance(other, self.__class__):
+                          return False

This comment was marked as spam.

Sign in to view

bigtable/google/cloud/bigtable/row_data.py Outdated

+                      return other._response_iterator == self._response_iterator
+                  def __ne__(self, other):
+                      return not self.__eq__(other)

This comment was marked as spam.

Sign in to view

bigtable/google/cloud/bigtable/row_data.py Outdated

+                      """
+                      # NOTE: To avoid duplicating large objects, this is just the
+                      #       mutable private data.
+                      return self._rows

This comment was marked as spam.

Sign in to view

bigtable/google/cloud/bigtable/row_data.py Outdated

+                  :param response_iterator: A streaming iterator returned from a
+                                            ``ReadRows`` request.
+                  """
+                  START = "Start"  # No responses yet processed.

This comment was marked as spam.

Sign in to view

bigtable/google/cloud/bigtable/row_data.py

+                  def state(self):
+                      """State machine state.
+                      :rtype: str

This comment was marked as spam.

Sign in to view

bigtable/google/cloud/bigtable/row_data.py Outdated

@@ @@ -425,18 +451,9 @@ def _copy_from_previous(self, cell): @@
                               cell.row_key = previous.row_key
                           if not cell.family_name:
                               cell.family_name = previous.family_name
-                          # NOTE: ``cell.qualifier`` **can** be empty string.
-                          if cell.qualifier is None:
+                          if not cell.qualifier:

This comment was marked as spam.

Sign in to view

bigtable/tests/unit/test_row_data.py Outdated

		@@ -189,117 +189,85 @@ def test_row_key_getter(self):


		class TestPartialRowsData(unittest.TestCase):
		@staticmethod

This comment was marked as spam.

Sign in to view

bigtable/tests/unit/test_row_data.py Outdated

                   @staticmethod
-                  def _get_target_class():
+                  def _get_partial_target_class():

This comment was marked as spam.

Sign in to view

tseaver added the api: bigtable label

Aneep Tandel and others added 3 commits

January 3, 2018 10:30


          Add test for yield_rows

346f8f8


          Refactore test_row_data.py

1e6ac6d


          Add code review changes on pull request

42eaa13

Contributor

sduskis commented Jan 9, 2018

Since consume_next() was a public method, does that mean that we need to maintain backwards compatibility? If so, can we just have it read a single row via the generator?

Contributor

dhermes commented Jan 9, 2018

does that mean that we need to maintain backwards compatibility

Not until 1.0 (current version is 0.28.1). We typically try to provide a "warning" period though before breaking an interface, but in a case like this it might not be possible and it'd be fine to make a breaking change in the release (we can just include that in the release notes).


          Create Performance Test

88cdf48

chemelnucfin assigned dhermes and chemelnucfin

chemelnucfin added the type: feature request label

Contributor

zakons commented Jan 18, 2018

All review requests made but not finding a solution for providing unit tests that provide coverage of iterator / generator code. Please provide any suggestions for this so this can be completed.


          Merge branch 'feature/row_iterator' of https://github.com/zakons/goog…

49f44c3

…le-cloud-python into feature/row_iterator

zakons mentioned this pull request

BigTable: PartialRowsData iterator #4761

Closed

Contributor

chemelnucfin commented Jan 21, 2018

This would close #1812 from discussion in #4761

aneepct changed the title ~~Adding a row generator on a table.~~ To: BigTable: Adding a row generator on a table.

zakons mentioned this pull request

Bigtable PartialRowsData is not an iterable. #1812

Closed

aneepct changed the title ~~To: BigTable: Adding a row generator on a table.~~ BigTable: Adding a row generator on a table.

Aneep Tandel added 4 commits

January 23, 2018 09:44


          Add test cases for yield_rows

ce5dfbf


          Remove test

e6895e7


          Move test case

67e09d3


          Removed unwated test codes

a9c133c


          Improve code coverage.

756e57c

Contributor

zakons commented Jan 24, 2018

@tseaver @sduskis We are getting 100% code coverage in PyCharm, which uses coverage.py:

However, circleci is identifying lines 338-341 as NOT having coverage. This is the only thing holding up the merge for this PR. Any suggestions or ideas would be very much appreciated so we can complete this PR.

Contributor

tseaver commented Jan 24, 2018

@zakons The output on CircleCI is 338 -> 341, which is a branch miss -- a normal gap would be denoted 338-341. Can you configure the PyCharm -> coverage bit so that it enables branch coverage?

To mitigate: the branch is the test if self._cell: -- that condition is always true, and therefore the call to self._save_current_cell() is always executed. We need a test which gets to that point without having a cell assigned, or else (if that is impossible) we can just take out the if altogether.

chemelnucfin mentioned this pull request

BigTable: Memory usage abnormally increasing when using table.read_rows method #4586

Closed


          Fix code coverage

13e0452

Contributor

zakons commented Jan 26, 2018

@tseaver That was the issue - branch coverage. Your help is very much appreciated. Can you please fix the cla so it is green (see OK on this from @aneepct) and I believe we are ready to merge. Thanks!

tseaver merged commit 166278f into googleapis:master

Contributor

sduskis commented Jan 26, 2018

Thanks!

chemelnucfin added cla: yes and removed cla: no labels

zakons deleted the feature/row_iterator branch

February 5, 2018 22:06

zunger-humu commented Mar 5, 2018 •

edited

Loading

So, guess which breaking change didn't get mentioned in the release notes? ;)

Also, the documentation is now out-of-date, so it ended up taking some source diving to figure out how to use the new code. I think this means that Table.read_rows needs to be marked as deprecated, or its documentation otherwise modified to indicate when you should use it and when you should use yield_rows. TBH, from the code I think read_rows can just be ripped out; if you need to pull all the information, you can just wrap a call to yield_rows in list().

theacodes unassigned dhermes and chemelnucfin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: bigtable cla: yes type: feature request