Code

feat: optimize read_rows#852

Code

Merged

daniel-sanche merged 79 commits intogoogleapis:v3googleapis/python-bigtable:v3from

daniel-sanche:optimize_read_rows_2daniel-sanche/python-bigtable:optimize_read_rows_2Copy head branch name to clipboard

Aug 17, 2023

Commits

Commits on Jul 24, 2023

• added function for populating shard

  daniel-sanche

  committed
  18650af

  Copy full SHA for 18650af

• Merge branch 'v3' into benchmarks_prod

  daniel-sanche

  committed
  df05171

  Copy full SHA for df05171

• got table population working

  daniel-sanche

  committed
  4db25b3

  Copy full SHA for 4db25b3

• finished table population

  daniel-sanche

  committed
  28cc1f4

  Copy full SHA for 28cc1f4

Commits on Jul 25, 2023

• pulled out table setup fixtures from system tests

  daniel-sanche

  committed
  9e97ac0

  Copy full SHA for 9e97ac0

• get system tests working

  daniel-sanche

  committed
  52550cd

  Copy full SHA for 52550cd

• added some basic throughput code

  daniel-sanche

  committed
  a191236

  Copy full SHA for a191236

• count operations

  daniel-sanche

  committed
  31dbbd9

  Copy full SHA for 31dbbd9

• added splits to table

  daniel-sanche

  committed
  eac206a

  Copy full SHA for eac206a

• extracted cluster config

  daniel-sanche

  committed
  ea771de

  Copy full SHA for ea771de

• chanted test name

  daniel-sanche

  committed
  2eda0ac

  Copy full SHA for 2eda0ac

• improved benchmark output

  daniel-sanche

  committed
  f0649fd

  Copy full SHA for f0649fd

• fixed population to use new format

  daniel-sanche

  committed
  fbf2636

  Copy full SHA for fbf2636

• added sharded scan benchmark

  daniel-sanche

  committed
  21df0c2

  Copy full SHA for 21df0c2

• more accurate time tracking

  daniel-sanche

  committed
  91ef008

  Copy full SHA for 91ef008

Commits on Jul 26, 2023

• added script to deploy to GCE

  daniel-sanche

  committed
  f91750f

  Copy full SHA for f91750f

• limit to one qualifier

  daniel-sanche

  committed
  0236f85

  Copy full SHA for 0236f85

• added separate test for point reads

  daniel-sanche

  committed
  fd94350

  Copy full SHA for fd94350

• made row size configurable

  daniel-sanche

  committed
  82ecaf2

  Copy full SHA for 82ecaf2

• don't timeout benchmarks at deadline

  daniel-sanche

  committed
  a35f985

  Copy full SHA for a35f985

• added fastapi benchmark

  daniel-sanche

  committed
  a440d76

  Copy full SHA for a440d76

• added driver for fastapi class

  daniel-sanche

  committed
  d0fcdb9

  Copy full SHA for d0fcdb9

Commits on Jul 27, 2023

• delete VM on complete; improved formatting

  daniel-sanche

  committed
  0bb70ef

  Copy full SHA for 0bb70ef

• improved fastapi benchmark

  daniel-sanche

  committed
  3f3f59a

  Copy full SHA for 3f3f59a

• run tests concurrently

  daniel-sanche

  committed
  5a6ffad

  Copy full SHA for 5a6ffad

• saved test_system changes

  daniel-sanche

  committed
  604071f

  Copy full SHA for 604071f

• added profling to point reads

  daniel-sanche

  committed
  3686ef6

  Copy full SHA for 3686ef6

• add duration to run script

  daniel-sanche

  committed
  c42f596

  Copy full SHA for c42f596

Commits on Jul 28, 2023

• build async generator manually

  daniel-sanche

  committed
  e2ac258

  Copy full SHA for e2ac258

• lazy parse cell data

  daniel-sanche

  committed
  1228c8a

  Copy full SHA for 1228c8a

• added profiling to scans

  daniel-sanche

  committed
  1a3a8e7

  Copy full SHA for 1a3a8e7

• removed iscouroutine check

  daniel-sanche

  committed
  0659161

  Copy full SHA for 0659161

• added legacy benchmark

  daniel-sanche

  committed
  ce2b733

  Copy full SHA for ce2b733

• added thread to legacy tests

  daniel-sanche

  committed
  4714e8a

  Copy full SHA for 4714e8a

• cut out row adapter

  daniel-sanche

  committed
  3111543

  Copy full SHA for 3111543

• lazy load rows

  daniel-sanche

  committed
  c717a0b

  Copy full SHA for c717a0b

Commits on Jul 29, 2023

• simplify generator layers for non-stremaing

  daniel-sanche

  committed
  b5cb253

  Copy full SHA for b5cb253

• change pool size

  daniel-sanche

  committed
  aa4bed4

  Copy full SHA for aa4bed4

Commits on Jul 30, 2023

• disabled lazy loading (75k rps)

  daniel-sanche

  committed
  613b032

  Copy full SHA for 613b032

• only check for last_scanned when there are no chunks (81k rps)

  daniel-sanche

  committed
  8e37967

  Copy full SHA for 8e37967

Commits on Jul 31, 2023

• refactored chunk parsing (80k rps)

  daniel-sanche

  committed
  6775e2b

  Copy full SHA for 6775e2b

• removed subclass for row

  daniel-sanche

  committed
  537f4ff

  Copy full SHA for 537f4ff

• added igor's row merger

  daniel-sanche

  committed
  58b3fbb

  Copy full SHA for 58b3fbb

Commits on Aug 7, 2023

• use stream by default

  daniel-sanche

  committed
  a7478ab

  Copy full SHA for a7478ab

• use our models

  daniel-sanche

  committed
  689f072

  Copy full SHA for 689f072

• wrapped in object

  daniel-sanche

  committed
  107b8c5

  Copy full SHA for 107b8c5

Commits on Aug 8, 2023

• updated row merger for acceptance tests

  daniel-sanche

  committed
  d76a7dc

  Copy full SHA for d76a7dc

• improve performance

  daniel-sanche

  committed
  1d9a475

  Copy full SHA for 1d9a475

• removed HasField from family

  daniel-sanche

  committed
  37bc3a0

  Copy full SHA for 37bc3a0

• added request revision logic

  daniel-sanche

  committed
  ea95dcc

  Copy full SHA for ea95dcc

• added back split cell verifications

  daniel-sanche

  committed
  9b60693

  Copy full SHA for 9b60693

• got all acceptance tests passing

  daniel-sanche

  committed
  24de29e

  Copy full SHA for 24de29e

• track last yielded key

  daniel-sanche

  committed
  9c9ba5e

  Copy full SHA for 9c9ba5e

• track reamining count

  daniel-sanche

  committed
  5d49ad8

  Copy full SHA for 5d49ad8

• added retries

  daniel-sanche

  committed
  5ecfa3e

  Copy full SHA for 5ecfa3e

• added exception conversion

  daniel-sanche

  committed
  13603ea

  Copy full SHA for 13603ea

• moved into single function

  daniel-sanche

  committed
  b40e1cb

  Copy full SHA for b40e1cb

Commits on Aug 9, 2023

• Fixed style issues

  daniel-sanche

  committed
  0f58066

  Copy full SHA for 0f58066

• Merge branch 'v3' into optimize_read_rows_2

  daniel-sanche

  committed
  07341da

  Copy full SHA for 07341da

• fixed row and helpers tests

  daniel-sanche

  committed
  90b85b3

  Copy full SHA for 90b85b3

• fixed some read rows tests

  daniel-sanche

  committed
  4298e38

  Copy full SHA for 4298e38

• comments and cleanup

  daniel-sanche

  committed
  62f1bf7

  Copy full SHA for 62f1bf7

• fixed issues with conformance tests

  daniel-sanche

  committed
  98510c3

  Copy full SHA for 98510c3

• improved proto conversion

  daniel-sanche

  committed
  4cf4aab

  Copy full SHA for 4cf4aab

Commits on Aug 10, 2023

• fixed read_rows tests

  daniel-sanche

  committed
  0f0ccee

  Copy full SHA for 0f0ccee

• fixed tests

  daniel-sanche

  committed
  42f44da

  Copy full SHA for 42f44da

• optimizing query class

  daniel-sanche

  committed
  3d1804c

  Copy full SHA for 3d1804c

• keep filter and limit out of proto storage

  daniel-sanche

  committed
  b0882af

  Copy full SHA for b0882af

• fixed tests

  daniel-sanche

  committed
  2f2286d

  Copy full SHA for 2f2286d

• skip check when not necessary

  daniel-sanche

  committed
  fbe298e

  Copy full SHA for fbe298e

• experiment: yield in row batches

  daniel-sanche

  committed
  2a82343

  Copy full SHA for 2a82343

• Revert "experiment: yield in row batches"

  Show description for 4e6dd6f
  daniel-sanche

  committed
  4e6dd6f

  Copy full SHA for 4e6dd6f

• removed benchmarks dir

  daniel-sanche

  committed
  daa7a59

  Copy full SHA for daa7a59

• fixed type annotation

  daniel-sanche

  committed
  e91d693

  Copy full SHA for e91d693

• added slots to query

  daniel-sanche

  committed
  cb28451

  Copy full SHA for cb28451

Commits on Aug 14, 2023

• use separate instances for each run

  daniel-sanche

  committed
  1dd4d0c

  Copy full SHA for 1dd4d0c

• made _ReadRowOperation into slots

  daniel-sanche

  committed
  6d2dab6

  Copy full SHA for 6d2dab6

• clean up style issues

  daniel-sanche

  committed
  309405a

  Copy full SHA for 309405a

• removed commented test

  daniel-sanche

  committed
  e36d70b

  Copy full SHA for e36d70b