Support configuring flush_count and max_row_bytes of WriteToBigTable #34761
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
c507fe4 to
483ba64
Compare
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
7262e6d to
19c2071
Compare
19c2071 to
b8ecbb0
Compare
|
Hi @nguymin4 , the 429 error is coming from the bigtable server side because there are too many outstanding requests. And I don't think updating the flsuh_count will solve this issue. In the issue description you mentioned that retrying the mutations are causing the increased latencies. Should the mutations be retried? Would disabling retry help? |
|
Hi @mutianf, @andre-sampaio, we currently have the same output size for all Dataflow jobs but with 2.46.0 there is no error at all, comparing to 2.64.0. So I don't know what wrong with this besides trying to increase the flush_count -> reduce number of requests -> avoid 429 errors. Does this make sense to you? In my opinion, if upgrading SDK version cause performance issue then the error is with SDK and not Bigtable server itself, isn't it? I'm out of ideas as well. This is the biggest blocker that I need to pin our version to 2.46.0. |
|
(sorry I accidentaly deleted my last comment, adding it again) Thanks for your contribution! The error you are seeing indicates you are sending more mutations than your bt cluster can handle, which causes requests to pile up on the server side until they can be processed and once too many requests get queued you start seeing this error message. Generally speaking increasing your batch sizes can make the problem worse, specially in bursty workloads. A good signal for whether or not this is what is happening for you is checking the cpu usage for your cluster during these bursts and seeing if they are at ~100%. I don't see anything wrong with this PR, but you may want to instead add knobs for MAX_OUTSTANDING_ELEMENTS and MAX_OUTSTANDING_BYTES and try reducing those (which one depends on your use case, if you are writing many small rows reduce elements, if you are writing large rows reduce outstanding bytes). Or alternatively reduce the number of workers on your beam job. Let me know if this explanation helps |
|
@andre-sampaio Could you check my answer above if it makes sense to you? In my opinion, with same output size, upgrading SDK version SHOULD NOT cause any performance degradation. About those two settings you suggested, I actually don't know, it may or may not help. But in general I feel like python-bigtable library does not welcome for contributor (I tried in the past) so it will take months or years or not ever for this issue to be resolved. I also created a Google support ticket for this as this is consider a huge business impact to us. Should I escalate this to someone else? |
|
hey there @nguymin4! I agree with you that the SDK is likely the culprit here. I'm sorry about your experience with In the meantime I don't mind if we merge this PR but I'm afraid it won't help much. I would also suggest trying temporarily to reduce the number of workers in your job (if this is something feasible). If you want to open a support ticket for this it would help us investigating your particular case, which can help us to make more directed suggestions. |
|
Thanks @andre-sampaio, I believe you understand apache-beam and python-bigtable better than me, so I would let you decide the fate of this PR. As I mentioned, I'm totally out of idea now because this is a known issue to me for more than 1.5 years. This PR is simply a desperate attempt of mine as I didn't see anything news has been added. I did contact Google support, with details about successful and erroneous Dataflow job. I do hope to get this resolved soon. For now I roll back to 2.46.0 to avoid triggering our error alert. |
|
@nguymin4 can you ask the support ticket owner to consult the BigTable team? |
|
@andre-sampaio @liferoad I got an suggestion from Vasant D - Bigtable team.
So I guess he is referring to lower the current flush_count from 1000 to 100? But again I don't know why this doesn't work as expected after 2.46.0. |
Since I do not know the support ticket number to get more details about your job, I guess you could lower flush_count or reduce the number of workers as @andre-sampaio mentioned above. If you want to check whether your PR works, you could easily test this by overriding |
|
@liferoad @andre-sampaio We only have max 1 dataflow job at a time. Each job has max 1 worker with machine type |
We got the support ticket now and will check these jobs with more details. Please also share the successful job ids and the failed ones through the support ticket. |
andre-sampaio
approved these changes
Jun 4, 2025
|
We will migrate the batcher that beam is using the the new MutationsBatcher which provides control of total inflight mutations. In the meantime I think we can merge this PR to get this mitigation in for the next release in case the migration doesn't happen by then. |
liferoad
reviewed
Jun 4, 2025
| SchemaAwareExternalTransform.discover_config( | ||
| self._expansion_service, self.URN)) | ||
|
|
||
| self._flush_count = flush_count |
There was a problem hiding this comment.
we probably need to log warnings if using cross-lang is true since these two parameters are not supported.
Note if you migrate to MutationsBatcher, you are not supposed to remove these public parameters. |
|
I am going to merge this for now to unblock the users. |
razvanculea
pushed a commit
to razvanculea/beam
that referenced
this pull request
Jun 17, 2025
shunping
added a commit
that referenced
this pull request
Jul 30, 2025
* Add support for streaming writes in IOBase (Python) * add triggering_frequency in iobase.Sink * fix whitespaces/newlines * fixes per #35137 (review) * implement pre_finalize_windowed and finalize_windowed_write in FileBasedSink handle file naming based on windowing for streaming in internal methods TextIO - expose the new filebasedsink capabilities to write in streaming * update changes.md * fix formatting * space formatting * space nagging * keep in sync with refactor in #35137 * keep in sync with iobase changes in #35137 * [Website] add akvelon case study (#34943) * feat: add akvelon logo * feat: add akvelon case study * fix: remove white space * feat: add akvelon to main page * feat: use new images * fix: typos * fix: change order of akvelon case-study * fix: update text * fix: update mainPage text * fix: update images * fix: about akvelon section update * fix: update akvelon card * fix: update akvelon header * fix: update code tag * fix: update about akvelon * fix: update date and order * fix: add link and change img * fix: change CDAP text * fix: add bold weight * fix: solve conflicts * fix: remove unused code * fix: delete whitespace * fix: indents format * fix: add bold text --------- Co-authored-by: Bulat Safiullin <[email protected]> * added the rate test for GenerateSequence (#35108) * added the rate test for GenerateSequence * keep the master yaml * Re-enable logging after importing vllm. (#35103) Co-authored-by: Claude <[email protected]> * Deprecate Java 8 (#35064) * Deprecate Java 8 * Java 8 client now using Java 11 SDK container * adjust non LTS fallback to use the next LTS instead of the nearest LTS. Previously Java18 falls back to Java17, which won't work * Emit warning when Java 8 is used. Java8 is still supported until Beam 3.0 * Clean up subproject build file requiring Java9+ * Require java 11 to build SDK container * fix workflow * Simplify XVR test workflow * Fix Samza PVR * Remove beam college banners (#35123) * feat: change text (#35130) Co-authored-by: Bulat Safiullin <[email protected]> * Update Custom Remote Inference example to use RemoteModelHandler (#35066) * Update Custom Remote Inference example to use RemoteModelHandler * restore old kernel metadata * Update examples/notebooks/beam-ml/custom_remote_inference.ipynb Co-authored-by: Danny McCormick <[email protected]> * Add DLQ --------- Co-authored-by: Danny McCormick <[email protected]> * Remove Java 8 container (#35125) * add extra_transforms block documentation to chain transform documentation (#35101) * add note about testing (#35075) * fix whitespaces/newlines * fixes per #35137 (review) * implement pre_finalize_windowed and finalize_windowed_write in FileBasedSink handle file naming based on windowing for streaming in internal methods TextIO - expose the new filebasedsink capabilities to write in streaming * update changes.md * fix formatting * space formatting * space nagging * keep in sync with refactor in #35137 * keep in sync with iobase changes in #35137 * [Website] update akvelon case study: update text and fix landing page (#35133) * feat: change text * fix: add learn more to quotes Akvelon --------- Co-authored-by: Bulat Safiullin <[email protected]> * Fix PostCommit Python Xlang IO Dataflow job (#35131) * Add support for iterable type * Fix formatting * Bump google.golang.org/grpc from 1.72.0 to 1.72.2 in /sdks (#35113) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.72.0 to 1.72.2. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.72.0...v1.72.2) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.72.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cloud.google.com/go/bigquery from 1.67.0 to 1.69.0 in /sdks (#35061) Bumps [cloud.google.com/go/bigquery](https://github.com/googleapis/google-cloud-go) from 1.67.0 to 1.69.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@spanner/v1.67.0...spanner/v1.69.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/bigquery dependency-version: 1.69.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add known issues. (#35138) * Add known issues. * Add fixes notes. --------- Co-authored-by: Claude <[email protected]> * Bump @octokit/plugin-paginate-rest and @octokit/rest (#34167) Bumps [@octokit/plugin-paginate-rest](https://github.com/octokit/plugin-paginate-rest.js) to 11.4.3 and updates ancestor dependency [@octokit/rest](https://github.com/octokit/rest.js). These dependencies need to be updated together. Updates `@octokit/plugin-paginate-rest` from 2.17.0 to 11.4.3 - [Release notes](https://github.com/octokit/plugin-paginate-rest.js/releases) - [Commits](octokit/plugin-paginate-rest.js@v2.17.0...v11.4.3) Updates `@octokit/rest` from 18.12.0 to 21.1.1 - [Release notes](https://github.com/octokit/rest.js/releases) - [Commits](octokit/rest.js@v18.12.0...v21.1.1) --- updated-dependencies: - dependency-name: "@octokit/plugin-paginate-rest" dependency-type: indirect - dependency-name: "@octokit/rest" dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Explicitly handle singleton iterators instead of using helper and catching exceptions which may be from generating iterable (#35124) * Build last snapshot against RC00 tag instead of release branch (#35142) The last snapshot Beam Java SDK (aka "RC00" release) build is triggered manually, to verify a RC1 build will be successful. It has been built against release-2.xx branch, where the Dataflow container tag replaced from beam-master to the 2.xx.0 However, the versioned containers are not yet released, causing a timing gap that Beam 2.xx.0-SNAPSHOT won't work on Dataflow between release branch cut and RC1 rolled out to Dataflow Since we now have a v2.xx.0-RC00 tag, build RC00 against this tag resolves the issue. * Bump nodemailer from 6.7.5 to 6.9.9 in /scripts/ci/issue-report (#35143) Bumps [nodemailer](https://github.com/nodemailer/nodemailer) from 6.7.5 to 6.9.9. - [Release notes](https://github.com/nodemailer/nodemailer/releases) - [Changelog](https://github.com/nodemailer/nodemailer/blob/master/CHANGELOG.md) - [Commits](nodemailer/nodemailer@v6.7.5...v6.9.9) --- updated-dependencies: - dependency-name: nodemailer dependency-version: 6.9.9 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix tests affected by Java 8 container turn down (#35145) * Fix tests affected by Java 8 container turn down * still use Java 8 for Samza runner * Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#35146) Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.79.3 to 1.80.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@service/s3/v1.79.3...service/s3/v1.80.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-version: 1.80.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix jdbc transform validation (#35141) * fix jdbc transform validation * add test * annotations * spotless * Fix Java Example ARM PostCommit * fix: add missed word (#35163) Co-authored-by: Bulat Safiullin <[email protected]> * Add postcommit yaml xlang workflow and split tests accordingly (#35119) * move precommit yaml xlang to postcommit * add json file to trigger path * update pull request targets * add readme workflow changes * add cloud sdk setup * switch out to ubuntu-latest * add back precommit workflow * switch names and add postcommit * switch out to postCommitYamlIT * add test_files_dir flag * add conftest.py file for capturing directory flag * shift yaml tests around to appropriate locations * add back precommit to readme * add license for conftest.py * revert precommit to previous name * remove github.event.comment.body trigger * Replace usages of deprecated pkg_resources package (#35153) * Remove usages of deprecated pkg_resources package * use stdlib importlib.resources * remove extra comma * linting * import order * Improve error message when accidentally using PBegin/Pipeline (#35156) * Create test * Implement new error message * Add beam.Create into unit test pipeline * add friendly error message for when transform is applied to no output (#35160) * add friendly error message for when transform is applied to no output * update test name * Fix pubsub unit tests that depend on old behavior * Add warning if temp location bucket has soft delete enabled for Go SD… (#34996) * Add warning if temp location bucket has soft delete enabled for Go SDK (resolves #31606) * Corrected Formatting * Applied suggested changes * Constrain DequeCoder type correctly, as it does not support nulls The DequeCoder uses ArrayDeque internally, which disallows null elements. We could switch Deque implementations, but this change is better. To quote the JDK docs: "While Deque implementations are not strictly required to prohibit the insertion of null elements, they are strongly encouraged to do so. Users of any Deque implementations that do allow null elements are strongly encouraged not to take advantage of the ability to insert nulls. This is so because null is used as a special return value by various methods to indicated that the deque is empty." * Do not overwrite class states if a cached dynamic class is returned in cloudpickle.load (#35063) * Fix class states overwritten after cloudpickle.load * Fix further * Fix lint * Make SDK harness change effective on Iceberg Dataflow test (#35173) * Fix beam_PostCommit_Java_Examples_Dataflow_V2 (#35172) * [YAML]: Update postgres IT test and readme (#35169) * update postgres test without driver_class_name * update readme on how to run integration tests * fix misspelling * fix trailing whitespace * Bump Java beam-master container (#35170) * Make WindowedValue a public interface The following mostly-automated changes: - Moved WindowedValue from util to values package - Make WindowedValue an interface with companion class WindowedValues * Run integration tests for moving WindowedValue and making public * Add timer tests to make sure event-time timer firing at the right time. (#35109) * Add timer tests to make sure event-time timer firing at the right time. * Add more tests. * Disable the failed event-time timer tests for FnApiRunner. * Fix lints and reformat. * Disable another new test in FnApiRunnerTest and PortableRunnerTest due to flakiness. * Disable a new test in FlinkRunnerTest * Take out the early firing test case because it depends on bundle size. * change to ubuntu-20.04 (#35182) * Fix IntelliJ sync project failure due to circular Beam dependency (#35167) * Fix IntelliJ sync project failure due to circular Beam dependency * address comments * Update workflows categories (#35162) * Add cloudpickle coder. (#35166) * Move examples from sql package (#35183) * Fix the beam interactive install problem when on Google Colab (#35148) * Fix Google Colab Issue * Update CHANGES.md * Bump github.com/docker/docker in /sdks (#35112) Bumps [github.com/docker/docker](https://github.com/docker/docker) from 28.0.4+incompatible to 28.2.2+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v28.0.4...v28.2.2) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-version: 28.2.2+incompatible dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/nats-io/nats-server/v2 from 2.11.3 to 2.11.4 in /sdks (#35161) Bumps [github.com/nats-io/nats-server/v2](https://github.com/nats-io/nats-server) from 2.11.3 to 2.11.4. - [Release notes](https://github.com/nats-io/nats-server/releases) - [Changelog](https://github.com/nats-io/nats-server/blob/main/.goreleaser.yml) - [Commits](nats-io/nats-server@v2.11.3...v2.11.4) --- updated-dependencies: - dependency-name: github.com/nats-io/nats-server/v2 dependency-version: 2.11.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Touch JPMS test trigger file * Use staged SDK harness & Dataflow worker jar in JPMS tests * Bump cloud.google.com/go/storage from 1.52.0 to 1.55.0 in /sdks (#35114) Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.52.0 to 1.55.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@spanner/v1.52.0...spanner/v1.55.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-version: 1.55.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/nats-io/nats.go from 1.42.0 to 1.43.0 in /sdks (#35147) Bumps [github.com/nats-io/nats.go](https://github.com/nats-io/nats.go) from 1.42.0 to 1.43.0. - [Release notes](https://github.com/nats-io/nats.go/releases) - [Commits](nats-io/nats.go@v1.42.0...v1.43.0) --- updated-dependencies: - dependency-name: github.com/nats-io/nats.go dependency-version: 1.43.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Move non-dill specific code out of dill_pickler.py (#35139) * Create consistent_pickle.py * Update dill_pickler.py * Update consistent_pickle.py * Add Apache license to consistent_pickle.py * Update dill_pickler.py * Update and rename consistent_pickle.py to code_object_pickler.py * Update dill_pickler.py to change consistent_pickle to code_object_pickler * Fix formatting issue in code_object_pickler.py * Fix formatting issue in dill_pickler.py * Remove apache license from code_object_pickler.py * Add apache license to code_object_pickler.py * Fix some lints introduced in a recent PR. (#35193) * small filesystem fixes (#34956) * Fix typehint for get_filesystem * Fix bug with creating files in cwd with LocalFileSystem * lint * revert precommit config change * isort * add file to start PR builder * Revert "add file to start PR builder" This reverts commit f80b361. * fix isort again * [YAML] Add a spec provider for transforms taking specifiable arguments (#35187) * Add a test provider for specifiable and try it on AnomalyDetection. Also add support on callable in spec. * Minor renaming * Fix lints. * Touch trigger files to test WindowedValueReceiver in runners * Introduce WindowedValue receivers and consolidate runner code to them We need a receiver for WindowedValue to create an OutputBuilder. This change introduces the interface and uses it in many places where it is appropriate, replacing and simplifying internal runner code. * Eliminate nullness errors from ByteBuddyDoFnInvokerFactory and DoFnOutputReceivers * Fix null check when fetching driverJars from value provider * Fix PostCommit Python ValidatesRunner Samza / Spark jobs (#35210) * Skip Samza and Spark runner tests * Fix formatting * Update pypi documentation 30145 (#34329) * Add Pypi friendly documentation * provided full link path * Add more YAML examples involving Kafka and Iceberg (#35151) * Add more YAML examples involving Kafka and Iceberg * Fix some missed details from rebasing * Adding unit tests for YAML examples * Clean up and address PR comments * Formatting * Formatting * Evaluate namedTuples as equivalent to rows (#35218) * Evaluate namedTuples as equivalent to rows * lint * Add a new experiment flag to enable real-time clock as processing time. (#35202) * Touch trigger files for lightweight runners * Eliminate WindowedValue.getPane() in preparation for making it a user-facing interface * Do not fail if there were failed deletes (#35222) * Force FnApiRunner in cases where prism can't handle use case (#35219) * Force FnApiRunner in cases where prism can't handle use case * yapf * Bump golang.org/x/net from 0.40.0 to 0.41.0 in /sdks (#35206) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.40.0 to 0.41.0. - [Commits](golang/net@v0.40.0...v0.41.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.41.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix incorrect typehints generated by FlatMap with default identity function (#35164) * create unit test * add some unit tests * fix bug where T is considered iterable * update strip_iterable to return Any for "stripped iterable" type of TypeVariable * remove typehint from identity function and add a test to test for proper typechecking * Move callablewrapp typehint test * Remove print * isort * isort * return any for yielded type of T * Parse values returned from Dataflow API to BoundedTrieData (#34738) * Parse struct returned from Dataflow API to BoundedTrieData fix checkstyle Use getBoundedTrie add debug log adapt to ArrayMap * Add test, clean up * Fix test: setTrie -> setBoundedTrie * Fix comments * Remove breaking PDone change (#35224) * Remove breaking PDone change * Update pipeline.py * Update pipeline.py * Update pipeline_test.py * Move none check * yapf * fmt * Generic Postgres + Cloudsql postgres embeddings. (#35215) * Add base Postgres vector writer, CloudSQL vector writer and refactor. * Trigger tests. * Linter fixes. * Fix test * Add back tests. Update changes.md. Fix unrelated lint. * Drop test tables. * Fix test * Fix tests. * Fix test. * Fix tests. --------- Co-authored-by: Claude <[email protected]> * Allow only one thread at a time to start the VLLM server. (#35234) * [IcebergIO] Create namespaces if needed (#35228) * create namespaces dynamically * cleanup namespaces in ITs * optimization * Support configuring flush_count and max_row_bytes of WriteToBigTable (#34761) * Update CHANGES.md (#35242) Highlight improvements to the vllm model handler. * [Beam SQL] Implement Catalog and CatalogManager (#35223) * beam sql catalog * api adjustment * cover more naming syntax (quotes, backticks, none) * spotless * fix * add documentation and cleanup * rename to dropCatalog; mark BeamSqlCli @internal; rename to EmptyCatalogManager * use registrars instead; remove initialize() method from Catalog * cleanupo * [IcebergIO] Fix conversion logic for arrays of structs and maps of structs; fix output Schema resolution with column pruning (#35230) * fix complex types; filx filter schema logic * update Changes * add missing changes from other PRs * trigger ITs * make separate impl for iterable * fix * fix long_description when the md file cannot be found (#35246) * fix long_description when the md file cannot be found * yapf * [IcebergIO] Create tables with a partition spec (#34966) * create table with partition spec * add description and regenerate config doc; trigger tests * spotless * log partition spec * fixes * Fix typo java-dependencies.md (#35251) * Adding project and database support in write transform for firestoreIO (#35017) * Adding project and database support in write transform for firestoreIO * Spotless Apply * Resolving comments * Removing useless comments * public to private * Spotless apply * Fix a logical type issue about JdbcDateType and JdbcTimeType (#35243) * Fix a logical type issue about JdbcDateType * Fix typo and also fix the logical class for java time. * Get rid of the workaround on logical type registration. Trigger tests. * Fix lints. * Remove internal code. (#35239) Co-authored-by: Claude <[email protected]> * enable setting max_writer_per_bundle for avroIO and other IO (#35092) * enable setting max_writer_per_bundle for avroIO and other IO * enable setting max_writer_per_bundle for avroIO and other IO * enable for TFRecordIO and corrections * Updated standard_external_transforms.yaml * [Java FnApi] Fix race in BeamFnStatusClient by initializing all fields before starting rpc. (#35252) * [Java FnApi] Fix race in BeamFnStatusClient by initializing all fields before starting rpc. * spotless * [IcebergIO] Integrate with Beam SQL (#34799) * add Iceberg table provider and tests * properties go in the tableprovider initialization * tobuilder * streaming integration test * spotless * extend test to include multi nested types; fix iceberg <-> conversion logic * add to changes.md * spotless * fix tests * clean * update CHANGES * add projection pushdown and column pruning * spotless * fixes * fixes * updates * sync with HEAD and use new Catalog implementation * mark new interfaces @internal * spotless * fix unparse method * update changes (#35256) * [YAML]: fix postcommit oracle bug and reorganize postcommit tests (#35191) * fix postcommit oracle test * add revision * switch to hosted runner to try with kafka test * add extended timeout * upgrade to 4.10 testcontainers * switch out to redpanda for kafka * remove redpandacontainer * tmp comment * add postgres comment * revert to old kafkaContainer * revert commented out code and revert testcontainer version change * change mysql image version * revert image change * revert image change again :) * add comments to mysql again * shift post commit files to different folders * rename to Data version * add databases version * add messaging version * update readme for three post commits * update gradle with new post commits * fix job names * uncomment fixture on mysql * switch back to one workflow and update readme as such * remove old workflow files * update order and remove comment * update gradle with parameterized options * update gradle command calls to correct location * update workflow to three jobs explicit * add back Bigquery deselect * fix mysql teardown error * Simplify down to one from three explicit jobs * remove tab * remove Data * fix parsing parameters * update hadoop prefix (#35257) * Set go version 1.24.4 (#35261) * Set go version 1.24.4 * Set go version 1.24.4 * upgrade org.apache.parquet:parquet-avro to to 1.15.2 (#35037) * upgrade org.apache.parquet:parquet-avro to to 1.15.2 * keep iceberg to 1.6.1 * use 1.9.0 for iceberg * run all iceberg tests * require Java 11 * fixed java11 for iceberg * switched back to iceberg_version = "1.6.1" * merged master --------- Co-authored-by: Vitaly Terentyev <[email protected]> * Bump google.golang.org/grpc from 1.72.2 to 1.73.0 in /sdks (#35236) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.72.2 to 1.73.0. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.72.2...v1.73.0) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.73.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * add more doc strings for integration tests (#35171) * [IcebergIO] extend table partitioning to Beam SQL (#35268) * add sql partitioning * trigger ITs and add to CHANGES * simplify test * include emoty namespace fix * add error when using partitioning for unsupported tables * fix test * Moving to 2.67.0-SNAPSHOT on master branch. * Update CHANGES.md to have fields for 2.67.0 release * add free disk space step (#35260) * [yaml]: Fix post commit yaml once more (#35273) * switch back to self hosted runner and comment out most of kafka test for now * add git issue to track * revision * remove free space step - seems to be causing more issues than helping * Polish anomaly detection notebook and get ready to be imported in public devsite. (#35278) * Polish anomaly detection zscore notebook for public doc. * Adjust formatting. * Adjust formatting. * Suppress Findbugs (#35274) * Support Java 17+ compiled Beam components for build, release, and xlang tests (#35232) * Install JDK 21 in release build Support Beam components requiring Java17+ for release workflows They will be compiled with JDK21 with byte code compatibility configured by applyJavaNature(requireJavaVersion) * Use Java 21 for Python PostCommit * Honor JAVA_HOME in JavaJarServer checks and tests * Disable Debezium test on Java17- * add example line * [IO] Update Debezium in DebeziumIO to 3.1.1 (#34763) * Updating Debezium IO to 3.1.1 Enforce JDK17 in build.gradle of Debezium IO * adjust to review comments by @Abacn * cleanup * mention which PR needs to merge before unpinning * Mention Beam version 2.66 in the README for Debezium --------- Co-authored-by: Shunping Huang <[email protected]> * Document BQ Storage API pipeline options (#35259) * Document BQ Storage API pipeline options * Fix whitespace * Fix whitespace * Speed up StreamingDataflowWorkerTest by removing 10 second wait from shutdown path. (#35275) Takes test time from 5m18s to 2m40s. The previous shutdown() call doesn't do anything since that just means future scheduling won't trigger but we only schedule on the executor once. Also cleanup test logs by making sure to stop all workers we start so they don't continue to run in the background and log. This shutdown paths is only used in testing. * Bump org.ajoberstar.grgit:grgit-gradle from 4.1.1 to 5.3.2 (#35301) Bumps [org.ajoberstar.grgit:grgit-gradle](https://github.com/ajoberstar/grgit) from 4.1.1 to 5.3.2. - [Release notes](https://github.com/ajoberstar/grgit/releases) - [Commits](ajoberstar/grgit@4.1.1...5.3.2) --- updated-dependencies: - dependency-name: org.ajoberstar.grgit:grgit-gradle dependency-version: 5.3.2 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump google.golang.org/api from 0.235.0 to 0.237.0 in /sdks (#35302) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.235.0 to 0.237.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.235.0...v0.237.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-version: 0.237.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/linkedin/goavro/v2 from 2.13.1 to 2.14.0 in /sdks (#35205) Bumps [github.com/linkedin/goavro/v2](https://github.com/linkedin/goavro) from 2.13.1 to 2.14.0. - [Release notes](https://github.com/linkedin/goavro/releases) - [Changelog](https://github.com/linkedin/goavro/blob/master/debug_release.go) - [Commits](linkedin/goavro@v2.13.1...v2.14.0) --- updated-dependencies: - dependency-name: github.com/linkedin/goavro/v2 dependency-version: 2.14.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update spotbugs version, fix runner ubuntu version, fix found spotbugs issues (#35303) * feat: Add option to control resource cleanup failure for IT (#35287) * Revert "Bump org.ajoberstar.grgit:grgit-gradle from 4.1.1 to 5.3.2 (#35301)" (#35305) This reverts commit 0fd77ce. * Add PeriodicStream in the new time series folder. (#35300) * Add PeriodicStream in the new time series folder. * Add some more docstrings and minor fix on test name. * Fix lints and docs. * try buildah to replace kaniko (#35289) * try buildah to replace kaniko * trigger post-commit * Adding error handler for SpannerReadSchemaTransformProvider and missi… (#35241) * Adding error handler for SpannerReadSchemaTransformProvider and missing tests for SpannerSchemaTransformProvider * Removed not used logging * Spotless Apply * Spotless Apply * Spotless Apply * Typo correction * requests vulnerability. (#35308) * [IcebergIO] create custom java container image for tests (#35307) * create custom java container image for tests * syntax * eval depends on df * add streaming support to iobase (python) (#35137) * Add support for streaming writes in IOBase (Python) * add triggering_frequency in iobase.Sink * fix whitespaces/newlines * fixes per #35137 (review) * refactor for code redability * refactor _expand_unbounded , default num_shards to 1 , if undef or 0 * fix formatter * space * keep num_shards = 0 the same as before for bounded write * reafactor and code clean --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: bullet03 <[email protected]> Co-authored-by: Bulat Safiullin <[email protected]> Co-authored-by: liferoad <[email protected]> Co-authored-by: claudevdm <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Yi Hu <[email protected]> Co-authored-by: Danny McCormick <[email protected]> Co-authored-by: Jack McCluskey <[email protected]> Co-authored-by: Derrick Williams <[email protected]> Co-authored-by: Vitaly Terentyev <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: scwhittle <[email protected]> Co-authored-by: Radosław Stankiewicz <[email protected]> Co-authored-by: Hai Joey Tran <[email protected]> Co-authored-by: Tanu Sharma <[email protected]> Co-authored-by: Kenneth Knowles <[email protected]> Co-authored-by: Minbo Bae <[email protected]> Co-authored-by: Shunping Huang <[email protected]> Co-authored-by: Chenzo <[email protected]> Co-authored-by: kristynsmith <[email protected]> Co-authored-by: Rakesh Kumar <[email protected]> Co-authored-by: Charles Nguyen <[email protected]> Co-authored-by: tvalentyn <[email protected]> Co-authored-by: Ahmed Abualsaud <[email protected]> Co-authored-by: Minh Son Nguyen <[email protected]> Co-authored-by: Amar3tto <actions@GitHub Actions 1000279405.local> Co-authored-by: Vitaly Terentyev <[email protected]> Co-authored-by: Tobias Kaymak <[email protected]> Co-authored-by: Veronica Wasson <[email protected]>
shunping
added a commit
that referenced
this pull request
Aug 7, 2025
* Add support for streaming writes in IOBase (Python) * add triggering_frequency in iobase.Sink * fix whitespaces/newlines * fixes per #35137 (review) * implement pre_finalize_windowed and finalize_windowed_write in FileBasedSink handle file naming based on windowing for streaming in internal methods TextIO - expose the new filebasedsink capabilities to write in streaming * update changes.md * fix formatting * space formatting * space nagging * keep in sync with refactor in #35137 * keep in sync with iobase changes in #35137 * [Website] add akvelon case study (#34943) * feat: add akvelon logo * feat: add akvelon case study * fix: remove white space * feat: add akvelon to main page * feat: use new images * fix: typos * fix: change order of akvelon case-study * fix: update text * fix: update mainPage text * fix: update images * fix: about akvelon section update * fix: update akvelon card * fix: update akvelon header * fix: update code tag * fix: update about akvelon * fix: update date and order * fix: add link and change img * fix: change CDAP text * fix: add bold weight * fix: solve conflicts * fix: remove unused code * fix: delete whitespace * fix: indents format * fix: add bold text --------- Co-authored-by: Bulat Safiullin <[email protected]> * added the rate test for GenerateSequence (#35108) * added the rate test for GenerateSequence * keep the master yaml * Re-enable logging after importing vllm. (#35103) Co-authored-by: Claude <[email protected]> * Deprecate Java 8 (#35064) * Deprecate Java 8 * Java 8 client now using Java 11 SDK container * adjust non LTS fallback to use the next LTS instead of the nearest LTS. Previously Java18 falls back to Java17, which won't work * Emit warning when Java 8 is used. Java8 is still supported until Beam 3.0 * Clean up subproject build file requiring Java9+ * Require java 11 to build SDK container * fix workflow * Simplify XVR test workflow * Fix Samza PVR * Remove beam college banners (#35123) * feat: change text (#35130) Co-authored-by: Bulat Safiullin <[email protected]> * Update Custom Remote Inference example to use RemoteModelHandler (#35066) * Update Custom Remote Inference example to use RemoteModelHandler * restore old kernel metadata * Update examples/notebooks/beam-ml/custom_remote_inference.ipynb Co-authored-by: Danny McCormick <[email protected]> * Add DLQ --------- Co-authored-by: Danny McCormick <[email protected]> * Remove Java 8 container (#35125) * add extra_transforms block documentation to chain transform documentation (#35101) * add note about testing (#35075) * fix whitespaces/newlines * fixes per #35137 (review) * implement pre_finalize_windowed and finalize_windowed_write in FileBasedSink handle file naming based on windowing for streaming in internal methods TextIO - expose the new filebasedsink capabilities to write in streaming * update changes.md * fix formatting * space formatting * space nagging * keep in sync with refactor in #35137 * keep in sync with iobase changes in #35137 * [Website] update akvelon case study: update text and fix landing page (#35133) * feat: change text * fix: add learn more to quotes Akvelon --------- Co-authored-by: Bulat Safiullin <[email protected]> * Fix PostCommit Python Xlang IO Dataflow job (#35131) * Add support for iterable type * Fix formatting * Bump google.golang.org/grpc from 1.72.0 to 1.72.2 in /sdks (#35113) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.72.0 to 1.72.2. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.72.0...v1.72.2) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.72.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cloud.google.com/go/bigquery from 1.67.0 to 1.69.0 in /sdks (#35061) Bumps [cloud.google.com/go/bigquery](https://github.com/googleapis/google-cloud-go) from 1.67.0 to 1.69.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@spanner/v1.67.0...spanner/v1.69.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/bigquery dependency-version: 1.69.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add known issues. (#35138) * Add known issues. * Add fixes notes. --------- Co-authored-by: Claude <[email protected]> * Bump @octokit/plugin-paginate-rest and @octokit/rest (#34167) Bumps [@octokit/plugin-paginate-rest](https://github.com/octokit/plugin-paginate-rest.js) to 11.4.3 and updates ancestor dependency [@octokit/rest](https://github.com/octokit/rest.js). These dependencies need to be updated together. Updates `@octokit/plugin-paginate-rest` from 2.17.0 to 11.4.3 - [Release notes](https://github.com/octokit/plugin-paginate-rest.js/releases) - [Commits](octokit/plugin-paginate-rest.js@v2.17.0...v11.4.3) Updates `@octokit/rest` from 18.12.0 to 21.1.1 - [Release notes](https://github.com/octokit/rest.js/releases) - [Commits](octokit/rest.js@v18.12.0...v21.1.1) --- updated-dependencies: - dependency-name: "@octokit/plugin-paginate-rest" dependency-type: indirect - dependency-name: "@octokit/rest" dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Explicitly handle singleton iterators instead of using helper and catching exceptions which may be from generating iterable (#35124) * Build last snapshot against RC00 tag instead of release branch (#35142) The last snapshot Beam Java SDK (aka "RC00" release) build is triggered manually, to verify a RC1 build will be successful. It has been built against release-2.xx branch, where the Dataflow container tag replaced from beam-master to the 2.xx.0 However, the versioned containers are not yet released, causing a timing gap that Beam 2.xx.0-SNAPSHOT won't work on Dataflow between release branch cut and RC1 rolled out to Dataflow Since we now have a v2.xx.0-RC00 tag, build RC00 against this tag resolves the issue. * Bump nodemailer from 6.7.5 to 6.9.9 in /scripts/ci/issue-report (#35143) Bumps [nodemailer](https://github.com/nodemailer/nodemailer) from 6.7.5 to 6.9.9. - [Release notes](https://github.com/nodemailer/nodemailer/releases) - [Changelog](https://github.com/nodemailer/nodemailer/blob/master/CHANGELOG.md) - [Commits](nodemailer/nodemailer@v6.7.5...v6.9.9) --- updated-dependencies: - dependency-name: nodemailer dependency-version: 6.9.9 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix tests affected by Java 8 container turn down (#35145) * Fix tests affected by Java 8 container turn down * still use Java 8 for Samza runner * Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#35146) Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.79.3 to 1.80.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@service/s3/v1.79.3...service/s3/v1.80.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-version: 1.80.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix jdbc transform validation (#35141) * fix jdbc transform validation * add test * annotations * spotless * Fix Java Example ARM PostCommit * fix: add missed word (#35163) Co-authored-by: Bulat Safiullin <[email protected]> * Add postcommit yaml xlang workflow and split tests accordingly (#35119) * move precommit yaml xlang to postcommit * add json file to trigger path * update pull request targets * add readme workflow changes * add cloud sdk setup * switch out to ubuntu-latest * add back precommit workflow * switch names and add postcommit * switch out to postCommitYamlIT * add test_files_dir flag * add conftest.py file for capturing directory flag * shift yaml tests around to appropriate locations * add back precommit to readme * add license for conftest.py * revert precommit to previous name * remove github.event.comment.body trigger * Replace usages of deprecated pkg_resources package (#35153) * Remove usages of deprecated pkg_resources package * use stdlib importlib.resources * remove extra comma * linting * import order * Improve error message when accidentally using PBegin/Pipeline (#35156) * Create test * Implement new error message * Add beam.Create into unit test pipeline * add friendly error message for when transform is applied to no output (#35160) * add friendly error message for when transform is applied to no output * update test name * Fix pubsub unit tests that depend on old behavior * Add warning if temp location bucket has soft delete enabled for Go SD… (#34996) * Add warning if temp location bucket has soft delete enabled for Go SDK (resolves #31606) * Corrected Formatting * Applied suggested changes * Constrain DequeCoder type correctly, as it does not support nulls The DequeCoder uses ArrayDeque internally, which disallows null elements. We could switch Deque implementations, but this change is better. To quote the JDK docs: "While Deque implementations are not strictly required to prohibit the insertion of null elements, they are strongly encouraged to do so. Users of any Deque implementations that do allow null elements are strongly encouraged not to take advantage of the ability to insert nulls. This is so because null is used as a special return value by various methods to indicated that the deque is empty." * Do not overwrite class states if a cached dynamic class is returned in cloudpickle.load (#35063) * Fix class states overwritten after cloudpickle.load * Fix further * Fix lint * Make SDK harness change effective on Iceberg Dataflow test (#35173) * Fix beam_PostCommit_Java_Examples_Dataflow_V2 (#35172) * [YAML]: Update postgres IT test and readme (#35169) * update postgres test without driver_class_name * update readme on how to run integration tests * fix misspelling * fix trailing whitespace * Bump Java beam-master container (#35170) * Make WindowedValue a public interface The following mostly-automated changes: - Moved WindowedValue from util to values package - Make WindowedValue an interface with companion class WindowedValues * Run integration tests for moving WindowedValue and making public * Add timer tests to make sure event-time timer firing at the right time. (#35109) * Add timer tests to make sure event-time timer firing at the right time. * Add more tests. * Disable the failed event-time timer tests for FnApiRunner. * Fix lints and reformat. * Disable another new test in FnApiRunnerTest and PortableRunnerTest due to flakiness. * Disable a new test in FlinkRunnerTest * Take out the early firing test case because it depends on bundle size. * change to ubuntu-20.04 (#35182) * Fix IntelliJ sync project failure due to circular Beam dependency (#35167) * Fix IntelliJ sync project failure due to circular Beam dependency * address comments * Update workflows categories (#35162) * Add cloudpickle coder. (#35166) * Move examples from sql package (#35183) * Fix the beam interactive install problem when on Google Colab (#35148) * Fix Google Colab Issue * Update CHANGES.md * Bump github.com/docker/docker in /sdks (#35112) Bumps [github.com/docker/docker](https://github.com/docker/docker) from 28.0.4+incompatible to 28.2.2+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v28.0.4...v28.2.2) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-version: 28.2.2+incompatible dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/nats-io/nats-server/v2 from 2.11.3 to 2.11.4 in /sdks (#35161) Bumps [github.com/nats-io/nats-server/v2](https://github.com/nats-io/nats-server) from 2.11.3 to 2.11.4. - [Release notes](https://github.com/nats-io/nats-server/releases) - [Changelog](https://github.com/nats-io/nats-server/blob/main/.goreleaser.yml) - [Commits](nats-io/nats-server@v2.11.3...v2.11.4) --- updated-dependencies: - dependency-name: github.com/nats-io/nats-server/v2 dependency-version: 2.11.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Touch JPMS test trigger file * Use staged SDK harness & Dataflow worker jar in JPMS tests * Bump cloud.google.com/go/storage from 1.52.0 to 1.55.0 in /sdks (#35114) Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.52.0 to 1.55.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@spanner/v1.52.0...spanner/v1.55.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-version: 1.55.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/nats-io/nats.go from 1.42.0 to 1.43.0 in /sdks (#35147) Bumps [github.com/nats-io/nats.go](https://github.com/nats-io/nats.go) from 1.42.0 to 1.43.0. - [Release notes](https://github.com/nats-io/nats.go/releases) - [Commits](nats-io/nats.go@v1.42.0...v1.43.0) --- updated-dependencies: - dependency-name: github.com/nats-io/nats.go dependency-version: 1.43.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Move non-dill specific code out of dill_pickler.py (#35139) * Create consistent_pickle.py * Update dill_pickler.py * Update consistent_pickle.py * Add Apache license to consistent_pickle.py * Update dill_pickler.py * Update and rename consistent_pickle.py to code_object_pickler.py * Update dill_pickler.py to change consistent_pickle to code_object_pickler * Fix formatting issue in code_object_pickler.py * Fix formatting issue in dill_pickler.py * Remove apache license from code_object_pickler.py * Add apache license to code_object_pickler.py * Fix some lints introduced in a recent PR. (#35193) * small filesystem fixes (#34956) * Fix typehint for get_filesystem * Fix bug with creating files in cwd with LocalFileSystem * lint * revert precommit config change * isort * add file to start PR builder * Revert "add file to start PR builder" This reverts commit f80b361. * fix isort again * [YAML] Add a spec provider for transforms taking specifiable arguments (#35187) * Add a test provider for specifiable and try it on AnomalyDetection. Also add support on callable in spec. * Minor renaming * Fix lints. * Touch trigger files to test WindowedValueReceiver in runners * Introduce WindowedValue receivers and consolidate runner code to them We need a receiver for WindowedValue to create an OutputBuilder. This change introduces the interface and uses it in many places where it is appropriate, replacing and simplifying internal runner code. * Eliminate nullness errors from ByteBuddyDoFnInvokerFactory and DoFnOutputReceivers * Fix null check when fetching driverJars from value provider * Fix PostCommit Python ValidatesRunner Samza / Spark jobs (#35210) * Skip Samza and Spark runner tests * Fix formatting * Update pypi documentation 30145 (#34329) * Add Pypi friendly documentation * provided full link path * Add more YAML examples involving Kafka and Iceberg (#35151) * Add more YAML examples involving Kafka and Iceberg * Fix some missed details from rebasing * Adding unit tests for YAML examples * Clean up and address PR comments * Formatting * Formatting * Evaluate namedTuples as equivalent to rows (#35218) * Evaluate namedTuples as equivalent to rows * lint * Add a new experiment flag to enable real-time clock as processing time. (#35202) * Touch trigger files for lightweight runners * Eliminate WindowedValue.getPane() in preparation for making it a user-facing interface * Do not fail if there were failed deletes (#35222) * Force FnApiRunner in cases where prism can't handle use case (#35219) * Force FnApiRunner in cases where prism can't handle use case * yapf * Bump golang.org/x/net from 0.40.0 to 0.41.0 in /sdks (#35206) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.40.0 to 0.41.0. - [Commits](golang/net@v0.40.0...v0.41.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.41.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix incorrect typehints generated by FlatMap with default identity function (#35164) * create unit test * add some unit tests * fix bug where T is considered iterable * update strip_iterable to return Any for "stripped iterable" type of TypeVariable * remove typehint from identity function and add a test to test for proper typechecking * Move callablewrapp typehint test * Remove print * isort * isort * return any for yielded type of T * Parse values returned from Dataflow API to BoundedTrieData (#34738) * Parse struct returned from Dataflow API to BoundedTrieData fix checkstyle Use getBoundedTrie add debug log adapt to ArrayMap * Add test, clean up * Fix test: setTrie -> setBoundedTrie * Fix comments * Remove breaking PDone change (#35224) * Remove breaking PDone change * Update pipeline.py * Update pipeline.py * Update pipeline_test.py * Move none check * yapf * fmt * Generic Postgres + Cloudsql postgres embeddings. (#35215) * Add base Postgres vector writer, CloudSQL vector writer and refactor. * Trigger tests. * Linter fixes. * Fix test * Add back tests. Update changes.md. Fix unrelated lint. * Drop test tables. * Fix test * Fix tests. * Fix test. * Fix tests. --------- Co-authored-by: Claude <[email protected]> * Allow only one thread at a time to start the VLLM server. (#35234) * [IcebergIO] Create namespaces if needed (#35228) * create namespaces dynamically * cleanup namespaces in ITs * optimization * Support configuring flush_count and max_row_bytes of WriteToBigTable (#34761) * Update CHANGES.md (#35242) Highlight improvements to the vllm model handler. * [Beam SQL] Implement Catalog and CatalogManager (#35223) * beam sql catalog * api adjustment * cover more naming syntax (quotes, backticks, none) * spotless * fix * add documentation and cleanup * rename to dropCatalog; mark BeamSqlCli @internal; rename to EmptyCatalogManager * use registrars instead; remove initialize() method from Catalog * cleanupo * [IcebergIO] Fix conversion logic for arrays of structs and maps of structs; fix output Schema resolution with column pruning (#35230) * fix complex types; filx filter schema logic * update Changes * add missing changes from other PRs * trigger ITs * make separate impl for iterable * fix * fix long_description when the md file cannot be found (#35246) * fix long_description when the md file cannot be found * yapf * [IcebergIO] Create tables with a partition spec (#34966) * create table with partition spec * add description and regenerate config doc; trigger tests * spotless * log partition spec * fixes * Fix typo java-dependencies.md (#35251) * Adding project and database support in write transform for firestoreIO (#35017) * Adding project and database support in write transform for firestoreIO * Spotless Apply * Resolving comments * Removing useless comments * public to private * Spotless apply * Fix a logical type issue about JdbcDateType and JdbcTimeType (#35243) * Fix a logical type issue about JdbcDateType * Fix typo and also fix the logical class for java time. * Get rid of the workaround on logical type registration. Trigger tests. * Fix lints. * Remove internal code. (#35239) Co-authored-by: Claude <[email protected]> * enable setting max_writer_per_bundle for avroIO and other IO (#35092) * enable setting max_writer_per_bundle for avroIO and other IO * enable setting max_writer_per_bundle for avroIO and other IO * enable for TFRecordIO and corrections * Updated standard_external_transforms.yaml * [Java FnApi] Fix race in BeamFnStatusClient by initializing all fields before starting rpc. (#35252) * [Java FnApi] Fix race in BeamFnStatusClient by initializing all fields before starting rpc. * spotless * [IcebergIO] Integrate with Beam SQL (#34799) * add Iceberg table provider and tests * properties go in the tableprovider initialization * tobuilder * streaming integration test * spotless * extend test to include multi nested types; fix iceberg <-> conversion logic * add to changes.md * spotless * fix tests * clean * update CHANGES * add projection pushdown and column pruning * spotless * fixes * fixes * updates * sync with HEAD and use new Catalog implementation * mark new interfaces @internal * spotless * fix unparse method * update changes (#35256) * [YAML]: fix postcommit oracle bug and reorganize postcommit tests (#35191) * fix postcommit oracle test * add revision * switch to hosted runner to try with kafka test * add extended timeout * upgrade to 4.10 testcontainers * switch out to redpanda for kafka * remove redpandacontainer * tmp comment * add postgres comment * revert to old kafkaContainer * revert commented out code and revert testcontainer version change * change mysql image version * revert image change * revert image change again :) * add comments to mysql again * shift post commit files to different folders * rename to Data version * add databases version * add messaging version * update readme for three post commits * update gradle with new post commits * fix job names * uncomment fixture on mysql * switch back to one workflow and update readme as such * remove old workflow files * update order and remove comment * update gradle with parameterized options * update gradle command calls to correct location * update workflow to three jobs explicit * add back Bigquery deselect * fix mysql teardown error * Simplify down to one from three explicit jobs * remove tab * remove Data * fix parsing parameters * update hadoop prefix (#35257) * Set go version 1.24.4 (#35261) * Set go version 1.24.4 * Set go version 1.24.4 * upgrade org.apache.parquet:parquet-avro to to 1.15.2 (#35037) * upgrade org.apache.parquet:parquet-avro to to 1.15.2 * keep iceberg to 1.6.1 * use 1.9.0 for iceberg * run all iceberg tests * require Java 11 * fixed java11 for iceberg * switched back to iceberg_version = "1.6.1" * merged master --------- Co-authored-by: Vitaly Terentyev <[email protected]> * Bump google.golang.org/grpc from 1.72.2 to 1.73.0 in /sdks (#35236) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.72.2 to 1.73.0. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.72.2...v1.73.0) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.73.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * add more doc strings for integration tests (#35171) * [IcebergIO] extend table partitioning to Beam SQL (#35268) * add sql partitioning * trigger ITs and add to CHANGES * simplify test * include emoty namespace fix * add error when using partitioning for unsupported tables * fix test * Moving to 2.67.0-SNAPSHOT on master branch. * Update CHANGES.md to have fields for 2.67.0 release * add free disk space step (#35260) * [yaml]: Fix post commit yaml once more (#35273) * switch back to self hosted runner and comment out most of kafka test for now * add git issue to track * revision * remove free space step - seems to be causing more issues than helping * Polish anomaly detection notebook and get ready to be imported in public devsite. (#35278) * Polish anomaly detection zscore notebook for public doc. * Adjust formatting. * Adjust formatting. * Suppress Findbugs (#35274) * Support Java 17+ compiled Beam components for build, release, and xlang tests (#35232) * Install JDK 21 in release build Support Beam components requiring Java17+ for release workflows They will be compiled with JDK21 with byte code compatibility configured by applyJavaNature(requireJavaVersion) * Use Java 21 for Python PostCommit * Honor JAVA_HOME in JavaJarServer checks and tests * Disable Debezium test on Java17- * add example line * [IO] Update Debezium in DebeziumIO to 3.1.1 (#34763) * Updating Debezium IO to 3.1.1 Enforce JDK17 in build.gradle of Debezium IO * adjust to review comments by @Abacn * cleanup * mention which PR needs to merge before unpinning * Mention Beam version 2.66 in the README for Debezium --------- Co-authored-by: Shunping Huang <[email protected]> * Document BQ Storage API pipeline options (#35259) * Document BQ Storage API pipeline options * Fix whitespace * Fix whitespace * Speed up StreamingDataflowWorkerTest by removing 10 second wait from shutdown path. (#35275) Takes test time from 5m18s to 2m40s. The previous shutdown() call doesn't do anything since that just means future scheduling won't trigger but we only schedule on the executor once. Also cleanup test logs by making sure to stop all workers we start so they don't continue to run in the background and log. This shutdown paths is only used in testing. * Bump org.ajoberstar.grgit:grgit-gradle from 4.1.1 to 5.3.2 (#35301) Bumps [org.ajoberstar.grgit:grgit-gradle](https://github.com/ajoberstar/grgit) from 4.1.1 to 5.3.2. - [Release notes](https://github.com/ajoberstar/grgit/releases) - [Commits](ajoberstar/grgit@4.1.1...5.3.2) --- updated-dependencies: - dependency-name: org.ajoberstar.grgit:grgit-gradle dependency-version: 5.3.2 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump google.golang.org/api from 0.235.0 to 0.237.0 in /sdks (#35302) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.235.0 to 0.237.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.235.0...v0.237.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-version: 0.237.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/linkedin/goavro/v2 from 2.13.1 to 2.14.0 in /sdks (#35205) Bumps [github.com/linkedin/goavro/v2](https://github.com/linkedin/goavro) from 2.13.1 to 2.14.0. - [Release notes](https://github.com/linkedin/goavro/releases) - [Changelog](https://github.com/linkedin/goavro/blob/master/debug_release.go) - [Commits](linkedin/goavro@v2.13.1...v2.14.0) --- updated-dependencies: - dependency-name: github.com/linkedin/goavro/v2 dependency-version: 2.14.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update spotbugs version, fix runner ubuntu version, fix found spotbugs issues (#35303) * feat: Add option to control resource cleanup failure for IT (#35287) * Revert "Bump org.ajoberstar.grgit:grgit-gradle from 4.1.1 to 5.3.2 (#35301)" (#35305) This reverts commit 0fd77ce. * Add PeriodicStream in the new time series folder. (#35300) * Add PeriodicStream in the new time series folder. * Add some more docstrings and minor fix on test name. * Fix lints and docs. * try buildah to replace kaniko (#35289) * try buildah to replace kaniko * trigger post-commit * Adding error handler for SpannerReadSchemaTransformProvider and missi… (#35241) * Adding error handler for SpannerReadSchemaTransformProvider and missing tests for SpannerSchemaTransformProvider * Removed not used logging * Spotless Apply * Spotless Apply * Spotless Apply * Typo correction * requests vulnerability. (#35308) * [IcebergIO] create custom java container image for tests (#35307) * create custom java container image for tests * syntax * eval depends on df * add streaming support to iobase (python) (#35137) * Add support for streaming writes in IOBase (Python) * add triggering_frequency in iobase.Sink * fix whitespaces/newlines * fixes per #35137 (review) * refactor for code redability * refactor _expand_unbounded , default num_shards to 1 , if undef or 0 * fix formatter * space * keep num_shards = 0 the same as before for bounded write * add streaming to AvroIO, ParquetIO, TFRecordsIO * reformat * typo and spaces * carry on the refactor from #35253 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: bullet03 <[email protected]> Co-authored-by: Bulat Safiullin <[email protected]> Co-authored-by: liferoad <[email protected]> Co-authored-by: claudevdm <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Yi Hu <[email protected]> Co-authored-by: Danny McCormick <[email protected]> Co-authored-by: Jack McCluskey <[email protected]> Co-authored-by: Derrick Williams <[email protected]> Co-authored-by: Vitaly Terentyev <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: scwhittle <[email protected]> Co-authored-by: Radosław Stankiewicz <[email protected]> Co-authored-by: Hai Joey Tran <[email protected]> Co-authored-by: Tanu Sharma <[email protected]> Co-authored-by: Kenneth Knowles <[email protected]> Co-authored-by: Minbo Bae <[email protected]> Co-authored-by: Shunping Huang <[email protected]> Co-authored-by: Chenzo <[email protected]> Co-authored-by: kristynsmith <[email protected]> Co-authored-by: Rakesh Kumar <[email protected]> Co-authored-by: Charles Nguyen <[email protected]> Co-authored-by: tvalentyn <[email protected]> Co-authored-by: Ahmed Abualsaud <[email protected]> Co-authored-by: Minh Son Nguyen <[email protected]> Co-authored-by: Amar3tto <actions@GitHub Actions 1000279405.local> Co-authored-by: Vitaly Terentyev <[email protected]> Co-authored-by: Tobias Kaymak <[email protected]> Co-authored-by: Veronica Wasson <[email protected]>
parveensania
pushed a commit
to parveensania/beam-dp
that referenced
this pull request
Aug 17, 2025
* Add support for streaming writes in IOBase (Python) * add triggering_frequency in iobase.Sink * fix whitespaces/newlines * fixes per apache#35137 (review) * implement pre_finalize_windowed and finalize_windowed_write in FileBasedSink handle file naming based on windowing for streaming in internal methods TextIO - expose the new filebasedsink capabilities to write in streaming * update changes.md * fix formatting * space formatting * space nagging * keep in sync with refactor in apache#35137 * keep in sync with iobase changes in apache#35137 * [Website] add akvelon case study (apache#34943) * feat: add akvelon logo * feat: add akvelon case study * fix: remove white space * feat: add akvelon to main page * feat: use new images * fix: typos * fix: change order of akvelon case-study * fix: update text * fix: update mainPage text * fix: update images * fix: about akvelon section update * fix: update akvelon card * fix: update akvelon header * fix: update code tag * fix: update about akvelon * fix: update date and order * fix: add link and change img * fix: change CDAP text * fix: add bold weight * fix: solve conflicts * fix: remove unused code * fix: delete whitespace * fix: indents format * fix: add bold text --------- Co-authored-by: Bulat Safiullin <[email protected]> * added the rate test for GenerateSequence (apache#35108) * added the rate test for GenerateSequence * keep the master yaml * Re-enable logging after importing vllm. (apache#35103) Co-authored-by: Claude <[email protected]> * Deprecate Java 8 (apache#35064) * Deprecate Java 8 * Java 8 client now using Java 11 SDK container * adjust non LTS fallback to use the next LTS instead of the nearest LTS. Previously Java18 falls back to Java17, which won't work * Emit warning when Java 8 is used. Java8 is still supported until Beam 3.0 * Clean up subproject build file requiring Java9+ * Require java 11 to build SDK container * fix workflow * Simplify XVR test workflow * Fix Samza PVR * Remove beam college banners (apache#35123) * feat: change text (apache#35130) Co-authored-by: Bulat Safiullin <[email protected]> * Update Custom Remote Inference example to use RemoteModelHandler (apache#35066) * Update Custom Remote Inference example to use RemoteModelHandler * restore old kernel metadata * Update examples/notebooks/beam-ml/custom_remote_inference.ipynb Co-authored-by: Danny McCormick <[email protected]> * Add DLQ --------- Co-authored-by: Danny McCormick <[email protected]> * Remove Java 8 container (apache#35125) * add extra_transforms block documentation to chain transform documentation (apache#35101) * add note about testing (apache#35075) * fix whitespaces/newlines * fixes per apache#35137 (review) * implement pre_finalize_windowed and finalize_windowed_write in FileBasedSink handle file naming based on windowing for streaming in internal methods TextIO - expose the new filebasedsink capabilities to write in streaming * update changes.md * fix formatting * space formatting * space nagging * keep in sync with refactor in apache#35137 * keep in sync with iobase changes in apache#35137 * [Website] update akvelon case study: update text and fix landing page (apache#35133) * feat: change text * fix: add learn more to quotes Akvelon --------- Co-authored-by: Bulat Safiullin <[email protected]> * Fix PostCommit Python Xlang IO Dataflow job (apache#35131) * Add support for iterable type * Fix formatting * Bump google.golang.org/grpc from 1.72.0 to 1.72.2 in /sdks (apache#35113) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.72.0 to 1.72.2. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.72.0...v1.72.2) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.72.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cloud.google.com/go/bigquery from 1.67.0 to 1.69.0 in /sdks (apache#35061) Bumps [cloud.google.com/go/bigquery](https://github.com/googleapis/google-cloud-go) from 1.67.0 to 1.69.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@spanner/v1.67.0...spanner/v1.69.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/bigquery dependency-version: 1.69.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add known issues. (apache#35138) * Add known issues. * Add fixes notes. --------- Co-authored-by: Claude <[email protected]> * Bump @octokit/plugin-paginate-rest and @octokit/rest (apache#34167) Bumps [@octokit/plugin-paginate-rest](https://github.com/octokit/plugin-paginate-rest.js) to 11.4.3 and updates ancestor dependency [@octokit/rest](https://github.com/octokit/rest.js). These dependencies need to be updated together. Updates `@octokit/plugin-paginate-rest` from 2.17.0 to 11.4.3 - [Release notes](https://github.com/octokit/plugin-paginate-rest.js/releases) - [Commits](octokit/plugin-paginate-rest.js@v2.17.0...v11.4.3) Updates `@octokit/rest` from 18.12.0 to 21.1.1 - [Release notes](https://github.com/octokit/rest.js/releases) - [Commits](octokit/rest.js@v18.12.0...v21.1.1) --- updated-dependencies: - dependency-name: "@octokit/plugin-paginate-rest" dependency-type: indirect - dependency-name: "@octokit/rest" dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Explicitly handle singleton iterators instead of using helper and catching exceptions which may be from generating iterable (apache#35124) * Build last snapshot against RC00 tag instead of release branch (apache#35142) The last snapshot Beam Java SDK (aka "RC00" release) build is triggered manually, to verify a RC1 build will be successful. It has been built against release-2.xx branch, where the Dataflow container tag replaced from beam-master to the 2.xx.0 However, the versioned containers are not yet released, causing a timing gap that Beam 2.xx.0-SNAPSHOT won't work on Dataflow between release branch cut and RC1 rolled out to Dataflow Since we now have a v2.xx.0-RC00 tag, build RC00 against this tag resolves the issue. * Bump nodemailer from 6.7.5 to 6.9.9 in /scripts/ci/issue-report (apache#35143) Bumps [nodemailer](https://github.com/nodemailer/nodemailer) from 6.7.5 to 6.9.9. - [Release notes](https://github.com/nodemailer/nodemailer/releases) - [Changelog](https://github.com/nodemailer/nodemailer/blob/master/CHANGELOG.md) - [Commits](nodemailer/nodemailer@v6.7.5...v6.9.9) --- updated-dependencies: - dependency-name: nodemailer dependency-version: 6.9.9 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix tests affected by Java 8 container turn down (apache#35145) * Fix tests affected by Java 8 container turn down * still use Java 8 for Samza runner * Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (apache#35146) Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.79.3 to 1.80.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@service/s3/v1.79.3...service/s3/v1.80.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-version: 1.80.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix jdbc transform validation (apache#35141) * fix jdbc transform validation * add test * annotations * spotless * Fix Java Example ARM PostCommit * fix: add missed word (apache#35163) Co-authored-by: Bulat Safiullin <[email protected]> * Add postcommit yaml xlang workflow and split tests accordingly (apache#35119) * move precommit yaml xlang to postcommit * add json file to trigger path * update pull request targets * add readme workflow changes * add cloud sdk setup * switch out to ubuntu-latest * add back precommit workflow * switch names and add postcommit * switch out to postCommitYamlIT * add test_files_dir flag * add conftest.py file for capturing directory flag * shift yaml tests around to appropriate locations * add back precommit to readme * add license for conftest.py * revert precommit to previous name * remove github.event.comment.body trigger * Replace usages of deprecated pkg_resources package (apache#35153) * Remove usages of deprecated pkg_resources package * use stdlib importlib.resources * remove extra comma * linting * import order * Improve error message when accidentally using PBegin/Pipeline (apache#35156) * Create test * Implement new error message * Add beam.Create into unit test pipeline * add friendly error message for when transform is applied to no output (apache#35160) * add friendly error message for when transform is applied to no output * update test name * Fix pubsub unit tests that depend on old behavior * Add warning if temp location bucket has soft delete enabled for Go SD… (apache#34996) * Add warning if temp location bucket has soft delete enabled for Go SDK (resolves apache#31606) * Corrected Formatting * Applied suggested changes * Constrain DequeCoder type correctly, as it does not support nulls The DequeCoder uses ArrayDeque internally, which disallows null elements. We could switch Deque implementations, but this change is better. To quote the JDK docs: "While Deque implementations are not strictly required to prohibit the insertion of null elements, they are strongly encouraged to do so. Users of any Deque implementations that do allow null elements are strongly encouraged not to take advantage of the ability to insert nulls. This is so because null is used as a special return value by various methods to indicated that the deque is empty." * Do not overwrite class states if a cached dynamic class is returned in cloudpickle.load (apache#35063) * Fix class states overwritten after cloudpickle.load * Fix further * Fix lint * Make SDK harness change effective on Iceberg Dataflow test (apache#35173) * Fix beam_PostCommit_Java_Examples_Dataflow_V2 (apache#35172) * [YAML]: Update postgres IT test and readme (apache#35169) * update postgres test without driver_class_name * update readme on how to run integration tests * fix misspelling * fix trailing whitespace * Bump Java beam-master container (apache#35170) * Make WindowedValue a public interface The following mostly-automated changes: - Moved WindowedValue from util to values package - Make WindowedValue an interface with companion class WindowedValues * Run integration tests for moving WindowedValue and making public * Add timer tests to make sure event-time timer firing at the right time. (apache#35109) * Add timer tests to make sure event-time timer firing at the right time. * Add more tests. * Disable the failed event-time timer tests for FnApiRunner. * Fix lints and reformat. * Disable another new test in FnApiRunnerTest and PortableRunnerTest due to flakiness. * Disable a new test in FlinkRunnerTest * Take out the early firing test case because it depends on bundle size. * change to ubuntu-20.04 (apache#35182) * Fix IntelliJ sync project failure due to circular Beam dependency (apache#35167) * Fix IntelliJ sync project failure due to circular Beam dependency * address comments * Update workflows categories (apache#35162) * Add cloudpickle coder. (apache#35166) * Move examples from sql package (apache#35183) * Fix the beam interactive install problem when on Google Colab (apache#35148) * Fix Google Colab Issue * Update CHANGES.md * Bump github.com/docker/docker in /sdks (apache#35112) Bumps [github.com/docker/docker](https://github.com/docker/docker) from 28.0.4+incompatible to 28.2.2+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v28.0.4...v28.2.2) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-version: 28.2.2+incompatible dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/nats-io/nats-server/v2 from 2.11.3 to 2.11.4 in /sdks (apache#35161) Bumps [github.com/nats-io/nats-server/v2](https://github.com/nats-io/nats-server) from 2.11.3 to 2.11.4. - [Release notes](https://github.com/nats-io/nats-server/releases) - [Changelog](https://github.com/nats-io/nats-server/blob/main/.goreleaser.yml) - [Commits](nats-io/nats-server@v2.11.3...v2.11.4) --- updated-dependencies: - dependency-name: github.com/nats-io/nats-server/v2 dependency-version: 2.11.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Touch JPMS test trigger file * Use staged SDK harness & Dataflow worker jar in JPMS tests * Bump cloud.google.com/go/storage from 1.52.0 to 1.55.0 in /sdks (apache#35114) Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.52.0 to 1.55.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@spanner/v1.52.0...spanner/v1.55.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-version: 1.55.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/nats-io/nats.go from 1.42.0 to 1.43.0 in /sdks (apache#35147) Bumps [github.com/nats-io/nats.go](https://github.com/nats-io/nats.go) from 1.42.0 to 1.43.0. - [Release notes](https://github.com/nats-io/nats.go/releases) - [Commits](nats-io/nats.go@v1.42.0...v1.43.0) --- updated-dependencies: - dependency-name: github.com/nats-io/nats.go dependency-version: 1.43.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Move non-dill specific code out of dill_pickler.py (apache#35139) * Create consistent_pickle.py * Update dill_pickler.py * Update consistent_pickle.py * Add Apache license to consistent_pickle.py * Update dill_pickler.py * Update and rename consistent_pickle.py to code_object_pickler.py * Update dill_pickler.py to change consistent_pickle to code_object_pickler * Fix formatting issue in code_object_pickler.py * Fix formatting issue in dill_pickler.py * Remove apache license from code_object_pickler.py * Add apache license to code_object_pickler.py * Fix some lints introduced in a recent PR. (apache#35193) * small filesystem fixes (apache#34956) * Fix typehint for get_filesystem * Fix bug with creating files in cwd with LocalFileSystem * lint * revert precommit config change * isort * add file to start PR builder * Revert "add file to start PR builder" This reverts commit f80b361. * fix isort again * [YAML] Add a spec provider for transforms taking specifiable arguments (apache#35187) * Add a test provider for specifiable and try it on AnomalyDetection. Also add support on callable in spec. * Minor renaming * Fix lints. * Touch trigger files to test WindowedValueReceiver in runners * Introduce WindowedValue receivers and consolidate runner code to them We need a receiver for WindowedValue to create an OutputBuilder. This change introduces the interface and uses it in many places where it is appropriate, replacing and simplifying internal runner code. * Eliminate nullness errors from ByteBuddyDoFnInvokerFactory and DoFnOutputReceivers * Fix null check when fetching driverJars from value provider * Fix PostCommit Python ValidatesRunner Samza / Spark jobs (apache#35210) * Skip Samza and Spark runner tests * Fix formatting * Update pypi documentation 30145 (apache#34329) * Add Pypi friendly documentation * provided full link path * Add more YAML examples involving Kafka and Iceberg (apache#35151) * Add more YAML examples involving Kafka and Iceberg * Fix some missed details from rebasing * Adding unit tests for YAML examples * Clean up and address PR comments * Formatting * Formatting * Evaluate namedTuples as equivalent to rows (apache#35218) * Evaluate namedTuples as equivalent to rows * lint * Add a new experiment flag to enable real-time clock as processing time. (apache#35202) * Touch trigger files for lightweight runners * Eliminate WindowedValue.getPane() in preparation for making it a user-facing interface * Do not fail if there were failed deletes (apache#35222) * Force FnApiRunner in cases where prism can't handle use case (apache#35219) * Force FnApiRunner in cases where prism can't handle use case * yapf * Bump golang.org/x/net from 0.40.0 to 0.41.0 in /sdks (apache#35206) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.40.0 to 0.41.0. - [Commits](golang/net@v0.40.0...v0.41.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.41.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix incorrect typehints generated by FlatMap with default identity function (apache#35164) * create unit test * add some unit tests * fix bug where T is considered iterable * update strip_iterable to return Any for "stripped iterable" type of TypeVariable * remove typehint from identity function and add a test to test for proper typechecking * Move callablewrapp typehint test * Remove print * isort * isort * return any for yielded type of T * Parse values returned from Dataflow API to BoundedTrieData (apache#34738) * Parse struct returned from Dataflow API to BoundedTrieData fix checkstyle Use getBoundedTrie add debug log adapt to ArrayMap * Add test, clean up * Fix test: setTrie -> setBoundedTrie * Fix comments * Remove breaking PDone change (apache#35224) * Remove breaking PDone change * Update pipeline.py * Update pipeline.py * Update pipeline_test.py * Move none check * yapf * fmt * Generic Postgres + Cloudsql postgres embeddings. (apache#35215) * Add base Postgres vector writer, CloudSQL vector writer and refactor. * Trigger tests. * Linter fixes. * Fix test * Add back tests. Update changes.md. Fix unrelated lint. * Drop test tables. * Fix test * Fix tests. * Fix test. * Fix tests. --------- Co-authored-by: Claude <[email protected]> * Allow only one thread at a time to start the VLLM server. (apache#35234) * [IcebergIO] Create namespaces if needed (apache#35228) * create namespaces dynamically * cleanup namespaces in ITs * optimization * Support configuring flush_count and max_row_bytes of WriteToBigTable (apache#34761) * Update CHANGES.md (apache#35242) Highlight improvements to the vllm model handler. * [Beam SQL] Implement Catalog and CatalogManager (apache#35223) * beam sql catalog * api adjustment * cover more naming syntax (quotes, backticks, none) * spotless * fix * add documentation and cleanup * rename to dropCatalog; mark BeamSqlCli @internal; rename to EmptyCatalogManager * use registrars instead; remove initialize() method from Catalog * cleanupo * [IcebergIO] Fix conversion logic for arrays of structs and maps of structs; fix output Schema resolution with column pruning (apache#35230) * fix complex types; filx filter schema logic * update Changes * add missing changes from other PRs * trigger ITs * make separate impl for iterable * fix * fix long_description when the md file cannot be found (apache#35246) * fix long_description when the md file cannot be found * yapf * [IcebergIO] Create tables with a partition spec (apache#34966) * create table with partition spec * add description and regenerate config doc; trigger tests * spotless * log partition spec * fixes * Fix typo java-dependencies.md (apache#35251) * Adding project and database support in write transform for firestoreIO (apache#35017) * Adding project and database support in write transform for firestoreIO * Spotless Apply * Resolving comments * Removing useless comments * public to private * Spotless apply * Fix a logical type issue about JdbcDateType and JdbcTimeType (apache#35243) * Fix a logical type issue about JdbcDateType * Fix typo and also fix the logical class for java time. * Get rid of the workaround on logical type registration. Trigger tests. * Fix lints. * Remove internal code. (apache#35239) Co-authored-by: Claude <[email protected]> * enable setting max_writer_per_bundle for avroIO and other IO (apache#35092) * enable setting max_writer_per_bundle for avroIO and other IO * enable setting max_writer_per_bundle for avroIO and other IO * enable for TFRecordIO and corrections * Updated standard_external_transforms.yaml * [Java FnApi] Fix race in BeamFnStatusClient by initializing all fields before starting rpc. (apache#35252) * [Java FnApi] Fix race in BeamFnStatusClient by initializing all fields before starting rpc. * spotless * [IcebergIO] Integrate with Beam SQL (apache#34799) * add Iceberg table provider and tests * properties go in the tableprovider initialization * tobuilder * streaming integration test * spotless * extend test to include multi nested types; fix iceberg <-> conversion logic * add to changes.md * spotless * fix tests * clean * update CHANGES * add projection pushdown and column pruning * spotless * fixes * fixes * updates * sync with HEAD and use new Catalog implementation * mark new interfaces @internal * spotless * fix unparse method * update changes (apache#35256) * [YAML]: fix postcommit oracle bug and reorganize postcommit tests (apache#35191) * fix postcommit oracle test * add revision * switch to hosted runner to try with kafka test * add extended timeout * upgrade to 4.10 testcontainers * switch out to redpanda for kafka * remove redpandacontainer * tmp comment * add postgres comment * revert to old kafkaContainer * revert commented out code and revert testcontainer version change * change mysql image version * revert image change * revert image change again :) * add comments to mysql again * shift post commit files to different folders * rename to Data version * add databases version * add messaging version * update readme for three post commits * update gradle with new post commits * fix job names * uncomment fixture on mysql * switch back to one workflow and update readme as such * remove old workflow files * update order and remove comment * update gradle with parameterized options * update gradle command calls to correct location * update workflow to three jobs explicit * add back Bigquery deselect * fix mysql teardown error * Simplify down to one from three explicit jobs * remove tab * remove Data * fix parsing parameters * update hadoop prefix (apache#35257) * Set go version 1.24.4 (apache#35261) * Set go version 1.24.4 * Set go version 1.24.4 * upgrade org.apache.parquet:parquet-avro to to 1.15.2 (apache#35037) * upgrade org.apache.parquet:parquet-avro to to 1.15.2 * keep iceberg to 1.6.1 * use 1.9.0 for iceberg * run all iceberg tests * require Java 11 * fixed java11 for iceberg * switched back to iceberg_version = "1.6.1" * merged master --------- Co-authored-by: Vitaly Terentyev <[email protected]> * Bump google.golang.org/grpc from 1.72.2 to 1.73.0 in /sdks (apache#35236) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.72.2 to 1.73.0. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.72.2...v1.73.0) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.73.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * add more doc strings for integration tests (apache#35171) * [IcebergIO] extend table partitioning to Beam SQL (apache#35268) * add sql partitioning * trigger ITs and add to CHANGES * simplify test * include emoty namespace fix * add error when using partitioning for unsupported tables * fix test * Moving to 2.67.0-SNAPSHOT on master branch. * Update CHANGES.md to have fields for 2.67.0 release * add free disk space step (apache#35260) * [yaml]: Fix post commit yaml once more (apache#35273) * switch back to self hosted runner and comment out most of kafka test for now * add git issue to track * revision * remove free space step - seems to be causing more issues than helping * Polish anomaly detection notebook and get ready to be imported in public devsite. (apache#35278) * Polish anomaly detection zscore notebook for public doc. * Adjust formatting. * Adjust formatting. * Suppress Findbugs (apache#35274) * Support Java 17+ compiled Beam components for build, release, and xlang tests (apache#35232) * Install JDK 21 in release build Support Beam components requiring Java17+ for release workflows They will be compiled with JDK21 with byte code compatibility configured by applyJavaNature(requireJavaVersion) * Use Java 21 for Python PostCommit * Honor JAVA_HOME in JavaJarServer checks and tests * Disable Debezium test on Java17- * add example line * [IO] Update Debezium in DebeziumIO to 3.1.1 (apache#34763) * Updating Debezium IO to 3.1.1 Enforce JDK17 in build.gradle of Debezium IO * adjust to review comments by @Abacn * cleanup * mention which PR needs to merge before unpinning * Mention Beam version 2.66 in the README for Debezium --------- Co-authored-by: Shunping Huang <[email protected]> * Document BQ Storage API pipeline options (apache#35259) * Document BQ Storage API pipeline options * Fix whitespace * Fix whitespace * Speed up StreamingDataflowWorkerTest by removing 10 second wait from shutdown path. (apache#35275) Takes test time from 5m18s to 2m40s. The previous shutdown() call doesn't do anything since that just means future scheduling won't trigger but we only schedule on the executor once. Also cleanup test logs by making sure to stop all workers we start so they don't continue to run in the background and log. This shutdown paths is only used in testing. * Bump org.ajoberstar.grgit:grgit-gradle from 4.1.1 to 5.3.2 (apache#35301) Bumps [org.ajoberstar.grgit:grgit-gradle](https://github.com/ajoberstar/grgit) from 4.1.1 to 5.3.2. - [Release notes](https://github.com/ajoberstar/grgit/releases) - [Commits](ajoberstar/grgit@4.1.1...5.3.2) --- updated-dependencies: - dependency-name: org.ajoberstar.grgit:grgit-gradle dependency-version: 5.3.2 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump google.golang.org/api from 0.235.0 to 0.237.0 in /sdks (apache#35302) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.235.0 to 0.237.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.235.0...v0.237.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-version: 0.237.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/linkedin/goavro/v2 from 2.13.1 to 2.14.0 in /sdks (apache#35205) Bumps [github.com/linkedin/goavro/v2](https://github.com/linkedin/goavro) from 2.13.1 to 2.14.0. - [Release notes](https://github.com/linkedin/goavro/releases) - [Changelog](https://github.com/linkedin/goavro/blob/master/debug_release.go) - [Commits](linkedin/goavro@v2.13.1...v2.14.0) --- updated-dependencies: - dependency-name: github.com/linkedin/goavro/v2 dependency-version: 2.14.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update spotbugs version, fix runner ubuntu version, fix found spotbugs issues (apache#35303) * feat: Add option to control resource cleanup failure for IT (apache#35287) * Revert "Bump org.ajoberstar.grgit:grgit-gradle from 4.1.1 to 5.3.2 (apache#35301)" (apache#35305) This reverts commit 0fd77ce. * Add PeriodicStream in the new time series folder. (apache#35300) * Add PeriodicStream in the new time series folder. * Add some more docstrings and minor fix on test name. * Fix lints and docs. * try buildah to replace kaniko (apache#35289) * try buildah to replace kaniko * trigger post-commit * Adding error handler for SpannerReadSchemaTransformProvider and missi… (apache#35241) * Adding error handler for SpannerReadSchemaTransformProvider and missing tests for SpannerSchemaTransformProvider * Removed not used logging * Spotless Apply * Spotless Apply * Spotless Apply * Typo correction * requests vulnerability. (apache#35308) * [IcebergIO] create custom java container image for tests (apache#35307) * create custom java container image for tests * syntax * eval depends on df * add streaming support to iobase (python) (apache#35137) * Add support for streaming writes in IOBase (Python) * add triggering_frequency in iobase.Sink * fix whitespaces/newlines * fixes per apache#35137 (review) * refactor for code redability * refactor _expand_unbounded , default num_shards to 1 , if undef or 0 * fix formatter * space * keep num_shards = 0 the same as before for bounded write * add streaming to AvroIO, ParquetIO, TFRecordsIO * reformat * typo and spaces * carry on the refactor from apache#35253 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: bullet03 <[email protected]> Co-authored-by: Bulat Safiullin <[email protected]> Co-authored-by: liferoad <[email protected]> Co-authored-by: claudevdm <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Yi Hu <[email protected]> Co-authored-by: Danny McCormick <[email protected]> Co-authored-by: Jack McCluskey <[email protected]> Co-authored-by: Derrick Williams <[email protected]> Co-authored-by: Vitaly Terentyev <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: scwhittle <[email protected]> Co-authored-by: Radosław Stankiewicz <[email protected]> Co-authored-by: Hai Joey Tran <[email protected]> Co-authored-by: Tanu Sharma <[email protected]> Co-authored-by: Kenneth Knowles <[email protected]> Co-authored-by: Minbo Bae <[email protected]> Co-authored-by: Shunping Huang <[email protected]> Co-authored-by: Chenzo <[email protected]> Co-authored-by: kristynsmith <[email protected]> Co-authored-by: Rakesh Kumar <[email protected]> Co-authored-by: Charles Nguyen <[email protected]> Co-authored-by: tvalentyn <[email protected]> Co-authored-by: Ahmed Abualsaud <[email protected]> Co-authored-by: Minh Son Nguyen <[email protected]> Co-authored-by: Amar3tto <actions@GitHub Actions 1000279405.local> Co-authored-by: Vitaly Terentyev <[email protected]> Co-authored-by: Tobias Kaymak <[email protected]> Co-authored-by: Veronica Wasson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Support configuring flush_count and max_row_bytes of WriteToBigTable
Related to this issue: #34760
Our pipeline write 1.3 millions data points every 5 mins with a short burst in < 1 min. Since upgrading to 2.64.0 we started observing this error
google.api_core.exceptions.ResourceExhausted: 429 You have reached the limit of total mutations in your queue. Throttle your usage and wait for operations to finishI guess with current FLUSH_COUNT=1000 it's too low for our usage and we want to be able to configure this
flush_countoption which will probably solve the429error above.GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.