43 Pull requests merged by 20 people

• Add yapf version upgrade doc to discussion-docs for 2025

  #34871 merged May 6, 2025

• Encode paneinfo with PaneInfoCoder. (#34824)

  #34864 merged May 6, 2025

• [release-2.65] Cherrypick PR #34867 to the release branch

  #34868 merged May 6, 2025

• fix: correct spanner column schema type parser

  #34867 merged May 6, 2025

• Improve clarity of PTransformRunnerFactory in Java SDK harness

  #34846 merged May 6, 2025

• Add label when PR gets reassigned

  #34866 merged May 6, 2025

• Update breaking changes with PaneInfoCoder change.

  #34865 merged May 6, 2025

• Fix Tour of Beam Go Integration tests

  #34854 merged May 6, 2025

• Revert "Bump @octokit/request and @octokit/rest in /scripts/ci/issue-report"

  #34857 merged May 6, 2025

• Use checkerframework Nullable/NonNull

  #34851 merged May 6, 2025

• Bump github.com/nats-io/nats-server/v2 from 2.11.2 to 2.11.3 in /sdks

  #34838 merged May 6, 2025

• Modify the assert to accept any instance of AvroCoder, including subclasses

  #34850 merged May 6, 2025

• Change Assert to Check instanceof AvroCoder

  #34843 merged May 5, 2025

• Cherrypick - Prism windowed value coder (#34830)

  #34842 merged May 5, 2025

• Fix Nexmark Dataflow V2 use runner v2

  #34823 merged May 5, 2025

• Update code-change-guide.md

  #34776 merged May 5, 2025

• use batch delete for GCS IO

  #34835 merged May 5, 2025

• Encode paneinfo with PaneInfoCoder.

  #34824 merged May 5, 2025

• Add OrderedList and Set state

  #34836 merged May 5, 2025

• Add powershell command 32307

  #34837 merged May 5, 2025

• Bump github.com/golang-cz/devslog from 0.0.11 to 0.0.13 in /sdks

  #34758 merged May 4, 2025

• Bump github.com/go-sql-driver/mysql from 1.9.1 to 1.9.2 in /sdks

  #34759 merged May 4, 2025

• Bump google.golang.org/api from 0.229.0 to 0.231.0 in /sdks

  #34784 merged May 4, 2025

• Prism windowed value coder

  #34830 merged May 4, 2025

• Fix loopback

  #34678 merged May 4, 2025

• Fix test retry on throttling

  #34786 merged May 2, 2025

• Add changes note about TFRecord support in beam yaml and rename integration test.

  #34635 merged May 2, 2025

• Add name to BQ File Loads lambda to ensure changes are update compat

  #34813 merged May 2, 2025

• Add name to BQ File Loads lambda to ensure changes are update compatible

  #34807 merged May 1, 2025

• Cherrypick Managed config validation revert

  #34805 merged May 1, 2025

• Fix issue link in trivial_inference.py

  #34803 merged May 1, 2025

• Revert "[ManagedIO] Fail expansion when encountering extra or unknown configuration"

  #34802 merged May 1, 2025

• Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.17.73 to 1.17.74 in /sdks

  #34785 merged May 1, 2025

• Update CHANGES.md after 2.64 release cut

  #34793 merged Apr 30, 2025

• Update CHANGES.md

  #34791 merged Apr 30, 2025

• Adjusting the YAML Kafka Managed I/O compat version

  #34790 merged Apr 30, 2025

• Fail Fast if Resources Do Not Exist in Kafka Cluster.

  #34659 merged Apr 30, 2025

• Updates YAML SDK to replace Kafka read/write transforms with equivalent managed transforms

  #34755 merged Apr 30, 2025

• [Dataflow Streaming] BoundedQueueExecutor: Add an experiment to use fair monitor

  #34787 merged Apr 30, 2025

• Spark Runner : Support for Streaming side-inputs for Spark Runner

  #34560 merged Apr 30, 2025

• [KafkaIO] Improve caching in backlog estimation and processing

  #34331 merged Apr 30, 2025

• [ManagedIO] Fail expansion when encountering extra or unknown configuration

  #34525 merged Apr 30, 2025

• Support writing to Pubsub with ordering key; Add PubsubMessage SchemaCoder

  #31608 merged Apr 30, 2025

31 Pull requests opened by 16 people

• [Do not merge] Test Fork Option compile

  #34788 opened Apr 30, 2025

• [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

  #34794 opened Apr 30, 2025

• Fix for emojis rendering issue for yaml examples (resolves #34770)

  #34795 opened May 1, 2025

• [IcebergIO] Add Iceberg SQL table provider and tests

  #34799 opened May 1, 2025

• [DO NOT MERGE] Prototype yapf 0.43.0 migration

  #34801 opened May 1, 2025

• Revert "Unpin Dataflow legacy worker container for Nexmark test (#33224)"

  #34804 opened May 1, 2025

• [DO NOT MERGE] bigquery pico brainstorming

  #34806 opened May 1, 2025

• [BEAM-12164]: Make the spanner change stream connector metadata table ParentTokens column nullable.

  #34812 opened May 2, 2025

• Bump github.com/testcontainers/testcontainers-go from 0.36.0 to 0.37.0 in /sdks

  #34815 opened May 2, 2025

• use blob.exists to check the GCS file

  #34818 opened May 2, 2025

• [DO NOT MERGE] prototyping ValueKind

  #34820 opened May 2, 2025

• Update Beam website to release 2.65.0

  #34821 opened May 2, 2025

• [IcebergIO] Support filter pushdown during reads

  #34827 opened May 3, 2025

• Add checkpoint during progress reporting.

  #34828 opened May 3, 2025

• Add Git commit SHA to artifacts using buildnumber-maven-plugin (#18227)

  #34833 opened May 4, 2025

• Bump github.com/nats-io/nats.go from 1.41.2 to 1.42.0 in /sdks

  #34839 opened May 5, 2025

• Update website documentation and add notebook example for python sdk RRIO

  #34841 opened May 5, 2025

• [AnomalyDetection] Add a notebook for using iforest for anomaly detection

  #34845 opened May 5, 2025

• Update Iceberg table field documentation

  #34847 opened May 5, 2025

• fix direct vulnerabilities from jetty

  #34849 opened May 5, 2025

• Log warning instead of raising an exception for unsupported pickle option

  #34852 opened May 6, 2025

• Bump golang.org/x/text from 0.24.0 to 0.25.0 in /sdks

  #34853 opened May 6, 2025

• [IcebergIO] Support column pruning

  #34856 opened May 6, 2025

• Enable certain Beam module compile with newer Java version

  #34858 opened May 6, 2025

• Fix parquet-avro vulnerability in io expansion service

  #34860 opened May 6, 2025

• [WIP] Update Error Prone for ThreadSafe analysis

  #34861 opened May 6, 2025

• Update trivial inference for Python 3.13

  #34870 opened May 6, 2025

• reorder opt-out review comment

  #34872 opened May 6, 2025

• 34749 added cache for avro coder to reduce memory footprint

  #34873 opened May 6, 2025

• Cloudpickle deterministic

  #34874 opened May 7, 2025

• Replace deprecated model version.

  #34875 opened May 7, 2025

22 Issues closed by 10 people

• [Bug]: Reshuffle with default windows encodes PaneInfo with FastPrmitivesCoder

  #34826 closed May 6, 2025

• [Failing Test]: Tour of Beam Go Integration Test broken

  #34817 closed May 6, 2025

• The Generate issue report job is flaky

  #34855 closed May 6, 2025

• [Failing Test]: PostCommit Nexmark Dataflow Runner v2 actually runs on Dataflow legacy runner

  #34822 closed May 5, 2025

• [Bug]: Slow when deleting the temp files from BQ EXPORT

  #34834 closed May 5, 2025

• Performance Regression or Improvement: pytorch_image_classification_benchmarks-resnet152-GPU-mean_inference_batch_latency_micro_secs:mean_inference_batch_latency_micro_secs

  #34810 closed May 5, 2025

• [Bug]: Prism failed on pipelines with reshuffle after windowing

  #34829 closed May 4, 2025

• [Bug]: External transforms cannot be instantiated with LOOPBACK mode.

  #34594 closed May 4, 2025

• [Bug]: ./gradlew :sdks:python:wordCount (Python environment setup check) fails for py3.9

  #34819 closed May 3, 2025

• [Failing Test]: TestGCSIORetry.test_retry_on_throttling failed due to unknown reasons

  #34736 closed May 2, 2025

• [Failing Test]: PostCommit Python Xlang IO Direct Failing

  #34796 closed May 2, 2025

• [Bug]: Managed transform incompatible schema

  #34797 closed May 2, 2025

• [Bug]: BQ File Loads breaks update compatability

  #34808 closed May 2, 2025

• The PostCommit Python Xlang IO Direct job is flaky

  #32809 closed May 2, 2025

• The PostCommit Python Xlang IO Dataflow job is flaky

  #33253 closed May 2, 2025

• The IcebergIO Managed Integration Tests on Dataflow job is flaky

  #34809 closed May 2, 2025

• [Feature Request]: Automatically replace Beam YAML Kafka source/sink with managed I/O

  #34767 closed Apr 30, 2025

• [Bug]: KafkaIO unbounded read requires kafka connection on pipeline submission time

  #34630 closed Apr 30, 2025

• [Bug]: Beam YAML provider docs show unsupported provider configuration

  #34646 closed Apr 30, 2025

• The PreCommit Python Coverage job is flaky

  #30813 closed Apr 30, 2025

• Support streaming side-inputs in the Spark runner.

  #18136 closed Apr 30, 2025

• Add ability to Write to GCP PubSub with an orderingKey

  #21162 closed Apr 30, 2025

8 Issues opened by 8 people

• [Feature Request]: Add Support for Python 3.13

  #34869 opened May 6, 2025

• [Bug]: Spanner schemas with ARRAY<STRING(xxx)> fail to parse

  #34863 opened May 6, 2025

• [Bug]: Samza runner fails to run a pipeline with Custom Window followed by Reshuffle

  #34831 opened May 3, 2025

• [Feature Request]: make some of the dependencies optional/behind a feature flag

  #34816 opened May 2, 2025

• Performance Regression or Improvement: pytorch_image_classification_benchmarks-resnet152-GPU-mean_load_model_latency_milli_secs:mean_load_model_latency_milli_secs

  #34811 opened May 1, 2025

• [Bug]: Workflow hangs during dependency installation with Pip 25.1

  #34798 opened May 1, 2025

• [Bug]: apache_beam.io.gcp.bigquery.ReadFromBigQuery with EXPORT is slow when handling large amount of data

  #34792 opened Apr 30, 2025

• [IcebergIO][Feature Request]: Add support for query filters and projection

  #34789 opened Apr 30, 2025

58 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• add graceful restart mechanism for GetWorkStream to prevent DEADLINE_…

  #34367 commented on May 5, 2025 • 13 new comments

• use WindmillChannelFactory to control what types of channels to generate

  #34653 commented on May 6, 2025 • 10 new comments

• Streamline non-cached state backed iterable.

  #34746 commented on May 6, 2025 • 5 new comments

• complete implementation of open ai text embedding with test #new

  #34700 commented on May 6, 2025 • 4 new comments

• Concat protos in BQStorageWriteAPI - solve edge cases during mering of nested repeated fields

  #34436 commented on May 6, 2025 • 3 new comments

• unbounded PCollections writes to files support in iobase derived IOs: AvroIO, ParquetIO , TextIO, TFRecordIO

  #34777 commented on May 6, 2025 • 2 new comments

• [WIP] Update Debezium in DebeziumIO to 3.1.1

  #34763 commented on Apr 30, 2025 • 2 new comments

• Improve failure message when passing Pipeline object / PBegin objects…

  #34716 commented on May 2, 2025 • 2 new comments

• Disable logical type cast of fastavro

  #34603 commented on Apr 30, 2025 • 1 new comment

• [AnomalyDetection] Add a notebook for anomaly detection with Z-Score

  #34459 commented on May 7, 2025 • 1 new comment

• ✨ Upgrade sidepanel extension to JupyterLab 4.x compatibility [DO NOT MERGE]

  #34495 commented on May 2, 2025 • 0 new comments

• SnowflakeIO: filter on db and schema when searching for existing table

  #34486 commented on May 5, 2025 • 0 new comments

• Enabling long-running jobs to use federated STS assume role authentication for AWS resources.

  #34440 commented on May 5, 2025 • 0 new comments

• sdks/python: enrich data with CloudSQL

  #34398 commented on May 6, 2025 • 0 new comments

• Bump @octokit/plugin-paginate-rest, @actions/github and @octokit/rest in /scripts/ci/pr-bot

  #34377 commented on May 5, 2025 • 0 new comments

• [Java]Add Map Type Support to JsonToRow Transformer

  #34347 commented on May 1, 2025 • 0 new comments

• Update pypi documentation 30145

  #34329 commented on May 6, 2025 • 0 new comments

• Add Triton Inference Server Support

  #34252 commented on Apr 30, 2025 • 0 new comments

• Add PyTorch DistilBERT Sentiment Analysis streaming pipeline for ML Benchmarks

  #34577 commented on May 6, 2025 • 0 new comments

• Python PTransform wrapper for AWS SQS

  #34581 commented on May 1, 2025 • 0 new comments

• Support customizing how built-in types are pickled for cloudpickle

  #34699 commented on May 6, 2025 • 0 new comments

• Attempt prism for pipelines with unbounded PCollections.

  #34721 commented on May 6, 2025 • 0 new comments

• Parse struct returned from Dataflow API to BoundedTrieData

  #34738 commented on Apr 30, 2025 • 0 new comments

• 34749 reflect datum factory cache

  #34750 commented on May 6, 2025 • 0 new comments

• Support configuring flush_count and max_row_bytes of WriteToBigTable

  #34761 commented on Apr 30, 2025 • 0 new comments

• Fix PreCommit YAML Xlang Direct job

  #34762 commented on May 6, 2025 • 0 new comments

• Drain Mode as WindowedValue extension

  #34764 commented on May 2, 2025 • 0 new comments

• Make dill optional and fix coders.

  #34769 commented on May 5, 2025 • 0 new comments

• Yaml IT - Phase 3a

  #34782 commented on Apr 30, 2025 • 0 new comments

• [Bug][Prism]: Prism gets stuck when trying to flatten 2 unbounded pcollections

  #33815 commented on May 1, 2025 • 0 new comments

• [Feature Request]: Allowed write to custom sink from unbounded source

  #25598 commented on May 1, 2025 • 0 new comments

• [Bug]: Support error handling in PyTransform

  #32332 commented on May 2, 2025 • 0 new comments

• Add documentation in Portable Runner to submit job in Java SDK

  #20617 commented on May 3, 2025 • 0 new comments

• [Feature Request]: Running Word-Count with Gradle for PowerShell

  #32307 commented on May 5, 2025 • 0 new comments

• [Bug]: YAML examples don't render emojis

  #34770 commented on May 5, 2025 • 0 new comments

• [Feature Request]: Deterministic serialization of DoFns

  #34410 commented on May 6, 2025 • 0 new comments

• [Bug]: AvroCoder uses a lot of memory because it keeps instantiating the same DatumReader and DatumWriter

  #34749 commented on May 6, 2025 • 0 new comments

• The PostCommit Python Dependency job is flaky

  #30799 commented on May 6, 2025 • 0 new comments

• [yaml]: Normalize BigtableIO

  #28672 commented on May 6, 2025 • 0 new comments

• [Feature Request]: Add a basic doc explaining Beam's security model

  #30911 commented on May 6, 2025 • 0 new comments

• Write just one file per window with WriteToFiles transform

  #20676 commented on May 6, 2025 • 0 new comments

• The pr-bot-pr-updates job is flaky

  #34731 commented on May 7, 2025 • 0 new comments

• The pr-bot-new-prs job is flaky

  #34724 commented on May 7, 2025 • 0 new comments

• Replace StorageV1 client with GCS client - Draft

  #28733 commented on May 4, 2025 • 0 new comments

• Modify JVM options if enableHeapDumps is specified to dump heap in directory that MemoryMonitor will look in.

  #32953 commented on Apr 30, 2025 • 0 new comments

• Adding Google Storage Requester pays feature to Golang SDK.

  #33236 commented on May 1, 2025 • 0 new comments

• Bump @octokit/request-error, @actions/github and @octokit/rest in /scripts/ci/pr-bot

  #33998 commented on May 5, 2025 • 0 new comments

• Switch to use registerFileSystemsOnce for SerializablePipelineOptions constructor

  #34028 commented on May 2, 2025 • 0 new comments

• [BEAM-6394] Add support to write protobuf data using ProtoParquetReader

  #34063 commented on May 3, 2025 • 0 new comments

• Fix Docker build error by adding fallback for python3.12-distutils

  #34144 commented on May 2, 2025 • 0 new comments

• Bump @octokit/plugin-paginate-rest and @octokit/rest in /scripts/ci/issue-report

  #34167 commented on May 2, 2025 • 0 new comments

• Rethrowing Exception from CassandraIO's ReadFn

  #34191 commented on May 6, 2025 • 0 new comments

• Fix ProtoCoder NoSuchMethodException

  #34194 commented on May 2, 2025 • 0 new comments

• [KafkaIO] Remove duplicate offset in range check

  #34201 commented on May 2, 2025 • 0 new comments

• Add support for top-level table properties table creation

  #34205 commented on May 6, 2025 • 0 new comments

• [Java] Add parsedData to Hl7v2Message and Update HL7v2IO Docs

  #34213 commented on May 6, 2025 • 0 new comments

• [Java] Ensure Pipeline Execution Requires Configuration Options or Logs Warning

  #34220 commented on May 2, 2025 • 0 new comments

• feat:large-row-skip-in-bigtable | added experimental options to skip …

  #34245 commented on May 3, 2025 • 0 new comments