sync with open source how #118

lesterhaynes · 2024-03-14T08:15:08Z

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

This reverts commit c72970d.

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.44.0 to 0.45.0. - [Commits](golang/crypto@v0.44.0...v0.45.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.45.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Upgrade bom. * Test fix. * License. * License.

* Remove dead code. * Lint.

Fixed documentation parameter names in WriteToKafka.__init__: - key_deserializer -> key_serializer - value_deserializer -> value_serializer Fixes #36931

…d HEAD state

* Update python-pipeline-dependencies.md * Update python-pipeline-dependencies.md

* Include Beam version details in MongoDB handshake * Linting * Formatting * Use __version__ * formatter

Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.255.0 to 0.256.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.255.0...v0.256.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-version: 0.256.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Reuse secret instead of creating every time * Reuse secret instead of creating every time * lint * lint

…d record in output builder

* Catch Import Error * import order

@ksobrenat32

* Update users.yml - fix bad role label @ksobrenat32 - is this correct? looks like only `beam_writer` failed. fyi @yalah5084 the previous change did not add the privileges so I'm trying to fix it here. * Update users.yml

…s. (#37294) Co-authored-by: Claude <cvandermerwe@google.com>

* Update neo4j resource manager * Fix a test log spam in it/common * Add back javadoc

Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.39.0 to 0.40.0. - [Commits](golang/sys@v0.39.0...v0.40.0) --- updated-dependencies: - dependency-name: golang.org/x/sys dependency-version: 0.40.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.48.0 to 0.49.0. - [Commits](golang/net@v0.48.0...v0.49.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.49.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…type_hints() (#37299) * [Python] Fix AttributeError in ExternalTransform.expand by using get_type_hints() (Fixes #37289) * Fix lint: wrap long comment at line 808 * Fix yapf formatting and ignore local venv

* Remove standalone uses of the apitools HttpError class * Update sdks/python/apache_beam/testing/pipeline_verifiers_test.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * linting and formatting * more linting and formatting * even more linting --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…37324)

* Add LRU cache eviction to CachingStateProvider Fixes #37213 Implements LRU (Least Recently Used) cache eviction to prevent unbounded memory growth in long-running workers. Adds configurable maxCacheSize parameter (default: 1000 entries) and maintains LRU order using JavaScript Map's insertion order. - Add maxCacheSize constructor parameter with default value of 1000 - Implement evictIfNeeded() to remove oldest entry when cache is full - Implement touchCacheEntry() to move accessed items to end (LRU) - Add comprehensive test coverage in state_provider_test.ts This addresses the TODO comment in the code and improves reliability for production workloads. * Address review comments: size-based LRU eviction for CachingStateProvider - Fixed bug: removed incorrect evictIfNeeded() call in promise callback - Removed unnecessary this_ variable (arrow functions capture this) - Changed from count-based to size-based eviction (similar to Python statecache.py) - Added estimateSize() to calculate memory weight of cached values - Default cache weight: 100MB - Updated tests to work with weight-based eviction * Fix prettier formatting * Address review comments: circular references, eviction ordering, tests - Fixed sizeof function to handle circular references using visited Set - Fixed eviction ordering: add to cache first, then evict (fixes edge case) - Added test for oversized item that exceeds maxCacheWeight - Implemented custom sizeof instead of object-sizeof package (has Node.js compatibility issues) * Address Gemini comments: fix race condition, optimize evictIfNeeded - Fixed critical race condition in promise callback: only update cache if the entry is still the same promise we're resolving - Optimized evictIfNeeded: use entries() iterator and removed redundant checks

…37325) Bumps [keras](https://github.com/keras-team/keras) from 3.12.0 to 3.13.1. - [Release notes](https://github.com/keras-team/keras/releases) - [Commits](keras-team/keras@v3.12.0...v3.13.1) --- updated-dependencies: - dependency-name: keras dependency-version: 3.13.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add Terraform files to deploy Envoy RateLimiter * fix variables * Add nat creation command to readme * add hpa for memory * fix redability comments * fix comments * add license * Update examples/terraform/envoy-ratelimiter/README.md Co-authored-by: Danny McCormick <dannymccormick@google.com> --------- Co-authored-by: Danny McCormick <dannymccormick@google.com>

…d has a flaky test. (#37317)

Co-authored-by: damccorm <actions@main-runner-vplcm-927dk.local>

Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.20.19 to 1.21.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@feature/s3/manager/v1.20.19...v1.21.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-version: 1.21.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* porting and some improvements * fix some publisher issues * checkstyle, import, etc changes * add changes info on datadog * updated formatting and correct issue link * fix format issue * fix comments * add precommit for datadog * fix nullable annotation

…7366) Bumps [github.com/linkedin/goavro/v2](https://github.com/linkedin/goavro) from 2.14.1 to 2.15.0. - [Release notes](https://github.com/linkedin/goavro/releases) - [Commits](linkedin/goavro@v2.14.1...v2.15.0) --- updated-dependencies: - dependency-name: github.com/linkedin/goavro/v2 dependency-version: 2.15.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

github-actions bot added build java python examples go infra kotlin learning model labels Mar 14, 2024

stankiewicz and others added 21 commits November 27, 2025 11:02

propagate offset and record in output builder

81a838c

Revert "Fix CSP issues (#36902)" (#36929)

782a0e7

This reverts commit c72970d.

Fix runQuickstartJavaFlinkLocal (#36946)

43fdd50

Update BOM to 26.72.0 (#36919)

dbf4e5a

* Upgrade bom. * Test fix. * License. * License.

Remove python 3.9

d40da98

Remove dead code. (#36943)

e96c8dd

* Remove dead code. * Lint.

Fix typo in WriteToKafka docstring: deserializer -> serializer (#36934)

efed19a

Fixed documentation parameter names in WriteToKafka.__init__: - key_deserializer -> key_serializer - value_deserializer -> value_serializer Fixes #36931

Enforce version upper bound for scikit-learn. (#36948)

0fb3736

Merge pull request #36924: Fix linkage check script to handle detache…

c490d2d

…d HEAD state

Reflect that dill is no longer a default pickler. (#36903)

0592f4f

* Update python-pipeline-dependencies.md * Update python-pipeline-dependencies.md

Add nvidia license

d137814

Enforce hypothesis version upper bound (#36952)

bb8373f

Include Beam version details in MongoDB handshake (#36949)

f242a88

* Include Beam version details in MongoDB handshake * Linting * Formatting * Use __version__ * formatter

Upgrade pip-licenses

065a3ae

Reuse secret instead of creating every time (#36955)

cbc31be

* Reuse secret instead of creating every time * Reuse secret instead of creating every time * lint * lint

Merge pull request #36926: Offset deduplication - Propagate offset an…

ead63ad

…d record in output builder

[Java] Dataflow runner v1 - Propagate drain mode (#36534)

81bb506

Remove nvidia requirements

bfea1e3

Add support for sharding while avro write (#36933)

79ea2e8

Abacn and others added 30 commits January 13, 2026 14:53

Bump Python dev container version (#37293)

1330346

Catch breaking import error (#37295)

752da38

* Catch Import Error * import order

Update users.yml - fix bad role label (#37192)

4dcc27d

* Update users.yml - fix bad role label @ksobrenat32 - is this correct? looks like only `beam_writer` failed. fyi @yalah5084 the previous change did not add the privileges so I'm trying to fix it here. * Update users.yml

Support picosecond tiemstamps when writing GenericRecord and Beam Row…

aad5864

…s. (#37294) Co-authored-by: Claude <cvandermerwe@google.com>

Add new features to changes.md (#37308)

331b08e

Bump golang.org/x/text from 0.32.0 to 0.33.0 in /sdks (#37301)

daa0f15

Update neo4j resource manager (#37310)

fc39ee6

* Update neo4j resource manager * Fix a test log spam in it/common * Add back javadoc

Revert spanner version (#37305)

097e90a

Fix java versions in release guide (#37314)

674763d

[Python] Fix AttributeError in ExternalTransform.expand by using get_…

68e0d66

…type_hints() (#37299) * [Python] Fix AttributeError in ExternalTransform.expand by using get_type_hints() (Fixes #37289) * Fix lint: wrap long comment at line 808 * Fix yapf formatting and ignore local venv

add back ticks to highlight command to be used (#37321)

3aaf839

Increase timeout for Python YAML RC validation job (#37322)

60f1981

[Do Not Merge] Test bumping httplib2 upper bound to modern version (#…

c489cb7

…37324)

fix: prevent NoSuchElementException in AppendFilesToTables (#37217)

17198be

Fix spanner io hang by closing connections (#37327)

cc53097

Pin spanner to old version in gcp bom (#37329)

82ebcb2

[Dataflow Streaming] Remove GetWorkBudgetRefresher which is unused an…

600bd61

…d has a flaky test. (#37317)

Update Python Dependencies (#37361)

fb06f1b

Co-authored-by: damccorm <actions@main-runner-vplcm-927dk.local>

Fix typo in workflow name for updating dependencies (#37362)

e4f48ad

Merge pull request #37286 Fix link due to trailing slashes.

8a43f5e

Add more info to CHANGES on logical type issue (#37368)

e7c80cd

Add Datadog IO (#37319)

84b467b

* porting and some improvements * fix some publisher issues * checkstyle, import, etc changes * add changes info on datadog * updated formatting and correct issue link * fix format issue * fix comments * add precommit for datadog * fix nullable annotation

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

sync with open source how #118

sync with open source how #118

Uh oh!

lesterhaynes commented Mar 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

sync with open source how #118

Are you sure you want to change the base?

sync with open source how #118

Uh oh!

Conversation

lesterhaynes commented Mar 14, 2024

GitHub Actions Tests Status (on master branch)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants