Flink integration: Fixed a bug incorrectly loading configuration in Event Emitter #3799
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for opening your first OpenLineage pull request! We appreciate your contribution. If you haven't already, please make sure you've reviewed our guide for new contributors (https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md). |
Collaborator
pawel-big-lebowski
left a comment
There was a problem hiding this comment.
Flink part looks good. Thanks for working on this.
Contributor
Author
|
Thanks. What's next? I removed the readme.md fragment from integrations/sql/iface-java about failing SQL tests, cause Maciej merged the fix. This change is now changing Flink code only, except the .toString() message. I never commited to OL before, is there anything pending on my side? If not, what is about to happen? |
Member
|
@fetta can you rebase? |
Contributor
Author
|
Yup, I'll do my best to do that today |
…vent Emitter, resulting in "disabled facets" feature not working (and probably others as well). Signed-off-by: Jan Siekierski <[email protected]>
…vent Emitter, resulting in "disabled facets" feature not working (and probably others as well). Signed-off-by: Jan Siekierski <[email protected]>
mobuchowski
approved these changes
Jun 23, 2025
|
Great job! Congrats on your first merged pull request in OpenLineage! |
marccampa
pushed a commit
to marccampa/OpenLineage-Collibra
that referenced
this pull request
Jun 26, 2025
…vent Emitter (OpenLineage#3799) * Flink integration: Fixed a bug incorrectly loading configuration in Event Emitter, resulting in "disabled facets" feature not working (and probably others as well). Signed-off-by: Jan Siekierski <[email protected]> * Flink integration: Fixed a bug incorrectly loading configuration in Event Emitter, resulting in "disabled facets" feature not working (and probably others as well). Signed-off-by: Jan Siekierski <[email protected]> --------- Co-authored-by: Jan Siekierski <[email protected]> Signed-off-by: marccampa <[email protected]>
mobuchowski
added a commit
that referenced
this pull request
Jul 3, 2025
* Update consumers.tsx * Add files via upload * Update consumers.tsx * Add files via upload * Update consumers.tsx * Update consumers.tsx * Update consumers.tsx * Update consumers.tsx * Delete Collibra-Logo-RGB.png * Update consumers.tsx * [Flink] Do not hide OpenLineage config parsing errors (#3724) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * Update consumers.tsx Signed-off-by: marccampa <[email protected]> * Add files via upload Signed-off-by: marccampa <[email protected]> * Update consumers.tsx Signed-off-by: marccampa <[email protected]> * Add files via upload Signed-off-by: marccampa <[email protected]> * Update consumers.tsx Signed-off-by: marccampa <[email protected]> * [DBT] Add processing_engine facet (#3725) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * java: prevent original events from being mutated in TransformTransport (#3728) - Add deepCopy utility method to OpenLineageClientUtils for safe object cloning - Modify TransformTransport to create deep copies of events before transformation Signed-off-by: Jakub Dardzinski <[email protected]> Signed-off-by: marccampa <[email protected]> * [Flink] Add processing_engine facet (#3726) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * Add Github stars statistics to Readme (#3730) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * [DBT] Document supported adapters (#3729) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * Prettify Spark JSON event examples (#3740) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * Prettify Flink JSON event examples (#3742) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * Prettify Airflow JSON event examples (#3741) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * Prettify DBT JSON event examples (#3743) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * Restyle cardmedia. (#3733) Signed-off-by: merobi-hub <[email protected]> Signed-off-by: marccampa <[email protected]> * dbt-ol should not error on job complete if there is no start event (#3749) Signed-off-by: Maciej Obuchowski <[email protected]> Signed-off-by: marccampa <[email protected]> * [Flink] Add facet with Flink jobId (#3744) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * Update consumers.tsx Signed-off-by: marccampa <[email protected]> * Update consumers.tsx Signed-off-by: marccampa <[email protected]> * [DBT] Initial support for Clickhouse (#3739) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * [SPEC] Add contentType to documentation facet (#3748) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * [spark] Update Spark 4 dependency to 4.0.0 (remove -preview1 suffix) (#3751) Signed-off-by: Dominik Dębowczyk <[email protected]> Signed-off-by: marccampa <[email protected]> * filter temp inner jobs for bigquery indirect mode (#3722) Signed-off-by: Pawel Leszczynski <[email protected]> Signed-off-by: marccampa <[email protected]> * [Docs] Add documentation for some facets (#3752) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * Tweak the Maven signing config (#3069) This tweak allows Gradle to default on using values set in `~/.gradle/gradle.properties` Signed-off-by: Julien Phalip <[email protected]> Signed-off-by: marccampa <[email protected]> * Run prettier on .json files (#3750) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * remove native proxy (#3680) * remove native proxy Signed-off-by: Maciej Obuchowski <[email protected]> # Conflicts: # proxy/backend/gradle.properties * remove leftover proxy gradle reference Signed-off-by: Kacper Muda <[email protected]> --------- Signed-off-by: Kacper Muda <[email protected]> Co-authored-by: Kacper Muda <[email protected]> Signed-off-by: marccampa <[email protected]> * [DBT] Add DbtRun facet (#3738) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * Remove Airflow < 2.5.0 support (#3669) Signed-off-by: Kacper Muda <[email protected]> Signed-off-by: marccampa <[email protected]> * nit: fix supported airflow versions (#3755) Signed-off-by: Kacper Muda <[email protected]> Signed-off-by: marccampa <[email protected]> * [Java] Speedup generateNewUUID (#3754) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * [DBT] Use adapter rows_affected as outputStatistics (#3731) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * fix variables in docs for setting of root parents in spark config (#3761) Signed-off-by: Humzah Kiani <[email protected]> Signed-off-by: marccampa <[email protected]> * [spark] Add support for Big Query Metastore catalog type (#3760) Signed-off-by: Dominik Dębowczyk <[email protected]> Signed-off-by: marccampa <[email protected]> * Fix visibility of GcpLineageTransportConfig.Mode (#3762) * Register GCP common job facet Signed-off-by: Natalia Gorchakova <[email protected]> * Add ACCEPT_CASE_INSENSITIVE_ENUMS for ObjectMapper to ensure that lower and upper case enum values are accepted for config Signed-off-by: Natalia Gorchakova <[email protected]> * Add ACCEPT_CASE_INSENSITIVE_ENUMS for ObjectMapper to ensure that lower and upper case enum values are accepted for config Signed-off-by: Natalia Gorchakova <[email protected]> * Add ACCEPT_CASE_INSENSITIVE_ENUMS for ObjectMapper to ensure that lower and upper case enum values are accepted for config Signed-off-by: Natalia Gorchakova <[email protected]> * Add ACCEPT_CASE_INSENSITIVE_ENUMS for ObjectMapper to ensure that lower and upper case enum values are accepted for config Signed-off-by: Natalia Gorchakova <[email protected]> --------- Signed-off-by: Natalia Gorchakova <[email protected]> Signed-off-by: marccampa <[email protected]> * Update consumers.tsx Signed-off-by: marccampa <[email protected]> * Delete Collibra-Logo-RGB.png Signed-off-by: marccampa <[email protected]> * update httpConfig Headers and TimeoutInMillis property values (#3767) Signed-off-by: Nidhin Varghese <[email protected]> Signed-off-by: marccampa <[email protected]> * [java] Add log if load from yaml fails (#3766) Signed-off-by: Fiore Mario Vitale <[email protected]> Signed-off-by: marccampa <[email protected]> * smart debug facet (#3715) Signed-off-by: Pawel Leszczynski <[email protected]> Signed-off-by: marccampa <[email protected]> * [Spark] Fix missing table path in InsertIntoHadoopFsRelationCommand (#3773) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * Github: mark Hive PRs with proper label (#3778) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * fix configurable test failin in CI (#3782) Signed-off-by: Pawel Leszczynski <[email protected]> Signed-off-by: marccampa <[email protected]> * Column level lineage for jdbc queries load (#3763) * test column level lineage for jdbc queries load Signed-off-by: Pawel Leszczynski <[email protected]> * refactor jdbc lineage visitor Signed-off-by: Pawel Leszczynski <[email protected]> --------- Signed-off-by: Pawel Leszczynski <[email protected]> Signed-off-by: marccampa <[email protected]> * chore: Use attr.define instead of attr.s (#3776) Signed-off-by: Kacper Muda <[email protected]> Signed-off-by: marccampa <[email protected]> * [Hive] Add job sql facet (#3777) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * build(deps): bump the integration-sql group (#3704) Updates the requirements on [pyo3](https://github.com/pyo3/pyo3) and [pyo3-build-config](https://github.com/pyo3/pyo3) to permit the latest version. Updates `pyo3` to 0.25.0 - [Release notes](https://github.com/pyo3/pyo3/releases) - [Changelog](https://github.com/PyO3/pyo3/blob/main/CHANGELOG.md) - [Commits](PyO3/pyo3@v0.24.0...v0.25.0) Updates `pyo3-build-config` to 0.25.0 - [Release notes](https://github.com/pyo3/pyo3/releases) - [Changelog](https://github.com/PyO3/pyo3/blob/main/CHANGELOG.md) - [Commits](PyO3/pyo3@v0.24.0...v0.25.0) --- updated-dependencies: - dependency-name: pyo3 dependency-version: 0.25.0 dependency-type: direct:production dependency-group: integration-sql - dependency-name: pyo3-build-config dependency-version: 0.25.0 dependency-type: direct:production dependency-group: integration-sql ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: marccampa <[email protected]> * [Hive] Add hive_query facet (#3781) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * Website: correct Node version in README (#3783) * Fix node version in website readme. Signed-off-by: merobi-hub <[email protected]> * Misc fixes. Signed-off-by: merobi-hub <[email protected]> --------- Signed-off-by: merobi-hub <[email protected]> Signed-off-by: marccampa <[email protected]> * [spark] Disable module metadata file generation (#3785) Signed-off-by: Dominik Dębowczyk <[email protected]> Signed-off-by: marccampa <[email protected]> * Add Debezium to producers (#3787) Signed-off-by: Fiore Mario Vitale <[email protected]> Signed-off-by: marccampa <[email protected]> * build(deps): bump requests from 2.32.0 to 2.32.4 in /dev (#3759) Bumps [requests](https://github.com/psf/requests) from 2.32.0 to 2.32.4. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.0...v2.32.4) --- updated-dependencies: - dependency-name: requests dependency-version: 2.32.4 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: marccampa <[email protected]> * [Hive] Add hive_session facet (#3786) * [Hive] Add hive_session facet Signed-off-by: Martynov Maxim <[email protected]> * [Hive] Record hive session creation time Signed-off-by: Martynov Maxim <[email protected]> --------- Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * changelog for release 1.34.0 (#3790) Signed-off-by: Maciej Obuchowski <[email protected]> Signed-off-by: marccampa <[email protected]> * Prepare for release 1.34.0 Signed-off-by: Maciej Obuchowski <[email protected]> Signed-off-by: marccampa <[email protected]> * Prepare next development version 1.35.0-SNAPSHOT Signed-off-by: Maciej Obuchowski <[email protected]> Signed-off-by: marccampa <[email protected]> * chore: Fix changelog item authors (#3791) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * [Hive] Add jobType facet (#3789) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * Website: update README (#3801) * Update website readme. Signed-off-by: merobi-hub <[email protected]> * Fix code blocks. Signed-off-by: merobi-hub <[email protected]> * Fix wordiness. Signed-off-by: merobi-hub <[email protected]> * More details in deployment sec. Signed-off-by: merobi-hub <[email protected]> * Continued. Signed-off-by: merobi-hub <[email protected]> * Continued. Signed-off-by: merobi-hub <[email protected]> * Continued. Signed-off-by: merobi-hub <[email protected]> --------- Signed-off-by: merobi-hub <[email protected]> Signed-off-by: marccampa <[email protected]> * fix spotless in hive integration (#3806) Signed-off-by: Maciej Obuchowski <[email protected]> Signed-off-by: marccampa <[email protected]> * run Java SQL tests (#3808) Signed-off-by: Maciej Obuchowski <[email protected]> Signed-off-by: marccampa <[email protected]> * [Hive] Add docker-compose example for local testing (#3800) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * [DBT] Make invocation_id field optional (#3796) Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> * Remove empty Flink page. (#3810) Signed-off-by: Jakub Dardzinski <[email protected]> Signed-off-by: marccampa <[email protected]> * Flink integration: Fixed a bug incorrectly loading configuration in Event Emitter (#3799) * Flink integration: Fixed a bug incorrectly loading configuration in Event Emitter, resulting in "disabled facets" feature not working (and probably others as well). Signed-off-by: Jan Siekierski <[email protected]> * Flink integration: Fixed a bug incorrectly loading configuration in Event Emitter, resulting in "disabled facets" feature not working (and probably others as well). Signed-off-by: Jan Siekierski <[email protected]> --------- Co-authored-by: Jan Siekierski <[email protected]> Signed-off-by: marccampa <[email protected]> * Website: add missing guidance to readme (#3807) * Add missing guidance to readme. Signed-off-by: merobi-hub <[email protected]> * Img file formats. Signed-off-by: merobi-hub <[email protected]> --------- Signed-off-by: merobi-hub <[email protected]> Signed-off-by: marccampa <[email protected]> * build(deps): bump urllib3 from 1.26.19 to 2.5.0 in /dev (#3794) Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.19 to 2.5.0. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@1.26.19...2.5.0) --- updated-dependencies: - dependency-name: urllib3 dependency-version: 2.5.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: marccampa <[email protected]> * dbt: fix log path, more precise file reading (#3793) Signed-off-by: Maciej Obuchowski <[email protected]> Signed-off-by: marccampa <[email protected]> * Spark: fix & upgrade databricks test (#3811) Signed-off-by: Pawel Leszczynski <[email protected]> Signed-off-by: marccampa <[email protected]> * Formalize dataset naming (#3775) * Formalize dataset naming --------- Signed-off-by: Dominik Dębowczyk <[email protected]> Signed-off-by: marccampa <[email protected]> * Update consumers.tsx Signed-off-by: marccampa <[email protected]> * Apply prettier fix. Signed-off-by: merobi-hub <[email protected]> --------- Signed-off-by: Martynov Maxim <[email protected]> Signed-off-by: marccampa <[email protected]> Signed-off-by: Jakub Dardzinski <[email protected]> Signed-off-by: merobi-hub <[email protected]> Signed-off-by: Maciej Obuchowski <[email protected]> Signed-off-by: Dominik Dębowczyk <[email protected]> Signed-off-by: Pawel Leszczynski <[email protected]> Signed-off-by: Julien Phalip <[email protected]> Signed-off-by: Kacper Muda <[email protected]> Signed-off-by: Humzah Kiani <[email protected]> Signed-off-by: Natalia Gorchakova <[email protected]> Signed-off-by: Nidhin Varghese <[email protected]> Signed-off-by: Fiore Mario Vitale <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Maxim Martynov <[email protected]> Co-authored-by: Jakub Dardzinski <[email protected]> Co-authored-by: Michael Robinson <[email protected]> Co-authored-by: Maciej Obuchowski <[email protected]> Co-authored-by: ddebowczyk92 <[email protected]> Co-authored-by: pawel.leszczynski <[email protected]> Co-authored-by: Julien Phalip <[email protected]> Co-authored-by: Kacper Muda <[email protected]> Co-authored-by: Humzah Kiani <[email protected]> Co-authored-by: ngorchakova <[email protected]> Co-authored-by: Nidhin Varghese <[email protected]> Co-authored-by: Fiore Mario Vitale <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Fiore Mario Vitale <[email protected]> Co-authored-by: Maciej Obuchowski <[email protected]> Co-authored-by: Jan Siekierski <[email protected]> Co-authored-by: Jan Siekierski <[email protected]> Co-authored-by: merobi-hub <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Incorrect configuration resulted in "disabled facets" feature not working (and probably others as well).
Problem
openlineage-flink has a bug where in EventEmitter's constructor, the OpenLineageClient is built with only part of the OpenLineageContext. This is a fix, as well as some logging that would have helped in the diagnosis.
Solution
Instead of only passing transport to the OpenLineageClient builder, we're now calling the Clients.newClient factory method, passing the whole configuration.
One-line summary:
Fixed incorrect EventEmitter constructor
Checklist
SPDX-License-Identifier: Apache-2.0
Copyright 2018-2025 contributors to the OpenLineage project