Fix airflow connections import corrupting JSON values in .env files - #73083
Fix airflow connections import corrupting JSON values in .env files#73083Eason09053360 wants to merge 1 commit into
Conversation
A connection exported with `--file-format env --serialization-format json` could not be read back: the value was always parsed as a URI, so the whole JSON payload silently ended up in the schema column while the command still reported success. That value is the very string one would put in AIRFLOW_CONN_*, where every other secrets backend already deserializes it as JSON.
2aefd6d to
80751d5
Compare
Vamsi-klu
left a comment
There was a problem hiding this comment.
The bug is real. Export writes unquoted {...}, Connection(uri=value) stuffed JSON into schema, and import printed success. The { prefix plus from_json is the right helper, and the round-trip test fails on revert.
Two leftovers: this reimplements the { test instead of BaseSecretsBackend._deserialize_connection_value (strip vs lstrip), and quoted dotenv JSON (CONN_ID='{...}') still fails the prefix check. Export does not quote, so the reported path is fixed. Hand-written quoted .env files are not.
| if isinstance(value, str): | ||
| # A URI can never start with "{" (RFC 3986: the scheme begins with a letter), and a ``.env`` | ||
| # value is the string one would put in ``AIRFLOW_CONN_*`` -- so JSON deserializes as it does there. | ||
| if value.lstrip().startswith("{"): |
There was a problem hiding this comment.
This reimplements the { test instead of calling BaseSecretsBackend._deserialize_connection_value. The helper does value.strip() then value[0] == "{"; this lstrip()s only for the check, then passes the original string.
Also not dotenv: quoted CONN_ID='{"conn_type":...}' still fails the { test and still corrupts. Export does not quote, so the reported path is fine. Hand-written .env files that quote JSON still die the old way.
Why
airflow connections export --file-format env --serialization-format jsonwrites a file thatairflow connections importcannot read back. In_create_connection, a string value from a.envfile was always treated as a URI, andurlsplit()never rejects anything, so the JSONpayload was parsed as a path:
conn_type,host,loginandportcame out empty and the restof the JSON blob landed in the
schemacolumn. The command printedImported connection <id>andexited 0, so the corruption is silent until a task fails to connect.
"conn_type": "mysql", ... }That value is the same string one would put in
AIRFLOW_CONN_*, and every other secrets backendalready accepts both forms there:
BaseSecretsBackend._deserialize_connection_valuetreats a{-prefixed value as JSON and anything else as a URI.local_filesystemwas the only backendmissing that test.
What
airflow-core/src/airflow/secrets/local_filesystem.py: a{-prefixed string value isdeserialized with
Connection.from_json()instead of being parsed as a URI. A URI can neverstart with
{(RFC 3986 requires the scheme to begin with a letter), so the two forms areunambiguous and existing URI files are untouched. Going through
from_json()rather than thismodule's object branch also keeps
.envvalues consistent withAIRFLOW_CONN_*forconn_typenormalization and
portcoercion. Malformed JSON, which previously produced a garbageconnection, now raises a
ValueErrornaming the connection.airflow-core/docs/security/secrets/secrets-backend/local-filesystem-secrets-backend.rst:document that a
.envvalue may be a URI or a JSON object.test_cli_connections_import_should_round_trip_env_file_exported_as_jsoncovers thereported export/import path;
test_env_file_json_connection_is_normalized_like_an_environment_variablepins thepostgresql->postgresand"5432"->5432parity with the environment variable backend..jsonand.yamlfiles still go through this module's object branch, which does not normalizeconn_typeor coerceport. That predates this change and is left alone here — happy to followup separately if reviewers want the two paths unified.