[SPARK-59477][SQL] Add CSV write option treatNullAsEmptyString - #58767
Closed
dejankrak-db wants to merge 1 commit into
Closed
[SPARK-59477][SQL] Add CSV write option treatNullAsEmptyString#58767dejankrak-db wants to merge 1 commit into
dejankrak-db wants to merge 1 commit into
Conversation
Adds a per-write CSV option `treatNullAsEmptyString` that overrides the session-level `spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv` config for a single write. When set to `false`, nulls are written as a bare, unquoted empty token while empty strings stay quoted (`""`), so downstream consumers can differentiate a null from an empty string. When set to `true`, nulls are written through the `emptyValue`. When unset (including an explicit `null` value), the session config decides the behavior, so existing writes are unaffected. The option is parsed via the existing `getBool` helper, so a non-boolean value raises the structured `paramIsNotBooleanValueError` naming the option. It only takes effect when `nullValue` is left at its default (empty string); a non-empty `nullValue` is written verbatim, making the option a no-op. Co-authored-by: Isaac <[email protected]>
HyukjinKwon
approved these changes
Sep 13, 2026
uros-b
approved these changes
Sep 13, 2026
uros-b
left a comment
Member
There was a problem hiding this comment.
Thank you @dejankrak-db and @HyukjinKwon!
uros-b
pushed a commit
that referenced
this pull request
Sep 13, 2026
### What changes were proposed in this pull request? Adds a per-write CSV option `treatNullAsEmptyString` that overrides the session-level `spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv` config for a single write. - When set to `false`, nulls are written as a bare, unquoted empty token while empty strings stay quoted (`""`), so downstream consumers can tell a null apart from an actual empty string. - When set to `true`, nulls are written through the `emptyValue` (a quoted empty string `""` by default). - When unset (including an explicit `null` value), the session config decides the behavior, so existing writes are unaffected. The option is parsed via the existing `getBool` helper, so a non-boolean value raises the structured `paramIsNotBooleanValueError` naming the option rather than a bare error. It only takes effect when `nullValue` is left at its default (empty string); a non-empty `nullValue` is written verbatim, making the option a no-op, matching how the SQL config it overrides already composes with `nullValue`. ### Why are the changes needed? The behavior of writing a null vs. an empty string in CSV is currently only controllable through the session-level `spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv` config. A workload that needs to differentiate null from an empty string in its output cannot do so per-write without changing the session/workspace default, which is not always settable. A per-write option lets a single write opt into the differentiation without affecting other writes. ### Does this PR introduce _any_ user-facing change? Yes, it adds a new CSV write option `treatNullAsEmptyString`. Behavior is unchanged when the option is not set (the session config continues to decide), so existing writes are unaffected. ### How was this patch tested? Added unit tests in `CSVSuite` covering: the option winning over the session config for both `true`/`false`, the no-op behavior when a non-empty `nullValue` is set, explicit-null handling (falls back to the session config), and the structured error raised for a non-boolean value. Also updated the `validate CSV Options` test for the new option. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac This pull request and its description were written by Isaac. Closes #58767 from dejankrak-db/csv-treat-null-as-empty-string. Authored-by: Dejan Krakovic <[email protected]> Signed-off-by: Uros Bojanic <[email protected]> (cherry picked from commit 66737e0) Signed-off-by: Uros Bojanic <[email protected]>
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adds a per-write CSV option
treatNullAsEmptyStringthat overrides thesession-level
spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsvconfigfor a single write.
false, nulls are written as a bare, unquoted empty token whileempty strings stay quoted (
""), so downstream consumers can tell a nullapart from an actual empty string.
true, nulls are written through theemptyValue(a quotedempty string
""by default).nullvalue), the session config decidesthe behavior, so existing writes are unaffected.
The option is parsed via the existing
getBoolhelper, so a non-boolean valueraises the structured
paramIsNotBooleanValueErrornaming the option ratherthan a bare error. It only takes effect when
nullValueis left at its default(empty string); a non-empty
nullValueis written verbatim, making the optiona no-op, matching how the SQL config it overrides already composes with
nullValue.Why are the changes needed?
The behavior of writing a null vs. an empty string in CSV is currently only
controllable through the session-level
spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsvconfig. A workloadthat needs to differentiate null from an empty string in its output cannot do
so per-write without changing the session/workspace default, which is not
always settable. A per-write option lets a single write opt into the
differentiation without affecting other writes.
Does this PR introduce any user-facing change?
Yes, it adds a new CSV write option
treatNullAsEmptyString. Behavior isunchanged when the option is not set (the session config continues to decide),
so existing writes are unaffected.
How was this patch tested?
Added unit tests in
CSVSuitecovering: the option winning over the sessionconfig for both
true/false, the no-op behavior when a non-emptynullValueis set, explicit-null handling (falls back to the session config), and the
structured error raised for a non-boolean value. Also updated the
validate CSV Optionstest for the new option.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.