Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[SPARK-59477][SQL] Add CSV write option treatNullAsEmptyString - #58767

Closed
dejankrak-db wants to merge 1 commit into
apache:masterfrom
dejankrak-db:csv-treat-null-as-empty-string
Closed

[SPARK-59477][SQL] Add CSV write option treatNullAsEmptyString#58767
dejankrak-db wants to merge 1 commit into
apache:masterfrom
dejankrak-db:csv-treat-null-as-empty-string

Conversation

@dejankrak-db

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Adds a per-write CSV option treatNullAsEmptyString that overrides the
session-level spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv config
for a single write.

  • When set to false, nulls are written as a bare, unquoted empty token while
    empty strings stay quoted (""), so downstream consumers can tell a null
    apart from an actual empty string.
  • When set to true, nulls are written through the emptyValue (a quoted
    empty string "" by default).
  • When unset (including an explicit null value), the session config decides
    the behavior, so existing writes are unaffected.

The option is parsed via the existing getBool helper, so a non-boolean value
raises the structured paramIsNotBooleanValueError naming the option rather
than a bare error. It only takes effect when nullValue is left at its default
(empty string); a non-empty nullValue is written verbatim, making the option
a no-op, matching how the SQL config it overrides already composes with
nullValue.

Why are the changes needed?

The behavior of writing a null vs. an empty string in CSV is currently only
controllable through the session-level
spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv config. A workload
that needs to differentiate null from an empty string in its output cannot do
so per-write without changing the session/workspace default, which is not
always settable. A per-write option lets a single write opt into the
differentiation without affecting other writes.

Does this PR introduce any user-facing change?

Yes, it adds a new CSV write option treatNullAsEmptyString. Behavior is
unchanged when the option is not set (the session config continues to decide),
so existing writes are unaffected.

How was this patch tested?

Added unit tests in CSVSuite covering: the option winning over the session
config for both true/false, the no-op behavior when a non-empty nullValue
is set, explicit-null handling (falls back to the session config), and the
structured error raised for a non-boolean value. Also updated the
validate CSV Options test for the new option.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Isaac

This pull request and its description were written by Isaac.

Adds a per-write CSV option `treatNullAsEmptyString` that overrides the
session-level `spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv` config
for a single write.

When set to `false`, nulls are written as a bare, unquoted empty token while
empty strings stay quoted (`""`), so downstream consumers can differentiate a
null from an empty string. When set to `true`, nulls are written through the
`emptyValue`. When unset (including an explicit `null` value), the session
config decides the behavior, so existing writes are unaffected.

The option is parsed via the existing `getBool` helper, so a non-boolean value
raises the structured `paramIsNotBooleanValueError` naming the option. It only
takes effect when `nullValue` is left at its default (empty string); a
non-empty `nullValue` is written verbatim, making the option a no-op.

Co-authored-by: Isaac <[email protected]>

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @dejankrak-db and @HyukjinKwon!

@uros-b uros-b closed this in 66737e0 Sep 13, 2026
uros-b pushed a commit that referenced this pull request Sep 13, 2026
### What changes were proposed in this pull request?

Adds a per-write CSV option `treatNullAsEmptyString` that overrides the
session-level `spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv` config
for a single write.

- When set to `false`, nulls are written as a bare, unquoted empty token while
  empty strings stay quoted (`""`), so downstream consumers can tell a null
  apart from an actual empty string.
- When set to `true`, nulls are written through the `emptyValue` (a quoted
  empty string `""` by default).
- When unset (including an explicit `null` value), the session config decides
  the behavior, so existing writes are unaffected.

The option is parsed via the existing `getBool` helper, so a non-boolean value
raises the structured `paramIsNotBooleanValueError` naming the option rather
than a bare error. It only takes effect when `nullValue` is left at its default
(empty string); a non-empty `nullValue` is written verbatim, making the option
a no-op, matching how the SQL config it overrides already composes with
`nullValue`.

### Why are the changes needed?

The behavior of writing a null vs. an empty string in CSV is currently only
controllable through the session-level
`spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv` config. A workload
that needs to differentiate null from an empty string in its output cannot do
so per-write without changing the session/workspace default, which is not
always settable. A per-write option lets a single write opt into the
differentiation without affecting other writes.

### Does this PR introduce _any_ user-facing change?

Yes, it adds a new CSV write option `treatNullAsEmptyString`. Behavior is
unchanged when the option is not set (the session config continues to decide),
so existing writes are unaffected.

### How was this patch tested?

Added unit tests in `CSVSuite` covering: the option winning over the session
config for both `true`/`false`, the no-op behavior when a non-empty `nullValue`
is set, explicit-null handling (falls back to the session config), and the
structured error raised for a non-boolean value. Also updated the
`validate CSV Options` test for the new option.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Isaac

This pull request and its description were written by Isaac.

Closes #58767 from dejankrak-db/csv-treat-null-as-empty-string.

Authored-by: Dejan Krakovic <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
(cherry picked from commit 66737e0)
Signed-off-by: Uros Bojanic <[email protected]>
@uros-b

uros-b commented Sep 13, 2026

Copy link
Copy Markdown
Member

Merge Summary:

Posted by merge_spark_pr.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants