[Bug] Fix invalid validation for sampled column statistics with ndv=0 and non-null min/max#64206
[Bug] Fix invalid validation for sampled column statistics with ndv=0 and non-null min/max#64206MilanTyagi2004 wants to merge 3 commits into
Conversation
Issue Number: close apache#64122 Problem Summary: Under sample analysis, count and nullCount are scaled/rounded estimates and can differ, while min/max are retrieved from full scans. This mismatch causes `ColStatsData.isValid()` to reject valid statistics because `ndv == 0 && (!isNull(minLit) || !isNull(maxLit)) && nullCount != count`. This PR fixes the issue by passing `isSample` to clamp `ndv` to 1. - Test: Unit Test (ColStatsDataTest) - Behavior changed: No - Does this need documentation: No
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
The TeamCity compile failure appears unrelated to the changes in this PR. The build fails while downloading gzip: stdin: not in gzip format This causes Could a maintainer please check or rerun the CI? Thank you. |
|
这个 pr 对应的场景是 ndv 不可用,但 min/max/num_nulls 可用. 在这种场景下希望继续使用 min/max/num_nulls, 所以将 ndv 设置为 1. 是吗? |
|
感谢反馈。 当前实现是参考 issue 讨论中提到的方向：当 sampled statistics 中 ndv=0，但 min/max 表明存在非 NULL 值时，将 ndv 从 0 归一化为 1，从而保留 min/max 和 nullCount 信息，而不是直接丢弃整条列统计信息。 不过我理解您的担忧：将 ndv 设置为 1 可能会影响后续 StatsCalculator 的统计推导。 您更倾向于哪种方案：
我可以按照您建议的方向修改。 |
FE Regression Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 28870 ms |
TPC-DS: Total hot run time: 169071 ms |
You can see this issue: #64122. |
…mple analyze success ### What problem does this PR solve? Issue Number: close apache#64122 Related PR: apache#64206 Problem Summary: SAMPLE ANALYZE fails because ColStatsData.isValid() rejects statistics where ndv = 0, min/max is non-null, and nullCount != count. To fix this, we allow statistics collection and persistence to succeed without ndv normalization. When statistics are loaded later, they are detected as suspicious and treated as UNKNOWN ColumnStatistic by the optimizer. ### Release note None ### Check List (For Author) - Test: - Unit Test (ColStatsDataTest) - Behavior changed: Yes (Suspicious statistics with ndv=0 and non-null min/max are now persisted instead of raising exceptions, and treated as UNKNOWN by Nereids optimizer) - Does this need documentation: No
|
run buildall |
|
The failing checks appear unrelated to the statistics changes in this PR. FE UT failure reports parser errors such as:
which do not correspond to any code introduced by this PR. The Performance job also appears to fail during environment cleanup/startup rather than statistics execution. I have reviewed the modified files in this PR and do not see any introduced lambda ( Could a maintainer help verify whether these failures are caused by an infrastructure or upstream issue, or rerun the checks after rebasing onto the latest master? |
|
@MilanTyagi2004 do you still work on this issue? |
|
can you open the compile pipeline log ? PR had compile failed |
|
apology for the delay |
@MilanTyagi2004, I'm very glad to hear that you will continue with this work. When can you start working on it again? This is a very important repair and we hope to complete it as soon as possible. |
I will start working today on this issue and present the best solution in next upcoming days |
### What problem does this PR solve? Issue Number: close apache#64122, close apache#64206 Problem Summary: In PR apache#50574, logic was added to StatsCalculator to treat statistics with ndv=0, min/max not null, and nullCount == count as UNKNOWN (which represents sampling an all-null tablet). However, this check was only implemented inside a single overload of StatsCalculator#getColumnStatistic, which is bypassed by scan optimization nodes (like OlapScan, HiveScan, and partition-stats merge). As a result, suspicious statistics under the nullCount == count case were not treated as UNKNOWN during CBO estimation. We unify the validation rules into ColumnStatistic.isSuspiciousStats(...) by removing the `nullCount != count` check. This ensures that any suspicious statistics (ndv == 0 but min/max not null) are automatically deserialized as ColumnStatistic.UNKNOWN at the cache level, making it consistent for all optimizer scan paths. Added unit tests to ColStatsDataTest to cover the nullCount == count case. ### Release note None ### Check List (For Author) - Test: Unit Test - Unit Test (ColStatsDataTest.java) - Behavior changed: No - Does this need documentation: No
|
/review |
|
run buildall |
|
Hi @morrySnow, |
What problem does this PR solve?
Issue Number: close #64122
Problem Summary:
ColStatsData.isValid()applies a validation rule that assumes all statistics originate from the same source. However, during SAMPLE ANALYZE this assumption does not always hold:min/maxvalues may come from a full-table scan.ndvmay be estimated from sampled data.nullCountmay be a scaled estimate and therefore differ fromcount.As a result, sampled column statistics can be incorrectly rejected when:
This PR introduces sample-aware handling by adding an
isSampleflag toColStatsData. For sampled statistics, whenndv == 0but non-nullmin/maxvalues exist,ndvis normalized to1before validation. This prevents valid sampled statistics from being rejected while preserving the existing validation behavior for full analyze jobs.Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Test Details
Added unit tests in
ColStatsDataTestcovering:Full analyze statistics:
ndv = 0min/maxnullCount != countExpected result: validation fails.
Sample analyze statistics:
ndv = 0min/maxnullCount != countExpected result:
ndvis normalized to1