[SPARK-54808] Qualified session view#53630
Open
srielau wants to merge 136 commits intoapache:masterfrom
Open
Conversation
- Persistent functions now cached with unqualified keys for compatibility - Temporary functions use composite keys (session.funcName) - Both can coexist in the same registry - Views correctly exclude temp functions from resolution Issue: View test still failing - needs investigation into function builder resolution
- Persistent functions now stored with qualified keys (catalog.db.func) - Prevents conflicts when multiple databases have same function name - Temporary functions still use composite keys (session.func) Known issues: - View function resolution test still failing - Possible function listing regressions to investigate
Added extensive debug logging to understand why views capture wrong function class. Ready for detailed tracing.
**THE BUG:** In resolveBuiltinOrTempFunctionInternal, the 'isBuiltin' parameter was incorrectly checking if the temp/builtin identifier existed in the session registry, instead of checking the static FunctionRegistry.builtin. This caused lookupTempFuncWithViewContext to treat temp functions as builtins, bypassing view context checks and allowing temp functions created AFTER a view to incorrectly shadow the persistent function that the view should use. **THE FIX:** Changed the isBuiltin check to use FunctionRegistry.builtin.functionExists and TableFunctionRegistry.builtin.functionExists directly, matching master's behavior. **TEST RESULTS:** ✅ All 62 tests pass (PersistedViewTestSuite + FunctionQualificationSuite) ✅ SPARK-33692 view test now passes ✅ View correctly uses MyDoubleAvg and ignores temp MyDoubleSum
Removed leftover test scripts that were causing compilation errors: - test_simple_function.scala - test_view_function.scala All code now compiles cleanly.
Analysis covers: - Complete API surface (read/write operations) - Current architecture and memory usage - Three proposed optimization approaches - Detailed feasibility assessment KEY DISCOVERY: Internal functions already use separate static registry! - FunctionRegistry.internal contains ~20 ML/Pandas/Connect functions - Resolved directly, bypassing SessionCatalog - Proves composite registry pattern works in production - Validates proposed optimization approach Memory savings potential: 98% reduction for high-session deployments Implementation effort: 2-3 days coding + testing Risk: Low (pattern already proven with internal functions)
Comprehensive comparison covering: - Registry architecture (cloned vs static) - User-facing vs implementation details - Resolution paths and shadowing behavior - Examples and use cases - Historical context (Spark 4 separation) KEY FINDINGS: - Builtin: ~500 user-facing SQL functions, cloned per session - Internal: ~20 implementation functions for Connect/ML/Pandas, single global registry - Internal functions already use separate static registry pattern - Proves composite registry approach is production-ready This validates our proposed optimization approach for builtins.
Updated test to expect INVALID_TEMP_OBJ_QUALIFIER (AnalysisException) instead of INVALID_SQL_SYNTAX.CREATE_TEMP_FUNC_WITH_DATABASE (ParseException) for invalid temporary function qualifications. This aligns with the user's request to treat invalid temp function qualifications as semantic errors (42602 SQLSTATE) rather than syntax errors. Test cases updated: - CREATE TEMPORARY FUNCTION a.b() - now expects INVALID_TEMP_OBJ_QUALIFIER - CREATE TEMPORARY FUNCTION a.b.c() - now expects INVALID_TEMP_OBJ_QUALIFIER All tests pass.
These files were working notes created during development and should not be committed to the repository: - BUILTIN_VS_INTERNAL_FUNCTIONS.md - CURSOR_IMPLEMENTATION_SUMMARY.md - CURSOR_TEST_RESULTS.md - FUNCTION_QUALIFICATION_ANALYSIS.md - FUNCTION_QUALIFICATION_COMPLETE.md - FUNCTION_QUALIFICATION_SUMMARY.md - FUNCTION_REGISTRY_API_ANALYSIS.md - IMPLEMENTATION_COMPLETE.md - TABLE_FUNCTION_REGISTRY_ANALYSIS.md - UNIFIED_FUNCTION_NAMESPACE.md Only production code and tests should be in the repository.
These were working test files created during development: - test_both.sql - test_namespace.sql - test_range.sql They should not be committed to the repository.
…apsulation Refactoring #1: Extract Scalar/Table Function Duplication - Added handleViewContext() helper to centralize view resolution logic - Added lookupFunctionWithShadowing() generic helper for both scalar and table functions - Eliminated ~50 lines of duplicated shadowing and view context logic - Simplified lookupBuiltinOrTempFunction() and lookupBuiltinOrTempTableFunction() Refactoring #2: Unified Qualification Checker - Added isQualifiedWithNamespace() helper to check namespace qualifications - Refactored maybeBuiltinFunctionName() and maybeTempFunctionName() to use common helper - Eliminated ~15 lines of duplication - Prepares for future PATH implementation Documentation Improvements: - Added comprehensive scaladoc to TEMP_FUNCTION_DB explaining composite key pattern - Added scaladoc to tempFunctionIdentifier() and isTempFunctionIdentifier() - Added detailed comments to new helper methods Benefits: - Single source of truth for shadowing logic - Easier to maintain and test - Better encapsulation of view resolution context - More consistent code structure All tests pass: - FunctionQualificationSuite: 24/24 tests passed - SessionCatalogSuite: 100/100 tests passed
Naming Improvements: - Renamed 'useTempIdentifier' → 'lookupAsTemporary' for clarity - More semantic boolean parameter names Code Quality: - Made handleViewContext() functional (removed imperative 'return') - Removed unused 'tempIdentifier' and 'builtinIdentifier' variables - Extracted resolveFunctionWithFallback() to eliminate duplication - Used Option.filter for more idiomatic Scala Benefits: - More functional programming style - Clearer intent with better parameter names - No unused variables cluttering the code - Extracted common pattern reduces duplication by ~30 lines - More readable and maintainable All 24 tests in FunctionQualificationSuite pass.
Removed sql-function-qualifiers.sql and its golden files. All test coverage is comprehensively provided by FunctionQualificationSuite.scala. Rationale: - Eliminates duplication between SQL and Scala tests - Scala tests provide better error validation with checkError() - Scala tests are easier to maintain and debug - Scala tests have better test isolation - No loss of coverage: Scala suite has 24 tests covering all scenarios FunctionQualificationSuite.scala provides complete coverage: - 8 tests: Reference qualification (SELECT statements) - 9 tests: DDL (CREATE/DROP TEMPORARY FUNCTION) - 4 tests: Type mismatch errors - 3 tests: Integration scenarios All 24 tests pass.
f22e093 to
3c2b3f5
Compare
Followup cleanup for function qualification PR
3c2b3f5 to
2d9fc86
Compare
Resolve conflicts from search-path merge: - Analyzer: keep sessionOrder legacy doc and catalogPath.toSeq style - CheckAnalysis/Catalog: use (catalog +: currentNamespace).toSeq - FunctionResolution: use (name +: namespace).toSeq, drop NonFatal case, keep expandIdentifier then try and re-expand in catch - SessionCatalog: keep HEAD lookupTempFuncWithViewContext and PATH-based resolution - QueryCompilationErrors: keep tableOrViewNotFoundWithSearchPath - ResolveSessionCatalog: keep ResolvedTempView + DropViewInSessionCatalog, use operationNotAllowedOnBuiltinNamespaceError - SparkSqlParser: keep normalizeTempViewIdentifier for session/system.session qualifiers - Tests/sql.out: add queryContext to checkError and JSON outputs where master had them
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?
Was this patch authored or co-authored using generative AI tooling?