dolthub/dolt#10030: --filter contribution for dolt diff#10097
Merged
dolthub/dolt#10030: --filter contribution for dolt diff#10097
--filter contribution for dolt diff#10097Conversation
Add Go unit tests for the diff filter feature to provide fast feedback and granular validation of filter logic. Test coverage includes: - diffTypeFilter struct validation (isValid method) - Filter inclusion methods (adds, drops, modifications) - Edge cases (empty strings, typos, case sensitivity) - Consistency checks across all filter types - Constant validation (values, uniqueness, lowercase) - Invalid filter behavior verification Tests added: - 12 test functions - 48+ individual test cases - 100% coverage of diffTypeFilter struct methods These tests complement the existing BATS integration tests and provide unit-level regression protection. Refs: #1430
Implement lazy table header initialization to fix a bug where empty table headers were printed when all rows were filtered out during data-only diffs. This occurred because BeginTable() was called before row filtering, causing headers to print even when no matching rows existed. The solution introduces a lazyRowWriter that delays the BeginTable() call until the first row is actually written. This wrapper is only used when: - A filter is active (added/modified/removed) - The diff is data-only (no schema changes or table renames) Implementation changes: - Add shouldUseLazyHeader() helper to determine when to use lazy initialization based on filter presence and diff type - Add lazyRowWriter type that wraps SqlRowDiffWriter and delays BeginTable() until first WriteRow() or WriteCombinedRow() call - Modify diffUserTable() to skip BeginTable when using lazy writer - Modify diffRows() to conditionally create lazyRowWriter vs normal rowWriter based on shouldUseLazyHeader() check - Add comprehensive unit tests for shouldUseLazyHeader logic and lazyRowWriter behavior (5 test functions, 8+ test cases) - Add mock implementations of diffWriter and SqlRowDiffWriter interfaces to enable testing without database dependencies - Fix BATS test assertions to match actual SQL output format (lowercase type names, MODIFY COLUMN vs DROP/ADD pattern) Test coverage: - TestShouldUseLazyHeader: validates lazy header logic conditions - TestLazyRowWriter_NoRowsWritten: verifies BeginTable not called when no rows written (core lazy behavior) - TestLazyRowWriter_RowsWritten: verifies BeginTable called on first write - TestLazyRowWriter_CombinedRowsWritten: tests combined row writes - TestLazyRowWriter_InitializedOnlyOnce: ensures BeginTable called exactly once Refs: #1430
Extract duplicated row filter checking logic into a reusable shouldSkipDiffType helper function. Refs: #1430
Contributor
Constants and Naming: - Add FilterParam constant to flags.go following existing conventions - Move filter type constants (DiffTypeAdded, DiffTypeModified, DiffTypeRemoved, DiffTypeAll) to table_deltas.go for centralization - Update all references throughout codebase to use diff.DiffType* constants for consistency Validation Consolidation: - Consolidate duplicate validation logic into single isValid() method - Introduce newDiffTypeFilter() constructor for proper initialization - Remove redundant validation checks in parseDiffArgs function Map-Based Filter Architecture: - Refactor diffTypeFilter struct to use map[string]bool instead of string field for more extensible filtering - Replace three separate includeXOrAll() methods with single shouldInclude() method that performs map lookup - Update both table-level and row-level filtering to use unified shouldInclude() approach Row-Level Filtering with DiffType: - Add ChangeTypeToDiffType() helper function in table_deltas.go to convert row-level ChangeType enum to table-level DiffType strings - Refactor shouldSkipDiffType() to shouldSkipRow() using the new conversion helper and map-based filtering - Ensure consistent terminology between table and row filtering by using same DiffType string values throughout Refs: #1430
Refactor lazyRowWriter to use a cleaner callback-based architecture that reduces struct complexity from 8 fields to 2 fields. This addresses PR review feedback requesting simplification of the lazy header implementation. Changes: - Replace parameter storage with closure-based callback pattern that captures BeginTable parameters from outer scope - Eliminate separate ensureInitialized() method in favor of inline initialization check in WriteRow() and WriteCombinedRow() - Remove initialized bool field by using callback presence (nil check) to track initialization state - Always create RowWriter upfront and only delay BeginTable call, eliminating the need for writer factory function - Simplify Close() method to always delegate to wrapped writer Refs: #1430
Address @elianddb feedback on PR #10030 by standardizing DiffType usage, simplifying filter logic, and fixing row-level filtering. Changes: - Fix row filtering bug: add early return for ChangeType.None to prevent incorrectly skipping added/removed rows - Standardize DiffType: replace string literals with constants (DiffTypeAdded/Modified/Removed) in table_deltas.go and merge.go - Simplify table filtering: use delta.DiffType directly - Update tests: rewrite to use map-based architecture and newDiffTypeFilter() constructor Refs: #1430
Add DiffTypeRenamed constant and "removed" as user-friendly alias for "dropped" filter option. This provides more granular filtering and improves user experience with familiar terminology. List of changes: - Add DiffTypeRenamed constant for renamed tables - Revert GetSummary to use DiffTypeRenamed instead of treating renames as modified (table_deltas.go:742) - Add "removed" as alias that maps to "dropped" internally for user convenience - Update filter validation to include renamed option - Update CLI help text to document all filter options including renamed and removed alias - Handle renamed tables in merge stats alongside modified - Update getDiffSummariesBetweenRefs and getSchemaDiffSummariesBetweenRefs to handle renamed diff type - Update all tests to use new constants and test renamed filtering Filter options now available: added, modified, renamed, dropped, removed (alias for dropped). Refs: #1430
Add tests for the --filter=renamed option and the --filter=removed alias that maps to dropped. Go tests: - Tests for filter=renamed checking all diff types - Tests for "removed" alias mapping to dropped internally - Verify renamed filter only includes renamed tables BATS tests: - Test --filter=renamed with table rename scenario - Test --filter=dropped with table drop scenario - Verify --filter=removed alias works same as dropped - Verify other filters correctly exclude renamed/dropped tables Refs: #1430
…e/1430/add-filter-opt-for-diff
Update the short help text for the --filter parameter to document all valid filter options (added, modified, renamed, dropped) and mention that 'removed' is accepted as an alias for 'dropped'. Refs: #1430
Contributor
Contributor
|
@coffeegoddd DOLT
|
--filter contribution for dolt diff
--filter contribution for dolt diff--filter contribution for dolt diff
|
@coffeegoddd DOLT
|
|
@coffeegoddd DOLT
|
|
@coffeegoddd DOLT
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Author @codeaucafe
Add
--filteroption todolt diff, enabling filtering by specific change types and fixing issues from the earlier stalled PR (#3499).Close #10030
Fix #1430