Changes to support Join Pushdown for JDBC Connectors #24992
Conversation
0c6e27c to
a707a80
Compare
Co-Authored-By: Haritha Koloth <[email protected]> Co-Authored-By: Glerin Pinhero <[email protected]> Co-Authored-By: Thanzeel Hassan <[email protected]> Co-Authored-By: Anant Aneja <[email protected]>
a707a80 to
70f9851
Compare
| @@ -2631,7 +2631,7 @@ private Session inferNullsFromJoinFiltersWithUseFunctionMetadata() | |||
| .build(); | |||
| } | |||
|
|
|||
| private long getTableRowCount(String tableName) | |||
| protected long getTableRowCount(String tableName) | |||
| } | ||
|
|
||
| @Test | ||
| public void testRowFieldAccessorInJoin() |
There was a problem hiding this comment.
The tests below this line are all identical to the base class. For these tests, please create a test class where AbstractTestQueryFramework#getSession is overriden and optimizer_inner_join_pushdown_enabled is enabled
| @@ -88,7 +88,7 @@ | |||
| <dependency> | |||
| <groupId>com.google.guava</groupId> | |||
| <artifactId>guava</artifactId> | |||
| <scope>test</scope> | |||
| <scope>provided</scope> | |||
| @@ -58,13 +58,15 @@ public Map<ConnectorId, Set<ConnectorPlanOptimizer>> getOptimizers(PlanPhase pha | |||
| return ImmutableMap.copyOf(transformValues(planOptimizerProviders, ConnectorPlanOptimizerProvider::getLogicalPlanOptimizers)); | |||
| case PHYSICAL: | |||
| return ImmutableMap.copyOf(transformValues(planOptimizerProviders, ConnectorPlanOptimizerProvider::getPhysicalPlanOptimizers)); | |||
| case STRUCTURAL: | |||
There was a problem hiding this comment.
Let's discuss offline on the need for STRUCTURAL. I think we can make do with just LOGICAL
| columns1.get(1), Domain.onlyNull(DOUBLE))); | ||
|
|
||
| Connection connection = database.getConnection(); | ||
| try (PreparedStatement preparedStatement = new QueryBuilder("\"").buildSql(jdbcClient, session, connection, "", "test_schema", "test_table_1", Optional.empty(), columns1, ImmutableMap.of(), tupleDomain, Optional.empty()); |
There was a problem hiding this comment.
Please split parameters on different lines and keep line length under 180
|
|
||
| List<ConnectorTableHandle> newConnectorTableHandles = new ArrayList<>(); | ||
| Map<VariableReferenceExpression, ColumnHandle> groupAssignments = new HashMap<>(); | ||
| final String aliasPrefix = "T"; |
There was a problem hiding this comment.
Can you add a comment here -
Generate aliases for each table being joined to avoid any ambuigious column name references. These aliases are used in QueryBuilder#buildSql
| handle.getSchemaName(), handle.getTableName(), handle.getJoinTables(), Optional.of(aliasPrefix + (++tableCount))); | ||
| newConnectorTableHandles.add(newHandle); | ||
|
|
||
| // Update column handles using Streams |
There was a problem hiding this comment.
I couldn't understand this comment
|
|
||
| /** | ||
| * JdbcJoinPushdown Optimizer: | ||
| * This Optimizer converts ConnectorHandleSet with various TableHandles to JdbcTableHandle which has list of all the JdbcTableHandles that can be pushed down. |
There was a problem hiding this comment.
| * This Optimizer converts ConnectorHandleSet with various TableHandles to JdbcTableHandle which has list of all the JdbcTableHandles that can be pushed down. | |
| * This Optimizer converts a {@link JoinTableSet} to a {@link JdbcTableHandle} with it's {@link JdbcTableHandle#joinTables} set to the | |
| * those from {@link JoinTableSet#innerJoinTableInfos} |
|
|
||
| RowExpression remainingPredicates = logicalRowExpressions.combineConjuncts(remainingExpressions); | ||
| /// !!! For now, we only create a new FilterNode with the remaining predicates | ||
| // As per https://github.com/prestodb/presto/pull/16864, this is not sufficient and we should re-use the existing filter predicate for correctness |
There was a problem hiding this comment.
@rschlussel @NikhilCollooru
Do you recall what was the scenario where a partial pushdown of a filter (while retaining the non-pushed down part in Presto) was causing correctness issues mentioned in #16864 ?
We want to reintroduce the changes from #16412, and we'd like to know if we need to put these behind a feature-flag
| Map<VariableReferenceExpression, ColumnHandle> symbolToColumnHandleMap) | ||
| { | ||
| if (tableHandle instanceof JoinTableSet) { | ||
| // We now have all relevant pieces to build a JdbcJoinPredicateToSqlTranslator to test if the passed in filter will translate to SQL |
There was a problem hiding this comment.
I think this comment can be removed
Description
This PR implements connector-side support for join pushdown in JDBC connectors, leveraging the foundational changes introduced in 24115. The changes enable JDBC connectors to perform join pushdown by rebuilding the query to execute joins directly on the remote datasource.
Motivation and Context
Fixes 23152
Impact
This update includes the connector-side changes required to enable join pushdown in JDBC connectors, building on the foundational SPI and core changes introduced earlier.
Detailed RFC - prestodb/rfcs#32
Test Plan
Unit tests added for newly added logic
Contributor checklist
Release Notes
Please follow release notes guidelines and fill in the release notes below.