Derive effective predicate for Values #743

martint · 2019-05-10T18:20:43Z

This allows Join inference to push the derived predicate
to the other side when one side is based on a Values node.

For a query like:

 SELECT 1
 FROM orders JOIN (values 'O', 'F') t(x) ON orders.orderstatus = t.x;

It changes the plan from:

  - Output[_col0]
    - Project[]
      - InnerJoin[("orderstatus" = "field")]
        - TableScan[tpch:orders:sf0.01]
            orderstatus := tpch:orderstatus
                 :: [[F], [O], [P]]
        - Values
             ('O')
             ('F')

to:

  - Output[_col0]
    - Project[]
      - InnerJoin[("orderstatus" = "field")]
        - ScanFilter[table = tpch:orders:sf0.01, filterPredicate = ("orderstatus" IN ('F', 'O'))]
             orderstatus := tpch:orderstatus
                  :: [[F], [O]]
        - Filter[filterPredicate = ("field" IN ('F', 'O'))]
          - Values
               ('O')
               ('F')

sopel39 · 2019-05-13T14:19:26Z

presto-main/src/main/java/io/prestosql/sql/planner/EffectivePredicateExtractor.java

@@ -15,12 +15,15 @@

 import com.google.common.collect.ImmutableBiMap;


Can we do something about excessive filter:

- Filter[filterPredicate = ("field" IN ('F', 'O'))]

I saw this problem before with table scans and partition columns.
When partition values from one side of join were pushed to the other side, then there is always filter above table scan with col IN partition_values.
I tried to tackle it in starburstdata/facebook-presto@7081f3e
but it didn't work unfortunately because inference cannot easily reduce

inheritedValue = 1 OR inheritedValue = 2 inheritedValue = tableScanValue tableScanValue = 1 OR tableScanValue = 2

to simply true (after simplification) and TupleDomain doesn't have true contains method

Unfortunately, there's not much we can do about it, and I haven't investigated it further. On the other hand, it's a filter over a ValuesNode, so it's inexpensive.

It could be meaningful though if filter contains 100s of partition values, but it's out of scope of this PR

sopel39 · 2019-05-20T09:22:45Z

presto-main/src/main/java/io/prestosql/sql/planner/EffectivePredicateExtractor.java

+                        return TRUE_LITERAL;
+                    }
+
+                    ExpressionInterpreter interpreter = ExpressionInterpreter.expressionOptimizer(value, metadata, session, typeAnalyzer.getTypes(session, types, value));


maybe ValuesNode should only contain Literal as rows (I think it implicitly does)

Eventuallly, we should probably have a physical vs logical values node or something else to represent that. Currently, a values node inside a correlated subquery could have a reference to the outer scope.

Currently, a values node inside a correlated subquery could have a reference to the outer scope.

That's a problem on it's own. See: #473

It’s not a “problem” per se. It’s legitimate for a query to reference variables from the enclosing scope. After all, filters, projections, subqueries, etc, are nothing but capturing lamba expressions (if you squint a little)

After all, filters, projections, subqueries, etc, are nothing but capturing lamba expressions (if you squint a little)

Still, I don't think correlated symbols should appear out of nothing in any node that contains Expression, but should rather be injected into subquery via some kind of special ProjectCorrelations node. This way subquery support can be narrowed down in various places.

That would be a “bind” operation, just like lambda(x) { ... } binds x.

The next generation of the IR that I’ve been prototyping for a while will likely unify both concepts. Anyway, topic for another time :)

sopel39 · 2019-05-20T09:24:48Z

presto-main/src/main/java/io/prestosql/sql/planner/EffectivePredicateExtractor.java

+                for (int row = 0; row < node.getRows().size(); row++) {
+                    Expression value = node.getRows().get(row).get(column);
+
+                    if (!DeterminismEvaluator.isDeterministic(value)) {


nit: you could still derive domains for other symbols

No, the predicate needs to be an “upper bound”. If we had a single row, we could do that, but otherwise, there’s no way to formulate such a predicate.

Use tables to store the points/polygons used in the tests. This is needed for an upcoming change to derivery predicates from inline tables, which causes the plans used in the tests to constant-fold all the way.

This allows Join inference to push the derived predicate to the other side when one side is based on a Values node. For a query like: SELECT 1 FROM orders JOIN (values 'O', 'F') t(x) ON orders.orderstatus = t.x; It changes the plan from: - Output[_col0] - Project[] - InnerJoin[("orderstatus" = "field")] - TableScan[tpch:orders:sf0.01] orderstatus := tpch:orderstatus :: [[F], [O], [P]] - Values ('O') ('F') to: - Output[_col0] - Project[] - InnerJoin[("orderstatus" = "field")] - ScanFilter[table = tpch:orders:sf0.01, filterPredicate = ("orderstatus" IN ('F', 'O'))] orderstatus := tpch:orderstatus :: [[F], [O]] - Filter[filterPredicate = ("field" IN ('F', 'O'))] - Values ('O') ('F')

cla-bot bot added the cla-signed label May 10, 2019

martint force-pushed the effective-predicate-values branch 9 times, most recently from 54add8e to 53e0e2a Compare May 13, 2019 01:23

sopel39 reviewed May 13, 2019

View reviewed changes

martint force-pushed the effective-predicate-values branch from 53e0e2a to 650ff6f Compare May 13, 2019 16:19

sopel39 approved these changes May 20, 2019

View reviewed changes

martint force-pushed the effective-predicate-values branch from 650ff6f to 1a11e03 Compare May 20, 2019 15:50

martint added 3 commits May 20, 2019 19:10

Format spatial join tests

1f6bd1a

Clean up spatial join tests

d52da46

Use tables to store the points/polygons used in the tests. This is needed for an upcoming change to derivery predicates from inline tables, which causes the plans used in the tests to constant-fold all the way.

martint force-pushed the effective-predicate-values branch from 1a11e03 to 4163de2 Compare May 21, 2019 02:10

martint merged commit 03fd878 into trinodb:master May 21, 2019

martint mentioned this pull request May 21, 2019

Release notes for 312 #776

Closed

6 tasks

weiatwork mentioned this pull request Feb 23, 2023

Skip non-comparable & non-orderable field when extracting VALUES predicate #16242

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Derive effective predicate for Values #743

Derive effective predicate for Values #743

martint commented May 10, 2019 •

edited

Loading

sopel39 May 13, 2019

martint May 13, 2019

sopel39 May 13, 2019 •

edited

Loading

sopel39 May 20, 2019

martint May 20, 2019

sopel39 May 20, 2019

martint May 20, 2019

sopel39 May 20, 2019

martint May 20, 2019

sopel39 May 20, 2019

martint May 20, 2019

		@@ -15,12 +15,15 @@

		import com.google.common.collect.ImmutableBiMap;

Derive effective predicate for Values #743

Derive effective predicate for Values #743

Conversation

martint commented May 10, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sopel39 May 13, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martint commented May 10, 2019 •

edited

Loading

sopel39 May 13, 2019 •

edited

Loading