Is your feature request related to a problem? Please describe.
cudf-spark would like to be able to cache and reuse broadcast hash builds across stream batches. Currently the Java APIs for left semi/anti joins construct a filtered join for each call.
Describe the solution you'd like
We can wrap the libcudf::filtered_join in Java/JNI, and add Table.leftSemiJoinGatherMap and Table.leftAntiJoinGatherMap APIs that accept the reusable FilteredJoin object. This would closely follow what has been done for HashJoin and DistinctHashJoin.
This request just concerns equi-joins, additional residual predicates would be out of scope.
Describe alternatives you've considered
Reusable semi/anti joins can be emulated with a reusable HashJoin to perform an inner join, and turning the result into a semi/anti gather maps, at the cost of enormous intermediate gather maps if there are many duplicates.
Additional context
Is your feature request related to a problem? Please describe.
cudf-spark would like to be able to cache and reuse broadcast hash builds across stream batches. Currently the Java APIs for left semi/anti joins construct a filtered join for each call.
Describe the solution you'd like
We can wrap the
libcudf::filtered_joinin Java/JNI, and addTable.leftSemiJoinGatherMapandTable.leftAntiJoinGatherMapAPIs that accept the reusableFilteredJoinobject. This would closely follow what has been done for HashJoin and DistinctHashJoin.This request just concerns equi-joins, additional residual predicates would be out of scope.
Describe alternatives you've considered
Reusable semi/anti joins can be emulated with a reusable HashJoin to perform an inner join, and turning the result into a semi/anti gather maps, at the cost of enormous intermediate gather maps if there are many duplicates.
Additional context