Allow conditional loading of plugins #24965

pdabre12 · 2025-04-23T22:52:14Z

Description

Adds logic for conditional loading of the presto-native-sidecar-plugin plugin module.

Motivation and Context

Solves part of #24964

Impact

Users would be able to include the presto-native-sidecar-plugin plugin in the build, and not load the functionalities if coordinator-sidecar-enabled is set to false.

Test Plan

Unit tests

Contributor checklist

Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

Release Notes

== NO RELEASE NOTE ==

pdabre12 · 2025-04-23T23:03:16Z

Logs from running PrestoServer locally:

2025-04-23T15:51:11.214-0700	INFO	main	com.facebook.presto.server.PluginManager	Registering connector hudi
2025-04-23T15:51:11.216-0700	INFO	main	com.facebook.presto.server.PluginManager	-- Finished loading plugin ../presto-hudi/pom.xml --
2025-04-23T15:51:11.216-0700	INFO	main	com.facebook.presto.server.PluginManager	-- Skipping loading disabled plugin ../presto-native-sidecar-plugin/pom.xml --
2025-04-23T15:51:11.227-0700	INFO	main	com.facebook.presto.metadata.StaticCatalogStore	-- Loading catalog properties etc/catalog/blackhole.properties --

rschlussel · 2025-04-24T06:13:59Z

The general model for all plugins has been that you should be able to load plugins without having any side effects if they aren't enabled. Also, as far as cluster configuration, it's inconvenient to have to disable the plugin in addition to setting use side car to false. instead, i think it would be better for the NativeSessionpropertyProviders to have different names (currently the coordinator based one and the sidecar based one are both called native-worker), and loadSystemSessionProperty providers only loads the providers that are relevant for the configuration (similar to how AccessControlManager.loadSystemAccessControl(), but with a different/slightly more complicated combination of configs.

pdabre12 · 2025-04-25T18:44:28Z

@rschlussel Thanks for the response.
I think it might be much simpler to have annotations on the sidecar specific functionalities as shown in this PR.
We can mark the other functionalities aka. NativeTypeManager, NativeFunctionNamespaceManager as dependent on coordinator-sidecar-enabled as well.
From a user's POV, all they would need to do even if the plugin was loaded is set coordinator-sidecar-enabled to false, to not use the sidecar functionalities. What do you think?

rschlussel

This seems like a fine option

is it possible to add a test that everything starts up properly now if the plugin is loaded but the config is not enabled?
For some sidecar features it might be nice to have an additional flag that enables them (e.g. you want to disable sidecar constant folding at first, but still use the function registry). Would this kind of approach still work if something needs two configs to be enabled?

pdabre12 · 2025-04-28T20:54:00Z

This seems like a fine option

is it possible to add a test that everything starts up properly now if the plugin is loaded but the config is not enabled?

For some sidecar features it might be nice to have an additional flag that enables them (e.g. you want to disable sidecar constant folding at first, but still use the function registry). Would this kind of approach still work if something needs two configs to be enabled?

For 1, I will try to come up with a test case.
For 2,
Maybe we could have the @CondtionalOnConfigs annotation taking in an array of ConditionalOnConfig annotations declared as,


public @interface ConditionalOnConfigs {
ConditionalOnConfig[] value(),

I cannot think of a reason why we should not be able to have additional configs, we even have individual load methods for the sidecar functionalities like loadFunctionNamespaceManager(), etc, which allows us to run different functions for checking if a functionality is enabled/disabled.

Though, there's a possibility using part of the native-sidecar-plugin functionalities and disabling the others might break something (maybe need test cases to ensure they work fine independently).

pdabre12 · 2025-05-01T18:40:44Z

@rschlussel
Added the unit test.
The logic is to install the NativeSidecarPlugin on both the java and c++ clusters but not set the coordinator-sidecar-enabled(default false). If we are still able to run queries , means both the clusters were up without any issues. Will this work?

For some sidecar features it might be nice to have an additional flag that enables them (e.g. you want to disable sidecar constant folding at first, but still use the function registry). Would this kind of approach still work if something needs two configs to be enabled?

For supporting multiple configs, added a new annotation which handles that, still could not find out how to unit test this though. Maybe we can let this change pass, create an issue and handle the unit test at a later time?

pdabre12 · 2025-05-01T18:42:47Z

We only need the annotation for NativeTypeManager and the NativeSessionPropertyProvider, the other functionalities aka NativeFunctionNamespaceManager (using the StaticFunctionNamespaceStore) and NativePlanChecker are loaded through a configuration file and if these configuration files aren't present, we will not load them.

ZacBlanco · 2025-05-01T21:41:22Z

presto-common/src/main/java/com/facebook/presto/common/ConditionalOnConfigs.java

+
+@Retention(RetentionPolicy.RUNTIME)
+@Target(ElementType.TYPE)
+public @interface ConditionalOnConfigs


Why do we need this variant?

It handles a list of @ConditionalOnConfig.
The idea is to have the ability to have additional config flags on the sidecar functionalities to conditionally load individual functionalities.
As in let's say on the NativeFunctionNamespaceManager, we have two @ConditionalOnConfig configs coordinator-sidecar-enabled and load-function-registry.

ZacBlanco · 2025-05-01T21:42:10Z

presto-main-base/src/main/java/com/facebook/presto/metadata/FunctionAndTypeManager.java

@@ -420,8 +427,11 @@ public void loadTypeManager(String typeManagerName)

    public void loadTypeManagers()
    {
-        for (String typeManagerName : typeManagerFactories.keySet()) {
-            loadTypeManager(typeManagerName);
+        Map<String, String> configMap = ImmutableMap.of("coordinator-sidecar-enabled", String.valueOf(serverConfig.isCoordinatorSidecarEnabled()));


Do we have a static constant anywhere that we can reference rather than repeating this string?

No, but can make one.

ZacBlanco · 2025-05-01T21:44:14Z

presto-main-base/src/main/java/com/facebook/presto/metadata/SessionPropertyManager.java

@@ -150,8 +157,13 @@ public void loadSessionPropertyProvider(String sessionPropertyProviderName, Opti

    public void loadSessionPropertyProviders()
    {
-        for (String sessionPropertyProviderName : workerSessionPropertyProviderFactories.keySet()) {
-            loadSessionPropertyProvider(sessionPropertyProviderName, functionAndTypeManager, nodeManager);
+        Map<String, String> configMap = ImmutableMap.of("coordinator-sidecar-enabled", String.valueOf(serverConfig.isCoordinatorSidecarEnabled()));


I'm wondering if there's a better way to load the map of properties so that this code is more generic. It seems tailored specifically for the sidecar. If any other plugin needed to make use of ConditionalOnConfig, the property would need to get added here which is not ideal.

It seems like the best option.
We could have the factories provide the required properties, but we would need to push the config values down to the factories, which doesn't seem to be possible.

It might be possible to have just one generic configMap with the properties, because if the properties aren't present in the provider class we just assume it to be true

@ZacBlanco
What if we get a list of all the config properties using injector.getInstance(ConfigurationFactory.class).getProperties(); in PrestoServer.java and pass it down to PluginManager.java.
While loading the plugin, check the annotations to see if all the properties are satisfied, if yes load it.

Alternatively , if we don't want to use annotations, we can add a getRequiredConfigs() method in both Plugin.java and CoordinatorPlugin.java which would return a map of required properties to load this plugin.
Any plugin can override this method, returns its own set of configs, which can be validated before loading the plugin, making this validation generic for all plugins.
What do you think of this approach?

However, for the second approach, loading individual components of plugins like NativeSidecarPlugin might be tricky. For that use case, the annotations approach would be easier to handle.

I think the second approach should be okay. If you have the sidecar configured, as long as it's possible to set the particular component (e.g. constant folding thing) to the non-sidecar version without issue, it should be okay to have it loaded as long as there's no side effect. I think just making sure different implementations of the component (e.g. java builtin vs. sidecar) are registered different names should generally be enough.

How to control the loading of functionalities of the sidecar? Like loading the function registry without the constant folding thing.
For eg: If you do not want to use NativeTypeManager and you load the NativeSidecarPlugin with coordinator-sidecar-enabled true, we load the NativeTypeManager which is the default type manager now, in this case, we want to skip loading the native type manager itself.

Also, to clarify we are talking about the getRequiredConfigs() approach right?
That approach has the config properties defined at the Plugin level not at the individual components level.

i think it's okay if it's loaded as long as it's not configured as the one to use and loading it in itself won't break anything.

and yes getRequiredConfigs() approach.

amitkdutta · 2025-05-05T23:49:11Z

@pdabre12 @ZacBlanco @rschlussel Folks will be really helpful if we can merge this one. Without this change, we can't include the plugin in build and can't test out any of the good changes coming with it. CC: @kewang1024

prestodb-ci added the from:IBM PR from IBM label Apr 23, 2025

pdabre12 force-pushed the conditionally-load-sidecar-plugin branch 2 times, most recently from dbf3e2f to 2fc78ff Compare April 23, 2025 22:58

pdabre12 force-pushed the conditionally-load-sidecar-plugin branch 2 times, most recently from 9b2570c to a019db9 Compare April 25, 2025 18:40

rschlussel previously approved these changes Apr 28, 2025

View reviewed changes

pdabre12 dismissed rschlussel’s stale review via 242315d May 1, 2025 18:30

pdabre12 force-pushed the conditionally-load-sidecar-plugin branch from a019db9 to 242315d Compare May 1, 2025 18:30

pdabre12 changed the title ~~Allow conditional loading of plugins~~ [sidecar] Allow conditional loading of sidecar plugin May 1, 2025

pdabre12 marked this pull request as ready for review May 1, 2025 18:44

pdabre12 requested review from elharo and a team as code owners May 1, 2025 18:44

pdabre12 requested a review from ZacBlanco May 1, 2025 18:44

prestodb-ci requested review from a team, ScrapCodes and imsayari404 and removed request for a team May 1, 2025 18:44

pdabre12 force-pushed the conditionally-load-sidecar-plugin branch 2 times, most recently from 5fb0555 to d4d22da Compare May 1, 2025 19:17

pdabre12 requested a review from rschlussel May 1, 2025 19:19

ZacBlanco requested changes May 1, 2025

View reviewed changes

pdabre12 requested a review from ZacBlanco May 6, 2025 15:09

pdabre12 force-pushed the conditionally-load-sidecar-plugin branch from d4d22da to 3fb82c6 Compare May 7, 2025 00:50

pdabre12 changed the title ~~[sidecar] Allow conditional loading of sidecar plugin~~ Allow conditional loading of plugins May 7, 2025

pdabre12 force-pushed the conditionally-load-sidecar-plugin branch from 3fb82c6 to 521129b Compare May 7, 2025 00:53

Allow conditional loading of plugins

4dbb12e

pdabre12 force-pushed the conditionally-load-sidecar-plugin branch from 521129b to 4dbb12e Compare May 7, 2025 01:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow conditional loading of plugins #24965

Allow conditional loading of plugins #24965

pdabre12 commented Apr 23, 2025 •

edited

Loading

pdabre12 commented Apr 23, 2025

rschlussel commented Apr 24, 2025

pdabre12 commented Apr 25, 2025 •

edited

Loading

rschlussel left a comment

pdabre12 commented Apr 28, 2025 •

edited

Loading

pdabre12 commented May 1, 2025

pdabre12 commented May 1, 2025

ZacBlanco May 1, 2025

pdabre12 May 1, 2025

ZacBlanco May 1, 2025

pdabre12 May 1, 2025

ZacBlanco May 1, 2025

pdabre12 May 1, 2025

pdabre12 May 1, 2025 •

edited

Loading

pdabre12 May 2, 2025

pdabre12 May 2, 2025 •

edited

Loading

rschlussel May 6, 2025

pdabre12 May 6, 2025

pdabre12 May 6, 2025

rschlussel May 6, 2025

rschlussel May 6, 2025

amitkdutta commented May 5, 2025

Allow conditional loading of plugins #24965

Are you sure you want to change the base?

Allow conditional loading of plugins #24965

Conversation

pdabre12 commented Apr 23, 2025 • edited Loading

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Release Notes

pdabre12 commented Apr 23, 2025

rschlussel commented Apr 24, 2025

pdabre12 commented Apr 25, 2025 • edited Loading

rschlussel left a comment

Choose a reason for hiding this comment

pdabre12 commented Apr 28, 2025 • edited Loading

pdabre12 commented May 1, 2025

pdabre12 commented May 1, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdabre12 May 1, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdabre12 May 2, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amitkdutta commented May 5, 2025

pdabre12 commented Apr 23, 2025 •

edited

Loading

pdabre12 commented Apr 25, 2025 •

edited

Loading

pdabre12 commented Apr 28, 2025 •

edited

Loading

pdabre12 May 1, 2025 •

edited

Loading

pdabre12 May 2, 2025 •

edited

Loading