Spark on k8s : changes for using supporting multi tenant(multi namespace) k8s cluster - #462

This pull request has been automatically marked as stale because it has had no activity for at least 3 months. If you are still working on this change or plan to move it forward, please leave a comment or push a new commit so we know to keep it open. Otherwise, this PR will be closed automatically in about one month. Thank you for your contribution to Apache Livy!

Rebased onto the latest master and resolved all conflicts — the PR is now mergeable. Summary of what changed since the last review:

Rebase / conflict resolution

• Rebased onto current master (the branch was ~42 commits behind).
• Dropped changes that master has since made independently: the commons.lang3 import and the SQLInterpreterSpec assertion (kept master's broader version).
• Removed redundant per-module pom overrides (maven-shade-plugin, commons-lang3, netty-all, hadoop-common) now that the parent pom manages those versions; one of them would otherwise have downgraded maven-shade-plugin 3.5.0 → 3.4.1.

Version bumps (security) — kept the higher version across master/PR: kubernetes-client 6.8.1, netty 4.1.108.Final. Verified the code compiles and unit tests pass against the fabric8 6.x client.

Bug fix in SparkApp.getNamespace — the previous version unconditionally read $SPARK_HOME/conf/spark-defaults.conf and called .get on sparkHome(), which threw FileNotFoundException/NoSuchElementException and broke batch-session creation (and several existing tests) for non-Kubernetes deployments. It now:

• resolves a namespace only on Kubernetes (empty string otherwise, no filesystem access),
• treats a missing or key-less spark-defaults.conf as "no value" and falls back to the default namespace,
• closes the input stream.

This also addresses @jahstreet's earlier question about whether the namespace can be empty initially.

Tests — added SparkAppSpec covering the full namespace-resolution precedence chain (session conf → spark-defaults.conf → default), plus the non-Kubernetes and missing-SPARK_HOME paths. All touched suites pass locally (SparkAppSpec, BatchSessionSpec, SessionManagerSpec, SparkKubernetesAppSpec); scalastyle is clean.

Commits were squashed into one. @gyogal — could you kick off the check jobs when you get a chance? Thanks!

Note on the watched-namespace truncation idea (@jahstreet): I've left that out of this PR to keep it focused; happy to follow up in a separate change since picking a safe truncation point needs more thought.

Pushed an update that folds in five additional Kubernetes/session fixes that were developed on top of this branch (authored by @vallenki). All are rebased onto current master, compile cleanly against kubernetes-client 6.8.1, pass scalastyle/checkstyle, and have unit tests:

1. Livy Interactive session fixes — adds livy.rsc.driver.address.use-hostname (RSCConf) so the driver's reported hostname is used instead of the socket IP, which is needed for Kubernetes + Istio setups.
2. Reduce Kubernetes API load in polling — narrows the driver lookup to spark-role=driver at the API server (single-pod LIST instead of all executor pods) and gates the executor LIST behind a new flag livy.server.kubernetes.executor-tracking.enabled (default true, so no behavior change). Large clusters can disable it to skip per-executor LISTs.
3. SessionManager refresh + /recovery operator endpoint — a superuser-guarded (livy.superusers, returns 403 otherwise) REST endpoint to re-scan the recovery state store and import sessions written by another Livy server, without a restart.
4. Fix recovery-file leak after session deletion — kill() now removes the app from the shared monitor queue and the monitor early-returns after a terminal state, so a deleted session's recovery file is no longer resurrected by an in-flight monitor tick. killed becomes AtomicBoolean for cross-thread visibility.
5. Fix Promise.failure race that falsely failed healthy apps — switches the catch path to Promise.tryFailure, matching the trySuccess on the success path, so a late failure on an already-completed promise is dropped instead of flipping a healthy app to FAILED. This became reachable once app-lookup.thread-pool.size > 1.

Combined diff is ~545 insertions across 16 files. Happy to split any of these into separate PRs if you'd prefer to keep this one scoped purely to multi-namespace support — just let me know.

@gyogal @jahstreet looks like tests are passing now. Just fyi, we are running Livy server in production with these changes to run spark on k8s jobs.

Hi @ashokkumarrathore , I noticed that there are multiple related PRs opened for Kubernetes enhancements (for example #544, #545 and #546 ) and some of these address issues that your PR also addresses. Could you please take a look to see how those are related to your change and if it would be possible to similarly break up your changes to individual smaller PRs that each focus on a single new feature? Also would it be possible to open a JIRA ticket for each of the remaining changes?

Thanks @gyogal — you're right, this PR had grown to bundle several independent changes, and there's real overlap with the newer Kubernetes PRs. Here's how they relate and how I've split things up:

Already merged upstream — dropped from this PR:

• The driver.address.use-hostname RSC flag is the same change as [LIVY-1071] Add driver.address.use-hostname RSC config flag #544 (LIVY-1071).
• The Kubernetes polling API-load reduction is the same as [LIVY-1072] Reduce Kubernetes API load in SparkKubernetesApp polling #545 (LIVY-1072).

I've rebased this branch onto current master, which already carries both.

Overlaps an open PR — dropped here:

• The SessionManager refresh + /recovery operator endpoint overlaps [LIVY-1073] Add SessionManager refresh and /recovery operator endpoint #546 (LIVY-1073). I'll let [LIVY-1073] Add SessionManager refresh and /recovery operator endpoint #546 be the single source for that.

This PR now focuses on exactly one feature:

• LIVY-461 — Multi-tenant (multi-namespace) Kubernetes support (namespace resolution + inNamespace() request threading). I've force-pushed the branch down to just this change.

Two remaining independent bug fixes — a focused PR + JIRA each:

1. Fix recovery-file leak after session deletion.
2. Fix Promise.failure race that falsely fails healthy Spark apps.

I'll file JIRAs for those two and open a small PR for each, then link them back here.

Codecov Report

❌ Patch coverage is 24.71910% with 67 lines in your changes missing coverage. Please review.
✅ Project coverage is 53.86%. Comparing base (501e822) to head (e1e60d2).
⚠️ Report is 8 commits behind head on master.

@@              Coverage Diff              @@
##             master     #462       +/-   ##
=============================================
- Coverage     68.68%   53.86%   -14.82%     
+ Complexity     1218      865      -353     
=============================================
  Files           106      107        +1     
  Lines          6815     6943      +128     
  Branches        836      846       +10     
=============================================
- Hits           4681     3740      -941     
- Misses         1666     2757     +1091     
+ Partials        468      446       -22     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@gyogal @Rchie121 Can we please merge this?

Hi @ashokkumarrathore , thank you for reworking your PR, it looks much clearer now. We have received multiple Kubernetes related changes from @soumyadeeplogin (#544, #545 and #546) and I would just like to make sure those are compatible with your changes, so if you have any feedback on those it would be much appreciated! Similarly, @soumyadeeplogin if you have any comments on this PR, please let us know, it would help us a great deal!

We are currently in the middle of the Spark 4 upgrade, so once that is merged (in a few days), we could look into merging these changes as well.

In the meantime, could you please update the name of the PR so that it includes the JIRA ticket and please also update the description to include the AI section according to the Livy PR template at https://github.com/apache/livy/blob/master/.github/PULL_REQUEST_TEMPLATE - this would help so that it is not lost among some older stale PRs.

Also, the JIRA issue https://issues.apache.org/jira/browse/LIVY-461 does not seem to be related. Can it be a typo?