I do not think lakeFSFS compatibility run automatically on PRs (because reasons), so I ran them here.

The S3 Gateway redirect failure is due to AWS SDK v2 (bundled with Hadoop 3.4.1) intercepting all 3xx responses at the S3 service layer and treating them as region redirects, rather than following them as HTTP redirects. The QueryStringSignerType signer, which was required to prevent double-signing of presigned URLs, also no longer exists in SDK v2. This is not fixable via configuration. The redirect test has been restricted to Spark 3 only.

1. We must provide a reference, for users and/or for ourselves.
2. Is it possible to use a suitable Hadoop version which can do this? (Possibly related to above question.)

lakeFSFS presigned mode provides the same direct-to-storage performance benefit as the redirect mode and is the recommended alternative for Spark 4 users.

I worry this is false. It provides the same direct-to-storage performance benefits. But it requires very different configuration - which our users might not have. It forces all users to change to that configuration - now they need to configure lakeFSFS with blockstore credentials. This configuration might not be possible.

@arielshaqed

1. We must provide a reference, for users and/or for ourselves.
2. Is it possible to use a suitable Hadoop version which can do this? (Possibly related to above question.)

Here's the additional context and sources that should have been in the original description:

Sources for the redirect incompatibility

The dependency chain:

• Spark 4.0.0 upgrades Hadoop from 3.3.4 to 3.4.1 and removes aws-java-sdk-bundle (AWS SDK V1 jar) (Spark 4.0.0 Release Notes)
• Hadoop 3.4.0 migrated S3A from AWS SDK v1 to v2 (HADOOP-18073)

AWS SDK v2 does not follow HTTP redirects at the transport layer. This appears to be an intentional design decision so that redirect handling can be done at the SDK layer rather than the HTTP layer (aws-sdk-java-v2#975, aws-sdk-java-v2#989).

Hadoop's S3A DefaultS3ClientFactory uses two SDK HTTP clients (configured in AWSClientConfig), neither of which follows redirects:

• ApacheHttpClient (sync): redirect following is explicitly disabled via .disableRedirectHandling() (source)
• NettyNioAsyncHttpClient (async): Netty doesn't support redirects by default, and the AWS SDK netty client doesn't seem to expose any way to enable it.

When the lakeFS S3 gateway returns a 307 redirect, the SDK treats it as an error and unmarshalls it into an S3Exception. Hadoop's S3AUtils.translateException() wraps 301/307 status codes as AWSRedirectException, expecting an S3 region redirect with an x-amz-bucket-region header. lakeFS doesn't set this header, so the region resolves to null. (S3AUtils.java)

Possible workaround

Spark 4 offers a Hadoop-free build where users provide their own Hadoop version. Pairing Spark 4 with Hadoop 3.3.x (SDK v1) could preserve redirect support. I haven't been able to find any information to know how well this would work. It's also not an option on managed platforms like Databricks or EMR where the Hadoop version is fixed.

Impact on users who cannot use lakeFSFS

The PR description recommended lakeFSFS presigned as the alternative. This works for users who can install custom JARs, but as you mentioned, that may not be an option.

From my understanding, these are the use cases that are impacted:

• Databricks SQL Warehouses do not allow external JARs (lakeFS Databricks docs), so lakeFSFS is not an option. These users are limited to the S3 Gateway, and on Spark 4 the loss of redirect means all data must proxy through lakeFS. For users with large data volumes this is a significant performance and cost concern.
• Databricks Unity Catalog "does not respect cluster configurations for filesystem settings" when accessing data through Unity Catalog (Databricks docs).

For these users, there is currently no direct-to-storage path on Spark 4.