Motivation

Following #9603, I've ran the OpenDAL integration test with OPENDAL_S3_ENABLE_VIRTUAL_HOST_STYLE=on to see if there would be any difference, and 2 tests failed. After investigation, it was because calling ListObjectsV2 would return the keys without the trailing slash, making the test assertions fail. After investigating the difference between the calls, I could see the parsed request was different between the 2 modes:
Virtual-Host before the fix:
PUT opendal-testing.s3.localhost.localstack.cloud:4566/CI/16b716e8-4cd2-4f6e-9d82-ee338b5e48b1/580cc505-82cf-424e-8448-ca4bf9dfd1ac/
-> PutObjectRequest({'ACL': None, 'Bucket': 'opendal-testing', ..., 'Key': 'CI/16b716e8-4cd2-4f6e-9d82-ee338b5e48b1/580cc505-82cf-424e-8448-ca4bf9dfd1ac', 'Metadata': {}...)

Path style:
PUT s3.localhost.localstack.cloud:4566/opendal-testing/CI/cc66a8ac-a924-42b2-950a-b5a9b6bca060/23a5e248-3249-436f-881f-cc1b8dcbffbe/
-> PutObjectRequest({'ACL': None, 'Bucket': 'opendal-testing', ..., 'Key': 'CI/cc66a8ac-a924-42b2-950a-b5a9b6bca060/23a5e248-3249-436f-881f-cc1b8dcbffbe/', 'Metadata': {}...)

As we can see, the Virtual Host mangled the trailing slash, thus creating an object with the wrong key. This was due to a previous assertion in the VirtualHostRewriter which was written to actually mangle it. I've changed it to be closer to our previous proxy, and simply prefix the path with the bucket.

Those test suites are really building confidence in our new S3 implementation, most of them are passing on the first try and some real edge cases are popping up. I'll continue to scour the internet to find S3 test suites to test LocalStack against. Maybe another Apache project which will use LocalStack 😄

Just for info, here are the 2 previously failing tests:

• test_list_nested_dir
• test_list_sub_dir

Changes

• Updated an existing test verifying this behavior for the regular path style (we already have a special case for this in the parser, see localstack.aws.protocol.parser.S3RequestParser._parse_shape) to also test for virtual host mode.
• Modified the VirtualHostRewriter to not mangle the trailing slash
• Also spotted a different issue in the V2 legacy provider, where the v-host proxy was not proxying the request with the raw path, so the test would fail.