Implement more ASF S3 operations in provider #6868
Conversation
ddc8710 to
a3f202f
Compare
a3f202f to
f6e0d64
Compare
Local tests results:Once the class is fully succeeding, it won't be showed in subsequent PRs.
|
| / | Before | After |
|---|---|---|
| ✅ Passed | 43 | 61 |
| 🙈 Ignored | 7 | 8 |
| ❗ Failed | 12 | 1 |
| Total | 62 | 70 |
TestS3TerraformRawRequests
| / | Before | After |
|---|---|---|
| ✅ Passed | 0 | 1 |
| 🙈 Ignored | 0 | 0 |
| ❗ Failed | 1 | 0 |
| Total | 1 | 1 |
TestS3PresignedURL
note: the succeeding ones are not validating authorisation
| / | Before | After |
|---|---|---|
| ✅ Passed | 11 | 15 |
| 🙈 Ignored | 3 | 2 |
| ❗ Failed | 14 | 14 |
| Total | 28 | 31 |
TestS3CORS
| / | Before | After |
|---|---|---|
| ✅ Passed | 4 | 0 |
| 🙈 Ignored | 0 | 0 |
| ❗ Failed | 10 | 2 |
| Total | 14 | 2 |
TestS3StorageClass
| / | Before | After |
|---|---|---|
| ✅ Passed | 1 | 2 |
| 🙈 Ignored | 0 | 0 |
| ❗ Failed | 0 | 0 |
| Total | 1 | 2 |
TestS3StaticWebsiteHosting
| / | Before | After |
|---|---|---|
| ✅ Passed | - | 0 |
| 🙈 Ignored | - | 0 |
| ❗ Failed | - | 2 |
| Total | 0 | 2 |
Total
| / | Before | After |
|---|---|---|
| ✅ Passed | 62 | 79 |
| 🙈 Ignored | 10 | 10 |
| ❗ Failed | 34 | 19 |
| Total | 106 | 108 |
f6e0d64 to
58ebd39
Compare
a3c4506 to
51a57c0
Compare
51a57c0 to
3776444
Compare
| { | ||
| "op": "remove", | ||
| "path": "/shapes/DeleteObjectsOutput" | ||
| }, | ||
| { | ||
| "op": "add", | ||
| "path": "/shapes/DeleteResult", |
There was a problem hiding this comment.
question: why exactly are we replacing DeleteObjectsOutput with this new shape? looks like the botocore spec is the same https://github.com/boto/botocore/blob/4d4af71c10187a7c26b5193fb53a4ff8bc8cbfe7/botocore/data/s3/2006-03-01/service-2.json#L3200-L3218
There was a problem hiding this comment.
If I remember well, it's because the response is wrapped in the name of the output shape. Like <DeleteObjectOutput>response members xml</DeleteObjectOutput>.
But S3 returns <DeleteResult> instead. Not sure if I can change the name in any other way. Doesn't seem to have locationName for output shapes. Seems weird and dirty though 🤷♂️
This PR adds more operations to the S3 provider, and reorganised the tests by class, grouping them more coherently.
This will reduces the number of failing test and implement a lot of basic functionality, and fixes some current bugs at the same time.
It is a bit hard to keep track of implemented operation and fixed exceptions, as a lot of fixes are done by moto patching.
Implemented operations:
GetBucketRequestPaymentPutBucketRequestPaymentGetBucketLifecycleConfiguration(had to reimplement it, moto is not validating correctly Transition actions, and will raise exceptions. Would need a bigger rewrite in moto. Needed for Terraform).PutBucketLifecycleConfigurationDeleteBucketLifecycleConfigurationfix object Expires parameter
fix
GetObjectwith range headers, and return proper status codefix restoring object from
DEEP_ARCHIVE, raise error if trying to get an object in that statefix exceptions missing in the specs, or missing members
implement proper DeleteResult for
DeleteObjectsAlso had to regenerate all s3 snapshots as the moving of tests made a lot of changes, it was simpler to regenerate everything from scratch.
Currently having unit tests failing as they are testing patched behaviour against original specs. Will need to fix this before merging.Done, seen with Alex.Sorry this PR is quite large, subsequent PR based on this one will be smaller and implement specific things. This one was quite general.