Object delete in one way Bucket Replication #11451
Description
Very simple bucket to bucket replication use case. Wanting deletes to propagate to destination buckets. Similar to issue 11440
Expected Behavior
Expecting deletes (delete markers) to replicate to destination bucket. So destination bucket when viewed appears like the source.
Put a file on source. ls of both src and dest match.
Delete file on source. ls of both src and dest match.
Current Behavior
Put a file on source. ls of both src and dest match.
Delete file on source. ls of src is missing the file and dest still has the file shown.
Possible Solution
There has been a lot of regression fixes to replication recently. The documentation updates helped me, but perhaps I misunderstand how it is supposed to work.
Steps to Reproduce
Uses admin user on both sides. Not using locks, just versioning on buckets. Two different minio instances running locally - different directories, different ports.
| alias | endpoint | user | secret |
|---|---|---|---|
| one | http://127.0.0.1:9000 | minioadmin | secretpassword |
| two | http://127.0.0.1:9002 | minioadmin | secretpassword2 |
# make 2 buckets and enable versioning on both
./mc mb one/srcbucket
./mc mb two/dstbucket
./mc version enable one/srcbucket
./mc version enable two/dstbucket
# define remote target for replication from one/srcbucket -> two/dstbucket
./mc admin bucket remote add one/srcbucket http://minioadmin:secretpassword2@localhost:9002/dstbucket --service replication
#Remote ARN = `arn:minio:replication::d25463f4a0e2ad55d2d93b13c7e82bfe9b7f6d7efd4bca5de5b30b8510cf949e:dstbucket`.
./mc replicate add one/srcbucket --priority 1 --remote-bucket dstbucket --arn arn:minio:replication::d25463f4a0e2ad55d2d93b13c7e82bfe9b7f6d7efd4bca5de5b30b8510cf949e:dstbucket --replicate "delete,delete-marker"
# Now to test
# Test 1 copy file in
./mc cp cheese.png one/srcbucket
# file is present in both buckets
./mc ls -r --versions one/srcbucket
./mc ls -r --versions two/dstbucket
# Remove from srcbucket
./mc rm one/srcbucket/cheese.png
# watch srcbucket shows:
# s3:ObjectRemoved:DeleteMarkerCreated http://127.0.0.1:9000/srcbucket/cheese.png
# s3:Replication:OperationCompletedReplication http://127.0.0.1:9000/srcbucket/cheese.png
# watch dstbucket shows nothing
./mc ls one/srcbucket
./mc ls two/dstbucket
# file is gone from srcbucket and present in dstbucket still
./mc ls -r --versions one/srcbucket
./mc ls -r --versions two/dstbucket
# srcbucket shows delete marker as v2, original file as v1, dstbucket still shows v1 only
# Expectation was delete marker to replicate to dstbucket, and ls on dstbucket to not show file, like srcbucket.
# This way a normal (non forced/versiond) delete gets the same result on the dstbucket.
# Test 2 copy file in
./mc cp cheese.png one/srcbucket/cheese2.png
# file is present in both buckets
./mc ls -r --versions one/srcbucket
./mc ls -r --versions two/dstbucket
# Remove from srcbucket and all versions (needs force)
./mc rm --versions --force one/srcbucket/cheese2.png
# watch srcbucket shows:
# s3:ObjectAccessed:Head http://127.0.0.1:9000/srcbucket/cheese2.png
# s3:ObjectRemoved:Delete http://127.0.0.1:9000/srcbucket/cheese2.png
# s3:Replication:OperationCompletedReplication http://127.0.0.1:9000/srcbucket/cheese2.png
# watch dstbucket shows
# s3:ObjectRemoved:Delete http://127.0.0.1:9002/dstbucket/cheese2.png
./mc ls one/srcbucket
./mc ls two/dstbucket
# file is not in either bucket
./mc ls -r --versions one/srcbucket
./mc ls -r --versions two/dstbucket
# file is not present, no delete markers either
# Expectation: this is what is expected. Both buckets match.Additional information
# Additional information
./mc replicate ls one/srcbucket -json
{
"op": "",
"status": "success",
"url": "",
"rule": {
"ID": "c0dp1c45sjf78dtmcsb0",
"Status": "Enabled",
"Priority": 1,
"DeleteMarkerReplication": {
"Status": "Enabled"
},
"DeleteReplication": {
"Status": "Enabled"
},
"Destination": {
"Bucket": "arn:aws:s3:::dstbucket"
},
"Filter": {
"And": {},
"Tag": {}
},
"SourceSelectionCriteria": {
"ReplicaModifications": {
"Status": ""
}
}
}
}
Context
Setting up simple fan out bucket replication. For basic testing just using admin user only (no users or policies involved).
Regression
Unknown
Your Environment
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)
$ ./minio -v
minio version RELEASE.2021-02-01T22-56-52Z
$ ./mc -v
mc version RELEASE.2021-01-30T00-50-42Z
$ uname -a
Linux xxxx 3.10.0-1160.11.1.el7.x86_64 #1 SMP Mon Nov 30 13:05:31 EST 2020 x86_64 x86_64 x86_64 GNU/Linux