Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@harshavardhana
Copy link
Member

Description

xl: remove cleanupDir instead use Delete()

Motivation and Context

use a single call to remove directly at disk
instead of doing recursively at the network layer.

How to test this PR?

Generate lots of failed requests such as PutObject()
with a slowReader{} and close the connection prematurely
before server times out, this starts the cleanup routine that
would be slow, then don't wait subsequently send more
requests and let them timeout as well.

The code change simply ensures that we use efficient calls
to delete the temporary folder upon client timeout.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Optimization (provides speedup with no functional changes)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Documentation updated
  • Unit tests added/updated

use a single call to remove directly at disk
instead of doing recursively at network layer.
@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-large-bucket.sh ✔️
mint-fs.sh ✔️
mint-gateway-s3.sh ✔️
mint-erasure.sh ✔️
mint-dist-erasure.sh ✔️
mint-zoned.sh ✔️
mint-gateway-nas.sh ✔️
mint-compress-encrypt-dist-erasure.sh more...

11880-873a7d0/mint-compress-encrypt-dist-erasure.sh.log:

Running with
SERVER_ENDPOINT:      minio-c3.minio.io:30555
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 4dfdd7dc2ffa:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 3 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 44 seconds
(4/15) Running aws-sdk-ruby tests ... done in 5 seconds
(5/15) Running awscli tests ... FAILED in 33 seconds
{
  "name": "awscli",
  "duration": 2685,
  "function": "aws --endpoint-url http://minio-c3.minio.io:30555 s3api copy-object --bucket awscli-mint-test-bucket-9516 --key datafile-1-kB-copy --copy-source awscli-mint-test-bucket-9516/datafile-1-kB\n",
  "status": "FAIL",
  "error": "Hash mismatch expected 084e1383b70fb0c51acc680fef370023, got ac57de7156d7fc25ac1a65f81fa3989b"
}
(5/15) Running healthcheck tests ... done in 0 seconds
(6/15) Running mc tests ... done in 46 seconds
(7/15) Running minio-dotnet tests ... done in 51 seconds
(8/15) Running minio-go tests ... FAILED in 2 minutes and 26 seconds
{
  "args": {},
  "duration": 549,
  "error": "At least one of the pre-conditions you specified did not hold",
  "function": "CopyObjectPart(destination, source)",
  "message": "CopyObjectPart call failed",
  "name": "minio-go: testUnencryptedToSSES3CopyObjectPart",
  "status": "FAIL"
}
(8/15) Running minio-java tests ... FAILED in 1 minutes and 52 seconds
{
  "name": "minio-java",
  "function": "copyObject()",
  "args": "[match etag]",
  "duration": 301,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = PreconditionFailed, message = At least one of the pre-conditions you specified did not hold, bucketName = minio-java-test-2inpqvu, objectName = minio-java-test-2palj4l-copy, resource = /minio-java-test-2inpqvu/minio-java-test-2palj4l-copy, requestId = 166F42CEA37FF9D1, hostId = 40c5b5c7-316e-4db9-b0ec-047ac135e822)\nrequest={method=PUT, url=http://minio-c3.minio.io:30555/minio-java-test-2inpqvu/minio-java-test-2palj4l-copy, headers=x-amz-copy-source-if-match: 71cff0a060f852067e443ad1e24ae26c-1\nx-amz-copy-source: /minio-java-test-cdhig7/minio-java-test-2palj4l\nHost: minio-c3.minio.io:30555\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20210324T111046Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20210324/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-copy-source;x-amz-copy-source-if-match;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=412, headers=Accept-Ranges: bytes\nContent-Length: 418\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nETag: \"71cff0a060f852067e443ad1e24ae26c\"\nLast-Modified: Wed, 24 Mar 2021 11:10:46 GMT\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 166F42CEA37FF9D1\nX-Xss-Protection: 1; mode=block\nDate: Wed, 24 Mar 2021 11:10:46 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.execute(MinioClient.java:563), io.minio.MinioClient.executePut(MinioClient.java:904), io.minio.MinioClient.copyObject(MinioClient.java:1232), FunctionalTest.testCopyObjectMatchETag(FunctionalTest.java:1850), FunctionalTest.copyObject(FunctionalTest.java:2016), FunctionalTest.runObjectTests(FunctionalTest.java:3757), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(8/15) Running minio-js tests ... done in 53 seconds
(9/15) Running minio-py tests ... done in 3 minutes and 51 seconds
(10/15) Running s3cmd tests ... FAILED in 5 seconds
{
  "name": "s3cmd",
  "duration": "2877",
  "function": "test_put_object_multipart",
  "status": "FAIL",
  "error": "WARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Too many failures. Giving up on '/mint/data/datafile-65-MB'\nERROR: \nUpload of '/mint/data/datafile-65-MB' part 1 failed. Use\n  /usr/local/bin/s3cmd abortmp s3://s3cmd-test-bucket-12301/s3cmd-test-object-20175 5e4aa64b-a07a-4bcd-8d68-9956afdc5751\nto abort the upload, or\n  /usr/local/bin/s3cmd --upload-id 5e4aa64b-a07a-4bcd-8d68-9956afdc5751 put ...\nto continue the upload.\nERROR: Upload of '/mint/data/datafile-65-MB' failed too many times (Last reason: )"
}
(10/15) Running s3select tests ... done in 9 seconds
(11/15) Running security tests ... done in 0 seconds

Executed 11 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

Copy link
Contributor

@klauspost klauspost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! LGTM.

Copy link
Member

@vadmeste vadmeste left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but with one nit comment

@harshavardhana harshavardhana merged commit 75741db into minio:master Mar 24, 2021
@harshavardhana harshavardhana deleted the fix-delete branch March 24, 2021 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants