fix: use getObjectNInfo to avoid bytes.Buffer usage #11428

harshavardhana · 2021-02-03T23:16:14Z

Description

fix: use getObjectNInfo to avoid bytes.Buffer usage

Motivation and Context

few places were still using legacy call GetObject()
which was mainly designed for client response writer,
use GetObjectNInfo() for internal calls instead.

How to test this PR?

Nothing should change in terms of functionality
reduces the occasional memory usage

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Optimization (provides speedup with no functional changes)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Fixes a regression (If yes, please add commit-id or PR # here)
Documentation updated
Unit tests added/updated

few places were still using legacy call GetObject() which was mainly designed for client response writer, use GetObjectNInfo() for internal calls instead.

minio-trusted · 2021-02-04T10:00:13Z

Mint Automation

Test	Result
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-zoned.sh	✔️
mint-gateway-nas.sh	✔️
mint-large-bucket.sh	❌ more...
mint-gateway-azure.sh	❌ more...
mint-compress-encrypt-dist-erasure.sh	❗ more...

11428-23b4c4f/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev6.minio.io:32381
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp da00587665d2:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 8 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 3 minutes and 46 seconds
(4/15) Running aws-sdk-ruby tests ... done in 1 minutes and 4 seconds
(5/15) Running awscli tests ... done in 3 minutes and 2 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 28 minutes and 12 seconds
(8/15) Running minio-dotnet tests ... done in 5 minutes and 21 seconds
(9/15) Running minio-go tests ... FAILED in 3 minutes and 11 seconds
{
  "args": {
    "bucketName": "minio-go-test-qlxld6f9a4m6co9a",
    "objectName": "test-object",
    "opts": "",
    "size": -1
  },
  "duration": 1165,
  "function": "PutObject(bucketName, objectName, reader,size,opts)",
  "message": "Unexpected size",
  "name": "minio-go: testPutObjectStreaming",
  "status": "FAIL"
}
(9/15) Running minio-java tests ... FAILED in 8 minutes and 40 seconds
{
  "name": "minio-java",
  "function": "putObject()",
  "args": "[user metadata]",
  "duration": 173,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = AuthenticationFailed, message = -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/[email protected]/azblob/zc_storage_error.go:42\n===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:03c3cb87-001e-001e-01d7-fa3703000000\nTime:2021-02-04T09:27:28.1447173Z, Details: \n   AuthenticationErrorDetail: The MAC signature found in the HTTP request 'f+C+Ztv34h9o6i8Ul3IjjZgvjujj0wzRtmVqgV8+GFs=' is not the same as any computed signature. Server used following string to sign: 'PUT\n\n\n128\n\napplication/xml\n\n\n\n\n\n\nx-ms-blob-cache-control:\nx-ms-blob-content-disposition:\nx-ms-blob-content-encoding:\nx-ms-blob-content-language:\nx-ms-blob-content-type:application/octet-stream\nx-ms-client-request-id:399dd66a-4aa3-4de8-4b49-ab94c9ce6162\nx-ms-date:Thu, 04 Feb 2021 09:27:28 GMT\nx-ms-meta-my_header1:a   b   c\nx-ms-meta-my_header2:\"a   b   c\"\nx-ms-meta-my_project:Project One\nx-ms-meta-my_unicode_tag:å•†å“�\nx-ms-version:2019-02-02\n/minioazure/minio-java-test-2hvbs4p/minio-java-test-1j5kha4\ncomp:blocklist\ntimeout:1501'.\n   Code: AuthenticationFailed\n   PUT https://minioazure.blob.core.windows.net/minio-java-test-2hvbs4p/minio-java-test-1j5kha4?comp=blocklist&timeout=1501\n   Authorization: REDACTED\n   Content-Length: [128]\n   Content-Type: [application/xml]\n   User-Agent: [APN/1.0 MinIO/1.0 MinIO/2021-02-04T08:27:40Z]\n   X-Ms-Blob-Cache-Control: []\n   X-Ms-Blob-Content-Disposition: []\n   X-Ms-Blob-Content-Encoding: []\n   X-Ms-Blob-Content-Language: []\n   X-Ms-Blob-Content-Type: [application/octet-stream]\n   X-Ms-Client-Request-Id: [399dd66a-4aa3-4de8-4b49-ab94c9ce6162]\n   X-Ms-Date: [Thu, 04 Feb 2021 09:27:28 GMT]\n   X-Ms-Meta-My_header1: [a   b   c]\n   X-Ms-Meta-My_header2: [\"a   b   c\"]\n   X-Ms-Meta-My_project: [Project One]\n   X-Ms-Meta-My_unicode_tag: [商品]\n   X-Ms-Version: [2019-02-02]\n   --------------------------------------------------------------------------------\n   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\n   Content-Length: [1092]\n   Content-Type: [application/xml]\n   Date: [Thu, 04 Feb 2021 09:27:27 GMT]\n   Server: [Microsoft-HTTPAPI/2.0]\n   X-Ms-Error-Code: [AuthenticationFailed]\n   X-Ms-Request-Id: [03c3cb87-001e-001e-01d7-fa3703000000]\n\n\n, bucketName = minio-java-test-2hvbs4p, objectName = minio-java-test-1j5kha4, resource = /minio-java-test-2hvbs4p/minio-java-test-1j5kha4, requestId = 1660815031CB8242, hostId = 731381a5-7ebb-47ca-8b4a-31cd1e23d806)\nrequest={method=PUT, url=http://minio-dev6.minio.io:32381/minio-java-test-2hvbs4p/minio-java-test-1j5kha4, headers=x-amz-meta-My-Unicode-Tag: 商品\nx-amz-meta-My-Project: Project One\nx-amz-meta-My-header1: a   b   c\nx-amz-meta-My-Header2: \"a   b   c\"\nContent-Type: application/octet-stream\nHost: minio-dev6.minio.io:32381\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: A9oFTxee7YVcJ9fWsgQeKg==\nx-amz-content-sha256: 1ff7959f86334ddc5c188a5083268f600146328b2b6c5185e75bf7d9387d6b74\nx-amz-date: 20210204T092728Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20210204/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-meta-my-header1;x-amz-meta-my-header2;x-amz-meta-my-project;x-amz-meta-my-unicode-tag, Signature=*REDACTED*\n}\nresponse={code=403, headers=Accept-Ranges: bytes\nContent-Length: 3086\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 1660815031CB8242\nX-Xss-Protection: 1; mode=block\nDate: Thu, 04 Feb 2021 09:27:28 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.putObject(MinioClient.java:4547), io.minio.MinioClient.putObject(MinioClient.java:2713), io.minio.MinioClient.putObject(MinioClient.java:2830), FunctionalTest.testPutObject(FunctionalTest.java:763), FunctionalTest.putObject(FunctionalTest.java:890), FunctionalTest.runObjectTests(FunctionalTest.java:3751), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(9/15) Running minio-js tests ... FAILED in 1 minutes and 2 seconds
{
  "name": "minio-js",
  "function": "\"after all\" hook in \"functional tests\"",
  "duration": 79,
  "status": "FAIL",
  "error": "S3Error: The bucket you tried to delete is not empty at Object.parseError (node_modules/minio/dist/main/xml-parsers.js:79:11) at /mint/run/core/minio-js/node_modules/minio/dist/main/transformers.js:156:22 at DestroyableTransform._flush (node_modules/minio/dist/main/transformers.js:80:10) at DestroyableTransform.prefinish (node_modules/readable-stream/lib/_stream_transform.js:129:10) at prefinish (node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (node_modules/readable-stream/lib/_stream_writable.js:643:3) at DestroyableTransform.Writable.end (node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (internal/streams/readable.js:684:10) at endReadableNT (internal/streams/readable.js:1327:12) at processTicksAndRejections (internal/process/task_queues.js:80:21)"
}
(9/15) Running minio-py tests ... done in 19 minutes and 18 seconds
(10/15) Running s3cmd tests ... done in 10 minutes and 54 seconds
(11/15) Running s3select tests ... done in 42 seconds
(12/15) Running security tests ... done in 0 seconds

Executed 12 out of 15 tests successfully.

11428-23b4c4f/mint-large-bucket.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:30876
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp cf2a9758e04f:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 2 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... done in 2 minutes and 11 seconds
(6/15) Running healthcheck tests ... done in 1 seconds
(7/15) Running mc tests ... done in 1 minutes and 10 seconds
(8/15) Running minio-dotnet tests ... done in 45 seconds
(9/15) Running minio-go tests ... done in 2 minutes and 10 seconds
(10/15) Running minio-java tests ... done in 1 minutes and 17 seconds
(11/15) Running minio-js tests ... FAILED in 16 seconds
{
  "name": "minio-js",
  "function": "fPutObject(bucketName, objectName, filePath, metaData)",
  "args": "bucketName:minio-js-test-4be66340-9f54-4d18-99df-8179cbe1f3af, objectName:datafile-65-MB, filePath:/tmp/datafile-65-MB",
  "duration": 1118,
  "status": "FAIL",
  "error": "Error: read ECONNRESET at TCP.onStreamRead (internal/stream_base_commons.js:209:20)"
}
(11/15) Running minio-py tests ... done in 3 minutes and 5 seconds
(12/15) Running s3cmd tests ... done in 19 seconds
(13/15) Running s3select tests ... done in 6 seconds
(14/15) Running security tests ... done in 0 seconds

Executed 14 out of 15 tests successfully.

11428-23b4c4f/mint-compress-encrypt-dist-erasure.sh.xtrace:

DISABLED TIME BEING

Deleting image on docker hub
Deleting image locally

vadmeste

One comment

cmd/data-crawler.go

vadmeste

LGTM

klauspost

LGTM

fix: use getObjectNInfo to avoid bytes.Buffer usage

8bf9b89

few places were still using legacy call GetObject() which was mainly designed for client response writer, use GetObjectNInfo() for internal calls instead.

harshavardhana requested review from klauspost and vadmeste February 3, 2021 23:16

Merge branch 'master' into fix-getobject

23b4c4f

vadmeste reviewed Feb 4, 2021

View reviewed changes

cmd/data-crawler.go Show resolved Hide resolved

harshavardhana requested a review from vadmeste February 4, 2021 18:18

vadmeste approved these changes Feb 4, 2021

View reviewed changes

klauspost approved these changes Feb 5, 2021

View reviewed changes

harshavardhana merged commit 8bb580a into minio:master Feb 5, 2021

harshavardhana deleted the fix-getobject branch February 5, 2021 17:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: use getObjectNInfo to avoid bytes.Buffer usage #11428

fix: use getObjectNInfo to avoid bytes.Buffer usage #11428

Uh oh!

harshavardhana commented Feb 3, 2021

Uh oh!

minio-trusted commented Feb 4, 2021

Uh oh!

vadmeste left a comment

Uh oh!

Uh oh!

vadmeste left a comment

Uh oh!

klauspost left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fix: use getObjectNInfo to avoid bytes.Buffer usage #11428

fix: use getObjectNInfo to avoid bytes.Buffer usage #11428

Uh oh!

Conversation

harshavardhana commented Feb 3, 2021

Description

Motivation and Context

How to test this PR?

Types of changes

Checklist:

Uh oh!

minio-trusted commented Feb 4, 2021

Mint Automation

11428-23b4c4f/mint-gateway-azure.sh.log:

11428-23b4c4f/mint-large-bucket.sh.log:

11428-23b4c4f/mint-compress-encrypt-dist-erasure.sh.xtrace:

Uh oh!

vadmeste left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vadmeste left a comment

Choose a reason for hiding this comment

Uh oh!

klauspost left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants