Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@donatello
Copy link
Member

Description

This change fixes handling of these types of queries:

  • Double quoted column names with special characters:
    SELECT "column.name" FROM s3object
  • Double quoted column names with reserved keywords:
    SELECT "CAST" FROM s3object
  • Table name as prefix for column names:
    SELECT S3Object."CAST" FROM s3object

Motivation and Context

Bugs found by a customer.

How to test this PR?

Includes unit tests, and example queries for the kinds of SQL this fixes.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Optimization (provides speedup with no functional changes)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Documentation updated
  • Unit tests added/updated

This change fixes handling of these types of queries:

- Double quoted column names with special characters:
    SELECT "column.name" FROM s3object
- Double quoted column names with reserved keywords:
    SELECT "CAST" FROM s3object
- Table name as prefix for column names:
    SELECT S3Object."CAST" FROM s3object
@minio minio deleted a comment from minio-trusted Apr 3, 2021
@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-large-bucket.sh ✔️
mint-fs.sh ✔️
mint-gateway-s3.sh ✔️
mint-erasure.sh ✔️
mint-dist-erasure.sh ✔️
mint-zoned.sh ✔️
mint-gateway-nas.sh ✔️
mint-compress-encrypt-dist-erasure.sh more...

11957-1318c9e/mint-compress-encrypt-dist-erasure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev8.minio.io:32101
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 8658bf2e5c88:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 1 seconds
(2/15) Running aws-sdk-java tests ... done in 2 seconds
(3/15) Running aws-sdk-php tests ... done in 43 seconds
(4/15) Running aws-sdk-ruby tests ... done in 3 seconds
(5/15) Running awscli tests ... FAILED in 32 seconds
{
  "name": "awscli",
  "duration": 2796,
  "function": "aws --endpoint-url http://minio-dev8.minio.io:32101 s3api copy-object --bucket awscli-mint-test-bucket-4420 --key datafile-1-kB-copy --copy-source awscli-mint-test-bucket-4420/datafile-1-kB\n",
  "status": "FAIL",
  "error": "Hash mismatch expected 084e1383b70fb0c51acc680fef370023, got ac57de7156d7fc25ac1a65f81fa3989b"
}
(5/15) Running healthcheck tests ... done in 0 seconds
(6/15) Running mc tests ... done in 45 seconds
(7/15) Running minio-dotnet tests ... done in 41 seconds
(8/15) Running minio-go tests ... FAILED in 1 minutes and 59 seconds
{
  "args": {},
  "duration": 833,
  "error": "At least one of the pre-conditions you specified did not hold",
  "function": "CopyObjectPart(destination, source)",
  "message": "CopyObjectPart call failed",
  "name": "minio-go: testUnencryptedToSSES3CopyObjectPart",
  "status": "FAIL"
}
(8/15) Running minio-java tests ... FAILED in 1 minutes and 30 seconds
{
  "name": "minio-java",
  "function": "copyObject()",
  "args": "[match etag]",
  "duration": 186,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = PreconditionFailed, message = At least one of the pre-conditions you specified did not hold, bucketName = minio-java-test-uaahvf, objectName = minio-java-test-18ns8ht-copy, resource = /minio-java-test-uaahvf/minio-java-test-18ns8ht-copy, requestId = 1672CEC50768E07D, hostId = d0c8ad0f-970b-4fce-b799-cae5e7d68431)\nrequest={method=PUT, url=http://minio-dev8.minio.io:32101/minio-java-test-uaahvf/minio-java-test-18ns8ht-copy, headers=x-amz-copy-source-if-match: 71cff0a060f852067e443ad1e24ae26c-1\nx-amz-copy-source: /minio-java-test-3psts77/minio-java-test-18ns8ht\nHost: minio-dev8.minio.io:32101\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20210405T002921Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20210405/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-copy-source;x-amz-copy-source-if-match;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=412, headers=Accept-Ranges: bytes\nContent-Length: 416\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nETag: \"71cff0a060f852067e443ad1e24ae26c\"\nLast-Modified: Mon, 05 Apr 2021 00:29:21 GMT\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 1672CEC50768E07D\nX-Xss-Protection: 1; mode=block\nDate: Mon, 05 Apr 2021 00:29:21 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.execute(MinioClient.java:563), io.minio.MinioClient.executePut(MinioClient.java:904), io.minio.MinioClient.copyObject(MinioClient.java:1232), FunctionalTest.testCopyObjectMatchETag(FunctionalTest.java:1850), FunctionalTest.copyObject(FunctionalTest.java:2016), FunctionalTest.runObjectTests(FunctionalTest.java:3757), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(8/15) Running minio-js tests ... done in 48 seconds
(9/15) Running minio-py tests ... done in 3 minutes and 6 seconds
(10/15) Running s3cmd tests ... FAILED in 5 seconds
{
  "name": "s3cmd",
  "duration": "3043",
  "function": "test_put_object_multipart",
  "status": "FAIL",
  "error": "WARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Too many failures. Giving up on '/mint/data/datafile-65-MB'\nERROR: \nUpload of '/mint/data/datafile-65-MB' part 1 failed. Use\n  /usr/local/bin/s3cmd abortmp s3://s3cmd-test-bucket-32005/s3cmd-test-object-869 0c429d12-15ea-4bde-b463-e72d79a14e33\nto abort the upload, or\n  /usr/local/bin/s3cmd --upload-id 0c429d12-15ea-4bde-b463-e72d79a14e33 put ...\nto continue the upload.\nERROR: Upload of '/mint/data/datafile-65-MB' failed too many times (Last reason: )"
}
(10/15) Running s3select tests ... done in 6 seconds
(11/15) Running security tests ... done in 0 seconds

Executed 11 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

@harshavardhana harshavardhana requested a review from vadmeste April 5, 2021 15:52
Copy link
Contributor

@klauspost klauspost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@harshavardhana harshavardhana merged commit b293624 into minio:master Apr 6, 2021
@donatello donatello deleted the select-parsing branch April 6, 2021 18:37
harshavardhana pushed a commit that referenced this pull request Apr 6, 2021
This change fixes handling of these types of queries:

- Double quoted column names with special characters:
    SELECT "column.name" FROM s3object
- Double quoted column names with reserved keywords:
    SELECT "CAST" FROM s3object
- Table name as prefix for column names:
    SELECT S3Object."CAST" FROM s3object
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants