Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a core issue in how prepared statements handle virtual table queries by overhauling the metadata fetching process. It introduces a synchronous metadata retrieval step during statement binding, ensuring that the parser has complete information about virtual tables before execution. This change enhances the robustness and correctness of query processing for complex virtual table structures. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Pull request overview
This PR updates stmt/vtable query parsing to support virtual-table metadata resolution by fetching catalog metadata and feeding it into the parser/translator, and adjusts related call sites and tests.
Changes:
- Extend
qStmtParseQuerySqlto accept optionalSMetaData*and populate a parse meta-cache from it. - Add a synchronous
catalogGetAllMetawrapper (built oncatalogAsyncGetAllMeta) and use it from stmt2 query bind/parse. - Update stmt2 tests to cover vtable/
last()query scenarios.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| source/libs/planner/test/planTestUtil.cpp | Update call site for new qStmtParseQuerySql signature. |
| source/libs/planner/src/planOptimizer.c | Null out pointers after list destruction to avoid stale pointers/double-destroys. |
| source/libs/parser/src/parser.c | Implement new qStmtParseQuerySql(..., SMetaData*) flow with meta-cache + cleanup. |
| source/libs/parser/src/parTranslater.c | Remove fallback behavior when VStbRefDbs is missing from cache. |
| source/libs/parser/inc/parUtil.h | Move/adjust meta-cache related declarations and add collectMetaKey prototype. |
| source/libs/catalog/src/catalog.c | Implement synchronous catalogGetAllMeta wrapper using semaphore + async API. |
| source/client/src/clientStmt2.c | Fetch metadata synchronously on stmt2 bind, then parse with metadata. |
| source/client/src/clientStmt.c | Update call site for new qStmtParseQuerySql signature (passes NULL). |
| include/libs/parser/parser.h | Update public signature and expose meta-cache-related types/APIs. |
| source/client/test/stmt2Test.cpp | Expand stmt2 tests for vtable/core last() queries and adjust query test options. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Copy metaData to pRequest->parseMeta for potential future use | ||
| // Similar to doAsyncQueryFromAnalyse when parseOnly is true | ||
| (void)memcpy(&pStmt->exec.pRequest->parseMeta, &metaData, sizeof(SMetaData)); | ||
| (void)memset(&metaData, 0, sizeof(SMetaData)); // Clear to avoid double free | ||
| } else { |
There was a problem hiding this comment.
This copies metaData into pRequest->parseMeta unconditionally, but request teardown (doDestroyRequest) does not free parseMeta via catalogFreeMetaData. This introduces a persistent leak for every stmt2 bind/parse. Either only store parseMeta when parseOnly is enabled (matching doAsyncQueryFromAnalyse), or ensure parseMeta is freed during request destruction/reset before overwriting it.
| int32_t catalogGetAllMeta(SCatalog* pCtg, SRequestConnInfo* pConn, const SCatalogReq* pReq, SMetaData* pRsp) { | ||
| CTG_API_ENTER(); | ||
|
|
||
| if (NULL == pCtg || NULL == pConn || NULL == pReq || NULL == pRsp) { | ||
| CTG_API_LEAVE(TSDB_CODE_CTG_INVALID_INPUT); | ||
| } | ||
|
|
||
| int32_t code = 0; | ||
| pRsp->pTableMeta = NULL; | ||
|
|
||
| if (pReq->pTableMeta) { | ||
| int32_t tbNum = (int32_t)taosArrayGetSize(pReq->pTableMeta); | ||
| if (tbNum <= 0) { | ||
| ctgError("empty table name list, tbNum:%d", tbNum); | ||
| CTG_ERR_JRET(TSDB_CODE_CTG_INVALID_INPUT); | ||
| } | ||
|
|
||
| pRsp->pTableMeta = taosArrayInit(tbNum, POINTER_BYTES); | ||
| if (NULL == pRsp->pTableMeta) { | ||
| ctgError("taosArrayInit %d failed", tbNum); | ||
| CTG_ERR_JRET(terrno); | ||
| } | ||
|
|
||
| for (int32_t i = 0; i < tbNum; ++i) { | ||
| SName* name = taosArrayGet(pReq->pTableMeta, i); | ||
| STableMeta* pTableMeta = NULL; | ||
| SCtgTbMetaCtx ctx = {0}; | ||
| ctx.pName = name; | ||
| ctx.flag = CTG_FLAG_UNKNOWN_STB; | ||
|
|
||
| CTG_ERR_JRET(ctgGetTbMeta(pCtg, pConn, &ctx, &pTableMeta)); | ||
|
|
||
| if (NULL == taosArrayPush(pRsp->pTableMeta, &pTableMeta)) { | ||
| ctgError("taosArrayPush failed, idx:%d", i); | ||
| taosMemoryFreeClear(pTableMeta); | ||
| CTG_ERR_JRET(terrno); | ||
| } | ||
| } | ||
| SCatalogSyncCbParam cbParam = {.pRsp = pRsp, .code = TSDB_CODE_SUCCESS}; | ||
| if (tsem_init(&cbParam.sem, 0, 0) != 0) { | ||
| CTG_API_LEAVE(TSDB_CODE_CTG_INTERNAL_ERROR); | ||
| } | ||
|
|
||
| if (pReq->qNodeRequired) { | ||
| pRsp->pQnodeList = taosArrayInit(10, sizeof(SQueryNodeLoad)); | ||
| CTG_ERR_JRET(ctgGetQnodeListFromMnode(pCtg, pConn, pRsp->pQnodeList, NULL)); | ||
| } | ||
|
|
||
| CTG_API_LEAVE(TSDB_CODE_SUCCESS); | ||
|
|
||
| _return: | ||
|
|
||
| if (pRsp->pTableMeta) { | ||
| int32_t aSize = taosArrayGetSize(pRsp->pTableMeta); | ||
| for (int32_t i = 0; i < aSize; ++i) { | ||
| STableMeta* pMeta = taosArrayGetP(pRsp->pTableMeta, i); | ||
| taosMemoryFreeClear(pMeta); | ||
| } | ||
|
|
||
| taosArrayDestroy(pRsp->pTableMeta); | ||
| pRsp->pTableMeta = NULL; | ||
| int32_t code = catalogAsyncGetAllMeta(pCtg, pConn, pReq, catalogSyncGetAllMetaCb, &cbParam, NULL); | ||
| if (TSDB_CODE_SUCCESS == code) { | ||
| tsem_wait(&cbParam.sem); | ||
| code = cbParam.code; | ||
| } |
There was a problem hiding this comment.
catalogGetAllMeta no longer initializes/clears *pRsp. If catalogAsyncGetAllMeta returns an error (or tsem_init fails), the function returns without touching pRsp, leaving any existing pointers/values in caller memory intact and risking later misuse/free. Consider zeroing *pRsp at entry (or documenting/guaranteeing that callers must pass a zeroed SMetaData).
| ASSERT_NE(row2, nullptr); | ||
| ASSERT_EQ(*(int64_t*)row2[0], 1591060630000); | ||
| row2 = taos_fetch_row(res2); | ||
| ASSERT_EQ(row2, nullptr); |
There was a problem hiding this comment.
This loop allocates/initializes a TAOS_STMT2 but never closes it. In the rest of this test file, statements are consistently released via taos_stmt2_close; skipping it here will leak resources and can destabilize later tests. Ensure each stmt created in the loop is closed (and that any associated result state is released) before the next iteration/end of test.
| ASSERT_EQ(row2, nullptr); | |
| ASSERT_EQ(row2, nullptr); | |
| taos_free_result(res2); | |
| taos_stmt2_close(stmt); |
| AsyncArgs args = {0, 0}; | ||
| ASSERT_EQ(tsem_init(&args.sem, 0, 0), TSDB_CODE_SUCCESS); | ||
| TAOS_STMT2_OPTION option[2] = {{0, true, true, stmtAsyncQueryCb, &args}, {0, true, true, NULL, NULL}}; | ||
|
|
||
| int32_t dataname_len = 3; | ||
| char* dataname = "abc"; | ||
| char* dataname2 = "def"; | ||
|
|
||
| for (int i = 1; i < 2; i++) { | ||
| TAOS_STMT2* stmt = taos_stmt2_init(taos, &option[i]); | ||
| ASSERT_NE(stmt, nullptr); | ||
|
|
||
| int code = | ||
| taos_stmt2_prepare(stmt, "select last(bool_v) from ts_kv_data tbv1 where dataname in (?) partition by tbname", 0); | ||
| checkError(stmt, code, __FILE__, __LINE__); | ||
| int32_t dataname_len = 3; | ||
| char* dataname = "abc"; | ||
| TAOS_STMT2_BIND params[1] = { | ||
| {TSDB_DATA_TYPE_BINARY, dataname, &dataname_len, NULL, 1}, | ||
| }; | ||
| TAOS_STMT2_BIND* paramv = ¶ms[0]; | ||
| TAOS_STMT2_BINDV bindv = {1, NULL, NULL, ¶mv}; | ||
| code = taos_stmt2_bind_param(stmt, &bindv, -1); | ||
| checkError(stmt, code, __FILE__, __LINE__); | ||
| code = taos_stmt2_exec(stmt, NULL); | ||
| checkError(stmt, code, __FILE__, __LINE__); | ||
| int code = taos_stmt2_prepare( | ||
| stmt, | ||
| "select last(ts,bool_v,float_v) from stmt2_testdb_32.ts_kv_data where dataname in (?,?) partition by tbname", | ||
| 0); | ||
| checkError(stmt, code, __FILE__, __LINE__); | ||
| TAOS_STMT2_BIND params[2] = { | ||
| {TSDB_DATA_TYPE_BINARY, dataname, &dataname_len, NULL, 1}, | ||
| {TSDB_DATA_TYPE_BINARY, dataname2, &dataname_len, NULL, 1}, | ||
| }; | ||
| TAOS_STMT2_BIND* paramv = ¶ms[0]; | ||
| TAOS_STMT2_BINDV bindv = {1, NULL, NULL, ¶mv}; | ||
| code = taos_stmt2_bind_param(stmt, &bindv, -1); | ||
| checkError(stmt, code, __FILE__, __LINE__); | ||
| code = taos_stmt2_exec(stmt, NULL); | ||
| checkError(stmt, code, __FILE__, __LINE__); | ||
|
|
||
| TAOS_RES* res = taos_stmt2_result(stmt); | ||
| ASSERT_NE(res, nullptr); | ||
| TAOS_ROW row = taos_fetch_row(res); | ||
| ASSERT_NE(row, nullptr); | ||
| ASSERT_EQ(*(bool*)row[0], false); | ||
| if (i == 0) { | ||
| tsem_wait(&args.sem); | ||
| } | ||
|
|
There was a problem hiding this comment.
The async test path is currently unreachable: the loop starts at i=1, but the wait is guarded by if (i == 0). As written, the semaphore/AsyncArgs setup is unused and the async option isn’t exercised. Either iterate over both options (0 and 1) and wait in the async case, or remove the async scaffolding if only the sync path is intended.
source/client/test/stmt2Test.cpp
Outdated
| for (int i = 1; i < 2; i++) { | ||
| TAOS_STMT2* stmt = taos_stmt2_init(taos, &option[i]); | ||
| ASSERT_NE(stmt, nullptr); | ||
|
|
||
| int code = taos_stmt2_prepare( | ||
| stmt, | ||
| "select last(ts,bool_v,float_v) from stmt2_testdb_32.ts_kv_data where dataname in (?,?) partition by tbname", | ||
| 0); | ||
| checkError(stmt, code, __FILE__, __LINE__); | ||
| TAOS_STMT2_BIND params[2] = { | ||
| {TSDB_DATA_TYPE_BINARY, dataname, &dataname_len, NULL, 1}, | ||
| {TSDB_DATA_TYPE_BINARY, dataname2, &dataname_len, NULL, 1}, | ||
| }; | ||
| TAOS_STMT2_BIND* paramv = ¶ms[0]; | ||
| TAOS_STMT2_BINDV bindv = {1, NULL, NULL, ¶mv}; | ||
| code = taos_stmt2_bind_param(stmt, &bindv, -1); | ||
| checkError(stmt, code, __FILE__, __LINE__); | ||
| code = taos_stmt2_exec(stmt, NULL); | ||
| checkError(stmt, code, __FILE__, __LINE__); | ||
|
|
||
| if (i == 0) { | ||
| tsem_wait(&args.sem); | ||
| } | ||
|
|
There was a problem hiding this comment.
Same issue as above: the loop starts at i=1 so the async option (i==0) is never executed, and the semaphore wait block is dead code. This reduces test coverage of the async stmt2 execution path and leaves unnecessary synchronization setup in the test.
| SArray* pVStbRefs = NULL; | ||
| code = getVStbRefDbsFromCache(pCxt->pMetaCache, pName, &pVStbRefs); | ||
|
|
||
| // Handle the case where VStbRefDbs data is not available (e.g., stmt scenario) | ||
| if (TSDB_CODE_PAR_TABLE_NOT_EXIST == code || TSDB_CODE_PAR_INTERNAL_ERROR == code) { | ||
| // VStbRefDbs not available in cache (stmt scenario without async metadata fetch) | ||
| // Use empty vgroup list - the executor will resolve vgroups at runtime | ||
| taosMemoryFreeClear(pRefScanTable->pVgroupList); | ||
| PAR_ERR_JRET(toVgroupsInfo(vgroupList, &pRefScanTable->pVgroupList)); | ||
| code = TSDB_CODE_SUCCESS; | ||
| goto _return; | ||
| } | ||
| PAR_ERR_JRET(code); | ||
|
|
There was a problem hiding this comment.
Removing the fallback handling here means translate will now hard-fail when VStbRefDbs is missing from the meta cache (getVStbRefDbsFromCache returns TSDB_CODE_PAR_TABLE_NOT_EXIST when pMetaCache/pVStbRefDbs is NULL). This breaks sync parse flows that do not populate meta cache (e.g., qParseSql/parseSqlIntoAst passes NULL meta cache) and stmt1 query parsing (qStmtParseQuerySql called with NULL metadata). Either restore the previous fallback to allow an empty vgroup list, or add a synchronous catalog fetch path for VStbRefDbs when it’s not present in cache.
source/client/src/clientStmt2.c
Outdated
| // Clean up metaData on failure - free all arrays | ||
| if (pMetaData->pVStbRefDbs) { | ||
| taosArrayDestroy(pMetaData->pVStbRefDbs); | ||
| pMetaData->pVStbRefDbs = NULL; | ||
| } | ||
| // Note: Other fields in metaData are managed by catalog module if ctgFree is true | ||
| // For now, we only need to clean up pVStbRefDbs which we explicitly need |
There was a problem hiding this comment.
On failure, this cleanup only destroys pMetaData->pVStbRefDbs, but catalogGetAllMeta can populate many other fields in SMetaData (pDbVgroup, pTableMeta, pUser, etc.). This will leak memory (and possibly leave partially-initialized pointers) on error. Prefer calling catalogFreeMetaData(pMetaData) (or the appropriate catalog cleanup API) for complete cleanup.
| // Clean up metaData on failure - free all arrays | |
| if (pMetaData->pVStbRefDbs) { | |
| taosArrayDestroy(pMetaData->pVStbRefDbs); | |
| pMetaData->pVStbRefDbs = NULL; | |
| } | |
| // Note: Other fields in metaData are managed by catalog module if ctgFree is true | |
| // For now, we only need to clean up pVStbRefDbs which we explicitly need | |
| // Clean up metaData on failure using catalog API to avoid leaks/partial init | |
| catalogFreeMetaData(pMetaData); |
There was a problem hiding this comment.
Code Review
This pull request introduces a mechanism to fetch metadata for prepared statements, which is crucial for handling virtual tables where metadata depends on query parameters. The overall approach is sound, involving passing metadata to qStmtParseQuerySql and refactoring catalogGetAllMeta for synchronous metadata retrieval. The code is well-structured, but I've identified a few areas for improvement, primarily in the test suite where parameter binding is incorrect for queries with multiple parameters. I've also noted a redundant function declaration and an opportunity to refactor duplicated code.
| TAOS_STMT2_BIND* paramv = ¶ms[0]; | ||
| TAOS_STMT2_BINDV bindv = {1, NULL, NULL, ¶mv}; | ||
| code = taos_stmt2_bind_param(stmt, &bindv, -1); |
There was a problem hiding this comment.
The parameter binding for the prepared statement is incorrect. The query ... where dataname in (?,?) ... has two parameter markers, but the TAOS_STMT2_BINDV is configured to bind only one parameter (num_params is 1). This means the test doesn't correctly cover the scenario with multiple parameters in an IN clause.
TAOS_STMT2_BIND* param_ptrs[] = {¶ms[0], ¶ms[1]};
TAOS_STMT2_BINDV bindv = {2, NULL, NULL, param_ptrs};
code = taos_stmt2_bind_param(stmt, &bindv, -1);
source/client/test/stmt2Test.cpp
Outdated
| TAOS_STMT2_BIND* paramv = ¶ms[0]; | ||
| TAOS_STMT2_BINDV bindv = {1, NULL, NULL, ¶mv}; | ||
| code = taos_stmt2_bind_param(stmt, &bindv, -1); |
There was a problem hiding this comment.
Similar to the previous test case, the parameter binding here is incorrect. The query uses two parameter markers (?,?), but the binding is set up for only a single parameter. This should be corrected to bind both parameters.
TAOS_STMT2_BIND* param_ptrs[] = {¶ms[0], ¶ms[1]};
TAOS_STMT2_BINDV bindv = {2, NULL, NULL, param_ptrs};
code = taos_stmt2_bind_param(stmt, &bindv, -1);
source/client/test/stmt2Test.cpp
Outdated
| TAOS_STMT2_BIND* paramv = ¶ms[0]; | ||
| TAOS_STMT2_BINDV bindv = {1, NULL, NULL, ¶mv}; | ||
| code = taos_stmt2_bind_param(stmt, &bindv, -1); |
There was a problem hiding this comment.
The parameter binding is incorrect for this test case as well. The query has two parameter markers, but the binding is configured for only one. Please adjust the TAOS_STMT2_BINDV to bind both parameters.
TAOS_STMT2_BIND* param_ptrs[] = {¶ms[0], ¶ms[1]};
TAOS_STMT2_BINDV bindv = {2, NULL, NULL, param_ptrs};
code = taos_stmt2_bind_param(stmt, &bindv, -1);| if (TSDB_CODE_SUCCESS != code) { | ||
| // Clean up metaData on failure - free all arrays | ||
| if (pMetaData->pVStbRefDbs) { | ||
| taosArrayDestroy(pMetaData->pVStbRefDbs); | ||
| pMetaData->pVStbRefDbs = NULL; | ||
| } | ||
| // Note: Other fields in metaData are managed by catalog module if ctgFree is true | ||
| // For now, we only need to clean up pVStbRefDbs which we explicitly need | ||
| } |
source/libs/parser/inc/parUtil.h
Outdated
| int32_t getVStbRefDbsFromCache(SParseMetaCache* pMetaCache, const SName* pName, SArray** pOutput); | ||
| int32_t getDnodeListFromCache(SParseMetaCache* pMetaCache, SArray** pDnodes); | ||
| void destoryParseMetaCache(SParseMetaCache* pMetaCache, bool request); | ||
| int32_t collectMetaKey(SParseContext* pParseCxt, SQuery* pQuery, SParseMetaCache* pMetaCache); |
Description
Issue(s)
Checklist
Please check the items in the checklist if applicable.