Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a CI failure related to decimal datatype tests by implementing several robustness improvements. It reorders client cleanup operations to prevent Use-After-Free vulnerabilities, makes test file path generation process-ID-aware to enable reliable parallel test execution, and enhances error handling in test scripts to provide more informative diagnostics when subprocess commands fail. These changes collectively aim to stabilize the CI pipeline and improve debugging capabilities for decimal-related tests. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request addresses a CI failure related to decimal types. The changes include adjusting the cleanup order in clientMain.c to prevent use-after-free errors, which is a crucial fix. In the Python test files, error handling for subprocess calls has been significantly improved, and temporary file naming is made more robust for parallel execution. However, the continued use of shell=True in subprocess.run calls across multiple test files introduces a security vulnerability. I have provided suggestions to refactor these calls to eliminate the security risk and improve code consistency.
| def query(self, sql: str): | ||
| with open(self.get_file_path(), "a+") as f: | ||
| f.truncate(0) | ||
| # Truncate output file before execution | ||
| tdLog.debug(f"Executing SQL: {sql}") | ||
| out_path = self.get_file_path() | ||
| with open(out_path, "w"): | ||
| pass | ||
| self.queryResult = [] | ||
| try: | ||
| command = f'taos -s "{sql} >> {self.get_file_path()}"' | ||
| result = subprocess.run( | ||
| command, shell=True, check=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE | ||
| command = f'taos -s "{sql} >> {out_path}"' | ||
| subprocess.run( | ||
| command, | ||
| shell=True, | ||
| check=True, | ||
| stderr=subprocess.PIPE, | ||
| stdout=subprocess.PIPE, | ||
| text=True, | ||
| ) | ||
| self.read_result() | ||
| except subprocess.CalledProcessError as e: | ||
| # Surface stderr (e.g., ASAN diagnostics) without causing secondary decode errors | ||
| err_msg = e.stderr or "" | ||
| tdLog.exit(f"Command '{sql}' failed ({type(e).__name__}):\n{err_msg}") | ||
| except Exception as e: | ||
| tdLog.exit(f"Command '{sql}' failed with error: {e.stderr.decode('utf-8')}") | ||
| self.queryResult = [] | ||
| raise | ||
| extra = getattr(e, 'stderr', None) or "" | ||
| tdLog.exit(f"Unexpected error ({type(e).__name__}) while running '{sql}':\n{extra or str(e)}") | ||
| return self.queryResult |
There was a problem hiding this comment.
Using shell=True with a formatted string command is a security risk due to the potential for shell injection. It's much safer to pass command arguments as a list and handle file redirection within Python. This refactoring also simplifies the file handling logic by removing the separate file truncation step.
def query(self, sql: str):
tdLog.debug(f"Executing SQL: {sql}")
out_path = self.get_file_path()
self.queryResult = []
try:
with open(out_path, "w", encoding="utf-8") as f_out:
subprocess.run(
['taos', '-s', sql],
stdout=f_out,
stderr=subprocess.PIPE,
text=True,
check=True,
)
self.read_result()
except subprocess.CalledProcessError as e:
# Surface stderr (e.g., ASAN diagnostics) without causing secondary decode errors
err_msg = e.stderr or ""
tdLog.exit(f"Command '{sql}' failed ({type(e).__name__}):\n{err_msg}")
except Exception as e:
extra = getattr(e, 'stderr', None) or ""
tdLog.exit(f"Unexpected error ({type(e).__name__}) while running '{sql}':\n{extra or str(e)}")
return self.queryResultThere was a problem hiding this comment.
Pull request overview
This PR targets CI failures around decimal datatype tests by reducing temporary output-file collisions and improving subprocess error reporting, and it also adjusts client cleanup ordering to avoid teardown-related UAF hazards.
Changes:
- Add PID to taos-shell temporary result file names to avoid collisions under multi-process execution.
- Improve taos CLI subprocess execution error reporting (surface stderr) and refactor invocation formatting.
- Reorder
nodesDestroyAllocatorSet()intaos_cleanup()with added ordering rationale comments.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| test/cases/01-DataTypes/test_datatype_decimal_last.py | Make shell output file PID-unique; improve subprocess error surfacing and text-mode capture. |
| test/cases/01-DataTypes/test_datatype_decimal64.py | Same PID-unique output file change + improved subprocess error surfacing and text-mode capture. |
| test/cases/01-DataTypes/test_datatype_decimal.py | PID-unique output file, refactor truncation + subprocess call, and add additional debug logging in operator checks. |
| source/client/src/clientMain.c | Move allocator-set destruction later in cleanup sequence and document intended teardown order. |
Comments suppressed due to low confidence (3)
test/cases/01-DataTypes/test_datatype_decimal.py:2265
- These new debug logs are inside a potentially large nested loop. Using f-strings here formats the message even when DEBUG is disabled, adding avoidable overhead; prefer passing the SQL as a logging argument (or guard with a debug-enabled check) so formatting happens only when emitted.
expr.check(res[0], tbname)
tdLog.debug(f"sql: {sql} got results")
else:
tdLog.info(f"sql: {sql} got no output")
tdLog.debug("check decimal binary expr with col results done***************")
test/cases/01-DataTypes/test_datatype_decimal.py:2334
- Same as above: this per-query
tdLog.debug(f"...")in a loop eagerly builds the string even when DEBUG logging is off. Prefer parameterized logging (or a guard) to avoid the formatting and large-string allocations on every iteration.
if len(res) > 0:
expr.check(res[0], tbname)
tdLog.debug(f"sql: {sql} got results")
source/client/src/clientMain.c:283
- The new cleanup-order comment says “Destroy allocator set last”, but
nodesDestroyAllocatorSet()is still followed by additional teardown (e.g.,sessMgtDestroy(),taosConvDestroy()). Either clarify that “last” refers to the ref-pool/app/rpc/task-queue group, or move allocator destruction to truly be the final step that could touch node allocators.
// Order requirement:
// 1) Close request/connection ref pools first, so destroyTscObj() can safely
// access pAppInfo/pTransporter during object teardown.
// 2) Then cleanup app instances and rpc/task queue to stop async activities.
// 3) Destroy allocator set last, otherwise background cleanup paths may hit UAF.
cleanupAppInfo();
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| check=True, | ||
| stderr=subprocess.PIPE, | ||
| stdout=subprocess.PIPE, | ||
| text=True, |
There was a problem hiding this comment.
subprocess.run(..., text=True) still decodes stdout/stderr using the platform encoding with errors='strict', which can raise UnicodeDecodeError when ASAN/other diagnostics include non-decodable bytes. To make this error-path robust, pass an explicit errors='replace' (and ideally an explicit encoding, e.g. utf-8) or keep bytes mode and decode with errors='replace' when formatting the exit message.
| text=True, | |
| text=True, | |
| encoding="utf-8", | |
| errors="replace", |
| check=True, | ||
| stderr=subprocess.PIPE, | ||
| stdout=subprocess.PIPE, | ||
| text=True, |
There was a problem hiding this comment.
subprocess.run(..., text=True) decodes stderr/stdout with errors='strict', so this can still fail with UnicodeDecodeError if the process writes non-UTF8 bytes (common with sanitizer output). Consider adding errors='replace' (and an explicit encoding) or capturing bytes and decoding with errors='replace' when constructing the error message.
| text=True, | |
| text=True, | |
| encoding="utf-8", | |
| errors="replace", |
| check=True, | ||
| stderr=subprocess.PIPE, | ||
| stdout=subprocess.PIPE, | ||
| text=True, |
There was a problem hiding this comment.
subprocess.run(..., text=True) will decode stderr/stdout using the locale encoding with errors='strict', which can still raise UnicodeDecodeError for non-decodable sanitizer output. Add errors='replace' (and preferably an explicit encoding) or capture bytes and decode with errors='replace' before logging.
| text=True, | |
| text=True, | |
| encoding="utf-8", | |
| errors="replace", |
Description
Issue(s)
Checklist
Please check the items in the checklist if applicable.