ADBDEV-7691, ADBDEV-7690: Track PQresult allocations in server side, Gracefully handle OOM when cancelling a query #1804

bandetto · 2025-06-27T12:06:32Z

When querying pg_locks (or using pg_lock_status()), approximately 75% of backend memory allocations for resulting tuples weren't registered with Vmtracker or Resource Group Control. This memory would also leak if the query was cancelled or failed.

This happened because CdbDispatchCommand(), which is used by pg_locks, internally uses libpq to obtain the results, which were allocated as PQresult structures with bare malloc(), even on the server side.

Another issue is that if a query fails under resource group manager due to a lack of memory, the query is cancelled due to an error. The coordinator would still try to receive tuples from segments upon cancellation. Retrieving tuples requires more memory allocations, which may cause the same OOM error again. The query will get cancelled due to an error, entering into a recursion and leading to an eventual crash.

The first commit fixes both the memory leak and untracked memory issues by enforcing Vmtracker routines for PGresult allocations on the server-side. The original code is unchanged from #1231, besides addressing errcode-related macro redefinition warnings by un-defining them first.

The second commit prevents error recursion, upon query cancellation. It implements proper query cancellation by explicitly ignoring any libpq data from segments without extra memory allocations. The said issue cannot be reproduced without first commit. The original code was updated since #1231, based on #1668.

The PR contains several patches as separate, atomic commits, and therefore should be rebased. ADCC extension tests should pass before merging the PR.

When approved the PR will be moved to Greengage repo for merge. ABI tests for adb-6.x-dev are broken.

When querying pg_locks (or using `pg_lock_status()`), approximately 75% of backend memory allocations for resulting tuples weren't registered with Vmtracker or Resource Group Control. This memory would also leak if the query was cancelled or failed. This happened because `CdbDispatchCommand()`, which is used by `pg_locks`, internally uses libpq to obtain the results, which were allocated as PQresult structures with bare `malloc()`, even on the server side. This patch fixes both untracked memory issues by enforcing Vmtracker routines for `PGresult` allocations on the server-side. Please note that the original error from ADBDEV-7145, which is an `AbortTransaction()` recurstion due to an OOM error while cancelling a query, is fixed only in the next commit. Ticket: ADBDEV-7691

Previosly, if a query fails under resource group manager due to a lack of memory, the query is cancelled due to an error. The coordinator would still try to receive tuples from segments upon cancellation. Retrieving tuples requires more memory allocations, which may cause the same OOM error again. The query will get cancelled due to an error, entering into a recursion and leading to an eventual crash. This patch implements proper query cancellation by explicitly ignoring any libpq data from segments without extra memory allocations. Ticket: ADBDEV-7690

When querying pg_locks (or using `pg_lock_status()`), approximately 75% of backend memory allocations for resulting tuples weren't registered with Vmtracker or Resource Group Control. This memory would also leak if the query was cancelled or failed. This happened because `CdbDispatchCommand()`, which is used by `pg_locks`, internally uses libpq to obtain the results, which were allocated as PQresult structures with bare `malloc()`, even on the server side. This patch fixes both untracked memory issues by enforcing Vmtracker routines for `PGresult` allocations on the server-side. Please note that the original error from ADBDEV-7145, which is an `AbortTransaction()` recurstion due to an OOM error while cancelling a query, is fixed only in the next commit. Ticket: ADBDEV-7691

Previosly, if a query fails under resource group manager due to a lack of memory, the query is cancelled due to an error. The coordinator would still try to receive tuples from segments upon cancellation. Retrieving tuples requires more memory allocations, which may cause the same OOM error again. The query will get cancelled due to an error, entering into a recursion and leading to an eventual crash. This patch implements proper query cancellation by explicitly ignoring any libpq data from segments without extra memory allocations. Ticket: ADBDEV-7690

When querying pg_locks (or using `pg_lock_status()`), approximately 75% of backend memory allocations for resulting tuples weren't registered with Vmtracker or Resource Group Control. This memory would also leak if the query was cancelled or failed. This happened because `CdbDispatchCommand()`, which is used by `pg_locks`, internally uses libpq to obtain the results, which were allocated as PQresult structures with bare `malloc()`, even on the server side. This patch fixes both untracked memory issues by enforcing Vmtracker routines for `PGresult` allocations on the server-side. Please note that the original error from ADBDEV-7145, which is an `AbortTransaction()` recurstion due to an OOM error while cancelling a query, is fixed only in the next commit. Ticket: ADBDEV-7691

Previosly, if a query fails under resource group manager due to a lack of memory, the query is cancelled due to an error. The coordinator would still try to receive tuples from segments upon cancellation. Retrieving tuples requires more memory allocations, which may cause the same OOM error again. The query will get cancelled due to an error, entering into a recursion and leading to an eventual crash. This patch implements proper query cancellation by explicitly ignoring any libpq data from segments without extra memory allocations. Ticket: ADBDEV-7690

Previosly, `cdbconn_discardResults()` has proved to be unrealiable in case of OOM, as it would still try to read the results, causing errors in recursion if called when aborting a query. This patch refactors the said function by transferring code from another similar procedure, to make sure we're discarding results in a more robust way, adressing all libpq pitfalls instead of mindlessly retrying, as opposed to the past behavior. Ticket: ADBDEV-7690

When querying pg_locks (or using `pg_lock_status()`), approximately 75% of backend memory allocations for resulting tuples weren't registered with Vmtracker or Resource Group Control. This memory would also leak if the query was cancelled or failed. This happened because `CdbDispatchCommand()`, which is used by `pg_locks`, internally uses libpq to obtain the results, which were allocated as PQresult structures with bare `malloc()`, even on the server side. This patch fixes both untracked memory issues by enforcing Vmtracker routines for `PGresult` allocations on the server-side. Please note that the original error from ADBDEV-7145, which is an `AbortTransaction()` recurstion due to an OOM error while cancelling a query, is fixed only in the next commit. Ticket: ADBDEV-7691

Previosly, if a query fails under resource group manager due to a lack of memory, the query is cancelled due to an error. The coordinator would still try to receive tuples from segments upon cancellation. Retrieving tuples requires more memory allocations, which may cause the same OOM error again. The query will get cancelled due to an error, entering into a recursion and leading to an eventual crash. This patch implements proper query cancellation by explicitly ignoring any libpq data from segments without extra memory allocations. Ticket: ADBDEV-7690

Previosly, `cdbconn_discardResults()` has proved to be unrealiable in case of OOM, as it would still try to read the results, causing errors in recursion if called when aborting a query. This patch refactors the said function by transferring code from another similar procedure, to make sure we're discarding results in a more robust way, adressing all libpq pitfalls instead of mindlessly retrying, as opposed to the past behavior. Ticket: ADBDEV-7690

When querying pg_locks (or using `pg_lock_status()`), approximately 75% of backend memory allocations for resulting tuples weren't registered with Vmtracker or Resource Group Control. This memory would also leak if the query was cancelled or failed. This happened because `CdbDispatchCommand()`, which is used by `pg_locks`, internally uses libpq to obtain the results, which were allocated as PQresult structures with bare `malloc()`, even on the server side. This patch fixes both untracked memory issues by enforcing Vmtracker routines for `PGresult` allocations on the server-side. Please note that the original error from ADBDEV-7145, which is an `AbortTransaction()` recurstion due to an OOM error while cancelling a query, is fixed only in the next commit. Ticket: ADBDEV-7691

Previosly, if a query fails under resource group manager due to a lack of memory, the query is cancelled due to an error. The coordinator would still try to receive tuples from segments upon cancellation. Retrieving tuples requires more memory allocations, which may cause the same OOM error again. The query will get cancelled due to an error, entering into a recursion and leading to an eventual crash. This patch implements proper query cancellation by explicitly ignoring any libpq data from segments without extra memory allocations. Ticket: ADBDEV-7690

Previosly, `cdbconn_discardResults()` has proved to be unrealiable in case of OOM, as it would still try to read the results, causing errors in recursion if called when aborting a query. This patch refactors the said function by transferring code from another similar procedure, to make sure we're discarding results in a more robust way, adressing all libpq pitfalls instead of mindlessly retrying, as opposed to the past behavior. Ticket: ADBDEV-7690

src/test/isolation2/input/resgroup/resgroup_oom.source

src/backend/cdb/dispatcher/cdbdisp_async.c

RekGRpth · 2025-07-25T04:11:34Z

src/backend/cdb/dispatcher/cdbdisp_async.c

+	pqSaveErrorResult(conn);
+
+	/* Make sure PQgetResult() calls are not blocking. */
+	PQconsumeInput(conn);


Why don't you check what this function returned?!

The function returns 0 on error, and you ignore it!

Why shouldn't it be ignored? Errors will be caught by PQgetResult() and a condition at the end.

see similar function processResults

I still can't understand you and I don't see a point in checking the return code. Can you please explain the reason?

src/backend/cdb/dispatcher/cdbdisp_async.c

src/test/isolation2/input/parallel_retrieve_cursor/fault_inject.source

When querying pg_locks (or using `pg_lock_status()`), approximately 75% of backend memory allocations for resulting tuples weren't registered with Vmtracker or Resource Group Control. This memory would also leak if the query was cancelled or failed. This happened because `CdbDispatchCommand()`, which is used by `pg_locks`, internally uses libpq to obtain the results, which were allocated as PQresult structures with bare `malloc()`, even on the server side. This patch fixes both untracked memory issues by enforcing Vmtracker routines for `PGresult` allocations on the server-side. Including `postgres.h` in frontend code causes several errcode-related macro redefinition warnings. They are now un-definined first. Please note that the original error from ADBDEV-7145, which is an `AbortTransaction()` recurstion due to an OOM error while cancelling a query, is fixed only in the next commit. Ticket: ADBDEV-7691

Previosly, if a query fails under resource group manager due to a lack of memory, the query is cancelled due to an error. The coordinator would still try to receive tuples from segments upon cancellation. Retrieving tuples requires more memory allocations, which may cause the same OOM error again. The query will get cancelled due to an error, entering into a recursion and leading to an eventual crash. This patch implements proper query cancellation by explicitly ignoring any libpq data from segments without extra memory allocations. Ticket: ADBDEV-7690

When querying pg_locks (or using `pg_lock_status()`), approximately 75% of backend memory allocations for resulting tuples weren't registered with Vmtracker or Resource Group Control. This memory would also leak if the query was cancelled or failed. This happened because `CdbDispatchCommand()`, which is used by `pg_locks`, internally uses libpq to obtain the results, which were allocated as PQresult structures with bare `malloc()`, even on the server side. This patch fixes both untracked memory issues by enforcing Vmtracker routines for `PGresult` allocations on the server-side. Including `postgres.h` in frontend code causes several errcode-related macro redefinition warnings. They are now un-definined first. Please note that the original error from ADBDEV-7145, which is an `AbortTransaction()` recurstion due to an OOM error while cancelling a query, is fixed only in the next commit. Ticket: ADBDEV-7691

Previosly, if a query fails under resource group manager due to a lack of memory, the query is cancelled due to an error. The coordinator would still try to receive tuples from segments upon cancellation. Retrieving tuples requires more memory allocations, which may cause the same OOM error again. The query will get cancelled due to an error, entering into a recursion and leading to an eventual crash. This patch implements proper query cancellation by explicitly ignoring any libpq data from segments without extra memory allocations. Ticket: ADBDEV-7690

RekGRpth · 2025-07-29T06:27:30Z

Still, I suggest separating the patch for correcting the recursive error into a separate PR and accepting it first.

Previosly, if a query fails under resource group manager due to a lack of memory, the query is cancelled due to an error. The coordinator would still try to receive tuples from segments upon cancellation. Retrieving tuples requires more memory allocations, which may cause the same OOM error again. The query will get cancelled due to an error, entering into a recursion and leading to an eventual crash. This patch implements proper query cancellation by explicitly ignoring any libpq data from segments without extra memory allocations. Ticket: ADBDEV-7690

bandetto · 2025-07-29T07:00:17Z

Still, I suggest separating the patch for correcting the recursive error into a separate PR and accepting it first.

I disagree. The current approach was chosen since the recursive error patch cannot be tested without the first commit.

RekGRpth · 2025-07-29T07:09:16Z

src/backend/cdb/dispatcher/cdbdisp_async.c

+			 * might not be enough memory to discard the result properly.
+			 * Let's get the big guns out.
+			 */
+			tryToWipeResults(dispatchResult);


Your tryToWipeResults function is very similar to the processResults function: in both functions

first the PQconsumeInput function is called

then there is a loop until PQisBusy

inside the loop the PQgetResult function is called
Maybe it would be easier to add the pParms->waitMode == DISPATCH_WAIT_CANCEL argument to the processResults function and its processing?

processResults() is complicated and I'll risk breaking something else when making changes there.

cdbconn_discardResults() is also almost identical to the added function. But tryToWipeResults() was added for a reason and does only what it needs to.

Again, why we cannot use cdbconn_discardResults()?
And I agree with @RekGRpth about avoiding injecting a special case this way. Are you sure there is not any approach that looks more organic?

Again, why we cannot use cdbconn_discardResults()

Because it's used in other context and I'd prefer not to change existing behavior.

Are you sure there is not any approach that looks more organic?

I think the current state of the patch is the most straightforward and safe way to make this change.

Previosly, if a query fails under resource group manager due to a lack of memory, the query is cancelled due to an error. The coordinator would still try to receive tuples from segments upon cancellation. Retrieving tuples requires more memory allocations, which may cause the same OOM error again. The query will get cancelled due to an error, entering into a recursion and leading to an eventual crash. This patch implements proper query cancellation by explicitly ignoring any libpq data from segments without extra memory allocations. Ticket: ADBDEV-7690

bimboterminator1

Why do you write about CdbDispatchCommand in pg_locks_status if it has been removed from there? Write description in way more consistent with current changes , please

bimboterminator1 · 2025-08-02T21:06:34Z

src/backend/cdb/dispatcher/cdbdisp_async.c

+			 * might not be enough memory to discard the result properly.
+			 * Let's get the big guns out.
+			 */
+			tryToWipeResults(dispatchResult);


Again, why we cannot use cdbconn_discardResults()?
And I agree with @RekGRpth about avoiding injecting a special case this way. Are you sure there is not any approach that looks more organic?

src/interfaces/libpq/fe-protocol3.c

src/backend/cdb/dispatcher/cdbdisp_async.c

When querying pg_locks (or using `pg_lock_status()`), approximately 75% of backend memory allocations for resulting tuples weren't registered with Vmtracker or Resource Group Control. This memory would also leak if the query was cancelled or failed. This happened because `CdbDispatchCommand()`, which was previously used by `pg_locks`, called libpq to obtain the results that were allocated as PQresult structures with bare `malloc()`, even on the server side. This patch fixes both untracked memory issues by enforcing Vmtracker routines for `PGresult` allocations on the server-side. Including `postgres.h` in frontend code causes several errcode-related macro redefinition warnings. They are now un-definined first. Please note that the original error from ADBDEV-7145, which is an `AbortTransaction()` recurstion due to an OOM error while cancelling a query, is fixed only in the next commit. Ticket: ADBDEV-7691

Previosly, if a query fails under resource group manager due to a lack of memory, the query is cancelled due to an error. The coordinator would still try to receive tuples from segments upon cancellation. Retrieving tuples requires more memory allocations, which may cause the same OOM error again. The query will get cancelled due to an error, entering into a recursion and leading to an eventual crash. This patch implements proper query cancellation by explicitly ignoring any libpq data from segments without extra memory allocations. Ticket: ADBDEV-7690

bimboterminator1

LGTM. Update PR description.

RekGRpth · 2025-08-07T16:31:53Z

Still, I suggest separating the patch for correcting the recursive error into a separate PR and accepting it first.

I disagree. The current approach was chosen since the recursive error patch cannot be tested without the first commit.

I seem to have managed to reproduce the bug fixed by the second commit without using the first commit.

diff --git a/src/backend/cdb/dispatcher/cdbdisp_async.c b/src/backend/cdb/dispatcher/cdbdisp_async.c
index c41dbe55a72..c8cea03a508 100644
--- a/src/backend/cdb/dispatcher/cdbdisp_async.c
+++ b/src/backend/cdb/dispatcher/cdbdisp_async.c
@@ -866,6 +866,11 @@ handlePollSuccess(CdbDispatchCmdAsync *pParms,
 		ELOG_DISPATCHER_DEBUG("PQsocket says there are results from %d of %d (%s)",
 							  i + 1, pParms->dispatchCount, segdbDesc->whoami);
 
+		if (pParms->waitMode == DISPATCH_WAIT_CANCEL)
+		{
+			elog(ERROR, "Out of memory was emulated");
+		}
+
 		/*
 		 * Receive and process results from this QE.
 		 */

or

diff --git a/src/backend/cdb/dispatcher/cdbdisp_async.c b/src/backend/cdb/dispatcher/cdbdisp_async.c
index c41dbe55a72..a6fdd0e4a6b 100644
--- a/src/backend/cdb/dispatcher/cdbdisp_async.c
+++ b/src/backend/cdb/dispatcher/cdbdisp_async.c
@@ -1076,6 +1076,11 @@ processResults(CdbDispatchResult *dispatchResult)
 		ELOG_DISPATCHER_DEBUG("PQgetResult");
 		pRes = PQgetResult(segdbDesc->conn);
 
+		if (dispatchResult->wasCanceled)
+		{
+			elog(ERROR, "Out of memory was emulated");
+		}
+
 		/*
 		 * Command is complete when PGgetResult() returns NULL. It is critical
 		 * that for any connection that had an asynchronous command sent thru

CREATE FUNCTION gp_mock_cdbdispatchcommand(amount int) RETURNS SETOF text AS '/home/gpadmin/src/gpdb6/src/test/isolation2/../regress/regress.so', 'gp_mock_cdbdispatchcommand' LANGUAGE C;

SELECT count(*) FROM gp_mock_cdbdispatchcommand(10000000);
^CCancel request sent
ERROR:  canceling statement due to user request
ERROR:  Out of memory was emulated (cdbdisp_async.c:871)
ERROR:  Out of memory was emulated (cdbdisp_async.c:871)
ERROR:  Out of memory was emulated (cdbdisp_async.c:871)
ERROR:  Out of memory was emulated (cdbdisp_async.c:871)
ERROR:  Out of memory was emulated (cdbdisp_async.c:871)
ERROR:  Out of memory was emulated (cdbdisp_async.c:871)
ERROR:  Out of memory was emulated (cdbdisp_async.c:871)
ERROR:  Out of memory was emulated (cdbdisp_async.c:871)
ERROR:  Out of memory was emulated (cdbdisp_async.c:871)
PANIC:  ERRORDATA_STACK_SIZE exceeded (elog.c:1661)
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!>

bandetto · 2025-08-08T06:02:56Z

I seem to have managed to reproduce the bug fixed by the second commit without using the first commit.

Injecting an error into the middle of a function does not really check the fundamental issue behind the task, which are failed memory allocations. Your solution checks only for re-entrance into processResults() and handlePollSuccess().

bandetto · 2025-08-11T18:04:39Z

Evil PR counterpart from Greengage: GreengageDB/greengage#45

bandetto force-pushed the ADBDEV-7690-7691 branch 2 times, most recently from ea58a4d to 3e21592 Compare June 27, 2025 12:50

bandetto force-pushed the ADBDEV-7690-7691 branch from 3e21592 to 2afc07d Compare June 27, 2025 15:06

bandetto changed the title ~~ADBDEV-7691: Track PQresult allocations in server code, ADBDEV-7690: Gracefully handle OOM when cancelling the query~~ ADBDEV-7691: Track PQresult allocations in server side, ADBDEV-7690: Gracefully handle OOM when cancelling a query Jun 27, 2025

bandetto changed the title ~~ADBDEV-7691: Track PQresult allocations in server side, ADBDEV-7690: Gracefully handle OOM when cancelling a query~~ ADBDEV-7691, ADBDEV-7690: Track PQresult allocations in server side, Gracefully handle OOM when cancelling a query Jun 27, 2025

bandetto force-pushed the ADBDEV-7690-7691 branch 2 times, most recently from 9b90def to 8013158 Compare July 1, 2025 08:22

bandetto force-pushed the ADBDEV-7690-7691 branch from 8013158 to 907999e Compare July 1, 2025 08:43

bandetto force-pushed the ADBDEV-7690-7691 branch from 907999e to c5f2f45 Compare July 1, 2025 09:02

bandetto force-pushed the ADBDEV-7690-7691 branch 4 times, most recently from c61249d to 4709631 Compare July 23, 2025 05:51

This comment was marked as resolved.

Sign in to view

RekGRpth reviewed Jul 25, 2025

View reviewed changes

src/test/isolation2/input/resgroup/resgroup_oom.source Outdated Show resolved Hide resolved

RekGRpth reviewed Jul 25, 2025

View reviewed changes

src/backend/cdb/dispatcher/cdbdisp_async.c Outdated Show resolved Hide resolved

RekGRpth reviewed Jul 25, 2025

View reviewed changes

src/backend/cdb/dispatcher/cdbdisp_async.c Show resolved Hide resolved

RekGRpth reviewed Jul 25, 2025

View reviewed changes

src/test/isolation2/input/parallel_retrieve_cursor/fault_inject.source Outdated Show resolved Hide resolved

RekGRpth reviewed Jul 25, 2025

View reviewed changes

src/test/isolation2/input/parallel_retrieve_cursor/fault_inject.source Show resolved Hide resolved

bandetto force-pushed the ADBDEV-7690-7691 branch from 4709631 to 118c338 Compare July 25, 2025 11:03

bandetto force-pushed the ADBDEV-7690-7691 branch from 118c338 to 6b2fe85 Compare July 25, 2025 11:04

bandetto force-pushed the ADBDEV-7690-7691 branch from 6b2fe85 to 6610c93 Compare July 29, 2025 06:55

RekGRpth reviewed Jul 29, 2025

View reviewed changes

bandetto force-pushed the ADBDEV-7690-7691 branch from 6610c93 to 0d087a1 Compare July 29, 2025 15:46

bimboterminator1 reviewed Aug 2, 2025

View reviewed changes

bandetto added 2 commits August 4, 2025 09:41

bandetto force-pushed the ADBDEV-7690-7691 branch from 0d087a1 to 8d681ef Compare August 4, 2025 06:42

bimboterminator1 approved these changes Aug 4, 2025

View reviewed changes

ADBDEV-7691, ADBDEV-7690: Track PQresult allocations in server side, Gracefully handle OOM when cancelling a query #1804

Are you sure you want to change the base?

ADBDEV-7691, ADBDEV-7690: Track PQresult allocations in server side, Gracefully handle OOM when cancelling a query #1804

Uh oh!

Conversation

bandetto commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

RekGRpth Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

bandetto Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

RekGRpth Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

bandetto Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

RekGRpth Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

bandetto Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RekGRpth commented Jul 29, 2025

Uh oh!

bandetto commented Jul 29, 2025

Uh oh!

RekGRpth Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

bandetto Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bimboterminator1 Aug 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bandetto Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

bimboterminator1 left a comment

Choose a reason for hiding this comment

Uh oh!

bimboterminator1 Aug 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bimboterminator1 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RekGRpth commented Aug 7, 2025

Uh oh!

bandetto commented Aug 8, 2025

Uh oh!

bandetto commented Aug 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bandetto commented Jun 27, 2025 •

edited

Loading

bandetto Jul 29, 2025 •

edited

Loading

bandetto Jul 29, 2025 •

edited

Loading

bimboterminator1 Aug 2, 2025 •

edited

Loading

bimboterminator1 Aug 2, 2025 •

edited

Loading

bimboterminator1 left a comment •

edited

Loading