Thanks to visit codestin.com
Credit goes to github.com

Skip to content

New ETag Commands Syntax + ETag Object Support#1478

Draft
TalZaccai wants to merge 347 commits intodevfrom
talzacc/storage-v2-etag
Draft

New ETag Commands Syntax + ETag Object Support#1478
TalZaccai wants to merge 347 commits intodevfrom
talzacc/storage-v2-etag

Conversation

@TalZaccai
Copy link
Contributor

@TalZaccai TalZaccai commented Dec 19, 2025

This PR introduces a notion of "meta commands" - commands that "envelop" existing data commands.
The meta commands introduced in this PR are ETag related - EXECWITHETAG (execute a command and add the dest key's etag to the output), as well as EXECIFMATCH, EXECIFNOTMATCH, EXECIFGREATER (conditionally-execute a command based on the dest key's ETag in relation to the ETag meta-argument).

These command are able to replace the existing etag-related commands and command arguments, and in-turn support many more (including object commands).
For example:
DELIFGREATER key etag -> EXECIFGREATER etag DEL key
GETIFNOTMATCH key etag -> EXECIFNOTMATCH etag GET key
GETWITHETAG key -> EXECWITHETAG GET key
SETIFMATCH key value etag -> EXECIFMATCH etag SET key value

To achieve this logic, we parse the meta command and its meta-arguments into the SessionParseState. Currently this logic supports a constant number of meta-arguments which is defined in the Arity property in the meta-command's RespCommandInfo (arg count = arity - 1).
When the parse state gets handed to the appropriate ISessionFunctions method, the method decides whether the operation should get executed (in the case of conditional-execution) and to what value should the record's ETag advance (if at all).
The "special output" (in case the ETag needs to be written along with the command output) is handled by the ISessionFunctions method in the main and unified case, and in the IGarnetObject.Operate method in the object case.

To-do:

  • Add new supported meta-commands, remove existing etag commands and command arguments & update usages
  • Implement meta-command and meta-argument parsing in the RESP command parser.
  • Update SessionParseState to parse and hold meta-arguments
  • Update InputHeader to include meta-command and update commands
  • Implement ETag-related meta-command handling in ISessionFunctions for all 3 contexts
  • Implement meta-command-based ETag output for all object commands
  • Handle ETags in operations that do not call into ISessionFunctions (internal transactions)
  • Further clean-up of ETag logic in ISessionFunctions
  • Introduce leaner AOF serialization for commands with single long meta-argument
  • Support programmatic commands supplying long etag without serializing to parse state

Testing & Benchmarking:

  • Expand existing main-store ETag tests to include all meta-commands
  • Verify all write commands result in ETag advancement
  • Verify ETag output added to all commands when meta-command present
  • Verify running meta-commands within transactions
  • Verify benchmarking numbers on commands without ETags
  • Consider adding benchmarking logic for meta-commands with basic commands / basic commands with ETags

Almost done with Migrate changeover to LogRecord
… for SpanByteAllocator. The AddressType change is a breaking on-disk format change: it shuffles bits around in RecordInfo to add an additional bit adjacent to the old ReadCache bit to mark an address as:

- 00: Reserved
- 11: ReadCache
- 10: InMemory portion of the main log
- 01: On-Disk
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 113 out of 114 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (10)

test/Garnet.test/RespBlockingCollectionTests.cs:1

  • The timeout has been changed from 5 seconds to 500 seconds. This will cause tests to hang for up to ~8 minutes if the blocking tasks fail to complete, making test failures very slow to detect. This appears to be an unintended change — the original 5-second timeout was reasonable for a test environment. Please revert this to a shorter timeout (e.g., TimeSpan.FromSeconds(30)) or the original value.
    libs/server/Resp/RespServerSession.cs:1
  • When cmd is not a data command, the error is written but execution continues: outputEtag is still set to true and the switch expression below will still be evaluated. This means the error response will be followed by an attempt to execute the command and then another write of the etag integer, producing a malformed response. There should be a return true; immediately after the error is sent (or after the SendAndReset() loop), similar to how other early-exit error paths are handled.
    libs/server/Storage/Functions/UnifiedStore/VarLenInputMethods.cs:1
  • For GetUpsertFieldInfo (used when upserting via a source log record — e.g., in RENAME), HasETag is only set when the meta-command is ExecWithEtag. However, if the source log record already had an ETag, the destination record should also preserve it. The logic should be similar to CheckModifiedRecordHasEtag, which accounts for the existing ETag on the source record. Currently, renaming a key that already has an ETag will silently drop the ETag from the new record unless EXECWITHETAG is explicitly used.
    libs/server/StringOutput.cs:1
  • The Error flag (bit 7) is used in InvalidTypeError and NaNOrInfinityError before it is defined in the enum. In C#, enum member values are evaluated in declaration order, so InvalidTypeError = Error | (1 << 4) will use the value of Error as 0 (its default), resulting in InvalidTypeError = 0 | 16 = 16 and NaNOrInfinityError = 0 | 32 = 32. These will no longer have the Error bit set, breaking the HasError check ((OutputFlags & StringOutputFlags.Error) != 0). The Error constant must be declared before InvalidTypeError and NaNOrInfinityError.
    test/Garnet.test/Resp/ETag/GeneralEtagCoverageTests.cs:1
  • After EXECIFMATCH 1 INCR key is executed twice in a transaction where the key has ETag 1, the second INCR should not execute (because after the first INCR, the ETag advances to 2, making the condition etag == 1 false). The test expects the numeric value to be 2, but the correct expected value after one successful INCR on the string "1" is 2 — however the ETag expectation of 2 (the third argument to VerifyResultAndETag) may also be off if only one INCR runs. Please verify the expected ETag and value match the actual conditional execution semantics.
    libs/server/Storage/Functions/ObjectStore/UpsertMethods.cs:1
  • The return value of logRecord.TryCopyFrom is silently discarded. If the copy fails, logRecord is in an indeterminate state but execution continues, potentially advancing the ETag of a partially-written record. The result should be checked: if it returns false, the method should return false.
    libs/server/Resp/RespServerSession.cs:1
  • The array header *2\r\n is written unconditionally before the command executes (line 741-742), but the ETag is only written after, as the second array element. If the command inside the meta-command itself writes an array (e.g., EXECWITHETAG SMEMBERS key), the outer *2 array will contain an array as its first element and the ETag integer as its second — this is a nested/non-flat structure. Verify that the response format (command result + ETag integer as a flat 2-element array) is correct and consistent with what clients expect for all possible wrapped commands, including those that return arrays.
    libs/server/Resp/BasicEtagCommands.cs:1
  • GETETAG is a standalone read command (not a meta-command), so output.IsOperationSkipped should never be true here — IsOperationSkipped reflects whether a meta-command's conditional check failed. Treating IsOperationSkipped as a reason to write null for GETETAG is misleading and may mask unexpected states. If the key is not found, status != GarnetStatus.OK already handles it. This condition should be simplified to just status != GarnetStatus.OK.
    test/Garnet.test/Resp/ETag/SetCommandsETagCoverageTests.cs:1
  • db.ExecWithEtag("SADD", ...) returns a RedisResult (not string[]). Casting directly to string[] will throw at runtime for a multi-bulk RESP reply. The correct cast is (RedisResult[]), and then individual elements should be cast to string or long. Compare with GeoCommandsETagCoverageTests.DataSetUp where (string[])db.ExecWithEtag("GEOADD", ...) is used but likely has the same issue.
    test/Garnet.test/Resp/ETag/GeoCommandsETagCoverageTests.cs:1
  • Same issue as in SetCommandsETagCoverageTests.DataSetUp: db.ExecWithEtag(...) returns a RedisResult, and casting to string[] will fail at runtime. The result should be cast to RedisResult[] instead.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@TalZaccai TalZaccai changed the title [WIP] New ETag Commands Syntax + ETag Object Support New ETag Commands Syntax + ETag Object Support Mar 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants