Codestin Search App

huniu20 · 2026-07-02T08:07:07Z

Summary

This PR adds two features to the MooncakeStore backend:

1. P2PHANDSHAKE Metadata Mode

Mooncake supports a peer-to-peer handshake mode for metadata discovery that does not require a separate HTTP metadata server. This is particularly useful in multi-NIC environments (e.g., clusters with both RDMA and business network interfaces) where the HTTP metadata server may advertise the wrong IP address for TCP transport.

Usage: Set metadata_server: P2PHANDSHAKE in the config (case-insensitive, normalized internally).

2. SSD Offload Support

As GPU HBM capacity grows rapidly (H100 80GB -> H200 141GB -> B200 192GB -> B300 288GB), the ratio of CPU DRAM to GPU HBM is decreasing. In future clusters, total CPU DRAM may be smaller than total GPU HBM. This makes it essential to offload TQ data to NVMe SSD when DRAM is scarce.

When offload.enabled: true is set in the config, the bootstrap will:

Start mooncake_master with eviction enabled
Start a standalone mooncake_client process that offloads evicted data to SSD
Automatically disable hard_pin (so objects can be evicted and offloaded)

All parameters are configured through TQ standard config system (OmegaConf/YAML).

Configuration Example

MooncakeStore:
  auto_init: true
  metadata_server: P2PHANDSHAKE
  master_server_address: localhost:50051
  protocol: tcp
  device_name: bond1
  global_segment_size: 4294967296
  local_buffer_size: 1073741824
  hard_pin: true  # auto-disabled when offload is enabled

  offload:
    enabled: true
    file_storage_path: /data/mooncake_offload
    eviction_high_watermark_ratio: 0.9
    eviction_ratio: 0.1

Backward Compatibility

Default offload.enabled: false - existing users are not affected
Default metadata_server: localhost:50050 - HTTP mode still works as before
Default hard_pin: true - existing behavior preserved when offload is disabled
P2PHANDSHAKE is only activated when explicitly configured

Key Design Decisions

hard_pin vs offload: Hard-pinned objects are never evicted by Mooncake. When offload is enabled, hard_pin is automatically set to false (unless explicitly overridden) so that eviction can trigger and data flows to SSD.
Offload client lifecycle: The standalone mooncake_client process is tracked and terminated on tq.close(). If it fails to start, the bootstrap raises RuntimeError (hard fail) to prevent silent degradation.
P2PHANDSHAKE normalization: The metadata_server value is case-insensitively matched and normalized to the exact string "P2PHANDSHAKE" before passing to Mooncake (which requires exact match).
Parameter validation: eviction_high_watermark_ratio and eviction_ratio are validated to be in (0.0, 1.0].

Files Changed

transfer_queue/config.yaml - Added offload sub-section, hard_pin option, and P2PHANDSHAKE documentation
transfer_queue/storage/bootstrap/mooncake_bootstrap.py - P2PHANDSHAKE support + SSD offload with lifecycle management
transfer_queue/storage/clients/mooncake_client.py - P2PHANDSHAKE normalization + configurable hard_pin
transfer_queue/interface.py - Cleanup offload client process on close()

Testing

Tested with a 4-node GPU cluster:

P2PHANDSHAKE mode: TCP transport connects correctly across nodes (no wrong-IP issues in multi-NIC environments)
SSD offload: mooncake_master + mooncake_client start correctly from config, data offloaded to NVMe SSD as expected
Offload client failure: Verified that bootstrap raises RuntimeError and terminates master when client fails
End-to-end: Full distributed workload completes successfully with data flowing through DRAM -> SSD offload path

ascend-robot · 2026-07-02T08:07:17Z

CLA Signature Pass

huniu20, thanks for your pull request. All authors of the commits have signed the CLA. 👍

ascend-robot · 2026-07-02T08:56:54Z

CLA Signature Pass

huniu20, thanks for your pull request. All authors of the commits have signed the CLA. 👍

…eStore - Add P2PHANDSHAKE metadata mode support in mooncake_bootstrap.py and mooncake_client.py. This is Mooncake's official peer-to-peer metadata discovery mode, which avoids IP detection issues in multi-NIC environments (e.g., clusters with both RDMA and business network interfaces). - Add SSD offload support via config.yaml's offload sub-section under MooncakeStore. When enabled, the bootstrap automatically starts a standalone mooncake_client process that offloads data from DRAM to NVMe SSD when memory usage exceeds the configured high watermark. This is essential for environments where total CPU DRAM < GPU HBM (increasingly common with H200/B200/B300 generation GPUs). - All offload parameters are configured through TQ's standard config system (OmegaConf/YAML), not environment variables. Default behavior is unchanged (offload.enabled: false). Backward compatible: existing users are not affected. Signed-off-by: huniu20 <[email protected]>

ascend-robot · 2026-07-02T09:15:55Z

CLA Signature Guide

@huniu20 , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit	Reason
[`2df5895` [feat] Support P2PHANDSHAKE met...](`2df5895`)	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

0oshowero0 · 2026-07-03T06:30:36Z

    # Network device name.
    # Set to "" to let Mooncake auto-select available devices.
    device_name: ""
+    # Whether to hard-pin objects in memory. Hard-pinned objects are never evicted.


Please also update the config in https://github.com/Ascend/TransferQueue/blob/main/scripts/performance_test/perftest_config.yaml

Thanks for the review! I've addressed all the feedback, including copilot reviews. All changes are in the latest commit:)

Copilot

Pull request overview

Adds new MooncakeStore backend capabilities to improve deployability in multi-NIC clusters and enable spilling evicted data to NVMe SSD when host DRAM is constrained.

Changes:

Add P2PHANDSHAKE metadata discovery mode (skip HTTP metadata server, normalize config).
Add SSD offload bootstrap flow (start mooncake_master with eviction/offload and a standalone mooncake_client offload process).
Add offload client lifecycle cleanup on tq.close() and extend default config with offload-related knobs.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
`transfer_queue/storage/clients/mooncake_client.py`	Normalizes `metadata_server` for P2PHANDSHAKE and makes hard-pin behavior configurable for offload.
`transfer_queue/storage/bootstrap/mooncake_bootstrap.py`	Adds P2PHANDSHAKE-aware master startup and optional SSD-offload client process management.
`transfer_queue/interface.py`	Terminates the standalone offload client process during `close()`.
`transfer_queue/config.yaml`	Documents P2PHANDSHAKE and introduces `hard_pin` + `offload` configuration section.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    # Whether to hard-pin objects in memory. Hard-pinned objects are never evicted.
+    # When offload is enabled, this is automatically set to false unless explicitly overridden.
+    # Set to true for maximum read performance when offload is not needed.
+    hard_pin: true


+        # When offload is enabled, hard_pin must be disabled so that objects can be evicted
+        # and offloaded to SSD. Hard-pinned objects are never evicted by Mooncake.
+        offload_conf = config.get("offload", {})
+        offload_enabled = offload_conf.get("enabled", False) if isinstance(offload_conf, dict) else False
+        self.replica_config.with_hard_pin = bool(config.get("hard_pin", not offload_enabled))


ascend-robot · 2026-07-03T06:37:07Z

CLA Signature Guide

@huniu20 , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit	Reason
[`2df5895` [feat] Support P2PHANDSHAKE met...](`2df5895`)	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

Signed-off-by: huniu20 <[email protected]>

ascend-robot · 2026-07-03T06:49:21Z

CLA Signature Guide

@huniu20 , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit	Reason
[`2df5895` [feat] Support P2PHANDSHAKE met...](`2df5895`)	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

…rd_pin auto-management - mooncake_bootstrap.py: Add wait(timeout=5) + fallback kill() after process.terminate() when offload client fails to start, preventing stray mooncake_master processes if SIGTERM is ignored. - interface.py: Add wait(timeout=5) + fallback kill() after offload_proc.terminate() during tq.close() cleanup, preventing zombie processes. - mooncake_client.py: Treat hard_pin=None as 'auto' mode. When not explicitly set by user, hard_pin is automatically disabled when offload is enabled. This fixes a bug where OmegaConf merge always provides hard_pin=true from defaults, preventing auto-disable. - config.yaml + perftest_config.yaml: Change hard_pin default from true to null to enable auto-management. Signed-off-by: huniu20 <[email protected]>

ascend-robot · 2026-07-03T07:11:00Z

CLA Signature Guide

@huniu20 , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit	Reason
[`2df5895` [feat] Support P2PHANDSHAKE met...](`2df5895`)	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

0oshowero0 · 2026-07-03T07:40:02Z

+        client_port = str(offload_conf.get("client_port", 42052))
+        local_buffer_size = str(offload_conf.get("local_buffer_size_bytes", 2147483648))
+        use_uring = "1" if offload_conf.get("use_uring", False) else "0"
+        heartbeat_interval = str(offload_conf.get("heartbeat_interval_seconds", 2))


Is this heartbeat interval is the same with "-client_ttl=30"? If so, we should align the params

They are different but related parameters:

-client_ttl=30 is on the master side: the timeout (in seconds) after which the master considers a client dead if no heartbeat is received.

heartbeat_interval_seconds=2 is on the offload client side: how often the client sends heartbeats to the master.

They form a pair: the client sends a heartbeat every 2s, and the master tolerates up to 30s of silence before evicting the client (i.e., ~15 missed heartbeats). This provides reasonable fault tolerance without being too aggressive.

0oshowero0 · 2026-07-03T07:43:37Z

+            metadata_server_url = f"http://{metadata_server_host}:{metadata_server_port}/metadata"
+        master_address = f"{master_server_parsed.hostname}:{master_server_port}"
+
+        client_cmd = [


This offload client seems only exist on the node that performs tq.init. So it's not a distributed offload? If so, we should clearly state this limitation in doc & codes, and suggest user to use a shared network address

Yes, the current implementation starts a single mooncake_client (offload) process on the node that calls tq.init(). This is by design in Mooncake's architecture:

The offload client acts as a centralized SSD storage pool that registers itself with mooncake_master.

When eviction is triggered, the master moves data from any node's memory to this offload client via TCP/RDMA — so it does serve the entire cluster, not just the local node.

The local_hostname parameter ensures the offload client is reachable from all other nodes in the cluster.

Besides, I agree this should be documented more clearly. I'll:

Add a comment in the code explaining that the offload client is a single-node centralized SSD pool serving the whole cluster.

Add a note in config.yaml suggesting users ensure the local_hostname is a cluster-reachable address and the SSD path has sufficient capacity.

…tl relationship Signed-off-by: nexhu <[email protected]> Signed-off-by: huniu20 <[email protected]>

ascend-robot · 2026-07-03T07:57:01Z

CLA Signature Guide

@huniu20 , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit	Reason
[`2df5895` [feat] Support P2PHANDSHAKE met...](`2df5895`)	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

0oshowero0 · 2026-07-03T08:44:04Z

+    # Start standalone mooncake_client for SSD offload if enabled.
+    #
+    # Architecture: The offload client is a single-node centralized SSD storage pool.
+    # It runs only on the node that calls tq.init(), but serves the entire cluster:


the node that calls tq.init() -> the first node that calls tq.init()

I have no other comments. Thanks for your contribution :)

Thanks for the review! I've updated the comment as suggested.

Signed-off-by: huniu20 <[email protected]>

ascend-robot · 2026-07-03T08:51:53Z

CLA Signature Guide

@huniu20 , thanks for your pull request.

The following commit(s) are not associated with a signed Contributor License Agreement (CLA).

Commit	Reason
[`2df5895` [feat] Support P2PHANDSHAKE met...](`2df5895`)	the email used in the commit is not linked to a signed CLA! please verify that it matches the email you used when signing the CLA.

To sign CLA, click here.

To check if your email is configured correctly, refer to the FAQs.

Once you've signed the CLA or updating your email, please comment /check-cla to revalidate CLA status.

ascend-robot added the ascend-cla/yes label Jul 2, 2026

huniu20 force-pushed the feat/mooncake-ssd-offload-and-p2phandshake branch from 85ed266 to 3007b1c Compare July 2, 2026 08:56

huniu20 force-pushed the feat/mooncake-ssd-offload-and-p2phandshake branch from 3007b1c to 2df5895 Compare July 2, 2026 09:15

ascend-robot added ascend-cla/no and removed ascend-cla/yes labels Jul 2, 2026

0oshowero0 requested a review from Copilot July 3, 2026 06:26

Copilot started reviewing on behalf of 0oshowero0 July 3, 2026 06:27 View session

0oshowero0 reviewed Jul 3, 2026

View reviewed changes

Copilot AI reviewed Jul 3, 2026

View reviewed changes

fix(review): sync hard_pin and offload config to perftest_config.yaml

375710b

Signed-off-by: huniu20 <[email protected]>

huniu20 force-pushed the feat/mooncake-ssd-offload-and-p2phandshake branch from 31e5eec to 375710b Compare July 3, 2026 06:49

0oshowero0 reviewed Jul 3, 2026

View reviewed changes

docs: clarify single-node offload architecture and heartbeat/client_t…

695c05d

…tl relationship Signed-off-by: nexhu <[email protected]> Signed-off-by: huniu20 <[email protected]>

0oshowero0 reviewed Jul 3, 2026

View reviewed changes

docs: clarify 'the first node that calls tq.init()' per review

22e20f8

Signed-off-by: huniu20 <[email protected]>

0oshowero0 approved these changes Jul 3, 2026

View reviewed changes

0oshowero0 merged commit b0e4bf5 into Ascend:main Jul 3, 2026
8 checks passed

Uh oh!

Conversation

huniu20 commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

1. P2PHANDSHAKE Metadata Mode

2. SSD Offload Support

Configuration Example

Backward Compatibility

Key Design Decisions

Files Changed

Testing

Uh oh!

ascend-robot commented Jul 2, 2026

CLA Signature Pass

Uh oh!

ascend-robot commented Jul 2, 2026

CLA Signature Pass

Uh oh!

ascend-robot commented Jul 2, 2026

CLA Signature Guide

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

ascend-robot commented Jul 3, 2026

CLA Signature Guide

Uh oh!

ascend-robot commented Jul 3, 2026

CLA Signature Guide

Uh oh!

ascend-robot commented Jul 3, 2026

CLA Signature Guide

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ascend-robot commented Jul 3, 2026

CLA Signature Guide

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ascend-robot commented Jul 3, 2026

CLA Signature Guide

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

huniu20 commented Jul 2, 2026 •

edited

Loading