A framework for building open & decentralized AI projects

This includes all the core components required to launch a decentralized AI application, including:

• Kademlia DHT (KAD-DHT) – for scalable, decentralized storage and routing
• Asyncio-based DHT Node – designed for fast, concurrent communications
• DHT Protocol – allows DHT nodes to request keys/neighbors from other DHT nodes, and manages routing tables
• DHT Record Storage – decentralized key-value storage with support for versioned and validated records with customizable predicate extensions
• Record Validators – attach custom validation logic to any stored record, such as key authentication and Pydantic schemas
• DHT Traversal Tools – Traverse the DHT graph
• Routing Tables – manage network topology and neighbor nodes. A data structure that contains DHT peers bucketed according to their distance to node_id. Follows Kademlia routing table
• P2P Servicer Base – register RPC methods to the DHT for nodes to call on one another with security authorizer extensions
• Proof-of-Stake Integration – incentivize and secure participation, i.e. must be staked to join the subnet
• Hypertensor Consensus – Ready to run in parallel to the Hypertensor consensus mechanism
• Substrate Integration – Connect to Hypertensor with an RPC endpoint
• Secure Communication – support for Ed25519 and RSA authentication for communication

💡 Focus on Logic, Not Plumbing The networking, cryptography, consensus, and storage layers are already handled. As a subnet builder, your only responsibility is to implement the application logic — the custom AI protocols and behaviors that live on top of the DHT.

https://docs.hypertensor.org

To install this container from source, simply run the following:

git clone https://github.com/hypertensor-blockchain/mesh.git
cd mesh
python -m venv .venv
source .venv/bin/activate
pip install .

If you would like to verify that your installation is working properly, you can install with pip install .[dev] instead. Then, you can run the tests with pytest tests/.

By default, the contatiner uses the precompiled binary of the go-libp2p-daemon library. If you face compatibility issues or want to build the binary yourself, you can recompile it by running pip install . --global-option="--buildgo".

Before running the compilation, please ensure that your machine has a recent version of Go toolchain (1.15 or 1.16 are supported).

• Linux is the default OS for which the container is developed and tested. We recommend Ubuntu 18.04+ (64-bit), but other 64-bit distros should work as well. Legacy 32-bit is not recommended.
• macOS is partially supported. If you have issues, you can run the container using Docker instead. We recommend using our Docker image.
• Windows 10+ (experimental) can run the container using WSL. You can configure WSL to use GPU by following sections 1–3 of this guide by NVIDIA. After that, you can simply follow the instructions above to install with pip or from source.

Create an .env file for the environmental variables.

touch .env

Fill out the .env file with the necessary variables from .env.example.

The template currently allows RSA and Ed25519 key types and is interoperable between the two.

• This is the hotkey of the node. It is used for validating and attesting only. Each hotkey is unique to each subnet node and no hotkey can be used twice.

• This will create 3 private key files for your peer
  • peer_id: Main peer ID for communication and signing
  • bootstrap_peer_id: (Optional usage) Peer ID to be used as a bootstrap node.
  • client_peer_id: (Optional usage) Peer ID to be used as a client. This is for those who want to build frontends to interact with the subnet.

Example Private Key Generation

keygen \
--path rsa-pk.key \
--bootstrap_path rsa-bootstrap-pk.key \
--client_path rsa-client-pk.key  \
--key_type rsa

• Call register_subnet_node
• Retrieve your start_epoch by querying your SubnetNodesData storage element on polkadot.js with your subnet node ID. This is the epoch you must activate your node on + the grace period

register-node \
--subnet_id 1 \
--hotkey 0x773539d4Ac0e786233D90A233654ccEE26a613D9 \
--peer_id QmTJ8uyLJBwVprejUQfYFAywdXWfdnUQbC1Xif6QiTNta9 \
--bootnode_peer_id QmSjcNmhbRvek3YDQAAQ3rV8GKR8WByfW8LC4aMxk6gj7v \
--client_peer_id QmSjcNmhbRvek3YDQAAQ3rV8GKR8WByfW8LC4aMxk6gj71 \
--bootnode /ip4/127.00.1/tcp/31330/p2p/QmSjcNmhbRvek3YDQAAQ3rV8GKR8WByfW8LC4aMxk6gj7v \
--client_peer_id QmbRz8Bt1pMcVnUzVQpL2icveZz2MF7VtELC44v8kVNwiG \
--delegate_reward_rate 0.125 \
--stake_to_be_added 100.00

• Call activate_subnet_node in Hypertensor on your start epoch up to the grace period.

activate-node --subnet_id 1

• Replace port 31330 with your port of choice.
• Replace {your_ip} with your IP.

Note: To be verified to have a proof-of-stake in-subnet, use the bootnode private key generated during the Generate peer private keys step that the subnet node was registered with under the bootnode_peer_id.

Note: Bootnodes are not required by subnet nodes and are expected to be managed by the subnet owner entity.

A bootnode is an entry point into a decentralized network and should be ran on its own server.

Each subnet must have at least one public and running bootnode at all times for nodes to enter and for for Overwatch Nodes to validate a subnet. See documentation for more information.

mesh-dht \
--host_maddrs /ip4/0.0.0.0/tcp/31330 /ip4/0.0.0.0/udp/31330/quic \
--announce_maddrs /ip4/{your_ip}/tcp/31330 /ip4/{your_ip}/udp/31330/quic \
--identity_path alith.id

Once you run it, look at the outputs and find the following line:

Mon 00 01:23:45.678 [INFO] Running a DHT instance. To connect other peers to this one, use --initial_peers /ip4/YOUR_ADDRESS_HERE/tcp/31337/p2p/QmTPAIfThisIsMyAddressGoFindYoursnCfj

Once the bootnode is deployed, copy the /ip4/YOUR_ADDRESS_HERE/tcp/31337/p2p/QmTPAIfThisIsMyAddressGoFindYoursnCfj into the PUBLIC_INITIAL_PEERS in the .env file.

• Get the bootnode multiaddresses from the subnets team or blockchain and add them to the initial_peers argument.

mesh-dht \
--host_maddrs /ip4/0.0.0.0/tcp/31330 /ip4/0.0.0.0/udp/31330/quic \
--announce_maddrs /ip4/{your_ip} \
--initial_peers /ip4/{ip}/p2p/{peer_id} /ip4/{ip}/p2p/{peer_id}

The following is example only for deploying a subnet with no use-case.

This will start a new subnet (fresh swarm as initial node/bootnode and server in one)

mesh-server-mock \
--host_maddrs /ip4/0.0.0.0/tcp/31330 /ip4/0.0.0.0/udp/31330/quic \
--announce_maddrs /ip4/{your_ip}/tcp/31330 /ip4/{your_ip}/udp/31330/quic \
--identity_path bootnode.id \
--new_swarm  \
--subnet_id 1 --subnet_node_id 1

Once you run it, look at the outputs and find the following line:

Mon 00 01:23:45.678 [INFO] Running a DHT instance. To connect other peers to this one, use --initial_peers /ip4/YOUR_ADDRESS_HERE/tcp/31337/p2p/QmTPAIfThisIsMyAddressGoFindYoursnCfj

Once the bootnode is deployed, copy the /ip4/YOUR_ADDRESS_HERE/tcp/31337/p2p/QmTPAIfThisIsMyAddressGoFindYoursnCfj into the PUBLIC_INITIAL_PEERS in the .env file.

mesh-server-mock \
--host_maddrs /ip4/0.0.0.0/tcp/31331 /ip4/0.0.0.0/udp/31331/quic \
--announce_maddrs /ip4/{your_ip}/tcp/31331 /ip4/{your_ip}/udp/31331/quic \
--identity_path alith.id \
--subnet_id 1 --subnet_node_id 1

Start a mesh locally with 3 nodes and no requirement for a blockchain connection:

(this is a node that as a bootnode and a server) Note: In production, a bootnode should not be ran as a server.

mesh-server-mock \
    --host_maddrs /ip4/0.0.0.0/tcp/31330 /ip4/0.0.0.0/udp/31330/quic \
    --announce_maddrs /ip4/{your_ip}/tcp/31330 /ip4/{your_ip}/udp/31330/quic \
    --identity_path bootnode.id \
    --subnet_id 1 --subnet_node_id 1 \
    --no_blockchain_rpc \
    --new_swarm

Once you run it, look at the outputs and find the following line:

Mon 00 01:23:45.678 [INFO] Running a DHT instance. To connect other peers to this one, use --initial_peers /ip4/YOUR_ADDRESS_HERE/tcp/31337/p2p/QmTPAIfThisIsMyAddressGoFindYoursnCfj

Once the bootnode is deployed, copy the /ip4/YOUR_ADDRESS_HERE/tcp/31337/p2p/QmTPAIfThisIsMyAddressGoFindYoursnCfj into the PUBLIC_INITIAL_PEERS in the .env file.

mesh-server-mock \
    --host_maddrs /ip4/0.0.0.0/tcp/31331 /ip4/0.0.0.0/udp/31331/quic \
    --announce_maddrs /ip4/{your_ip}/tcp/31331 /ip4/{your_ip}/udp/31331/quic \
    --identity_path alith.id \
    --subnet_id 1 --subnet_node_id 2 \
    --no_blockchain_rpc

mesh-server-mock \
    --host_maddrs /ip4/0.0.0.0/tcp/31332 /ip4/0.0.0.0/udp/31332/quic \
    --announce_maddrs /ip4/{your_ip}/tcp/31332 /ip4/{your_ip}/udp/31332/quic \
    --identity_path baltathar.id \
    --subnet_id 1 --subnet_node_id 3 \
    --no_blockchain_rpc

• charleth.id
• dorothy.id
• ethan.id
• faith.id

The following example will use Alith as the subnet owner and as the node example (registering and running the node). To add more nodes for testing with a running local blockchain, see mesh/mesh_cli/hypertensor/README.md to view each test identity path, it's hotkeys, coldkeys, and their peer IDs.

See mesh/mesh_cli/hypertensor/node/register.py to register more test accounts.

1. Register subnet
2. Register at least 3 nodes
3. Run the nodes
4. Activate the subnet

Once the subnet is activated, consensus will begin on the following epoch between all of the nodes in the subnet.

With Alith's coldkey as the owner and with Alith, Baltathar, Charleth, and Dorothy as the initial coldkeys:

register-subnet \
  --max_cost 100.00 \
  --name subnet-1 \
  --repo github.com/subnet-1 \
  --description "artificial intelligence" \
  --misc "cool subnet" \
  --churn_limit 64 \
  --min_stake 100.00 \
  --max_stake  1000.00 \
  --delegate_stake_percentage 0.1 \
  --subnet_node_queue_epochs 10 \
  --idle_classification_epochs 10 \
  --included_classification_epochs 10 \
  --max_node_penalties 10 \
  --initial_coldkeys "0xf24FF3a9CF04c71Dbc94D0b566f7A27B94566cac" "0x3Cd0A705a2DC65e5b1E1205896BaA2be8A07c6e0" "0x798d4Ba9baf0064Ec19eB4F0a1a45785ae9D6DFc" "0x773539d4Ac0e786233D90A233654ccEE26a613D9" \
  --max_registered_nodes 10 \
  --key_types "Rsa" \
  --bootnodes "test_bootnode" \
  --private_key "0x5fb92d6e98884f76de468fa3f6278f8807c48bebc13595d45af5bdc4da702133" \
  --local_rpc

Note: The client peer ID, bootnode peer ID, and bootnode are only for testing purposes. In production, the client peer ID and bootnode peer ID should be generated beforehand and each have its own identity paths (the bootnode will be derived from the bootnode peer ID if utilized). The client and bootnode peer ID are required on-chain but not required to be used off-chain in the subnet. The bootnode is optional.

Using Alith's coldkey private key:

register-node \
  --subnet_id 1 \
  --hotkey 0x317D7a5a2ba5787A99BE4693Eb340a10C71d680b \
  --peer_id QmShJYgxNoKn7xqdRQj5PBcNfPSsbWkgFBPA4mK5PH73JB \
  --bootnode_peer_id QmShJYgxNoKn7xqdRQj5PBcNfPSsbWkgFBPA4mK5PH73JC \
  --bootnode /ip4/127.00.1/tcp/31330/p2p/QmShJYgxNoKn7xqdRQj5PBcNfPSsbWkgFBPA4mK5PH73JC \
  --client_peer_id QmShJYgxNoKn7xqdRQj5PBcNfPSsbWkgFBPA4mK5PH73JD \
  --delegate_reward_rate 0.125 \
  --stake_to_be_added 100.00 \
  --max_burn_amount 100.00 \
  --private_key "0x5fb92d6e98884f76de468fa3f6278f8807c48bebc13595d45af5bdc4da702133" \
  --local_rpc

Get the subnet node ID after registration

Using Alith's hotkey private key:

mesh-server-mock \
    --host_maddrs /ip4/0.0.0.0/tcp/31331 /ip4/0.0.0.0/udp/31331/quic \
    --announce_maddrs /ip4/{your_ip}/tcp/31331 /ip4/{your_ip}/udp/31331/quic \
    --identity_path alith.id \
    --subnet_id 1 --subnet_node_id 2 \
    --local_rpc \
    --private_key "0x51b7c50c1cd27de89a361210431e8f03a7ddda1a0c8c5ff4e4658ca81ac02720"

• Migrate to py-libp2p (from go-libp2p-daemon).
• Random-walk
• Gossip protocol integration
• Noise Protocol (Diffie-Hellman key exchange)
• Onion routing or mixnet options
• Multiple encryption options
• Explore alternative tensor/AI parameter compression options
• DHT Record uniqueness options
• Runtime upgrades
• In-subnet epochs (synced to blockchain clock) standard with ledger
• Ledger integration
• Runtime upgrade
  • Versioned protocols
  • Orchestration tooling
  • Blue-green deployment standardizations
• Etc.

This is currently at the active development stage, and we welcome all contributions. Everything, from bug fixes and documentation improvements to entirely new features, is appreciated.

If you want to contribute to this mesh template but don't know where to start, take a look at the unresolved issues.

Open a new issue or join our chat room in case you want to discuss new functionality or report a possible bug. Bug fixes are always welcome, but new features should be preferably discussed with maintainers beforehand.

[0]: Maymounkov, P., & Mazières, D. (2002). Kademlia: A Peer-to-Peer Information System Based on the XOR Metric. In P. Druschel, F. Kaashoek, & A. Rowstron (Eds.), Peer-to-Peer Systems (pp. 53–65). Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/3-540-45748-8_5

[1]: Baumgart, I., & Mies, S. (2014). S / Kademlia : A practicable approach towards secure key-based routing S / Kademlia : A Practicable Approach Towards Secure Key-Based Routing, (June). https://doi.org/10.1109/ICPADS.2007.4447808

[2]: Freedman, M. J., & Mazières, D. (2003). Sloppy Hashing and Self-Organizing Clusters. In IPTPS. Springer Berlin / Heidelberg. Retrieved from https://www.cs.princeton.edu/~mfreed/docs/coral-iptps03.pdf