attsim - Attestation Simulation

A GossipSub simulation framework for testing Ethereum attestation propagation using the Shadow discrete-event network simulator.

Overview

This repository provides tools to simulate GossipSub-based peer-to-peer networks using Shadow, focusing on realistic network conditions and Ethereum consensus layer parameters.

Key Features:

• libp2p GossipSub implementation with Ethereum consensus-specs parameters
• Multiple network topology generators (mesh, tree, random-regular)
• Shadow discrete-event network simulation
• Deterministic peer IDs and reproducible simulations
• Client diversity simulation (Prysm/Lighthouse with realistic validation timing)
• Slot-based attestation publishing with configurable slot timing
• Percentile-based BLS batch verification and validation times

Prerequisites

Required

• Go 1.23+ - for building simulation binaries
• Shadow - discrete-event network simulator (installation guide)
• Python 3.12+ - for configuration generation and visualization
• uv - Python package manager (installation guide)

# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Python dependencies are automatically managed by uv via pyproject.toml
# No manual installation needed - uv will handle dependencies when running scripts

Verify Installation

# Check Shadow is installed
shadow --version

# Check Go is installed
go version

# Check uv is installed
uv --version

Quick Start

# Run simulation with default parameters (random-regular topology, 10 nodes)
# This runs the simulation, tests results, and generates a propagation plot
make all

# Or with custom parameters
make run-sim NODE_COUNT=20 MESH_NODE_COUNT=12 MESH_ATTESTER_COUNT=4 NON_MESH_ATTESTER_COUNT=4

# Change topology type
make run-sim TOPOLOGY_TYPE=mesh NODE_COUNT=10

Directory Structure

attsim/
├── README.md              # This file
├── Makefile               # Build and run commands
├── network_graph.py       # Shadow network configuration generator
├── shadow.template.yaml   # Shadow configuration template
├── simconfig.yaml         # Simulation configuration (client split, timing, etc.)
├── plot_propagation.py    # Message propagation plotting script
├── test_results.py        # Simulation results validation
├── simconfig/             # Simulation config parsing
│   └── simconfig.go       # Config data structures
├── topology/              # Network topology generation
│   ├── topology.go        # Topology data structures
│   ├── generators.go      # Topology generation algorithms
│   └── gen/               # Topology generator CLI tool
│       └── main.go
├── gossipsub/             # GossipSub simulation
│   ├── main.go            # Simulation entry point
│   └── slot_ticker.go     # Slot timing management
├── plots/                 # Generated plot outputs
└── updated-plots/         # Latest simulation plots

Usage

The Makefile handles the complete workflow: topology generation, building, and running the Shadow simulation.

Run Simulation

# Run with defaults (random-regular topology, 10 nodes)
make all

# Custom node count and node type distribution
make run-sim NODE_COUNT=20 MESH_NODE_COUNT=10 MESH_ATTESTER_COUNT=5 NON_MESH_ATTESTER_COUNT=5

# Different topologies
make run-sim TOPOLOGY_TYPE=mesh NODE_COUNT=10
make run-sim TOPOLOGY_TYPE=tree NODE_COUNT=31 BRANCHING=2
make run-sim TOPOLOGY_TYPE=random-regular NODE_COUNT=20 PEER_COUNT=6

# With progress bar
make run-sim PROGRESS=true

Makefile Targets

• make all - Complete workflow (build, generate topology, run simulation, test, plot)
• make build - Build the GossipSub binary
• make build-topology-gen - Build the topology generator
• make generate-topology - Generate topology file
• make generate-config - Generate Shadow configuration
• make generate-config-only - Regenerate configuration without rebuild
• make run-shadow - Run Shadow simulation (assumes config exists)
• make run-sim - Run complete simulation (shadow + test + plot)
• make test - Test simulation results
• make plot - Plot message propagation over time
• make clean - Clean up artifacts and results
• make help - Show help with all options

Topology Options

The Makefile supports three topology types:

• mesh: Full connectivity (all-to-all)
• tree: Hierarchical tree with configurable branching factor
• random-regular: Random graph where mesh nodes have uniform degree

Configuration Variables

Node Configuration:

• NODE_COUNT - Total number of nodes (default: 10)
• MESH_NODE_COUNT - Number of mesh subscriber nodes (default: 6)
• MESH_ATTESTER_COUNT - Number of mesh attester nodes that publish & subscribe (default: 2)
• NON_MESH_ATTESTER_COUNT - Number of non-mesh attester nodes that only publish (default: 2)

Topology Configuration:

• TOPOLOGY_TYPE - Topology type: mesh, tree, random-regular (default: random-regular)
• PEER_COUNT - Peer count for mesh nodes in random-regular (default: 4)
• NON_MESH_NODE_PEER_COUNT - Peer count for non-mesh nodes (default: 4)
• BRANCHING - Branching factor for tree topology (default: 2)

Simulation Configuration:

• PROGRESS - Show Shadow progress bar (default: false)
• LOG_LEVEL - Log level (default: info)

Additional parameters are configured in simconfig.yaml:

• client_split - Prysm/Lighthouse client distribution
• prysm_validator - Batch interval and verification times
• lighthouse_validator - Validation times
• slots_to_run - Number of slots to simulate
• msg_size - Message size in bytes

GossipSub Parameters

The simulation uses Ethereum consensus-specs parameters for realistic behavior:

Parameter	Value	Description
D	8	Target mesh size (number of peers)
Dlo	6	Lower bound on mesh size
Dhi	12	Upper bound on mesh size
Dlazy	6	Gossip target (lazy push)
Heartbeat	700ms	Heartbeat interval
Fanout TTL	60s	Time-to-live for fanout peers
History Length	6	Message cache windows
History Gossip	3	Windows to gossip about

Parameters from: Ethereum consensus-specs

Simulation Parameters

Node Types

The simulation supports three types of nodes:

• Mesh Nodes: Subscribe to the topic and receive messages (do not publish)
• Mesh Attester Nodes: Subscribe to the topic AND publish attestation messages
• Non-Mesh Attester Nodes: Only publish messages (fanout peers, do not subscribe)

Client Types

Each node is assigned a client type based on the client_split in simconfig.yaml:

• Prysm: Uses BLS batch verification with configurable batch intervals and percentile-based verification times
• Lighthouse: Uses direct validation with percentile-based validation times

Configuration File (simconfig.yaml)

genesis_time: 946665000        # Unix timestamp for genesis
slot_time: 12                  # Slot duration in seconds
slots_to_run: 30               # Number of slots to simulate
msg_size: 192                  # Message size in bytes

client_split:
  prysm: 44                    # Percentage of Prysm clients
  lighthouse: 56               # Percentage of Lighthouse clients

prysm_validator:
  batch_interval: 5            # Batch interval in milliseconds
  p99_batch_verifier_time: 32500  # Verification times in microseconds
  p95_batch_verifier_time: 8500
  p90_batch_verifier_time: 5000
  p75_batch_verifier_time: 4350
  p50_batch_verifier_time: 3900
  p25_batch_verifier_time: 3900

lighthouse_validator:
  p99_validator_time: 9500     # Validation times in microseconds
  p95_validator_time: 4900
  p90_validator_time: 4650
  p75_validator_time: 3700
  p50_validator_time: 2600
  p25_validator_time: 1300

General Parameters

• Node Count: Total number of participating nodes
• Topology: Network connection graph (mesh, tree, random-regular)
• Log Level: debug, info, warn, error

Development

Building

# Build everything and run simulation
make all

# Or build components separately
make build-topology-gen  # Build topology generator
make build              # Build GossipSub binary

Testing Different Configurations

# Test different node counts and distributions
make run-sim NODE_COUNT=10 MESH_NODE_COUNT=6 MESH_ATTESTER_COUNT=2 NON_MESH_ATTESTER_COUNT=2
make run-sim NODE_COUNT=50 MESH_NODE_COUNT=30 MESH_ATTESTER_COUNT=10 NON_MESH_ATTESTER_COUNT=10
make run-sim NODE_COUNT=100 MESH_NODE_COUNT=60 MESH_ATTESTER_COUNT=20 NON_MESH_ATTESTER_COUNT=20

# Test different topologies
make run-sim TOPOLOGY_TYPE=mesh NODE_COUNT=10
make run-sim TOPOLOGY_TYPE=tree NODE_COUNT=31 BRANCHING=2
make run-sim TOPOLOGY_TYPE=random-regular NODE_COUNT=20 PEER_COUNT=6

# Test different peer counts for mesh vs non-mesh nodes
make run-sim PEER_COUNT=8 NON_MESH_NODE_PEER_COUNT=4

Viewing Results

# View all logs
cat shadow.data/hosts/node*/gossipsub.*.stdout

# View specific node logs
cat shadow.data/hosts/node0/gossipsub.*.stdout
cat shadow.data/hosts/node1/gossipsub.*.stdout

# Check for errors
grep -i error shadow.data/hosts/*/gossipsub.*.stderr

# Plot message propagation
make plot

Clean Up

# Clean build artifacts and simulation results
make clean

Architecture

Simulation Workflow

1. Topology Generation

   • The Makefile automatically generates a topology file based on parameters
   • Supported types: mesh (full connectivity), tree (hierarchical), random-regular (uniform degree)
   • Topology includes node type assignments (mesh, mesh-attester, non-mesh-attester)
   • Client type distribution (Prysm/Lighthouse) based on simconfig.yaml
   • Topology saved as JSON with node connections and metadata

2. Shadow Configuration

   • Network graph generated with realistic global latencies (Australia, Europe, Asia, Americas, Africa)
   • Node types: supernodes (1024 Mbps) and fullnodes (50 Mbps)
   • Nodes distributed across geographic locations
   • Reads parameters from both topology and simconfig files

3. GossipSub Simulation

   • Each node runs as a separate process in Shadow
   • Nodes use deterministic peer IDs for reproducibility
   • All nodes start synchronized at 2000/01/01 00:02:00
   • Slot-based message publishing (attester nodes publish once per slot)
   • Message validation based on client type (Prysm batch vs Lighthouse direct)

Topology JSON Format

{
  "node_count": 10,
  "mesh_node_ids": [0, 1, 2, 3, 4, 5],
  "mesh_attester_node_ids": [6, 7],
  "non_mesh_attester_node_ids": [8, 9],
  "prysm_node_ids": [0, 2, 4, 6, 8],
  "lighthouse_node_ids": [1, 3, 5, 7, 9],
  "connections": [
    {"from": 0, "to": 1},
    {"from": 1, "to": 2},
    {"from": 2, "to": 3}
  ]
}

Node Behavior

Mesh Nodes (subscribers only):

1. Join GossipSub topic and subscribe
2. Receive and log messages from attesters
3. Participate in gossip protocol mesh

Mesh Attester Nodes (publish & subscribe):

1. Join GossipSub topic and subscribe
2. Publish one attestation message per slot
3. Receive and log messages from other attesters
4. Validate messages using client-specific validation (Prysm batch or Lighthouse direct)

Non-Mesh Attester Nodes (fanout publishers):

1. Join GossipSub topic (without subscribing)
2. Publish one attestation message per slot
3. Use fanout mechanism to reach mesh peers

Client Validation

Prysm Client:

• Uses BLS batch verification queue
• Messages are batched at configurable intervals (default: 5ms)
• Batch verification time follows percentile distribution from real-world data

Lighthouse Client:

• Uses direct per-message validation
• Validation time follows percentile distribution from real-world data

Key Features

• Deterministic peer IDs for reproducibility
• Shadow-aware timing (synchronized start)
• Topology-based peering (controlled network structure)
• Slot-based message publishing
• Realistic client validation timing (Prysm/Lighthouse)
• Percentile-based verification time distributions

Plots

the 50ms_batch_interval_plot.png represents the message propagation CDF with the 50ms batch interval.

the 5ms_batch_interval_plot.png represents the message propagation CDF with the 5ms batch interval.

More details on the topology, client_split, batch_verifier times, lighthouse validator time should be self-explanatory in the plot

License

Same as parent project.

Related Projects

• eth-ec-broadcast - Parent project with erasure coding improvements
• Shadow - Discrete-event network simulator
• libp2p - Modular peer-to-peer networking stack
• go-libp2p-pubsub - GossipSub implementation