Description

This is a patch release addressing a parachain block production halt that occurred on Kusama on November 25.

The runtime was fixed, and an additional runtime patch was embedded into the parachain chain specification to ensure successful execution of block #9091612, completing the migration.

Post-Mortem

Incident Tracking

A full timeline of events and technical discussion is available in the issue:
paritytech/polkadot-sdk#10428

Background

The runtime upgrade to version 4.0 introduced a major redesign of the parachain block production mechanism.

Previously, the parachain used a simple open relay consensus, where any participant could propose a block.

Starting from runtime v4.0, a new Aura-based consensus algorithm adapted for parachains was introduced.
This new model requires a predefined set of authorities — a dedicated group of nodes authorized to produce blocks.

The initial list of authorities was planned to be set during the runtime upgrade via the OnRuntimeUpgrade hook.
The runtime was tested on a fork of the chain using Chopsticks and prepared for release.

What Went Wrong

On the live network, it was later discovered that OnRuntimeUpgrade is not executed atomically.

Instead, the upgrade process occurs in two phases:

1. The runtime structure changes first (including the installation of the new validation function)
2. The migration logic runs in the following block

This sequencing caused a critical deadlock:

• v3.x collators were unable to produce blocks because they did not support the new runtime structure
• v4.0 collators could not start producing blocks because the authorities set had not yet been initialized

As a result, the parachain became stuck midway through the upgrade, with no collator version capable of proposing a valid block.

Mitigation & Recovery

Resolving the issue required intervention at the relay chain level.

A patched runtime was prepared that temporarily allowed blocks authored by any address in cases where the authorities field was not yet initialized. This made it possible to produce a block and complete the migration.

The situation was further complicated by the fact that, in order to replace the runtime from the relay chain side, the parachain first had to be unlocked.
This operation requires root privileges and therefore involved a Kusama governance vote, which significantly extended the recovery time.

We would like to express our sincere gratitude to the Kusama community and the Parity team for their support and responsiveness throughout this process.

Once the lock was removed, the patch was applied successfully, allowing the migration block to be executed and block production to resume.

Lessons Learned & Preventive Measures

To prevent similar incidents in the future, we are taking the following steps:

• Move from Chopsticks-based testing to Zombienet-based testing
• Validate runtime upgrades in environments that more closely resemble real network conditions
• Explicitly test non-atomic upgrade paths and intermediate runtime states

Summary

This incident highlighted an important edge case in parachain runtime upgrades involving consensus transitions.
The applied patch resolved the immediate issue, and the lessons learned will directly improve the safety and reliability of future upgrades.

Upgrade priority: CRITICAL

Runtimes

Robonomics

✨ spec_version:                40
🏋 Runtime Size:                774570
🗜 Compressed:                  Yes
🎁 Metadata version:            14
🗳️ sha256:                      0x26d974c0aeaae53839930d18b11041a59c2d4a5eb3116ee0f2b3f9e3ce0fca66
🗳️ blake2-256:                  0x359b99b8a5137b69c11df9c143f8a43d208e6c4d525a85e6f65683bb42a54c3a
🗳️ proposal (authorizeUpgrade): 0x280b013620dd22a5803496a3658fe57a3395d8315a47b11fe07005e6da0340ed
📦 IPFS:                        QmS1jSeihoTqQd8gS8SeTgy7hBC7KvuNKatPSngggjxaW2

Build Info

WASM runtime built using rustc 1.88.0 (6b00bc388 2025-06-23)

Robonomics Node 4.0

This release completes the full migration to the Polkadot OMNI node architecture.

One Library to Rule Them All

This version brings major simplifications to the codebase. All custom client code has been removed, and the node now relies entirely on the polkadot-omni-node-lib for collator functionality. This reduces maintenance overhead and unifies behavior with the broader Polkadot OMNI ecosystem.

Async Baking Enabled

Robonomics runtime now supports Async Backing. With this upgrade, bandwidth usage and block production timings are optimized to fully leverage the performance of the Polkadot relay chain.

Keep It Simple

By removing all client-side code, we free up significant space for custom pallets and runtime logic. The Robonomics repository is now streamlined and focused primarily on the Robonomics runtime and its pallets—making development simpler and more maintainable.

Upgrade priority: HIGH

Runtimes

Robonomics

✨ spec_version:                40
🏋 Runtime Size:                778816
🗜 Compressed:                  Yes
🎁 Metadata version:            14
🗳️ sha256:                      0xb0fa7078d81d1608e1f642cedf22f6f3af86700c8670ef41397068a301fcc98f
🗳️ blake2-256:                  0x9db98273c77b937df8a60259e9f264c228035323eb8224628f09e81a6047bd79
🗳️ proposal (authorizeUpgrade): 0xe4fc4dc56425815e6be976e6399ec693b1a0ec9b55c40e544a4e48ad4524e79a
📦 IPFS:                        QmbTGtba3FaDohVzEjTnj1eJX3P9WtYNeviHmFAqM3v2NM

Build Info

WASM runtime built using rustc 1.88.0 (6b00bc388 2025-06-23)

Changes

Client

• Robonomics v4 :: OMNI node release (#386)
• Feat/polkadot sdk uplift (#374)
• Sdk fixes (#380)
• Fix dev node finalization (#384)

Runtime

• ADD: datalog weights (#375)

Others

• Workflow permissions autofixes (#393)
• Bump @babel/runtime from 7.19.4 to 7.28.4 in /.github/scripts (#392)
• Bump @octokit/plugin-paginate-rest from 9.1.5 to 9.2.2 in /.github/scripts (#391)
• Bump @octokit/request-error from 5.0.1 to 5.1.1 in /.github/scripts (#390)
• Bump @octokit/request from 8.1.6 to 8.4.1 in /.github/scripts (#389)

Dependency Changes

Robonomics: v3.3.0...v4.0.0

Download Links

Description

This is runtime only release, on the way to support Snowbridge for Robonomics on Polkadot.

Upgrade priority: LOW

Runtimes

Main

✨ spec_version:                35
🏋 Runtime Size:                932276
🗜 Compressed:                  Yes
🎁 Metadata version:            14
🗳️ sha256:                      0x41a4388535206106a093b2776267852d90aba8db43a8cb1e3a0f813da080274d
🗳️ blake2-256:                  0xf69781179ab8057162c33e66c4e15bbf953b6e1198baaa0483866eb5f41260ec
🗳️ proposal (authorizeUpgrade): 0xf63029532c86a88ab1f50485e474913336ea7be03576e76d8740ff978951bf6f
📦 IPFS:                        QmfFkbePuNujVg5Dz1zBAYUJRPridFsVqG5qjCtotqo99P

Build Info

WASM runtime built using rustc 1.81.0 (eeb90cda1 2024-09-04)

Changes

Client

None

Runtime

• Added Snowbridge support (XCM tags for Ethereum bridged XRT)

Others

• fix release action: docker and srtool (#373)

Dependency Changes

Robonomics: v3.2.0...v3.3.0
Polkadot-sdk:
Frontier:

Download Links

Description

From this release Robonomics runtime become single one for Polkadot and Kusama parachains.

Upgrade priority: MID

Runtimes

Main

✨ spec_version:                34
🏋 Runtime Size:                925586
🗜 Compressed:                  Yes
🎁 Metadata version:            14
🗳️ sha256:                      0xf32ea7618a3d52475b0ae3bac86b65c3ca159945b817e58534eaaefd094e71e5
🗳️ blake2-256:                  0x855017afb39d98bef69c7b2bb18eb89e24c20990bd56e9507dd441d785589120
🗳️ proposal (authorizeUpgrade): 0x226a7627afae08ac8e79865d380a0bcd17480e1131f696d234f84d33b2783f7f
📦 IPFS:                        QmdUSHG4wYmEufdRBJBtkcB9jKpE5w96dx6sdeuXxfhDkp

Build Info

WASM runtime built using rustc 1.81.0 (eeb90cda1 2024-09-04)

Changes

Client

None

Runtime

• Added XcmInfo pallet for manage asset -> multi-location linkage
• Merge Polkadot technical runtime to Kusama runtime
• Remove sudo from polkadot runtime

Others

None

Dependency Changes

Robonomics: v3.0.0...v3.2.0

Download Links

Description

This release marks a significant milestone in the development of the Robonomics parachain with the launch on the Polkadot. Built on the stable Polkadot SDK release 1.0, the primary focus of this update is on enhancing stability and security.

The launch parameters for the Robonomics node have been updated. From this release onward, the target chain must be specified: use 'kusama' for the Kusama relay chain and 'polkadot' for the Polkadot relay-based parachain.

example:

robonomics —chain polkadot

Upgrade priority: HIGH

Runtimes

Main

✨ spec_version:                32
🏋 Runtime Size:                899854
🗜 Compressed:                  Yes
🎁 Metadata version:            14
🗳️ sha256:                      0xaaf2291f4a8cbb2e104fbbb708a3fec1e0daeb0545fc0dde52b1a21f4243fdf4
🗳️ blake2-256:                  0x14cc74e150b5020f16207be4150b7ddaf3f7c88e891d2d8370dbf42cfc7d3d6b
🗳️ proposal (authorizeUpgrade): 0x8cb255e79d721766278d83a11fa024a5d74717ba50eca23e2bcf25045a9eb24b
📦 IPFS:                        QmWBVwQjfXbmWzsJcLrsv86dNzghWNYEgDrGiFMv26px9z

Build Info

WASM runtime built using rustc 1.77.0 (aedd173a2 2024-03-17)

Changes

Client

• Uplift to Polkadot SDK 1.0
• Updated bootnodes list

Others

• Added Polkadot parachain spec file
• Temporary removed IPCI parachain support

Dependency Changes

Robonomics: v2.7.0-1...v3.0.0

Download Links

Native runtimes: robonomics-29, robonomics-alpha-12

Changelog

• Add install protobuf compiler to release actions
• Pass arch as a docker build argument
• Fix build script binary file deletion

Downloads

Native runtimes: robonomics-28, robonomics-alpha-11

Changelog

• Drop crowdloan pallet (fix runtime version section)

Downloads

Native runtimes: robonomics-26, robonomics-alpha-9

Changelog

• Robonomics crowdloan pallet (#312 by akru)

Downloads

Native runtimes: robonomics-25, robonomics-alpha-8, ipci-4

Changelog

• Bump to polkadot-v0.9.28
• Enable IPCI governance & XCM config
• Added pallet_assets into local runtime
• Use evercity official repository for IPCI carbon assets pallet

Downloads

Native runtimes: robonomics-23, robonomics-alpha-6

Changelog

• Reduce democracy waiting time on main runtime and add Crust XCM Token
• Build fixes
• Update ipci spec (#302 by dergudzon)
• [CI] bump rust, add ipci runtime

Downloads