While it currently serves reasonably well, Capnp has several drawbacks for Rain:

• The interface and support for some languages is limited (e.g. no RPC for some, Pytohn API is very low-level).
  • No support in browser means we need REST(-like) API for monitoring anyway.
• Compiling in some HPC environments may be an additional problem (may not improve with other solutions).
• The message size is limited to 64MB (see Capnp message size caps limits one submit size #8) which makes large submits (and possibly other information calls) complicated.
• We do not really need the advanced features of Capnp such as remote objects (these also need an additional packet to free even if they have no methods, see RPC: Avoid redundant finish packet when return carries no capabilities capnproto/capnproto#534).
  • The current capnp-rust sends every message in 2 packets - a further minor slowdown
  • However we do need bi-directional RPC calls, so no simple server/client RPC would do.

Therefore it might be better to use different protocols for different interfaces:

• For client, low latency is not critical while portability and ease of use are, so it may make sense to have a REST API (or gRPC). The monitoring javascript app already uses REST for information.
  • It would be nice to leverage serde-json for the messages
  • API can be specified with a tool to generate code stubs and verifiers (like Apiary?)
• For server-worker communication, it would be ergonomic to use serde, tarpc or possibly even abomonation with some simple framing.
• For worker-subworker communication, it would be best to have some multi-platform and simple and yet fast protocol. Here the solution is less clear, but the communication schema is simpler (even if ideally with bi-directional messages).

This issue is a basis for discussion of requirements and options. The transition from Capnp is not critical.