This repository tracks the network latency over different GNES versions. As a part of CICD pipeline, this repo gets automatically updated when the GNES master is updated or a new GNES version is released.

Please don't change the content of this file manually, as it will be overwritten during the update anyway.

In this benchmark, we setup multiple workflows to represent typical pipelines in the everyday usage of GNES. We then do a "load testing" to determine the system's behavior under normal/peak conditions.

All experiments use gnes/gnes:{version}-alpine as the base image. All microservices are simplified using BaseRouter and BlockRouter.

For the purpose of evaluating this benchmark in CICD pipeline, some environment

For example, to run the third test case on GNES version latest-alpine:

export GNES_IMG_TAG=latest-alpine
export GNES_BENCHMARK_ID=3

make pull && make build && make test d=500 b=10 s=1000000 && make clean

The client will generate 500 documents with the batch size of 10, which yields 50 requests in total. Each document has the size of 1MB. Hence each request is 10MB.

Due to the memory limit of our CD node (t2-micro, 1GB memory), we better keep the number of document in this way.

Time units are in seconds, the smaller the better. Numbers are the best average over three runs.

• roundtrip: the average latency in seconds for a request travel from Frontend and through the whole workflow and finally back to Frontend.
• MB/s: megabyte per second (MB/s) is a unit of data transfer rate over the whole workflow.
• f:send: the average latency in seconds between sending every two requests at the Frontend.
• f:recv: the average latency in seconds between receiving every two requests at the Frontend.
• f->r1:send: the average latency in seconds for Router1 receiving a request sent from Frontend.
• r1->r2:send: the average latency in seconds for Router2 receiving a request sent from Router1 (or all Router1 from the last layer).
• r2->f:send: the average latency in seconds for Frontend receiving a request sent from Router2 (or all Router2 from the last layer).

• version_vcs: corresponds to the gnes-ai/gnes@ commit's SHA hash.
• version_tag: corresponds to the version tag of a GNES docker image.
• timestamp_build: timestamp when the docker image was built.
• timestamp_eval: timestamp when the benchmark was evaluated.

Table results are sorted by timestamp_build with the most recent build at first.

The workflow is as follows:

The ideal roundtrip latency is 0. The smaller the better.

The workflow is as follows:

It simulates a pipeline with uneven workload, Router2 block the pipeline for 1s.

Hence, a naive synchronized pipeline will take 50s to finish 50 requests.

The workflow is as follows:

The ideal roundtrip latency is 0. The smaller the better.

The workflow is as follows:

It simulates a parallel pipeline with heavy workload. Both Router1 and Router2 will block the pipeline for 1s.

As Router1 and Router2 are parallel, a naive synchronized implementation will take 25s to finish 50 requests.