Fixes #40

Checklist

• My branch is up-to-date with upstream/develop branch.
• Everything works and tested for Python 3.8.0 and above.
• I ran pre-commit checks against my changes.
• I've written tests against my changes and all the current present tests are passing.

Current behaviour

There are no APIs that expose the processing status of a snapshot according to the state transitions it goes through before being finalized.

For more details, refer #40

New expected behaviour

The changes included in this PR capture the state transitions as detailed in #40 as per the following data models as an epoch is released until SnapshotFinalized event is received by the processor distributor for the specific epoch.

EPOCH_RELEASED

Redis key being set:

https://github.com/PowerLoom/pooler/blob/bcc245d228acce504ba803b9b50fd89c8eb05984/snapshotter/utils/redis/redis_keys.py#L90-L91

https://github.com/PowerLoom/pooler/blob/bcc245d228acce504ba803b9b50fd89c8eb05984/snapshotter/processor_distributor.py#L649-L652

It contains the UNIX timestamp at which the EpochReleased event was received from the protocol state contract.

For each of the state IDs corresponding to a state transition, except EPOCH_RELEASED, a Redis hashtable entry is set against every project ID or type being processed by the snapshotter. The key of this hashtable is in the following format

https://github.com/PowerLoom/pooler/blob/bcc245d228acce504ba803b9b50fd89c8eb05984/snapshotter/utils/redis/redis_keys.py#L94-L95

and each value against a hashtable entry of a project ID follows the data model:

https://github.com/PowerLoom/pooler/blob/bcc245d228acce504ba803b9b50fd89c8eb05984/snapshotter/utils/models/data_models.py#L48-L52

PRELOAD

For every project type's preloader specifications, the status of all the preloading dependencies being satisfied is captured here:

https://github.com/PowerLoom/pooler/blob/bcc245d228acce504ba803b9b50fd89c8eb05984/snapshotter/processor_distributor.py#L227-L251

SNAPSHOT_BUILD

The snapshot builders as configured in projects.json are executed. Also refer to the case study of the current implementation of Pooler for a detailed look at snapshot building for base as well as aggregates.

https://github.com/PowerLoom/pooler/blob/bcc245d228acce504ba803b9b50fd89c8eb05984/snapshotter/utils/snapshot_worker.py#L100-L120

SNAPSHOT_SUBMIT_PAYLOAD_COMMIT

Captures the status of propagation of the built snapshot to the payload commit service in Audit Protocol for further submission to the protocol state contract.

https://github.com/PowerLoom/pooler/blob/bcc245d228acce504ba803b9b50fd89c8eb05984/snapshotter/utils/generic_worker.py#L166-L195

SNAPSHOT_SUBMIT_PROTOCOL_CONTRACT

The snapshot submission transaction from the relayer to the protocol state smart contract was successful and a SnapshotSubmitted event was generated

https://github.com/PowerLoom/pooler/blob/bcc245d228acce504ba803b9b50fd89c8eb05984/snapshotter/processor_distributor.py#L655-L668

SNAPSHOT_FINALIZE

Captures the finalized snapshot accepted against an epoch via a SnapshotFinalized event.

https://github.com/PowerLoom/pooler/blob/bcc245d228acce504ba803b9b50fd89c8eb05984/snapshotter/processor_distributor.py#L475-L482

Core API ([core_api.py](https://github.com/PowerLoom/pooler/blob/bcc245d228acce504ba803b9b50fd89c8eb05984/snapshotter/core_api.py)) now exposes an internal API endpoint GET /internal/snapshotter/epochProcessingStatus that returns a paginated response of the state transition status of snapshots against project IDs, upto 30 most recent epochs.

Request:

curl -X 'GET' \
  'http://localhost:8002/internal/snapshotter/epochProcessingStatus?page=1&size=10' \
  -H 'accept: application/json'

Sample response:

{
    "items": [
      {
        "epochId": 43523,
        "transitionStatus": {
          "EPOCH_RELEASED": {
            "status": "success",
            "error": null,
            "extra": null,
            "timestamp": 1692530595
          },
          "PRELOAD": {
            "pairContract_pair_total_reserves": {
              "status": "success",
              "error": null,
              "extra": null,
              "timestamp": 1692530595
            },
          },
          "SNAPSHOT_BUILD": {
            "aggregate_24h_stats_lite:35ee1886fa4665255a0d0486c6079c4719c82f0f62ef9e96a98f26fde2e8a106:UNISWAPV2": {
              "status": "success",
              "error": null,
              "extra": null,
              "timestamp": 1692530596
            },
          },
          "SNAPSHOT_SUBMIT_PAYLOAD_COMMIT": {

          },
         "RELAYER_SEND": {

         },
        "SNAPSHOT_FINALIZE": {

        },
      },
    }
   ],
   "total": 30,
   "page": 1,
   "size": 10,
   "pages": 3
}

Change logs

Added

• Core API query endpoint GET /internal/snapshotter/epochProcessingStatus

• Record state transition of a snapshot at every step

• Send health pings to onchain consensus reporting service every 2 seconds

https://github.com/PowerLoom/pooler/blob/bcc245d228acce504ba803b9b50fd89c8eb05984/snapshotter/process_hub_core.py#L266-L278

• Send reports on crashed callback workers as well as complete shutdown of running snapshotter instances

https://github.com/PowerLoom/pooler/blob/bcc245d228acce504ba803b9b50fd89c8eb05984/snapshotter/process_hub_core.py#L186-L199

https://github.com/PowerLoom/pooler/blob/bcc245d228acce504ba803b9b50fd89c8eb05984/snapshotter/process_hub_core.py#L147-L166

Deployment Instructions

Pull the latest code and restart.

Ensure the issue reporting service URL in config/settings.json is set the base URL of the reporting service entry point of onchain-consensus

https://github.com/PowerLoom/pooler/blob/bcc245d228acce504ba803b9b50fd89c8eb05984/config/settings.example.json#L56-L59