Navi

Cache Warmer Tool

Current Version: 1.11.1

Next Release: 1.11.2

Client Current Version: 0.2.2

Client Next Version: 0.2.3

Worker Current Version: 1.9.0

Worker Next Version: 1.9.1

Tagging a navi release now automatically publishes deku-swarm to npm and pushes a matching worker-x.y.z tag whenever worker/ changed since the last worker release (or force_worker_build was explicitly set for that pipeline run) and that version isn't already published — no separate manual worker release step is needed anymore. Use bump_version.sh worker [version] beforehand to bump the worker version and the badges above. force_worker_build is a CircleCI pipeline parameter that can only be set by explicitly triggering a pipeline (UI/API "Trigger Pipeline"), not by a normal tag push.

---

Table of Contents

• Overview
• Quick Start
• Configuration File
• Running Navi
• Development
• Running Tests
• Roadmap
• Installation
• How to Use Navi in Your Project

---

Overview

Navi is a queue-based cache-warmer written in Node.js and distributed as a Docker image. It reads a YAML configuration file, enqueues HTTP requests as jobs, and processes them concurrently using a configurable pool of workers.

Key features:

• Concurrent HTTP request execution via a worker pool.
• URL templates with placeholder parameters (e.g. {:id}).
• Response-driven actions: after each successful request, configurable actions extract variables from the response and trigger follow-up processing.
• Paginated resource support: paginated_actions fan out one request per page based on a page-count expression evaluated against the response.
• Automatic retry of failed requests after the main queue is exhausted.
• Config splitting: split resources/clients across multiple files with top-level include/namespace keys, with validated cross-namespace references. See How to Use Navi in Your Project for details.

---

Quick Start

The darthjee/navi-hey Docker image ships with a minimal, production-ready configuration (web — see dockerfiles/production_navi_hey/config/web.yml) baked in, so it works out of the box with zero volume mounts:

docker run -p 3000:3000 darthjee/navi-hey:latest

This brings up the monitoring web UI immediately at http://localhost:3000, and it stays up indefinitely (no auto-shutdown). The packed config declares no resources:/clients: — add those afterwards through the Navi client/API. Every setting in the packed config is overridable via an environment variable, without editing or rebuilding the image:

Env var	Default	Config field
NAVI_CONFIG	./config/web.yml	Path to the config file navi-hey loads (-c/--config). Selects which packed config runs.
PORT	3000	web.port
LOGS_PAGE_SIZE	20	web.logs_page_size
ENABLE_SHUTDOWN	false	web.enable_shutdown
AUTOSTART	true	web.autostart
IDLE_TIMEOUT	0 (disabled)	web.idle_timeout
API_TOKEN	empty (disabled)	web.api.token
WORKERS	1	workers.quantity
RETRY_COOLDOWN	2000	workers.retry_cooldown
WORKERS_SLEEP	500	workers.sleep
MAX_RETRIES	3	workers.max-retries

A few loader/CLI controls (not packed-config fields) round out the surface:

Env var	Default	Meaning
NAVI_EXTENSIONS_ENABLED	unset (off)	Load extra backend routes + frontend pages from the extensions mount. Truthy = 1/true/yes/on.
NAVI_EXTENSIONS_DIR	/navi/extensions	Mount point scanned for backend/ and frontend/ subtrees.
NAVI_MENU	./config/menu.yml	Menu config file (-m / --menu).

Extending Navi

Add your own backend routes and dashboard pages on top of the stock image without forking it — build a small project, mount its dist/ folder, and set NAVI_EXTENSIONS_ENABLED=true. See Extending Navi with Your Own Routes and Pages, and Configuring the Internal Navigation Menu to customise the nav menu.

For example, to change the exposed port:

docker run -p 8080:8080 -e PORT=8080 darthjee/navi-hey:latest

To bring your own full configuration (resources, clients, and more) instead, see Custom Configuration below.

---

Configuration File

Navi is configured via a YAML file that defines HTTP clients, resources, and the worker pool size. See the configuration schema for the full field-by-field reference.

workers:
  quantity: 5          # number of concurrent workers (default: 1)

log:
  size: 100            # max number of log entries kept in memory (default: 100)

web:
  port: 3000           # port for the monitoring web UI (omit to disable)

clients:
  default:
    base_url: https://example.com

resources:
  categories:
    - url: /categories.json
      status: 200

Custom Configuration

To bring your own full configuration (with resources:/clients: of your own) instead of the packed zero-config default (see Quick Start), mount the YAML file as a volume:

docker run --rm \
  -v /path/to/your/config.yml:/home/node/app/config/navi_config.yml \
  darthjee/navi-hey:latest \
  node navi.js config/navi_config.yml

In the development environment the config file lives at docker_volumes/config/navi_config.yml and is automatically mounted into the container.

---

Installation

Via npm / yarn (npx)

No installation required — run Navi directly using npx:

npx navi-hey --config /path/to/your/config.yml

Or install globally:

# npm
npm install -g navi-hey

# yarn
yarn global add navi-hey

Then run:

navi-hey --config /path/to/your/config.yml

Note: The web UI frontend is bundled directly with the Navi package and served from source/static/. After making changes to the frontend code, run yarn build inside the navi_frontend Docker Compose service to update the bundled assets.

Docker

See the Running Navi section below.

---

Running Navi

Docker (recommended)

1. Build the production image:

   make build

2. Run Navi with the packed zero-config default — see Quick Start:

   docker run -p 3000:3000 darthjee/navi-hey:latest

   Or with your own configuration file — see Custom Configuration:

   docker run --rm \
     -v /path/to/your/config.yml:/home/node/app/config/navi_config.yml \
     darthjee/navi-hey:latest \
     node navi.js config/navi_config.yml

Local execution (Node.js)

Requires Node.js (see source/package.json for the engine version).

cd source
yarn install
node ../navi.js /path/to/your/config.yml

---

Development

The development workflow is Docker-based. Docker and Docker Compose must be installed.

First-time setup

make setup

This command:

1. Copies .env.sample to .env.
2. Copies docker_volumes/config/navi_config.yml.sample to docker_volumes/config/navi_config.yml (if it does not already exist).
3. Builds the base_build Docker image.
4. Installs Node.js dependencies inside the container via yarn install.

Note: Always use Yarn to manage dependencies. Do not use npm install.

Starting a development shell

make dev

This opens an interactive Bash shell inside the navi_app container, where you can run yarn test, yarn lint, and other commands.

Available Makefile commands

Command	Description
make setup	First-time environment setup (copies .env and config.yml from samples, builds image, installs deps).
make dev	Opens a shell in the navi_app container.
make tests	Opens a shell in the isolated navi_tests container.
make build-dev	Builds the development Docker image (navi:dev).
make build	Builds the production Docker image (darthjee/navi-hey:latest).
make build-client	Builds the production client Docker image (darthjee/navi-hey-client:latest).

---

Running Tests

Tests are written with Jasmine and use c8 for code coverage.

Inside the development container (via make dev or make tests):

yarn test    # run tests with coverage report
yarn lint    # run ESLint
yarn report  # run copy/paste analysis (JSCPD)
yarn docs    # generate JSDoc API documentation

---

Actions & Response Chaining

After a successful HTTP response, Navi executes each configured action for every item in the response body. If the body is a JSON array, each action runs once per element; if it is a single object, each action runs once.

For each action, the parameters map is applied to the response wrapper to produce a set of named variables:

• With parameters: each value is a path expression (e.g. parsedBody.id, headers['page']) resolved against a wrapper exposing the parsed JSON body and response headers. Only the explicitly mapped fields are included.
• Without parameters: the parsed body item is passed through unchanged.

The mapped variables are then used to resolve {:placeholder} tokens in the target resource's URL templates. For example, if the response body contains { "id": 1 } and the action has parameters: { id: parsedBody.id }, the target resource's URL /categories/{:id}.json resolves to /categories/1.json. Header values can also be extracted, e.g. page: headers['page'].

Note: HTTP response header names are always lowercase after Node.js normalization. Use lowercase keys in path expressions (e.g. headers['x-total-pages']), regardless of how the server set them.

Each action is enqueued as an ActionProcessingJob, which looks up the target resource, creates a ResourceRequestJob for each URL entry in that resource with the resolved parameters, and enqueues them for processing by the worker pool. This enables multi-level resource chaining — a response can trigger further requests whose responses trigger even more requests.

Error handling: an action whose resource field is missing is skipped and logged. An action whose path expression cannot be resolved against the response is also skipped and logged. Other actions continue normally. A response body that is not valid JSON raises an error for the whole request.

---

Paginated Actions

When a resource response indicates multiple pages, paginated_actions fan out one downstream ResourceRequestJob per page. This is configured alongside (or instead of) actions in a resource definition.

Each paginated_action entry specifies a resource and a pagination block:

• pages — a path expression evaluated against the response (e.g. parsedBody.pagination.pages) to determine the total page count.
• page_key — the parameter name injected as the current page number into each downstream request.
• zero_indexed — boolean; when true pages run from 0 to pages-1; when false (default) from 1 to pages.

Optionally, each paginated_action entry may also specify a parameters map (same syntax as actions[].parameters) — path expressions resolved against the same response used for pages, letting you forward server-reported metadata (e.g. a per_page value from a response header) into every paginated request without a separate fetch.

Each downstream request's final parameter set is built in this order (later wins on a key collision):

1. Parameters inherited from the parent chain (existing behavior).
2. The resolved parameters map, when present — overrides same-named inherited parameters.
3. page_key's page number — always wins, even over a same-named parameters entry.

So {:page} (or whatever page_key is set to) can be used as a {:placeholder} in the target resource's URL template alongside other variables (e.g. {:category_id}, or any key from parameters). If a parameters path expression can't be resolved against the response, that one paginated action fails in isolation (no pages enqueued, logged, dead-lettered, no retry) — everything else in the run is unaffected.

Each paginated action is enqueued as a PaginatedActionProcessingJob. Unlike ActionProcessingJob, it operates on the whole response wrapper (not individual array items) and has no retry rights.

Example

resources:
  categories:
    - url: /categories.json
      status: 200
      paginated_actions:
        - resource: products_page
          pagination:
            - pages: parsedBody.pagination.pages
            - page_key: page
            - zero_indexed: false
          parameters:
            per_page: headers['x-per-page']
  products_page:
    - url: /products/{:page}.json?per_page={:per_page}
      status: 200

If the /categories.json response contains { "pagination": { "pages": 3 } } with a X-Per-Page: 25 response header, Navi enqueues three jobs for /products/1.json?per_page=25, /products/2.json?per_page=25, and /products/3.json?per_page=25.

Capping pages with max_page

Sometimes you don't want to warm every page a paginated_actions caller reports — just the first few, most-likely-to-be-hit ones. max_page caps this from the target resource's side, independent of who calls it:

resources:
  categories:
    - url: /categories.json
      status: 200
      paginated_actions:
        - resource: products_page
          pagination:
            - pages: parsedBody.pagination.pages
            - page_key: page
            - zero_indexed: false
          parameters:
            per_page: headers['x-per-page']
  products_page:
    - url: /products/{:page}.json?per_page={:per_page}
      status: 200
      max_page: 2

Even though /categories.json reports 3 pages, products_page caps itself at 2 — Navi enqueues only /products/1.json?per_page=25 and /products/2.json?per_page=25. max_page is a property of products_page itself: every caller that fans out into it is capped the same way, not just this one. It counts pages, not page numbers, so it composes the same way regardless of zero_indexed (a zero_indexed: true caller capped at max_page: 2 would enqueue pages 0 and 1, not 1 and 2).

Omitted, null, 0, or any other non-positive-integer value means unlimited (all pages the caller resolves are enqueued) — the default. A present-but-invalid value (e.g. a negative number or a non-numeric string) also logs a warning.

---

Data Extraction and Emission

After a successful response, a resource-request entry may optionally declare a parser and an emit. The parser extracts one or more structured items from the raw response body; emit then sends each extracted item as its own HTTP request to an external endpoint. This runs independently of (in parallel with) actions/paginated_actions chaining — a resource can have only actions, only parser/emit, or both at once, and neither path interferes with the other.

Three parser types are available, each producing the same shape of extracted item(s) regardless of which one is used:

• regex — applies a regular expression to the raw response body and captures a single field.
• json_path — navigates to an array within the parsed JSON body — or, when match is omitted, treats the response body's own root as that array — optionally filters it, and maps selected fields into each extracted item.
• css — applies a CSS selector to an HTML response body and maps selected fields (and/or attributes) into each extracted item.

See the Configuration File Fields table for the full field-by-field breakdown of parser and emit.

Example: json_path extraction with emit

The loot_catalog resource below fetches a miniature catalog, extracts every miniature-typed item from its bundleObjs array, and posts each one to the majora_api client:

clients:
  lootstudios:
    base_url: https://app.lootstudios.com
  majora_api:
    base_url: https://majora.example.com
    headers:
      Authorization: Bearer 

resources:
  loot_catalog:
    - url: /wp-admin/admin-ajax.php?action=GetMyLootsCache
      status: 200
      client: lootstudios
      parser:
        type: json_path
        match: bundleObjs
        filter:
          - field: obj_type
            equals: miniature
        fields:
          obj_inid: inid
          obj_title: name
          obj_post_id: post_id
          bnd_title: bundle
      emit:
        client: majora_api
        method: POST
        url: /api/miniatures
        headers:
          Authorization: Bearer $MAJORA_API_TOKEN

For each of the 28 matched items, Navi enqueues one POST https://majora.example.com/api/miniatures request with a body built from the mapped fields ({ inid, name, post_id, bundle }).

Root-level array: when the response body is itself a JSON array (e.g. [ { "obj_type": "miniature", … }, … ]) rather than an object with a bundleObjs wrapper key, omit match entirely — match: '' and match: '.' are accepted aliases for the same thing. fields stays required and filter still applies, exactly as for a nested path; a non-array body raises the same "did not resolve to an array" error as a bad nested path. regex and css still require match.

resources:
  loot_catalog:
    - url: /wp-admin/admin-ajax.php?action=GetMyLootsCache
      status: 200
      client: lootstudios
      parser:
        type: json_path
        # match omitted — the whole response body is the array of items
        filter:
          - field: obj_type
            equals: miniature
        fields:
          obj_inid: inid
          obj_title: name
          obj_post_id: post_id
          bnd_title: bundle
      emit:
        client: majora_api
        method: POST
        url: /api/miniatures

Example: regex standalone

A parser doesn't need json_path's nested fields/filter — a regex parser captures a single field directly from the raw body, useful for pulling a value out of HTML that isn't itself JSON:

resources:
  bundle_page:
    - url: /bundle/tidal-aberrations/?logged-in
      status: 200
      client: lootstudios
      parser:
        type: regex
        match: 'postid-(\d+)'
        field: post_id
      emit:
        client: majora_api
        method: POST
        url: /api/bundles/resolve

Here the regex postid-(\d+) captures 880433 out of the response body's postid-880433 class name, and Navi enqueues POST https://majora.example.com/api/bundles/resolve with { "post_id": "880433" }.

Emit retry policy

Each EmitJob retries independently of the global workers.max-retries/workers.retry_cooldown policy: 5 retries, 5000ms cooldown between attempts by default, since external endpoints are more likely to be transiently flaky than Navi's own crawl targets. Override either value per resource via emit.retries/emit.cooldown (see the Fields table above). An EmitJob retries on any 5xx, 429, 408, or network-level (no response) failure; any other 4xx dead-letters immediately, since those represent bad requests/config/auth issues that won't resolve by waiting. A 429 response honors a Retry-After header (capped at 60 seconds) instead of the normal cooldown; a malformed or missing value falls back to the normal cooldown.

Tracking extraction and emission

Navi tracks every extraction run and emission attempt in memory, exposed through two unauthenticated GET endpoints alongside the monitoring web UI:

• GET /extractions.json — returns { counts, extractions }. counts is { extracted }, the monotonic total number of items produced across every ExtractionJob run (exact even past ring-buffer eviction). extractions is a page of records — one per ExtractionJob run, not per item — each shaped { id, parserType, originUrl, itemCount, timestamp }.
• GET /emissions.json — returns { counts, emissions }. counts is { extracted, emitted, failed, dead }. emissions is a page of records, each shaped { id, extractionId, status, url, method, httpStatus, error, itemRef, timestamp }, where status is one of success/failed/dead and extractionId links back to the GET /extractions.json record whose items produced it (null when it can't be traced).

Both endpoints page with a ?last_id=<id> cursor and cap each page at web.logs_page_size records (default 20, shared with /logs.json), ordered oldest-first. Their underlying ring buffers are sized independently via the top-level emit.size / extraction.size config keys (default 100 each, see the Fields table above); the counters themselves stay exact for the whole run regardless of ring-buffer eviction. GET /stats.json also summarizes the same counters under its emissions key. All of this data resets when the engine stops, the same as the log buffers.

See docs/agents/future/crawler/flows.md for further worked examples, including how extraction/emit interacts with paginated_actions.

See it live

The public navi-hey demo runs a live crawl-and-emit example alongside its cache-warming: while crawling the Oak application it extracts data from four resources — one per parser type — and emits every item to a collector client backed by a logging POST /collector/:source endpoint on the demo app.

• oak_categories — json_path over the bare-array GET /categories.json → POST /collector/oak-categories.
• oak_paginated_category_items — json_path + body_template, once per page → POST /collector/oak-category-items/{category_slug}?page={page}.
• oak_home — css over the SPA shell's <head> <link> tags → POST /collector/oak-home.
• oak_templates — regex capturing the hashed JS bundle name → POST /collector/oak-templates.

Watch it on the demo's Extractions and Emissions dashboards; the full config is at dockerfiles/demo_navi_hey/navi-config.yml.

---

Roadmap

The following features are planned but not yet implemented:

• WorkersFactory — the factory responsible for instantiating Worker instances is planned but not yet implemented. Workers are currently initialized directly inside WorkersRegistry.

Web UI

Navi includes a built-in read-only monitoring web UI (built with React + React Bootstrap). Enable it by adding a web: section to your configuration file:

web:
  port: 3000
  autostart: true # optional, defaults to true

By default, the engine starts processing jobs immediately at boot. Setting autostart: false boots the web server without starting the engine — job processing stays paused until an operator calls PATCH /engine/start (optionally with a { "resources": [...] } body naming which configured resources to enqueue).

When enabled, the UI is accessible at http://localhost:<port> and includes the following screens:

Dashboard (/#/) — displays the real-time state of all job queues:

• Jobs currently in queue.
• Jobs being processed.
• Finished jobs.
• Failed jobs (with last failure reason).
• Dead jobs (exceeded retry limit).

Jobs list (/#/jobs) — shows a table of all jobs across every status, with links to each job's detail page.

Job detail (/#/job/:id) — shows the full details of a specific job (ID, status, and attempt count).

Memory status (/#/memory/status) — shows current process memory usage against the resolved maximum, color-coded by status:

• Current vs. maximum usage, formatted (e.g. 512 MB / 2 GB).
• Usage percentage.
• Status label (low/medium/high/over), colored per status — a distinct color when usage exceeds 100% of the maximum.

Extractions (/#/extractions) — shows a table of ExtractionJob runs (parser type, origin URL, item count, timestamp), each linked to the emissions it produced, with a running extracted total.

Emissions (/#/emissions) — shows a table of individual EmitJob emissions (status, target URL/method, HTTP status, error when failed/dead, linked extraction, timestamp), with running extracted/emitted/failed/dead totals.