oy-scrape-jitter

usage: oy-scrape-jitter [<flags>]


Flags:
  -h, --[no-]help                Show context-sensitive help (also try
                                 --help-long and --help-man).
      --metric="up"              Metric to use to determine jitter.
      --query.timestamp=QUERY.TIMESTAMP  
                                 Timestamp of the query.
      --plot.file=file.png       Path to a file to write an image of the
                                 results.
      --[no-]plot.log-y          Use logarithmic Y axis.
      --lookback=1h              How much time to look in the past for scrapes.
      --divisor=1s               Divisor to use to determine if a scrape is
                                 aligned.
      --[no-]log.unaligned-only  Only take unaligned targets in logging.
      --[no-]plot.unaligned-only  
                                 Only take unaligned targets in plot.
      --[no-]log.results-only    Only log final result.
      --prometheus.url="http://127.0.0.1:9090"  
                                 URL of the Prometheus server.
      --client.config=CLIENT.CONFIG  
                                 Path to a HTTP client configuration.
      --log.level=info           Only log messages with the given severity or
                                 above. One of: [debug, info, warn, error]
      --log.format=logfmt        Output format of log messages. One of: [logfmt,
                                 json]
      --[no-]version             Show application version.

oy-scrape-jitter queries a Prometheus server to see how regular scrapes are. Perfect scrape alignment happens when the distance between all the scrapes are exactly the same. It enables Prometheus delta-of-delta encoding to reduce significantly the size of the blocks.

$ ./oy-scrape-jitter --prometheus.url=https://prometheus.demo.do.prometheus.io/ --log.results-only
ts=2022-04-26T12:11:34.659Z caller=main.go:117 level=info msg="overall results" aligned_targets=0 unaligned_targets=10 max_ms=25

This means that the maximum deviation seen in your scrape jobs is 25ms. You could set --scrape.timestamp-tolerance=25ms to reduce your disk usage over time, by enabling Prometheus to correct timestamps up to 25ms.

Prometheus will only apply timestamp tolerance up to 1%. If your scrape interval is 30s, you can only adjust timestamps up to 300ms. Setting a 500ms tolerance will have no effects on jobs with a scrape interval lower than 500s, even if the deviation is tiny.

By using --plot.file=scrape.png, you can generate a PNG file which shows the scrape timestamps jitter with an histogram.

wget https://github.com/o11ydev/oy-toolkit/releases/download/main/oy-scrape-jitter -O oy-scrape-jitter && chmod +x oy-scrape-jitter && ./oy-scrape-jitter --help

docker run quay.io/o11y/oy-toolkit:oy-scrape-jitter --help

nix run github:o11ydev/oy-toolkit#oy-scrape-jitter -- --help