Prometheus is an excellent systems monitoring and alerting toolkit, which uses a pull model for collecting metrics. The pull model is problematic when a firewall separates a Prometheus server and its metrics endpoints.

Prometheus Proxy enables Prometheus to scrape metrics endpoints running behind a firewall and preserves the native pull-based model architecture.

• Architecture
• Quick Start
• Building from Source
• Configuration Examples
• Docker Usage
• Advanced Features
• Monitoring & Observability
• Configuration Options
• Security & TLS
• Troubleshooting
• API Documentation
• License

The prometheus-proxy runtime comprises two services:

• proxy: runs in the same network domain as Prometheus server (outside the firewall) and proxies calls from Prometheus to the agent behind the firewall.
• agent: runs in the same network domain as all the monitored hosts/services/apps (inside the firewall). It maps the scraping queries coming from the proxy to the actual /metrics scraping endpoints of the hosts/services/apps.

Prometheus Proxy solves the firewall problem by using a persistent gRPC connection initiated from inside the firewall. Here's a simplified network diagram of how the deployed proxy and agent work:

Endpoints running behind a firewall require a prometheus-agent (the agent) to be run inside the firewall. An agent can run as a stand-alone server, embedded in another java server, or as a java agent. Agents connect to a prometheus-proxy (the proxy) and register the paths for which they will provide data. One proxy can work with one or many agents.

• 🌐 Proxy - Runs outside the firewall alongside Prometheus

  • HTTP server (port 8080) - Serves metrics to Prometheus
  • gRPC server (port 50051) - Accepts agent connections
  • Service discovery support for dynamic targets

• 🔗 Agent - Runs inside the firewall with monitored services

  • Connects to proxy via gRPC (outbound connection only)
  • Scrapes local metrics endpoints
  • Registers available paths with proxy

• ✅ Firewall-friendly - Only requires outbound connection from agent
• ✅ Preserves pull model - Prometheus continues to pull as normal
• ✅ High performance - Built with Kotlin coroutines and gRPC
• ✅ Secure - Optional TLS with mutual authentication
• ✅ Scalable - One proxy supports many agents
• ✅ Zero changes to existing Prometheus configuration patterns

Requirements: Java 17 or newer

1. Download the latest proxy and agent JAR files from releases

2. Start the proxy (runs outside the firewall with Prometheus):

   java -jar prometheus-proxy.jar

3. Start the agent (runs inside the firewall with your services):

   java -jar prometheus-agent.jar \
     -Dagent.proxy.hostname=mymachine.local \
     --config https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/myapps.conf
   
   # or use --proxy option
   java -jar prometheus-agent.jar \
     --proxy mymachine.local \
     --config https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/myapps.conf

4. Configure Prometheus to scrape from the proxy at http://mymachine.local:8080

5. Verify it works with:

   curl -s http://mymachine.local:8080/app1_metrics | head
   
   # or, if SD is enabled
   curl -s http://mymachine.local:8080/discovery | jq '.'

If you prefer to build the project from source:

1. Clone the repository:

   git clone https://github.com/pambrose/prometheus-proxy.git
   cd prometheus-proxy

2. Build the fat JARs:

   ./gradlew shadowJar

3. The JARs will be available in build/libs/:

• build/libs/prometheus-proxy.jar
• build/libs/prometheus-agent.jar

# Start proxy
docker run --rm -p 8080:8080 -p 50051:50051 pambrose/prometheus-proxy:3.0.0

# Start agent
docker run --rm \
  --env AGENT_CONFIG='https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/simple.conf' \
  pambrose/prometheus-agent:3.0.0

If the prometheus-proxy were running on a machine named mymachine.local and the agent.pathConfigs value in the myapps.conf config file had the contents:

agent {
  pathConfigs: [
    {
      name: "App1 metrics"
      path: app1_metrics
      labels: "{\"key1\": \"value1\", \"key2\": 2}"
      url: "http://app1.local:9100/metrics"
    },
    {
      name: "App2 metrics"
      path: app2_metrics
      labels: "{\"key3\": \"value3\", \"key4\": 4}"
      url: "http://app2.local:9100/metrics"
    },
    {
      name: "App3 metrics"
      path: app3_metrics
      labels: "{\"key5\": \"value5\", \"key6\": 6}"
      url: "http://app3.local:9100/metrics"
    }
  ]
}

then the prometheus.yml scrape_config would target the three apps with:

• http://mymachine.local:8080/app1_metrics
• http://mymachine.local:8080/app2_metrics
• http://mymachine.local:8080/app3_metrics

If the endpoints were restricted with basic auth/bearer authentication, you could either include the basic-auth credentials in the URL with: http://user:pass@hostname/metrics or they could be configured with basic_auth/ bearer_token in the scrape-config.

The prometheus.yml file would include:

scrape_configs:
  - job_name: 'app1 metrics'
    metrics_path: '/app1_metrics'
    bearer_token: 'eyJ....hH9rloA'
    static_configs:
      - targets: [ 'mymachine.local:8080' ]
  - job_name: 'app2 metrics'
    metrics_path: '/app2_metrics'
    basic_auth:
      username: 'user'
      password: 's3cr3t'
    static_configs:
      - targets: [ 'mymachine.local:8080' ]
  - job_name: 'app3 metrics'
    metrics_path: '/app3_metrics'
    static_configs:
      - targets: [ 'mymachine.local:8080' ]

The docker images support multiple architectures (amd64, arm64, s390x):

docker pull pambrose/prometheus-proxy:3.0.0
docker pull pambrose/prometheus-agent:3.0.0

Start a proxy container with:

# Proxy with admin and metrics enabled
docker run --rm -p 8082:8082 -p 8092:8092 -p 50051:50051 -p 8080:8080 \
        --env ADMIN_ENABLED=true \
        --env METRICS_ENABLED=true \
        --restart unless-stopped \
        pambrose/prometheus-proxy:3.0.0

Start an agent container with:

# Agent with remote config file
docker run --rm -p 8083:8083 -p 8093:8093 \
        --env AGENT_CONFIG='https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/simple.conf' \
        --restart unless-stopped \
        pambrose/prometheus-agent:3.0.0

Or use docker-compose: see etc/compose/proxy.yml for a working example.

Using the config file simple.conf, the proxy and the agent metrics would be available from the proxy on localhost at:

• http://localhost:8082/proxy_metrics
• http://localhost:8083/agent_metrics

If you want to use a local config file with a docker container (instead of the above HTTP-served config file), use the docker mount option. Assuming the config file prom-agent.conf is in your current directory, run an agent container with:

# Agent with local config file
docker run --rm -p 8083:8083 -p 8093:8093 \
    --mount type=bind,source="$(pwd)"/prom-agent.conf,target=/app/prom-agent.conf \
    --env AGENT_CONFIG=prom-agent.conf \
    pambrose/prometheus-agent:3.0.0

Note: The WORKDIR of the proxy and agent images is /app, so make sure to use /app as the base directory in the target for --mount options.

If you are running a JVM-based program, you can run with the agent embedded directly in your app and not use an external agent. This approach eliminates the need for a separate agent process when your application already runs on the JVM.

// Start embedded agent
EmbeddedAgentInfo agentInfo = startAsyncAgent("configFile.conf", true);

// Your application code runs here
// The agent runs in the background and does not block your application

// Shutdown the agent when the application terminates
agentInfo.close();

Enable Prometheus service discovery support:

java -jar prometheus-proxy.jar \
  --sd_enabled \
  --sd_path discovery \
  --sd_target_prefix http://proxy-host:8080/

Access discovery endpoint at: http://proxy-host:8080/discovery

Configure concurrent scraping:

java -jar prometheus-agent.jar \
  --max_concurrent_clients 5 \
  --client_timeout_secs 30 \
  --config myconfig.conf

Both proxy and agent expose their own metrics:

• Proxy metrics: http://proxy-host:8082/proxy_metrics
• Agent metrics: http://agent-host:8083/agent_metrics
• Admin endpoints: http://host:admin-port/ping, /healthcheck, /version

The proxy and agent use the Typesafe Config library. Configuration values are evaluated in order: CLI options → environment variables → config file values.

Typesafe Config highlights include:

• support for files in three formats: Java properties, JSON, and a human-friendly JSON superset (HOCON)
• config files can be files or urls
• config values can come from CLI options, environment variables, Java system properties, and/or config files.
• config files can reference environment variables

📖 Complete configuration reference: CLI Options & Environment Variables Reference

• Formats: Supports HOCON (.conf), JSON (.json), and Java Properties (.properties).
• Logging: Customize with -Dlogback.configurationFile=/path/to/logback.xml.
• Dynamic Props: Use -Dproperty.name=value for any configuration key.
• Keepalives: See the gRPC keepalive guide for tuning details.

• 🏢 Enterprise environments - Scrape metrics across firewall boundaries
• ☁️ Multi-cloud deployments - Bridge different network segments
• 🔒 Secure environments - Monitor internal services without opening inbound ports
• 🌐 Federation - Scrape existing Prometheus instances via /federate endpoint
• 🚀 Kubernetes - Monitor services across clusters or namespaces

Scrape an existing Prometheus instance via the /federate endpoint:

agent.pathConfigs: [{
  name: "Federated Prometheus"
  path: "federated_metrics"
  url: "http://prometheus-server:9090/federate?match[]={__name__=~\"job:.*\"}"
}]

This leverages the existing service discovery features already built into Prometheus.

Another service discovery example config can be found in federate.conf.

To use with Nginx as a reverse proxy, disable the transport filter on both Proxy and Agent:

java -jar prometheus-proxy.jar --tf_disabled
java -jar prometheus-agent.jar --tf_disabled --config myconfig.conf

An example nginx conf file is here, and an example agent/proxy conf file is here

⚠️ Note: With transportFilterDisabled, agent disconnections aren't immediately detected. Agent contexts on the proxy are removed after inactivity timeout (default: 1 minute, controlled by proxy.internal.maxAgentInactivitySecs).

gRPC Reflection is enabled by default for debugging and tooling.

Test with grpcurl:

# List available services
grpcurl -plaintext localhost:50051 list

# Describe a service
grpcurl -plaintext localhost:50051 describe io.prometheus.ProxyService

If you use grpcurl -plaintext option, make sure that you run the proxy in plaintext mode, i.e., do not define any TLS properties.

Disable reflection: Use --ref_disabled, REFLECTION_DISABLED, or proxy.reflectionDisabled=true.

Agents connect to a proxy using gRPC. gRPC supports TLS with or without mutual authentication. The necessary certificate and key file paths can be specified via CLI args, environment variables, and configuration file settings.

The gRPC docs describe how to set up TLS. The repo includes the certificates and keys necessary to test TLS support.

Running TLS without mutual authentication requires these settings:

• certChainFilePath and privateKeyFilePath on the proxy
• trustCertCollectionFilePath on the agent

Running TLS with mutual authentication requires these settings:

• certChainFilePath, privateKeyFilePath and trustCertCollectionFilePath on the proxy
• certChainFilePath, privateKeyFilePath and trustCertCollectionFilePath on the agent

Run a proxy and an agent with TLS (with mutual authentication) with:

# Proxy with TLS
java -jar prometheus-proxy.jar \
  --cert /path/to/server.crt \
  --key /path/to/server.key \
  --trust /path/to/ca.crt

# Agent with TLS
java -jar prometheus-agent.jar \
  --config myconfig.conf \
  --trust /path/to/ca.crt \
  --cert /path/to/client.crt \
  --key /path/to/client.key

Run a proxy and an agent with TLS (no mutual authentication) using the included testing certs and keys with:

java -jar prometheus-proxy.jar --config examples/tls-no-mutual-auth.conf

java -jar prometheus-agent.jar --config examples/tls-no-mutual-auth.conf

Run a proxy and an agent docker container with TLS (no mutual authentication) using the included testing certs and keys with:

docker run --rm -p 8082:8082 -p 8092:8092 -p 50440:50440 -p 8080:8080 \
    --mount type=bind,source="$(pwd)"/testing/certs,target=/app/testing/certs \
    --mount type=bind,source="$(pwd)"/examples/tls-no-mutual-auth.conf,target=/app/tls-no-mutual-auth.conf \
    --env PROXY_CONFIG=tls-no-mutual-auth.conf \
    --env ADMIN_ENABLED=true \
    --env METRICS_ENABLED=true \
    pambrose/prometheus-proxy:3.0.0

docker run --rm -p 8083:8083 -p 8093:8093 \
    --mount type=bind,source="$(pwd)"/testing/certs,target=/app/testing/certs \
    --mount type=bind,source="$(pwd)"/examples/tls-no-mutual-auth.conf,target=/app/tls-no-mutual-auth.conf \
    --env AGENT_CONFIG=tls-no-mutual-auth.conf \
    --env PROXY_HOSTNAME=mymachine.lan:50440 \
    --name docker-agent \
    pambrose/prometheus-agent:3.0.0

Note: The WORKDIR of the proxy and agent images is /app, so make sure to use /app as the base directory in the target for --mount options.

When Prometheus scrape configurations include basic_auth or bearer_token, the proxy forwards the Authorization header to the agent over the gRPC channel. If TLS is not configured, these credentials are transmitted in plaintext and could be intercepted on the network between the proxy and agent.

Enable TLS when forwarding auth headers:

# Proxy with TLS to protect forwarded credentials
java -jar prometheus-proxy.jar \
  --cert /path/to/server.crt \
  --key /path/to/server.key

# Agent with TLS
java -jar prometheus-agent.jar \
  --config myconfig.conf \
  --trust /path/to/ca.crt

The proxy logs a warning on the first request that includes an Authorization header when TLS is not enabled.

Disable SSL verification for agent https endpoints with the TRUST_ALL_X509_CERTIFICATES environment var, the --trust_all_x509 CLI option, or the agent.http.enableTrustAllX509Certificates property.

To scrape HTTPS endpoints with a self-signed certificate:

java -jar prometheus-agent.jar --trust_all_x509 --config myconfig.conf

⚠️ Security Note: Only use --trust_all_x509 in development/testing environments.

Agent can't connect to proxy:

• Verify proxy hostname and port
• Check firewall rules (agent needs outbound access to proxy port)
• Ensure proxy is running and listening

TLS connection failures:

• Verify certificate paths and file permissions
• Check certificate validity and chain
• Ensure clock synchronization between proxy and agent

Metrics not appearing:

• Verify agent path configuration matches Prometheus scrape paths
• Check agent logs for scraping errors
• Confirm target endpoints are accessible from agent

Performance issues:

• Increase max_concurrent_clients for high-throughput scenarios
• Tune HTTP client cache settings
• Consider running multiple agents for load distribution

KDoc API documentation is published at pambrose.github.io/prometheus-proxy.

This project is licensed under the Apache License 2.0 - see License.txt for details.

• Prometheus.io
• gRPC
• Typesafe Config