Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Traefik

Kolin edited this page Nov 6, 2025 · 2 revisions

Traefik Configuration Guide for LogLynx

This guide provides recommended Traefik configurations optimized for LogLynx log ingestion and analysis.

Table of Contents

  1. Quick Start
  2. JSON Format Configuration (Recommended)
  3. CLF Format Configuration
  4. Access Log Settings
  5. Performance Tuning
  6. Troubleshooting

Quick Start

For immediate setup with LogLynx, add this to your Traefik configuration:

Docker Compose Example

version: '3.8'

services:
  traefik:
    image: traefik:v2.10
    command:
      - "--accesslog=true"
      - "--accesslog.filepath=/logs/access.log"
      - "--accesslog.format=json"
      - "--accesslog.bufferingsize=100"
    volumes:
      - ./logs:/logs
      - /var/run/docker.sock:/var/run/docker.sock:ro

Static Configuration File (traefik.yml)

accessLog:
  filePath: "/var/log/traefik/access.log"
  format: json
  bufferingSize: 100

JSON Format Configuration (Recommended)

LogLynx works best with Traefik's JSON format, as it captures all available fields with maximum precision.

Full Configuration Example

# traefik.yml (static configuration)
accessLog:
  # File path
  filePath: "/var/log/traefik/access.log"

  # Format: json is highly recommended for LogLynx
  format: json

  # Buffer size (number of log lines to buffer before writing)
  # Higher values = better performance, but potential data loss on crash
  # Recommended: 100-1000 for production
  bufferingSize: 100

  # Fields to log (optional - logs all fields by default)
  fields:
    defaultMode: keep

    # Additional headers to capture (optional)
    headers:
      defaultMode: keep
      names:
        User-Agent: keep
        Referer: keep
        X-Forwarded-For: keep
        X-Forwarded-Proto: keep
        X-Real-Ip: keep

What Fields Are Captured

When using JSON format, LogLynx captures:

Client Information:

  • ClientAddr → ClientIP + ClientPort
  • ClientHost → ClientHostname
  • ClientUsername → ClientUser

Request Information:

  • RequestMethod → Method
  • RequestProtocol → Protocol
  • RequestHost → Host
  • RequestPath → Path
  • RequestContentSize → RequestLength
  • request_X-Forwarded-Proto → RequestScheme

Response Information:

  • DownstreamStatus → StatusCode
  • DownstreamContentSize → ResponseSize
  • downstream_Content-Type → ResponseContentType

Timing Information:

  • Duration → Duration (nanoseconds)
  • StartUTC → StartUTC (nanosecond precision timestamp)
  • RequestTime → ResponseTimeMs

Proxy/Backend Information:

  • ServiceName → BackendName
  • ServiceURL → BackendURL
  • RouterName → RouterName
  • OriginStatus → UpstreamStatus
  • OriginDuration → UpstreamResponseTimeMs
  • origin_Content-Type → UpstreamContentType
  • RetryAttempts → RetryAttempts

Example JSON Log Entry

{
  "ClientAddr": "103.4.250.66:48952",
  "ClientHost": "103.4.250.66",
  "ClientPort": "48952",
  "ClientUsername": "-",
  "DownstreamContentSize": 20,
  "DownstreamStatus": 200,
  "Duration": 8528869,
  "OriginDuration": 8000000,
  "OriginStatus": 200,
  "Overhead": 528869,
  "RequestAddr": "example.com",
  "RequestContentSize": 0,
  "RequestCount": 49384,
  "RequestHost": "example.com",
  "RequestMethod": "GET",
  "RequestPath": "/_app/version.json",
  "RequestPort": "-",
  "RequestProtocol": "HTTP/2.0",
  "RequestScheme": "https",
  "RetryAttempts": 0,
  "RouterName": "llm@file",
  "ServiceAddr": "192.168.241.152:8181",
  "ServiceName": "llm@file",
  "ServiceURL": "http://192.168.241.152:8181/",
  "StartLocal": "2025-11-06T22:06:44.123456789+00:00",
  "StartUTC": "2025-11-06T22:06:44.123456789Z",
  "downstream_Content-Type": "application/json",
  "origin_Content-Type": "application/json",
  "request_User-Agent": "Mozilla/5.0...",
  "request_X-Forwarded-For": "103.4.250.66",
  "request_X-Forwarded-Proto": "https"
}

CLF Format Configuration

If you prefer Common Log Format (CLF), LogLynx supports it but with reduced field capture.

Configuration

# traefik.yml
accessLog:
  filePath: "/var/log/traefik/access.log"
  format: common  # CLF format
  bufferingSize: 100

Example CLF Log Entry

192.168.241.1 - - [06/Nov/2025:22:06:44 +0000] "GET /_app/version.json HTTP/2.0" 200 20 "-" "-" 49384 "llm@file" "http://192.168.241.152:8181/" 9ms

CLF Field Mapping

CLF Position Field LogLynx Field
1 Client IP ClientIP
2 Ident (not used)
3 Username ClientUser
4 Timestamp Timestamp
5 Method + Path + Protocol Method, Path, Protocol
6 Status Code StatusCode
7 Response Size ResponseSize
8 Referer Referer
9 User-Agent UserAgent
10 Request Count (not used)
11 Router Name RouterName
12 Service URL BackendURL
13 Duration Duration (converted ms→ns)

Note: CLF format does not capture:

  • StartUTC (constructed from timestamp, second precision only)
  • RetryAttempts
  • UpstreamStatus
  • Content-Type headers
  • Client hostname
  • Request scheme
  • Nanosecond precision timing

Recommendation: Use JSON format for complete field capture.


Access Log Settings

File Path Recommendations

Docker/Container Environments:

accessLog:
  filePath: "/logs/access.log"

Mount /logs volume to host for LogLynx ingestion.

Bare Metal/VM:

accessLog:
  filePath: "/var/log/traefik/access.log"

Ensure LogLynx has read permissions.

Kubernetes:

accessLog:
  filePath: "/dev/stdout"  # Send to stdout for log aggregator

Use log shipper (Fluentd/Fluent Bit) to forward to LogLynx.

Buffering Configuration

accessLog:
  bufferingSize: 100  # Number of log lines to buffer

Buffer Size Guidelines:

Environment Buffer Size Reason
Development 10-50 Fast visibility, frequent writes
Production (Low Traffic) 100-500 Balance performance/safety
Production (High Traffic) 500-1000 Maximum performance
Mission Critical 0 (disabled) Immediate writes, no loss risk

Trade-offs:

  • Higher buffer = Better performance, potential data loss on crash
  • Lower buffer = More disk I/O, safer against crashes
  • No buffer (0) = Real-time writes, highest I/O overhead

Filtering (Optional)

Exclude health checks or internal traffic:

accessLog:
  filePath: "/var/log/traefik/access.log"
  format: json
  filters:
    statusCodes:
      - "200-299"  # Only successful requests
    retryAttempts: true  # Only requests with retries
    minDuration: "10ms"  # Only slow requests

Warning: LogLynx performs its own filtering. Using Traefik filters may limit analysis capabilities.


Performance Tuning

High-Traffic Environments

For environments with >1000 req/s:

accessLog:
  filePath: "/var/log/traefik/access.log"
  format: json
  bufferingSize: 1000  # Large buffer

  # Optional: Reduce logged fields
  fields:
    defaultMode: keep
    headers:
      defaultMode: drop  # Don't log all headers
      names:
        User-Agent: keep
        Referer: keep

Disk I/O Optimization

Mount log volume on fast storage:

volumes:
  - type: bind
    source: /mnt/ssd/traefik-logs  # SSD storage
    target: /logs

Or use tmpfs (RAM) for extreme performance:

volumes:
  - type: tmpfs
    target: /logs
    tmpfs:
      size: 1G  # 1GB RAM disk

Warning: tmpfs logs are lost on restart. Use log shipper to forward immediately.

LogLynx Integration Settings

Configure LogLynx source for optimal performance:

# LogLynx config.yaml
sources:
  - name: "traefik-prod"
    type: "traefik"
    path: "/var/log/traefik/access.log"
    format: "json"

    # Polling interval (how often to check for new logs)
    poll_interval: "1s"  # Default

    # Batch size (how many log lines to process at once)
    batch_size: 1000  # Default

    # Buffer size (internal LogLynx buffer)
    buffer_size: 10000  # Default

Performance Guidelines:

Traffic Level poll_interval batch_size buffer_size
Low (<100 req/s) 5s 500 5000
Medium (100-500 req/s) 2s 1000 10000
High (500-2000 req/s) 1s 2000 20000
Very High (>2000 req/s) 500ms 5000 50000

Troubleshooting

Problem: Logs Not Appearing in LogLynx

Symptoms: LogLynx shows no new logs, but Traefik is receiving traffic

Checks:

  1. Verify log file exists:

    ls -lh /var/log/traefik/access.log
  2. Check file permissions:

    # LogLynx needs read permission
    chmod 644 /var/log/traefik/access.log
  3. Verify log format:

    head -n 1 /var/log/traefik/access.log

    Should see JSON if format: json is configured.

  4. Check Traefik is writing logs:

    tail -f /var/log/traefik/access.log

    Send a test request and verify new log line appears.

  5. Verify LogLynx source configuration:

    sources:
      - name: "traefik-prod"
        type: "traefik"
        path: "/var/log/traefik/access.log"  # Must match actual path
        format: "json"  # Must match Traefik config

Problem: High CPU Usage

Symptoms: LogLynx or Traefik consuming excessive CPU

Solutions:

  1. Increase buffer size (Traefik writes less frequently):

    accessLog:
      bufferingSize: 1000  # Increase from 100
  2. Reduce LogLynx polling frequency:

    sources:
      - name: "traefik-prod"
        poll_interval: "5s"  # Increase from 1s
  3. Use batch processing:

    sources:
      - name: "traefik-prod"
        batch_size: 5000  # Increase from 1000

Problem: High Disk I/O

Symptoms: Disk at 100% utilization, slow log processing

Solutions:

  1. Enable log compression (logrotate):

    /var/log/traefik/access.log {
        compress
        delaycompress
    }
    
  2. Use faster storage (SSD instead of HDD)

  3. Reduce logged fields:

    accessLog:
      fields:
        headers:
          defaultMode: drop  # Don't log all headers
  4. Use tmpfs for temporary log storage:

    volumes:
      - type: tmpfs
        target: /logs
        tmpfs:
          size: 1G

Problem: Missing Fields in LogLynx

Symptoms: Some fields show empty in LogLynx dashboard

Checks:

  1. Verify JSON format (not CLF):

    accessLog:
      format: json  # Not "common"
  2. Check Traefik version (older versions may not have all fields):

    traefik version
    # Recommended: v2.10 or later
  3. Verify field logging is enabled:

    accessLog:
      fields:
        defaultMode: keep  # Not "drop"
  4. Check raw log contains field:

    cat /var/log/traefik/access.log | jq '.[0]'

Problem: Log File Growing Too Large

Symptoms: Disk space filling up, large log files

Solutions:

  1. Enable log rotation:

    /var/log/traefik/access.log {
        daily
        rotate 30
        size 100M
    }
    
  2. Reduce retention period:

    rotate 7  # Keep only 7 days instead of 30
    
  3. Use filtering (Traefik side):

    accessLog:
      filters:
        statusCodes:
          - "400-599"  # Only log errors
  4. Archive old logs:

    # Compress logs older than 7 days
    find /var/log/traefik/ -name "*.log" -mtime +7 -exec gzip {} \;

Problem: Duplicate Log Entries

Symptoms: Same request appears multiple times in LogLynx

Causes & Solutions:

  1. Multiple LogLynx instances reading same log:

    • Use single instance
    • Or configure deduplication properly
  2. Log file was re-imported:

    • LogLynx's deduplication system should handle this
    • Check hash is calculated correctly (9 fields)
  3. Hash collision (extremely rare):

    • Check if requests are actually different
    • SHA256 collision probability: ~0% for practical purposes

Best Practices Summary

DO:

  • Use JSON format for maximum field capture
  • Enable buffering (100-1000) for better performance
  • Set up log rotation to manage disk space
  • Use SSD storage for log files in high-traffic environments
  • Monitor disk space and I/O utilization
  • Test configuration with sample traffic before production
  • Set correct file permissions (644 for log files)
  • Use log shipping in distributed deployments

DON'T:

  • Don't use CLF format unless necessary (fewer fields)
  • Don't disable buffering in high-traffic environments (performance impact)
  • Don't filter logs at Traefik level (limits LogLynx analysis)
  • Don't use HDD storage for high-traffic logs (I/O bottleneck)
  • Don't forget log rotation (disk will fill up)
  • Don't mix JSON and CLF formats in same file
  • Don't manually edit log files (corrupts LogLynx tracking)
  • Don't use world-writable permissions (security risk)

Additional Resources


Last Updated: November 2025

LogLynx Wiki

Getting Started

Deployment

Reverse proxy recommended configuration

Key features

Side features

Quick Links

API Endpoints

Configuration

Deployment

Support

Clone this wiki locally