Readme
# LogUtil - Real-time Nginx Log Analyzer
Overview
LogUtil is a powerful, real-time log analysis tool written in Rust that provides an interactive terminal interface for monitoring and analyzing Nginx access logs. It offers comprehensive log parsing, real-time monitoring, and detailed analytics with a beautiful TUI (Terminal User Interface).
Key Features
🔍 Real-time Log Monitoring
Live tail functionality that monitors log files as they grow
Automatic detection and processing of new log entries
Real-time updates without manual refresh
📊 Comprehensive Analytics
IP Address Analysis : Track requests by IP addresses with detailed statistics
URL Analysis : Monitor most accessed URLs and their patterns
Request Type Tracking : Categorize requests by HTTP methods (GET, POST, etc.)
Domain Analysis : Track requests by domain names
Time-based Analytics : Analyze request patterns over time
🎨 Interactive TUI Interface
9 Main Tabs : Overview, Requests, Detailed, Sparkline, Heatmap, Security, Performance, Errors, and Bots
Overview Tab : Shows top IPs and URLs with real-time statistics
Requests Tab : Searchable log entries with pagination
Detailed Tab : Drill-down view for specific IP addresses
Sparkline Tab : Real-time request timeline visualization
Heatmap Tab : Hourly request patterns across multiple days
Security Tab : Detection of suspicious IPs and attack patterns
Performance Tab : Response time analysis and slow request identification
Errors Tab : HTTP error code analysis and categorization
Bots Tab : Bot and crawler detection and classification
Efficient memory management with automatic cleanup
Optimized for large log files (handles 10,000+ entries)
Minimal CPU usage during real-time monitoring
Configurable cleanup policies
🔧 Flexible Configuration
Customizable regex patterns for different log formats
Support for various date formats
Configurable top N entries display
Optional automatic cleanup of outdated entries
📁 Interactive File Selection Mode
When you run LogUtil without specifying a log file, it launches an interactive file selection mode:
File Selector Features:
📂 Directory Navigation : Browse through folders with intuitive navigation
📄 File Browser : View all files with clear icons (📁 for folders, 📄 for files, ⬆️ for parent directory)
🔍 Visual Selection : Highlighted current selection with clear visual feedback
⚡ Quick Access : Navigate with arrow keys and select with Enter
Settings Configuration:
After selecting a log file, you'll see an interactive settings screen where you can configure:
📊 Analysis Parameters : Enable/disable specific analysis tabs (Security, Performance, Errors, Bots, Sparkline, Heatmap)
🔧 Processing Options : Set count limits, regex patterns, date formats
⚙️ Real-time Settings : Configure cleanup policies and monitoring options
▶️ Start Analysis : Launch the analysis with your custom configuration
Usage:
# Launch interactive mode
logutil
# Or specify a file directly
logutil /path/to/access.log
🎮 Keyboard Shortcuts
Navigation:
Tab / T : Switch to next tab
Shift+Tab / Shift+T : Switch to previous tab
↑/↓ : Navigate through lists and options
←/→ : Switch between panels (in tabs with multiple panels)
Enter : Select item or toggle boolean values
Esc : Go back or cancel
File Selector:
↑/↓ : Navigate through files and folders
Enter : Select file or enter folder
Esc : Go back to parent directory or exit
Settings:
↑/↓ : Navigate through settings
Enter : Edit setting or toggle boolean values
Esc : Cancel editing or go back
TUI Controls:
Q / Ctrl+C : Quit application
Enter : Copy selected item to clipboard (Overview tab)
📋 Command Line Examples
Interactive mode with pre-configured parameters:
# Launch file selector with all analysis tabs enabled
cargo run "" --enable-security --enable-performance --enable-errors --enable-bots --enable-sparkline --enable-heatmap --count=1000
# Launch with custom settings
cargo run "" --enable-security --enable-performance --top=20 --count=500
# Launch with specific analysis tabs only
cargo run "" --enable-security --enable-errors --count=2000
Direct file analysis with all features:
# Analyze with all tabs enabled
logutil access.log --enable-security --enable-performance --enable-errors --enable-bots --enable-sparkline --enable-heatmap --count=1000
# Security-focused analysis
logutil access.log --enable-security --enable-errors --top=50
# Performance monitoring
logutil access.log --enable-performance --enable-sparkline --enable-heatmap
Installation
Quick Install (Recommended)
Download the latest release for your platform:
Linux (x86_64):
curl - L - o /usr/local/bin/logutil https://github.com/s00d/logutil/releases/latest/download/logutil-x86_64-unknown-linux-gnu
chmod +x /usr/local/bin/logutil
Linux (ARM64):
curl - L - o /usr/local/bin/logutil https://github.com/s00d/logutil/releases/latest/download/logutil-aarch64-unknown-linux-gnu
chmod +x /usr/local/bin/logutil
macOS:
curl - L - o /usr/local/bin/logutil https://github.com/s00d/logutil/releases/latest/download/logutil-x86_64-apple-darwin
chmod +x /usr/local/bin/logutil
Build from Source
Install Rust:
curl -- proto ' =https' -- tlsv1 .2 - sSf https://sh.rust-lang.org | sh
Clone and Build:
git clone https://github.com/s00d/logutil.git
cd logutil
cargo build -- release
Install:
sudo cp target/release/logutil /usr/local/bin/
Usage
Basic Usage
Monitor a log file in real-time:
logutil /var/log/nginx/access.log
Analyze the entire log file:
logutil /var/log/nginx/access.log -- count= -1
Show only the last 1000 lines:
logutil /var/log/nginx/access.log -- count= 1000
Advanced Usage
Custom regex pattern:
logutil /var/log/nginx/access.log -- regex= ' ^(\S+) - - \[(.*?)\] "(\S+) (\S+) HTTP/\d+\.\d+" (\d+) (\d+) "([^"]*)" "([^"]*)"$'
Show top 20 entries:
logutil /var/log/nginx/access.log -- top= 20
Disable automatic cleanup:
logutil /var/log/nginx/access.log -- no-clear
Custom date format:
logutil /var/log/nginx/access.log -- date-format= " %d/%b/%Y:%H:%M:%S %z"
Load regex from file:
logutil /var/log/nginx/access.log -- regex= /path/to/regex.txt
Tab Management
By default, only the core tabs (Overview, Requests, Detailed) are enabled. Additional tabs can be enabled using command-line flags:
Enable Security tab (detect suspicious activity, attacks, etc.):
logutil /var/log/nginx/access.log -- enable-security
Enable Performance tab (monitor response times, slow requests):
logutil /var/log/nginx/access.log -- enable-performance
Enable Errors tab (track error codes and failed requests):
logutil /var/log/nginx/access.log -- enable-errors
Enable Bots tab (detect bot traffic and crawlers):
logutil /var/log/nginx/access.log -- enable-bots
Enable Sparkline tab (real-time request rate visualization):
logutil /var/log/nginx/access.log -- enable-sparkline
Enable Heatmap tab (hourly traffic patterns visualization):
logutil /var/log/nginx/access.log -- enable-heatmap
Enable all tabs:
logutil /var/log/nginx/access.log -- enable-security -- enable-performance -- enable-errors -- enable-bots -- enable-sparkline -- enable-heatmap
Console Output Mode
Show top URLs in console:
logutil /var/log/nginx/access.log -- show-urls -- top= 10
Show top IPs in console:
logutil /var/log/nginx/access.log -- show-ips -- top= 10
Interactive Interface
Navigation
Tab or t : Switch between tabs (Overview → Requests → Detailed → Sparkline → Heatmap)
↑/ ↓ : Navigate through lists
←/ → : Switch between panels or pages
Enter : Copy selected item to clipboard (in Overview tab)
q or Ctrl+ C : Quit the application
Tabs Overview
1. Overview Tab
Left Panel : Top IP addresses with request counts and last update times
Right Panel : Top URLs with request types, domains, and statistics
Bottom Panel : Full URL display for selected URL
Navigation : Use arrow keys to switch between panels
2. Requests Tab
Search Functionality : Type to filter log entries
Pagination : Navigate through large result sets
Real-time Updates : New requests appear automatically
3. Detailed Tab
IP List : Select an IP address to view its details
Request Details : View all requests from the selected IP
Drill-down Analysis : Understand traffic patterns per IP
4. Sparkline Tab
Real-time Timeline : Visual representation of request patterns
Statistics : Min, max, average, and current request counts
Time Range : Shows the time span of the data
5. Heatmap Tab
Hourly Patterns : Color-coded request intensity by hour
Multi-day View : Track patterns across multiple days
Legend : Blue (low) → Green → Red (high) intensity
6. Security Tab
Advanced Threat Detection : SQL Injection, XSS, Path Traversal, Command Injection, Brute Force
Log Analysis : View highlighted suspicious patterns in log entries with visual indicators
IP Blocking : Block/unblock suspicious IP addresses with Enter key
Threat Levels : Visual indicators (🔴🟡🟢) for threat severity assessment
Pattern Highlighting : Suspicious patterns are highlighted with icons in log details
Security Summary : Comprehensive overview of all security threats and violations
Log Detail View : Press Enter to view detailed logs with highlighted suspicious patterns
Real-time Monitoring : Continuous monitoring and detection of security threats
Response Time Analysis : Tracks average, min, and max response times
Slow Request Identification : Highlights requests taking longer than 1 second
Requests Per Second : Real-time RPS calculation and monitoring
Performance Tracking : Detailed tracking of slow requests with timestamps
Performance Metrics : Real-time performance statistics
Resource Usage : Total response size and throughput analysis
8. Errors Tab
HTTP Error Analysis : Categorizes and counts error codes (4xx, 5xx)
Error Pattern Detection : Identifies common error sources
Error Distribution : Shows which URLs and IPs generate most errors
Error Trends : Tracks error patterns over time
9. Bots Tab
Bot Detection : Identifies crawlers, scrapers, and automated traffic
Bot Classification : Categorizes different types of bots (Googlebot, Bingbot, etc.)
Bot Activity Analysis : Tracks bot behavior patterns
Bot Traffic Statistics : Quantifies bot vs human traffic
Configuration
Command Line Options
Option
Description
Default
file
Path to the log file
Required
--count
Lines to read from end (0=tail, -1=all)
0
--regex
Regex pattern or file path
Nginx default
--date -format
Date parsing format
% d/ % b/ % Y: % H: % M: % S % z
--top
Number of top entries to show
10
--no-clear
Disable automatic cleanup
false
--show-urls
Output top URLs to console
false
--show-ips
Output top IPs to console
false
--log-to-file
Enable logging to app.log
false
--enable-security
Enable Security tab
false
--enable-performance
Enable Performance tab
false
--enable-errors
Enable Errors tab
false
--enable-bots
Enable Bots tab
false
--enable-sparkline
Enable Sparkline tab
false
--enable-heatmap
Enable Heatmap tab
false
1. Nginx Access Log (Default)
^ ( \S+ ) - " .+" \[ ( . * ? ) \] \d+ \. \d+ " (\S+)" " (\S+) (\S+?)(?:\?.*?)? "
Date Format: % d/ % b/ % Y: % H: % M: % S % z
2. Apache Access Log
^ ( \S+ ) \S+ \S+ \[ . * ? \] " \S+ (\S+?)(?:\?.*?)? \S+" \d+ \d+
Date Format: % d/ % b/ % Y: % H: % M: % S % z
^ ( \S+ ) - - \[ ( . * ? ) \] " (\S+) (\S+) HTTP/\d+\.\d+" ( \d+ ) ( \d+ ) " ([^" ]* )" " ( [ ^ " ]*)" $
Date Format: % d/ % b/ % Y: % H: % M: % S % z
Examples
Basic Monitoring
# Monitor nginx access logs in real-time
logutil /var/log/nginx/access.log
# Analyze entire log file
logutil /var/log/nginx/access.log --count=-1 --top=20
# Apache access logs
logutil /var/log/apache2/access.log --regex='^(\S+) \S+ \S+ \[.*?\] "\S+ (\S+?)(?:\?.*?)? \S+" \d+ \d+'
# Custom application logs
logutil /var/log/app/access.log --regex='^(\S+) \[(.*?)\] (\S+) (\S+)'
Console Output
# Get top URLs for reporting
logutil /var/log/nginx/access.log --show-urls --top=10
# Get top IPs for security analysis
logutil /var/log/nginx/access.log --show-ips --top=20
Advanced Configuration
# Custom regex from file
logutil /var/log/nginx/access.log --regex=/etc/logutil/patterns.txt
# Disable cleanup for long-term analysis
logutil /var/log/nginx/access.log --no-clear --count=-1
# Custom date format
logutil /var/log/nginx/access.log --date-format="%Y-%m-%d %H:%M:%S"
Memory Management
Automatic Cleanup : Removes entries older than 20 minutes when over 10,000 entries
Configurable : Use --no-clear to disable automatic cleanup
Efficient : Minimal memory footprint even with large log files
Processing Speed
Real-time : Processes new lines as they appear
Optimized : Efficient regex matching and data structures
Scalable : Handles high-traffic logs without performance degradation
File Handling
Smart Reading : Only processes new lines when tailing
Error Recovery : Gracefully handles file rotation and truncation
Progress Tracking : Shows loading progress for large files
Troubleshooting
Common Issues
1. "No match for line" errors
Check your regex pattern with --regex option
Verify log format matches the expected pattern
Use --log-to-file to debug parsing issues
2. High memory usage
Enable automatic cleanup (default behavior)
Use --count to limit initial processing
Consider using --no-clear only for short-term analysis
3. Slow performance with large files
Use --count=1000 to limit initial processing
Ensure regex pattern is optimized
Check system resources (CPU, memory)
Debug Mode
# Enable debug logging
logutil /var/log/nginx/access.log --log-to-file
# Check the generated app.log file for errors
tail -f app.log
Contributing
We welcome contributions! Please see our Contributing Guidelines for details.
Development Setup
git clone https://github.com/s00d/logutil.git
cd logutil
cargo build
cargo test