Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@ktsaou
Copy link
Member

@ktsaou ktsaou commented Jun 22, 2025

  • libnetdata converts yaml to json-c objects and vice versa
  • added significantly more tests to log2journal yaml parsing and operation
  • log2journal custom yaml parser replaced with libnetdata's yaml parser
  • inicfg populated from netdata.yaml or netdata.json
  • inicfg to produce yaml and json config files
  • /netdata.yaml to export netdata's configuration in yaml
  • /netdata.json to export netdata's configuration in json

@github-actions github-actions bot added area/docs area/collectors Everything related to data collection area/daemon area/build Build system (autotools and cmake). labels Jun 22, 2025
ktsaou and others added 5 commits June 27, 2025 20:41
Implements a new module in src/libnetdata/yaml that provides:
- YAML parsing from string, file, or file descriptor to json-c objects
- YAML generation from json-c objects to string, file, or file descriptor
- Support for YAML subset matching JSON 100% (strings, numbers, null, maps, arrays)
- Comprehensive unit tests covering edge cases and libyaml limitations
- Round-trip consistency for data conversion

The module handles known libyaml limitations gracefully:
- Octal escape sequences are not supported
- Single-quoted literal newlines become spaces
- Null bytes in strings may cause issues
- Complex multiline indentation may not be preserved exactly

Tests have been adjusted to accept libyaml's actual behavior while
maintaining robustness for the supported use cases.

πŸ€– Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
…rage

- Migrate from legacy filename-based CLI tests to clean .cmd format
- Convert comment-based failure tests to explicit .fail files
- Replace tests_v2.sh with enhanced tests.sh as primary test runner
- Add --verbose and --test flags for improved debugging capabilities
- Implement version-agnostic testing to prevent build-dependent failures
- Add comprehensive test suite covering 81 scenarios:
  * Core logic (7 tests): rewrite pipeline, variables, renaming, filters
  * YAML parsing (13 tests): edge cases, multiline, type handling
  * Unicode/encoding (8 tests): UTF-8, control chars, escape sequences
  * Error handling (6 tests): syntax, regex, type validation
  * Advanced features (5 tests): complex patterns, edge cases
  * Real-world formats (5 tests): Apache, nginx, Docker, syslog logs
  * Boundary testing (12 tests): limits, empty values, long strings
  * CLI integration (2 tests): inject, filter behavior

All tests pass and framework includes detailed troubleshooting documentation.

πŸ€– Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
…log2journal

Major architectural improvements to log2journal configuration parsing:

πŸ”§ **YAML Parser Migration**
- Replace custom event-based YAML parser with libnetdata's unified parser
- Add YAML2JSON_FLAGS enum with YAML2JSON_ALL_VALUES_AS_STRINGS support
- Ensure consistent parsing behavior across all Netdata components
- Delete 500+ lines of custom parser code in favor of 200 lines using libnetdata

πŸš€ **JSON-C Parser Macro Enhancements**
- Enhanced all scalar conversion macros for maximum type flexibility
- JSONC_PARSE_INT64/DOUBLE/BOOL now convert between all scalar types
- String macros now accept int/double/boolean with proper conversion
- Replace snprintf with optimized Netdata print_int64()/print_netdata_double()
- Add comprehensive endptr validation for string-to-number conversions

πŸ”’ **Robust Type Validation**
- Fix JSONC_PARSE_SUBOBJECT/ARRAY to always validate types when field present
- Implement "try everything, fail only when impossible" conversion philosophy
- Proper error logic: required=false allows missing fields but rejects invalid types
- Enhanced boolean parsing: strings ("yes"/"no"), integers (0/1), doubles (0.0/1.0)

πŸ§ͺ **Enhanced Testing Framework**
- Comprehensive unit test coverage with 100+ test cases
- Add .fail file support for testing error conditions
- Version-agnostic test comparisons for error messages
- Improved test runner with better error reporting and exit code validation
- Test boundary conditions, edge cases, and real-world log formats

✨ **Configuration Robustness**
- Support scalar type coercion (string "123" β†’ int 123, bool "yes" β†’ true)
- Reject fundamentally incompatible types (string β†’ array/object)
- Maintain backward compatibility while adding flexibility
- Consistent error messages across all validation failures

This migration unifies log2journal with Netdata's standard parsing infrastructure
while dramatically improving type conversion flexibility and error handling.

πŸ€– Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/build Build system (autotools and cmake). area/collectors Everything related to data collection area/daemon area/docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant