The documentation repository for the Karafka ecosystem - a Ruby and Rails efficient Kafka processing framework.
This repository contains all the documentation for Karafka and its related components, organized as a collection of Markdown files that are automatically processed and deployed to karafka.io/docs.
This repository includes several automated processes to keep documentation current and consistent.
The repository automatically fetches and updates dynamic content daily at 6:00 AM UTC and on-demand through repository dispatch events:
- Component Changelogs: Pulls the latest CHANGELOG.md files from all Karafka ecosystem repositories
- librdkafka Releases: Fetches and formats librdkafka release information from GitHub
- librdkafka Documentation: Generates error reference, statistics, and configuration documentation
The documentation is automatically mirrored to the GitHub Wiki on every merge to master. The sync process:
- Copies and flattens the directory structure for GitHub Wiki compatibility
- Adds a header to each page directing users to the main documentation site
- Removes stale pages that no longer exist in the source
- Preserves wiki-specific files like
_Footer.mdand_Sidebar.md
The header template is located in .gh-templates/_GH_Header.md.
Located in the bin/ directory:
| Script | Description |
|---|---|
refresh_remote_content |
Downloads changelogs and commercial license from component repositories |
refresh_librdkafka_errors |
Generates librdkafka error documentation using the rdkafka gem |
refresh_librdkafka_releases |
Generates librdkafka releases documentation from GitHub |
refresh_librdkafka_statistics |
Generates librdkafka statistics documentation |
refresh_librdkafka_configuration |
Generates librdkafka configuration documentation |
refresh_karafka_integrations_catalog |
Generates integration tests catalog |
refresh_instrumentation_events |
Generates instrumentation events documentation |
align_structure |
Flattens nested documentation structure for MkDocs compatibility |
mklint |
Validates documentation structure, references, and builds with MkDocs |
sync_gh |
Synchronizes content with the GitHub Wiki |
The CI pipeline ensures documentation quality through:
- Markdown Linting: Uses
markdownlint-cli2to enforce consistent formatting - Structure Validation: Validates cross-references, anchor links, and documentation structure
- MkDocs Build: Builds documentation in strict mode to catch errors
- Security: Verifies all GitHub Actions use SHA-pinned versions
When automatic content updates are available, the system:
- Fetches the latest content from all sources
- Regenerates dynamic documentation
- Creates a pull request with the changes
- Maintains a clean commit history
Several files in this repository are automatically generated and should not be edited manually:
Changelog/*.md- Component changelogsLibrdkafka/Changelog.md- librdkafka release notesLibrdkafka/Errors.md- librdkafka error referenceLibrdkafka/Statistics.md- librdkafka statistics documentationLibrdkafka/Configuration.md- librdkafka configuration documentationDevelopment/Karafka-Integration-Tests-Catalog.md- Integration tests catalogOperations/Instrumentation-Events.md- Instrumentation eventsWaterDrop/Instrumentation-Events.md- WaterDrop instrumentation events
These files include header comments indicating their auto-generated status.
See LICENSE for details.