- Introduction
- Project Overview & Learning Objectives
- Application Architecture: The
AuditFlow Platform - Technology Stack
- Directory Structure
- Getting Started
- DevOps Pillars & Their Integration
- Contributing
- License
- Contact
Project Genesis is a comprehensive, hands-on learning initiative designed to build and manage a tangible, multi-service application within a modern DevOps ecosystem. This project serves as a real-world sandbox, demonstrating best practices across various disciplines, including DevOps, Site Reliability Engineering (SRE), DevSecOps, and FinDevOps.
At its core, Project Genesis leverages a Kubernetes (K3s) cluster to orchestrate the AuditFlow Platform. The project provides a holistic view of the software delivery lifecycle, from infrastructure provisioning to application deployment and ongoing operations, all managed as code.
This project aims to provide a practical foundation for aspiring and current DevOps practitioners. By working through Project Genesis, you will gain hands-on experience in:
- Designing and deploying the AuditFlow Platform in a Kubernetes environment.
- Implementing Infrastructure as Code (IaC) for consistent and automated infrastructure provisioning.
- Building robust CI/CD pipelines for continuous integration and delivery.
- Managing Kubernetes resources effectively using Helm and Kustomize.
- Integrating testing and code quality into the development workflow.
- Establishing observability for monitoring application health and performance.
- Understanding and applying DevSecOps principles, specifically gaining hands-on experience with Static Application Security Testing (SAST), Software Composition Analysis (SCA) and integrating security analysis directly into CI/CD pipelines.
- Exploring FinDevOps concepts for cost-effective cloud-native operations.
The AuditFlow Platform is a conceptual multi-service application designed to generate, process, and display audit events. It typically consists of:
audit_event_generator: A service responsible for creating synthetic audit events.audit-log-analysis: Consumes events, performs analysis, and stores results.event-audit-dashboard: Provides a user interface to visualize audit events and alerts.notification-service: Handles alerts and notifications based on analysis.postgres: A relational database for persistent data storage.rabbitmq: A message broker for inter-service communication.redis: A key-value store for caching or temporary data.
These services communicate primarily via RabbitMQ, enabling an event-driven architecture that showcases real-world microservice interaction patterns. For further details about the application architecture, please refer to the Architecture Documentation.
Project Genesis utilizes a diverse set of industry-standard tools and technologies:
- Containerization: Docker
- Source Code Management: GitHub
- CI/CD Automation: GitHub Actions
- Container Registry: Docker Hub
- Kubernetes Distribution: K3s (Lightweight Kubernetes)
- Programming Language: Python
- Kubernetes Package Manager: Helm
- Testing Framework: Pytest (for Python application testing)
- Code Coverage: Coverage.py (integrated with Pytest)
- Infrastructure as Code (IaC):
- Ansible: For configuration management and K3s cluster setup.
This project is a dynamic and evolving learning initiative. Its status and ongoing development can be tracked via the GitHub Project board and GitHub Issues.
The project is meticulously organized to separate application code from infrastructure, documentation, and CI/CD configurations. This structure promotes clarity, maintainability, and scalability.
.
├── .github/ # GitHub Actions workflows for CI/CD
├── docs/ # Project documentation (architecture, pillars, setup guides)
├── infra/ # Infrastructure as Code (Terraform for provisioning, Ansible for configuration)
│ ├── terraform/ # Terraform configurations for base infrastructure
│ └── ansible/ # Ansible playbooks and roles for configuration management (e.g., K3s installation)
├── k8s/ # Kubernetes manifests and Helm charts
│ ├── base/ # Base Kubernetes YAMLs for individual services (can be integrated into Helm)
│ ├── charts/ # Helm charts for the entire application and individual services
│ └── overlays/ # Kustomize overlays for environment-specific configurations
├── monitoring/ # Configurations for Prometheus, Grafana, and alert rules
├── reports/ # Generated reports (e.g., test coverage XML)
├── scripts/ # Helper scripts for deployment, setup, etc.
└── src/ # Application source code (the 'AuditFlow Platform' microservices)
├── audit_event_generator/
├── audit-log-analysis/
├── event-audit-dashboard/
└── notification-service/
└── tests/ # Unit and integration tests for each service
├── .gitignore # Files and directories to ignore in Git
├── .pytest.ini # Global Pytest configuration
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── README.md # This file
└── TODO.md # Project tasks and notes
For a more detailed breakdown of the directory contents, refer to the docs/ directory.
🎥 Want to see Project Genesis in action and follow its development? Check out our video tutorials and development logs on our YouTube Channel!
Project Genesis is structured to explicitly demonstrate key DevOps principles:
- DevOps: Full automation from code commit to deployment using a hybrid runner strategy, fostering collaboration and rapid feedback cycles for the
AuditFlow Platform. - Site Reliability Engineering (SRE): Focus on observability (Prometheus, Grafana), defining SLOs/SLAs, and building resilient, self-healing systems for the
AuditFlow Platform. - DevSecOps: Integration of security best practices throughout the SDLC, including automated Static Application Security Testing (SAST) with Bandit for Python services, Software Composition Analysis (SCA) and vulnerability scanning of Docker images with Trivy, and secure Kubernetes configurations for the
AuditFlow Platform. - FinDevOps: Awareness and implementation of cost-efficient strategies in cloud-native environments, such as resource optimization, effective scaling, and cost monitoring of the
AuditFlow Platformand its underlying infrastructure.
Project Genesis incorporates several security best practices:
- Static Application Security Testing (SAST): Python services are scanned using Bandit during the CI/CD pipeline. Findings are automatically uploaded to GitHub Code Scanning.
- Scan Tool: Bandit
- Output Format: SARIF
- Integration Point: GitHub Actions (
.github/workflows/build-single-service.yml) - View Results: Check the Code scanning alerts in the GitHub repository's Security tab.
- Software Composition Analysis (SCA) & Vulnerability Scanning:
- Goal: Identify known vulnerabilities in third-party libraries/dependencies and in Docker images (OS packages, installed software).
- Purpose: Ensure the security of your supply chain by scanning for components with known CVEs.
- Scan Tool: Trivy
- Scan Types:
- Filesystem Scan: Scans Python project dependencies (e.g.,
requirements.txt,pyproject.toml) for vulnerabilities. - Image Scan: Scans built Docker images for vulnerabilities in OS packages, programming language dependencies, and configuration issues.
- Filesystem Scan: Scans Python project dependencies (e.g.,
- Output Format: SARIF
- Integration Point: GitHub Actions (
.github/workflows/build-single-service.yml) - View Results: Check the Code scanning alerts in the GitHub repository's Security tab.
- Secure Configuration: Kubernetes security best practices are enforced through Infrastructure as Code. The *self-hosted runner deployment and its lifecycle are managed by Ansible, ensuring a consistent and secure setup without manual intervention.
This project is licensed under the MIT License - see the LICENSE file for details.
- Author: Joji Vithayathil Johny
- Email: [email protected]
- GitHub Repository: https://github.com/jojees/project-genesis