Gthulhu API Server

Gthulhu API Server is a Golang-based API server designed to integrate the Linux kernel scheduler (sched_ext) with the Kubernetes orchestration system. The core objective of this project is to collect scheduling metrics from eBPF programs and provide intelligent scheduling decisions to the kernel scheduler based on Kubernetes Pod labels and user-defined strategies.

DEMO

Click the image below to watch our DEMO on YouTube!

System Architecture

This API Server adopts a dual-mode architecture, consisting of two independent services:

┌─────────────────────────────────────────────────────────────────────────────────┐
│                              Gthulhu Architecture                               │
├─────────────────────────────────────────────────────────────────────────────────┤
│                                                                                 │
│   ┌─────────────┐         ┌─────────────────────┐         ┌─────────────────┐  │
│   │    User     │ ──────▶ │      Manager        │ ──────▶ │    MongoDB      │  │
│   │  (Web UI)   │         │ (Central Management)│         │  (Persistence)  │  │
│   └─────────────┘         └──────────┬──────────┘         └─────────────────┘  │
│                                      │                                          │
│                                      │ Query Pods via K8s API                   │
│                                      ▼                                          │
│                           ┌─────────────────────┐                               │
│                           │   Kubernetes API    │                               │
│                           │   (Pod Informer)    │                               │
│                           └─────────────────────┘                               │
│                                      │                                          │
│              ┌───────────────────────┼───────────────────────┐                  │
│              │                       │                       │                  │
│              ▼                       ▼                       ▼                  │
│   ┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐          │
│   │ Decision Maker  │     │ Decision Maker  │     │ Decision Maker  │          │
│   │   (Node 1)      │     │   (Node 2)      │     │   (Node N)      │          │
│   └────────┬────────┘     └────────┬────────┘     └────────┬────────┘          │
│            │                       │                       │                    │
│            ▼                       ▼                       ▼                    │
│   ┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐          │
│   │  sched_ext      │     │  sched_ext      │     │  sched_ext      │          │
│   │ (eBPF Scheduler)│     │ (eBPF Scheduler)│     │ (eBPF Scheduler)│          │
│   └─────────────────┘     └─────────────────┘     └─────────────────┘          │
│                                                                                 │
└─────────────────────────────────────────────────────────────────────────────────┘

Manager Mode

Purpose: Serves as the central management service, responsible for handling user requests, managing scheduling strategies, user authentication, and access control.

Key Features:

User authentication and authorization (JWT Token)
Role and permission management (RBAC)
CRUD operations for scheduling strategies
Monitor Pod status via Kubernetes Informer
Distribute scheduling intents to Decision Makers on each node
Data persistence to MongoDB

Default Port: :8081

Decision Maker Mode

Purpose: Deployed on each Kubernetes node (as DaemonSet), responsible for receiving scheduling intents and interacting with the local sched_ext scheduler.

Key Features:

Receive scheduling intents from Manager
Scan /proc filesystem to discover Pod processes
Convert scheduling strategies into concrete PID-based scheduling decisions
Collect eBPF scheduler metrics
Expose metrics via Prometheus endpoint

Default Port: :8080

Core Features

Manager Service Features

User Management: Create, query users, password reset
Role & Permission Management: RBAC role management, permission assignment
Scheduling Strategy Management: Create Pod label-based scheduling strategies
Scheduling Intent Tracking: Track strategy execution status
Kubernetes Integration: Real-time Pod monitoring via Pod Informer
JWT Authentication: RSA asymmetric encryption Token authentication

Decision Maker Service Features

Intent Processing: Receive and process scheduling intents from Manager
Process Discovery: Parse cgroup information to map PIDs to Pods
Scheduling Strategy Provider: Provide concrete PID scheduling strategies to sched_ext
Metrics Collection: Collect and expose eBPF scheduler metrics to Prometheus
Token Authentication: Validate requests from Manager

API Endpoints

Manager Endpoints

System Endpoints

Endpoint	Method	Description
`/health`	GET	Health check
`/version`	GET	Version information
`/swagger/*`	GET	Swagger documentation

Authentication Endpoints

Endpoint	Method	Description
`/api/v1/auth/login`	POST	User login

User Management Endpoints

Endpoint	Method	Description
`/api/v1/users`	POST	Create user
`/api/v1/users`	GET	List users
`/api/v1/users/password`	PUT	Reset password
`/api/v1/users/permissions`	PUT	Update permissions
`/api/v1/users/self/password`	PUT	Change own password
`/api/v1/users/self`	GET	Get own information

Role Management Endpoints

Endpoint	Method	Description
`/api/v1/roles`	POST	Create role
`/api/v1/roles`	GET	List roles
`/api/v1/roles`	PUT	Update role
`/api/v1/roles`	DELETE	Delete role
`/api/v1/permissions`	GET	List permissions

Scheduling Strategy Endpoints

Endpoint	Method	Description
`/api/v1/strategies`	POST	Create scheduling strategy
`/api/v1/strategies/self`	GET	List own strategies
`/api/v1/intents/self`	GET	List own scheduling intents

Decision Maker Endpoints

Endpoint	Method	Description
`/health`	GET	Health check
`/version`	GET	Version information
`/metrics`	GET	Prometheus metrics
`/api/v1/auth/token`	POST	Get authentication token
`/api/v1/intents`	POST	Receive scheduling intents
`/api/v1/scheduling/strategies`	GET	Get scheduling strategies
`/api/v1/metrics`	POST	Update metrics data

Data Structures

ScheduleStrategy

Field	Type	Description
`strategyNamespace`	string	Strategy namespace
`labelSelectors`	[]LabelSelector	Pod label selectors
`k8sNamespace`	[]string	Kubernetes namespaces
`commandRegex`	string	Process command regex
`priority`	int	Priority level
`executionTime`	int64	Execution time (nanoseconds)

ScheduleIntent

Field	Type	Description
`podID`	string	Pod UID
`podName`	string	Pod name
`nodeID`	string	Node name
`k8sNamespace`	string	Kubernetes namespace
`commandRegex`	string	Process command regex
`priority`	int	Priority level
`executionTime`	int64	Execution time (nanoseconds)
`podLabels`	map[string]string	Pod labels
`state`	string	Intent state

MetricSet

Field	Type	Description
`usersched_last_run_at`	uint64	User scheduler last run timestamp
`nr_queued`	uint64	Number of tasks in scheduling queue
`nr_scheduled`	uint64	Number of scheduled tasks
`nr_running`	uint64	Number of running tasks
`nr_online_cpus`	uint64	Number of online CPUs
`nr_user_dispatches`	uint64	Number of user-space dispatches
`nr_kernel_dispatches`	uint64	Number of kernel-space dispatches
`nr_cancel_dispatches`	uint64	Number of cancelled dispatches
`nr_bounce_dispatches`	uint64	Number of bounce dispatches
`nr_failed_dispatches`	uint64	Number of failed dispatches
`nr_sched_congested`	uint64	Number of scheduler congestion events

Quick Start

0. Test Environment Setup

For local development, you need to set up a MongoDB instance. You can use Docker to start the infrastructure:

# Start MongoDB using Docker
$ docker run --rm --network host mongo:6.0 mongosh "mongodb://test:test@localhost:27017/?authSource=admin&authMechanism=SCRAM-SHA-256" --eval 'db.runCommand({ ping: 1 })'

After starting MongoDB, create the required user and verify the connection:

# Create a test user with root privileges in MongoDB
$ docker exec mongodb mongosh --eval 'db.getSiblingDB("admin").createUser({user:"test",pwd:"test",roles:[{role:"root",db:"admin"}]})'

# Verify the connection with the created user credentials
$ docker exec mongodb mongosh "mongodb://test:test@localhost:27017/?authSource=admin&authMechanism=SCRAM-SHA-256" --eval 'db.runCommand({ ping: 1 })'

1. Install Dependencies

go mod tidy

2. Configuration

Manager Configuration (`config/manager_config.toml`)

[server]
host = ":8081"

[logging]
level = "info"

[mongodb]
host = "localhost"
port = "27017"
user = "test"
password = "test"
database = "manager"

[k8s]
kube_config_path = "/path/to/.kube/config"
in_cluster = false

[key]
rsa_private_key_pem = "..."
dm_public_key_pem = "..."
client_id = "your-client-id"

[account]
admin_email = "[email protected]"
admin_password = "your-password"

Decision Maker Configuration (`config/dm_config.toml`)

[server]
host = ":8080"

[logging]
level = "info"

[token]
rsa_private_key_pem = "..."
token_duration_hr = 24

3. Start Services

Start Manager

go run main.go manager

# With custom configuration
go run main.go manager -c manager_config -d /path/to/config

Start Decision Maker

go run main.go decisionmaker

# With custom configuration
go run main.go decisionmaker -c dm_config -d /path/to/config

4. Test API

Health Check

# Manager
curl http://localhost:8081/health

# Decision Maker
curl http://localhost:8080/health

User Login

curl -X POST http://localhost:8081/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{
    "username": "[email protected]",
    "password": "your-password-here"
  }'

Create Scheduling Strategy

curl -X POST http://localhost:8081/api/v1/strategies \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <your-token>" \
  -d '{
    "strategyNamespace": "default",
    "labelSelectors": [
      {"key": "app", "value": "nginx"}
    ],
    "k8sNamespace": ["default"],
    "priority": 10,
    "executionTime": 20000000
  }'

Kubernetes Deployment

Deployment Architecture

Manager: Deployed as a Deployment, typically single replica
Decision Maker: Deployed as a DaemonSet on every node

Deployment Manifest Locations

deployment/
├── kind/                    # Kind local test environment
│   ├── local_setup.sh
│   ├── decisonmaker/
│   │   ├── daemonset.yaml
│   │   └── service.yaml
│   ├── manager/
│   │   ├── deployment.yaml
│   │   └── service.yaml
│   └── mongo/
│       ├── secret.yaml
│       ├── service.yaml
│       └── statefulset.yaml
└── local/
    └── docker-compose.infra.yaml  # Docker Compose for local development

Quick Deployment

# Local testing with Kind
cd deployment/kind
./local_setup.sh

# Or manual deployment
kubectl apply -f deployment/kind/mongo/
kubectl apply -f deployment/kind/manager/
kubectl apply -f deployment/kind/decisonmaker/

RBAC Permission Requirements

Manager requires the following Kubernetes RBAC permissions:

pods: list, watch, get
namespaces: list, get

Development Guide

Project Structure

.
├── main.go                 # Entry point, uses Cobra for subcommands
├── config/                 # Configuration definition and parsing
├── manager/               # Manager service
│   ├── app/              # Application initialization
│   ├── cmd/              # Cobra commands
│   ├── domain/           # Domain models and interfaces
│   ├── k8s_adapter/      # Kubernetes client
│   ├── client/           # Decision Maker client
│   ├── repository/       # MongoDB data access
│   ├── rest/             # HTTP handlers
│   ├── service/          # Business logic
│   └── migration/        # Database migrations
├── decisionmaker/        # Decision Maker service
│   ├── app/             # Application initialization
│   ├── cmd/             # Cobra commands
│   ├── domain/          # Domain models
│   ├── rest/            # HTTP handlers
│   └── service/         # Business logic and process discovery
├── pkg/                  # Shared packages
│   ├── logger/          # Logging utilities
│   ├── middleware/      # HTTP middleware
│   └── util/            # Common utility functions
└── deployment/          # Deployment configurations

Tech Stack

Web Framework: Echo v4
Dependency Injection: Uber fx
CLI: Cobra
Logging: zerolog
Database: MongoDB (mongo-driver v2)
Kubernetes Client: client-go
Metrics Collection: Prometheus client_golang
Configuration Management: Viper

Build and Test

# Build
make build

# Run tests
make test

# Build Docker image
make image

Container Images

This project uses GitHub Actions to automatically build and publish to GitHub Container Registry.

# Use latest version
docker pull ghcr.io/gthulhu/api:main

# Use specific version
docker pull ghcr.io/gthulhu/api:v1.0.0

License

This project is open source.

Name		Name	Last commit message	Last commit date
Latest commit History 63 Commits
.github		.github
config		config
decisionmaker		decisionmaker
deployment		deployment
docs/manager		docs/manager
manager		manager
pkg		pkg
web/static		web/static
.gitignore		.gitignore
.mockery.yaml		.mockery.yaml
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
go.mod		go.mod
go.sum		go.sum
main.go		main.go
notice.txt		notice.txt

License

Gthulhu/api

Folders and files

Latest commit

History

Repository files navigation

Gthulhu API Server

DEMO

System Architecture

Manager Mode

Decision Maker Mode

Core Features

Manager Service Features

Decision Maker Service Features

API Endpoints

Manager Endpoints

System Endpoints

Authentication Endpoints

User Management Endpoints

Role Management Endpoints

Scheduling Strategy Endpoints

Decision Maker Endpoints

Data Structures

ScheduleStrategy

ScheduleIntent

MetricSet

Quick Start

0. Test Environment Setup

1. Install Dependencies

2. Configuration

Manager Configuration (config/manager_config.toml)

Decision Maker Configuration (config/dm_config.toml)

3. Start Services

Start Manager

Start Decision Maker

4. Test API

Health Check

User Login

Create Scheduling Strategy

Kubernetes Deployment

Deployment Architecture

Deployment Manifest Locations

Quick Deployment

RBAC Permission Requirements

Development Guide

Project Structure

Tech Stack

Build and Test

Container Images

License

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 4

Packages 0

Uh oh!

Contributors 5

Uh oh!

Languages

Manager Configuration (`config/manager_config.toml`)

Decision Maker Configuration (`config/dm_config.toml`)

Packages