# EVE Kernel Documentation

This document describes how EVE's Linux kernel is built, configured, and maintained.

## TL;DR for Developers

To quickly build a custom kernel for development (example shows disabling `MODULE_SIG_FORCE`, but you can modify any kernel option):

```bash
# Clone the kernel repository
git clone https://github.com/lf-edge/eve-kernel/
cd eve-kernel

# Checkout the branch you want to modify (example for amd64 v6.12.49)
git checkout eve-kernel-amd64-v6.12.49-generic

# Generate base config (see "Kernel Config Flavors" section for other options: rt, hwe)
make eve-core_defconfig

# Modify kernel configuration interactively
make menuconfig
# Example: disable MODULE_SIG_FORCE (or modify any other option you need)
# - Press '/' to search for an option
# - Type the option name (e.g., "MODULE_SIG_FORCE")
# - Press the number shown to jump to the option
# - Toggle to enable/disable it
# - Save and exit

# IMPORTANT: Save as defconfig (do not manually edit .config!)
make savedefconfig
cp defconfig arch/x86/configs/eve-core_defconfig

# Build kernel in container
make -f Makefile.eve kernel-gcc

# Option 1: Build EVE on the same machine using local kernel
cd /path/to/eve
# Note: -s flag is crucial (silent mode) to output only the tag without make noise
make KERNEL_TAG=$(make -C /path/to/eve-kernel -s -f Makefile.eve docker-tag-gcc) pkgs rootfs

# Option 2: Push to DockerHub and build EVE elsewhere
# Push your kernel container to your DockerHub account, then:
cd /path/to/eve
make KERNEL_TAG=<your-dockerhub-tag> pkgs rootfs
```

## Kernel Repository

The EVE kernel is maintained in a separate repository:

**Repository:** <https://github.com/lf-edge/eve-kernel>

### Historical Note

The kernel was previously built from source within EVE's main repository under `pkg/kernel/`.
As of 2024, the kernel has been moved to its own repository (`lf-edge/eve-kernel`) and is
consumed by EVE as a pre-built container image.

## Kernel Versioning and Branches

EVE supports multiple kernel versions across different architectures and platforms. The kernel
version selection is handled automatically based on the target architecture (`ZARCH`) and
platform (`PLATFORM`).

### Current Kernel Versions

- **amd64**: v6.12.49 (generic and rt platforms)
- **arm64**:
  - v6.1.155 (generic)
  - v5.10.192 (nvidia-jp5)
  - v5.15.136 (nvidia-jp6)
- **riscv64**: v6.1.112 (generic)

### Branch Naming Convention

Kernel branches in the `eve-kernel` repository follow this naming pattern:

```text
eve-kernel-<ARCH>-<VERSION>-<FLAVOR>
```

Where `<FLAVOR>` corresponds to the EVE `PLATFORM` variable.

Examples:

- `eve-kernel-amd64-v6.12.49-generic` (PLATFORM=generic)
- `eve-kernel-arm64-v5.10.192-nvidia-jp5` (PLATFORM=nvidia-jp5)
- `eve-kernel-arm64-v5.15.136-nvidia-jp6` (PLATFORM=nvidia-jp6)
- `eve-kernel-amd64-next` (next LTS development branch)

## Important Files

### kernel-commits.mk

Location: `/kernel-commits.mk` (root of EVE repository)

**⚠️ Warning:** This file is auto-generated by `tools/update_kernel_commits.py` and should NOT be manually modified. Always use the update script to change kernel commits.

This file contains the specific commit hashes for each kernel branch that EVE uses. Each entry
maps a combination of architecture, version, and flavor to a specific commit:

```makefile
KERNEL_COMMIT_amd64_v6.12.49_generic = ef7ccc4d151c
KERNEL_COMMIT_arm64_v5.10.192_nvidia-jp5 = b4464b03583e
KERNEL_COMMIT_arm64_v6.1.155_generic = ece043d8b403
KERNEL_COMMIT_riscv64_v6.1.112_generic = 18e1d313b90b
```

**Purpose:** Pins EVE to specific kernel commits to ensure reproducible builds.

### kernel-version.mk

Location: `/kernel-version.mk` (root of EVE repository)

This file defines the kernel version selection logic based on architecture and platform. It:

- Maps platforms to kernel versions
- Sets kernel flavors and config flavors
- Constructs Docker tags for kernel images
- Includes `kernel-commits.mk` to get commit hashes

**Key Variables:**

- `KERNEL_VERSION`: The kernel version (e.g., v6.12.49)
- `KERNEL_FLAVOR`: The kernel flavor matching the EVE `PLATFORM` (e.g., generic, nvidia-jp5, nvidia-jp6)
- `KERNEL_CONFIG_FLAVOR`: Config variant (e.g., core, rt, hwe) - see detailed description below
- `KERNEL_COMMIT`: The specific commit hash from kernel-commits.mk
- `KERNEL_TAG`: The full Docker image tag

### tools/update_kernel_commits.py

Location: `/tools/update_kernel_commits.py`

This Python script automates the process of updating kernel commits in EVE.

**Features:**

- Queries GitHub API to fetch latest commits from eve-kernel branches
- Generates updated `kernel-commits.mk` file
- Creates detailed commit messages with changelog
- Validates kernel tags against DockerHub
- Supports testing mode for dry-run updates

**Usage:**

```bash
# Update all kernel branches
./tools/update_kernel_commits.py

# Update specific branch
./tools/update_kernel_commits.py --branch eve-kernel-amd64-v6.12.49-generic

# Dry run to see what would change
./tools/update_kernel_commits.py --test

# Use custom GitHub token
./tools/update_kernel_commits.py --token <your-github-token>
```

**GitHub Token:** The script requires a GitHub personal access token for API access. It will
prompt you to create one if not found. The token is stored in `~/.config/eve/github_token`.

## Branching Strategy

EVE's kernel repository uses a multi-branch strategy:

### 1. Versioned Branches

These track specific stable kernel versions:

- Example: `eve-kernel-amd64-v6.12.49-generic`
- Based on upstream stable kernel releases
- Receive backported fixes and EVE-specific patches
- Used for production EVE releases

### 2. Next Branch

The "next" branch is prepared for the next LTS kernel version, not for current development:

- Example: `eve-kernel-amd64-next`
- Represents the next planned LTS version (at time of writing, the next LTS version is not yet defined)
- All current patches are applied to next as well as stable branches

**LTS Transition Process:**
When time comes to move to a new LTS:

1. The floating `next` branch content is used to create a new versioned branch (e.g., eve-kernel-amd64-v6.13.x-generic)
2. The old stable version (e.g., v6.12.x) is replaced with this new version
3. The `next` branch is rebased on top of the next upcoming LTS

This strategy ensures EVE is always prepared for kernel updates in terms of CI, patch rework, and smoke testing.

### 3. Branch Selection

The appropriate branch is automatically selected based on:

- Architecture (amd64, arm64, riscv64)
- Platform (generic, rt, nvidia-jp5, etc.)
- Configuration defined in `kernel-version.mk`

## Kernel Config Flavors

**⚠️ Important Distinction:** `KERNEL_FLAVOR` and `KERNEL_CONFIG_FLAVOR` are different concepts and can be confusing:

- **`KERNEL_FLAVOR`** (also called just `FLAVOR` in branch names): Corresponds to the EVE `PLATFORM` variable and determines the kernel branch (e.g., generic, nvidia-jp5, nvidia-jp6). This is part of the branch name like `eve-kernel-amd64-v6.12.49-generic`.
- **`KERNEL_CONFIG_FLAVOR`**: Controls which configuration variant is used when building from a specific branch (e.g., core, rt, hwe). This is NOT part of the branch name but determines which defconfig file is used.

The `KERNEL_CONFIG_FLAVOR` variable controls which kernel configuration variant to use. This allows
multiple configurations to be built from the same branch without maintaining separate branches with
identical patch sets.

### Available Config Flavors

**For x86/amd64:**

- `core`: Standard EVE kernel configuration with essential drivers
- `hwe`: Hardware Enablement - includes many additional drivers for broader hardware support
- `rt`: Real-time kernel configuration with RT patches for deterministic latency

**Not all flavors are available on all branches.** The flavor determines which defconfig file is used:

- `arch/x86/configs/eve-core_defconfig`
- `arch/x86/configs/eve-hwe_defconfig`
- `arch/x86/configs/eve-rt_defconfig`

## Building the Kernel

### In eve-kernel Repository

The kernel is built using the `Makefile.eve` in the eve-kernel repository:

```bash
# Build with GCC compiler (standard method)
make -f Makefile.eve kernel-gcc

# Build with specific config flavor
make KERNEL_CONFIG_FLAVOR=rt -f Makefile.eve kernel-gcc
make KERNEL_CONFIG_FLAVOR=hwe -f Makefile.eve kernel-gcc

# Build with LLVM/Clang (experimental, not currently used in production)
make -f Makefile.eve kernel-llvm

# Get the Docker tag that was built
make -f Makefile.eve docker-tag-gcc
```

### Kernel Configuration Files

Kernel configurations are stored in the standard Linux kernel location:

- `arch/x86/configs/eve-<FLAVOR>_defconfig` (for amd64)
- `arch/arm64/configs/eve-<FLAVOR>_defconfig` (for arm64)
- `arch/riscv/configs/eve-<FLAVOR>_defconfig` (for riscv64)

Where `<FLAVOR>` is determined by the `KERNEL_CONFIG_FLAVOR` make variable:

- `eve-core_defconfig`: Standard EVE kernel config
- `eve-rt_defconfig`: Real-time kernel config
- `eve-hwe_defconfig`: Hardware enablement config (many drivers enabled)

The asterisk (`*`) in config paths is controlled by the `KERNEL_CONFIG_FLAVOR` variable.

## Modifying Kernel Configuration

**⚠️ Important:** Always use the kernel build system to modify configuration. Manual edits to
`.config` can break dependencies.

### Step-by-Step Process

1. **Start with base config:**

   ```bash
   make eve-core_defconfig
   # Or for other flavors:
   # make eve-rt_defconfig
   # make eve-hwe_defconfig
   ```

2. **Modify interactively:**

   ```bash
   make menuconfig
   ```

   - Use `/` to search for options (e.g., search for "MODULE_SIG_FORCE")
   - Navigate with arrow keys
   - Use space to toggle options
   - Save and exit

   **Example:** The TL;DR section shows disabling `MODULE_SIG_FORCE` as a common development task.

3. **Save as defconfig:**

   ```bash
   make savedefconfig
   ```

   This creates a minimal `defconfig` file with only the changes from defaults.

4. **Copy to proper location:**

   ```bash
   # For x86/amd64
   cp defconfig arch/x86/configs/eve-core_defconfig

   # For arm64
   cp defconfig arch/arm64/configs/eve-core_defconfig

   # For riscv64
   cp defconfig arch/riscv/configs/eve-core_defconfig
   ```

5. **Commit and push:**

   ```bash
   git add arch/*/configs/eve-*_defconfig
   git commit -m "kernel: update configuration"
   git push
   ```

### Why Use savedefconfig?

- `savedefconfig` generates a minimal config with only non-default values
- Keeps config files small and manageable
- Automatically handles option dependencies
- Makes config changes easier to review in git diffs
- Prevents configuration conflicts

## Integrating Kernel Changes into EVE

### Option 1: Use Local Build

Build EVE with your locally-built kernel:

```bash
cd /path/to/eve
make KERNEL_TAG=$(make -C /path/to/eve-kernel -s -f Makefile.eve docker-tag-gcc) pkgs rootfs
```

This must be done on the same machine where you built the kernel.

### Option 2: Push to Docker Registry

1. **Tag your kernel:**

   ```bash
   docker tag lfedge/eve-kernel:$(make -f Makefile.eve docker-tag-gcc) \
       yourusername/eve-kernel:custom-tag
   ```

2. **Push to DockerHub:**

   ```bash
   docker push yourusername/eve-kernel:custom-tag
   ```

3. **Build EVE with custom kernel:**

   ```bash
   cd /path/to/eve
   make KERNEL_TAG=yourusername/eve-kernel:custom-tag pkgs rootfs
   ```

### Option 3: Update Official Kernel

1. **Make changes in eve-kernel repository**
2. **Push to your fork and create PR**
3. **After PR is merged, update EVE:**

   ```bash
   cd /path/to/eve
   ./tools/update_kernel_commits.py --branch eve-kernel-amd64-v6.12.49-generic
   git add kernel-commits.mk
   git commit -F kernel-update-commit-message.txt
   ```

## Container-Based Build

The kernel is built inside a container to ensure reproducibility and consistency. The build
container includes:

- Specific toolchain versions (GCC or LLVM)
- Required build dependencies
- Deterministic build environment

This approach ensures that the same kernel source produces identical binaries regardless of the
host system.

## Reproducible Builds

EVE kernels are configured for bit-by-bit reproducible builds. This means:

- Build timestamps are fixed
- Toolchain versions are pinned
- Build flags are consistent
- Output is deterministic

For more details, see [EVE-IMAGE-REPRODUCIBILITY.md](EVE-IMAGE-REPRODUCIBILITY.md).

## Kernel Modules and Signing

EVE supports signed kernel modules for enhanced security. By default:

- Module signing is enabled in production builds
- Modules must be signed with the correct key
- `MODULE_SIG_FORCE` can be disabled for development

To use custom module signing keys in the eve-kernel repository:

1. Generate your key pair
2. Place the signing certificate in the appropriate location in the eve-kernel repository
3. Update the kernel build configuration to use your custom keys

**Note:** Disabling `MODULE_SIG_FORCE` (shown in the TL;DR section) is just one example of a
common kernel configuration change for development. You may need to modify other options depending
on your development needs.

## Platform-Specific Kernels

### NVIDIA Jetson Platforms

- Use vendor-specific kernel versions (JP5: v5.10.192, JP6: v5.15.136)
- Include NVIDIA-specific drivers and patches
- Branch names include platform identifier (nvidia-jp5, nvidia-jp6)

### Real-Time (RT) Kernel

- amd64 with `PLATFORM=rt` uses real-time kernel patches
- Uses the `KERNEL_CONFIG_FLAVOR=rt` configuration
- Built from the same branch as generic, but with RT-specific config
- Provides deterministic latency for time-critical applications

### Generic Platforms

- Standard mainline kernel with EVE patches
- Used for most hardware platforms
- Optimized for virtualization and container workloads

## See Also

- [BUILD.md](BUILD.md) - General EVE build instructions
- [CUSTOM-BUILD.md](CUSTOM-BUILD.md) - Customizing EVE components
- [EVE-IMAGE-SOURCES.md](EVE-IMAGE-SOURCES.md) - Source tracing and provenance
- [DEBUGGING.md](DEBUGGING.md) - Kernel debugging techniques
- [KERNEL-DUMPS.md](KERNEL-DUMPS.md) - Capturing and analyzing kernel dumps

## Additional Resources

- EVE Kernel Repository: <https://github.com/lf-edge/eve-kernel>
- Linux Kernel Documentation: <https://www.kernel.org/doc/>
- Kernel Newbies: <https://kernelnewbies.org/>