Cloud Operating Systems: Minik8s Lab 2025

For this final project, you are required to build a lightweight container orchestrator called Minik8s. It must manage CRI-compatible containers across multiple hosts, supporting container lifecycle management, dynamic scaling, container-to-container networking, and automatic scaling. On top of these core capabilities, each team must implement an elective feature chosen from either (1) a MicroService framework or (2) a Serverless platform integration. You are encouraged, though not obliged, to leverage existing infrastructure such as etcd, ZooKeeper, or RabbitMQ, and to base your runtime on containerd; Docker is also acceptable if you prefer.

The assignment specifies only the functional specs; you may consult the lecture slides, follow Kubernetes-style patterns, or invent and extend designs entirely on your own.

Group Assignments

Basic Features

Implement Pod Abstraction to Manage Containers' Lifecycle

Minik8s must provide a Pod abstraction.

• Users define and launch Pods through YAML. The manifest must support:
  • kind: object type -> Pod
  • name: Pod identifier
  • container image name and image tag
  • container entry-point command and args
  • volume: shared volumes
  • port: exposed network ports
  • resource requests (e.g., 1 CPU, 128 MB memory)
  • namespace
  • labels
• Built on this abstraction, Minik8s handles the full Pod lifecycle: start and stop Pod on user command.
• Fault-tolerance: if any container in a Pod crashes, Minik8s should automatically restart it.
• Visibility: a single command lists every Pod with at least:
  • Pod name
  • status (and uptime if running)
  • namespace
  • labels

Implement CNI to Support Inter-Pod Communication

Minik8s must enable inter-Pod communication. We recommend adopting the Kubernetes model: implement the feature as a CNI plugin (e.g., Flannel; Weave is discouraged because it is no longer maintained). Required capabilities:

• Assign a unique internal IP to every Pod at creation time.
• Allow Pods to reach one another directly, within the same node or across nodes, using these IPs via the CNI plugin.

Implement Service Abstraction

Based on CNI, Minik8s must provide a Service abstraction.

A Service acts as a front-end proxy that aggregates the network endpoints of a set of Pods and presents a single, stable virtual IP. Users reach the Service by this IP; Minik8s components then forward traffic to the appropriate Pods.

Mandatory Service types to support:

1. ClusterIP – cluster-internal access only.
2. NodePort – additionally exposes the Service on a static port of every node so that external clients can reach it.

Functional requirements:

• YAML creation – users create Services via YAML. The manifest must support:
  • kind: Service
  • type: ClusterIP or NodePort
  • name: Service identifier
  • selector: label query to pick Pods
  • ports: – port: VIP port inside the cluster – targetPort: port the Pod actually listens on – nodePort: node-wide port (NodePort type only)
  • namespace
  • labels
• Traffic forwarding – Minik8s must forward packets sent to the virtual IP to the selected Pods; implementation should leverage Linux IPVS or iptables.
• Load balancing – When multiple Pods match, traffic must be spread across them (random or round-robin).
• Dynamic endpoints – Pods that die are removed immediately; newly created Pods that match the selector are added automatically.
• Deletion – a single user command deletes the Service and all associated state.
• Visibility – minik8s get service displays at least:
  • Service name
  • Virtual IP, port, targetPort
  • Endpoints (real Pod IPs + ports)
  • namespace
  • labels

Implement ReplicaSet Abstraction

Minik8s must support ReplicaSets for declarative Pod management.

A ReplicaSet uses label selectors to “own” a set of Pods and continuously reconciles reality to the user-defined replica count.

Required capabilities:

• YAML creation – users declare a ReplicaSet via YAML. Mandatory fields:
  • kind: ReplicaSet
  • name: ReplicaSet identifier
  • selector: label query that picks managed Pods
  • replicas: desired number of identical Pod instances
  • namespace
  • labels
• Self-healing – if live Pods exceed the desired count, the ReplicaSet deletes the excess; if too few exist, it creates new ones.
• Deletion – a single command removes the ReplicaSet and every Pod it owns.
• Visibility – minik8s get replicaset shows at least:
  • ReplicaSet name
  • desired and current replica counts, expected output format: current counts / desired counts
  • namespace
  • labels

Resource Monitoring and Auto Scaling

Beyond the fixed replica count of a ReplicaSet, Minik8s can also elastically scale Pods up and down through an Auto-Scaling ReplicaSet abstraction that reacts to real-time load.

Functional requirements:

• Resource monitoring: Minik8s must periodically collect per-Pod utilization metrics for at least two resource types, CPU (mandatory) plus one optional resource type (memory, memory bandwidth, I/O, network, etc.). We recommend reusing existing collectors, such as cadvisor.
• YAML-driven configuration: Users declare autoscaling behavior in a manifest with the following fields.
  • kind: HorizontalPodAutoscaler
  • name: autoscaler identifier
  • targetWorkload: points to the scaled workload (Pod, ReplicaSet, or Deployment)
  • minReplicas / maxReplicas: hard limits on replica count (max ≥ min)
  • metrics: resource type + target value or utilization percentage (e.g., CPU 60 %, memory 500 Mi). At least two resource types should be supported by the system; a single manifest may specify one or both.
• Scaling policy: The autoscaler must implement the following features.
  • When to scale (trigger conditions, cooldown windows)
  • How fast to scale (step size & rate, e.g., “add one replica every 15 s”)

DNS and Traffic Routing

Minik8s must allow users to assign human-readable hostnames and sub-paths to any Service through a YAML manifest. Once configured:

• Clients inside the cluster can reach the Service by domain name alone—no need to remember virtual IPs or ports.
• The same parent domain can map multiple sub-paths to different Services

While the idea is inspired by Kubernetes Ingress, the exact schema and implementation are not required to mirror Ingress; you are free to define your own YAML format and routing logic as long as the two bullets above are satisfied.

Example: After a user creates Service1 and Service2, they are reachable at Service1-IP: Port and Service2-IP: Port, respectively. Next, the user creates a DNS-config object that:

1. Declares the parent domain example.com.
2. Maps sub-path /path1 -> Service1 and /path2 -> Service2.

From that moment on, both human users and Pods inside the cluster can call:

• example:80/path1 (routed to Service1)
• example:80/path2 (routed to Service2)

The experience is equivalent to contacting the original Service1-IP: Port and Service2-IP: Port, but only the friendly domain name is ever exposed.

DNS Functional Requirements:

• Users can create a DNS configuration object via YAML with the following fields:
  • kind: object type -> DNS
  • name: identifier of the DNS object
  • host: parent domain (e.g., example.com)
  • paths: list of sub-path rules; each rule contains:
    • the path prefix
    • target Service name
    • target Service port
  • namespace
  • labels
• A single command can delete the DNS object and clean up all associated state.
• Once created, the configuration is visible cluster-wide: both end-users and Pods can reach Services through the declared domain and sub-paths; multiple Services can share the same parent domain differentiated only by path.
• Visibility: minik8s get dns displays at least:
  • host
  • paths
  • namespace
  • labels

Multi-Machine Deployment

Minik8s must orchestrate containers across at least three physical or virtual machines. The following capabilities are required:

• Node abstraction:
  • Every machine is modeled as a Node.
  • The control-plane Node starts first and acts as the cluster entry point.
  • Additional worker Nodes join by posting a YAML manifest (fields designed by yourself) to the control plane’s API endpoint, such as the API-Server in Kubernetes.
  • minik8s get node lists each Node’s name, role (control-plane / worker), and status (Ready / NotReady).
• Scheduler: When a Pod is created, the scheduler is consulted to decide which Node will control this new Pod. You may implement:
  • A trivial policy (round-robin or random)
  • A slightly smarter policy that respects resource requests.
  • Even the simplest working scheduler earns > 80 % of this item’s credit.
• Location transparency: The Service and DNS abstractions must hide the Pods' physical location; clients reach the Pod through the Service’s stable virtual IP or domain name, regardless of which Node hosts it.
• Cluster-wide ReplicaSet & AutoScaling: Both ReplicaSets and the Horizontal Pod Autoscaler should spread all replicas across the entire cluster.

Fault-Tolerance

Support Fault-Tolerance for Minik8s, the required functionalities are listed as follows:

• If the control plane crashes, every Pod already running on worker nodes must stay up and healthy
• After the ·Minik8s· control plane restarts, deployed Pods, ReplicaSets, Services, and DNS objects can be restored, and their status can be updated to the status before the crash.
• There should be heartbeat communication between the Minik8s control plane and data plane, so the control plane can detect whether a data plane Node is alive; when a data plane Node crash is detected, new Pods should avoid being scheduled there until the Node recovers and rejoins the cluster.

Optional Features

In addition to the basic features, each team must implement at least one optional feature.

At the final demo, only one optional feature will be graded; if multiple are implemented, the team must choose which one to present.

Note: the optional feature is not extra credit. It is part of the 100-point total.

MicroService

Extend Minik8s networking so that, at control-plane startup, users may choose a Service Mesh architecture instead of the classic Service model. This architecture gives fine-grained traffic control without changing application code, enabling MicroService operation (You can refer to the implementation of Istio). Concrete requirements are as follows:

• Traffic hijack: adopt a SideCar pattern: every Pod’s inbound/outbound traffic is intercepted by a proxy SideCar.

• Service registry: expose a virtual IP for each Service; external clients hit this Virtual IP, and the mesh forwards to the correct Pod.

• Automated service discovery: the Service Mesh must automatically detect every deployed Service and Pod, notify the network proxy inside each Pod, and ensure hijacked traffic is routed correctly; when a Pod’s IP changes, the registry updates itself and the sidecar proxies reconfigure so the service remains reachable (users can access it and services can call each other).

• Use SideCars to deliver advanced traffic-control features:

  • Rolling upgrade: via a custom CLI, instruct the mesh to shift traffic gracefully so a Service can be upgraded Pod-by-Pod with zero downtime.
  • Canary release: expose a custom API where users declare rules that split traffic for the same Service across different Pods; rules must support both percentage-based weights and regex matches on HTTP headers or URLs.

• Adopt or build a non-trivial MicroService application that:

  • Solves a realistic problem (e.g., the image-processing pipeline shown below); trivial arithmetic or “Hello World” is not acceptable.

  • Incorporates model-based workloads—small models for image processing, text generation, classification, etc.; large models are not required.

  • Deploy the application on Minik8s and use it to demonstrate the MicroService features above; in the final report, briefly explain why a MicroService architecture is necessary and advantageous for this scenario.

  • Example codebases may be drawn from GitHub - delimitrou/DeathStarBench: Open-source benchmark suite for cloud microservices , or GitHub - kubeshark/sock-shop-demo: Sock Shop example with additions

  • A working implementation of the basic functionality earns full base marks; extra credit is available for richer complexity and practical utility.

    We recommend finding suitable models from HuggingFace , such as https://huggingface.co/openai-community/gpt2，https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0

Serverless

Build a Serverless platform on top of Minik8s that:

• Executes user code at function granularity
• Scales to zero when idle and automatically scales up on demand
• Supports function chaining and communication between functions in a chain

You are encouraged to study existing open-source platforms such as Knative or OpenFaaS for design guidance. Detailed requirements are as follows:

• Support Function abstraction. Users define a function in a single file (either source code or a zip archive), upload it to Minik8s with a CLI command, and invoke it through an HTTP trigger and an event trigger.

  • Functions must support Python at a minimum.
  • The layout of the function, the format of its return value, and the syntax of the update/invoke commands may be defined by the implementation.
  • Invocation methods: expose the function via HTTP requests and allow it to be bound to events. Users specify the event source through a command; event types may include cron schedules, file modifications, or any custom source.

• Support the Serverless Workflow abstraction: users can define a serverless DAG that contains the following elements:

  • Function chain: invoke the first function with arguments, then call each subsequent function in sequence, using the output of the previous function as the input to the next, and return the final result.
  • Conditional branch: the control plane decides which branch to execute next based on the output of the preceding function.
  • Serverless Workflows are declared in a configuration file; follow AWS Step Functions or Knative, or invent your own programming model—any approach that satisfies the above behavior is acceptable.

• Serverless autoscaling with scale-to-zero

  • Instances are created on the first incoming request (cold start) and deleted after a prolonged idle period (scale-to-zero). The platform must also monitor request volume and automatically scale out to >1 instance when traffic rises.
  • The implementation must gracefully handle tens of concurrent requests; demonstrate this with wrk2 or JMeter during the presentation.

• Find or build a non-trivial Serverless application that:

  • Solves a realistic problem (e.g., the image-processing pipeline shown below); trivial arithmetic or “Hello World” is not acceptable.

  • Uses model-based workloads—small models for image processing, text generation, classification, etc.; large models are not required.

  • Deploy the application on Minik8s and use it to demonstrate serverless features; in the final report, briefly explain why a serverless architecture is necessary and advantageous for this scenario.

  • A sample reference is the Alexa skill in ServerlessBench/Testcase4-Application-breakdown/alexa at master · SJTU-IPADS/ServerlessBench .

  • A working implementation of the basic functionality earns full base marks; extra credit is available for richer complexity and practical utility.

    We recommend finding suitable models from HuggingFace , such as https://huggingface.co/openai-community/gpt2，https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0

Individual Assignments

Persistent Storage

Minik8s must provide persistent storage (refer to the Persistent Volumes documentation). When a Pod (e.g., running a database like MySQL) crashes or is deleted, its application data must survive and be mountable by any replacement Pod. Required capabilities are as follows:

1. PV / PVC abstractions: Implement both PersistentVolume (PV) and PersistentVolumeClaim (PVC) and clearly reflect their different roles and use-cases.
2. Support two provisioning modes
   1. Static: cluster admin pre-creates PVs.
   2. Dynamic: Minik8s creates a suitable PV automatically when a PVC is submitted.
3. Support at least two PV types
   1. hostPath (simple, single-node only).
   2. At least one type that can be accessed from multiple nodes (See item 5).
4. Support Basic PV/PVC life-cycle A Pod obtains storage via a PVC. Deleting the Pod releases the PVC-to-PV bond but retains the data. A new Pod that rebinds the same PVC (or an equivalent one) must see the previous data. Data becomes inaccessible only when the PV itself is deleted.
5. Multi-node persistent storage (demo required) Show that Pods scheduled on different nodes, one after another, can successfully bind to and read/write the same PV.
   1. Hint: deploy a dedicated storage node and export volumes through NFS (or similar).
   2. A plain single-node NFS setup is acceptable, but scores lower; more sophisticated, fault-tolerant storage back-ends are encouraged and will receive higher marks.

GPU Application Support

The course will grant access to the “Jiao Wo Suan” super-computing platform, which uses Slurm for job scheduling (guide: https://docs.hpc.sjtu.edu.cn/job/slurm.html). Requirements are as follows:

1. Minik8s allows users to write GPU programs (e.g., CUDA) and handle submission to “Jiao Wo Suan” for compilation and execution.
2. Users supply only the CUDA source and a build script, and submit this job to Minik8s using a YAML configuration file. An internal Minik8s service uploads the code to the platform, runs it, and returns results to the user.
3. Jobs must be isolated: different job names trigger different service processes when submitted to Slurm. (Hint: mimic a Kubernetes Job to package the GPU code in a Pod that contains the submission service.)
4. The YAML carries the usual fields (name, kind, …) plus Slurm-aligned job parameters; exact fields are left to the implementation.
5. Implement at least matrix multiplication and matrix addition in CUDA, exploiting hardware concurrency. During the presentation, explain how you use the CUDA parallelism.
6. Basic CUDA earns full base marks; richer concurrency or more sophisticated CUDA code yields bonus points.

Security Context Support

Minik8s must implement Security Context (see official Security Context documentation: “Set the security context for a Pod”) to configure container privileges and reduce unnecessary access. Requirements are as follows:

1. Pod-wide Security Context (same for every container in the Pod)
   1. runAsUser: set UID for all container processes. If this field is omitted, UID = 0 (root).
      1. Demo for presentation: ps on the host must show the process UID equal to runAsUser.
   2. runAsGroup: set GID for all container processes. If this field is omitted, GID =0 (root).
      1. Demo for presentation: every container in the Pod shows the same GID, matching runAsGroup.
   3. fsGroup: set GID for all files created in volumes mounted by the Pod.
      1. Demo: ls -l on a newly created file inside the volume shows the fsGroup GID.
2. Per-Container Security Context (overrides Pod-wide values)
   1. Must support at least runAsUser, runAsGroup, and fsGroup.
   2. Demo for presentation: settings apply only to the target container; other containers remain unchanged.
   3. Demo for presentation: per-container values take precedence over Pod-wide values.
3. Bonus – implement a supplementalGroupsPolicy equivalent (see official documentation Supplemental Groups Control , a related example is provided in Security Context documentation: "Configure fine-grained SupplementalGroups control for a Pod" part) supporting both Merge and Strict policies.

Correct implementation of items 1–2 earns full base marks; item 3 yields bonus points.

Assessment Method

This lab is highly open-ended: the implementation language and approach are not prescribed (though following Kubernetes patterns is strongly recommended). Assessment is mainly through a final presentation that verifies functionality, performance, and reliability.

• Group Assignments: Required features are split into group and individual tasks. Each team divides the group assignments internally and presents together; the team receives a collective score adjusted by individual contribution.
• Individual Assignments: Every member must implement one personal feature. Each student presents this feature alone and is graded separately.

Staged Assessment

The lab is completed in multiple iterations, each followed by a checkpoint presentation: one mid-term and one final.

• Mid-term Presentation (not graded) – let TAs review your progress and give feedback. After this presentation, you must submit a short report summarising the current status.
• Final Presentation (graded) – you must implement and clearly demonstrate every required feature.
• Grading breakdown:
  • Functional requirements & representation performance: 80%
  • Engineering quality: 20%

Engineering Requirements

• Zero tolerance for plagiarism – any detected duplication (vs. current or past submissions) earns 0 marks for everyone involved!
• 3-person teams, self-organized; elect one team leader.
• Clear internal division of labor: – At kickoff, list per-iteration tasks and owner names. – Mid-term Presentation must show what was actually finished, who did it, and whether the schedule is on track; adjust next-iteration plans as needed.
• The final report must state each member’s percentage contribution.
• Open-source workflow: – Every feature lives in its own Git branch; merge only via pull request .
• CI/CD: – PRs must pass automated tests before merging; commits on the main branch auto-deploy. – Any CI/CD stack is allowed (Travis, Jenkins, GitHub Actions, GitLab CI, etc.).
• Best Practice examples:
  • JavaScript Project Best Practice: https://github.com/elsewhencode/project-guidelines
  • Go Project Layout: https://github.com/golang-standards/project-layout