tl;dr
The TES spec doesn’t define how container images should behave during task execution—leaving security sandboxing entirely to implementors. This can lead to failures (e.g., tools needing root access) or serious security risks (e.g., images trying to bind ports, run syscalls, or tamper with the cluster).

In Poiesis, I hit this when an execution container failed due to permission errors. My current fix uses Kubernetes securityContext to block dangerous capabilities, but this feels fragile and ad hoc.

Problem

The TES specification currently does not formally define or restrict the kind of container images that can be used for task execution. While this is understandable—TES aims to stay implementation-agnostic and flexible—this flexibility creates real-world problems when running untrusted or poorly constructed container images in secure environments, particularly in Kubernetes-based deployments.

One key concern is that TES implementors are implicitly expected to sandbox execution, but without any guidance or enforcement mechanism. For example, nothing in the TES spec prevents a user from specifying a container image that:

• Installs tools at root (/) without proper permissions
• Assumes root access for execution
• Attempts to make privileged syscalls (e.g., mounting filesystems, binding to low-numbered ports, etc.)
• Tries to perform unauthorized network operations

This can result in tasks failing silently due to permission issues, or worse—security vulnerabilities in shared or multi-tenant environments.

Real-World Example (Poiesis TES on Kubernetes)

While developing [poiesis](https://github.com/jaeaeich/poiesis), a TES implementation running on Kubernetes, I encountered a concrete example of this issue.

I ran a Nextflow workflow using a custom container image. Though the TES request passed validation, the image had tools copied to / and required root or escalated permissions to run. Since Kubernetes pods by default were not sandboxed via securityContext, these images could previously execute with broad capabilities. After tightening security using Kubernetes' securityContext, the same workflow failed due to permission errors.

TES lacks the concept of sandboxing or even guidance around it, leaving every implementation to handle it ad-hoc.

Security Risk

In cluster environments, particularly in research institutions or shared clouds, TES becomes a potential vector for privilege escalation or cluster compromise unless sandboxing is explicitly and securely handled.

A malicious actor could:

• Bind to network interfaces
• Attempt port scans or exfiltration via raw sockets
• Use kernel-level capabilities to affect host or node-level settings
• Mount system filesystems or tamper with runtime state

Current Mitigation (Work-in-Progress Sandbox Approach)

In poiesis, I’ve opted for a sandboxing model that:

• Allows users to run arbitrary images with minimal friction
• Restricts dangerous syscalls and capabilities using Kubernetes security policies

Example securityContext:

securityContext:
  runAsNonRoot: false
  allowPrivilegeEscalation: true
  runAsUser: 1001
  runAsGroup: 1001
  privileged: false
  capabilities:
    drop:
      - "SYS_ADMIN"
      - "SYS_MODULE"
      - "SYS_PTRACE"
      - "SYS_TIME"
      - "NET_ADMIN"
      - "NET_RAW"
      - "NET_BIND_SERVICE"
      - "NET_BROADCAST"

This blocks most cluster-impacting actions while still allowing most tools to run. Additionally, TTL and culling policies are used to terminate long-running or suspicious jobs early.

However, this feels fragile. It’s easy to miss a dangerous capability or overlook a security regression. Every implementor might invent their own partial sandbox, leading to inconsistencies and potential vulnerabilities.

Proposed Paths for Discussion

1. TES Extension for Execution Policy Metadata

   • Introduce an optional field in the TES spec allowing users or admins to declare what execution restrictions are expected (e.g., "requires root", "needs elevated net access", etc.)
   • This could be a hint for the scheduler or executor to deny/allow a task or apply a stricter profile.

2. TES Implementation Guidelines

   • Publish a community document (linked from the main TES repo) outlining best practices for securely running containers.
   • Could include example Kubernetes securityContext, runtime sandboxing options (gVisor, Kata containers), and failure-handling strategies.

3. Validation Layer / Pre-execution Checks

   • Build a standard plugin or utility that inspects container images (e.g., via trivy, docker inspect, or crane) before executing them.
   • It could reject images that run as root, lack a defined ENTRYPOINT, or contain known dangerous packages (like netcat or tcpdump).

4. Network Sandboxing

   • Suggest that TES implementations, especially in shared environments, default to dropping network access unless explicitly requested.

Questions for the Community

• Has anyone else encountered similar issues with user-submitted images?
• Are there existing efforts or patterns in TES implementations for sandboxing?