Flyte is a K8s-native, multi-tenant, distributed orchestration engine that manages highly scalable compute workflows.

Extensibility is provided by Flyte plugins and K8s operators, including executors for Spark, Hive, AWS Batch, in addition to raw K8s compute. Flyte can orchestrate arbitrary containers at scale, providing feature fanout, automatic task parallelization, strongly-typed paramerter passing, automatic cluster scale up and scale down, cluster fail-over, and fine-grained programmatic control over retry policies, error handling, and notifications.

See docs at https://lyft.github.io/flyte/