GPU as a product: Fair-share GPUs with preemption on Kubernetes

Zaid Fakhruddin presents a practical blueprint for "GPU as a product" for platform teams supporting GPU-backed AI/ML workloads: predictable access, high utilization, and explainable spend.

He walks through how to build a fair-share GPU platform on Kubernetes:
• Define 3 GPU SKUs (interactive, batch, training) with SLAs (TTL, checkpoint required, preemptible)
• Model tenants with Kueue ClusterQueues, cohorts, weights, nominal quota, and fair-sharing preemption
• Add hard guardrails: PriorityClasses, admission policies, and quota and budget caps per team
• Reduce fragmentation with MIG where it helps, and measure it with DCGM metrics

Takeaways include the exact Kubernetes objects and policies to create, the 5 metrics to track (queue time, utilization, fragmentation, job completion time, cost per team), and a rollout plan that platform teams can ship.