Job Description & Details
This is a heavy-duty infrastructure role for someone who knows how to wrangle massive clusters without breaking a sweat. You'll be bridging the gap between traditional HPC workloads like Slurm and modern cloud-native orchestration with Kubernetes while pushing heavy-duty NVIDIA GPUs to their absolute limits for AI training and inference. If you enjoy solving gnarly networking and scheduling bottlenecks at scale, this gig is worth a serious look.
What You'll Actually Be Doing
Expect to spend your days designing, deploying, and troubleshooting large-scale Kubernetes platforms that power serious AI/ML firepower. You will be managing cluster upgrades, tweaking container network interfaces, and ensuring that expensive GPU nodes aren't sitting idle due to poor scheduling or storage latency. When things inevitably break at 500+ node scale, you'll be the one diving into the metrics to figure out whether it's an InfiniBand fabric issue, a Lustre filesystem timeout, or a misconfigured Karpenter autoscaler.
The Core Tech Stack
You absolutely cannot fake your way through this without deep, production-level experience in Kubernetes administration and Slurm. The infrastructure relies heavily on AWS services like EKS and FSx for Lustre, paired with Infrastructure as Code via Terraform to keep deployments reproducible. Python automation is a must for scripting daily fixes, and you need a solid grasp of NVIDIA GPU infrastructure, CUDA, and RDMA to make sure distributed training jobs actually perform as expected.
Interview Expectations
Be ready for deep-dive troubleshooting scenarios where an interviewer drops a hypothetical cluster failure in your lap, such as a multi-node distributed training job stalling due to silent InfiniBand packet drops or CNI misconfigurations. They want to see your systematic approach to observability using Prometheus and Grafana, and how you isolate hardware failures from software issues under pressure. They'll also likely grill you on advanced Kubernetes scheduling and how you handle node management at scale without dropping active workloads.
Application Advice
Your resume needs to scream scale immediately. Don't just list Kubernetes and Terraform; explicitly mention the node counts you've managed, the size of the GPU clusters you've supported, and specific instances where you optimized distributed training performance. Weave in keywords like Slurm, FSx for Lustre, Karpenter, and Cilium naturally into your past experience bullets so the ATS flags you as a heavy infrastructure specialist right out of the gate.