Job Description & Details
This is a heavy-duty platform and observability gig where you won't just be writing alerts—you'll be architecting and scaling massive enterprise telemetry pipelines. If you enjoy wrangling massive Splunk clusters, Elasticsearch nodes, and modern distributed tracing stacks like OpenTelemetry and Grafana Tempo, this contract will keep you deeply challenged.
What You'll Actually Be Doing
Your day-to-day will revolve around keeping enterprise-scale observability platforms fast, reliable, and actually useful for engineering teams. You'll spend a good chunk of your time tuning Splunk environments, optimizing SPL queries, scaling Elasticsearch clusters, and setting up end-to-end distributed tracing using OpenTelemetry and Grafana Tempo. When things break or pipelines get bottlenecked by massive log volume, you'll be the one diving into Kafka streams, fixing Terraform scripts, and automating the infrastructure back to health.
The Core Tech Stack
They need someone who genuinely knows their way around Splunk Enterprise or Cloud administration—we're talking Search Head Clusters, Indexers, and Heavy Forwarders, not just basic querying. Beyond Splunk, you must have solid chops in Elasticsearch, Prometheus, Grafana, and Kafka. Modern observability is a core pillar here, so deep hands-on experience with OpenTelemetry and Grafana Tempo for distributed tracing is non-negotiable. Tie all of that together with Terraform for Infrastructure as Code and some Python, Go, or Bash for scripting, and you've got the exact toolkit this team is running on.
Interview Expectations
Expect the hiring manager to throw a heavy architecture scenario at you, likely asking how you would design a multi-region distributed tracing pipeline using OpenTelemetry and Grafana Tempo while keeping storage costs and ingestion latency under control. They are secretly looking to see if you understand the actual performance trade-offs of heavy instrumentation versus network overhead. You'll also probably get grilled on advanced Splunk SPL optimization or how you'd troubleshoot a cascading failure in a massive Elasticsearch cluster where queries are timing out.
Application Advice
Skip the generic DevOps buzzwords and make sure your resume explicitly calls out your enterprise-level experience with Splunk, Elasticsearch, and OpenTelemetry. ATS filters are going to scan heavily for exact matches on Terraform, Kafka, and distributed tracing, so weave those terms naturally into your project achievements rather than hiding them in a skills matrix. Highlight any past work where you successfully scaled observability pipelines or reduced noise in alerting systems, as that's the exact pain point they are paying top dollar to solve.