Job Description & Details
This is a heavy-duty 12-month contract focused entirely on building and scaling enterprise observability platforms for large-scale cloud infrastructure. If you enjoy wrangling massive Splunk deployments, Elasticsearch clusters, and distributed tracing pipelines, this role puts you right at the architectural helm. Plus, you will be operating within strict FedRAMP High or IL-5 environments, meaning security and compliance rigor are non-negotiable.
What You'll Actually Be Doing
Expect to spend your days knee-deep in telemetry data, designing and maintaining platforms for logging, metrics, tracing, and alerting. You won't just be maintaining existing dashboards; you will be architecting end-to-end tracing pipelines, standardizing instrumentation with OpenTelemetry, and scaling out complex clusters of Prometheus, Kafka, and Grafana Tempo. Automation is a massive part of the gig, so you will be writing Terraform constantly to provision and manage your infrastructure cleanly.
The Core Tech Stack
Your bread and butter here will be Splunk Enterprise or Splunk Cloud, paired with deep expertise in Splunk SPL for building out high-performance queries and alerts. You also need heavy production experience with Elasticsearch, Prometheus, Grafana, and distributed tracing via OpenTelemetry and Tempo. Because this role touches regulated government environments, knowing your way around Linux, Kubernetes, and IaC tools like Terraform will make or break your ability to survive the scale and compliance demands.
Interview Expectations
When you sit down with the hiring team, expect them to probe deeply into how you handle blast radius and failure modes in massive telemetry pipelines. They will likely ask you how you would troubleshoot a scenario where an Elasticsearch cluster is dropping writes or a Kafka topic is backing up during a traffic spike. The interviewer is secretly trying to figure out if you panic when distributed systems degrade or if you methodically isolate bottlenecks using metrics and tracing data without guessing.
Application Advice
Your resume needs to scream observability scale right at the top. Don't just list tools; highlight the size of the clusters you have managed, the volume of logs or traces you have handled per second, and your specific experience with Splunk and OpenTelemetry. Ensure keywords like 'FedRAMP', 'Terraform', 'Splunk SPL', and 'Distributed Tracing' are woven naturally into your experience bullets so you easily clear the ATS and catch the engineering manager's eye.