Job Description & Details
This role puts you right in the middle of modernizing data infrastructure for public sector programs, focusing heavily on building out resilient cloud-native data pipelines. If you enjoy the nitty-gritty of distributed processing, optimizing Spark jobs, and managing infrastructure-as-code without the usual startup chaos, this is a solid gig. You will be dealing with real-scale data transformations, production operations, and mentoring engineers who want to level up their craft.
What You'll Actually Be Doing
Your day-to-day will involve designing, building, and babysitting heavy-duty data pipelines that ingest structured and unstructured data via batch, streaming, APIs, and Change Data Capture (CDC). You are going to spend a good chunk of your time troubleshooting why a Spark job is dragging, rewriting inefficient queries with better partitioning strategies, and tightening up CI/CD automation. Aside from writing clean Python and SQL, you'll collaborate closely with data governance and product teams to make sure lineage, quality checks, and security guardrails aren't an afterthought.
The Core Tech Stack
You need to be completely fluent in Python, SQL, and Apache Spark for distributed data processing. On the infrastructure side, deep familiarity with major cloud ecosystems like Azure (ADLS, ADF, Databricks) or AWS is non-negotiable, along with Infrastructure-as-Code using Terraform or Bicep. Orchestration tools like Airflow or Databricks Jobs will be your best friends for keeping workflows running smoothly. Experience with modern analytics engineering tools like dbt and Delta Lake is a massive plus, especially since performance tuning and observability are baked into everything you touch here.
Interview Expectations
Expect the hiring team to hand you a scenario where a production streaming pipeline is dropping messages and spiking cloud costs, and they'll want to hear how you triage, debug, and optimize it under pressure. They are secretly looking to see if you understand the underlying mechanics of distributed computing, like memory management in Spark and smart partitioning strategies, rather than just memorized syntax. They'll also probe your architectural mindset around CI/CD automation, testing data quality, and enforcing security controls like RBAC in a cloud environment.
Application Advice
When updating your resume for this position, ditch the generic bullet points and explicitly highlight your hands-on metrics around data pipeline optimization and cost reduction. Make sure keywords like Spark, Python, Terraform, CI/CD, and data lineage are prominent so you easily clear the ATS screening. Since this role sits in a public sector environment, emphasizing your experience with rigorous data governance, cross-functional collaboration, and maintaining production SLOs will instantly catch the hiring manager's eye.