Job Description & Details
This is a heavy-duty data engineering contract supporting the Indiana Department of Health's Office of Data & Analytics. You will be building and optimizing large-scale data pipelines on Azure, dealing with sensitive public health data while bridging the gap between Databricks lakehouse patterns and Azure Synapse Analytics.
What You'll Actually Be Doing
Your day-to-day will revolve around architecting, deploying, and tuning medallion-style data architectures (bronze, silver, gold layers) using PySpark and Spark SQL inside Azure Databricks. You'll spend a lot of time writing Delta Lake transactions, setting up Auto Loader for incremental ingestion, and ensuring data governance via Unity Catalog to protect PII/PHI. Expect to troubleshoot shuffling and partitioning issues in the Spark UI, optimize sluggish Synapse SQL pools, and orchestrate end-to-end pipelines utilizing Azure Data Factory and Databricks Workflows.
The Core Tech Stack
You absolutely need hands-on production experience with Azure Databricks, PySpark, Spark SQL, and Delta Lake fundamentals like ACID transactions, MERGE statements, and OPTIMIZE/Z-order maintenance. Beyond Databricks, a deep understanding of Azure Synapse Analytics (dedicated/serverless pools, MPP tables, distribution strategies), ADLS Gen2, and orchestration tools like ADF or Databricks Workflows is mandatory. Strong CI/CD practices using Git and Azure DevOps/GitHub are also non-negotiable since you'll be managing Databricks Asset Bundles.
Interview Expectations
Expect the hiring team to grill you on Spark optimization strategies, particularly how you diagnose and resolve data skew, memory bottlenecks, and inefficient joins using the Spark UI. They will also likely ask you to walk through a scenario where you design a medallion architecture for streaming data, focusing on how you handle schema evolution and late-arriving data. The interviewer is secretly looking for deep, practical troubleshooting experience rather than just textbook definitions, wanting to see if you can debug live pipeline failures under pressure.
Application Advice
To get past the ATS for this role, ensure your resume heavily features exact keyword matches for "PySpark", "Delta Lake", "Unity Catalog", and "Medallion Architecture". Don't just list technologies; write descriptive bullet points highlighting how you optimized cluster performance, reduced query runtimes in Synapse, or managed incremental ingestion using Auto Loader. Since this is a local hybrid role in Indianapolis, make sure your resume clearly states your current local residency right at the top.