Job Description & Details
This is a heavy-duty data engineering role focused entirely on building out modern analytics infrastructure on Azure and Databricks. If you enjoy wrangling massive datasets, setting up Medallion architectures, and optimizing Spark jobs rather than just writing basic CRUD queries, this position offers a great chance to own the data lifecycle end-to-end.
What You'll Actually Be Doing
Expect to spend your days knee-deep in pipeline development, migrating legacy data sources, and keeping streaming workflows healthy on the Databricks Lakehouse Platform. You will collaborate closely with product owners and source application teams to ingest data via Azure Data Factory and Event Hubs, map it through Bronze, Silver, and Gold layers using Delta Lake, and ensure everything stays performant and cost-efficient. Production support and troubleshooting flaky CI/CD pipelines via GitHub Actions will also take up a solid chunk of your sprint.
The Core Tech Stack
You need absolute fluency in Azure cloud services and Databricks. This means deep, practical experience with PySpark or Scala, optimizing Spark jobs, and managing Delta tables. SQL and dimensional data modeling are non-negotiable because you'll be shaping raw enterprise data into clean, curated datasets for business KPIs. Knowing your way around Azure Data Factory, ADLS, and GitHub Actions for CI/CD automation will make or break your day-to-day execution here.
Interview Expectations
Expect the interview panel to drill down into your Spark optimization skills by asking you how you would debug a slow-running PySpark job suffering from data skew or excessive shuffle partitions. They want to see how you reason about cluster sizing, broadcast joins, and caching strategies. You will also likely be tested on your architectural design chops regarding the Medallion architecture—be prepared to defend how you handle schema evolution, idempotency, and data quality checks in the Bronze-to-Silver transition.
Application Advice
Your resume needs to make it immediately obvious that you've built enterprise-scale data lakes on Azure, not just small scripts. Front-load keywords like Azure Databricks, Delta Lake, PySpark, Medallion Architecture, and GitHub Actions in your experience bullets. If you hold a Databricks Certified Data Engineer Associate or Professional certification, put that right at the top, as it's explicitly called out as a strong preference for this team.