Job Description & Details
This Lead Data Engineer position sits inside a major media streaming ecosystem, where you will be tasked with wrangling massive volumes of real-time viewer data. It's a high-visibility technical role where your pipeline architecture directly impacts how business analytics and emerging AI models consume terabytes of daily event data. If you enjoy solving heavy distributed computing problems and want to own the data lifecycle from ingestion to consumption, this one is worth a serious look.
What You'll Actually Be Doing
Your day-to-day will heavily revolve around designing, building, and maintaining scalable batch and streaming data pipelines using the Medallion Architecture. You aren't just writing basic SQL scripts; you will be architecting robust Delta Lake environments, optimizing expensive Spark jobs, and enforcing strict data quality and observability across Unity Catalog. Expect to spend a solid chunk of your time troubleshooting production bottlenecks, conducting thorough code reviews, and mentoring mid-level engineers who are still mastering distributed systems.
The Core Tech Stack
You absolutely must have recent, hands-on production experience with Databricks, Python, PySpark, and advanced SQL. The hiring team is looking for someone who doesn't just know how to run a notebook, but truly understands Databricks Lakehouse internals, Delta Lake optimization, and workflow orchestration. Having AWS experience, Auto Loader, or structured streaming in your back pocket will immediately push your resume to the top of the stack, as they deal with heavy clickstream and media ingestion patterns.
Interview Expectations
Be prepared to defend your choices around Spark performance tuning, specifically how you would diagnose and fix a massive shuffle read bottleneck or memory leak on a multi-node cluster. The interviewer is secretly testing whether you've actually hit production scaling walls or if you've only built toy pipelines in development. You should also expect a deep dive into data modeling strategies for Lakehouse architectures, where they will test your understanding of how to balance denormalization against storage costs in Delta Lake.
Application Advice
To get past the ATS filter, make sure your resume explicitly highlights quantifiable achievements tied to Databricks cost reduction, pipeline optimization, and PySpark tuning rather than just listing tools. Weave in exact keywords like Unity Catalog, Delta Lake, Medallion Architecture, and CI/CD directly into your experience bullets so the automated screen immediately tags you as a senior-level platform builder.