Job Description & Details
This is a heavy-hitting leadership role for a senior engineer who doesn't want to get stuck just drawing architecture diagrams all day. You'll be architecting and building an AI-first data hub for a major financial services and insurance client, merging heavy-duty data engineering with modern Generative AI systems. If you like scaling out enterprise pipelines while messing around with LLMs and RAG architectures, this is a solid gig.
What You'll Actually Be Doing
Your day-to-day will be a mix of deep technical execution and guiding a team of engineers. You'll be designing and writing production code for complex PySpark and Azure Databricks pipelines, setting up medallion architectures, and integrating LLMs and RAG workflows directly into the data streams. Expect to spend a chunk of your time troubleshooting performance bottlenecks on massive datasets, ensuring data governance is tight in a heavily regulated financial domain, and mentoring devs who need help navigating distributed systems and AI toolchains.
The Core Tech Stack
You're going to need serious mileage in Python, PySpark, and Scala, paired with a rock-solid grasp of the Azure ecosystem—specifically Azure Databricks, Delta Lake, ADLS Gen2, and ADF. Because this is an AI-first platform, you aren't just moving data around; you need proven production experience with LLMs, RAG, and frameworks like LangChain, LangGraph, or LlamaIndex. Terraform and Azure DevOps will round out your daily toolkit for managing infrastructure and CI/CD pipelines.
Interview Expectations
Expect the hiring team to hit you with a scenario where a RAG pipeline feeding into a financial reporting tool starts hallucinating or experiencing massive latency spikes under high concurrency, and they'll want to hear how you'd debug the retrieval and embedding generation layers in Databricks. They are secretly testing whether you actually know how to deploy LLMs in production or if you've just read tutorials. You'll also likely be grilled on how to optimize a sluggish PySpark shuffle operation while maintaining strict data governance and lineage in a medallion architecture.
Application Advice
Make sure your resume doesn't just read like a list of buzzwords; highlight the actual scale of the data pipelines you've built and the specific business impact of your LLM deployments. Explicitly weave in keywords like Azure Databricks, PySpark, Delta Lake, LangChain, and RAG to clear the ATS, but back them up with bullet points detailing how you handled performance tuning, infrastructure as code with Terraform, and technical leadership.