Job Description & Details
This is a heavy-hitting architecture role blending enterprise-grade lakehouse engineering with bleeding-edge AI applications for financial data. You won't just be writing basic Spark jobs; you'll be building autonomous, metadata-driven ingestion frameworks and GPT-powered conversational interfaces backed by Unity Catalog. It is a demanding senior position requiring deep technical credibility and the ability to guide both engineers and stakeholders through complex cloud migrations.
What You'll Actually Be Doing
You will spend your days owning the end-to-end medallion architecture on Azure and Databricks, designing everything from Bronze to Gold layers, schema evolution, and cost-optimization strategies. Half your time will go into designing AI-augmented ingestion and semantic knowledge graphs that power natural-language querying for financial data. The rest involves reviewing pull requests, mentoring engineers, defending designs in architecture boards, and ensuring your code is clean enough for the client's internal team to run autonomously.
The Core Tech Stack
You need absolute fluency in Python, Scala, and PySpark alongside deep enterprise experience with Databricks, Delta Lake, and Unity Catalog. On the cloud side, mastery of Azure ADLS Gen2, Azure Data Factory, and Event Hubs is mandatory. Beyond data engineering, you need production-grade experience shipping LLM systems using LangChain, LlamaIndex, or LangGraph, coupled with strong Infrastructure as Code capabilities using Terraform and Azure DevOps.
Interview Expectations
Expect the hiring team to grill you on how you handle state and partition tuning for massive streaming workloads without blowing up your compute budget. They will likely present a scenario where an AI-driven schema mapping hallucinates or drifts in production, and they'll want to hear your exact strategy for implementing human-in-the-loop review gates and deterministic fallbacks. They are looking for architectural pragmatism—knowing when a simple SQL constraint beats an expensive LLM call.
Application Advice
Skip the generic bullet points on your resume and spotlight enterprise-scale lakehouse migrations you've personally architected from scratch. Make sure keywords like Unity Catalog, Delta Lake, metadata-driven ingestion, and RAG pipelines are prominent to get past the initial ATS filters. Emphasize your governance background, particularly around PII handling, RBAC, and Terraform, as this role requires defending your decisions to strict enterprise security boards.