Job Description & Details
This is a hands-on data engineering role centered around building fast-turnaround AI applications and analytics solutions using Databricks. If you enjoy moving quickly to prototype, automate, and ship real business tools rather than maintaining massive legacy infrastructure, this gig will keep you on your toes. You are essentially bridging the gap between raw enterprise data and bleeding-edge AI workflows.
What You'll Actually Be Doing
Your day-to-day will involve writing robust PySpark and Python code to ingest, clean, and model messy data sources into pristine Delta Lake tables. You'll be collaborating closely with product and analytics stakeholders to wire up APIs, build data pipelines, and deploy rapid solutions like internal data apps or GenAI prototypes. The pace is fast, meaning you'll need to balance quick turnaround times with data quality, pipeline performance, and proper governance through Unity Catalog.
The Core Tech Stack
Databricks is the absolute heartbeat of this role, so you need to be deeply comfortable navigating notebooks, Delta Lake, Databricks Workflows, and Unity Catalog without hand-holding. Alongside that, you'll live and breathe Python, PySpark, and advanced SQL for heavy data lifting. If you also have exposure to Streamlit or FastAPI for spinning up quick UIs, and familiarity with LLMs or MLflow, you'll easily stand out from the typical applicant pool.
Interview Expectations
Expect the hiring team to grill you on how you handle performance bottlenecks in Spark, such as optimizing shuffle partitions, handling data skew, or tuning memory usage on Databricks clusters. They aren't just looking for textbook definitions; they want to hear war stories about how you diagnosed a crashing pipeline in production and fixed it. You'll also likely field architecture questions on how you would design an end-to-end data pipeline that feeds an AI agent or downstream LLM application while maintaining strict data quality standards.
Application Advice
To get past the ATS and grab the hiring manager's attention, make sure your resume explicitly highlights your Databricks ecosystem experience—don't just list 'Big Data' as a skill. Emphasize projects where you built production-grade PySpark pipelines, managed Delta Lake architectures, or integrated APIs into analytical applications. Weave in keywords like Unity Catalog, MLflow, and workflow automation directly into your experience bullet points to show you already speak their language.