Job Description & Details
This is a hands-on, 6-month contract gig focused on modernizing cloud data infrastructure using Azure, PySpark, and Databricks. You will be knee-deep in heavy ETL pipelines, data warehousing, and performance tuning for massive datasets. If you enjoy untangling messy data workflows and building scalable architectures in a hybrid setup (2 days onsite in Iselin, NJ), this is a solid project to sink your teeth into.
What You'll Actually Be Doing
Your day-to-day will revolve around designing, building, and maintaining scalable data pipelines that ingest, transform, and load diverse datasets. You'll spend a good chunk of your time writing and optimizing PySpark code within Databricks, dealing with Delta Lakes, and tuning jobs that crawl through massive volumes of structured and unstructured data. Expect to troubleshoot bottlenecks, handle performance tuning, enforce data governance, and occasionally play the architectural advisor as the team migrates and scales their cloud data platforms.
The Core Tech Stack
You need to be fluent in Python and advanced SQL, with at least 5 years under your belt building ETL pipelines specifically using Databricks and PySpark. Experience with a modern cloud data warehouse like Azure Synapse, Snowflake, or Redshift is non-negotiable, alongside a solid grasp of dimensional data modeling (Facts, Dimensions, OLAP, OLTP). If you have hands-on experience with Delta Lake, event-based streaming, and Airflow, you'll feel right at home here.
Interview Expectations
Expect the interviewers to probe deeply into your Databricks optimization skills, likely asking you how you handle data skew or memory spill issues in large PySpark jobs. They want to see if you actually understand the underlying architecture or if you just know how to write basic notebooks. Be ready to whiteboard a complete ETL pipeline from ingestion to a dimensional data warehouse, explaining how you would design for idempotency, error handling, and incremental loads.
Application Advice
Your resume needs to heavily feature concrete metrics around the size of the datasets you have managed and the performance gains you've unlocked in Databricks. Make sure keywords like PySpark, Delta Lake, Data Modeling, ETL pipelines, and Azure Synapse are explicitly stated to clear the ATS filters. Tie your past migration experiences directly to cloud architecture to show you can step in and immediately lead technical decisions without hand-holding.