Job Description & Details
This Data Architect gig in the NY/NJ area is a heavy-hitting migration and modernizing role focused entirely on moving legacy workloads into the Databricks ecosystem. You aren't just drawing architecture diagrams here; you are expected to roll up your sleeves and build scalable ingestion pipelines using PySpark and Delta Lake. If you enjoy untangling messy legacy data setups and replacing them with clean Lakehouse architectures, this is a solid project to sink your teeth into.
What You'll Actually Be Doing
Your day-to-day will revolve around untangling existing data pipelines, assessing old ETL workloads, and orchestrating massive migrations over to Databricks using Unity Catalog and Lakeflow. You will spend a good chunk of your time building reusable, metadata-driven frameworks that automate migrations so the team isn't reinventing the wheel for every data source. Expect to balance hands-on PySpark development with leading onshore and offshore engineering teams, translating client requirements into solid technical execution.
The Core Tech Stack
Databricks is the absolute center of gravity here—you need deep fluency with Delta Lake, Unity Catalog, and PySpark. A strong grasp of Medallion architecture is non-negotiable since you'll be designing ingestion and transformation layers from scratch. On top of that, pipeline orchestration, CI/CD, and data quality validation frameworks are crucial to keep these large-scale migrations from breaking in production. Bonus points if you know your way around Alteryx, especially when it comes to migrating legacy Alteryx workflows into native Databricks code.
Interview Expectations
Be ready for them to test your practical knowledge of Databricks optimization and Unity Catalog governance by asking how you would handle data lineage and access control across multiple business units during a migration. They want to see how you troubleshoot performance bottlenecks in PySpark jobs, so expect a scenario-based question where you have to explain how you diagnose memory leaks or shuffle skews in a massive distributed join. The hiring manager is secretly looking for architecture pragmatism—they want to know you can balance enterprise-grade security with the speed required to get migrations done on time.
Application Advice
When updating your resume for this role, make sure 'Databricks', 'Unity Catalog', 'Delta Lake', and 'PySpark' are front and center, ideally with metric-driven examples of past data migrations you've personally led. Don't just list tools; emphasize your experience in building metadata-driven frameworks and automation scripts, as the JD heavily stresses a 'framework mindset'. Weave in keywords like 'Lakehouse architecture', 'ETL modernization', and 'pipeline orchestration' naturally into your experience bullets to ensure you easily clear the ATS keyword filters.