Job Description & Details
This is a heavy-duty cloud data engineering gig focused on migrating massive on-prem data footprints to Google Cloud while keeping real-time streaming pipelines humming. You aren't just writing basic ETL scripts here; you are architecting lakehouse solutions and building event-driven systems that actually scale under heavy enterprise loads.
What You'll Actually Be Doing
Your day-to-day will revolve around designing, building, and maintaining scalable data pipelines using Python, PySpark, and Kafka. You will spend a good chunk of your time orchestrating workflows with Airflow and Cloud Composer, ensuring that data moving from Hadoop or on-prem environments lands cleanly into GCP BigQuery and Cloud Storage. Expect to tackle gnarly data migration bottlenecks, optimize streaming jobs with Flink and Spark Streaming, and embed automated data governance controls right into the pipeline architecture.
The Core Tech Stack
You absolutely need deep, battle-tested experience with Google Cloud Platform—specifically BigQuery, Dataproc, and Cloud Storage—alongside heavy Python and PySpark chops. Apache Kafka and Spark Streaming are non-negotiable for the real-time messaging requirements, and familiarity with modern lakehouse formats like Iceberg and Delta Tables will make or break your ability to handle their storage layer. Knowing your way around NoSQL databases, Airflow orchestration, and containerized deployments with Docker and Kubernetes rounds out the core toolkit.
Interview Expectations
Be ready to walk through a time you optimized a slow-running Spark job or debugged a lagging Kafka consumer group under high throughput. The interviewer is going to probe deep into your understanding of partitioning strategies, memory tuning, and how you handle backpressure in streaming pipelines. They want to see if you can troubleshoot distributed systems intuitively without guessing.
Application Advice
Skip the generic bullet points on your resume and make sure your experience with GCP, PySpark, and Kafka is front and center. Explicitly highlight any large-scale data migrations you've led from on-prem Hadoop to public cloud data lakes, as that is the primary pain point this role is funded to solve. Weave in keywords like Lakehouse architecture, Airflow orchestration, and CI/CD pipelines to easily clear their automated ATS filters.