Job Description & Details
This is a heavy-duty data engineering role disguised as an AI position, leaning heavily into massive scale and distributed systems. You won't just be playing with wrappers; you'll be building the robust data pipelines and cloud infrastructure required to feed modern AI workloads. If you enjoy wrangling massive datasets and untangling messy migration projects, this one's worth a look.
What You'll Actually Be Doing
Expect to spend your days knee-deep in distributed data processing, migrating heavy workloads from on-prem Hadoop clusters over to Google Cloud Platform. You will design and implement scalable, modular data pipelines using Spark, Flink, and Kafka, ensuring data governance and automated controls are baked right in from the start. A good chunk of your time will also go toward orchestrating lakehouse architectures and occasionally poking at GenAI frameworks like LangChain to stitch together agent-based capabilities.
The Core Tech Stack
Your bread and butter here will be Python, PySpark, and Apache Kafka running on top of Google Cloud's ecosystem—specifically BigQuery, Dataproc, Cloud Storage, and Cloud Composer. You need solid hands-on experience with lakehouse formats like Iceberg or Delta, plus a firm grasp of NoSQL databases and Airflow orchestration. Don't gloss over the DevOps requirements either; you'll be expected to manage deployments using Docker, Kubernetes, Jenkins, and Git without holding your hand.
Interview Expectations
When you get on the technical screen, be ready for someone to ask you how you'd handle a massive stateful stream processing failure in Flink or Kafka under heavy load. The hiring manager isn't just looking for syntax here; they want to see how you reason through distributed system bottlenecks, backpressure, and data consistency. You'll likely also get grilled on cloud migration strategies, specifically how you would safely transition petabyte-scale data from an on-prem Hadoop environment to BigQuery without taking production offline.
Application Advice
Make sure your resume doesn't just read like a list of tools, but rather a track record of scale. Explicitly highlight your production experience with GCP, Kafka streaming, and PySpark, ensuring exact keywords from the JD like "lakehouse architecture", "Airflow", and "Iceberg" are visible so you easily glide past the ATS filters. If you have any public cloud data certifications, put them right at the top where the recruiter can see them immediately.