Job Description & Details
This is a heavy-duty, senior-level data engineering gig focused on building out sales channel residual payment and reporting systems with AI-driven intelligence. You will be stepping into an enterprise environment where handling massive, multi-platform data pipelines efficiently is the baseline expectation. If you enjoy wrangling complex data architectures and working directly with cloud-native tooling, this role offers a chance to build mission-critical financial applications.
What You'll Actually Be Doing
Expect to spend your days designing, building, and troubleshooting event-driven data ingestion pipelines using Python and PySpark alongside AWS cloud services. You will be transforming and merging data from disparate platforms, ensuring everything flows smoothly into downstream reporting models. Aside from writing clean code, you will actively participate in rigorous code reviews, optimize system performance, and troubleshoot stubborn bugs during UAT and post-go-live phases. The real challenge here is ensuring high performance and data integrity for complex financial calculations at scale.
The Core Tech Stack
You absolutely must have deep, hands-on expertise in Python and PySpark paired with robust AWS data services like S3, Glue, and Lambda. Solid SQL database skills and production-level experience with CI/CD pipelines and DevOps practices are non-negotiable since you will be shipping code frequently. If you also happen to know your way around Snowflake, Snowpark, and Snowflake Cortex for generative AI integrations, you will immediately stand out as someone who can hit the ground running on their modern data stack.
Interview Expectations
Expect the hiring team to grill you on how you handle performance bottlenecks in large-scale PySpark jobs running on AWS Glue, specifically asking you to walk through a time you optimized a slow-moving data pipeline. They want to see how you troubleshoot memory overhead and partition strategies. You will also likely face architectural questions regarding event-driven data ingestion, where they will test your understanding of decoupling services using S3 and Lambda while maintaining transactional consistency.
Application Advice
To get past the automated screeners and land an interview, ensure your resume explicitly highlights your hands-on years with Python, PySpark, and core AWS data tools like Glue and S3. Don't just list technologies; describe the scale of the pipelines you have built and the CI/CD processes you used to deploy them. If you have experience with commission calculation systems, financial data models, or Snowflake, put those front and center because those domain-specific skills are what the client is ultimately prioritizing.