Job Description & Details
This 12-month hybrid contract in Charlotte is a heavy-duty data engineering gig centered around building out massive AWS data lakes and real-time streaming pipelines. If you enjoy wrangling massive datasets with PySpark and spinning up serverless workflows without breaking a sweat, this role will keep you deeply engaged in modern cloud architecture.
What You'll Actually Be Doing
You'll spend your days designing, building, and maintaining end-to-end data pipelines that ingest terabytes of information. Your main focus will be writing high-performance PySpark code to run on EMR clusters, orchestrating serverless workflows via AWS Lambda, and setting up real-time streaming infrastructure with Confluent Kafka. Expect to constantly troubleshoot bottlenecks in data lakes, optimize slow-running Redshift queries, and manage cloud infrastructure using Terraform.
The Core Tech Stack
You need absolute fluency in Python 3.8+ and PySpark for heavy data lifting. AWS is your playground here, meaning you must know your way around S3 data lake architecture, AWS Glue ETL jobs, Amazon Athena, and EMR. On top of batch processing, Confluent Kafka experience with Schema Registry and connectors is non-negotiable for their real-time streaming needs. Toss in Terraform for Infrastructure as Code and CloudWatch for monitoring, and you have a complete picture of what keeps their data ecosystem running.
Interview Expectations
Be ready for deep-dive technical questions on how you handle data skew in large PySpark jobs running on EMR, and make sure you can explain the exact steps you take to optimize a sluggish query in Amazon Redshift. The hiring manager is looking for someone who doesn't just write scripts, but truly understands distributed computing principles, memory management in Spark, and how to structure raw versus curated data layers in S3 without hitting cost or performance walls.
Application Advice
Skip the generic tech buzzwords and make sure your resume immediately highlights concrete examples of scaling PySpark jobs on EMR and building production-grade Kafka streaming pipelines. Explicitly mention your hands-on experience with Terraform, AWS Glue, and S3 partitioning strategies so the ATS picks up the exact keywords this team relies on to filter candidates.