Senior AWS Data Engineer

We are seeking a skilled AWS Data Engineer with Data bricks to join our dynamic team. The ideal candidate will have hands-on experience in designing, building, and optimizing data pipelines on AWS using Data bricks, Spark, and Python. They will collaborate with cross-functional teams to develop scalable and efficient data solutions for enterprise applications.

Key Responsibilities:

• Design, develop, and maintain ETL/ELT data pipelines using AWS services (S3, Glue, Lambda, Redshift, etc.) and Data bricks.
• Build and optimize large-scale data processing frameworks using Apache Spark and Python.
• Develop data lakehouse architectures to handle structured and unstructured data efficiently.
• Implement data governance, security, and compliance best practices.
• Work with business and data analysts to understand data requirements and translate them into technical solutions.
• Optimize query performance and manage data partitioning and indexing in Databricks.
• Integrate third-party APIs and data sources into the AWS data ecosystem.
• Automate data pipeline deployments using CI/CD pipelines and Infrastructure as Code (IaC) tools like Terraform or CloudFormation.
• Monitor and troubleshoot data pipeline failures, bottlenecks, and performance issues.

Required Skills Experience:

• 4-6 years of experience as a Data Engineer working with AWS cloud services.
• Hands-on experience with Data bricks and Apache Spark.
• Strong programming skills in Python (or Scala).
• Experience with SQL for querying and data transformation.
• Proficiency in AWS services like S3, Glue, Lambda, Redshift, EMR, Athena, and IAM.
• Experience with data lake and data warehouse architectures.
• Familiarity with CI/CD pipelines for data applications.
• Strong understanding of data modeling, schema design, and ETL best practices.
• Knowledge of data security, encryption, and governance frameworks.

Good to Have:

• Experience with Terraform or CloudFormation for IaC.
• Knowledge of Kafka/Kinesis for real-time data streaming.
• Familiarity with dbt ( data build tool).
• Understanding of machine learning workflows on Databricks.
• Certification in AWS ( AWS Certified Data Analytics, AWS Certified Solutions Architect, etc.).