AppSierra Solutions - AWS Data Engineer - PySpark/Databricks (5-12 yrs)

Description :

We are seeking a skilled AWS Data Engineer with expertise in Databricks and Pyspark to join our dynamic team. In this crucial role, you will be responsible for designing and implementing big data processing pipelines that efficiently handle and analyze large volumes of data. You will work closely with cross-functional teams to gather requirements and translate them into scalable solutions that leverage the power of the cloud.

Your proficiency with AWS services, particularly in conjunction with Databricks, will enable you to optimize performance, enhance data integrity, and ensure security compliance throughout our data architecture. As you contribute to our data analytics and machine learning initiatives, you will help drive actionable insights, ensuring our organization maintains a competitive edge.

The ideal candidate will possess a strong analytical mindset, excellent problem-solving skills, and the ability to communicate technical concepts to non-technical stakeholders. If you are passionate about data engineering and eager to work in an innovative environment, we encourage you to apply and join us on this exciting journey to unlock the full potential of data-driven decision-making.

Responsibilities :

- Design, develop, and deploy scalable data processing pipelines using Databricks and Pyspark on AWS.

- Collaborate with cross-functional teams to gather and document data requirements for various projects.

- Implement ETL processes and ensure efficient data extraction, transformation, and loading into data warehouses.

- Optimize existing data solutions for performance and scalability while maintaining data integrity and reliability.

- Utilize AWS services such as S3, EC2, Lambda, and Redshift to enhance data workflows and architecture.

- Monitor data pipeline performance and troubleshoot issues as they arise, ensuring minimal downtime.

- Maintain documentation and adhere to best practices in data engineering and cloud development.

Requirements :

- Bachelor's degree in Computer Science, Engineering, or a related field.

- Proven experience as a Data Engineer with a strong portfolio of successful projects.

- Expertise in Databricks and Pyspark for data processing and analytics.

- Solid understanding of AWS services, particularly S3, EC2, Lambda, and Redshift.

- Experience with SQL and NoSQL databases to support data management and retrieval.

- Strong programming skills in Python, Scala, or Java for data manipulation and pipeline development.

- Excellent communication skills and the ability to work collaboratively in a team environment.