AWS Data Engineer - Python/Apache Airflow (6-12 yrs)

JD : AWS Data Engineer

Job location : Hyderabad

Must Have :

- Design and develop efficient data pipelines using Python and orchestrate workflows with Airflow, ensuring reliable data ingestion, transformation, and storage.

- Utilize Spark (Glue, EMR) for largescale data processing tasks, optimizing for performance and scalability.

- Implement and manage streaming data solutions with Kafka, managing high-volume data with proficiency in SQS and Event Bridge for even- driven architectures.

- Leverage AWS services extensively, including S3 for data storage, ensuring best practices in cloud-based data handling and security.

- Work closely with data architects, analysts, and other stakeholders, providing technical expertise and support in data-related projects and initiatives.

Good to Have :

- Knowledge of Data Warehousing (DW) concepts and Data Integration (DI) methodologies.

- Experience with data modelling and familiarity with ETL processes and tools.

- Experience in API integration, with a solid understanding of RESTful services.

- Developing and consuming APIs to facilitate seamless data exchange and system interoperability.

Qualifications :

- Bachelor's degree in computer science, Engineering, Data Science, or related field.

- 4-8 years of relevant experience in data engineering, with a proven track record of developing and managing data solutions.

- Strong proficiency in Python and experience with Airflow for data pipeline management.

- Expertise in Spark for big data processing, with experience in Glue and/or EMR.

- Demonstrated experience with Kafka for real-time data streaming and familiarity with AWS messaging services like SQS and Event Bridge.

- In-depth knowledge of AWS cloud services, especially S3, and best practices in cloud security and data management.