Data Engineer - AWS/Python (2-5 yrs)

Experience Required : 2+ years

Role Summary :

As a Senior Data Engineer, with over 2 years of expertise in Python, PySpark,- SQL to- design, develop and optimize complex data pipelines, support data modeling, and contribute to the architecture that supports big data processing and analytics to cutting-edge cloud solutions that drive business growth.

You will lead the design and implementation of scalable, high-performance data solutions on AWS and mentor junior team members.

This role demands a deep understanding of AWS services, big data tools, and complex architectures to support large-scale data processing and advanced analytics.

Key Responsibilities :

- Design and develop robust, scalable data pipelines using AWS services, Python, PySpark, and SQL that integrate seamlessly with the broader data and product ecosystem.

- Lead the migration of legacy data warehouses and data marts to AWS cloud-based data lake and data warehouse solutions.

- Optimize data processing and storage for performance and cost.

- Implement data security and compliance best practices, in collaboration with the IT security team.

- Build flexible and scalable systems to handle the growing demands of real-time analytics and big data processing.

- Work closely with data scientists and analysts to support their data needs and assist in building complex queries and data analysis pipelines.

- Collaborate with cross-functional teams to understand their data needs and translate them into technical requirements.

- Continuously evaluate new technologies and AWS services to enhance data capabilities and performance.

- Create and maintain comprehensive documentation of data pipelines, architectures, and workflows.

- Participate in code reviews and ensure that all solutions are aligned to pre-defined architectural specifications.

- Present findings to executive leadership and recommend data-driven strategies for business growth.

- Communicate effectively with different levels of management to gather use cases/requirements and provide designs that cater to those stakeholders.

- Handle clients in multiple industries at the same time, balancing their unique needs.

- Provide mentoring and guidance to junior data engineers and team members.

Requirements :

- 2+ years of experience in a data engineering role with a strong focus on AWS, Python, PySpark, Hive, and SQL.

- Proven experience in designing and delivering large-scale data warehousing and data processing solutions.

- Lead the design and implementation of complex, scalable data pipelines using AWS services such as S3, EC2, EMR, RDS, Redshift, Glue, Lambda, Athena, and AWS Lake Formation.

- Bachelor's or Master's degree in Computer Science, Engineering, or a related technical field.

- Deep knowledge of big data technologies and ETL tools, such as Apache Spark, PySpark, Hadoop, Kafka, and Spark Streaming.

- Implement data architecture patterns, including event-driven pipelines, Lambda architectures, and data lakes.

- Incorporate modern tools like Databricks, Airflow, and Terraform for orchestration and infrastructure as code.

- Implement CI/CD using GitLab, Jenkins, and AWS CodePipeline.

- Ensure data security, governance, and compliance by leveraging tools such as IAM, KMS, and AWS CloudTrail.

- Mentor junior engineers, fostering a culture of continuous learning and improvement.

- Excellent problem-solving and analytical skills, with a strategic mindset.

- Strong communication and leadership skills, with the ability to influence stakeholders at all levels.

- Ability to work independently as well as part of a team in a fast-paced environment.

- Advanced data visualization skills and the ability to present complex data in a clear and concise manner.

- Excellent communication skills, both written and verbal, to collaborate effectively across teams and levels.

Preferred Skills :

- Experience with Databricks, Snowflake, and machine learning pipelines.

- Exposure to real-time data streaming technologies and architectures.

- Familiarity with containerization and serverless computing (Docker, Kubernetes, AWS Lambda).