Lead Software Engineer - Python, AWS

Job responsibilities

• Design, develop, and maintain ETL processes (using Python, AWS) to support data integration and data warehousing solutions.
• Collaborate with data architects, data analysts, and business stakeholders to understand data requirements and translate them into technical specifications.
• Optimize and tune ETL processes (using Python, AWS) for performance and scalability.
• Ensure data quality and integrity through rigorous testing and validation procedures.
• Monitor and troubleshoot ETL processes to resolve issues and ensure timely data availability.
• Develop and maintain documentation for ETL processes, data flows, and data models.
• Mentor and provide guidance to junior ETL engineers and data team members.
• Stay current with industry trends and best practices in data engineering and ETL technologies.

Required qualifications, capabilities, and skills

• Formal training or certification on software engineering concepts and 5+ years of applied experience
• Hands-on practical experience in ETL development (using Python, AWS) and data engineering
• Preferred experience with Apache Spark
• Strong SQL skills and experience with relational databases (e.g., Oracle, SQL Server, MySQL)
• Good Debugging and Optimization Capabilities
• Understanding of different data formats like CSV, Parquet and ORC and data sources like HDFS, S3, SFTP, Apache Iceberg etc
• Proficient in all aspects of the Software Development Life Cycle
• Advanced understanding of agile methodologies such as CI/CD, Application Resiliency, and Security
• Familiarity with other programming languages such as Java or Scala
• Experience with cloud platforms (e.g., AWS, Azure, GCP) and their data services

Preferred qualifications, capabilities, and skills

• Experience with big data technologies (e.g., Hadoop).
• Knowledge of data governance and data security best practices.
• Experience with real-time data processing and streaming technologies (e.g., Kafka, Flink).