Lead Data Engineer - ETL/Python (8-10 yrs)

Job Description :

- Lead and mentor a team while remaining hands-on with development and problem-solving.

- Work closely with stakeholders to understand business requirements and translate them into technical solutions.

- Design, develop, and optimize complex SQL queries for data extraction, transformation, and analysis.

- Utilize Python for automation, scripting, and data processing.

- Leverage Google Cloud Platform (GCP) services for data storage, processing, and analytics.

- Participate in code reviews, ensure best practices, and optimize data workflows.

- Troubleshoot and resolve performance bottlenecks in SQL queries and data pipelines.

Required Skills :

- Strong communication skills - ability to interact with stakeholders, lead discussions, and present insights.

- Expertise in SQL - deep understanding of database design, query optimization, and performance tuning.

- Hands-on leadership - ability to guide the team while contributing actively to development.

- Proficiency in Python - for scripting, data processing, and automation tasks.

- Good to have Knowledge of GCP services - BigQuery, Cloud Storage, Dataflow, Pub/Sub, etc.

- Experience in migration projects (preferred) - especially from on-prem to cloud.

Responsibilities :

- Lead and manage the design, development, and implementation of scalable data pipelines and data models for both batch and real-time processing.

- Develop and optimize ETL processes to collect, transform, and store data from multiple sources into the company's data warehouse or data lake.

- Work with teams to ensure the efficient integration of data sources, providing end-to-end solutions that support analytics and reporting needs.

- Design and implement strategies for data governance, security, and quality assurance to ensure compliance and data integrity.

- Build and manage data infrastructure (data lakes, data warehouses, cloud platforms like AWS, GCP, Azure) ensuring optimal performance, scalability, and cost-efficiency.

- Mentor and guide a team of data engineers, fostering a collaborative environment and promoting best practices in coding, architecture, and development.

- Optimize database queries and data processing pipelines to enhance performance and reduce costs.

- Troubleshoot and resolve data issues, collaborating with data scientists, product managers, and IT teams.

- Implement and maintain continuous integration/continuous deployment (CI/CD) pipelines for data engineering projects.

- Stay up to date with industry trends and best practices, driving innovation in data engineering practices.

Requirements :

- 8+ years of experience in data engineering, data architecture, or a related field.

- Strong proficiency in programming languages such as Python, Java, or Scala.

- Expertise in designing and optimizing large-scale data pipelines and ETL processes.

- Hands-on experience with cloud platforms (AWS, Google Cloud, Azure) and big data technologies (e.g., Hadoop, Spark, Kafka).

- Advanced knowledge of relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra).

- Familiarity with data warehousing solutions (e.g., Snowflake, Redshift, BigQuery).

- Strong understanding of data modeling, data governance, and data security practices.

- Experience with version control systems (e.g., Git) and CI/CD tools.

- Proven leadership experience, with the ability to mentor junior team members and collaborate effectively with cross-functional teams.

- Excellent problem-solving, analytical, and communication skills.