Lead Data Engineer - Python/Hadoop (5-12 yrs)

Job Description : Senior Data Engineer (6+ Years of Experience)

Position Overview :

We are looking for a Senior Data Engineer who has a strong background in designing, developing, and maintaining complex data systems. The ideal candidate will have experience working with large datasets, real-time data processing, and cloud technologies. You will work closely with data scientists, analysts, and software engineers to develop end-to-end data solutions that support business intelligence, analytics, and reporting.

Key Responsibilities :

- Data Pipeline Development : Design, build, and maintain scalable data pipelines that efficiently ingest, process, and store data from various sources, ensuring high data quality and accessibility.

- Data Architecture & Modeling : Develop and implement data architecture strategies to support data warehousing, data lakes, and other big data initiatives. Design and optimize database structures and data models.

- ETL/ELT Processes : Lead the development of ETL/ELT workflows to transform raw data into actionable insights. Optimize existing workflows for performance and reliability.

- Collaboration with Cross-Functional Teams : Work closely with Data Scientists, Analysts, and Product Teams to understand business requirements and provide solutions that enable data-driven decision-making.

- Cloud Technologies : Implement and manage data solutions in cloud platforms (AWS, Azure, GCP), including leveraging services like S3, Redshift, BigQuery, and others for scalable data storage and processing.

- Data Quality & Governance : Ensure that data pipelines are compliant with data governance and quality standards. Implement monitoring and alerting systems to track data accuracy and availability.

- Optimization & Performance Tuning : Regularly optimize database performance, queries, and data pipelines for speed and cost-effectiveness. Troubleshoot performance issues and suggest improvements.

- Documentation & Reporting : Document data engineering processes, tools, and methodologies to maintain a clear and organized framework for team collaboration and knowledge sharing.

- Mentorship & Leadership : Mentor and provide technical guidance to junior engineers and other team members. Lead code reviews and help shape best practices for the engineering team.

Skills & Qualifications :

- Experience : 6+ years of experience in data engineering, data pipeline development, and data management in large-scale environments.

- Programming Languages : Expertise in Python, SQL, and familiarity with other programming languages like Java, Scala, or Go.

- Data Technologies : Strong experience with big data technologies such as Hadoop, Spark, Kafka, and related frameworks.

- Cloud Platforms : Proficient in cloud data services such as AWS, Google Cloud, or Azure (preferably AWS).

- Database Management : Solid experience with relational and NoSQL databases (e.g., PostgreSQL, MySQL, MongoDB, Cassandra).

- Data Warehousing & BI : Knowledge of data warehousing concepts and tools like Redshift, Snowflake, or BigQuery, and experience working with BI tools like Tableau, Looker, or Power BI.

- Data Modeling & Architecture : Strong understanding of data modeling, schema design, and architectural best practices for scalable data solutions.

- Version Control & CI/CD : Proficiency in version control tools (Git) and experience with CI/CD pipelines to ensure efficient and reliable deployments.

- Problem-Solving : Excellent troubleshooting skills and the ability to solve complex data engineering challenges.

- Communication : Strong verbal and written communication skills to interact with both technical and non-technical stakeholders.

Required Skills & Qualifications :

Education : Bachelor's or master's degree in computer science, Information Technology, or a related field.

Experience : At least 6 years of experience in data engineering or a related role.

Technical Expertise :

Programming Languages : Proficiency in Python, Java, or Scala.

Big Data Technologies : Hands-on experience with tools like Hadoop, Spark, and Kafka.

Cloud Platforms : Proficiency with AWS (e.g., S3, Redshift, Glue), Azure (e.g., Synapse, Data Factory), or GCP (e.g., BigQuery, Dataflow).

Databases : Expertise in SQL Server, MySQL, PostgreSQL.

Reporting : Expertise in Power BI, SSRS.

Tools : Familiarity with orchestration tools like Apache Airflow and dbt.

Technology - Understanding the latest trends in Data technology and updated on offerings like Databricks, MS Fabrics, AWS Sagemaker Lakehouse

Soft Skills :

- Strong problem-solving and analytical abilities.

- Excellent communication and collaboration skills.

- Ability to mentor junior engineers and lead data engineering initiatives.