Lead Data Engineer

• Assembling complex data sets that meet both functional and non-functional business requirements.
• Identifying and implementing internal process improvements, including redesigning infrastructure for scalability, optimizing data delivery, and automating manual processes.
• Building infrastructure for optimal extraction, transformation, and loading (ETL) of data from various data sources using GCP/Azure and SQL technologies.
• Developing analytical tools to leverage the data pipeline and provide actionable insights on key business performance metrics, including operational efficiency and customer acquisition.
• Collaborating with stakeholders, including data, design, product, and executive teams, to support their data infrastructure needs and address data-related technical issues.
• Overseeing the integration of new technologies and initiatives into data standards and structures.
• Ensuring high standards for data warehouse design, scalability, and data pipeline optimization.

Key Responsibilities:

• 20% : Requirements gathering and design
• 60% : Coding & testing
• 10% : Reviewing code done by developers, analyzing, and troubleshooting issues
• 10% : Deployments and release planning

Required Skills & Qualifications:

• Education : Bachelors degree in Computer Science, Computer Engineering, or a related field. A Masters degree is a plus.
• Experience :
• 6+ years of experience in Data Warehouse and Hadoop/Big Data technologies.
• 3+ years of strategic data planning, governance, and standard procedures.
• 4+ years of hands-on experience with Scala, Spark, PySpark, Python, and SQL.
• 3+ years of experience in an Agile environment.
• Strong background in data pipeline creation, optimization, and troubleshooting.
• Experience with cloud platforms such as GCP or Azure, data migration, ETL processes, and data validation.
• Technical Skills :
• Languages : Scala, Spark, PySpark, Python, SQL.
• Big Data : Hadoop, Hive, Pig, MapReduce.
• Tools : Apache Hadoop, Airflow, Kubernetes, Containers.
• Data Technologies : Data Warehouse Design, ETL, Data Analytics, Data Mining, Data Cleansing.
• Cloud Platforms : GCP, Azure.
• Additional Skills : Experience with data pipeline tools like Airflow, and understanding of Hadoop log files and multiple data processing engines is a plus.

Desirable Skills:

• Experience in Data Analytics, Machine Learning, and optimization.
• Understanding of web-based technologies like Java, ReactJS, Node. js.
• Knowledge of managing big data workloads and containerized environments.
• Experience in analyzing large datasets and optimizing data workflows.

Core Competencies:

• Strong problem-solving skills and the ability to think critically and strategically.
• Proactive sharing of knowledge, accomplishments, and lessons learned across teams.
• Ability to work effectively in a fast-paced environment.
• Excellent verbal and written communication skills.
• Leadership skills in guiding teams and driving technical solutions.
• Experience in deploying and releasing software in large organizations.