Lead Data Engineer - Python/Spark (10-20 yrs)

Role : Lead Data Engineer

Experience : 7+ Years

Location : Bhilai

Job Description :

Drive the design and implementation of robust data architectures, specializing in Machine Learning, and AI implementation. Familiarity with implementing Generative AI use cases is a Plus. Provide thought leadership on developing analytical tools, create high performance analytic and ML solutions, implementing efficient ETL processes, and deploying well architected data infrastructures. Shape the future of data excellence. Report to the Director of Data Engineering and Data Science based in the US, need to accommodate overlapping working hours.

Responsibilities :

AI/ML Leadership :

- Lead AI/ML projects, guiding the team through research, development, and deployment.

- Oversees Business Intelligence and Reporting, integrating data solutions to facilitate insightful analytics.

Data Modeling :

- Apply advanced data modeling techniques to uncover patterns and insights.

AI Solutions :

- Collaborate on designing and deploying innovative AI solutions to address business challenges.

OCI and Snowflake :

- Leverage expertise in Oracle Cloud Infrastructure (OCI) and Snowflake to enhance data solutions.

Data Architecture :

- Design and implement robust, scalable, and maintainable data architectures.

- Collaborate with cross-functional teams to ensure data solutions align with business requirements.

Data Integration :

- Lead efforts to integrate and consolidate data from various sources for analysis.

- Implement efficient ETL processes to support real-time and batch data processing.

Database Management :

- Manage and optimize database systems for performance, reliability, and scalability.

- Implement best practices in data modeling and database design.

- Proficiently manages Cloud-based and on-premises databases.

Quality Assurance :

- Establish and enforce data engineering best practices and standards.

- Implement data quality checks and monitoring to ensure accuracy and reliability.

Technology Evaluation :

- Stay abreast of emerging technologies and trends in data engineering.

- Evaluate and recommend tools and frameworks to enhance the data engineering stack.

Collaboration :

- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.

- Facilitate communication between data engineering and other technical teams.

Technical Leadership :

- Provide technical leadership and mentorship to a team of AI Engineers, Data Engineers, and Data Analysts.

- Oversee the end-to-end development and deployment of scalable and efficient data architectures.

- Leads Data Governance initiatives, ensuring adherence to Data Quality standards and Regulatory Compliance.

- Implements robust data security measures, safeguarding sensitive information across cloud and on-premises databases.

- Establishes comprehensive monitoring for data pipelines and architecture, proactively identifying and addressing potential issues to maintain optimal performance and reliability.

Requirements :

- Proven experience in Designing, Building, Testing, Deploying and Supporting Data Architectures, AI/ML use cases and Business analytics.

- Proficient in programming languages such as Python, SQL, TensorFlow, or PyTorch.

- Experience with big data technologies (e.g., Hadoop, Spark) and cloud platforms (e.g., AWS, Azure, GCP). Hands on Databricks, Snowflake.

- Solid understanding of data modeling, database design, and optimization techniques.

- Excellent communication skills and the ability to convey complex technical concepts to non-technical stakeholders.

- 5+ years of experience managing data engineering teams, data analytics teams, and AI engineers.

- Strong understanding of data governance, data security, and data privacy best practices.

- Experience as a player/coach that fosters a culture of continuous improvement and learning.