Aidetic - Data Engineer III - Python/PySpark (4-8 yrs)

About the role :

We are seeking a Data Engineer 3 (Lead Data Engineer) with expertise in modern data warehousing tools like Databricks or Snowflake, large scale data processing frameworks like Apache Spark, scripting in Python and SQL, and cloud platforms like Azure and AWS. This role will focus on leading a team of data engineers who are building,maintaining, and optimizing scalable data architectures and ETL pipelines for real world,mission critical projects for our clients. The candidate will oversee end to end design and deployment of these projects.

About the company :

Aidetic is an AI/ML consulting company working at the cutting edge of AI since 2018, before it was cool. We're a small but highly talented team of 50 people passionate about solving real world problems with AI. With clients such as Target, Tata Steel, InsuranceDekho etc. we have put 200+ AI systems in production.

Experience :

- 4+ years of data engineering experience

Key Responsibilities :

Lead the Design and Implementation : Oversee end-to-end design, development, and deployment of data engineering solutions using Databricks or Snowflake for large-scale data processing and integration projects.

Team Leadership : Manage and mentor a team of data engineers, providing technical guidance and ensuring best practices are followed.

Architect Scalable Solutions : Build and optimize scalable data architectures and ETL pipelines, ensuring data quality, reliability, and performance.

Cross Team Collaboration : Collaborate with DevOps, infrastructure teams, and client stakeholders to optimize the performance and scalability of Databricks or Snowflake clusters.

Data Security and Privacy : Ensure adherence to data governance, data security, and data privacy standards within the Databricks or Snowflake environments.

Complex Data Processing : Understand and optimize complex stored procedures for data manipulation and processing in relational databases like MySQL and MS SQL.

Innovation and Improvement : Stay updated with industry trends and introduce innovative solutions to improve data engineering practices.

Stakeholder Communication : Effectively communicate technical concepts to non-technical stakeholders and manage expectations.

Must-Have Skills :

1. 4+ Years of Experience : Extensive experience with big data technologies, including Databricks or Snowflake and Apache Spark.

2. Leadership Abilities : Proven experience in leading and mentoring data engineering teams.

3. Advanced Programming Skills : Strong proficiency in Python, PySpark, Scala, or Java for complex data processing.

4. ETL Expertise : Demonstrated experience in designing and setting up large-scale ETL pipelines from scratch and managing data migration projects.

5. Apache Airflow Proficiency : Expertise in working with Apache Airflow and DAGs for workflow orchestration.

6. Database Mastery : In-depth experience with SQL and relational/NoSQL databases (Elasticsearch, MySQL, MS SQL) for data extraction and manipulation.

7. Cloud Platform Proficiency : Preferred experience with Azure; familiarity with AWS or GCP is acceptable. Knowledge of services like ADLS, Eventbridge, or Azure Kafka.

8. Data Governance and Security : Strong knowledge of data modeling, data warehousing, and implementing data governance, security, and quality best practices.

9. DevOps and Scripting : Experience with Docker, Unix/Linux scripting, and Agile methodologies.

10. Problem-Solving Skills : Exceptional ability to troubleshoot complex data and infrastructure issues.

Good-to-Have Skills :

1. Professional Certifications : Certifications such as Databricks Engineer Professional or Microsoft AzureData Engineer.

2. Real-Time Streaming : Experience with real-time data streaming technologies like Kafka or Redpanda.

3. Modern ETL Tools : Familiarity with modern ETL tools like dbt.

4. Documentation Skills : Strong documentation and white-boarding abilities.

5. Stakeholder Management : Excellent communication skills and experience in managing client relationships.

6. Innovative Mindset : Proactive in learning new technologies and introducing best practices.