DEzen Technology Solutions - Principal Data Engineer - Apache Airflow/Kafka (6-8 yrs)

Principal Data Engineer

Key Responsibilities :

- Develop and maintain data architectures that are scalable, flexible, and aligned with business needs.

- Design and implement data pipelines using a variety of tools and technologies (Apache Airflow, Kafka, Spark, SQL, NoSQL).

- Define data models and schemas to ensure data consistency and integrity.

- Develop and optimize ETL/ELT processes to extract, transform, and load data from various sources (databases, APIs, cloud storage).

- Implement data quality checks and monitoring to ensure data accuracy and completeness.

- Build and maintain data lakes and data warehouses to store and manage large volumes of data.

- Collaborate with data analysts and business stakeholders to identify data-driven insights.

- Develop and optimize SQL queries and data analysis techniques to extract meaningful information.

- Utilize data visualization tools to create interactive dashboards and reports.

- Lead and mentor a team of data engineers.

- Foster a culture of innovation and continuous learning.

- Provide technical guidance and support to team members.

- Leverage cloud platforms (AWS, GCP, Azure) to build scalable and cost-effective data solutions.

- Implement cloud-native technologies and best practices.

Required Skills and Experience :

- Strong proficiency in SQL and data modeling techniques.

- Experience with big data technologies like Hadoop, Spark, and Kafka.

- Expertise in data warehousing and data lake concepts.

- Proficiency in cloud platforms (AWS, GCP, Azure).

- Strong programming skills in Python or Scala.

- Experience with data visualization tools (Tableau, Power BI).

- Excellent problem-solving and analytical skills.

- Strong communication and collaboration skills.

- Experience leading and mentoring teams.

Preferred Skills :

- Experience with machine learning and AI techniques.

- Knowledge of data governance and data security best practices.

- Experience with data integration tools (Informatica, Talend)