Senior Data Engineer - PySpark/Python (5-10 yrs)

Required Skills : Python, spark/pyspark, Hive, Databricks, SQL ,Airflow/ Github.

Data Engineer

We are seeking an experienced Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, implementing, and maintaining data pipelines, ETL processes, and data infrastructure.

The ideal candidate should have a strong background in Python, Spark, SQL, and Databricks.

Responsibilities :

- Design, develop, and maintain data pipelines to extract, transform, and load data from various sources into our data lake or data warehouse.

- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure data quality.

- Optimize and tune data processing workflows for performance and scalability.

- Implement data governance and security best practices.

- Troubleshoot and resolve data-related issues.

- Monitor and maintain data infrastructure, ensuring high availability and reliability.

- Stay up-to-date with industry trends and emerging technologies in data engineering.

Qualifications :

- Bachelor's degree in Computer Science, Engineering, or a related field.

- 3+ years of experience as a Data Engineer.

- Strong proficiency in Python for data processing and ETL.

- Experience with Apache Spark for distributed data processing.

- Proficiency in SQL for querying and manipulating data.

- Hands-on experience with Databricks or similar cloud-based data platforms.

- Familiarity with data modeling, data warehousing, and data lake concepts.

- Excellent problem-solving skills and attention to detail.

- Strong communication and collaboration skills.