Data Engineer

Overview:

We are seeking a skilled and experienced Data Engineer to join our dynamic team. The ideal candidate will have hands-on experience with Databricks, Fabric, and PySpark to design, develop, and maintain scalable data pipelines and platforms. You will work closely with data scientists, analysts, and other stakeholders to ensure efficient data management and delivery.

Key Responsibilities:

• Design, build, and maintain data pipelines and ETL processes using Databricks, Fabric, and PySpark.
• Collaborate with cross-functional teams to understand data requirements and deliver solutions that drive business value.
• Develop and optimize complex SQL queries and data workflows for large-scale data processing.
• Create and manage data lakes, data warehouses, and other data storage solutions.
• Implement best practices for data modeling, quality, and governance.
• Develop automation scripts for data integration and transformation using Python and PySpark.
• Build and optimize performance across various data processing workflows.
• Troubleshoot, debug, and optimize issues related to data pipeline performance and reliability.
• Ensure data security and compliance across systems.
• Provide technical leadership and mentorship to junior engineers and other team members.
• Stay up-to-date with new technologies and trends in data engineering and cloud computing.

Skill & Qualification:

• 5-8 years of hands-on experience in data engineering, with significant expertise in Databricks, Fabric, and PySpark.
• Strong experience in building and optimizing large-scale data pipelines and workflows.
• Proficient in Python and SQL for data manipulation, integration, and analysis.
• Experience with Azure Data Factory, Azure Synapse Analytics, and other cloud-based data services.
• Knowledge of data modeling, ETL frameworks, and best practices for data governance.
• Expertise in Databricks platforms for distributed data processing, data lake formation, and real-time analytics.
• Experience with cloud data storage and data warehousing (Azure, AWS, GCP).
• Familiarity with containerization technologies (Docker, Kubernetes) and CI/CD pipelines is a plus.
• Strong problem-solving skills, with the ability to optimize complex data systems and workflows.
• Excellent communication skills and ability to collaborate with cross-functional teams.