Data Engineer (Databricks, Hadoop)

As a Data Engineer, you will leverage Databricks and Hadoop ecosystems to build robust and efficient data pipelines, enable real-time analytics, and deliver data-driven solutions. You will collaborate closely with cross-functional teams to support data processing, model development, and infrastructure projects.

Key Responsibilities :

• Design, implement, and optimize scalable data pipelines using Databricks and Hadoop.
• Leverage Databricks for big data processing and cloud-based analytics, ensuring high-performance, reliability, and cost-effectiveness.
• Develop and maintain data models and ETL (Extract, Transform, Load) processes to streamline data ingestion from multiple sources.
• Optimize Hadoop clusters (via tools like Hive, Pig, HBase) for high-volume, distributed data processing.
• Ensure data quality, consistency, and integrity in the systems.
• Collaborate with data scientists and analysts to build machine learning models and deploy data-driven solutions.
• Troubleshoot and resolve performance issues related to data workflows and pipelines.
• Work with cloud platforms (AWS, Azure, GCP) to integrate Databricks and Hadoop with cloud-based infrastructure.
• Automate and optimize recurring data workflows to improve overall efficiency.
• Maintain documentation and reports on system configurations, processes, and best practices.