Data Engineer

• Design, build, and maintain scalable, high-performance data pipelines in Databricks to support analytics and machine learning workflows.
• Collaborate with data scientists, analysts, and other engineers to define and implement data solutions.
• Develop ETL processes for ingesting, transforming, and storing data across a variety of data sources, ensuring data quality and integrity.
• Optimize and tune existing Databricks pipelines and workflows for performance, scalability, and reliability.
• Build and maintain data models and data warehouses, leveraging tools such as Delta Lake and Apache Spark.
• Write efficient SQL queries for data analysis and reporting.
• Automate data workflows, monitor pipeline performance, and implement proactive solutions to resolve any issues.
• Ensure data security, privacy, and compliance by adhering to best practices and organizational policies.
• Participate in code reviews, mentorship, and knowledge-sharing sessions with the team.
• Troubleshoot and debug complex data issues to ensure the robustness and efficiency of data solutions.

• Experience: 4-5 years of hands-on experience as a Data Engineer, with a strong focus on Databricks and data pipeline development.

Technologies:

• Proficiency in Databricks (Spark, Delta Lake).
• Strong experience in SQL , Python , and Scala .
• Familiarity with Apache Spark and distributed computing concepts.
• Experience with ETL tools and frameworks (eg, Airflow, Azure Data Factory).
• Experience working with cloud platforms like Azure , AWS , or Google Cloud .
• Knowledge of data warehousing concepts and architecture (eg, Redshift, Snowflake).

• Database Systems: Strong knowledge of relational and non-relational databases, with hands-on experience in SQL and NoSQL systems.
• Version Control: Familiarity with Git or other version control systems.
• Problem Solving: Strong analytical and problem-solving skills with the ability to troubleshoot complex data issues.
• Collaboration: Excellent communication and collaboration skills, with the ability to work cross-functionally in a fast-paced environment.