Azure Databricks Developer

• As an Azure Databricks Developer, you will be responsible for designing, implementing, and optimizing data solutions using Azure Databricks, Azure Data Factory, LogicApp etc.
• You will collaborate closely with multi-functional teams to ensure the efficient management, processing, and analysis of large-scale datasets.
• Your expertise will play a crucial role in enhancing our data-driven decision-making processes.
• Additionally, you'll take the lead in guiding other developers to deliver high-quality projects end to end.

Responsibilities:

• Design, develop, and implement scalable data pipelines using Azure Databricks to ingest, transform, and store structured and unstructured data.
• Analyze, optimize, and tune existing data pipelines for performance, reliability, and efficiency.
• Collaborate with data scientists, analysts, functional consultant, and other collaborators to understand business requirements and translate them into technical solutions.
• Provide expertise and guidance in selecting appropriate Azure services and tools to build robust data solutions.
• Implement security measures and standard processes to ensure data privacy and compliance with regulatory standards.
• Solve issues related to data processing, performance, and system failures.
• Stay updated with the latest Azure Databricks features, tools, and best practices and provide recommendations for improvements.

Requirements:

• Proven experience working with Databricks in large-scale data environments
• 6-8 years of experience with Databricks or similar technologies like Spark, Hive, Hadoop etc.
• Must be well versed with Databricks Lakehouse, Unity Catalog concept and its implementation in enterprise environments.
• Prior experience handling ERP data like SAP, JD Edwards and high-level understanding of different functional modules is required.
• In-depth knowledge of data engineering concepts, ETL processes, and data architecture principles. Strong understanding of cloud computing principles, particularly within the Azure ecosystem.
• Proficiency in programming languages such as Python /PySpark and query languages like SQL.

Preferred Qualifications:

• Bachelors degree or higher in computer science or engineering, or equivalent work experience within Data Engineering.
• 7+ years of rigorous experience developing in Databricks, PySpark, SQL (any flavor)
• Strong understanding of Data warehousing and various governance and security standards around Databricks.
• Experience with Databricks Unity Catalog, Databricks Workflow, Delta Sharing, Auto Loader, Delta Live Table, Overwatch, Databricks Volume etc.
• Good to have knowledge of RESTful API and its implementation.
• Experience with data modeling, data warehousing, and working with both structured and unstructured data.
• Databricks Data Engineer Associate certification will be preferred.
• Experience with other Azure services such as Azure SQL Database, Azure Data Lake Storage, LogicApp, Azure Data Factory etc.
• Excellent problem-solving skills and the ability to work in a collaborative team environment.
• Strong communication skills to convey complex technical concepts to non-technical partners.
• Knowledge of machine learning and GenAI concepts is an added advantage.