Databricks Architect

Role Overview:

Exponentia.ai is seeking a highly skilled and experienced Databricks Architect to lead the design and implementation of data solutions on Azure Databricks. This role involves working closely with clients and internal teams to architect scalable, efficient, and secure data platforms that leverage Databricks and related Azure services. The ideal candidate will have extensive hands-on experience with Databricks and a deep understanding of big data processing, data engineering, and advanced analytics.

Job Responsibilities:

• Lead discussions with stakeholders to gather requirements, understand the data landscape, and identify opportunities for leveraging Databricks to solve complex data challenges. Facilitate discovery exercises and demonstrate proof of concepts.
• Architect end-to-end data solutions using Azure Databricks, Delta Lake, and other relevant Azure services. Focus on designing scalable ELT/ETL pipelines, optimizing data models for performance, and ensuring data quality and governance.
• Provide technical guidance to the implementation team, conducting detailed code reviews and ensuring that best practices in Databricks development, including performance optimization and cost management, are followed. Mentor and upskill team members in Databricks and related technologies.
• Develop and optimize complex data pipelines, leveraging the power of Databricks for big data processing, real-time streaming, and batch processing. Implement robust data engineering practices to ensure data reliability, efficiency, and scalability.
• Identify and resolve performance bottlenecks in Databricks workflows, optimize Spark jobs, and ensure efficient resource utilization in the cloud environment.
• Ensure robust data governance practices are integrated into Databricks solutions, including data security, lineage, and compliance with regulatory requirements.
• Work closely with cross-functional teams, including data scientists, analysts, and DevOps engineers, to integrate Databricks solutions into broader data architectures.

Technical Skills:

• Extensive experience in designing, building, and optimizing data pipelines and analytics solutions using Azure Databricks and Delta Lake.
• Advanced skills in PySpark, with the ability to write, debug, and optimize complex Spark jobs for both batch and streaming data.
• Strong experience in designing both logical and physical data models within Databricks, ensuring that models are optimized for performance and scalability.
• Proficient in integrating Databricks with other Azure services, such as Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Azure Synapse Analytics, and Azure SQL.
• Proven ability to tune Spark jobs for optimal performance, including resource management, cluster configuration, and efficient data partitioning strategies.
• Familiarity with implementing CI/CD pipelines for Databricks projects using tools like Azure DevOps and Terraform.
• Knowledge of implementing security best practices in Databricks, including encryption, role-based access control (RBAC), and integration with Azure Active Directory.

Why join Exponentia:

• Expand your knowledge and work with cutting-edge technologies.
• Opportunity to work with some of the best minds and collaborate with them.
• Learn from Industry experts.
• Get Certified with latest technologies and platforms.
• Get access to networks of OEM partners and business leaders who are setting new standards at the cutting-edge of technology.
• Exponentia.ai is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.