Amossys Consulting Services - Senior Azure Databricks Specialist - Python/PySpark (5-8 yrs)

Job Description :

Key Responsibilities :

- Design, develop, and implement data solutions using Azure Data Bricks and Azure Data Factory.

- Build and optimize data pipelines for data ingestion, processing, and storage.

- Develop data transformation logic using Python, PySpark, Scala, or Hive.

- Work extensively with Azure SQL Database, Azure SQL Warehouse, and Synapse Analytics for data integration and processing.

- Design and implement data models and schemas optimized for performance and scalability.

- Build and maintain CI/CD pipelines in Azure DevOps for automated deployment and testing of data solutions.

- Implement best practices for code management, testing, and release processes.

- Collaborate with cross-functional teams (data engineers, data scientists, business analysts) to understand data requirements and design optimal data workflows.

- Identify and resolve performance bottlenecks and optimize data pipelines for efficiency and cost-effectiveness.

- Adhere to best practices for data engineering, data governance, and security.

- Mentor and guide junior team members on Azure data technologies and best practices.

Qualifications :

Experience : 5+ years of hands-on experience with Azure Data Bricks and Azure Data Factory, including designing, developing, and deploying production-ready data solutions.

Technical Skills :

- Deep expertise in Azure Data Bricks and its functionalities.

- Strong experience with Azure Data Factory for building and managing data pipelines.

- Proficiency in at least one of the following languages: Python, PySpark, Scala, or Hive.

- Hands-on experience with Azure SQL Database, Azure SQL Warehouse, and Synapse Analytics.

- Expertise in building and managing CI/CD pipelines in Azure DevOps.

Other Skills :

- Excellent problem-solving and analytical skills.

- Strong communication and collaboration abilities.

- Experience working in Agile development methodologies is a plus.

Preferred Skills :

- Experience with other Azure data services (Azure Cosmos DB, Azure Blob Storage).

- Knowledge of data warehousing concepts and best practices.

- Experience with data visualization tools (Power BI, Tableau).

- Experience with real-time data streaming technologies (Azure Event Hubs, Apache Kafka)