Big Data Engineer - Azure Data Services (5-6 yrs)

Job Title : Big Data Engineer

Location : Chennai, Bangalore, Hyderabad

Key Responsibilities :

- Design, develop, and implement data pipelines, leveraging Azure Data Factory, Azure Synapse Analytics, Azure SQL, and Azure Data Lake.

- Create scalable, efficient, and robust data solutions to manage data ingestion through APIs and streaming data sources, optimizing performance and reliability.

- Develop and deploy cloud-based analytics and data solutions to support various business intelligence and data analysis requirements.

- Build infrastructure for data analysis and machine learning workflows using best practices in Azure cloud environments.

- Build, test, and maintain efficient data pipelines to collect, manage, and transform large datasets from diverse sources, ensuring data quality and accessibility.

- Use SQL scripting and Stream Analytics to handle real-time and batch processing requirements effectively.

- Implement DevOps processes including CI/CD and Infrastructure as Code (IaC) to streamline development, testing, and deployment of data solutions.

- Collaborate with DevOps teams to integrate automation and version control in data pipeline processes.

- Develop and enforce security models to protect sensitive data within the cloud environment.

- Ensure compliance with data privacy regulations and implement robust security protocols for all data solutions.

- Analyze complex data sets to support decision-making and derive actionable insights.

- Work closely with data analysts and business teams to understand requirements and troubleshoot any data issues promptly.

Required Skills & Qualifications :

- 5+ years in IT service delivery with experience working in global onshore and offshore teams.

- 3+ years of hands-on experience in the Azure Analytics space, including Azure Data Factory, Azure Synapse, Azure SQL, and Data Lake.

- Strong knowledge of SQL scripting and familiarity with Python or Spark-SQL for data processing tasks.

- Experience with data ingestion, transformation, and pipeline development using API and streaming methods.

- Familiarity with DevOps tools and practices, including CI/CD and infrastructure management.

- Ability to develop and implement secure data models in cloud environments, with a focus on data protection and regulatory compliance.

- Excellent data analysis, problem-solving, and troubleshooting skills, with attention to detail and a strong analytical mindset.

- Ability to work effectively in a fast-paced, agile development environment and collaborate with cross-functional teams.

- Certified as an Azure Data Engineer or higher, which is a plus.