Epergne Solutions is looking for Azure Data Engineers across Locations. Work Location :Bangalore / Hyderabad / Pune. Mode:Hybrid. Role:Azure Data Engineer to design, build, and maintain data infrastructure in Microsoft Azure, ensuring data flows smoothly from various sources to destinations for analytics and reporting purposes.
Key Responsibilities. Design and Develop Data Pipelines:. Build and optimize scalable data pipelines using Azure Data Factory (ADF) and Azure Synapse Analytics.
Develop, maintain, and improve data integration, processing, and storage pipelines for ETL/ELT processes.
Ensure data consistency, accuracy, and reliability across the pipelines.
Data Modeling and Architecture:. Design and implement data models (Star, Snowflake, and OLAP) based on business requirements.
Collaborate with data architects to develop and refine the data warehouse/lake architecture using Azure Synapse Analytics or Azure Data Lake Storage.
Data Transformation and Storage:. Implement data transformation using Azure Databricks, SQL, and other tools.
Manage and monitor data storage using Azure Data Lake and SQL Databases.
Optimization and Performance Tuning:. Optimize data processes for performance, scalability, and cost-efficiency in an Azure cloud environment.
Security and Compliance:. Implement data security best practices, including data encryption, role-based access control, and monitoring using Azure Security and Azure Monitor.
Collaboration:. Work closely with data scientists, business analysts, and other stakeholders to gather requirements and ensure data pipelines meet business needs.
Monitoring and Maintenance:. Set up automated monitoring, alerts, and logging for data pipelines.
Troubleshoot and resolve data pipeline issues and ensure minimal downtime.
Technical Skills. ETL/ELT design and development experience using Azure Data Factory or similar tools.
Expertise in SQL, PySpark, and Python for data manipulation and transformation.
Proficient in Azure DevOps for CI/CD pipeline management and version control.
Strong understanding of data warehousing and data lake architectures.
Hands-on experience with Azure Databricks for big data processing and machine learning integration