Azure Data Engineer - DataLake/SQL (5-12 yrs)

Key Responsibilities :

Design and Architecture :

- Design and implement end-to-end data solutions leveraging Azure Data services such as Azure Data Lake, Azure SQL, Azure Synapse Analytics, and Azure Databricks.

- Collaborate with cross-functional teams to define data requirements and translate them into scalable and efficient data pipelines.

- Architect data models, design ETL/ELT processes, and optimize data storage strategies.

Data Engineering & ETL Pipelines :

- Develop, implement, and maintain data ingestion, transformation, and storage pipelines using Azure Data Factory, Azure Databricks, or Azure Synapse.

- Automate data workflows and ensure high data quality, performance, and reliability of data pipelines.

- Integrate and optimize data from multiple sources including structured, semi-structured, and unstructured data.

Cloud Data Solutions :

- Develop solutions in Azure cloud environments that ensure scalability, security, and cost optimization.

- Manage large-scale data processing and storage within Azure Data Lake, Azure SQL Database, and other cloud-native services.

- Utilize Azure DevOps for CI/CD integration and deployment of data pipeline projects.

Collaboration & Stakeholder Management :

- Work closely with business analysts, data scientists, and other stakeholders to ensure data solutions meet business needs.

- Translate business requirements into technical specifications and deliverables.

- Provide expert guidance on best practices for data governance, security, and compliance.

Performance Tuning & Optimization :

- Monitor and optimize data pipelines and processes for performance, scalability, and cost-effectiveness.

- Troubleshoot and resolve data pipeline issues to ensure seamless data operations.

Documentation & Reporting :

- Document processes, technical designs, and architecture for data solutions and pipelines.

- Generate reports on pipeline performance, data quality metrics, and optimization efforts.

Skills & Qualifications :

Technical Skills :

- Strong expertise in Azure Data Services (Azure Data Lake, Azure SQL Database, Azure Synapse Analytics, Azure Data Factory, Azure Databricks, etc.).

- Extensive experience in designing, developing, and managing ETL/ELT pipelines.

- Proficiency in SQL and hands-on experience with Azure SQL, Azure Cosmos DB, Azure Synapse Analytics, and Azure Blob Storage.

- Knowledge of data modeling, data warehousing concepts, and designing data solutions.

- Experience with Python, Scala, Spark, and Databricks is highly desirable.

- Hands-on experience with Azure DevOps for CI/CD processes, version control, and automating deployment pipelines.

Cloud & Infrastructure :

- Strong understanding of cloud architectures, infrastructure, and security principles in Azure.

- Familiarity with Azure Security best practices, Azure Active Directory, and data governance principles.

Data Engineering & Big Data :

- Experience working with big data technologies (Apache Spark, Hadoop, Kafka) and real-time data processing.

- Knowledge of working with both structured and unstructured data.

Analytical & Problem-Solving :

- Excellent troubleshooting and performance tuning skills.

- Strong analytical thinking and ability to optimize complex systems.

Soft Skills :

- Excellent communication and interpersonal skills to collaborate effectively with cross-functional teams and stakeholders.

- Ability to articulate complex technical concepts to non-technical audiences.

- Strong problem-solving and critical-thinking abilities.