Senior Data Engineer - Azure Cloud Services (5-8 yrs)

Technical Experience and Expertise :

- When working as a Senior Data Engineer you will leverage azure stack tools based on requirements for developing data integration workflows.

- Familiarity with these tools and an understanding of when to use each one is crucial for optimizing data processing, movement, and transformation in a cloud-based environment.

1. Azure Data Factory, Fivetran, Talend, Informatica etc.

2. Azure Logic Apps\Function Apps.

3. Azure Stream Analytics.

4. Azure Data Lake Storge.

5. Azure Databricks.

6. Azure Synapse Analytics & Azure Data Explorer.

7. Snowflake.

8. Power BI - Optional.

Data Integration and Design :

- Implement data integration and ETL processes to ingest data from various sources into Azure data storage solutions.

- Ensure efficient and secure data transfer and transformation.

- Pattern based templates combined with Control Framework models for processing large volumes of initial and incremental data with observability.

- Design and implement scalable, high-performance data solutions on the Azure cloud platform.

- Develop and maintain data models, schemas, and structures to support business needs.

Azure Services :

- Utilize a range of Azure data services such as Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Snowflake, and others to build end-to-end data solutions.

- Implement and optimize data storage solutions using Snowflake, Azure SQL Database, Azure Data Lake Storage, etc.

Data Processing and Analysis :

- Develop and optimize queries for data analysis and reporting.

- Implement data processing solutions for real-time and batch processing scenarios.

Data Security and Compliance :

- Implement security measures to protect sensitive data.

- Ensure compliance with data governance and regulatory requirements.

Monitoring and Optimization :

- Monitor and optimize data pipelines and processing for performance and reliability.

- Troubleshoot and resolve issues related to data engineering processes.

Collaboration :

- Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver effective solutions.

Documentation :

- Document data architecture, data pipelines, processes, and best practices.

- Create and maintain documentation for data engineering solutions.

Scalability :

- Design solutions that can scale with the growth of data and business needs.

- Implement strategies for data partitioning, indexing, and optimization.

Qualifications :

- Bachelor's degree in computer science, Information Technology, or related field.

- 5+ years of experience in data engineering, with a focus on cloud platforms.

- Strong proficiency in Azure services for data engineering.

- Experience with ETL tools, data modelling, and database design.

- Solid understanding of data warehousing concepts and best practices.

- Programming skills in languages such as Python, SQL, or others.

- Excellent problem-solving and troubleshooting skills.

- Strong communication and collaboration skills.