Azure Data Engineer Data Lakes

We are seeking a skilled Azure Data Engineer with expertise in designing, implementing, and managing Azure Data Lakes. The successful candidate will have deep experience in building scalable and efficient data lake solutions, integrating structured and unstructured data, and enabling advanced analytics. This role involves collaborating with data scientists, analysts, and other stakeholders to develop high-performing data platforms on Azure.

Key Responsibilities:

1. Data Lake Design and Implementation:
   
• Architect, build, and optimize Azure Data Lake Storage (ADLS) for efficient data storage and retrieval.
• Manage data lake structures (e.g., raw, processed, curated layers) and implement data lake best practices.
2. Data Ingestion and Integration:
   
• Design and implement robust ETL/ELT pipelines to ingest data from diverse sources into the data lake using Azure Data Factory or Databricks.
• Implement data streaming solutions with Azure Event Hubs, Kafka, or Azure Stream Analytics.
3. Data Transformation and Processing:
   
• Process large-scale datasets using Databricks, Spark, Python, or PySpark.
• Enable real-time and batch processing within the data lake environment.
4. Security and Governance:
   
• Enforce data security protocols, including Azure RBAC, encryption, and access controls.
• Implement data governance frameworks with tools like Azure Purview to ensure compliance and data quality.
5. Data Modeling and Analytics Enablement:
   
• Develop logical and physical data models to support analytical workloads.
• Collaborate with data scientists and analysts to expose data for BI and machine learning use cases.
6. Performance Optimization:
   
• Optimize data storage, access, and compute within the Azure ecosystem.
• Monitor and troubleshoot data lake pipelines and workflows for performance and reliability.
7. Collaboration and Documentation:
   
• Work closely with stakeholders to understand data requirements and ensure alignment with business goals.
• Document architecture, data pipelines, and processes for scalability and reusability.

Qualifications and Skills:

• Required:
  
• Bachelors degree in Computer Science, Information Systems, or a related field.
• 3+ years of experience in building and managing Azure Data Lakes.
• Proficiency in Azure Data Factory, Databricks, Azure Synapse Analytics, and Azure Data Lake Storage (Gen2).
• Expertise in Python, PySpark, and SQL.
• Familiarity with data lake architecture principles, including partitioning, schema evolution, and metadata management.
• Knowledge of Delta Lake for handling data consistency in large datasets.
• Preferred:
  
• Azure certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect Expert).
• Experience with big data ecosystems (e.g., Hadoop, Hive).
• Understanding of data governance and lineage tools like Azure Purview.
• Familiarity with containerization and orchestration tools like Docker and Kubernetes.

Key Competencies:

• Strong problem-solving and analytical skills.
• Excellent communication and collaboration abilities.
• Adaptability to emerging Azure technologies and best practices.
• Detail-oriented with a focus on data quality and accuracy.