Azure Data Engineer

Knowledge and working experience in

1. Strong SQL Knowledge
2. Experience in Scala, SPARK is a must.
3. Python/Pyspark frameworks/libraries is optional.
4. Expertise of Azure Databricks; Azure Databricks integration with other services, Read and write data in Azure Databricks
5. Knowledge of Unity catalog, AZCopy,DataBrick Workflow, DeltaLake, Dockers , AKS.
6. Hands-on on Azure Data Factory Objects - ADF pipeline, configuration, parameters, variables, Integration services runtime, other activities(such as Copy, SP, lkp etc) and DataFlows , ADF data Ingestion and Integration with other services
7. Synapse Analytics- Import data into Azure Synapse Analytics with and without using PolyBase , Implement a Data Warehouse with Azure Synapse Analytics, Query data in Azure Synapse Analytics
8. End to end understand of Cluster management , able to debug issues if required.
9. Understand different types of data structure, Semi-Structure data, unstructured
10. Good to have Azure DevOps knowledge, Repository , Configuration Management, CI/CD.
11. Experienced Scheduling jobs using Airflow/ESP
12. Working with business and peers to define, estimate, and deliver functionality.
13. Should have gone through entire development life cycle including system integration testing
14. Provide Production Support for any issues raised by the Business and experience on
15. Production/Integration release management
    
16. Experience with GITLAB
17. Good knowledge of Data Warehousing concepts & Data Modelling
18. Should have experience working on complex data architecture
19. Experience in working for a healthcare client is a plus
20. Communicate progress and other relevant information the project stakeholders
21. Excellent communication and documentation skills
22. Any AZURE certification is good to have like DP-900, DP-203, DP-204

Experience:

4 -7 years total experience

Soft skills:

1. Right communication skills to be able to work with Business.
2. Good logical reasoning, analytical abilities.
3. Experience working with remote teams.
4. Work to become the trusted partner of the Architects by regular engagement and proposals on best practices which can be implemented and would bring change to the process
5. Support peers by sharing knowledge and mentoring.

Agile skills:

1. Experience in Agile development methodology.
2. Create Acceptance Criteria for all the stories/ tasks that are testable
3. Participate in all Agile ceremony meetings and update task status and adhere to timelines and quality standards for the task.

Roles and Responsibilities:

• Create ADF pipeline, Create a data brick Notebook , Scheduling workflow job, analysis the execution.
• Perform POC to Load data from different system Oracle, Sql server, files to databricks.
• Execution of workflow using external orchestration tool like Airflow, ESP
• Prepare reports, extract data to Azure Storage.
• Review business requirement specifications, technical specification, walkthrough.
• Test case creation, review and walkthrough.
• Involved in Sanity testing, System Integration testing, Regression testing, Integration testing, UAT support.
• Test Execution, Results Analyzing and Defect Reporting.
• Data analysis, data mapping, data loading and data validation.
• Coordination with offshore team for project execution