Data_Engineering_M

1. Understand, implement, and automate ETL pipelines with better industry standards
2. Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, design infrastructure for greater scalability, etc
3. Developing, integrating, testing, and maintaining existing and new applications
4. Design, and create data pipelines (data lake / data warehouses) for real world energy analytical solutions
5. Expert-level proficiency in Python (preferred) for automating everyday tasks
6. Strong understanding and experience in distributed computing frameworks, particularly Spark, Spark-SQL, Kafka, Spark Streaming, Hive, Azure Databricks etc
7. Limited experience in using other leading cloud platforms preferably Azure.
8. Hands on experience on Azure data factory, logic app, Analysis service, Azure blob storage etc.