Senior Data Engineer

Senior Data Engineer

Diggibyte Technologies is a technology consulting and services company. We help organizations build innovative products and solutions using big data and analytics on the cloud premises.

We are looking for an experienced Senior Data Engineer (Exp. 7-10 years) to join our team in Bengaluru/Hosur/Remote. Someone who can help a client to build scalable, reliable, and secure Data analytic solutions.

Skills Required:
4+ years of total IT experience with 4+ years in big data engineering and Microsoft Azure.
Experience in implementing Data Lake with technologies like Azure Data Factory (ADF), PySpark, Databricks, ADLS, Azure SQL Database.
A comprehensive foundation with working knowledge of Azure Full stack, Event Hub & Streaming Analytics.
A passion for writing high-quality code and the code should be modular, scalable, and free of bugs (debugging skills in SQL, Python, or Scala/Java).
Enthuse to collaborate with various stakeholders across the organization and take complete ownership of deliverables.
Experience in using big data technologies like Hadoop, Spark, Databricks, Airflow, Kafka. 
Adept understanding of different file formats like Delta Lake, Avro, Parquet, JSON, and CSV.
Good knowledge of building and designing REST APIs with real-time experience working on Data Lake/Lakehouse projects.
Experience in supporting BI and Data Science teams in consuming the data in a secure and governed manner.
Certifications like Data Engineering on Microsoft Azure (DP-203) or Databricks Certified Developer (DE) are a valuable addition.

Responsibilities:
As a Big Data Engineer (Azure), you will build and learn about a variety of analytics solutions & platforms, data lakes, modern data platforms, data fabric solutions, etc. using different Open Source, Big Data, and Cloud technologies on Microsoft Azure.
Design and build scalable & metadata-driven data ingestion pipelines (For Batch and Streaming Datasets).
Conceptualize and execute high-performance data processing for structured and unstructured data, and data harmonization.
Schedule, orchestrate, and validate pipelines.
Design exception handling and log monitoring for debugging.
Ideate with your peers to make tech stack and tools-related decisions.
Interact and collaborate with multiple teams (Vendors/Consultants/Data Science & Dev team) and various stakeholders to meet deadlines, to bring Analytical Solutions to life.
Should be able to understand complex architectures and be comfortable working with multiple teams.
Delivery teams to build, automate, and deploy cloud solutions on Microsoft Azure.

Monitor production, staging, test, and development environments for a myriad of applications in an agile and dynamic organization.
Should be well-versed in Security Skills.
Should have a Customer-Focused approach and understanding the problems.
Proactiveness is must and self-driven.