Big Data Developer

Role Purpose

The purpose of this role is to design, test and maintain software programs for operating systems or applications which needs to be deployed at a client end and ensure its meet 100% quality assurance parameters

We are seeking a skilled Big Data Engineer with expertise in managing complex migration or modernization use-cases in various platforms including Azure, Snowflake, Postgres, and programming languages such as SQL, & Python. The ideal candidate will have a strong understanding of data engineering concepts and hands-on experience.

Responsibilities:

• Responsible for Development, implementation & maintenance of various data feeds.
• Build data movement / aggregation pipelines that meet all functional & non-functional requirements
• Process structured and unstructured data, validate data quality, and help design data quality tests in a big data environment.
• Maintain the parser and automated processes (scheduler). Perform daily check to verify the integrity.
• Develop capabilities to support proper data extraction & curation for analytical purposes
• Creation and management of data pipelines, datasets, tables, and views to fulfil complex migration use-cases
• Implementing Python based data processing modules which runs on Kubernetes or Docker containers
• Implementing Micro batch Data Ingestion to Remote Databases, ADLS, Snowflake using Python
• Strong understanding and hands-on experience with data engineering concepts like Queues, Pub-Sub, NoSQL, and Streaming
• Modernising legacy scripts and workflows is an added advantage

Experience and Skillset:

• 3+ years of hands-on experience as a Data engineer / Developer (in Big Data technologies) Azure or On-prem.
• 3+ years of hands-on experience in developing and supporting Spark(Mandatory) applications in a distributed platform.
• 3+ years of hands-on experience in one of the programming languages like JAVA or Scala or Python
• Writing complex SQL statements and performing data discovery (Hive, RDBMS Mandatory)
• Good to have experience optimizing the data pipelines on cloud solutions such as Azure ADLS, Azure log analytics. (Preferred), Knowledge on HBase and Snowflake