Lead Data Engineer - PySpark/SQL (5-9 yrs)

Job Title : Data Engineer.

Contract Duration : 12 months.

Total Experience Required : 5-9 years.

Relevant Experience Required : 5+ years.

Job Description :

We are seeking experienced Data Engineers to join our team for a 12-month contract position.

As a Data Engineer, you will play a key role in the development, deployment, and management of data engineering solutions.

You will collaborate closely with cross-functional teams, contributing to the data warehousing and reporting infrastructure, ensuring smooth data processing pipelines, and enabling business insights.

Mandatory Skills :

- Scripting & Automation : Expertise in Python/Pyspark for scripting, data manipulation, and automation.

- SQL Development : More than 3 years of expert-level experience in SQL development, writing complex queries.

- Data Analysis : Perform data analysis, analyze SQL scripts to understand business logic, and assess Unix job operations.

- PL/SQL & UNIX Shell Scripting : Hands-on experience in writing PL/SQL and UNIX shell scripting.

- Scheduling & Testing : Exposure to Control-M scheduling and the ability to perform efficient testing.

- Version Control & CI Tools : Experience with Git, Jenkins, GitHub, and Atlassian tools such as Jira/Confluence.

- Banking Domain Experience : At least 3 years of experience in the banking domain.

- Data Solutions Management : Experience in deploying and managing data solutions and data warehouse environments.

- Data Engineering Technologies : Proficiency in technologies like Spark, Hadoop, and Kafka.

- Data Warehousing Knowledge : Experience with SQL, OLTP, OLAP, and DSS systems.

- Programming Language : Comfort with Python, Scala, or Java.

- AWS Fundamentals : Strong understanding of AWS cloud services.

- Knowledge of NoSQL databases such as MongoDB or Cassandra for handling unstructured or semi-structured data.

Good to Have Skills :

- Cloud Technologies Knowledge : Experience with cloud technologies and have worked in a data operations position for a large Enterprise.

- Infrastructure as Code : Familiarity with Ansible, Terraform, CloudFormation, Docker, ECS, Kubernetes, or Packer.

- Data Modeling : Experience in data modeling is advantageous.

Responsibilities :

- Collaborate effectively with the development team to implement data warehouse and reporting systems.

- Work closely with data scientists to determine metadata querying requirements.

- Understand and determine business needs for data reporting and contribute to solutions.

- Lead initiatives to drive change across the project and continuously improve data engineering processes.

- Maintain hands-on involvement in coding, problem-solving, and researching new technologies.

- Note : Background verification will be conducted before onboarding.

- The interview will be conducted either Face-to-Face or on Skype.