Lead I - Big Data Developer

Senior Big Data Developer

About the Role:
We are looking for a Senior Big Data Developer with 5-6 years of hands-on experience in Hadoop, Spark, and AWS technologies. The role requires a deep understanding of Big Data tools like Hive, HBase, and Sqoop, along with expertise in Python and/or Scala. You will be working in a dynamic environment, contributing to data engineering efforts and optimizing data workflows to help scale our platforms.

Responsibilities:

• Develop and maintain Big Data solutions using Hadoop, Spark, HBase, Hive, and Sqoop.
• Build optimized Directed Acyclic Graphs (DAGs) in Airflow for scheduling and workflow management.
• Work with AWS services like Lambda, EC2, ECS, EMR, Athena, S3, DynamoDB, and IAM roles for cloud-based data processing and storage.
• Write efficient unit tests (positive and negative scenarios) and perform testing to ensure high-quality deliverables.
• Debug and troubleshoot issues in AWS CloudWatch and resolve cloud infrastructure-related issues.
• Contribute to the optimization and automation of data pipelines and workflows.
• Collaborate with cross-functional teams to design and implement scalable data solutions.
• Follow Agile practices to deliver timely solutions.

Requirements:

• Bachelor s degree in Computer Science, Engineering, or a related field.
• Minimum of 5-6 years of hands-on experience in Big Data technologies like Hadoop, Spark, HBase, Hive, and Sqoop.
• Proficient in Python and/or Scala programming languages for data processing.
• Strong understanding of RDBMS and No-SQL database architectures.
• Experience with AWS services such as Lambda, EC2, ECS, EMR, Athena, S3, DynamoDB, and IAM roles.
• Proficient in Git/Bitbucket for version control.
• Strong experience in debugging AWS CloudWatch for issue resolution.
• Experience working in Agile development environments.
• Good communication skills and a positive attitude.

Preferred Qualifications:

• Expertise in building and optimizing DAGs using Apache Airflow.
• Experience with data processing constructs like Joins and MapReduce.
• Familiarity with containerization and orchestration tools like Docker or Kubernetes.

Technical Skills:

• Big Data Tools: Hadoop, Spark, Hive, HBase, Sqoop
• Programming Languages: Python, Scala
• Cloud Services: AWS (Lambda, EC2, ECS, EMR, Athena, S3, DynamoDB, IAM)
• Version Control: Git, Bitbucket
• Other Tools: Airflow, CloudWatch
• Testing: Unit Testing (positive and negative scenarios)