Big Data Engineer - Hadoop/Cloudera (6-12 yrs)

Responsibilities of the Candidate :

- Be responsible for the design and development of big data solutions.

- Partner with domain experts, product managers, analysts, and data scientists to develop Big Data pipelines in Hadoop

- Be responsible for moving all legacy workloads to a cloud platform

- Work with data scientists to build Client pipelines using heterogeneous sources and provide engineering services for data science applications

- Ensure automation through CI/CD across platforms both in cloud and on-premises

- Define needs around maintainability, testability, performance, security, quality, and usability for the data platform

- Drive implementation, consistent patterns, reusable components, and coding standards for data engineering processes

- Convert SAS-based pipelines into languages like PySpark, and Scala to execute on Hadoop and non-Hadoop ecosystems

- Tune Big data applications on Hadoop and non-Hadoop platforms for optimal performance

- Apply an in-depth understanding of how data analytics collectively integrate within the sub-function as well as coordinate and contribute to the objectives of the entire function.

- Produce a detailed analysis of issues where the best course of action is not evident from the information available, but actions must be recommended/taken.

- Assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients, and assets, by driving compliance with applicable laws, rules, and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct, and business practices, and escalating, managing and reporting control issues with transparency

Requirements :

- 6+ years of total IT experience

- 4+ years of experience with Hadoop (Cloudera)/big data technologies

- Knowledge of the Hadoop ecosystem and Big Data technologies Hands-on experience with the Hadoop eco-system (HDFS, MapReduce, Hive, Pig, Impala, Spark, Kafka, Kudu, Solr)

- Experience in designing and developing Data Pipelines for Data Ingestion or Transformation using Java Scala or Python.

- Experience with Spark programming (Pyspark, Scala, or Java)

- Hands-on experience with Python/Pyspark/Scala and basic libraries for machine learning is required.

- Proficient in programming in Java or Python with prior Apache Beam/Spark experience a plus.

- System level understanding - Data structures, algorithms, distributed storage & compute

- Can-do attitude on solving complex business problems, good interpersonal and teamwork skills.