Data Engineer - SQL/Spark/Spark (2-5 yrs)

Job Description :

- Proven experience in in SQL, Spark, Hadoop ecosystem

- Have worked on multiple TBsof data volume from ingestion to consumption.

- Work with business stakeholders to identify and document high impact business problems and potential solutions.

- Good understanding of Data Lake/Lakehouse architecture and experience/exposure to Hadoop (cloudera, hortonworks) and/or AWS.

- Work on end-to-end data lifecycle from Data Ingestion, Data Transformation and Data Consumption layer.

- Versed with API and its usability.

- Suitable candidate will also be proficient in Spark, Spark Streaming, hive, SQLs.

- Suitable candidate will also demonstrate experience with big data infrastructure inclusive of MapReduce, Hive, HDFS, YARN, HBase, Oozie, etc.

- The candidate will additionally demonstrate substantial experience and a deep knowledge of relational databases.

- Good skills in technical debugging of the code in case of issues.

- Also, working with git for code versioning

- Creating Technical Design Documentation of the projects/pipelines

Secondary Responsibilities :

- Ability to work independently and handle your own development effort.

- Excellent oral and written communication skills Learn and use internally available analytic technologies.

- Identify key performance indicators and create educational/deliverables path to achieve the same.

- Use educational background in data engineering and perform data mining analysis.

- Work with BI analysts/engineers to create prototypes.

- Engage in the delivery and presentation of solutions.