Gloify - Data Engineer - ETL/Spark/Big Data Tools (4-8 yrs)

Responsibilities :

- Build, maintain and optimize the data pipeline architecture

- Develop ETL pipelines based on the requirements following best practices, specifications, and standards

- Work with project teams to provide input and assistance in the design, development, and support of ETL and database components following standard software development lifecycle.

- Work with the team to ensure that ETL and database components conform to global standards via thorough code reviews

- Mentor junior programmers in the team and plan their skill development

- Determine the technical needs for the team to work on a project assigned or third-party tools

- Documenting the data requirements, data format and tool dependencies

- Flexibility and ability to adapt to new technological changes as per project need.

Requirements :

- 3+ years of experience in Big data.

- Knowledge of multiple cloud storage options like S3, Redshift, or Bigquery is a must.

- Should be well versed with MySQL, NoSQL.

- Knowledge of Pandas and Spark is a must.

- Knowledge of Python, SQL, shell script is a must and Javascript (Node.js) is good to have.

- Knowledge of streaming services like AWS streams / Kafka / Google Pub/sub is good to have.

- Knowledge of Queue systems like Rabbit MQ, SQS and building pipelines using the same is very much needed.

- Knowledge on Dockers, Kubernetes is good to have.

- Familiarity with multiple cloud providers (AWS, Azure & GCP) is good to have.

- Should be familiar with strategies for Real-time analytics and batch processing.

- Masters or Bachelor's degree in Computer Science or the equivalent in experience.