Vertoz - Data Engineer - Python/Apache Spark (3-5 yrs)

What You Will Do :

- Responsible for the documentation, design, development, and architecture of Hadoop applications.

- Must have at least 3 years of hands-on working knowledge on Big Data technologies such as Impala, Hive, Hadoop, Spark, Spark streaming, Kafka etc.

- Excellent programming skills in python.

- Experience with stream-processing systems : Storm, Spark-Streaming, etc.

- Experience with relational SQL and NoSQL databases, including Vertica.

- Experience with cloud services.

- Cloudera Hadoop Distribution, Shell Scripting, Superset, Hands on with cluster management.

- Development : Create and maintain scalable big data applications using Python, Spark, Hive, and Impala.

- Data Pipelines : Develop and optimize data processing pipelines to handle large datasets.

- Integration : Implement data ingestion, transformation, and loading processes.

- Collaboration : Work with data scientists and analysts to meet data requirements.

- Quality Control : Ensure data quality, integrity, and security.

- Performance : Monitor and troubleshoot performance issues to improve efficiency.

- Documentation : Participate in code reviews, testing, and documentation.

- Learning : Stay updated with industry trends and advancements in big data technologies.

Requirements :

- Bachelor's or Master's degree in Computer Science, IT, or related field.

- 3 years in a Big Data Developer role.

- Proficiency in Python.

- Strong experience with Apache Spark.

- Hands-on experience with Hive and Impala.

- Familiarity with Hadoop, HDFS, Kafka, and other big data tools.

- Knowledge of data modeling, ETL processes, and data warehousing concepts.

- Soft Skills : Excellent problem-solving, communication, and teamwork skills.