PIQUOTA - Senior Data Engineer - Hadoop/Python (5-7 yrs)

Key Responsibilities :

- Design and develop data ingestion frameworks using technologies like Kafka.

- Build real-time data processing solutions using Apache Spark (SQL, Scala, Java).

- Develop and implement data processing and transformation frameworks using Python and Hadoop.

- Experience with GCP Cloud services like Cloud Composer, BigQuery, and DataProc.

- Operationalize open-source data analytics tools for enterprise use.

- Ensure data governance policies are followed by implementing or validating data lineage, quality checks, and data classification.

- Understand and follow the company's software development lifecycle for development, deployment, and delivery of solutions.

- Participate in on-call support rotation with other team members.

Qualifications :

- Bachelor's degree in Computer Science, Computer Information Systems (CIS), or a related field.

- 5-7 years of experience in software engineering or a related field.

- Proven experience with implementing software development life cycles (SDLC) in projects.

Strong hands-on experience with :

- Kafka

- Apache Spark (SQL, Scala, Java)

- Python

- Hadoop Platform

- Hive

- Airflow

- Experience with GCP Cloud services : Cloud Composer, BigQuery, DataProc

- Strong understanding of data warehousing and data lake concepts.

- Experience with data quality and data governance best practices.

- Excellent problem-solving and analytical skills.

- Strong communication and collaboration skills.

Preferred Qualifications :

- Experience with containerization technologies (Docker, Kubernetes)

- Experience with NoSQL databases (MongoDB, Cassandra)

- Experience with data visualization tools (Tableau, Power BI)