Data Engineer - Spark / Hadoop

Job Summary :

We are seeking an experienced GCP Data Engineer to join our team. The ideal candidate will have expertise in designing, building, and maintaining scalable data architectures on Google Cloud Platform (GCP).

Key Skills and Requirements :

- Kafka

- Apache Spark (SQL, Scala, Java)

- Python

- Hadoop Platform

- Hive

- AirFlow

- Cloud Composer

- PySpark

- GCP : experience with GCP services, including GCS, GDS, BigQuery, and Cloud Dataflow.

- Data Engineering : experience in data engineering, including data warehousing, ETL, and data pipelines.

- Programming languages : Proficiency in programming languages, including Python, Java, or Scala.

- Data Modeling : Understanding of data modeling concepts, including entity relationships and data normalization.

- Cloud Computing : Experience with cloud computing platforms, including GCP, AWS, or Azure.

- Data Security : Knowledge of data security and compliance, including data encryption and access control.

Key Responsibilities :

- Design and develop data-ingestion frameworks, real-time processing solutions, and data transformation frameworks using open-source tools.

- Operationalize open-source data-analytic tools for enterprise use.

- Ensure adherence to data governance policies by implementing or validating data lineage, quality checks, and classification.

- Follow the company development lifecycle to develop, deploy, and deliver solutions.

- Perform root-cause analysis on data-related issues.

- Design, build, and maintain scalable data architectures on GCP.

- Collaborate with cross-functional teams to identify and prioritize project requirements.

- Develop and maintain detailed technical documentation.

- Troubleshoot and resolve technical issues.

- Participate in code reviews and ensure adherence to coding standards.

- Stay up-to-date with the latest GCP features and best practices.

Preferred Qualifications :

- GCP certifications (e.g., Professional Data Engineer).

- Experience with data governance and DevOps practices.

Nice to Have Skills :

1. Experience with GCP Machine Learning and AI capabilities.

2. Knowledge of NoSQL databases, including Bigtable and Cloud Firestore.

3. Familiarity with DevOps tools, including Git and Jenkins.

4. Experience with Agile development methodologies, including Scrum and Kanban.

5. Certification in GCP development, including Professional Data Engineer certification.