Big Data Engineer - ETL/SQL (10-17 yrs)

Job Description :

We are looking for a Big Data Engineer to work on collecting, storing, processing, and analyzing large data sets. The primary focus will be on choosing optimal solutions for these purposes, then maintaining, implementing, and monitoring them. You will also be responsible for integrating these solutions with the architecture used across our clients.

Roles and Responsibilities :

- Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities.

- Developing and maintaining data pipelines, implementing ETL processes, monitoring performance, and advising on necessary infrastructure changes.

- Translating complex technical and functional requirements into detailed designs.

- Investigating and analyzing alternative solutions for data storage, processing, etc., to ensure the most streamlined approaches are implemented.

- Serving as a mentor to junior staff by conducting technical training sessions and reviewing project outputs.

Skills and Qualifications :

- Strong understanding of data warehousing and data modeling techniques.

- Proficient understanding of distributed computing principles (e.g., Hadoop v2, MapReduce, HDFS).

- Strong data engineering skills on GCP platforms, including Airflow, Cloud Composer, Data Fusion, Data Flow, Data Proc, BigQuery.

- Experience with building stream-processing systems, using solutions such as Storm or Spark Streaming.

- Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala.

- Experience with Spark, SQL, and Linux.

- Knowledge of various ETL techniques and frameworks, such as Flume, Apache NiFi, or DBT.

- Experience with messaging systems like Kafka or RabbitMQ.

- Good understanding of Lambda Architecture, along with its advantages and drawbacks.