Senior Data Engineer - Spark/Python/Scala

Mandatory Skills :

GCP Cloud Services, Spark, Scala or Python, Hive, Hadoop, BigQuery Architecture, LookML, DataProc, Data Modeling, Query Optimization,

Location : Bengaluru [Hybrid - 2 days/week]

Skills :

- Developing and maintaining data pipelines for efficient data extraction, transformation, and loading (ETL) processes.

- Designing and optimizing data storage solutions, including data warehouses and data lakes.

- Ensuring data quality and integrity through data validation, cleansing, and error handling.

- Collaborating with data analysts, data architects, and software engineers to understand data requirements and deliver relevant data sets (i.e., for business intelligence).

- Implementing data security measures and access controls to protect sensitive information.

- Automating and improving data processes and workflows for scalability and efficiency.

- Monitoring data infrastructure for performance and reliability to address issues promptly.

- Keeping abreast of industry trends and emerging technologies in data engineering.

- Documenting data pipelines, processes, and best practices for knowledge sharing.

- Participating in data governance and compliance efforts to meet regulatory requirements.

- Providing technical support and mentoring to junior data engineers, if applicable.

- Continuously optimizing data architecture to support the company's evolving data needs.

- Collaborating with cross-functional teams to drive data-driven decision-making within the organization.

Requirements :

- Strong knowledge of data processing in Spark (Scala or Python), Hive, Hadoop.

- Experience with data modeling and query optimization.

- Deep understanding of BigQuery architecture, best practices, and performance optimization.

- Proficiency in LookML for building data models and metrics.

- Experience with DataProc for running Hadoop/ Spark jobs on GCP.

- Knowledge of configuring and optimizing DataProc clusters