Senior Data Engineer - ETL/Python (8-12 yrs)

This job description outlines the responsibilities and qualifications for a Data Engineer, focusing on experience with cloud platforms, particularly Google Cloud Platform (GCP) and AWS. It seeks an individual with solid technical expertise and the ability to collaborate with various teams to design, develop, and optimize data workflows. The role requires working with large datasets, ensuring data integrity, and leveraging GCP tools to manage and process data efficiently.

Key Responsibilities

1. Design, Develop, and Optimize Scalable Data Pipelines :

The primary responsibility is to design and build robust data pipelines on GCP, ensuring they are scalable to handle increasing data loads. This includes optimizing the pipelines for speed, efficiency, and reliability.

2. Collaboration Across Teams :

The role involves working with cross-functional teams, which could include data scientists, analysts, product teams, and more. This collaboration is essential for collecting, processing, and analyzing large datasets, ensuring that data is readily available for various use cases across the organization.

3. ETL Implementation and Management :

The candidate will be tasked with implementing ETL (Extract, Transform, Load) processes. These processes are crucial for extracting data from various sources, transforming it into the required format, and loading it into storage systems like data warehouses. Ensuring the availability and integrity of this data is a key part of the role.

4. Utilizing GCP Tools :

The job requires expertise in various GCP tools like BigQuery, Dataflow, and Dataproc. BigQuery is used for analyzing large datasets, Dataflow for streamlining ETL pipelines, and Dataproc for running big data workloads. Knowledge of these tools will be essential for handling data efficiently in a cloud environment.

5. Performance Monitoring and Optimization :

The Data Engineer will monitor and fine-tune the performance of data workflows and pipelines. This includes identifying bottlenecks and finding ways to enhance the speed and reliability of the data processing systems.

6. Data Governance and Security :

Adherence to data governance and security standards is another crucial responsibility. This ensures that all data engineering tasks comply with internal data management policies, industry standards, and regulatory requirements, safeguarding the integrity and confidentiality of the data.

7. Automation of Repetitive Tasks :

The candidate will automate recurring data engineering tasks to enhance productivity and reduce human error. This could involve scripting, job scheduling, and other automation practices to streamline the workflow.

8. Technical Guidance :

Providing mentorship and technical support to junior engineers and other stakeholders is a part of the role. The candidate is expected to lead by example and guide others in best practices and technical problem-solving.

Required Experience and Skills :

1. Experience :

The ideal candidate should have between 6-8 years of experience in data engineering, with at least 3 years specifically focused on GCP. This experience ensures that the candidate has a deep understanding of cloud-based data processing and can effectively use GCP's suite of tools.

2. Technical Skills :

A strong grasp of key technologies like GCP tools (BigQuery, Dataflow, Dataproc), Python, SQL, and ETL processes is essential. These skills are fundamental to building, optimizing, and maintaining the data pipelines and workflows.

3. Educational Background :

A bachelor's degree in Computer Science, Information Technology, or a related field is expected. This ensures the candidate has a foundational understanding of the technical aspects of data engineering.

4. Certifications :

Certifications like the GCP Professional Data Engineer certification are preferred, as they demonstrate the candidate's formal knowledge of Google Cloud services and best practices in data engineering.

5. Soft Skills :

The role demands excellent analytical, problem-solving, and communication skills. The ability to work in fast-paced environments and collaborate effectively with multiple teams is also important.

6. Additional Knowledge and Tools :

Familiarity with CI/CD pipelines, version control (e.g., Git), and DevOps practices would be beneficial, as these tools and practices are often used in modern data engineering workflows.

In summary, this job description calls for an experienced and technically proficient Data Engineer with specialized skills in GCP and AWS, capable of building scalable data pipelines, managing large datasets, ensuring data integrity, and automating tasks to improve efficiency. The role requires strong collaboration, problem-solving skills, and a solid understanding of data governance, cloud architecture, and modern data processing tools.