Senior Data Engineer - SQL/Python/ETL (5-9 yrs)

About the Role :

We are seeking a highly skilled and experienced Senior Data Engineer to join our growing data team. In this role, you will be responsible for designing, building, and maintaining robust and scalable data pipelines and infrastructure. You will work with large datasets, implement data processing frameworks, and ensure data quality and reliability. The ideal candidate will have a strong understanding of data engineering principles, excellent programming skills, and a passion for building efficient and reliable data systems.

Responsibilities :

Data Pipeline Development & Maintenance :

- Design, develop, and maintain scalable and reliable data pipelines for data ingestion, processing, and storage.

- Implement ETL/ELT processes using various data integration tools and technologies.

- Optimize data pipelines for performance, scalability, and cost-effectiveness.

- Monitor and troubleshoot data pipeline issues, ensuring data quality and reliability.

Data Infrastructure Management :

- Design and implement data storage solutions using cloud-based data warehouses and data lakes (e.g., Snowflake, BigQuery, Redshift, S3, ADLS).

- Manage and optimize data infrastructure, including data storage, processing, and compute resources.

- Implement data governance and security best practices.

Data Modeling & Architecture :

- Design and implement data models for relational and NoSQL databases.

- Develop and maintain data dictionaries and metadata repositories.

- Collaborate with data scientists and analysts to understand data requirements and design efficient data architectures.

Automation & Scripting :

- Automate data pipeline deployments and infrastructure provisioning using infrastructure-as-code (IaC) tools (e.g., Terraform, CloudFormation).

- Develop and maintain scripts for data processing and automation tasks (Python, SQL, Bash).

Performance Optimization & Tuning :

- Identify and resolve performance bottlenecks in data pipelines and infrastructure.

- Tune database queries and data processing jobs for optimal performance.

- Conduct performance testing and capacity planning.

Collaboration & Communication :

- Collaborate with cross-functional teams, including data scientists, analysts, and software engineers.

- Communicate technical concepts and solutions effectively to both technical and non-technical audiences.

- Document data engineering processes and best practices.

Technical Skills :

Programming Languages :

- Proficiency in Python and SQL.

- Experience with other programming languages (e.g., Java, Scala) is a plus.

Data Processing Frameworks :

- Experience with distributed data processing frameworks (e.g., Spark, Hadoop, Flink).

- Familiarity with data streaming technologies (e.g., Kafka, Kinesis).

Cloud Platforms & Data Warehouses :

- Experience with cloud platforms (AWS, Azure, GCP) and their data services (e.g., S3, ADLS, BigQuery, Redshift, Snowflake).

- Strong understanding of data warehousing and data lake concepts.

Databases :

- Experience with relational and NoSQL databases (e.g., PostgreSQL, MySQL, MongoDB, Cassandra).

- Strong SQL skills and experience with database optimization.

ETL/ELT Tools :

- Experience with ETL/ELT tools (e.g., Airflow, Talend, Informatica).

- Experience with data integration and orchestration tools.

Infrastructure as Code (IaC) :

- Proficiency in Terraform, CloudFormation, or similar tools.

Version Control :

- Proficiency in Git and version control best practices.

Operating Systems :

- Proficiency in Linux or other Unix-like operating systems.

Data Modeling :

- Experience with data modeling techniques and tools.

Qualifications :

- Experience : 5-9 years of professional experience in data engineering or a related field.

- Education : Bachelor's degree in Computer Science, Information Technology, or a related field.

Soft Skills :

- Excellent problem-solving and analytical skills.

- Strong communication and interpersonal skills.

- Ability to work independently and as part of a team. -

- Strong attention to detail and a commitment to quality.

- Ability to learn and adapt quickly.

Benefits :

- Competitive salary and benefits package.

- Opportunity to work with cutting-edge data technologies.

- Collaborative and supportive work environment.

- Opportunities for professional development and growth.

- Chance to make a significant impact on the company's data infrastructure