Senior Data Engineer - Python Programming (5-10 yrs)

Job Description :

We are seeking a skilled and experienced Data Engineer to join our dynamic team. The Data Engineer will be responsible for designing, implementing, and maintaining our data infrastructure, pipelines, and systems.

This role involves collaborating with cross-functional teams to ensure efficient data flow, data quality, and optimal performance. The ideal candidate should have a strong background in software engineering, database management, and data architecture, along with proficiency in relevant technologies and tools.

Responsibilities :

- Lead the design and implementation of robust data pipelines for efficient extraction, transformation, and loading (ETL) of large datasets from diverse sources.

- Collaborate with stakeholders to understand business requirements, translating them into scalable data solutions tailored to end-user needs.

- Enhance and maintain our data architecture, ensuring scalability, reliability, and performance.

- Investigate and resolve data anomalies within ETL pipelines, ensuring data integrity and accuracy.

- Develop and maintain documentation for data processes, ensuring clarity and sustainability.

- Utilize AWS platform capabilities to architect and operate scalable production solutions.

- Implement data security practices and compliance measures to protect sensitive information.

- Build and manage data warehouses and data lakes to store and organize large datasets.

- Implement data quality checks and procedures to ensure data accuracy and consistency.

- Automate data workflows and processes using scripting languages and data pipeline orchestration tools.

- Collaborate with business analysts, onboarding teams, and business stakeholders to understand data needs and translate them into technical solutions.

- Monitor and troubleshoot data pipelines, identify bottlenecks, and optimize performance of data infrastructure to ensure scalability and efficiency.

- Troubleshoot issues with production data and/or inconsistencies and propose sustainable solutions to address them

- Document data pipelines and processes for maintainability and knowledge sharing.

- Stay updated with emerging technologies and best practices in data engineering, and propose innovative solutions.

- Document system architectures, processes, and procedures for knowledge sharing and future reference.

- Provide technical guidance and support to junior team members, and contribute to their professional development

Skills and Qualifications :

- Bachelor's degree in Computer Science, Information Technology, or a related field (or equivalent experience).

- A minimum of 5 years of experience as a Data Engineer or similar role.

- Strong proficiency in SQL and experience with relational databases (e.g., MySQL, PostgreSQL).

- Experience with cloud-based data platforms (e.g., AWS, Azure, GCP).

- Experience with data warehousing solutions (e.g., Snowflake, Redshift, BigQuery).

- Experience with data ingestion tools (e.g., Fivetran, Stitch, Kafka).

- Programming experience in Python, Java, or similar languages.

- Experience with scripting languages (e.g., Bash, Shell).

- Experience with data workflow management tools (e.g., Airflow, Luigi).

- Excellent problem-solving and analytical skills.

- Strong communication and collaboration skills.

- Ability to work independently and as part of a team.