Data Engineer - Python/SQL (1-6 yrs)

Job Description :

The ideal candidate will have a deep understanding of data engineering principles and practices, and significant experience building and maintaining robust data pipelines and data infrastructure. This role will be heavily involved in designing, developing, and implementing data solutions that directly support our business needs, leveraging Python and SQL as primary tools.

Responsibilities :

- Design, build, and maintain scalable and reliable data pipelines using various technologies, with a focus on Python-based solutions (e.g., Spark with PySpark, Dask, Pandas, etc.) and SQL for data transformation and manipulation.

- Develop and implement ETL (Extract, Transform, Load) processes for integrating data from diverse sources, utilizing strong SQL skills for data extraction and transformation and Python for orchestration and complex logic.

- Ensure data quality and integrity through data validation and testing, implementing Python-based testing frameworks and SQL queries for data profiling and quality checks.

- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements, translating them into efficient Python and SQL code.

- Monitor and optimize the performance of data pipelines and data infrastructure, using Python for performance analysis and SQL for query optimization.

- Research and evaluate new data engineering technologies and tools, with a focus on Python libraries and SQL database technologies.

- Document data engineering processes and best practices, including Python code documentation and SQL schema definitions.

- Troubleshoot and resolve data-related issues, using Python debugging tools and SQL query analysis.

- Contribute to the development of data governance policies and procedures, utilizing Python for automation and SQL for data access control.

- Work in an Agile/Scrum environment. (If applicable)

- Proficiency in Python and Python Libraries especially Pandas.

- Strong understanding of architectural patterns, design principles, and coding best practices.

- Familiarity with cloud platforms (such as AWS, Azure, or Google Cloud) and their data-related services.

- Excellent problem-solving and analytical skills with the ability to translate business requirements into technical designs.

- Knowledge of air flows is advantageous.

- Effective communication skills, both written and verbal, to collaborate with stakeholders at different levels of the organization.