Data Engineer - Python

Responsibilities:

Creating, maintaining data infrastructure that can support large amounts of data processing Designing, building scalable data pipelines and API integrations to support growing data volumes Implementing right design patterns, involving message queues, multi-threading / processing Implement best practices, code reviewing, and ability to adhere to overall architecture principles Making improvement and process recommendations that have an impact on the business Solving complex issues involving very large data volumes, with minimal supervision Able to design pipelines which are cost effective, being mindful of the infra optimization goals Follow, learn and apply new big data technologies and innovations

Required experience, skills

6+ years of hands-on data engineering development experience in Python Advanced knowledge of Python and Linux shell scripting Proficiency in SQL and Data Structures, experience in handling (transformations, read / write) huge volume of data using Python

Experience in performance improvement on data pipelines and a good understanding of Volume, Variety, Veracity, Velocity, and Value concepts in Big Data

• Good knowledge of algorithms and software design principles
• Experience in CI/CD pipelines with one of: Azure DevOps, AWS Code Pipeline, GitHub Actions Familiarity with Docker, Kubernetes, and cloud services (Azure, AWS, GCP) Proficient with Kafka, Flink, RabbitMQ or similar technologies Self-motivated, able to self-improve, learn new and complex Technologies
• In-depth knowledge and experience in agile processes and principles
• Summary: This JD is targeting a senior-level data engineer with extensive experience in building scalable data infrastructures, working with large datasets, optimizing performance, and applying cutting-edge technologies
• Strong Python and cloud experience are critical, as well as proficiency in Big Data tools and design patterns
• You will also need to be proactive in improving systems and aligning your work with business goals