Data Pipeline Engineer

• Aggregate data from both internal and external data sources to build a data processing system
• Create new pipelines and build reusable components at scale to support reporting & analytics data products
• Write complex queries to transform raw data sources into accessible models by coding
• Clean, prepare and optimize data at scale for integration and consumption
• Implement data management projects and re-structure of current web architecture
• Implement automated workflow using workflow scheduling processes like apache airflow
• Solve complex data issues and perform root cause analysis to proactively resolve product issues
• Own the data pipeline and support systems failure
• Collaborate with relevant stakeholders taking learning and leadership opportunities

Job Requirements:

• Bachelor s/Master s degree in Engineering, Computer Science (or equivalent experience)
• At least 5+ years of relevant experience as a software engineer
• Extensive experience in Python, SQL, AWS, Airflow, or Glue
• Nice to have experience in Elasticsearch or Apache Druid, Kafka and Parquet, or Avro
• A minimum of 3+ years of experience in SQL, Python, and Bash in a data-focused role
• At least 3+ years of proven experience in engineering complex data pipelines with low latency and redundancy
• Must have 3+ years of experience building cloud scalable and high-performance data warehouse solutions using AWS products, including S3, Athena, Redshift, EMR
• Exposure to open source and proprietary cloud data pipeline tools such as Airflow and Glue
• Solid understanding of data structures and algorithms
• Understanding of code versioning tools such as Git
• Passionate about data solutions
• Nice to have familiarity with other distributed data stores (Elasticsearch, Apache Druid)
• Familiarity with streaming data experience with Kafka
• Demonstrable experience with binary data serialization formats such as Parquet, Avro
• Fluent in verbal and written English