Data Engineer

• Take ownership of the crawl pipeline, architecting, and implementing new solutions
• Keep crawl pipelines running with high uptime and scale them up to handle the load
• Establish an effective roadmap to meet development cycle deadlines
• Work with a wide range of systems and technologies

Job Requirements:

• Bachelor s/Master s degree in Engineering, Computer Science (or equivalent experience)
• At least 3+ years of relevant experience as a data engineer
• Prolific experience working with Ruby, Python, Crawlers, and GCP
• Demonstrable experience with JavaScript, SQL, and R
• Experience with large distributed infrastructure, consisting of hundreds of systems ranging from scrapers
• Familiarity with data processing clusters, databases, API servers, etc.
• Experience writing good software is a must
• Experience with distributed systems and big data
• Good intuition about what good architecture
• Excellent English communication skills, both written and verbal