Thoughtspot - Data Engineer - Spark/Kafka (4-8 yrs)

We are seeking an experienced Data Engineer to join our AI search team. As a Data Engineer at ThoughtSpot, you will be responsible for designing, building, and maintaining the data infrastructure that powers our analytics platform and underlying Natural Language Search engine.

Responsibilities :

- Develop and maintain scalable data pipelines to process large volumes of data from various sources.

- Working closely with our product teams and ML engineers to build features that directly impact the robustness and accuracy of our search infrastructure and NLS engine.

- Ensure data quality and consistency through rigorous testing and validation processes.

- Monitor and troubleshoot data pipeline performance and resolve any issues.

Requirements :

- 4+ years of experience.

- Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases/warehouses.

- Experience building and maintaining large data pipelines, data infrastructure, (Kubernetes preferred), and Machine learning infrastructure.

- Working knowledge of message queuing, stream processing, and highly scalable data stores.

- Experience working with data tools: Spark, Kafka, etc.

- Knowledge of data warehousing concepts and experience with databases like Snowflake, and Redshift with experience with data model design, and optimization for data warehousing.

- Experience with data pipeline and workflow management tools like Dragster or Airflow with code and no-code ETL implementations.

- Experience with cloud services such as AWS, Azure, etc.

- Proficiency in programming languages such as Python and data processing libraries such as Pandas and Numpy.

- Experience with Java is good to have.

- Strong problem-solving skills and the ability to work independently and as part of a team.

- Advanced knowledge of development good practices such as testing, code reviews, and git.