Software Architect - Spark/Hadoop (7-12 yrs)

Job Description :

Responsibilities :

- Support hyper growth/new products from the Platform & DevOps perspective.

- Enhance and modernize the existing systems to be more reliable and efficient

- High performance systems : Experience with building high performance distributed systems that can scale to 10,000s QPS

- Core Infrastructure : Experience with developing and running large scale distributed storage systems, service oriented architectures, and reliable monitoring and deployment infrastructure. We use Python for the majority of our systems.

- Data Processing : Experience with building and maintaining large scale and/or real-time complex data processing pipelines using Spark, Kafka, Hadoop, Hive, Storm, and Zookeeper.

- Scale AI Palette backend services that support millions of data points with 100% accuracy

- Architect our platform systems to be more generic in order to support various fast-growing verticals and features

- Develop creative ways to increase organization's engagement from the platform's perspective. E. g., apply machine learning to increase relevancy of our data, scalably updating recommendations on real time data updates

- Adapt and optimize key pieces of evolving infrastructure

- Set best engineering practices and mentor junior team members.

- MLOps and LLMOps are good to have.

Requirements :

- 6-12 years of engineering experience.

- Expertise in Python. Go and Java are good to have

- Experience with developing complex software systems scaling to at least tens of thousands of users with production quality deployment, monitoring, and reliability.

- Experience with large-scale distributed storage and database systems (SQL & NoSQL, e. g. MySQL, ElasticSearch, Postgres ).

- Ability to decompose complex business problems and lead a team in solving them.

- Bonus points on experience in deploying LLMs at scale.

- Attractive salary

- Early joiners