Kloud9 - Senior Data Engineer - PySpark (7-10 yrs)

Company : Kloud9 (https://www.kloud9.nyc/)

Location : Bangalore

Experience: 7-9 years

Mode : Hybrid (2 days work from office in a week)

Company Overview :

Founded in 2016 and headquartered in New York, Kloud9 is one of the leading technology services and solutions providers specializing in Data Science, Artificial Intelligence (AI) and Machine Learning (ML). At Kloud9, we work closely with our customers across Retail, Consumer Goods, Healthcare & Manufacturing to achieve a satisfying customer experience using modern technologies in AI, ML, Cloud and Automation, that transform Data into action-driven insights.

As a leading provider of cutting-edge cloud solutions, Kloud9 empowers businesses to harness the full potential of cloud technology, enabling them to scale, innovate, and thrive in the digital era. With a strong commitment to excellence and a customer-centric approach, Kloud9 is redefining the way organizations leverage the power of the cloud to drive growth and efficiency.

Through our customer-centric approach and relentless commitment to excellence, we have helped numerous organizations achieve remarkable results in their cloud journey. Our success stories highlight the transformative impact of our cloud solutions, including cost savings, increased productivity, improved scalability, and enhanced business agility. With our strong technology expertise, we understand & visualize a new world with an AI-First Approach. Kloud9 is constantly on the lookout for top-tier talent to join us in this exciting journey!

Responsibilities :

- Drive the implementation of new data management projects and re-structure of the current data architecture

- Clean, prepare and optimize data at scale for ingestion and consumption by machine learning models

- Implement complex automated workflows and routines using workflow scheduling tools

- Build continuous integration, test-driven development and production deployment frameworks

- Drive collaborative reviews of design, code, test plans and dataset implementation performed by other data engineers in support of maintaining data engineering standards

- Anticipate, identify and solve issues concerning data management to improve data quality

- Design and build reusable components, frameworks and libraries at scale to support machine learning products

- Design and implement product features in collaboration with business and Technology stakeholders

- Analyze and profile data for the purpose of designing scalable solutions

- Troubleshoot complex data issues and perform root cause analysis to proactively resolve product and operational issues

- Mentor and develop other data engineers in adopting best practices

- Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders

Qualifications :

- 6+ years of experience developing scalable Pyspark applications or solutions on distributed platforms

- Experience in Google Cloud Platform (GCP) and good to have other cloud platform tools

- Experience working with Data warehousing tools, including DynamoDB, SQL, and Snowflake

- Experience architecting data products in Streaming, Serverless and Microservices Architecture and platform.

- Experience with Pyspark, Spark (Scala/Python/Java) and Kafka

- Work experience with using Databricks (Data Engineering and Delta Lake components)

- Experience working with Big Data platforms, including Dataproc, Data Bricks etc

- Experience working with distributed technology tools including Spark, Presto, Databricks, Airflow

- Working knowledge of Data warehousing, Data modeling

- Experience working in Agile and Scrum development process

- Bachelor's degree in Computer Science, Information Systems, Business, or other relevant subject area