Data Engineer (Spark)

About Clairvoyant Clairvoyant is a global technology consulting and services company founded in 2012, headquartered in Chandler, US, and has delivery centers across the globe. We help organizations maximize the value of data by providing data engineering, analytics, machine learning, and user experience consulting and development projects to multiple Fortune 500 clients. Clairvoyant clients rely on its deep vertical knowledge and best-in-class services to drive revenue growth, boost operational efficiencies, and manage risk and compliance. Our team of experts with direct industry experience in data engineering, analytics, machine learning, and user experience has your back at every step. Our Values: Passion, Continuous Learning, Adaptability, Teamwork, Customer Centricity, Reliability Position - Data Engineer (Spark) Location - Bangalore Experience - 5+ We are seeking an experienced and skilled Data Engineer (Spark) to join our dynamic team. In this role, you will leverage your expertise in Apache Spark to design, develop, and maintain high-performance data processing pipelines. As a key member of the data engineering team, you will be responsible for creating efficient and scalable solutions that enable the organization to process large datasets and drive data-driven insights. This is an exciting opportunity for a hands-on coder who thrives in a fast-paced environment and is passionate about working with big data technologies. While a basic understanding of cloud platforms is preferred, your core focus will be on mastering Spark and building robust data systems. Key Responsibilities: Design, develop, and maintain data processing pipelines using Apache Spark in a distributed environment. Implement high-performance Spark jobs to process large datasets and ensure efficient data flow. Collaborate with cross-functional teams to define data requirements and provide scalable data solutions. Write clean, efficient, and maintainable code for data transformation, aggregation, and analysis usingSpark SQL, PySpark, or Scala. Troubleshoot, optimize, and debug Spark jobs to ensure high performance and minimal downtime. Automate data workflows and integrate Spark jobs with other tools in the data pipeline. Continuously improve data quality, speed, and performance of data processing systems. Keep up with the latest developments in Spark and contribute to the improvement of our data systems. Must-Have Qualifications: 5 to 7 years of experience in data engineering with strong expertise in Apache Spark (Spark SQL, PySpark, Scala). Demonstrated experience writing complex, optimized Spark code for batch and stream processing. Strong problem-solving skills and the ability to develop solutions on the fly. Excellent knowledge of SQL and ability to perform complex data transformations. Hands-on experience in building, deploying, and managing data pipelines and ETL processes using Spark. Strong understanding of distributed computing principles and parallel processing techniques. Proficiency in coding and debugging Spark applications, optimizing job execution, and working with Spark clusters. Experience with version control tools (Git, etc.). Good-to-Have Skills: Cloud Knowledge (e.g., AWS, Azure, GCP) and familiarity with cloud-native tools for data processing. Knowledge of data warehousing concepts and technologies (e.g., Redshift, BigQuery, Snowflake). Familiarity with other big data tools like Hadoop, Hive, Kafka, and Airflow. Familiarity with containerization technologies like Docker and orchestration tools likeKubernetes. Experience with data lakes and NoSQL databases. Good understanding of data governance, security, and privacy protocols. Soft Skills: Ability to work independently and take ownership of complex tasks. Strong communication skills to collaborate effectively with both technical and non-technical stakeholders. Strong attention to detail and ability to troubleshoot and resolve issues swiftly. Ability to learn new technologies quickly and adapt to changing requirements. What we provide: Opportunities to develop and grow as an engineer. We are at the forefront of our industry, always expanding into new areas, and working with open-source and new technologies. A set of hardworking and dedicated peers. Growth and mentorship. We believe in growing engineers through ownership and leadership opportunities. We also believe mentors help both sides of the equation. Education: BE/B.Tech from a Tier 1 or 2 institute You are important to us, let s stay connected! Every individual comes with a different set of skills and qualities so even if you don t tick all the boxes for the role today, we urge you to apply as there might be a suitable/unique role for you tomorrow!