Softpath Technologies - Senior Software Engineer - Azure Databricks (4-6 yrs)

Job Title : Senior Software Engineer - Data Bricks

Location : Pune, India

Experience : 4 to 6 years

Notice Period : Immediate to 10 days

Interview Process : Assessment followed by 2 Virtual Interviews

Job Description :

We are seeking a highly motivated and skilled Senior Software Engineer specializing in Data Bricks to join our dynamic team. This position offers the opportunity to work with cutting-edge technologies and drive impactful data engineering and analytics initiatives.

You will play a critical role in implementing and optimizing data pipelines, processing large-scale datasets, and delivering high-quality solutions to business problems using Data Bricks and other related technologies.

As a Senior Software Engineer in our team, you will be responsible for the following :

Key Responsibilities :

Data Engineering & Pipeline Development :

- Design, develop, and maintain robust, scalable, and efficient data pipelines and workflows using Data Bricks.

- Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality data solutions.

- Implement ETL processes to handle large-scale datasets and ensure data quality and integrity.

- Utilize Spark, PySpark, and SQL to process and analyze large volumes of data.

Optimization & Performance Tuning :

- Optimize existing data pipelines to ensure maximum performance, scalability, and cost-effectiveness.

- Monitor and troubleshoot performance issues in data workflows and work towards timely resolutions.

- Apply best practices for data partitioning, caching, and query optimization.

Collaboration & Stakeholder Management :

- Work closely with cross-functional teams to understand data requirements and design solutions that align with business goals.

- Provide technical guidance and mentorship to junior engineers.

- Communicate complex technical concepts in a clear and concise manner to non-technical stakeholders.

Tool and Platform Management :

- Work with Data Bricks, Apache Spark, and other cloud technologies to build, scale, and maintain data-driven applications.

- Continuously evaluate and integrate new tools and technologies to improve the efficiency and scalability of our data infrastructure.

Automation & Continuous Integration :

- Develop automated testing frameworks and integrate them with the continuous integration and deployment pipeline.

- Ensure that data pipelines are automated, self-monitoring, and resilient to failures.

Documentation & Best Practices :

- Maintain comprehensive and accurate documentation for data pipelines, workflows, and other solutions.

- Promote and follow industry best practices for coding, version control, and data governance.

Required Skills & Qualifications :

- Experience: 4 to 6 years of experience in software engineering with a focus on data engineering and data pipelines.

Technical Expertise :

- Strong hands-on experience with Data Bricks and Apache Spark (PySpark/Scala).

- Solid proficiency in SQL for querying, data transformation, and reporting.

- Familiarity with cloud platforms such as AWS, Azure, or Google Cloud, and related services like Data Lake, Data Warehousing, and Serverless Architectures.

- Knowledge of ETL frameworks, data integration, and data quality processes.

- Experience with version control systems such as Git and CI/CD pipelines.

- Programming Languages : Proficiency in Python, with experience in PySpark preferred. Familiarity with Scala is a plus.

- Problem-Solving : Strong analytical and problem-solving skills with the ability to troubleshoot complex data pipeline and performance issues.

- Collaboration & Communication : Excellent verbal and written communication skills with the ability to work well in cross-functional teams.

Education : A degree in Computer Science, Engineering, or a related field is preferred.

Preferred Qualifications :

- Experience working with Data Lakes, Data Warehouses, and Big Data technologies.

- Experience with Kafka, Hive, Hadoop, or similar big data technologies.

- Familiarity with machine learning frameworks or data science workflows is a plus.

- Knowledge of data governance and security best practices in data handling.

Benefits :

- Competitive salary and performance-based bonuses.

- Comprehensive health and wellness benefits.

- Opportunities for career growth and skill development.

- A collaborative and inclusive work environment.

Interview Process :

- Assessment : Initial assessment that will evaluate your problem-solving, programming, and data engineering skills.

- Virtual Interview 1: Technical interview to assess your proficiency with Data Bricks, Spark, SQL, and cloud platforms.

- Virtual Interview 2 : A final round to assess your overall fitment within the team and evaluate your communication skills, collaboration abilities, and past experiences.