Scala & Databricks Developer - Big Data Solutions (4-5 yrs)

Job Description :

We are looking for an experienced Scala and Databricks Developer with strong expertise in building data solutions on AWS. The ideal candidate will have hands-on experience in developing data pipelines, optimizing big data solutions, and implementing best practices using Scala, Databricks, and AWS services. You will play a critical role in designing and delivering scalable, reliable, and high-performance data solutions to support our business objectives.

Key Responsibilities :

- Collaborate with data architects and analysts to design and implement data models and processing frameworks.

- Optimize performance of data workflows and address bottlenecks in Databricks and AWS environments.

- Integrate Databricks with AWS services like S3, Glue, Lambda, and Redshift.

- Implement data validation and quality checks to ensure accuracy and reliability of data pipelines.

- Work with stakeholders to understand business requirements and translate them into technical solutions.

- Monitor, troubleshoot, and resolve issues in production data pipelines.

- Document processes, configurations, and solutions for future reference.

- 4-5 years of experience in developing data solutions using Scala.

- Hands-on experience with Databricks for building and managing large-scale data workflows.

- Proficiency in AWS services, including S3, Glue, Lambda, Redshift, and IAM.

- Strong understanding of distributed computing and big data concepts like Spark, Hadoop, or similar frameworks.

- Experience with CI/CD pipelines and version control systems like Git.

- Knowledge of data security, governance, and compliance practices.

- Strong problem-solving skills and the ability to work in a fast-paced environment.

Preferred Qualifications :

- Familiarity with AWS tools such as Athena, EMR, or Step Functions.

- Knowledge of data lakes and lakehouse architectures.

- Experience in Python for scripting and auxiliary tasks.

- Understanding of monitoring and logging tools like CloudWatch, Datadog, or Prometheus.

- Exposure to Agile methodologies and tools like JIRA.