CirrusLabs - Databricks Platform Engineer - ETL (6-10 yrs)

We are hiring a talented Databricks Platform Engineer to join our team.

If you're excited to be part of a winning team, CirrusLabs ( http://www.cirruslabs.io ) is a great place to grow your career.

Experience : 6- 10 years.

Location : Bengaluru and Hyderabad.

Experience in API and SDT tools is mandatory.

Job Overview :

We are looking for a talented Databricks Platform Engineer to join our data engineering team.

The ideal candidate will have extensive experience with Databricks, Big Data technologies, and data processing frameworks.

You will be responsible for designing, implementing, and managing data pipelines and analytics solutions on the Databricks platform, ensuring efficient data processing and integration.

Key Responsibilities :

- Databricks Implementation : Design and implement scalable data solutions using Databricks, ensuring optimal performance and cost efficiency.

- Data Pipeline Development : Build and maintain robust ETL/ELT data pipelines to process large datasets from various sources, leveraging Databricks notebooks and Spark.

- Collaboration : Work closely with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.

- Big Data Technologies : Utilize various Big Data technologies and frameworks, including Apache Spark, Delta Lake, and others, to enhance data processing capabilities.

- Data Integration : Integrate data from multiple sources, ensuring data quality and consistency across the platform.

- Performance Optimization : Monitor and optimize the performance of data processing jobs, identifying and addressing bottlenecks and inefficiencies.

- Documentation : Create and maintain comprehensive documentation of data workflows, architecture, and best practices for the Databricks environment.

- Security and Compliance : Implement best practices for data security, privacy, and compliance within the Databricks platform.

Qualifications :

- Bachelor's or master's degree in computer science, Data Engineering, or a related field.

- 6+ years of experience in data engineering or data platform development, with a strong focus on Databricks.

- Proficiency in Apache Spark, with hands-on experience in Spark SQL and DataFrames.

- Familiarity with cloud platforms (e.g, AWS, Azure, Google Cloud) and their respective data services.

- Experience with data integration tools and technologies, such as Apache Kafka, Airflow, or similar.

Skills :

Technical Skills :

- Strong expertise in Databricks and Big Data technologies, with a solid understanding of data modeling and data warehousing concepts.

- Analytical Skills : Excellent problem-solving abilities, with a focus on developing efficient solutions to complex data challenges.

- Communication : Strong verbal and written communication skills, capable of conveying technical concepts clearly to both technical and non-technical stakeholders.

- Collaboration : Proven ability to work effectively in cross-functional teams and agile environments.

- Adaptability : Willingness to learn new technologies and adapt to evolving project requirements.