Databricks Developer - Spark Streaming :: Freelancer :: Remote

Role & responsibilities

• Over 10 years of IT experience in the big data domain, specializing in Hadoop, data lakes, and data engineering using Python and Spark programming languages.
• At least 3 years of experience with Databricks workspace, including working with Databricks notebooks, job clusters, Delta Lake, Databricks Lakehouse, and Unity Catalog.
• A minimum of 5 years of experience in PySpark and Python development.
• Over 6 years of experience in designing and implementing data pipelines.
• Proficient in Spark Streaming and its integration into data processing workflows.
• Experienced in designing and developing data pipelines, as well as ETL/ELT jobs for data ingestion and processing in data lakes.
• Strong expertise in SQL Server, NoSQL, Spark SQL, data modeling, identity & access management, query optimization, and parallel processing.
• Proven problem-solving abilities along with excellent verbal and written communication skills.
• Familiar with processing streaming data using Kafka and Pub/Sub technologies.
• Experienced in Agile development, SCRUM methodologies, and Application Lifecycle Management (ALM).
• In-depth knowledge of data warehouse concepts.

Overall Experience: 10 Years (IT)

Relevant Experience: 4-5 Years

Timings: 3 PM- 12 PM (IST)

Work Type: Remote

Immediate joiners preferred.

Interested candidates can share their profiles to nitin.paul@msr-it.com