Data Engineer

Job Title: Data Engineer

Experience: 4+Years

Locations: Hyderabad

Data Engineer

We are looking for a Data Engineer with strong expertise in Databricks, PySpark, SQL, and Azure, who can take a primary role in building and managing Databricks-based data solutions. The ideal candidate should have experience designing and optimizing Spark jobs, managing Delta Lake implementations, configuring and maintaining Databricks clusters, and developing scalable ETL pipelines. A deep understanding of Databricks Workspace, advanced features such as Databricks SQL and structured streaming, and integration with Azure services like ADLS, Azure Data Factory, and Azure DevOps is essential. Familiarity with Databricks performance tuning, cost optimization strategies, and implementing data security measures such as RBAC and encryption is highly desirable.

Key Responsibilities:

• Design, develop, and optimize data pipelines and workflows within the Databricks environment.
• Configure and maintain Databricks clusters to support various data processing and analytics workloads.
• Implement and manage Delta Lake for robust data storage, ensuring ACID compliance and data versioning.
• Build, deploy, and maintain scalable ETL/ELT pipelines using PySpark and SQL.
• Transform, cleanse, and process large datasets to prepare them for analytics and machine learning models.
• Automate data ingestion processes from various structured and unstructured data sources.
• Integrate Databricks with Azure services such as Azure Data Lake Storage (ADLS), Azure Synapse, Azure Data Factory, and Azure DevOps.
• Ensure efficient data movement and management across cloud storage and processing layers.
• Implement Azure-native security features, including Managed Identity and role-based access control (RBAC).
• Monitor and tune the performance of Spark jobs and Databricks workflows to minimize runtime and cost.
• Optimize cluster configurations for diverse workloads and manage auto-scaling.
• Implement and enforce data governance policies, including data masking, encryption, and lineage tracking.
• Ensure compliance with industry standards like GDPR, HIPAA, or other relevant regulations.
• Collaborate with data scientists, analysts, and stakeholders to understand business requirements and translate them into technical solutions.
• Document data workflows, architecture designs, and best practices to ensure knowledge sharing and continuity.
• Troubleshoot and resolve issues in Databricks workflows, clusters, and data pipelines.
• Proactively monitor the health of Databricks jobs and implement alerts for failures or bottlenecks.
• Stay updated on new features and improvements within Databricks and Azure ecosystems.
• Innovate and implement best practices for data engineering to improve efficiency and reliability.

Qualifications:

• Bachelors or Masters degree in Computer Science, Information Technology, Data Science, or a related field.
• Proven experience in developing and managing solutions within the Databricks platform.
• Strong understanding of Delta Lake, Databricks SQL, and structured streaming.
• Hands-on experience with Databricks cluster configuration and optimization.
• Proficiency in PySpark and Python for data processing and workflow automation.
• Advanced SQL skills for querying and transforming data.
• Solid experience working with Azure services such as Azure Data Lake Storage (ADLS), Azure Synapse Analytics, Azure Data Factory, and Azure DevOps.
• Understanding of cloud infrastructure concepts such as VPC, IAM, and security best practices.
• Expertise in building and maintaining ETL pipelines for large-scale data processing.
• Knowledge of data modeling, schema design, and data lake best practices.
• Experience with performance tuning for Apache Spark and Databricks workloads.
• Familiarity with strategies for cost optimization in a cloud-based environment.
• Minimum 35 years of experience in data engineering, with at least 2 years focused on Databricks and Apache Spark.
• Demonstrated experience working with big data technologies and managing large datasets.

Preferred Certifications

• Databricks Certified Data Engineer Associate or Professional.
• Microsoft Azure certifications such as Azure Data Engineer Associate.