Azure Data Engineer - ETL Tools (5-10 yrs)

About RiDiK:

RiDiK, a subsidiary of CLPS Incorporation (Nasdaq: CLPS), is a leading technology solutions provider, delivering innovative and high-quality services to global clients. We are seeking a highly skilled and experienced Advanced Azure Data Engineer to join our team in Bengaluru or Hyderabad.

Position Overview:

As an Advanced Azure Data Engineer at RiDiK, you will be responsible for designing, building, and maintaining robust data pipelines and data solutions within the Azure ecosystem. You will leverage your expertise in Azure Databricks, PySpark, Python, SQL, and data modeling to deliver advanced data engineering solutions. This role requires a deep understanding of data engineering principles, cloud technologies, and a passion for building scalable and efficient data systems.

Responsibilities:

Azure Databricks Development:

- Design, develop, and optimize data pipelines and transformations using Azure Databricks and PySpark.

- Build and maintain scalable data processing solutions using Apache Spark.

- Implement data warehousing and data lake solutions using Databricks.

PySpark and Python Development:

- Develop and maintain complex data processing scripts and applications using PySpark and Python.

- Automate data ingestion, transformation, and loading processes.

- Optimize code for performance and scalability.

SQL Development and Data Modeling:

- Design and implement database schemas and data models.

- Develop and optimize SQL queries for data extraction, transformation, and analysis.

- Ensure data integrity and quality.

ETL/ELT Process Development :

- Design and implement efficient ETL/ELT processes for data migration and integration.

- Develop data integration solutions using various data sources and targets.

- Implement data quality checks and validation processes.

Data Integration and Workflow Orchestration :

- Design and implement data integration solutions using Azure Data Factory or similar tools.

- Orchestrate data pipelines and workflows using relevant tools and technologies.

Real-Time Data Processing :

- Design and implement real-time data processing solutions using Azure Stream Analytics or similar technologies.

- Process and analyze streaming data for real-time insights.

Azure Cloud Services :

- Utilize various Azure cloud services for data engineering tasks, including Azure Storage, Azure SQL Database, and Azure Data Lake Storage.

- Manage and optimize data storage and processing on Azure.

Data Security and Compliance :

- Implement data security measures and ensure compliance with relevant regulations.

- Manage data access and authorization controls.

Collaboration and Communication :

- Work closely with data scientists, analysts, and other engineers.

- Communicate effectively with stakeholders and provide regular updates on project progress.

Required Skills :

Education : Master's degree in Computer Science, Information Technology, or a related field.

Experience : 5 to 10 years of experience as an Azure Data Engineer or similar role.

Azure Databricks :

- Strong expertise in Azure Databricks and Apache Spark.

PySpark and Python :

- Advanced proficiency in PySpark and Python for data processing and automation.