Thoucentric - Azure Data Engineer - SQL/ETL (3-5 yrs)

Job Description :

We are looking for a highly skilled Azure Data Engineer to join our team. The ideal candidate will have strong expertise in implementing data solutions on Azure using technologies like Azure Data Lake Storage Gen2 (ADLS2), Azure Data Factory (ADF), Azure Databricks, and Azure SQL/other RDBMS. The role involves building robust and scalable data pipelines, transforming raw data into actionable insights, and supporting advanced analytics initiatives.

Key Responsibilities :

Design and Develop Data Solutions :

- Build and optimize data pipelines using Azure Data Factory and Azure Databricks.

- Implement solutions for structured and unstructured data storage using ADLS Gen2.

- Integrate data from various sources into Azure SQL or other relational databases.

Data Transformation and Processing :

- Design ETL/ELT workflows for data ingestion, cleansing, transformation, and aggregation.

- Process and analyze large datasets using Databricks (PySpark/Scala/Spark SQL).

Performance Optimization :

- Optimize data storage and retrieval in ADLS2 and relational databases to ensure high performance.

- Implement partitioning, indexing, and caching strategies for efficient data operations.

Collaboration and Support :

- Collaborate with data scientists, analysts, and business stakeholders to gather requirements and deliver solutions.

- Ensure data security, governance, and compliance with organizational standards.

Monitoring and Maintenance :

- Monitor data pipelines for reliability and scalability.

- Troubleshoot and resolve issues related to data processing and integration.

Requirements :

Required Skills and Qualifications :

Technical Expertise :

- Strong experience with Azure Data Lake Storage Gen2 (ADLS2) for data storage and management.

- Proficiency in building data pipelines with Azure Data Factory (ADF).

- Hands-on experience in Databricks (including PySpark, Scala, or Spark SQL).

- Advanced knowledge of Azure SQL or any other RDBMS (e.g., SQL Server, PostgreSQL, MySQL).

Programming Skills : Proficiency in Python, SQL, Pyspark for data engineering tasks.

Cloud Knowledge : Solid understanding of Azure ecosystem, including services like Azure Synapse Analytics, Azure Blob Storage, and Azure Key Vault (optional but preferred).

Problem-Solving and Analytical Skills : Ability to design and implement efficient solutions for large-scale data processing and analytics.

Soft Skills :

- Strong communication and collaboration skills.

- Ability to work in agile, fast-paced environments

Preferred Qualifications :

- Azure certifications (e.g., Microsoft Certified : Azure Data Engineer Associate).

- Familiarity with data governance frameworks and tools.

- Knowledge of distributed systems and big data frameworks like Apache Kafka (optional).

Experience Level : Minimum 35 years in data engineering or related roles.

Benefits :

- Opportunity to define your career path and not as enforced by a manager

- A great consulting environment with a chance to work with Fortune 500 companies and startups alike.

- A dynamic but relaxed and supportive working environment that encourages personal development.

- Be part of One Extended Family. We bond beyond work - sports, get-togethers, common interests etc.

- Work in a very enriching environment with Open Culture, Flat Organization and Excellent Peer Group.

- Be part of the exciting Growth Story of Thoucentric!