UST - Azure Data Engineer - Scala/Python (1-3 yrs)

Job Title : Azure Data Engineer.

Experience : 1-3 Years.

Location : Bengaluru.

Job Description :

We are seeking an Azure Data Engineer with strong communication skills and a passion for learning. The ideal candidate should have hands-on experience with Azure Data Lake, Azure Data Factory, Azure Databricks, Apache Spark, and Scala/Python programming..

Key Responsibilities :

- Design, develop, and maintain data pipelines using Azure Data Factory to ingest, transform, and load data from various sources.

- Build scalable and efficient data processing workflows using Azure Databricks and Apache Spark.

- Implement data quality checks and validation processes to ensure data accuracy and reliability.

- Manage and maintain Azure Data Lake Storage, ensuring data security, accessibility, and performance.

- Implement data partitioning and optimization strategies for efficient data retrieval.

- Collaborate with data scientists and analysts to provide access to data for analysis and reporting.

- Develop data transformation logic using Scala or Python within Azure Databricks.

- Implement data cleansing and standardization processes to ensure data consistency.

- Optimize data processing jobs for performance and resource utilization.

- Integrate Azure Data Lake, Azure Data Factory, and Azure Databricks with other Azure services, such as Azure SQL Database, Azure Synapse Analytics, and Azure Event Hubs.

- Develop and maintain data connectors and APIs for data integration.

- Write clean, efficient, and well-documented code in Scala or Python.

- Utilize version control systems (Git) for code management and collaboration.

- Participate in code reviews and provide constructive feedback.

- Troubleshoot and resolve data pipeline and data processing issues.

- Monitor system performance and identify areas for improvement.

- Provide technical support to data scientists and analysts.

- Stay up-to-date on the latest Azure data services and technologies.

- Participate in training and development activities to enhance technical skills.

- Contribute to knowledge sharing and documentation efforts.

Technical Skills & Qualifications :

- 1-3 years of experience in data engineering or related fields.

- Hands-on experience with Azure Data Lake Storage.

- Hands-on experience with Azure Data Factory.

- Hands-on experience with Azure Databricks and Apache Spark.

- Proficiency in Scala or Python programming.

- Basic understanding of data warehousing concepts.

- Strong communication and interpersonal skills.

- Passion for learning and a proactive attitude.

- Ability to work effectively in a team environment.

Preferred Qualifications :

- Bachelor's or Master's degree in Computer Science, Data Science, or a related field.

- Experience with SQL and database management.

- Experience with cloud platforms (AWS, GCP).

- Knowledge of data modeling and data architecture.

- Azure certifications (e.g., Azure Data Fundamentals, Azure Data Engineer Associate).