Youngsoft India - Cloud Data Engineer - Azure (5-8 yrs)

Cloud Data Engineer - Azure

Job Description :

Responsibilities :

- Design, develop, and implement complex data pipelines using Azure Databricks, PySpark, and other relevant Azure services (e.g., Data Factory, Data Lake Storage).

- Optimize data pipelines for performance, cost-effectiveness, and scalability.

- Monitor and maintain data pipelines, ensuring data quality, accuracy, and timely delivery.

- Troubleshoot and resolve data pipeline issues proactively.

- Leverage Azure cloud services effectively, including compute, storage, and networking services.

- Implement security best practices and data governance policies within the Azure environment.

- Stay updated on the latest Azure cloud technologies and best practices.

- Advocate for and implement data engineering best practices, including data modeling, data quality checks, and data lineage.

- Contribute to the development and improvement of data engineering standards and processes.

- Collaborate effectively with data scientists, analysts, business stakeholders, and other engineers.

- Mentor junior engineers and provide guidance on data engineering best practices.

- Actively participate in team discussions and contribute to the overall success of data engineering projects.

- Document data pipelines, architectures, and processes clearly and concisely.

- Create and maintain technical documentation for internal and external audiences.

Requirements :

- 5+ years of hands-on experience in software development with a strong focus on data engineering.

- Proven expertise in designing, developing, and implementing data pipelines using Azure Databricks and PySpark.

- Solid understanding of SQL and experience with relational and NoSQL databases.

- Experience with data modeling, data warehousing concepts, and data quality best practices.

- Strong analytical and problem-solving skills with the ability to analyze complex data problems and develop effective solutions.

- Experience with Agile development methodologies and working in a fast-paced environment.

- Excellent communication and collaboration skills with the ability to effectively communicate technical concepts to both technical and non-technical audiences.

- A strong work ethic with a focus on delivering high-quality results.

Preferred Skills :

- Experience with Java, C#, Kafka, Node.js, Azure Data Factory, and big data toolsets.

- Experience with Linux and shell scripting.

- Experience with data visualization and business intelligence tools.

- Experience with cloud security best practices and data governance frameworks.

- Experience with DevOps practices and tools (e.g., CI/CD pipelines).