TVARIT GmbH - Senior Data Engineer - ETL/Python (4-6 yrs)

Job Description :

We are seeking a skilled and experienced Senior Data Engineer to join our team.

The ideal candidate will play a crucial role in designing, building, and maintaining robust data pipelines and architectures to support advanced analytics and decision-making processes in the manufacturing industry, with a strong focus on steel/metal production.

Key Responsibilities :

- Design, build, and manage scalable ETL/ELT pipelines to process structured and unstructured data from sources such as IoT devices, MES, SCADA, LIMS, and ERP systems.

- Develop scalable, high-performance data architectures for analytics and data science initiatives.

- Develop and optimize real-time and batch data processing workflows to support critical business needs, ensuring low latency and high throughput.

- Implement real-time data streaming pipelines using tools like Kafka, Azure Event Hub, or similar technologies, ensuring low latency.

- Architect and maintain efficient data storage solutions, including data lakes and data warehouses, and implement modern methodologies like medallion architecture to ensure scalability and efficient data processing.

- Work with cloud platforms (preferably Azure) to design and deploy robust data solutions.

- Monitor and optimize data workflows, implementing alerting mechanisms to ensure quality, reliability, and stability.

- Optimize data pipelines to improve performance and support both batch and real-time processing.

- Automate data workflows using DevOps practices while ensuring security and compliance with industry standards.

- Collaborate with data scientists, analysts, process engineers, and stakeholders to deliver data solutions aligned with manufacturing KPIs.

- Ensure high data availability, reliability, and proactive resolution of quality issues.

- Implement best practices for data governance and regulatory compliance.

Desired Skills and Qualifications :

- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.

- 4+ years of experience in data engineering, preferably in the manufacturing industry.

- Hands-on experience with cloud platforms such as AWS, Azure, or Google Cloud (preferably Azure).

- Certifications are an added advantage.

- Strong experience with data engineering tools and frameworks such as Apache, Spark, Hadoop, Kafka, or Flink.

- Proficiency in programming languages like Python, Java, PySpark, or Scala.

- Expertise in relational databases (e.g. , SQL Server, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).

- Knowledge of data warehouse and lakehouse methodologies, including medallion architecture.

- Proven expertise in designing and implementing ETL pipelines using tools such as Azure Data Factory (ADF) or Databricks to orchestrate workflows.

- Familiarity with manufacturing systems and protocols such as SCADA, MES, and OPC-UA.

- Expertise in data pre-processing techniques, including data cleaning, normalization, deduplication, transformation, encoding, scaling, and type conversion.

- Familiarity with predictive analytics, AI/ML applications, and IoT in manufacturing is a plus.

- Strong analytical and problem-solving skills with attention to detail.

- Excellent communication and collaboration skills to work effectively with technical and non-technical stakeholders.

- Excellent communication and collaboration skills, with a proven ability to work effectively as a team player.

- Comfortable working in a dynamic, fast-paced startup environment, adapting quickly to changing priorities and responsibilities.