Palantir Engineer

Job description

Role: Developer Azure Databricks

Position Summary:

At Arthya Tech we are building India Competency center - ICC for our client - an US based producer of dairy products.

As a Tech Lead / Senior Developer, you will be responsible for technical leadership of Agile Team(s) with our Business Analytics & Insights (BAI) team. You will work closely with cross-functional teams to deliver high-quality solutions in domains such as Supply Chain, Finance, Operations, Customer Experience, HR, Risk Management, and Global IT.

Responsibilities:

• Develop a comprehensive technical plan for the migration, including data ingestion, transformation, storage, and access control in Azure's Data Factory and data lake.
• Design and implement scalable and efficient data pipelines to ensure smooth data movement from multiple sources using Azure Databricks
• Developing scalable and re-usable frameworks for ingesting of data sets
• Ensure data quality and integrity throughout the entire data pipeline, implementing robust data validation and cleansing mechanisms.
• Working with event based / streaming technologies to ingest and process data.
• Provide technical guidance and support to the team, resolving any technical challenges or issues that may arise during the migration and post-migration phases.
• Stay up to date with the latest advancements in cloud computing, data engineering, and analytics technologies, and recommend best practices and industry standards for implementing the data lake solution.

Requirements:

• 3 to 6 years of IT experience
• Min. 3 years of Experience working with Azure Databricks & Palantir
• Experience in Data Modelling, Source System Analysis,
• Familiarity with PySpark
• Mastery of SQL
• Knowledge of components: Azure Data Factory, Azure Data Lake, Azure SQL DW, Azure SQL • Experience with Python programming language used for data Engineering purpose.
• Ability to conduct data profiling, cataloging, and mapping for technical design and construction of technical data flows.
• Experience in data visualization/exploration tools

Will be considered as an advantage but are not required:

• Microsoft Certified: Azure Data Engineer Associate
• Experience preparing data for Data Science and Machine Learning
• Knowledge of Jupyter Notebooks or Databricks Notebooks for Python development

Personal Attributes

• Providing Technical leadership and guidance to Teams
• Strong problem-solving skills and the ability to translate business requirements into actionable data science solutions.
• Excellent communication skills, with the ability to effectively convey complex ideas to technical and non-technical stakeholders.
• Strong team player with excellent interpersonal and collaboration skills.
• Ability to manage multiple projects simultaneously and deliver high-quality results within specified timelines.