Data Architect - Python/Spark

Responsibilities for the job :Key Responsibilities :- Architectural Leadership: Design and implement scalable data architectures using Databricks, ensuring they meet the specific needs of financial services such as risk management, compliance, and customer analytics.- Conduct assessment of existing data architectures, identify areas of improvement, and recommend remediation plans to optimise data management process and systems.- Develop data migration strategies and oversee the execution of data migration activities, ensuring data integrity and minimal disruption to client operations.- Data Pipeline Development: Build and optimize data ingestion and processing pipelines to facilitate real-time analytics and reporting.- Collaboration with Stakeholders: Work closely with business units to understand their data requirements and translate them into technical specifications that align with business goals.- Compliance and Governance: Ensure data solutions adhere to financial regulations and best practices in data governance, including data security and privacy standards.- Performance Optimization: Monitor and tune the performance of data systems to ensure efficiency and reliability, addressing any bottlenecks or issues that arise.Mentorship and Training: Provide guidance to junior data engineers and analysts, sharing best practices and promoting a culture of continuous improvementOther Responsibilities (If Any):- Availability during month-end Deck generation, may be sometime during week-end/holidays.Eligibility Criteria for the JobEducation : B.E/B. Tech in any specialization, BCA, M.Tech in any specialization, MCAAge : No BarWork Experience :

- Proven experience, 7 years as a Data Architect, Data Solutions Architect, or similar role in a consulting environment.- Strong hands-on experience on various cloud services like ADF/Lambda, ADLS/S3, Security, Monitoring, Governance & Compliance.- Hands-on Experience to design platform and build- Databricks- based solution on any cloud platform as well as design and build solution powered by- DBT- models and integrate with databricks.- Must have good: - End-to-End design solution on cloud platform.- Knowledge of Data Engineering concept and related services of cloud.- Experience in Python and Spark and setting up development best practices.- Knowledge of docker and Kubernetes.- Experience with claims-based authentication (SAML/OAuth/OIDC), MFA,- RBAC, SSO etc.- Knowledge of cloud security controls including tenant isolation, encryption at rest, encryption in transit, key management, vulnerability assessments, application firewalls, SIEM, etc.Primary Skill :

- Effectively communicate complex technical concepts and recommendations to non-technical stakeholders, including C-level executives and business leaders, to facilitate decision-making and secure buy-in.- The ability to lead the design, implementation, and optimization of our Databricks platform.- Work closely with our data engineering team to ensure that our Databricks platform is optimized for performance, scalability, and reliability.- Develop and maintain a comprehensive understanding of our data pipeline and data architecture.- Develop and maintain documentation for our Databricks platform, including architecture diagrams, deployment guides, and operational procedures.- Experience in a Platform architecture role using service and hosting solutions such as private/public cloud IaaS, PaaS, and SaaS platforms.- Provide guidance and support to our data engineering team on Databricks-related issues.- Maintains close awareness of new and emerging technologies and their potential application for service offerings and products.Technical Skills :

- Must have orchestrated at least 3 projects using any of the cloud platforms (GCP, Azure, AWS etc.) is a must. - Must have orchestrated at least 2 projects using Databricks over AWS/Azure

- Mush have worked on any cloud PaaS/SaaS database/DWH such as AWS redshift/ Big Query/ Snowflake

- Expertise in programming languages like Python or Pyspark or Scala or SQL/PLSQL. - Lead the design and implementation of data artefacts, including data modelling, data integration, data storage, and data governance based on industry best practices and client-specific requirements.- Hands-on experience on data modelling techniques to create Enterprise Data Models

- Implementing data management concepts (Metadata Management, Data Quality, Data Testing)

- Python/Spark Hands -on Exp from data engineering perspective is a must

- Experience in at least one of the major ETL tools (Talend + TAC, SSIS, Informatica) will be added advantage

- Experience with Github and Jenkins CI/CD or similar Devops tools. - Experience with ETL tooling, migrating ETL code from one technology to another will be a benefit

- Hands-on experience in Visualization/Dashboard tools (PowerBI, Tableau, Qliksense)