Data Architect - Azure Cloud (8-10 yrs)

Job Description : Data Architect

Company : TechWise Digital

Location : Bengaluru, Karnataka, India

Role : Data Architect

About TechWise Digital :

TechWise Digital is a dynamic and innovative technology company focused on delivering cutting-edge digital solutions. We are seeking a highly skilled and experienced Data Architect to join our team and lead the design and implementation of robust and scalable data architectures leveraging Azure and Databricks.

About the Job :

We are seeking a seasoned Data Architect with deep expertise in Azure and Databricks to design and implement scalable, high-performance data architectures capable of handling both real-time and batch processing workloads. This role is pivotal in building and maintaining ETL pipelines, seamlessly integrating data from diverse sources, and ensuring stringent data security and governance. The successful candidate will play a crucial role in enabling advanced analytics and Power BI applications.

Key Responsibilities :

- Design and implement scalable and efficient data lakehouse architectures, including bronze, silver, and gold layers, optimized for analytics and reporting.

- Develop comprehensive data models and schemas that support business requirements and ensure data integrity.

- Design and implement data integration strategies, including ETL/ELT processes, to ingest and transform data from various sources.

- Evaluate and select appropriate Azure data services and Databricks features to meet performance, scalability, and cost requirements.

- Design and implement real-time data pipelines using Azure Event Hub, Azure Stream Analytics, and Databricks Structured Streaming.

- Develop solutions for real-time data ingestion, processing, and analysis to support immediate business insights.

- Ensure low-latency data delivery and high availability of real-time data pipelines.

- Develop and optimize batch data processing pipelines using Databricks and Azure Data Factory.

- Design and implement efficient data transformations and aggregations for large datasets.

- Ensure data quality and consistency throughout the batch processing lifecycle.

- Implement robust data security measures, including access control, encryption, and data masking.

- Ensure compliance with data governance policies and regulations.

- Establish and maintain data lineage and metadata management processes.

- Implement and maintain data quality checks and data validation processes.

- Optimize data pipelines and queries for performance and scalability.

- Monitor and troubleshoot data infrastructure performance issues.

- Implement strategies for horizontal and vertical scaling of data systems.

- Collaborate with data engineers, data scientists, and business stakeholders to understand data requirements and deliver effective solutions.

- Communicate technical concepts and solutions clearly and effectively.

- Document data architectures, data flows, and data processing procedures.

- Design and implement data models that are optimized for Power BI reporting and analysis.

- Ensure that the data architecture supports the needs of Power BI dashboards and reports.

Required Skills and Experience :

- Minimum of 8+ years of experience in data architecture and data engineering.

- Extensive experience with Azure Data Services, including Azure Data Lake Storage, Azure Data Factory, Azure Event Hub, Azure Stream Analytics, Azure SQL Database, and Azure Synapse Analytics.

- Strong proficiency in Databricks, including Spark SQL, Delta Lake, and Databricks Structured Streaming.

- Expertise in SQL and Python programming languages.

- Proven experience in designing and implementing ETL/ELT pipelines.

- Hands-on experience with real-time data processing and stream analytics.

- Strong understanding of data modeling, data warehousing, and data lakehouse architectures.

- Excellent problem-solving and analytical skills.

- Strong understanding of data security and governance principles.

- Experience in data quality and data validation.

Preferred Qualifications :

- Experience with machine learning and AI applications.

- Experience with CI/CD pipelines for data engineering.

- Experience with cloud cost optimization strategies.

- Experience with other cloud platforms like AWS or GCP.

- Experience working with large scale data migrations.

- Relevant Azure and Databricks certifications.

Skills :

- ETL/ELT

- Data Architecture

- Azure Data Services

- Azure

- Databricks

- SQL

- Python

- Data Modeling

- Data Warehousing

- Data Lakehouse

- Real-time data processing

- Data Security

- Data Governance

- Power BI