Databricks Architect / Sr. Data Engineer

Role & responsibilities

Good communications and stakeholder management skills.

7 +years of expertise in Big Data Analytics, i.e., Hadoop Ecosystem, Cloud Platform Azure, AWS, etc.

Design, develop, and maintain scalable and robust data pipelines to ingest, process, transform, and store large volumes of data from various sources.

Design and implement data models, schemas, and data architecture to support the needs of the business, ensuring data integrity, consistency, and performance.

Integrate data from disparate sources and systems, ensuring compatibility, consistency, and accuracy across the data ecosystem.

Implement data quality checks and validation processes to ensure data accuracy, completeness, and consistency throughout the data lifecycle.

Optimize data pipelines and processes for performance, scalability, and efficiency, addressing issues such as latency, throughput, and resource utilization.

Implement data orchestration framework and setting up CI/CD pipelines.

Develop consumers for stream processing platforms such as Kafka.

Implement Big Data solutions for Data Ingestion, Transformation and Analytics, including ACID transactions and Change Data Capture (CDC) scenarios.

Tools stack:

Azure - Data Lake, Synapse, Databricks, Data Factory (Must have)

Spark, Python, Hive, HDFS, Sqoop, Kafka

AWS - EMR, EC2, S3, Redshift, RDS

Postgres, MySQL, Teradata, Oracle, DB2, HBase, MS SQL

GitHub, Bit Bucket, Control-M, JIRA, Jenkins, Confluence

Preferred candidate profile

Cloud Certification

Perks and benefits

• Competitive Salary with Performance-Based Bonuses
• Health Insurance for you and your family
• Flexible Work Hours and Remote Work Options
• Opportunities for Upskilling and Certifications in Big Data and Cloud Technologies
• Access to Cutting-Edge Technologies and Challenging Projects
• Collaborative Work Environment with a focus on career growth