Principal Data Engineer - Python/SQL (12-19 yrs)

Base Location : Noida

Work Mode : Remote

Mandatory Skill :

DATA Engineering GCP, Data platform , Lakeset up,Data Modelling, Big Query, Python , cloud dataflow,End to end data services & Data platform in GCP

Job Description :

As a Principal Data Engineer, you will be responsible for ideation, architecture, design and development of new enterprise data platform.

- You will collaborate with other cloud and security architects to ensure seamless alignment within our overarching technology strategy.

- Architect and design core components with a microservices architecture, abstracting platform, and infrastructure intricacies.

- Create and maintain essential data platform SDKs and libraries, adhering to industry best practices.

- Design and develop connector frameworks and modern connectors to source data from disparate systems both on-prem and cloud.

- Design and optimize data storage, processing, and querying performance for large-scale datasets using industry best practices while keeping costs in check.

- Architect and design the best security patterns and practices

- Design and develop data quality frameworks and processes to ensure the accuracy and reliability of data.

- Collaborate with data scientists, analysts, and cross functional teams to design data models, database schemas and data storage solutions.

- Design and develop advanced analytics and machine learning capabilities on the data platform.

'- Design and develop observability and data governance frameworks and practices.

- Stay up to date with the latest data engineering trends, technologies, and best practices.

- Drive the deployment and release cycles, ensuring a robust and scalable platform.

Requirements :

- 12+ years of proven experience in modern cloud data engineering, broader data landscape experience and exposure and solid software engineering experience.

- Prior experience architecting and building successful enterprise scale data platforms in a green field environment is a must.

- Proficiency in building end to end data platforms and data services in GCP is a must.

- Proficiency in tools and technologies : BigQuery, Cloud Functions, Cloud Run, Dataform, Dataflow, Dataproc, SQL, Python, Airflow, PubSub.

- Experience with Microservices architectures - Kubernetes, Docker and Cloud Run

- Experience building Symantec layers.

- Proficiency in architecting and designing and development experience with batch and real time streaming infrastructure and workloads.

- Solid experience with architecting and implementing metadata management including data catalogues, data lineage, data quality and data observability for big data workflows.

- Hands-on experience with GCP ecosystem and data lakehouse architectures.

- Strong understanding of data modeling, data architecture, and data governance principles.

- Excellent experience with DataOps principles and test automation.

- Excellent experience with observability tooling : Grafana, Datadog.

Requirements :

Nice to have :

- Experience with Data Mesh architecture.

- Experience building Semantic layers for data platforms.

- Experience building scalable IoT architectures Non-Technical/ Behavioural competencies required :

- Must have worked with US/Europe based clients in onsite/offshore delivery model

- Should have very good verbal and written communication, technical articulation, listening and presentation skills

- Should have proven analytical and problem solving skills

- Should have demonstrated effective task prioritization, time management and internal/external stakeholder management skills

- Should be a quick learner, self starter, go-getter and team player

- Should have experience of working under stringent deadlines in a Matrix organization structure

- Should have demonstrated appreciable Organizational Citizenship Behavior (OCB) in past organization

Qualification :

- Bachelor of Science in Computer Science or equivalent technical training and professional work experience.