Cloudsufi - Principal Data Engineer - Python/Spark (15-20 yrs)

About Us :

CLOUDSUFI is a Data Science and Product Engineering organization building Products and Solutions for Technology and Enterprise industries. We firmly believe in the power of data to transform businesses and make better decisions. We combine unmatched experience in business processes with cutting edge infrastructure and cloud services. We partner with our customers to monetize their data and make enterprise data dance.

Our Values :

We are a passionate and empathetic team that prioritizes human values. Our purpose isto elevate the quality of lives for our family, customers, partners, and the community.

Diversity and Inclusivity :

CLOUDSUFI is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All qualified candidates receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, and national origin status. We provide equal opportunities in employment, advancement ,and all other areas of our workplace. Please explore more at https : //www.cloudsufi.com/

What We Are Looking For :

We are seeking a dynamic and highly skilled Principal Data Engineer who has extensive experience building enterprise-scale data platforms and lead these foundational efforts. This role demands someone who not only possesses a profound understanding of the data engineering landscape but is also at the forefront of their game. The ideal candidate will contribute significantly to platform development with a diverse skill set while also being very hands-on coding and actively shaping the future of our data ecosystem.

Job Location :

The location for this role will be Noida, India

Key Responsibilities :

- As a Principal Data Engineer, you will be responsible for ideation, architecture, design, and development of new enterprise data platform. You will collaborate with other senior architects to ensure seamless alignment within our overarching technology strategy.

- Architect and design core components with a microservices architecture, abstracting platform, and infrastructure intricacies.

- Create and maintain essential data platform SDKs and libraries, adhering to industry best practices.

- Design and develop connector frameworks and modern connectors to source data from disparate systems both on-prem and cloud.

- Design and optimize data storage, processing, and querying performance for large-scale datasets using industry best practices while keeping costs in check.

- Architect and design the best security patterns and practices.

- Design and develop data quality frameworks and processes to ensure the accuracy and reliability of data.

- Collaborate with data scientists, analysts, and cross-functional teams to design data models, database schemas and data storage solutions.

- Design and develop advanced analytics and machine learning capabilities on the data platform.

- Design and develop observability and data governance frameworks and practices.

- Stay up to date with the latest data engineering trends, technologies, and best practices.

- Drive the deployment and release cycles, ensuring a robust and scalable platform.

Key Qualifications/Experience :

- Education Background : BTech/ BE / BS / MS / MBA

- A minimum of 12+ years of proven experience in modern cloud data engineering, broader data landscape experience and exposure and solid software engineering experience.

- Prior experience architecting and building successful enterprise scale data platforms in a greenfield environment is a must.

- Hands-on experience with GCP ecosystem and data lakehouse architectures, with proficiency in building end to end data platforms and services.

- Strong understanding of data modeling, data architecture, and data governance principles.

- Proficiency in GCP native services : BigQuery, Cloud Functions, Dataform, Dataproc, Dataflow, Airflow, PubSub.

- Proficiency in programming languages : Python, Spark, SQL, Java

- Experience with Microservices architectures- Kubernetes, Docker, and Cloud Run

- Experience building Symantec layers.

- Proficiency in architecting and designing and development experience with batch and real time streaming infrastructure and workloads.

- Solid experience with architecting and implementing metadata management including data

catalogues, data lineage, data quality and data observability for big data workflows.

Good to have :

- Experience with Data Mesh architecture.

- Experience with DataOps principles and test automation.

- Experience with observability tooling : Grafana, Datadog.

- Experience building scalable IoT architectures.

Behavioral competencies :

- Must have worked with Europe/US based clients in onsite/offshore delivery model.

- Should have very good verbal and written communication, technical articulation, listening and presentation skills - 8/10.

- Should have proven analytical and problem-solving skills.

- Should have demonstrated effective task prioritization, time management and internal/external stakeholder management skills.

- Should be a quick learner, self-starter, go-getter and team player.

- Should have experience of working under stringent deadlines in a Matrix organization structure.

- Should have demonstrated appreciable Organizational Citizenship Behavior (OCB) in past organizations.