Data Engineer - Sr. Consultant-Java/SQL/Big Data

The Universal Data Catalog (UDC) / Visa Data Catalog (VDC) is a comprehensive metadata management platform designed to provide a centralized repository for all data-related information across the organization. The platform enables efficient data discovery, governance, and utilization by offering detailed metadata definitions for tables, columns, and other data assets. It supports various business units by improving data accessibility, enhancing data quality, and facilitating compliance with data governance policies. Our implementation leverages the open-source Datahub project, ensuring a robust, flexible, and scalable solution.

Key Responsibilities

This is a hybrid position. Expectation of days in office will be confirmed by your Hiring Manager.

Basic Qualifications
8+ years of relevant work experience with a Bachelor s Degree or at least 5 years of experience with an Advanced Degree (e.g. Masters, MBA, JD, MD) or 2 years of work experience with a PhD, OR 11+ years of relevant work experience.

Preferred Qualifications
9 or more years of relevant work experience with a Bachelor Degree or 7 or more relevant years of experience with an Advanced Degree (e.g. Masters, MBA, JD, MD) or 3 or more years of experience with a PhD
10 yrs+ of experience in architecture design and development of large-scale data management platforms and data application with simple solutions
Must have extensive hands-on coding and designing skills on
oJava/Python or Node.js/Javascript for backend
oMVC (model-view-controller) for end-to-end development
oSQL/NoSQL technology. Familiar with Databases like Oracle, DB2, SQL Server, etc.
oWeb Services (REST/ SOAP/gRPC)
oReact/Angular for front-end (UI front-end nice to have)
Expertise in design and management of complex data structures and data processes
Expertise in efficiently leveraging the power of distributed big data systems, including but not limited to Hadoop Hive, Spark, Kafka streaming, etc.
Deep knowledge and hands on experience on big data and cloud computing technologies.
Strong service architecture and development experience with high performance and scalability
Technical background in data with deep understanding of issues in multiple areas such as data acquisition, ingestion and processing, data management, distributed processing, and high availability is required.
Strong on driving for results and self-motivated, strong learning mindset, with good understanding of related advanced/new technology. Keep up with the technology development in the related areas in the industry, which could be leveraged to enhance current architectures and build durable new ones.
Bachelor s degree in computer science or related technical discipline required. Advanced degree is a plus.
Strong leadership and team player.
Strong skills on mentoring/growing junior people
Payment industry experience is a plus.

What will also help:
Experience with ETL/ELT tools / applications
Experience with Apache NiFi and Apache Spark for processing large data sets
Experience on Elastic Search
Knowledge on Data Catalog tools
Experience in building Data Pipeline development tools
Experience with Data Governance and Data Quality tools