Lead Data Engineer- Java/Big Data/SQL/Architect

The Universal Data Catalog (UDC) / Visa Data Catalog (VDC) is a comprehensive metadata management platform designed to provide a centralized repository for all data-related information across the organization. The platform enables efficient data discovery, governance, and utilization by offering detailed metadata definitions for tables, columns, and other data assets. It supports various business units by improving data accessibility, enhancing data quality, and facilitating compliance with data governance policies. Our implementation leverages the open-source Datahub project, ensuring a robust, flexible, and scalable solution.

Key Responsibilities

This is a hybrid position. Expectation of days in office will be confirmed by your hiring manager.

Basic Qualification
12 to 15 yrs+ of experience in architecture design and development of large-
scale data management platforms and data application with simple solutions
Must have extensive hands-on coding and designing skills on
- Java/Python for backend
- MVC (model-view-controller) for end-to-end development
- SQL/NoSQL technology. Familiar with Databases like Oracle, DB2, SQL Server, etc.
- Web Services (REST/ SOAP/gRPC)
- React/Angular for front-end (UI front-end nice to have)
Expertise in design and management of complex data structures and data
processes
Expertise in efficiently leveraging the power of distributed big data systems,
including but not limited to Hadoop Hive, Spark, Kafka streaming, etc.
Deep knowledge and hands on experience on big data and cloud computing
technologies.
Strong service architecture and development experience with high
performance and scalability
Technical background in data with deep understanding of issues in multiple
areas such as data acquisition, ingestion and processing, data management,
distributed processing, and high availability is required.
Strong on driving for results and self-motivated, strong learning mindset, with
good understanding of related advanced/new technology. Keep up with the
technology development in the related areas in the industry, which could be
leveraged to enhance current architectures and build durable new ones.
Bachelors degree in Computer Science or related technical discipline required.
Advanced degree is a plus.
Strong leadership and team player.
Strong skills on mentoring/growing junior people
Payment industry experience is a plus.

Preferred Qualification
Experience with ETL / ELT tools / applications
Experience with Apache NiFi and Apache Spark for processing large data sets
Experience on Elastic Search
Knowledge on Data Catalog tools
Experience in building Data Pipeline development tools
Experience with Data Governance and Data Quality tools