Data Architect

This architecture includes the definition and details of relevant endpoints and computational services necessary for process for data ingestion, processing and storage, persistent data source and sink schema design including the product catalog itself and intermediate data sources to facilitate necessary pipeline jobs, performance optimization and parallelized distributed computation design, as well as resource definition for inference endpoints.

The product catalog structure and knowledge graph itself must be flexible in its ontology to accommodate the wide variety of product attribute sets across products. In application, products are vended from cribs (tool and product vending machines) composed of configured bins (slots) to support the needs of customers based upon specific products. The primary application of this catalog is for point of use crib solution recommendation. Critically the product catalog and knowledge graph must incorporate a proprietary understanding of compatibility and degree of preference for usability in certain machines, known as vending configuration information, which directs recommendation of a given product to prospective crib and bin combinations.

There are several required processes including but not limited to curation and cleaning, duplicate resolution, entity expansion, relational discovery, searches for similar products by product attributes. The product catalog is itself a live structure that constantly expands upon the receipt of new data which primarily serves the goals of intelligent product query or search, having applications in the overall recommendation of point of use crib solutions of as well as insights that require relationships arising from product attributes.

This role will not be the only role required to implement the corresponding architecture, but this role will be hands on, laying down the foundation of implementation through initial framework design and setup in the form of POCs and must be capable of the required implementation even if collaborating with other engineers.

Azure Data Architect with Graph databases, Knowledge Graphs

1. Product Catalog (Gold Standard) data structure definition in the form of established schema type (relational, hierarchical, structured), specified attribute and value mappings, specified persistent data storage resource to facilitate efficient queries
2. Product Knowledge Graph design in the form of specified and desired persistent data storage service (database) with demonstration of suitability for variety of required product data and applicable relationship of ontology and data attributes
3. Raw Data Ingestion architecture for receipt of raw product data in form of documented diagram, resource definition, schema standardization and support for variety of known input formats (POC required)
4. Product Attribute extraction service architecture in form of documented diagram, resource definition for retrieval of attributes from product descriptions and explicit attribute key value mapping pairs (POC required)
5. Entity Incorporation (into internal Knowledge Graph) architecture design in form of documented diagram, resource definition for corresponding endpoints for data processing and taxonomy enrichment (POC required)
6. Product duplicate resolution (into internal Knowledge Graph) architecture in form of documented diagram, resource definition (POC required)
7. Product Catalog output update architecture in form of documented diagram, necessary compute resources and endpoints (POC required)
8. Similar Product Search service architecture in form of documented diagram, resource definition to facilitate efficient queries for similar products based on provided attributes across millions of candidates (POC required)
9. Product Catalog and Knowledge Graph snapshot architecture, in form of documented diagram or resource for maintaining historical snapshot states of the everchanging data structures
10. Architecture Performance Optimization Plan in form of documented execution limits (time and space) and latency of various stages of data processing
11. Architecture Job and Pipeline Scheduling and Orchestration Plan in form of documented steps, timings with dependencies and desired orchestration service

Architecture Implementation Verification and Validation in the form of documented end to end pipeline execution and proven functionality