17 Data Art Solutions Jobs
1-3 years
Kolkata, Mumbai, New Delhi + 4 more
1 vacancy
Data Engineer
Data Art Solutions
posted 13d ago
Key skills for the job
Design and implement robust ETL/ELT pipelines to ingest API metadata from diverse sources (e.g., Gateways, Workflow platform such as ServiceNow, Development platforms such as Gitlab and Network platforms such as Kafka and F5)
Build scalable and reusable data engineering frameworks to streamline data processing
Ensure data pipelines handle large volumes of structured and semi-structured data efficientlyIntegrate data from APIs, databases, and other platforms into MongoDB or ElasticsearchTransform raw data into clean, enriched, and usable formats for API catalogue ingestion
Maintain and manage unified metadata models for versioning, standards adherence, and API quality metrics
Optimize data pipelines for performance, scalability, and reliabilityImplement monitoring, logging, and alerting for pipeline health and data quality
Collaborate with DevOps teams to implement CI/CD workflows for data engineering tasksWork closely with product teams, business analysts, and solution architects to understand data requirements
Provide technical expertise for data-related decisions and troubleshooting
Collaborate with enterprise architects to align pipelines with organizational standardsEnsure compliance with data security and privacy policies, including role-based access control (RBAC)Implement data validation and quality checks within pipelinesContribute to the development of data governance frameworks for catalogue metadata
Proficiency in data engineering tools and technologies (e.g., Apache Spark, Kafka, Airflow)
Expertise in building and managing data pipelines using mainly Java, or PythonStrong experience with database technologies like MongoDB, Elasticsearch, and SQL/NoSQL databases
Familiarity with API integration, RESTful services, and JSON/XML data formats
Knowledge of cloud platforms (AWS preferred) and their data services (e.g., S3, Lambda)
Experience with containerization and orchestration tools (Docker, Kubernetes)
Understanding of CI/CD practices and tools (e.g., GitLab)Excellent problem-solving and analytical skills
Strong communication skills for technical and non-technical stakeholders
Ability to work in an agile and collaborative environment
Detail-oriented with a focus on delivering high-quality outputs5+ years of experience in data engineering or related roles
Employment Type: Full Time, Permanent
Read full job description