Software Development Engineer II

Basic concepts of Data Engineering; Ingestion from diverse sources and file formats; Hadoop; Data Warehousing; Designing & Implementing large scale Distributed Data platforms & Data Lakes
- Building distributed platforms or services
- SQL; Spark; Query Tuning & Performance Optimization
- Expertise in advanced Scala (e.g Functional Programming; using Case classes; Complex Data Structures & Algorithms)
- Experience on SOLID & Dry principles and Good Software Architecture & Design Experience
- Languages: Python; Java; Scala
- Good experience in Big Data Unit; System; Integration & Regression Testing
- Devops experience in Jenkins; Maven ;Github; Artifactory/Jfrog; CI/CD
- Big Data Processing: Hadoop; Sqoop; Spark and Spark Streaming
- Hadoop Distributions: Cloudea / Hortonworks experience
- Data Streaming: Experience on Kafka and Spark Streaming
- Data Validation & Data Quality
- Data Lake & Medallion Architecture
- Shell Scripting & Automation using Ansible or related Configuration management tools
- Agile processes & tools like Jira & Confluence
- Code Management tools like Git
- File formats like ORC; Avro; Parquet; Json & CSV
- Big Data Orchastration: Airflow; Spark on Kubernetes; Yarn; Oozie;
Good to have:
- Building Managing Large Scale on-prem Hadoop clusters or on Cloud
- Data Catalog & Data Governance
- Customer facing experience
- Experience with wide range of Range of Big Data tools; databases & frameworks e.g Kafka Connect; Apache Arrow; Apache Iceberg; Flink
- Cloud Native Storage like Ceph or S3
- Cloud Data Engineering (Azure; AWS & GCP)
- SQL & NoSQL DB: Hive; Cassandra; Dremio; Presto
- Infrastructure: Hands on experience on Kubernetes; Docker and related container technologies
- Data Security & Compliance: Experience working on GDPR & Regulatory Frameworks
- Security & Secrets Management like Ansible Vault; Snyk; Git Secrets; Hashicorp Vault

Big Data expertise with Spark; Scala; Hive; Airflow; CICD pipelines