Data Engineer / Senior Data Engineer

Job Description:

The Senior Data Engineer with Databricks experience at Koantek will use comprehensive modern data engineer techniques and methods with Advanced Analytics to support business decisions for our clients. You will help design and build data pipelines, data streams, reporting tools, information dashboards, data service APIs, data generators, and other end-user information portals and insight tools. You will be a critical part of the data supply chain, ensuring that stakeholders can access and manipulate data for routine and ad hoc analysis to drive business outcomes using Advanced Analytics

Role: Senior Data Engineer (With Any cloud platform AWS / GCP / Azure)

Job Requirements:

* Strong experience as a data engineer (Databricks preferred)

* Expert proficiency in Spark Scala,Advanced SQL & Python (mandate), and PySpark is a plus

* Must have data migration experience from on prem to cloud

* Hands-on experience in Kinesis to process & analyze Streaming data, and cloud DynamoDB

* In depth understanding of cloud, data lake, and analytics solutions.

* Expert level hands-on development in designing and developing applications on Databricks,

Databricks Workflows, cloud Managed Airflow, and Apache Airflow is required.

* Extensive hands-on experience implementing data migration and data processing using Cloud

services: VPC/SG, EC2, S3, AutoScaling, CloudFormation, LakeFormation, DMS, Kinesis, Kafka,

Nifi, CDC processing, Amazon S3, EMR, Redshift, Athena, Snowflake, RDS, Aurora, Neptune,

DynamoDB, Cloudtrail, CloudWatch, Docker, Lambda, Spark, Glue, SageMaker, AI/ML, API GW,

etc.

* Hands-on experience with the Technology stack available in the industry for data management,

data ingestion, capture, processing, and curation: Kafka, StreamSets, Attunity, GoldenGate, Map

Reduce, Hadoop, Hive, Hbase, Cassandra, Spark, Flume, Hive, Impala, etc.

* Knowledge of different programming and scripting languages

* Good working knowledge of code versioning tools [such as Git, Bitbucket or SVN]

* Hands-on experience in using Spark SQL with various data sources like JSON, Parquet and Key

Value Pair

* Experience preparing data for use in SageMaker and Databricks.

* Demonstrated experience preparing data, automating and building data pipelines for AI Use

Cases (text, voice, image, IoT data etc....).

* Experience in creating tables, partitioning, bucketing, loading and aggregating data using Spark

Scala, Spark SQL/PySpark

* Knowledge of any cloud DevOps processes like CI/CD as well as Agile tools and processes

including Git, Jenkins, Jira, and Confluence

* Strong understanding of Data Modeling and defining conceptual logical and physical data

models.

Responsibilities:

* Work closely with team members to lead and drive enterprise solutions, advising on key decision points on trade-offs, best practices, and risk mitigation

* Manage data related requests, analyze issues, and provide efficient resolution. Design all

program specifications and perform required tests

* Design and Develop data Ingestion using Glue, Any cloud Managed Airflow, Apache Airflow and

processing layer using Databricks.

* Work with the SMEs to implement data strategies and build data flows and prepare codes for all modules according to required specification.

* Monitor all production issues and inquiries and provide efficient resolution.

* Evaluate all functional requirements, map documents, and troubleshoot all development

processes

* Document all technical specifications and associates project deliverables and design all test cases to provide support to all systems and perform unit tests.