Senior Data Engineer

Under minimal to moderate supervision, responsible for designing, building, and managing the infrastructure and tools that allow for the efficient processing and analysis of large data sets. Propose short/long term solutions. Automate the and eliminate manual process. Should be familiar with AWS services as we will be using most of them.

Essential Duties & Responsibilities
Develops software applications and supporting infrastructure using established coding standards and methodologies
Sets example for software quality through multiple levels of automated tests, including but not limited to unit, End to End, and load.
Self-starter and self-organized - able to work without supervision
Develops tooling, test harnesses and innovative solutions to understand and monitor the quality of the product
Develops infrastructure as code to reliably deploy applications on demand or through automation
Understands cloud managed services and builds scalable and secure applications using them
Creates proof of concepts for new ideas that answer key questions of feasibility, desirability, and viability
Work with other technical leaders to establish coding standards, development best practices and technology direction
Performs thorough code reviews that promote better understanding throughout the team
Work with architects, designers, business analysts and others to design and implement high quality software solutions
Builds intuitive user interfaces with the end user persona in mind using front end frameworks and styling
Assist product owners in backlog grooming, story breakdown and story estimation
Collaborate and communicate effectively with team members and other stakeholders throughout the organization
Document software changes for use by other engineers, quality assurance and documentation specialists
Master the technologies, languages, and practices used by the team and project assigned
Train others in the technologies, languages, and practices used by the team
Troubleshoot, instrument and debug existing software resolving root causes of defective behavior
Other duties as assigned

 

Education
BE/BTech or equivalent work experience

 

Minimum Qualifications
6 years experience in Data Engineering Data Processing using various tools
Excellent understanding and hands on experience with T-SQL programming
Excellent understanding and hands on experience with Microsoft SQL Server, other RDBMS, and No-SQL systems
Excellent understanding and hands on experience with ETL, Data Warehouse and Data Lake Concepts
Exposure to cloud database technologies

 

Skills/Knowledge

Python, Spark, Jupyter, AWS Lambda, AWS S3, AWS EMR, AWS Glue, AWS Step Functions
Good to have AWS QuickSight
DocumentDB / DynamoDB / RDS / Athena / MSK / Amazone Redshift/ Apache Iceberg
Big data experience (Pyspark/Hadoop/Bigdata)
Technical expertise with data models, data mining and segmentation techniques and software architecture.
Good in performance tuning, especially tuning complex queries, procedure, and indexing strategies
Working experience with relational databases, queries optimization (DBA or database developer with proven hands on application and understanding) with high emphasis on postgres and SQL.
Proficient with Database Normalization process.
Must be proficient in MS-SQL programming and can handle complex logic.
Expert in SQL Queries, Stored Procedures, Performance Measurement & Tuning.
Designing and developing Database solutions, schemas for structured and unstructured data, stored procedures, and functions.
Experience on various ETL tools and techniques to handle data for cleaning, formatting, extraction, transforming and loading into Database Data Warehouse/Data Lake
Build data systems, pipelines, and workflows.
Work with software engineering teams to develop and scale our data infrastructure.
Develop and test stored procedures, views, triggers, and functions.
Sound knowledge and practical experience covering end to end Data Science life cycle.
Write well designed, testable, and efficient code.
Design, develop and implement the complete life cycle of data warehouses, data lakes etc.
Ensure compliance of standards and conventions in developing programs.
Good to have .Net/Java