Data Engineer - R/ETL/Python (4-10 yrs)

Responsibilities :

- Responsible for designing, building, and maintaining the infrastructure, systems, processes needed to handle large volumes of data and work with various tools and technologies to develop solutions that
enable our organization to process, analyse, and derive insights from massive datasets.

- Collaborate with cross-functional teams to identify data needs, design solutions, and enhance data
driven decision-making capabilities.

- Maintain and ensure the integrity and reliability of data pipelines, implementing best practices in data
security and compliance.

- Execute end-to-end data engineering projects, from conceptualization to deployment, demonstrating
project management and technical prowess.

- Set the standard for data engineering practices within the company, guiding the architectural approaches, data pipeline designs, and the integration of cutting-edge technologies to foster a culture
of innovation and continuous improvement.

- Spearhead the discovery, evaluation, and integration of new datasets, collaborating (incl. pipeline development and data modelling/documentation), and working closely with key data stakeholders to
understand their impact and relevance to our core products and the business domain.

Skills & Qualifications :

- Experience in Data engineering, Data modelling, Structured/Non-Structured Data, ETL, Big database, Data warehouse, Data Lake & administration of data infrastructure, build highly reliable & scalable data
systems with good data governance that provide complete, secure, accurate and timely data for
analytics.

- Hands on administration of big data databases/compute engines (Data Lake, Redshift, EMR, Spark or
similar) and optimization/tuning technique

- Design, Development & implementation of OLTP & OLAP databases with efficient design, optimization
techniques & data ingestion, aggregation, labelling, anonymization, archival, retrival mechanisms.

- Proficient in Python, SQL and PL-SQL, query tuning, optimization, ETL, ELT, Data Modelling and Data
Warehouse systems.

- Strong Proficiency With AWS Technologies Like S3, EC2, Lambda, Batch, CloudWatch, Redshift, RDS,
DynamoDB, DMS, Data Lake, Glue, Athena, Kinesis.

- Experience in building pipelines, ingesting data, data orchestration from heterogeneous systems onto
data lake environments in real time using tools like Airbyte, AirFlow, Trino, Minio, Hive etc.

- Proficient in a variety of big-data tools and technologies, including Hadoop, Spark, etc., handling
different storage formats like Parquet, Delta/Apache Hudi/Apache Iceberg

- Employ advanced SQL, Python, R, Spark, and Bash scripting to optimize data processing, analysis, and
automation tasks. Advanced SQL, Python and associated libraries like Pandas, Numpy etc., Pyspark ,
Shell scripting, Data-Modelling, Big data, Hadoop, Hive, ETL pipelines.

- Experience with data visualization tools and analytics technologies (Meta, Power BI, Looker, Tableau,
etc.)

- Proficient in Business Intelligence (BI), analytic database and products, able to take business requirements and translate into database tables, pipeline design and tasks.

- Experience with technology stacks available for Metadata Management, Data Governance, Data Quality, MDM, Data Catalog etc.

- Proficient in integrating data from and to diverse platforms using API's

- Experience with building data pipelines in streaming and batch mode.

- Experience with Git and CI/CD pipelines to deploy cloud applications