Data Engineer

Cab Facility- Yes

Shift Time - 12:30 pm- 9:30 pm (Flexible)

People Manager role: No

Required education and certifications critical for the role-

Bachelor's degree in computer science, engineering, or a related field

Required years of experience - 4-8 years of relevant experience

Aon is in the business of better decisions

At Aon, we shape decisions for the better to protect and enrich the lives of people around the world.

As an organization, we are united through trust as one inclusive, diverse team, and we are passionate about helping our colleagues and clients succeed.

Competencies

- Independent worker

- Research/reading around the subject

- Abstraction of the problem

Required Skills

- 4+ years Python + pyTest, UnitTest, numpy, pandas

- Understanding of functional/modular development

- TDD

- DevOps CI/CD

- Schema / Database design

- ETL / ELT

- Relational (RDBS) vs Non-Relational (NoSql data stores)

- Formats: Xml, Json, Parquet, Excel, CSV, TxT

- Github/ SourceControl

Desirable / Awareness of

- Docker

- Airflow

- Spark

- Rabbit

- Kafka

- ML tools

1. Technical Skills

• Programming Languages: Proficiency in languages like Python, SQL, and sometimes Scala or Java for data processing.
• Database Management: Experience with both relational (SQL-based) and NoSQL databases, such as MS SQL
• Data Warehousing: Knowledge of data warehousing solutions like Amazon Redshift, Google BigQuery, Snowflake, or Microsoft Azure SQL Data Warehouse.
• ETL Tools: Proficiency with Extract, Transform, Load (ETL) tools such as Apache Airflow, Talend, or Informatica.
• Big Data Technologies: Experience with Hadoop ecosystem (Hive, Pig, HDFS) and Spark, Databricks for processing and managing large datasets.
• Machine Learning Modelling: Faimiliarity with XGBoost and creating decision trees.
• Cloud Platforms: Familiarity with cloud services (AWS, Google Cloud Platform, or Microsoft Azure) and their data storage, computing, and analysis services.
• Modular Development: Class design and familiarity with testing modular code
• Testing: Familiarity with unit testing concepts such as pytest and unittest.

2. Data Modeling and Pipeline Design

• Ability to design, implement, and optimize data pipelines that enable efficient data flow and retrieval for downstream use.
• Knowledge of data modeling and schema design, including dimensional modeling for analytics.
• Knowledge of streaming concepts and transformations required to convert data. Medallion approach to transformations.

3. Data Quality and Monitoring

• Experience with tools and processes to ensure data accuracy, consistency, and integrity.
• Setting up data validation, monitoring, and error-handling systems to maintain high data quality standards.

4. Scalability and Optimization

• Skills in optimizing data systems to handle scaling and increasing data volumes effectively.
• Knowledge of distributed systems, caching, and query optimization to improve performance.

5. Problem-Solving and Analytical Thinking

• Strong analytical skills for identifying data issues and understanding business requirements.
• Creative problem-solving ability to develop efficient data solutions that fit project needs.

6. Soft Skills

• Collaboration: Working effectively with data scientists, analysts, and other teams to understand data needs and ensure proper data access.
• Communication: Clearly communicating technical processes and insights to both technical and non-technical stakeholders.
• Attention to Detail: Ensuring data accuracy and completeness in all data processes.