Big Data Engineer - Azure Data Factory (6-8 yrs)

Job Description :

As a Big Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines to extract, transform, and load (ETL) data from various sources into target systems.

You will work closely with data analysts and data scientists to ensure data quality, accuracy, and consistency.

Key Responsibilities :

- Design and implement robust data ingestion pipelines to extract data from various sources such as databases, APIs, and cloud storage systems.

- Utilize tools like Apache Spark, Apache Kafka, or Azure Data Factory to automate data extraction and loading processes.

- Handle missing values, outliers, and inconsistencies in data.

- Design and implement efficient data models to support business intelligence and analytics.

- Optimize data models for performance and scalability.

- Develop and maintain ETL and ELT pipelines using tools like Apache Airflow, Apache NiFi, or Azure Data Factory.

- Optimize ETL processes for performance and cost-efficiency.

- Design and implement real-time data pipelines using tools like Apache Kafka or Apache Flink.

- Monitor data quality and identify data anomalies.

- Leverage cloud platforms like AWS, Azure, or GCP to build scalable and cost-effective data solutions.

- Utilize cloud-native services like AWS Glue, Azure Data Factory, or Google Cloud Dataflow.

Required Skills and Experience :

Programming Languages : Strong proficiency in Python and Scala.

Big Data Technologies: Experience with Hadoop, Spark, Hive, HDFS, and other big data tools.

Data Warehousing and ETL: Understanding of data warehousing concepts and data modeling techniques.

Experience with ETL and ELT processes and tools.

Cloud Platforms: Knowledge of cloud platforms (AWS, Azure, GCP) and cloud-native services.

Databases: Proficiency in SQL and experience with relational and NoSQL databases.

Data Pipelines: Experience with building and managing data pipelines.

CI/CD: Experience with CI/CD tools like Jenkins, GitLab CI/CD, or Azure DevOps.

Containerization: Experience with containerization tools like Docker and Kubernetes.

Cloud Orchestration: Experience with cloud orchestration tools like Kubernetes, Terraform, or Ansible.

Soft Skills: Strong problem-solving and analytical skills.

Excellent communication and collaboration skills.

Preferred Skills :

- Experience with machine learning and AI frameworks (e.g, TensorFlow, PyTorch).

- Knowledge of data visualization tools (e.g, Tableau, Power BI).

- Experience with data streaming technologies (e.g, Kafka, Kinesis).