Senior Data Engineer - ETL/Data Pipeline (6-8 yrs)

Key Responsibilities :

Data Pipeline Development :

- Design, develop, and maintain robust data pipelines using tools like Airflow, Databricks, and Pyspark.

- Extract, transform, and load data from various sources, including databases, APIs, and cloud storage.

- Ensure data quality and integrity through rigorous testing and validation procedures.

Big Data Technologies :

- Leverage big data technologies like Hadoop, Spark, and Snowflake to process and analyze large datasets.

- Optimize data pipelines for performance and scalability.

- Implement data warehousing and data lake solutions.

Cloud Infrastructure :

- Utilize AWS services (i.e., S3, EMR, Redshift) to build and manage cloud-based data infrastructure.

- Automate infrastructure provisioning and management using tools like Terraform and CloudFormation.

API Integration :

- Integrate with external APIs (i.e., Facebook Graph API, GA, Segment, Google Ad Words, Pinterest, Snapchat) to extract valuable data.

- Handle API rate limits, authentication, and error handling.

Data Quality Assurance :

- Implement data quality checks and monitoring to ensure data accuracy and consistency.

- Identify and resolve data quality issues proactively.

Collaboration :

- Collaborate with data analysts, data scientists, and other stakeholders to understand their data needs and deliver solutions.

- Contribute to a culture of data-driven decision-making.

Required Qualifications :

- 6+ years of experience in data engineering and ETL pipeline development.

- Proficiency in SQL, Python, and Pyspark.

- Strong understanding of big data technologies (Hadoop, Spark, etc.

- Experience with cloud platforms, especially AWS.

- Proficiency in data pipeline orchestration tools like Airflow.

- Experience with data warehousing and data lake architectures.

- Knowledge of data modeling and data warehousing concepts.

- Experience with data quality and validation techniques.

- Strong problem-solving and analytical skills.

- Excellent communication and collaboration skills.

Preferred Qualifications :

- Experience with Kubernetes and Docker.

- Knowledge of machine learning and AI concepts.

- Experience with data visualization tools (i.e., Tableau, PowerBI).

- Certifications in AWS, Azure, or GCP