Senior Data Engineer - ETL/Hadoop (5-8 yrs)

Job Title : Senior Data Engineer

Location : Bangalore

Experience Level : 5-8 years

Job Summary :

We are looking for a skilled and experienced Senior Data Engineer to join our dynamic team. In this role, you will be responsible for designing, implementing, and maintaining robust and scalable data pipelines and architectures that support our business intelligence, data analytics, and machine learning initiatives. You will work closely with data scientists, analysts, and other engineering teams to ensure high-quality data is available for critical business decision-making.

Key Responsibilities :

Data Pipeline Development : Build and maintain scalable, high-performance data pipelines for collecting, transforming, and storing large datasets.

ETL/ELT Processes : Design and implement ETL (Extract, Transform, Load) or ELT processes to integrate data from a variety of sources such as APIs, databases, and third-party services.

Data Architecture : Develop and optimize data architectures, including data warehouses, data lakes, and big data platforms, ensuring they support business requirements for analytics and reporting.

Collaboration : Work closely with data scientists, business analysts, and other engineering teams to understand data requirements and deliver high-quality solutions.

Data Integration : Integrate data from multiple systems and sources to provide a unified view for business intelligence and analytics purposes.

Automation and Optimization : Continuously improve the performance, scalability, and reliability of data pipelines and systems by identifying bottlenecks and implementing optimizations.

Data Quality : Ensure data quality by implementing data validation, error handling, and monitoring processes.

Big Data Technologies : Leverage big data technologies such as Hadoop, Spark, and Kafka to process large volumes of data efficiently.

Cloud Platforms : Develop and maintain cloud-based data infrastructure on platforms such as AWS, Azure, or Google Cloud.

Monitoring & Reporting : Build monitoring and alerting systems for data pipelines and data quality to ensure continuous operation and reduce downtime.

Mentorship : Mentor junior data engineers, providing guidance on best practices, architecture design, and code quality.

Technical Skills Required :

Programming Languages : Expertise in Python, Java, or Scala for data engineering tasks and pipeline development.

Big Data Frameworks : Experience with Hadoop, Apache Spark, Kafka, Airflow, and other big data tools for distributed data processing.

Cloud Platforms : Hands-on experience with cloud services such as AWS (Redshift, S3, Lambda), Azure, or Google Cloud (BigQuery, Dataflow).

Data Warehousing/Database Systems : Strong understanding of data warehousing concepts and experience with SQL databases (e.g., MySQL, PostgreSQL, SQL Server) and NoSQL databases (e.g., MongoDB, Cassandra).

ETL Tools : Proficiency in ETL tools such as Apache NiFi, Talend, or Matillion for data integration and processing.

Containerization & Orchestration : Experience with containerization technologies like Docker and orchestration tools such as Kubernetes.

Data Modeling : Strong knowledge of data modeling concepts for designing schemas and database structures that are optimized for performance.

Version Control : Experience with Git or similar version control systems for collaborative software development.