Lead Big Data Engineer - Hadoop/Hive/Spark (4-9 yrs)

Job Title : Senior Data Engineer (Big Data)

Experience : 4-9 Years

Location : Bangalore

Job Overview :

We are seeking a skilled Senior Data Engineer to coordinate with cross-functional teams, manage and prioritize projects, and drive key data engineering initiatives. The successful candidate will have a deep understanding of big data tools and the ability to translate business needs into data models. This role requires proficiency in creating robust data pipelines, developing and maintaining data lakes, and implementing CI/CD pipelines for data solutions.

Key Responsibilities :

- Requirement Analysis : Understand and translate business needs into scalable data models to support long-term solutions.

- Data Modeling : Collaborate with the Business team to implement data strategies, design data flows, and develop conceptual data models.

- Data Pipeline Design : Create robust and scalable data pipelines across multiple domains.

- Data Integration : Develop and maintain data lakes by acquiring data from primary and secondary sources, ensuring flexibility and scalability in data evaluation.

- Testing and Quality Assurance : Define and manage data load procedures to ensure data quality, adhering to business rules and quality thresholds.

- Deployment : Implement data strategies and develop physical data models in collaboration with development teams, data analysts, and information system teams to ensure operational excellence.

- Team Leadership : Coordinate with other teams to manage and prioritize data-related projects.

- Stakeholder Management : Collaborate with internal stakeholders to understand and deliver solutions based on their data requirements.

- CI/CD Implementation : Build and maintain CI/CD pipelines for data solutions, ensuring smooth updates to the data environment.

Technical Skills :

- Big Data Tools : Expertise in Hadoop, Hive, and Spark.

- Programming Languages : Proficient in Scala, Python, SQL, and PySpark.

- Cloud Services : Experience with AWS or Azure.

- Stream Processing : Experience with Kafka, Spark-Streaming, etc.

- Database Management : Hands-on experience with relational (Postgres) and NoSQL (Cassandra) databases.

- Unix/Linux : Experience with Unix/Linux systems and scripting (Bash).

- Workflow Management : Familiarity with tools like Airflow for managing data pipelines.

- Data Transformation : Deep understanding of data transformation, metadata, and workload management processes.

Preferred Qualifications :

- Educational Qualification : Bachelor's degree in a relevant field.

- Industry Experience : Experience in the Satellite/Telecom industry will be an added advantage.

- Leadership Skills : Proven experience in leading large data engineering projects and cross-functional teams