Techstar Software - Big Data Engineer - Spark/Hadoop (7-15 yrs)

A Big Data Engineer Lead plays a critical role in managing and guiding the development of big data solutions within an organization. Here's a typical job description for this role.

Job Title : Big Data Engineer Lead

Exp : 5+ Years

Location : Hyderabad

Notice Period : Immediate to 30days

Job Summary :

The Big Data Engineer Lead is responsible for overseeing the design, implementation, and management of large-scale data processing systems. This role involves leading a team of data engineers, collaborating with cross-functional teams, and ensuring the organization effectively utilizes big data technologies to derive actionable insights.

Key Responsibilities :

Team Leadership :

- Lead and mentor a team of data engineers, fostering a collaborative and innovative work environment.

- Provide technical guidance and support for team members in project execution

Architecture Design :

- Design and implement scalable big data architectures and solutions.

- Ensure systems are optimized for performance and scalability.

Data Processing :

- Develop and maintain data pipelines for processing structured and unstructured data.

- Utilize big data technologies (e.g., Hadoop, Spark, Kafka) to manage large datasets.

Collaboration :

- Work closely with data scientists, analysts, and stakeholders to understand data needs and requirements.

- Communicate effectively with technical and non-technical teams.

Quality Assurance :

- Establish data governance and data quality standards.

- Implement testing and validation processes for data integrity.

Innovation and Strategy :

- Stay updated with industry trends and emerging technologies in big data.

- Propose and implement best practices for data engineering and analytics.

Documentation and Reporting :

- Create and maintain documentation for data architecture, processes, and systems.

- Report on project progress and performance metrics to upper management.

Qualifications :

Education :

- Bachelor's degree in computer science, Engineering, Data Science, or a related field; a Master's degree is a plus.

Experience:

- 5+ years of experience in data engineering or related fields, with at least 2 years in a leadership role.

- Proven experience with big data technologies (e.g., Hadoop, PySpark, Hive, NoSQL databases).

Technical Skills :

- Proficiency in programming languages such as Python or Java, and Scala.

- Experience with Cloud Platform CI/CD Pipelines

- Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and data warehousing solutions.

Soft Skills :

- Strong analytical and problem-solving abilities.

- Excellent communication and interpersonal skills.

- Ability to manage multiple projects and priorities effectively.

Preferred Skills :

- Familiarity with machine learning frameworks and tools.

- Knowledge of data visualization tools (e.g., Tableau, Power BI).

- Experience with DevOps practices and CI/CD pipelines.

Work Environment :

This position may require occasional overtime and the flexibility to adapt to changing project requirements. Remote or hybrid work options may be available.