Big Data Engineer - Java/Python/Scala (5-10 yrs)

Job Description :

- Develop and maintain robust data pipelines for ingesting data from various sources such as databases, APIs, and streaming platforms.

- Design and implement data transformation and cleaning processes to ensure data quality and consistency.

- Develop data models and schemas for efficient data storage and retrieval.

- Expertise in working with big data technologies such as Hadoop, Spark, Hive, HBase, Kafka, and cloud platforms (AWS, Azure, GCP).

- Experience with NoSQL databases such as Cassandra, MongoDB, and Elasticsearch.

- Strong understanding of data warehousing concepts and methodologies.

- Collaborate with data scientists and analysts to perform exploratory data analysis and generate insights.

- Develop and maintain data dashboards and visualizations using tools like Tableau, Power BI, or similar.

- Identify and optimize performance bottlenecks in data pipelines and queries.

- Implement strategies for improving data processing speed and efficiency.

- Experience with cloud-based big data services such as AWS EMR, Azure Databricks, or GCP Dataflow.

- Understanding of cloud infrastructure and security best practices.

- Work closely with cross-functional teams (data scientists, engineers, analysts, product managers) to understand business

requirements and translate them into technical solutions.

- Communicate complex technical concepts effectively to both technical and non-technical audiences.

Technical Skills :

- Strong proficiency in Java, Python, or Scala.

- Expertise in SQL and NoSQL databases.

- Experience with big data frameworks such as Hadoop, Spark, Hive, HBase.

- Familiarity with cloud platforms (AWS, Azure, GCP) and their big data services.

- Experience with data visualization tools (Tableau, Power BI).