Raasoft Infotech - Senior Data Engineer - AWS/Spark/Kafka (5-7 yrs)

Job Title : Senior Data Engineer

Company : Raasoft Infotech

Location : Mangalore/Bangalore

Experience : 5+ years

Employment Type : WFO/Hybrid

About the Role :

We are seeking a highly skilled and experienced Senior Data Engineer to join our growing team. In this critical role, you will be responsible for designing, developing, and optimizing data pipelines and infrastructure that power our data-driven decision-making.

You will work closely with data scientists, analysts, and software engineers to ensure the timely and accurate flow of data across our organization.

Key Responsibilities :

- Design, build, and maintain robust and scalable data pipelines using industry-standard tools and technologies (i.e., AWS services, Apache Spark, Kafka).

- Develop and implement batch and real-time data processing solutions to meet evolving business needs.

- Ensure the efficient and timely extraction, transformation, and loading (ETL/ELT) of data from various sources.

- Integrate data from diverse sources, including databases, APIs, cloud storage, and other data systems.

- Develop and implement data integration strategies to ensure data consistency and accuracy.

- Build and manage cloud-based or on-premises data platforms (i.e., data lakes, data warehouses, data marts) on AWS.

- Ensure the availability, performance, and security of the data infrastructure.

- Optimize data storage and processing costs.

- Design and implement optimized data models (i.e., star schema, snowflake schema) to support business intelligence and analytics.

- Create and maintain data dictionaries and metadata.

- Ensure data quality, integrity, and security by implementing appropriate data governance policies and procedures.

- Monitor data usage and access to ensure compliance with regulations and security standards.

- Work closely with data scientists, analysts, and software engineers to understand business requirements and translate them into technical solutions.

- Collaborate with stakeholders to identify data needs and prioritize projects.

- Continuously monitor and analyze data pipeline performance.

- Identify and implement performance optimization techniques to improve data processing speed and reduce latency.

- Mentor junior data engineers and provide technical guidance.

- Contribute to the development and documentation of best practices.

- Stay abreast of the latest advancements in data engineering technologies.

- Establish and maintain robust monitoring systems to track data pipeline health and identify potential issues.

- Proactively troubleshoot and resolve data quality and infrastructure issues.

Required Skills :

Strong proficiency in :

- AWS services : S3, EC2, EMR, Glue, Redshift, Athena, Lambda, etc.

- Data warehousing and data lake technologies : Snowflake, Redshift, S3, etc.

- Programming languages : Python (with libraries like Pandas, NumPy, Scikit-learn)

- SQL : Advanced SQL skills for data manipulation, querying, and analysis

- Data pipeline tools : Apache Airflow, Prefect, Luigi

Experience with :

-Agile development methodologies

- Data modeling techniques (star schema, snowflake schema)

- Data quality and validation techniques

- Data security and privacy best practices

- Excellent problem-solving and analytical skills

- Strong communication and interpersonal skills

- Ability to work independently and as part of a team -

Preferred Skills :

- Experience with containerization technologies (Docker, Kubernetes)

- Experience with stream processing technologies (Kafka, Kinesis)

- Experience with NoSQL databases (MongoDB, Cassandra)

- Experience with data visualization tools (Tableau, Power BI)