Urgent Hiring || Sr Data Engineer - ThinkSys Software Noida

Job Description

We seek a highly skilled and experienced Big Data Architect to join our dynamic team for an esteemed overseas client. The ideal candidate will be responsible for designing, implementing, and managing large-scale data architectures. This role requires a deep understanding of big data technologies, data lakes, data warehousing, ETL processes, and data pipeline orchestration. The candidate will play a critical role in handling large databases and data sets, ensuring data integrity, performance, and security.

Key Responsibilities

Architecture Design: Design and implement scalable and robust big data architectures to support data processing, storage, and analysis.

Data Management: Develop and maintain data lake and data warehousing solutions for efficient data storage and retrieval.

ETL Processes: Design, develop, and optimize ETL processes to ensure seamless data integration from various sources.

Database Management: Manage and optimize Amazon Redshift clusters for high performance and cost efficiency.

Big Data Technologies: Leverage big data technologies such as Hadoop, Spark, and other relevant tools to handle large-scale data processing.

Scripting: Write and maintain efficient Python scripts for data processing and automation tasks.

Pipeline Orchestration: Utilize Airflow or similar orchestration tools to automate and manage

complex data pipelines.

Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver suitable solutions.

Quality Assurance: Ensure data quality, integrity, and security across all data platforms.

Monitoring & Troubleshooting: Monitor and troubleshoot data pipelines and architectures to ensure high availability and performance.

Continuous Learning: Stay updated with the latest industry trends and best practices in big data

technologies and architectures.

Nice to have: Data Platform Engineering

Qualifications

Bachelors or Masters degree in Computer Science, Information Technology, or a related field.

5+ years of relevant experience in big data architecture, data engineering, or a related field.

Strong understanding of data lake and data warehousing concepts and architectures.

Proven experience with ETL technologies and tools.

Hands-on experience with Amazon Redshift and managing large-scale data warehouses.

Proficiency in big data technologies, including Hadoop, Spark, and related ecosystems.

Strong Python scripting skills for data processing and automation.

Experience with Airflow or similar orchestration tools.

Experience in handling large databases and data sets, ensuring data quality and performance.

Excellent problem-solving skills and ability to troubleshoot complex data issues.

Strong communication skills and ability to work collaboratively with cross-functional teams.

Knowledge of data governance, security, and compliance best practices.

Preferred Skills

Experience with cloud platforms such as AWS, Azure, or Google Cloud.

Familiarity with additional programming languages like SQL, Java, or Scala.

Experience with data visualization tools and techniques.