Big Data Engineer

Job Description:

We are seeking a high-energy Big Data Engineer to join our team. The ideal candidate will have strong expertise in Python, PySpark, Big Data technologies, and Google Cloud Platform (GCP). This role requires hands-on experience in designing, developing, and optimizing data processing applications in a cloud-based environment.

Key Responsibilities:

• Design and implement scalable data processing solutions using PySpark, SQL, and Big Data technologies.
• Develop and maintain ETL pipelines for handling large-scale data processing.
• Work with Big Data technologies such as Spark, Hive, MapReduce, and HBase to process and analyze structured and unstructured data.
• Optimize database queries and indexing for high-performance data retrieval.
• Develop scripts using UNIX, Shell, Perl, and Python for automation and data processing tasks.
• Implement RESTful/SOAP web services to facilitate data integration and exchange.
• Collaborate with cross-functional teams to analyze business requirements and translate them into technical solutions.
• Leverage Google Cloud Platform (GCP) for cloud-based data solutions.
• Ensure best practices in data engineering, including code quality, performance tuning, and security.
• Work in an Agile environment, participating in sprint planning and story analysis.

Required Skills & Qualifications:

• Bachelor's degree in Computer Science, Computer Engineering, or a related technical field, or equivalent work experience.
• 6 - 9 years of experience in software development and data engineering.
• Strong proficiency in SQL and PySpark.
• Hands-on experience with Big Data technologies (Spark, Hive, MapReduce, HBase).
• Proficiency in UNIX/Linux scripting (Shell, Perl, Python).
• Experience with Google Cloud Platform (GCP).
• Understanding of database query optimization and indexing techniques.
• Experience in designing and implementing REST/SOAP web services (Java is a plus).
• Strong analytical and problem-solving skills.
• Experience in Agile methodologies and working closely with business stakeholders.

Nice to Have:

• Knowledge of additional cloud platforms such as AWS or Azure.
• Experience with CI/CD pipelines for data engineering workflows.
• Exposure to machine learning and data science concepts.