Data Engineer

OVERVIEW

Do you have a passion for building the foundation for data-driven decision making? Are you an expert in data wrangling, storage, and pipelines? Then this Data Engineer role is the perfect opportunity for you!

In this exciting role, you will play a critical role in our data team by designing, developing, and maintaining the infrastructure that supports our ever-growing data needs. You'll leverage your technical expertise to ensure data is collected, transformed, and delivered efficiently and reliably for analysis and consumption.

RESPONSIBILITIES

• Data Ingestion and Integration:
• Design and implement data pipelines to extract data from various sources (databases, APIs, log files) in real-time or batch mode.
• Develop data transformation processes to clean, validate, and prepare data for analysis.
• Ensure data quality and consistency throughout the data lifecycle.
• Data Storage and Management:
• Design and implement data storage solutions using technologies like data warehouses, data lakes, and cloud storage platforms (AWS S3, Azure Blob Storage, etc.).
• Optimize data storage for performance, scalability, and cost-effectiveness.
• Implement data security best practices to protect sensitive information.
• Data Pipelines and Automation:
• Develop and automate data pipelines using workflow orchestration tools (Airflow, Luigi) to ensure smooth data flow.
• Monitor data pipelines for errors and implement alerting mechanisms.
• Continuously improve data pipelines for efficiency and performance.
• Collaboration and Communication:
• Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements and design solutions.
• Document data pipelines and processes for maintainability and future reference.
• Communicate technical concepts related to data infrastructure to non-technical audiences.

SKILLS AND EXPERIENCE

• Bachelor's degree in Computer Science, Information Technology, or a related field (or equivalent experience).
• Minimum 3+ years of experience as a Data Engineer or a related role.
• Strong programming skills in languages like Python, Java, or Scala.
• Experience with SQL and relational databases (e.g., MySQL, PostgreSQL).
• Experience with data warehousing concepts and technologies (e.g., Apache Spark, Hadoop).
• Familiarity with cloud platforms (AWS, Azure, GCP) is a plus.
• Excellent problem-solving and analytical skills.
• Strong attention to detail and commitment to data quality.
• Effective communication and collaboration skills