Data Engineer

Job Title: Data Engineer

We are seeking a skilled and detail-oriented Data Engineer to design, build, and maintain robust data pipelines and infrastructure. The ideal candidate will extract, transform, and load data from various sources, ensuring high-quality data is available for business analysis and decision-making. You will collaborate with cross-functional teams to enable efficient data workflows and ensure the accuracy and usability of data systems.

Key Responsibilities:

Data Extraction: Extract data from diverse sources such as APIs, databases, websites, and other repositories.

Data Transformation: Transform raw data into structured and usable formats using ETL tools and techniques.

Data Loading: Load transformed data into data warehouses, data lakes, or other storage systems while optimizing for performance and scalability.

Data Quality Assurance: Validate and ensure the accuracy, consistency, and reliability of data.

Pipeline Maintenance: Monitor and troubleshoot data pipelines to maintain optimal performance.

Collaboration: Work closely with data analysts, scientists, and stakeholders to understand data requirements and implement effective solutions.

Documentation: Maintain clear documentation of data processes, pipelines, and systems.

Required Skills and Qualifications:

Proficiency in programming languages like Python, SQL, or Scala.

Hands-on experience with ETL tools such as Apache Airflow, Talend, or Informatica.

Familiarity with data storage solutions, including data warehouses (e.g., Snowflake, Redshift) and data lakes.

Strong understanding of relational and non-relational databases (e.g., PostgreSQL, MySQL, MongoDB).

Experience with big data technologies such as Hadoop, Spark, or Kafka.

Knowledge of cloud platforms like AWS (S3, Glue, Redshift), Azure (Data Factory, Synapse), or GCP (BigQuery, Dataflow).

Solid understanding of data modeling and data architecture principles.

Strong analytical and problem-solving skills.

Preferred skillsets

Knowledge of API integration and web scraping for data extraction.

Familiarity with version control tools like Git.

Understanding of CI/CD pipelines.

Experience with data visualization tools (e.g., Tableau, Power BI).

Certification in cloud platforms (AWS, Azure, or GCP).