Data Engineer

Development:
Design, build, and maintain robust, scalable, and high-performance data pipelines to ingest, process, and store large volumes of structured and unstructured data.
Utilize Apache Spark within Databricks to process big data efficiently, leveraging distributed computing to process large datasets in parallel.
Integrate data from a variety of internal and external sources, including databases, APIs, cloud storage, and real-time streaming data.
Data Integration & Storage:
Implement and maintain data lakes and warehouses, using technologies like Databricks, Azure Synapse, Redshift, BigQuery to store and retrieve data.
Design and implement data models, schemas, and architecture for efficient querying and storage.
Data Transformation & Optimization:
Leverage Databricks and Apache Spark to perform data transformations at scale, ensuring data is cleaned, transformed, and optimized for analytics.
Write and optimize Spark SQL, PySpark, and Scala code to process large datasets in real-time and batch jobs.
Work on ETL processes to extract, transform, and load data from various sources into cloud-based data environments.
Big Data Tools & Technologies:
Utilize cloud-based big data platforms (e.g., AWS, Azure, Google Cloud) in conjunction with Databricks for distributed data processing and storage.
Implement and maintain data pipelines using Apache Kafka, Apache Flink, and other data streaming technologies for real-time data processing.
Collaboration & Stakeholder Engagement:
Work with data scientists, data analysts, and business stakeholders to define data requirements and deliver solutions that align with business objectives.
Collaborate with cloud engineers, data architects, and other teams to ensure smooth integration and data flow between systems.
Monitoring & Automation:
Build and implement monitoring solutions for data pipelines, ensuring consistent performance, identifying issues, and optimizing workflows.
Automate data ingestion, transformation, and validation processes to reduce manual intervention and increase efficiency.
Document data pipeline processes, architectures, and data models to ensure clarity and maintainability.
Adhere to best practices in data engineering, software development, version control, and code review.
Required Skills & Qualifications:
Education: Bachelors degree in Computer Science, Engineering, Data Science, or a related field (or equivalent experience).

Technical Skills:
Apache Spark: Strong hands-on experience with Spark, specifically within Databricks (PySpark, Scala, Spark SQL).
Experience working with cloud-based platforms such as AWS, Azure, or Google Cloud, particularly in the context of big data processing and storage.
Proficiency in SQL and experience with cloud data warehouses (e.g., Redshift, BigQuery, Snowflake).
Strong programming skills in Python, Scala, or Java.
Big Data & Cloud Technologies:
Experience with distributed computing concepts and scalable data processing architectures.
Familiarity with data lake architectures and frameworks (e.g., AWS S3, Azure Data Lake).
Data Engineering Concepts:
Strong understanding of ETL processes, data modeling, and database design.
Experience with batch and real-time data processing techniques.
Familiarity with data quality, data governance, and privacy regulations.
Problem Solving & Analytical Skills:
Strong troubleshooting skills for resolving issues in data pipelines and performance optimization.
Ability to work with large, complex datasets, and perform data wrangling and cleaning.