Senior Data Engineer - Data Pipeline (3-5 yrs)

Job Summary :

We are seeking a highly skilled and experienced Senior Data Engineer to join our growing data engineering team. You will play a critical role in designing, building, and maintaining our data infrastructure and pipelines.

This role requires a deep understanding of data engineering principles, expertise in a variety of data processing technologies, and a passion for building robust, scalable, and reliable data solutions. You will collaborate with data scientists, analysts, and other engineers to deliver high-quality data that empowers data-driven decision-making.

Key Responsibilities :

- Data Pipeline Development : Design, develop, and maintain robust and scalable data pipelines for ingestion, processing, and storage of large datasets.

- Data Warehousing : Design and implement data warehousing solutions, including dimensional modeling, ETL processes, and data quality management.

- Data Lake Development : Build and maintain data lakes to store raw and processed data in various formats.

- ETL Development : Develop and optimize ETL (Extract, Transform, Load) processes for data integration and transformation.

- Data Modeling : Design and implement data models that meet business requirements and ensure data integrity.

- Data Governance : Implement data governance policies and procedures to ensure data quality, consistency, and security.

- Performance Optimization : Monitor and optimize data pipeline performance, identifying and addressing bottlenecks.

- Technology Evaluation : Evaluate and recommend new data engineering technologies and tools.

- Collaboration : Collaborate with data scientists, analysts, and other engineers to understand data needs and deliver effective solutions.

- Mentorship : Mentor and guide junior data engineers.

- Documentation : Create and maintain clear and comprehensive documentation for data pipelines and infrastructure.

Required Technical Skills :

- Programming Languages : Strong proficiency in at least one programming language (i.e., Python, Java, Scala). Python is highly preferred.

- Big Data Technologies : Extensive experience with big data technologies like Hadoop, Spark, Hive, Kafka, or similar.

- Data Warehousing : Solid understanding of data warehousing concepts, dimensional modeling, and ETL processes.

- SQL : Strong SQL skills for data querying and manipulation.

- Cloud Platforms : Hands-on experience with cloud platforms like AWS, Azure, or GCP, and their data engineering services.

- Data Storage : Experience with various data storage solutions, including relational databases (i.e., PostgreSQL, MySQL), NoSQL databases (i.e. MongoDB, Cassandra), and cloud-based storage services (i.e., S3, Azure Blob Storage).

- Data Pipeline Tools : Experience with data pipeline orchestration tools like Airflow, Luigi, or similar.

- Version Control : Proficiency with Git.

Preferred Skills :

- Data Governance Tools : Familiarity with data governance tools and frameworks.

- Data Visualization : Experience with data visualization tools like Tableau or Power BI.

- DevOps Practices : Understanding of DevOps principles and practices.

- Machine Learning : Basic understanding of machine learning concepts and algorithms.

Required Experience & Qualifications :

- Experience : 3+ years of experience as a Data Engineer, with a focus on building and maintaining data pipelines and infrastructure.

- Education : Bachelor's or Master's degree in Computer Science, Information Technology, or a related field preferred.

Personal Attributes :

- Strong problem-solving and analytical skills.

- Excellent communication and interpersonal skills.

- Ability to work effectively in a team environment.

- Proactive, self-motivated, and results-oriented.

- Attention to detail and a commitment to quality