Senior Data Engineer - SQL/Python (8-10 yrs)

Job Description :

We are seeking a highly motivated and skilled Data Engineer to join our dynamic team. The ideal candidate will have expertise in data engineering principles, proficiency in programming languages such as Python, and experience with cloud platforms, as well as a working knowledge of machine learning frameworks like TensorFlow.

This position offers an exciting opportunity to work with cutting-edge technologies in data processing, machine learning, and cloud computing to support data-driven decision-making within the organization.

Key Responsibilities :

Data Pipeline Development :

- Design, build, and maintain robust data pipelines that enable efficient data extraction, transformation, and loading (ETL) processes.

- Integrate data from various sources, ensuring data consistency and quality.

- Develop solutions to process large volumes of structured and unstructured data.

Data Modeling & Management :

- Collaborate with cross-functional teams to create efficient data models, optimize data storage, and ensure the performance of the databases.

- Use SQL to query, manipulate, and manage data in relational and NoSQL databases.

- Implement and manage data governance, ensuring data accuracy, availability, and security.

Machine Learning Integration :

- Work closely with data scientists and machine learning engineers to integrate machine learning models (including TensorFlow) into data pipelines.

- Support the deployment of machine learning models in production, ensuring scalability, maintainability, and real-time performance.

- Develop processes to monitor model performance and implement necessary improvements based on business needs.

Cloud Architecture & Data Infrastructure :

- Leverage cloud platforms (AWS, Azure, or Google Cloud) to build and scale data infrastructure, ensuring high availability and scalability.

- Design and implement cloud-based storage and computing solutions, including data lakes, warehouses, and distributed systems.

- Ensure seamless integration between on-premise and cloud-based data environments.

Automation & Optimization :

- Automate manual processes to ensure the efficient movement, transformation, and storage of data.

- Continuously optimize existing pipelines to improve performance, reduce costs, and scale with growing data needs.

- Identify and implement opportunities for streamlining data workflows, enhancing operational efficiency.

Collaboration & Documentation :

- Work collaboratively with data scientists, analysts, software engineers, and business stakeholders to ensure the successful delivery of data solutions.

- Maintain clear and comprehensive documentation for data pipeline processes, architecture, and system integrations.

- Participate in code reviews and knowledge-sharing sessions to ensure best practices and code quality.

Required Skills & Qualifications :

- Proficiency in Python : Strong knowledge of Python for data engineering, ETL scripting, and integration with data storage systems.

- SQL Expertise : Advanced skills in writing complex SQL queries for data extraction, transformation, and analysis across relational and NoSQL databases (e.g., PostgreSQL, MySQL, MongoDB).

- TensorFlow Experience : Familiarity with TensorFlow for integrating machine learning models into production environments, along with knowledge of data preprocessing for ML.

- Cloud Platform Experience : Hands-on experience with at least one major cloud platform (AWS, Azure, or Google Cloud), including familiarity with cloud storage services (e.g., S3, BigQuery) and computing frameworks (e.g., EC2, Lambda).

- Data Engineering Experience : 5+ years of experience in data engineering, working on data pipelines, big data tools (e.g., Hadoop, Spark), and data management.

- Version Control & CI/CD : Familiarity with version control systems (e.g., Git) and CI/CD pipelines for automated deployment and testing of data pipelines.

- Problem-Solving Skills : Strong analytical and problem-solving abilities, with a focus on improving performance and scalability of data processes.

Preferred Qualifications :

- Experience with containerization technologies (e.g., Docker, Kubernetes).

- Familiarity with data warehousing tools such as Redshift, Snowflake, or BigQuery.

- Knowledge of data processing frameworks (e.g., Apache Kafka, Apache Airflow).

- This role offers a fantastic opportunity for a data engineer to leverage their expertise in cloud computing, Python programming, SQL, and machine learning technologies like TensorFlow to drive innovation and improve data workflows within a fast-paced environment. If you are passionate about working with big data and want to make an impact on the organization's data capabilities, we encourage you to apply.