Data Engineer - SQL/Python (4-6 yrs)

Job Description :

Key Responsibilities :

- Design and develop data pipelines using tools like Apache Spark and PySpark.

- Extract data from various sources (databases, APIs, files) using appropriate techniques.

- Transform and clean data to ensure data quality and consistency.

- Load data into data warehouses, data lakes, or data marts.

- Design and implement data models that are efficient, scalable, and aligned with business requirements.

- Create data pipelines that are reliable, fault-tolerant, and maintainable.

- Optimize data pipelines for performance and cost-effectiveness.

- Leverage cloud platforms (AWS, GCP, or Azure) to build scalable and cost-effective data solutions.

- Utilize cloud services like data warehouses, data lakes, and data processing services.

- Implement data security and privacy best practices in cloud environments.

- Implement data governance policies and procedures to ensure data quality and security.

- Monitor data pipelines and proactively identify and resolve issues.

- Collaborate with data analysts and data scientists to understand their data needs and provide solutions.

Required Skills and Experience :

- Strong proficiency in Python and PySpark.

- Experience with Apache Spark and its core components (Spark SQL, Spark Streaming, Spark MLlib).

- Knowledge of SQL and database technologies.

- Experience with cloud platforms (AWS, GCP, or Azure).

- Understanding of data warehousing and data lake concepts.

- Experience with data visualization tools (Tableau, Power BI).

- Strong problem-solving and analytical skills.

- Excellent communication and collaboration skills