Senior Data Engineer - Data Pipeline & Governance (5-7 yrs)

Job Title : Senior Data Engineer (Lakehouse Migration).

Experience Level : 5+ years of relevant experience.

Location: Bangalore / Hyderabad/ Pune/ Chennai.

Job Responsibilities :

- Design, build, and maintain scalable and efficient data pipelines in Databricks following the Medallion architecture (bronze, silver, gold layers) using Delta Lake.

- Implement complex data workflows using Databricks workflows and Unity Catalog for data governance.

- Build and manage data ingestion processes from diverse sources into AWS S3, using tools such as AWS Lambda, AWS Glue, RDS, Postgres, and ensure secure access with AWS IAM.

- Develop and optimize data pipelines within Snowflake and write complex SQL queries for data transformation and analysis.

- Utilize PySpark to process large datasets in distributed environments.

- Architect, design, and maintain Big Data/Hadoop ecosystem solutions and leverage workflow orchestration tools like Airflow for job scheduling and monitoring.

- Create and optimize Near-Real Time and Batch Data Pipelines to support high-volume data integration and processing.

- Work with diverse file formats such as Parquet, Avro, CSV, Excel, JDBC, and integrate them into the data architecture.

- Collaborate with other data engineers, data scientists, and stakeholders to implement data solutions that align with business objectives.

- Focus on performance optimization for Spark, SQL, and Python to ensure efficient data processing and query execution.

- Strong skills in troubleshooting and issue tracing within complex data pipelines and distributed systems.

- Exhibit strong team spirit and collaboration with technical and non-technical team members.

Required Skills & Qualifications :

- Extensive hands-on experience with Databricks (Delta Lake, Workflow, Unity Catalog) and knowledge of Medallion architecture.

- Strong proficiency in AWS services (S3, Lambda, Glue, RDS, Postgres, IAM).

- Advanced experience with Snowflake and SQL for data warehousing and query performance optimization.

- Advanced hands-on experience with PySpark and the Big Data ecosystem.

- Proficiency in data workflow orchestration tools such as Apache Airflow.

- Proficiency in working with varied file formats including Parquet, Avro, CSV, Excel, JDBC.

- Demonstrated ability to design and optimize Near-Real Time and Batch Data Pipelines.

- Strong understanding of data pipeline performance optimization (Spark, SQL, Python).

- Excellent problem-solving and analytical skills.

- Strong communication skills and ability to work effectively in a team environment.

Preferred Skills (Optional) :

- Experience with Scala Spark.

- Proven track record of creating framework utilities and reusable components for enhancing development efficiency.

- Knowledge of Agile methodologies and tools like Jira.

- Experience with Git, Bitbucket or similar version control tools.