Data Engineering -SE

Hi «Candidate_NameÂ,
We're seeking a seasoned Lead Data Engineer to architect and implement robust data pipelines. Strong PySpark skills are a must. If you're passionate about data and want to lead our data initiatives, reply with your updated resume.
Role: Lead Data Engineer
Location: Hyderabad
Job Type: Full Time
Work Mode: Hybrid

Roles & Responsibilities
● Lead the design, development, and maintenance of data pipelines and ETL processes Architect and implement scalable data solutions using Databricks and AWS
● Optimize data storage and retrieval systems using Rockset, Clickhouse, and CrateDB
● Develop and maintain data APIs using FastAPI
● Orchestrate and automate data workflows using Airflow
● Collaborate with data scientists and analysts to support their data needs
● Ensure data quality, security, and compliance across all data systems
● Mentor junior data engineers and promote best practices in data engineering
● Evaluate and implement new data technologies to improve the data infrastructure
● Participate in cross-functional projects and provide technical leadership
● Manage and optimize data storage solutions using AWS S3, implementing best practices for data lakes and data warehouses
● Implement and manage Databricks Unity Catalog for centralized data governance and access control across the organization
Qualifications
● Bachelor's or Master's degree in Computer Science, Engineering, or related field
● 3+ years of experience in data engineering
● Strong proficiency in Python, PySpark, and SQL
● Extensive experience with Databricks and AWS cloud services
● Hands-on experience with Airflow for workflow orchestration
● Familiarity with FastAPI for building high-performance APIs
● Experience with columnar databases like Rockset, Clickhouse, and CrateDB
● Solid understanding of data modeling, data warehousing, and ETL processes
● Experience with version control systems (e.g., Git) and CI/CD pipelines
● Excellent problem-solving skills and ability to work in a fast-paced environment
● Strong communication skills and ability to work effectively in cross-functional teams
● Knowledge of data governance, security, and compliance best practices
● Proficiency in designing and implementing data lake architectures using AWS S3
● Experience with Databricks Unity Catalog or similar data governance and metadata management tools
Preferred Qualifications

1. Experience with real-time data processing and streaming technologies
2. Familiarity with machine learning workflows and MLOps
3. Certifications in Databricks, AWS
4. Experience implementing data mesh or data fabric architectures
5. Knowledge of data lineage and metadata management best practices

Tech Stack

Databricks, Python, PySpark, SQL, Airflow, FastAPI, AWS (S3, IAM, ECR, Lambda), Rockset, Clickhouse, CrateDB

Best Regards,
Sai Kiran Ibrahimpatnam
Deputy Manager – TA
https://www.fissionlabs.com/careers