Data Engineer

Job Description: Data Engineer

Location: Remote

Experience: 5-6 years

Annual Salary: ₹10 lacs to ₹12 lacs

Key Responsibilities:

• Design, build, automate, and maintain sophisticated ETL programs that extract, convert, and load data.
• Gather and process data into a unified format and load it into Data Lake or Data Warehouse.
• Perform system testing to ensure data accuracy, troubleshoot defects, and make necessary coding adjustments.
• Collaborate with development teams to design storage solutions and optimize performance.
• Work on real-time and batch data processing using tools like Apache Spark, EventHub, and Kafka.
• Utilize cloud services such as Azure and AWS to build and optimize data pipelines.
• Troubleshoot data-related performance issues and optimize queries for high efficiency.
• Ensure timely resolution of models and performance issues related to data storage, databases, and data warehousing.
• Validate data through SQL queries, ensuring it meets business requirements.
• Contribute to the continuous improvement of ETL systems and pipelines.

Key Skills:

• Cloud Platforms & Services: Azure (preferred), AWS (plus), Azure Data Factory (ADF), Databricks, Azure Data Lake (ADLS Gen2), Blob storage, Synapse Analytics.
• ETL & Data Engineering: Proficiency in ETL systems and tools such as Databricks (PySpark/Scala/SparkSQL), Kafka, and Azure Data Factory (ADF).
• Programming Languages: Python, Scala, PySpark, SQL.
• Data Processing: Real-time and batch processing using Spark, EventHub, Kafka.
• Data Storage: Data Lake, Data Lakehouse, Blob storage, SQL DB, Postgres, and Azure Synapse.
• Data Engineering Tools: Familiarity with Pandas, NumPy, UDFs on Spark, and SQL performance tuning (index management, query optimization).
• Other Skills: Strong troubleshooting abilities and understanding of data modeling concepts.

Preferred Qualifications:

• 3-5 years of hands-on experience working with ETL systems and Databricks.
• Knowledge of Snowflake is an added advantage.
• Proficient in handling large and complex data sets, data modeling, and storage techniques.
• Excellent verbal and written communication skills.
• Ability to work effectively in a collaborative, fast-paced, and iterative environment.