Data Engineer

Position Name : Sr Data Engineer

Location : Gurgaon

Work Model : Work from Office (WFO)

Notice Period : Immediate/ Early joiners (serving Notice Period)

Must Have skills:

• Azure Data factory (ADF), Azure Databricks (ADB), Azure Synapse Analytics (SQL DW), Azure Analysis Services (AAS), Azure Data Lake Storage (ADLS), Azure SQL Database (SQL DB), SQL
• Hands on experience designing and developing Data Pipelines for Data Ingestion or Transformation using Python (PySpark)/Spark SQL.
• Experience working with version control GitHub and CI/CD pipelines using Azure.
• Good to have: Power BI
• Azure Certified (DP-200 DP201 preferred)
• Having good experience on Apache Spark-based analytics platform like Azure Data bricks

Job Description:

1. Data Ingestion and Integration: You may need to ingest data from various sources such as databases, files, APIs, and streaming platforms into Azure. For example, you could use Azure Data Factory to orchestrate the extraction, transformation, and loading (ETL) of data from an on-premises SQL Server database to Azure Blob Storage.
2. Data Storage and Management: Azure provides various data storage options, and as a Data Engineer, youll need to select and manage the appropriate storage solutions for your data. For instance, you could use Azure SQL Database or Azure Cosmos DB for structured data, Azure Data Lake Storage for big data and unstructured data, or Azure Blob Storage for file storage.
3. Data Warehousing: Implementing data warehousing solutions is a common use case for Azure Data Engineers. Azure Synapse Analytics (formerly Azure SQL Data Warehouse) offers scalable and distributed analytics capabilities for handling large datasets.
4. Data Modeling and Analysis: Data Engineers may collaborate with data scientists and analysts to design and implement data models for analytical workloads. Azure Analysis Services or Azure Databricks can be used for building and deploying models to support interactive data analysis.
5. Real-time Data Processing: Azure Stream Analytics is a platform that enables real-time processing of streaming data from sources like IoT devices, social media feeds, or log files. As a Data Engineer, you may configure and optimize streaming pipelines to extract insights from live data streams.
6. Data Governance and Security: Azure Data Engineers play a crucial role in ensuring data governance and security practices are implemented. They set up data access controls, implement encryption, monitor data quality, and establish data retention policies.
7. DevOps and Automation: Automating data pipelines, monitoring data processes, and managing deployments are important tasks for Azure Data Engineers. They may use tools like Azure DevOps, Azure Monitor, and Azure Automation to achieve continuous integration and delivery.