Azure Data Engineer - Python/SQL (5-7 yrs)

About Us :

AceNet Consulting is a fast-growing global business and technology consulting firm specializing in business strategy, digital transformation, technology consulting, product development, start-up advisory and fund-raising services to our global clients across banking & financial services, healthcare, supply chain & logistics, consumer retail, manufacturing, eGovernance and other industry sectors.

We are looking for hungry, highly skilled and motivated individuals to join our dynamic team. If you're passionate about technology and thrive in a fast-paced environment, we want to hear from you.

Job Summary :

We are seeking a skilled and innovative Data Engineer to join our team. In this role, you will design and implement scalable data pipelines and architectures that empower our organization to make data-driven decisions. You'll work closely with cross-functional teams to ensure the reliability, security, and efficiency of our data systems.

Key Responsibilities :

1.Staging Layer Development :

- Create and Manage Staging Layers : Design efficient staging tables for preliminary data storage.

- Historical and Incremental Data Loads : Implement strategies for managing both historical and incremental data loads using Azure Synapse SQL pools and other relevant tools.

- Performance Optimization : Optimize data load processes to ensure minimal impact on system performance.

2. Data Quality Management :

- Data Quality Checks : Develop and implement robust data quality checks using custom scripts or data quality tools.

- Source & Master Mapping : Develop source-to-target mapping documents and master data mapping in the silver layer.

- Data Profiling : Conduct detailed data profiling and data anomaly detection.

3. Operational Data Store (ODS) Layer Development :

- Design and Implementation : Architect and impleme nt the ODS layer using Azure Synapse Analytics.

- Integration : Integrate ODS with diverse data sources ensuring real-time and batch processing capabilities.

- Refresh Strategies : Develop strategies for refreshing data within the ODS layer.

4. Gold Layer Development :

- Business-Critical Data Handling : Design the gold layer for high-value, business-critical data using Azure Synapse dedicated SQL pools.

- Aggregation & Calculation : Implement complex aggregations and calculations in the gold layer.

- Performance Tuning : Continuously monitor and tune performance for optimal query execution.

5. Architecture Implementation :

- Medallion Architecture : Design and implement data solutions adhering to the Medallion Architecture principles for reliable and scalable data lakes.

- Microsoft Fabric & Azure Synapse Integration : Integrate Microsoft Fabric services with Azure Synapse environments.

- Delta Lakehouse Implementation : Utilize Delta Lake for building scalable and fault-tolerant data lakehouses.

6. Data Engineering Solutions :

- Business Data Model Implementation : Translate business data models into robust data engineering solutions using Azure Synapse, Azure Data Factory, and other Azure data services.

- Workflow Orchestration : Orchestrate complex data workflows ensuring data consistency and reliability.

7. Data Warehouse & Data Lake Creation :

- Data Warehouse Design : Design and build data warehouses leveraging Azure Synapse Analytics.

- Data Lake Construction : Construct and manage data lakes and delta lakehouses for seamless data integration and storage.

- Compliance & Security : Ensure data solutions adhere to compliance and security best practices.

Role Requirements and Qualifications :

- Bachelor's degree in Computer Science, Data Engineering, or a related field.

- Proficiency in programming languages like Python, Java, or Scala.

- Experience with data processing frameworks (e.g., Apache Spark, Kafka).

- Strong knowledge of database systems (e.g., PostgreSQL, MySQL, MongoDB).

- Familiarity with cloud platforms (AWS, Azure, or GCP) and their data services.

- Excellent problem-solving and communication skills.

Why Join Us :

- Opportunities to work on transformative projects, cutting-edge technology and innovative solutions with leading global firms across industry sectors.

- Continuous investment in employee growth and professional development with a strong focus on up & re-skilling.

- Competitive compensation & benefits, ESOPs and international assignments.

- Supportive environment with healthy work-life balance and a focus on employee well-being.

- Open culture that values diverse perspectives, encourages transparent communication and rewards contributions.