Flatworld Solutions - Data Engineer - ETL/Synapse Analytics (6-10 yrs)

Role : ETL Developer / Azure Synapse Data Engineer.

Location : Bengaluru - Hennur location.

Experience : 6+ years.

Job Description :

We are seeking a skilled Data Engineer to design, develop, and maintain data pipelines that automate the flow of data from NetSuite into Azure Synapse, enabling robust reporting and analytics within Power BI.

This role will be crucial in building and optimizing our data infrastructure, ensuring data accuracy, and improving the efficiency of our data processes.

Responsibilities :

- Develop and maintain data pipelines : Design, implement, and maintain automated data pipelines from NetSuite to Azure Synapse using Azure Data Factory, leveraging the Suite Analytics ODBC Connector and custom API solutions.

- Data Modeling and Transformation : Design and implement efficient data transformations within Azure Synapse to prepare data for consumption in Power BI. This includes developing staging, bronze, and final data models.

- ETL Process Development : Build and maintain robust Extract, Transform, Load (ETL) processes using Azure Data Factory, ensuring data quality and integrity throughout the pipeline.

- Performance Optimization : Optimize data pipeline performance by identifying and addressing bottlenecks, improving query efficiency, and implementing best practices for data storage and retrieval in Azure Synapse.

- Collaboration and Communication : Collaborate with business stakeholders, data analysts, and other engineers to understand data requirements and ensure the delivery of accurate and reliable data.

- Monitoring and Troubleshooting : Monitor data pipeline health, proactively identify and resolve issues, and implement monitoring and alerting mechanisms.

- Documentation : Maintain comprehensive documentation of data pipelines, data models, and ETL processes.

- Celigo Integration : Work with existing Celigo integrations for certain data processes, ensuring seamless integration with the overall data pipeline.

Example Tasks :

- Develop an Azure Data Factory pipeline that extracts data from NetSuite using the Suite Analytics ODBC connector. Example : Extracting daily sales order data.

- Implement data transformations within Azure Synapse to cleanse and prepare the NetSuite data. Example : Converting date formats, handling null values, and aggregating data.

- Load transformed data into the appropriate layer (staging, bronze, final) within Azure Synapse. Example : Loading aggregated sales data into a dimensional model in the final layer.

- Optimize the performance of a data pipeline by implementing partitioning or indexing strategies in Azure Synapse. Example : Partitioning a large sales table by date to improve query performance.

- Work with the Celigo team to ensure data consistency between Celigo managed processes and custom-built pipelines. Example : Ensuring data lineage between a Celigo integration and an Azure Data Factory pipeline.

Qualifications :

- Bachelor's degree in Computer Science, Engineering, or a related field.

- Proven experience in developing and maintaining data pipelines using Azure Data Factory.

- Strong understanding of Azure Synapse architecture and functionalities, specifically dedicated SQL pools (formerly Azure SQL DW).

- Experience working with NetSuite and the Suite Analytics ODBC Connector.

- Proficiency in SQL and data manipulation techniques.

- Familiarity with Celigo or other iPaaS solutions is a plus.

Bonus Points :

- Experience with Azure DevOps for CI/CD.

- Experience with data governance and data quality best practices.

- Familiarity with Power BI report development.