Etl Developer

We are looking for an experienced ETL Developer with strong hands-on experience in designing and building ETL pipelines using Azure Data Factory (ADF). The ideal candidate will be responsible for developing, implementing, and maintaining ETL processes to efficiently extract, transform, and load data from various sources into data warehouses, databases, or other storage systems. If youre passionate about data integration and cloud technologies, this is the perfect opportunity for you to contribute to a growing and innovative team.

• Design and Develop ETL Pipelines: Build and implement robust ETL pipelines using Azure Data Factory (ADF) for extracting, transforming, and loading data across various systems and platforms.

• Data Integration: Integrate data from a wide variety of sources such as SQL databases, APIs, flat files, cloud storage (Azure Blob Storage, Data Lake), and other enterprise systems.

• Data Transformation: Write complex data transformation logic using ADF Data Flows, SQL, and other tools to ensure data is cleansed, enriched, and transformed according to business requirements.

• Data Validation: Implement data validation steps to ensure the integrity and accuracy of data throughout the ETL process.

• Monitor and Optimize Pipelines: Continuously monitor pipeline performance, troubleshoot issues, and optimize for performance and reliability.

• Automation: Automate ETL workflows and jobs using ADF triggers, schedules, and custom error-handling mechanisms.

• Collaboration with Stakeholders: Work closely with Data Engineers, Data Analysts, and other team members to understand data requirements and deliver solutions that meet business and technical objectives.

• Documentation: Maintain clear documentation for ETL processes, pipeline architectures, and data flows for ease of understanding and troubleshooting.

• ETL Development: Proven experience in developing ETL pipelines using Azure Data Factory (ADF).

• SQL Data Transformation: Strong experience in writing SQL queries for data manipulation and transformation. Knowledge of Data Flows and Mapping Data Flows in ADF is highly desirable.

• Cloud Technologies: Experience working with Azure cloud services, including Azure Blob Storage, Azure SQL Database, Azure Data Lake, and other related services.

• Data Integration: Expertise in integrating various data sources (e.g., relational databases, flat files, APIs, cloud storage).

• Data Warehousing: Familiarity with designing and implementing data warehouses, data marts, or similar solutions.

• Scripting Automation: Experience with Azure Functions, PowerShell, or other scripting tools for automation and orchestration of ETL jobs.

• Problem-Solving Debugging: Strong troubleshooting skills and the ability to resolve issues with pipelines, data extraction, or transformation logic.

• Azure Synapse Analytics: Experience working with Azure Synapse Analytics or other cloud-based data warehousing solutions.

• CI/CD: Familiarity with Continuous Integration/Continuous Deployment (CI/CD) practices for deploying ADF pipelines and automating deployment workflows.

• Big Data Technologies: Knowledge of Apache Spark or other big data processing frameworks is a plus.

• Data Quality: Experience with data quality tools and techniques to ensure high data integrity and accuracy.

• Agile Methodologies: Familiarity with Agile development and working in cross-functional teams to deliver ETL solutions iteratively.