Infosys Azure Data Engineer Interview Questions and Answers

SCD type 1 overwrites existing data, while SCD type 2 maintains historical data

• SCD type 1 updates existing records with new information, losing historical data

• SCD type 2 creates new records for changes, preserving historical data with effective start and end dates

• In Databricks, SCD type 1 can be implemented using overwrite function, while SCD type 2 can be implemented using merge function

Parameters can be passed from Azure Data Factory (ADF) to Azure Data Factory (ADB) using linked services and datasets.

• Create a linked service in ADF to connect to ADB

• Define parameters in ADF pipeline and pass them to ADB activities

• Use dynamic content expressions to pass parameters between ADF and ADB

• Example: Pass a parameter for a SQL query in ADB from ADF pipeline

Challenges in production deployment include scalability, data consistency, and monitoring.

• Ensuring scalability to handle increasing data volumes and user loads

• Maintaining data consistency across different databases and systems

• Implementing effective monitoring and alerting to quickly identify and resolve issues

Developed a data pipeline to ingest, process, and analyze customer feedback data for a retail company.

• Used Azure Data Factory to orchestrate data flow

• Implemented Azure Databricks for data processing and analysis

• Utilized Azure Synapse Analytics for data warehousing

• Generated visualizations using Power BI for insights

• Implemented machine learning models for sentiment analysis