HCLTech Data Engineer Interview Questions, Process, and Tips

3 SQL question -2 window, join and cte

Q1. How to connect to adls gen 2 from Databricks
Ans. 
To connect to ADLS Gen2 from Databricks, you can use the Azure Blob Storage API.
• Use the Azure Blob Storage API to connect to ADLS Gen2 from Databricks

• Provide the storage account name and key for authentication

• Use the storage account name as the filesystem

• Example: spark.conf.set('fs.azure.account.key..blob.core.windows.net', '')
Answered by AI
Add your answer
Q2. Explain activities used in your pipeline
Ans. 
Activities in the pipeline include data extraction, transformation, loading, and monitoring.
• Data extraction: Retrieving data from various sources such as databases, APIs, and files.

• Data transformation: Cleaning, filtering, and structuring the data for analysis.

• Data loading: Storing the processed data into a data warehouse or database.

• Monitoring: Tracking the pipeline performance, data quality, and handling errors.
Answered by AI
Add your answer
Q3. Diff b/w adls gen1 and gen2
Ans. 
ADLS Gen2 offers better performance, security, and scalability compared to Gen1.
• ADLS Gen2 supports hierarchical file system, while Gen1 does not

• ADLS Gen2 integrates with Azure Blob Storage, providing a unified data lake solution

• ADLS Gen2 offers better performance and scalability compared to Gen1

• ADLS Gen2 provides better security features such as Azure Data Lake Storage firewall and Azure Active Directory integration
Answered by AI
Add your answer
Q4. Did you work on AKV
Ans. 
Yes, I have worked on Azure Key Vault (AKV) for securely storing and managing sensitive information.
• Implemented AKV to securely store and manage sensitive information such as passwords, certificates, and keys

• Utilized AKV to encrypt data at rest and in transit

• Integrated AKV with applications for secure access to secrets
Answered by AI
Add your answer

Q1. Salary discussion
Add your answer
Q2. Roles and responsibilties
Ans. 
Data Engineers are responsible for designing, constructing, installing, and maintaining data management systems.
• Designing and implementing data pipelines

• Building and maintaining data warehouses

• Ensuring data quality and integrity

• Collaborating with data scientists and analysts

• Optimizing data processes for performance and scalability
Answered by AI
Add your answer

Q1. Sql commands group by and order by
Add your answer

Q1. Cicd pipeline and kafka
Add your answer

Q1. Reparitition and pipelines
Add your answer

Q1. Joins operations
Add your answer

Q1. Self Introduction
Add your answer
Q2. Coalesce and Repartition
Add your answer

Q1. General Data Warehousing questions like explain your pipeline, how you implemented scd2?
Add your answer
Q2. SQL questions like increment top 5th salary by 10k, last day of month, etc.
Add your answer