L&T Technology Services Data Engineer Interview Questions and Answers for Experienced

Q1. What Volume of data have you handled in your POCs ?
Ans. 
I have handled terabytes of data in my POCs, including data from various sources and formats.
• Handled terabytes of data in POCs

• Worked with data from various sources and formats

• Used tools like Hadoop, Spark, and SQL for data processing
Answered by AI
View 1 more amswer
Q2. When will you decide to use repartition and coalesce?
Ans. 
Repartition is used for increasing partitions for parallelism, while coalesce is used for decreasing partitions to reduce shuffling.
• Repartition is used when there is a need for more partitions to increase parallelism.

• Coalesce is used when there are too many partitions and need to reduce them to avoid shuffling.

• Example: Repartition can be used before a join operation to evenly distribute data across partitions for bette...read more
Answered by AI
Add your answer
Q3. How will you design/configure a cluster if you have given 10 petabytes of data.
Ans. 
Designing/configuring a cluster for 10 petabytes of data involves considerations for storage capacity, processing power, network bandwidth, and fault tolerance.
• Consider using a distributed file system like HDFS or object storage like Amazon S3 to store and manage the large volume of data.

• Implement a scalable processing framework like Apache Spark or Hadoop to efficiently process and analyze the data in parallel.

• Utilize...read more
Answered by AI
Add your answer

Q1. Aws sevices you have worked on
Ans. 
I have worked on various AWS services including S3, EC2, Lambda, Glue, and Redshift.
• S3 - Used for storing and retrieving data

• EC2 - Used for running virtual servers

• Lambda - Used for serverless computing

• Glue - Used for ETL (Extract, Transform, Load) processes

• Redshift - Used for data warehousing and analytics
Answered by AI
Add your answer
Q2. Question on python list and dictionary
Add your answer

Q1. Questions on project and skillset
Add your answer

Q1. Project architechture
Add your answer
Q2. What is adf triggers
Ans. 
ADF triggers are used in Azure Data Factory to schedule and orchestrate data pipelines.
• ADF triggers enable the automation of data movement and data transformation activities.

• Triggers can be scheduled to run at specific times or based on event-based triggers.

• They can be used to start or stop pipelines, and can be configured with parameters and dependencies.

• Examples of triggers include time-based schedules, event-based t...read more
Answered by AI
Add your answer
Q3. What is IR, and difference between dataset and linked service
Ans. 
IR stands for Integration Runtime. Dataset is a representation of data, while linked service is a connection to the data source.
• IR is a compute infrastructure used to provide data integration capabilities

• Dataset is a structured representation of data used in data engineering tasks

• Linked service is a connection to a data source, providing access to the data

• IR enables data movement and transformation between different da...read more
Answered by AI
Add your answer
Q4. Optimization techniques in spark
Ans. 
Optimization techniques in Spark
• Partitioning data to optimize data locality

• Caching frequently accessed data

• Using broadcast variables for small data sets

• Using appropriate data structures and algorithms

• Avoiding unnecessary shuffling of data
Answered by AI
Add your answer

Q1. Tell. me about youself
Add your answer
Q2. Sql question , pyspark questions
Add your answer

Q1. Explain Spark Architecture
Ans. 
Spark Architecture is a distributed computing framework that provides high-level APIs for various languages.
• Spark Architecture consists of a cluster manager, worker nodes, and a driver program.

• It uses Resilient Distributed Datasets (RDDs) for fault-tolerant distributed data processing.

• Spark supports multiple programming languages such as Scala, Java, Python, and R.

• It includes components like Spark Core, Spark SQL, Spar...read more
Answered by AI
Add your answer
Q2. SQL question on Joins
Add your answer

Pyspark and SQL coding questions

Q1. Previous project worked on? ETL projects? SQL proficiency
Ans. 
Worked on ETL projects involving data extraction, transformation, and loading using SQL proficiency.
• Developed ETL pipelines to extract data from various sources such as databases, APIs, and flat files

• Transformed the extracted data using SQL queries to meet business requirements

• Loaded the transformed data into a data warehouse or other target systems

• Optimized SQL queries for performance and efficiency

• Collaborated with c...read more
Answered by AI
Add your answer
Q2. ETL projects in previous company
Add your answer

Q1. Salary expectations? Working hours?
Add your answer