HCLTech Big Data Engineer Interview Questions and Answers for Experienced

Q1. Coalesce vs repartition
Ans. 
Coalesce is used to reduce the number of partitions in a DataFrame, while repartition is used to increase the number of partitions.
• Coalesce is a narrow transformation that can only decrease the number of partitions without shuffling data.

• Repartition is a wide transformation that can both increase or decrease the number of partitions and involves shuffling data across the cluster.

• Coalesce is more efficient for reducing ...read more
Answered by AI
Add your answer

Rank vs dense rank quetions ctes

Python data structure

Q1. Basic Questions of Scala Functional Programming concepts.
Add your answer
Q2. Spark internal working and optimization techniques
Ans. 
Spark internal working and optimization techniques
• Spark uses Directed Acyclic Graph (DAG) for optimizing workflows

• Lazy evaluation helps in optimizing transformations by combining them into a single stage

• Caching and persistence of intermediate results can improve performance

• Partitioning data can help in parallel processing and reducing shuffle operations
Answered by AI
Add your answer

Q1. What is speculative execution in Hadoop?
Ans. 
Speculative execution in Hadoop is a feature that allows the framework to launch duplicate tasks for a job, with the goal of completing the job faster.
• Speculative execution is used when a task is taking longer to complete than expected.

• Hadoop identifies slow-running tasks and launches duplicate tasks on other nodes.

• The first task to complete is used, while the others are killed to avoid duplication of results.

• This help...read more
Answered by AI
Add your answer
Q2. Spark Architecture
Add your answer
Q3. Difference in list , tuple,sets
Ans. 
Lists are mutable ordered collections, tuples are immutable ordered collections, and sets are mutable unordered collections.
• Lists are mutable and ordered, allowing for duplicate elements. Example: [1, 2, 3, 3]

• Tuples are immutable and ordered, allowing for duplicate elements. Example: (1, 2, 3, 3)

• Sets are mutable and unordered, not allowing for duplicate elements. Example: {1, 2, 3}
Answered by AI
Add your answer
Q4. Difference in rank, dense rank in sql
Ans. 
Rank assigns a unique rank to each distinct row, while dense rank assigns consecutive ranks to rows with the same values.
• Rank function assigns unique ranks to each distinct row in the result set

• Dense rank function assigns consecutive ranks to rows with the same values

• Rank function leaves gaps in the ranking sequence if there are ties, while dense rank does not
Answered by AI
Add your answer
Q5. Add data into a partitioned hive table
Ans. 
To add data into a partitioned hive table, you can use the INSERT INTO statement with the PARTITION clause.
• Use INSERT INTO statement to add data into the table.

• Specify the partition column values using the PARTITION clause.

• Example: INSERT INTO table_name PARTITION (partition_column=value) VALUES (data);
Answered by AI
Add your answer

WAP to add index wise elements of a list . A=[1,2,3] , B=[4,5,7] C should be [5,7,10]

Q1. Aws sevices you have worked on
Ans. 
I have worked on various AWS services including S3, EC2, Lambda, Glue, and Redshift.
• S3 - Used for storing and retrieving data

• EC2 - Used for running virtual servers

• Lambda - Used for serverless computing

• Glue - Used for ETL (Extract, Transform, Load) processes

• Redshift - Used for data warehousing and analytics
Answered by AI
Add your answer
Q2. Question on python list and dictionary
Add your answer

Q1. Questions on project and skillset
Add your answer

Q1. Machine learning related questions and the theory of its operation
Add your answer

Q1. Tell. me about youself
Add your answer
Q2. Sql question , pyspark questions
Add your answer

Q1. Self introduced and my skills knowledge
Add your answer
Q2. My name name is Aasif and I have done my graduation b.com. I have good knowledge English and Hindi in oral and writing. I have done excel, ms word, PowerPoint Data entry.
Add your answer

Q1. Study and skills
Add your answer
Q2. Like my work speciality’s
Add your answer
Q3. Study speciality’s
Add your answer