eClerx Data Engineer Interview Questions and Answers

Q1. Write code in regular expression to remove the special characters
Ans. 
Use regular expression to remove special characters from a string
• Use the regex pattern [^a-zA-Z0-9\s] to match any character that is not a letter, digit, or whitespace

• Use the replace() function in your programming language to replace the matched special characters with an empty string

• Example: input string 'Hello! How are you?' will become 'Hello How are you' after removing special characters
Answered by AI
Add your answer
Q2. Questions on resume
Add your answer

Q1. Spark Architecture
Add your answer
Q2. Cache vs persist, lazy evaluation
Add your answer

Q1. What is Databricks, Internals of it, Optimization technique
Ans. 
Databricks is a unified analytics platform that provides a collaborative environment for data scientists, engineers, and analysts.
• Databricks is built on top of Apache Spark, providing a unified platform for data engineering, data science, and business analytics.

• Internals of Databricks include a cluster manager, job scheduler, and workspace for collaboration.

• Optimization techniques in Databricks include query optimizati...read more
Answered by AI
Add your answer
Q2. SQL questions of joins,group by
Add your answer

Q1. Scenario based azure data factory questions
Add your answer
Q2. Project structres, pyspark dataframes related
Add your answer

General question around data engineering

Q1. Different apache tech and tools
Add your answer
Q2. What is snowflake
Add your answer
Q3. Coding questions on python
Add your answer

Q1. About location preference
Add your answer
Q2. Behavioural questions
Add your answer

Q1. What Volume of data have you handled in your POCs ?
Ans. 
I have handled terabytes of data in my POCs, including data from various sources and formats.
• Handled terabytes of data in POCs

• Worked with data from various sources and formats

• Used tools like Hadoop, Spark, and SQL for data processing
Answered by AI
View 1 more amswer
Q2. When will you decide to use repartition and coalesce?
Ans. 
Repartition is used for increasing partitions for parallelism, while coalesce is used for decreasing partitions to reduce shuffling.
• Repartition is used when there is a need for more partitions to increase parallelism.

• Coalesce is used when there are too many partitions and need to reduce them to avoid shuffling.

• Example: Repartition can be used before a join operation to evenly distribute data across partitions for bette...read more
Answered by AI
Add your answer
Q3. How will you design/configure a cluster if you have given 10 petabytes of data.
Ans. 
Designing/configuring a cluster for 10 petabytes of data involves considerations for storage capacity, processing power, network bandwidth, and fault tolerance.
• Consider using a distributed file system like HDFS or object storage like Amazon S3 to store and manage the large volume of data.

• Implement a scalable processing framework like Apache Spark or Hadoop to efficiently process and analyze the data in parallel.

• Utilize...read more
Answered by AI
Add your answer

Q1. Python list comprehension
Add your answer
Q2. SQL- product with second highest total price
Ans. 
Use SQL query to find the product with the second highest total price.
• Use the ORDER BY clause to sort the products by total price in descending order

• Use the LIMIT clause to select the second row after sorting
Answered by AI
Add your answer

Its mandatory test even for experience people

Q1. Related to technology
Add your answer

Q1. Very good discussion towards work culture, salary and all
Add your answer

Q1. Data Structures
Add your answer
Q2. Python Lambda Function
Add your answer

Q1. What are your Strengths
Ans. 
My strengths include strong analytical skills, attention to detail, and problem-solving abilities.
• Strong analytical skills - able to analyze complex data sets and derive meaningful insights

• Attention to detail - meticulous in ensuring data accuracy and quality

• Problem-solving abilities - adept at identifying and resolving data-related issues

• Experience with data manipulation tools like SQL, Python, and Spark
Answered by AI
Add your answer
Q2. Why you are looking for job change
Ans. 
Seeking new challenges and growth opportunities in a different environment.
• Looking for new challenges to enhance my skills and knowledge

• Seeking growth opportunities that align with my career goals

• Interested in exploring different technologies and industries

• Want to work in a more collaborative team environment

• Seeking better work-life balance or location proximity
Answered by AI
Add your answer