Niveus Solution Data Engineer Interview Questions and Answers

Q1. What is Informatica tool
Ans. 
Informatica is a data integration tool used for ETL (Extract, Transform, Load) processes in data engineering.
• Informatica is used for extracting data from various sources like databases, flat files, etc.

• It can transform the data according to business rules and load it into a target data warehouse or database.

• Informatica provides a visual interface for designing ETL workflows and monitoring data integration processes.

• It ...read more
Answered by AI
Add your answer
Q2. What is Datastage tool
Ans. 
Datastage is an ETL tool used for extracting, transforming, and loading data from various sources to a target destination.
• Datastage is part of the IBM Information Server suite.

• It provides a graphical interface to design and run data integration jobs.

• Datastage supports parallel processing for high performance.

• It can connect to a variety of data sources such as databases, flat files, and web services.

• Datastage jobs can b...read more
Answered by AI
Add your answer

Q1. Project Worked on and brief overview
Add your answer
Q2. Technologies worked on
Ans. 
I have worked on a variety of technologies including Python, SQL, Hadoop, Spark, and AWS.
• Python

• SQL

• Hadoop

• Spark

• AWS
Answered by AI
Add your answer
Q3. Rate in skills out of 5
Ans. 
Rate in skills out of 5
• Programming: 5

• Data Analysis: 4

• Database Management: 4

• Machine Learning: 3
Answered by AI
Add your answer

Q1. What transformation done in project
Add your answer
Q2. Explain explode function
Ans. 
Explode function is used to split a string into an array of strings based on a delimiter.
• The explode function takes in a delimiter and a string as input

• It splits the string into an array of strings based on the delimiter

• For example, explode(',', 'apple,banana,orange') would return ['apple', 'banana', 'orange']
Answered by AI
Add your answer
Q3. Python question to convert string to dictionary
Add your answer
Q4. Spark coding question to flatten a Json
Add your answer
Q5. Persist, cache and unpersist in pyspark
Add your answer

General Aptitude test

Q1. On SQL functions
Add your answer
Q2. Python coding questions
Add your answer

Q1. Databricks compute and config
Add your answer
Q2. Dremio optimization process and.migration
Add your answer

Q1. Data Engineering and Data modelling
Add your answer

Q1. People management and salary
Add your answer

Q1. Project history
Add your answer
Q2. Tech stack used
Ans. 
Our tech stack includes Python, SQL, Apache Spark, Hadoop, AWS, and Docker.
• Python is used for data processing and analysis

• SQL is used for querying databases

• Apache Spark is used for big data processing

• Hadoop is used for distributed storage and processing

• AWS is used for cloud infrastructure

• Docker is used for containerization
Answered by AI
Add your answer

Q1. Handled team or not?
Add your answer
Q2. Project experience
Add your answer

Q1. Spark and airflow related questions
Add your answer
Q2. AWS services and it's related questions
Add your answer

Q1. Sql joins,dense rank and rank,window function
Add your answer

Q1. Duplicate in list
Ans. 
Find duplicates in a list of elements.
• Iterate through the list and keep track of elements seen so far

• If an element is already in the seen elements, it is a duplicate

• Use a hash set to efficiently check for duplicates
Answered by AI
Add your answer
Q2. Anagram in list
Ans. 
Check if any two strings in the list are anagrams of each other
• Iterate through the list and for each string, sort the characters and check if it already exists in a hashmap

• If it exists, then the current string is an anagram of the existing string

• Return true if anagrams are found, false otherwise
Answered by AI
Add your answer