Happiest Minds Technologies Data Engineer Interview Questions and Answers

Q1. More on Technical area
Add your answer

Q1. More on Technical area
Add your answer

Q1. Technical + Behaviour
Add your answer

Q1. Technical + Behaviour
Add your answer

Q1. Expectation and Genaral
Add your answer

Q1. Diff between Coalesce and repatriation
Add your answer

Q1. SCD questions. Iceberg questions
Add your answer
Q2. Basic python programing, pyspark architechture.
Add your answer

Q1. Data Structures
Add your answer
Q2. Python Lambda Function
Add your answer

Q1. What are your Strengths
Ans. 
My strengths include strong analytical skills, attention to detail, and problem-solving abilities.
• Strong analytical skills - able to analyze complex data sets and derive meaningful insights

• Attention to detail - meticulous in ensuring data accuracy and quality

• Problem-solving abilities - adept at identifying and resolving data-related issues

• Experience with data manipulation tools like SQL, Python, and Spark
Answered by AI
Add your answer
Q2. Why you are looking for job change
Ans. 
Seeking new challenges and growth opportunities in a different environment.
• Looking for new challenges to enhance my skills and knowledge

• Seeking growth opportunities that align with my career goals

• Interested in exploring different technologies and industries

• Want to work in a more collaborative team environment

• Seeking better work-life balance or location proximity
Answered by AI
Add your answer

Just focus on the basics of pyspark.

Q1. Normalisation of database, views,stored procedures
Ans. 
Normalization is a process of organizing data in a database to reduce redundancy and improve data integrity.
• Normalization involves breaking down a table into smaller tables and defining relationships between them.

• It helps in reducing data redundancy and inconsistencies.

• Views are virtual tables that are created based on the result of a query. They can be used to simplify complex queries.

• Stored procedures are precompiled...read more
Answered by AI
Add your answer

Q1. How to create pipeline in adf?
Ans. 
To create a pipeline in ADF, you can use the Azure Data Factory UI or code-based approach.
• Use Azure Data Factory UI to visually create and manage pipelines

• Use code-based approach with JSON to define pipelines and activities

• Add activities such as data movement, data transformation, and data processing to the pipeline

• Set up triggers and schedules for the pipeline to run automatically
Answered by AI
Add your answer
Q2. Diffrent types of activities in pipelines
Ans. 
Activities in pipelines include data extraction, transformation, loading, and monitoring.
• Data extraction: Retrieving data from various sources such as databases, APIs, and files.

• Data transformation: Cleaning, filtering, and structuring data for analysis.

• Data loading: Loading processed data into a data warehouse or database.

• Monitoring: Tracking the performance and health of the pipeline to ensure data quality and reliab
Answered by AI
Add your answer
Q3. What is use of getmetadata
Ans. 
getmetadata is used to retrieve metadata information about a dataset or data source.
• getmetadata can provide information about the structure, format, and properties of the data.

• It can be used to understand the data schema, column names, data types, and any constraints or relationships.

• This information is helpful for data engineers to properly process, transform, and analyze the data.

• For example, getmetadata can be used ...read more
Answered by AI
Add your answer
Q4. Diffrent types of triggers
Ans. 
Triggers in databases are special stored procedures that are automatically executed when certain events occur.
• Types of triggers include: DML triggers (for INSERT, UPDATE, DELETE operations), DDL triggers (for CREATE, ALTER, DROP operations), and logon triggers.

• Triggers can be classified as row-level triggers (executed once for each row affected by the triggering event) or statement-level triggers (executed once for eac...read more
Answered by AI
Add your answer
Q5. Diffrence between normal cluster and job cluster in databricks
Ans. 
Normal cluster is used for interactive workloads while job cluster is used for batch processing in Databricks.
• Normal cluster is used for ad-hoc queries and exploratory data analysis.

• Job cluster is used for running scheduled jobs and batch processing tasks.

• Normal cluster is terminated after a period of inactivity, while job cluster is terminated after the job completes.

• Normal cluster is more cost-effective for short-liv...read more
Answered by AI
Add your answer
Q6. What is slowly changing dimensions
Ans. 
Slowly changing dimensions refer to data warehouse dimensions that change slowly over time.
• SCDs are used to track historical changes in data over time.

• There are three types of SCDs - Type 1, Type 2, and Type 3.

• Type 1 SCDs overwrite old data with new data, Type 2 creates new records for changes, and Type 3 maintains both old and new data in separate columns.

• Example: A customer's address changing would be a Type 2 SCD.

• Ex...read more
Answered by AI
Add your answer
Q7. Incremental load
Add your answer
Q8. With use in python
Ans. 
Use Python's 'with' statement to ensure proper resource management and exception handling.
• Use 'with' statement to automatically close files after use

• Helps in managing resources like database connections

• Ensures proper cleanup even in case of exceptions
Answered by AI
Add your answer
Q9. List vs tuple in python
Ans. 
List is mutable, tuple is immutable in Python.
• List can be modified after creation, tuple cannot be modified.

• List uses square brackets [], tuple uses parentheses ().

• Lists are used for collections of items that may need to be changed, tuples are used for fixed collections of items.

• Example: list_example = [1, 2, 3], tuple_example = (4, 5, 6)
Answered by AI
Add your answer
Q10. Datalake 1 vs datalake2
Ans. 
Datalake 1 and Datalake 2 are both storage systems for big data, but they may differ in terms of architecture, scalability, and use cases.
• Datalake 1 may use a Hadoop-based architecture while Datalake 2 may use a cloud-based architecture like AWS S3 or Azure Data Lake Storage.

• Datalake 1 may be more suitable for on-premise data storage and processing, while Datalake 2 may offer better scalability and flexibility for clou...read more
Answered by AI
Add your answer
Q11. How to read a file in databricks
Ans. 
To read a file in Databricks, you can use the Databricks File System (DBFS) or Spark APIs.
• Use dbutils.fs.ls('dbfs:/path/to/file') to list files in DBFS

• Use spark.read.format('csv').load('dbfs:/path/to/file') to read a CSV file

• Use spark.read.format('parquet').load('dbfs:/path/to/file') to read a Parquet file
Answered by AI
Add your answer
Q12. Star vs snowflake schema
Ans. 
Star schema is denormalized with one central fact table surrounded by dimension tables, while snowflake schema is normalized with multiple related dimension tables.
• Star schema is easier to understand and query due to denormalization.

• Snowflake schema saves storage space by normalizing data.

• Star schema is better for data warehousing and OLAP applications.

• Snowflake schema is better for OLTP systems with complex relationsh
Answered by AI
Add your answer
Q13. Repartition vs coalesece
Ans. 
repartition increases partitions while coalesce decreases partitions in Spark
• repartition shuffles data and can be used for increasing partitions for parallelism

• coalesce reduces partitions without shuffling data, useful for reducing overhead

• repartition is more expensive than coalesce as it involves data movement

• example: df.repartition(10) vs df.coalesce(5)
Answered by AI
Add your answer
Q14. Parquet file uses
Ans. 
Parquet file format is a columnar storage format used for efficient data storage and processing.
• Parquet files store data in a columnar format, which allows for efficient querying and processing of specific columns without reading the entire file.

• It supports complex nested data structures like arrays and maps.

• Parquet files are highly compressed, reducing storage space and improving query performance.

• It is commonly used ...read more
Answered by AI
Add your answer