CAPG Data Engineer Interview Questions and Answers

Q1. I was asked to write code and some scenario based question on Big Data etc
Add your answer

2nd with VP which is easy but he seems not okay with my ECTC

Q1. General Data Warehousing questions like explain your pipeline, how you implemented scd2?
Add your answer
Q2. SQL questions like increment top 5th salary by 10k, last day of month, etc.
Add your answer

Q1. Project question like how connected to s3 bucket through pyspark,HDFS basic commands likhe copy etc ,how and where created hive table (if used in your project).
Add your answer
Q2. Share variables in pyspark like broadcast and accumulator..(i will suggest go through pyspark official documentation once)
Add your answer
Q3. Sql joins- , how to read file in pyspark, job scheduling means resource allocation within and across spark..(ans- can go through job scheduling of pyspark documentaion )
Add your answer

Basic aptitude test asking questions very basic

Coding questions based on dsa

Q1. Basic hr question
Add your answer

Q1. Mostly on cloud tools
Add your answer

Q1. What is the difference between supervised and unsupervised learning?
Ans. 
Supervised learning uses labeled data to train the model, while unsupervised learning uses unlabeled data.
• Supervised learning requires a target variable to be predicted, while unsupervised learning does not.

• In supervised learning, the model learns from labeled training data, whereas in unsupervised learning, the model finds patterns in unlabeled data.

• Examples of supervised learning include regression and classification...read more
Answered by AI
Add your answer
Q2. Elaborate concepts of Object Oriented Programming in Python.
Ans. 
Object Oriented Programming in Python focuses on creating classes and objects to organize code and data.
• Python supports classes, objects, inheritance, polymorphism, and encapsulation.

• Classes are blueprints for creating objects, which are instances of classes.

• Inheritance allows a class to inherit attributes and methods from another class.

• Polymorphism enables objects to be treated as instances of their parent class.

• Encap...read more
Answered by AI
Add your answer
Q3. How to find delta between two tables in SQL?
Ans. 
To find delta between two tables in SQL, use the EXCEPT or MINUS operator.
• Use the EXCEPT operator in SQL to return rows from the first table that do not exist in the second table.

• Use the MINUS operator in SQL to return distinct rows from the first table that do not exist in the second table.
Answered by AI
View 1 more answer
Q4. Illustrate exception handling in python.
Ans. 
Exception handling in Python allows for graceful handling of errors and preventing program crashes.
• Use try-except blocks to catch and handle exceptions.

• Multiple except blocks can be used to handle different types of exceptions.

• Finally block can be used to execute code regardless of whether an exception was raised or not.

• Custom exceptions can be defined by creating a new class that inherits from the built-in Exception c
Answered by AI
Add your answer
Q5. Give an example of decorators in Python?
Ans. 
Decorators in Python are functions that modify the behavior of other functions.
• Decorators are defined using the @decorator_name syntax before the function definition.

• They can be used for logging, timing, authentication, etc.

• Example: @staticmethod decorator in Python makes a method static.
Answered by AI
Add your answer

Q1. What are examples of iaas,paas,saas
Ans. 
Examples of IaaS, PaaS, and SaaS include AWS (IaaS), Google App Engine (PaaS), and Salesforce (SaaS).
• IaaS - Infrastructure as a Service: AWS, Microsoft Azure, Google Cloud Platform

• PaaS - Platform as a Service: Google App Engine, Heroku, Microsoft Azure App Service

• SaaS - Software as a Service: Salesforce, Google Workspace, Microsoft Office 365
Answered by AI
Add your answer
Q2. Difference between Adf and ADB
Ans. 
ADF stands for Azure Data Factory, a cloud-based data integration service. ADB stands for Azure Databricks, an Apache Spark-based analytics platform.
• ADF is used for data integration and orchestration, while ADB is used for big data analytics and machine learning.

• ADF provides a visual interface for building data pipelines, while ADB offers collaborative notebooks for data exploration and analysis.

• ADF supports various da...read more
Answered by AI
Add your answer

Q1. Architecture of Spark
Ans. 
Spark is a distributed computing system that provides an interface for programming clusters with implicit data parallelism.
• Spark is built on the concept of Resilient Distributed Datasets (RDDs), which are fault-tolerant collections of objects.

• It supports various programming languages such as Scala, Java, Python, and R.

• Spark provides high-level APIs for distributed data processing, including transformations and actions.

• ...read more
Answered by AI
Add your answer
Q2. What is Lazy evaluation in spark
Ans. 
Lazy evaluation is a strategy used by Spark to delay the execution of transformations until an action is called.
• Lazy evaluation improves performance by optimizing the execution plan

• Transformations in Spark are not executed immediately, but rather recorded as a lineage graph

• Actions trigger the execution of the transformations and produce a result

• Lazy evaluation allows Spark to optimize the execution plan by combining an...read more
Answered by AI
Add your answer
Q3. Difference between Left join and inner join
Ans. 
Left join returns all records from the left table and the matching records from the right table.
• Inner join returns only the matching records from both tables.

• Left join includes all records from the left table, even if there are no matches in the right table.

• Inner join excludes the non-matching records from both tables.

• Left join is used to retrieve all records from one table and the matching records from another table.

• I...read more
Answered by AI
Add your answer