EPAM Systems Senior Data Engineer Interview Questions, Process, and Tips

Q1. Current day to day tasks
Ans. 
My current day-to-day tasks involve designing and implementing data pipelines, optimizing data storage and retrieval, and collaborating with cross-functional teams.
• Designing and implementing data pipelines to extract, transform, and load data from various sources

• Optimizing data storage and retrieval processes for efficiency and scalability

• Collaborating with cross-functional teams to understand data requirements and del...read more
Answered by AI
Add your answer
Q2. Question on data modelling(star vs snowflake)
Add your answer
Q3. Questions on data warehousing Dimensions, Facts, CDC, SCD types
Add your answer
Q4. Questions on snowflake product and variety on objects there use case and optimizations, partition pruning.
Add your answer
Q5. SQL queries: as per snowflake documentation 1. Read json file and extract data 2. sql query with subqueries to update records 3. sql query on window functions
Add your answer
Q6. Easy Pyspark coding question.
Add your answer

Q1. Project end to end detailed explanation and follow up questions.
Add your answer
Q2. Questions on data warehousing concepts like facts & dimensions in detail
Add your answer
Q3. Question on semi structured file formats
Add your answer
Q4. Questions on snowflake product: 1. tasks 2. streams 3. time travel 4. meta data caching 5. views vs materialized views 6. how would you spin up an warehouse cluster
Add your answer
Q5. Questions on spark 1. lazy evaluation 2. handling skew 3. data partitioning 4. pandas vs spark
Add your answer
Q6. Questions on aws 1. lambda functions, synchronous vs asynchronous calls 2. dynamic frames, transformations 3. glue crawler 4. s3 storage classes
Add your answer
Q7. SQL question using cross join, python question on string
Add your answer

Q1. End to end project architecture.
Ans. 
The end-to-end project architecture involves designing and implementing the entire data pipeline from data ingestion to data visualization.
• Data ingestion: Collecting data from various sources such as databases, APIs, and files.

• Data processing: Cleaning, transforming, and aggregating the data using tools like Apache Spark or Hadoop.

• Data storage: Storing the processed data in data warehouses or data lakes like Amazon S3 ...read more
Answered by AI
Add your answer
Q2. Questions on data modelling: 1. facts and types of facts 2. dimensions and its types 3. normalization vs denormalization
Add your answer
Q3. Questions on automating spark notebooks and optimization on spark job.
Add your answer
Q4. How to migrate 1000s of tables using spark(databricks) notebooks
Ans. 
Use Spark (Databricks) notebooks to migrate 1000s of tables efficiently.
• Utilize Spark's parallel processing capabilities to handle large volumes of data

• Leverage Databricks notebooks for interactive data exploration and transformation

• Automate the migration process using scripts or workflows

• Optimize performance by tuning Spark configurations and cluster settings
Answered by AI
Add your answer
Q5. Questions on delta format
Add your answer
Q6. Sql question on joins and numbers of records for each type of join.
Add your answer

Q1. Reason for change
Ans. 
Seeking new challenges and opportunities for growth in a more dynamic environment.
• Desire for new challenges and learning opportunities

• Seeking a more dynamic work environment

• Looking to expand skill set and experience

• Interested in working on cutting-edge technologies

• Seeking better career advancement prospects
Answered by AI
Add your answer
Q2. Expectations from EPAM
Ans. 
I expect EPAM to provide challenging projects, opportunities for growth, a collaborative work environment, and support for continuous learning.
• Challenging projects that allow me to utilize my skills and knowledge

• Opportunities for professional growth and advancement within the company

• A collaborative work environment where teamwork is valued

• Support for continuous learning through training programs and resources
Answered by AI
Add your answer
Q3. Willing to relocate? when can you join
Ans. 
Yes, I am willing to relocate for the right opportunity. I can join the company within 4 weeks.
• Willing to relocate for the right opportunity

• Can join within 4 weeks

• Open to discussing relocation assistance if needed
Answered by AI
Add your answer

Big data related multiple questions around 10 Multiple choice
One sql question to find the salary details medium difficulty
One python related question

Q1. All basic to complex questions related to spark and its architecture
Add your answer
Q2. Big data and python questions and how to build new data source from sql server to data warehouse. Need to give detail tech stack and complete process information
Add your answer

Q1. What is the process for finding the missing number from a list?
Add your answer
Q2. What are some methods for optimizing Spark performance?
Add your answer

Q1. Lineage graph,rdd,python decorators,python dict problem,sql self join probelm
Add your answer
Q2. Types of transformations,no of jobs,tasks,actions
Ans. 
The question is asking about types of transformations, number of jobs, tasks, and actions in the context of a Senior Data Engineer role.
• Types of transformations: Extract, Transform, Load (ETL), MapReduce, Spark transformations, SQL transformations

• Number of jobs: Depends on the complexity and scale of the data engineering projects

• Number of tasks: Varies based on the number of data sources, data transformations, and data...read more
Answered by AI
Add your answer
Q3. Fact,dimesnion,star and snowflake schema
Add your answer
Q4. Architecture of spark,airflow,bigquery,
Ans. 
Spark is a distributed processing engine, Airflow is a workflow management system, and BigQuery is a fully managed data warehouse.
• Spark is designed for big data processing and provides in-memory computation capabilities.

• Airflow is used for orchestrating and scheduling data pipelines.

• BigQuery is a serverless data warehouse that allows for fast and scalable analytics.

• Spark can be integrated with Airflow to schedule and m...read more
Answered by AI
Add your answer
Q5. Optimisation in spark,sql,bigquery,airflow
Ans. 
Optimization techniques in Spark, SQL, BigQuery, and Airflow.
• Use partitioning and bucketing in Spark to optimize data processing.

• Optimize SQL queries by using indexes, query rewriting, and query optimization techniques.

• In BigQuery, use partitioning and clustering to improve query performance.

• Leverage Airflow's task parallelism and resource allocation to optimize workflow execution.
Answered by AI
Add your answer
Q6. Delete duplicates from table in spark and sql
Ans. 
To delete duplicates from a table in Spark and SQL, you can use the DISTINCT keyword or the dropDuplicates() function.
• In SQL, you can use the DISTINCT keyword in a SELECT statement to retrieve unique rows from a table.

• In Spark, you can use the dropDuplicates() function on a DataFrame to remove duplicate rows.

• Both methods compare all columns by default, but you can specify specific columns to consider for duplicates.

• You...read more
Answered by AI
Add your answer
Q7. Coaelesce,repartition,shuffling,CDC,acid properties,CDC types
Add your answer

Q1. Dataflow vs Dataproc, layering processing and curated environments in gcp , Data cleaning
Ans. 
Dataflow and Dataproc are both processing services in GCP, but with different approaches and use cases.
• Dataflow is a fully managed service for executing batch and streaming data processing pipelines.

• Dataproc is a managed Spark and Hadoop service for running big data processing and analytics workloads.

• Dataflow provides a serverless and auto-scaling environment, while Dataproc offers more control and flexibility.

• Dataflow...read more
Answered by AI
Add your answer

Q1. Pyspark field delimited questions with group by transformations, Sql window function question Discussion on arch of spark,bq,airflow. How to define dependency between tasks and two diff dags. star and sno...read more
Add your answer

This was final round with Client.
They ask questions based on my work in pyspark area.
Questions are like:
What kind of transformations you used.
Broadcast join internals.
Spark internal joins
Spark catalyst optimizer.What are the joins happens in catalyst optimizer.
Window function question: 3rd highest salary of an emp
Discussion on airflow arch and how to deploy a airflow dag in gcp.
Discussion on BQ on what kind of work i have done till now.