Lead Data Engineer - Azure Data Lake Job

8+ years

8+ years of professional experience in development and support
4+ years of experience in hands on work on PySpark
Areas of expertise include Big Data Technologies like Hadoop, Spark, Impala, HDFS, Hive, Sqoop, Yarn, Java, Oracle database.
Good knowledge on Cloud technologies like Azure and AWS
Worked on Azure data factory, blobs, Azure SQL, and Azure Data Lake Storage.
Worked with Data frames and Datasets in Spark for Transforming data
Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse
Migrating and recreating existing application logic and functionality in the Azure Data Lake, Data Factory, SQL Database and SQL Datawarehouse environment
A logical, analytical thinker with excellent team player skills and possesses good experience as a Data Engineer
Excellent interpersonal skills, confident and poised in interaction with individuals at all levels
Focus on following a structured approach and Quality check methodology to generate error free deliverables

• Technical Skills: Spark, Impala, Hadoop, HDFS, Hive, Sqoop, AWS, Azure, Databricks

Functional Skills: Banking and Finance, Insurance

Accountability : Takes responsibility for and ensures accuracy of own work, as well as the work and deadlines of the team.

Collaboration : Shares information within team, participates in team activities, asks questions to understand other points of view.

Agility : Demonstrates readiness for change, asking questions and determining how changes could impact own work.

Customer Focus : Identifies trends and patterns emerging from customer preferences and works towards customizing/ refining existing services to exceed customer needs and expectations.

Communication : Targets communications for the appropriate audience, clearly articulating and presenting his/her position or decision.

Drives Results : Sets realistic stretch goals for self & others to achieve and exceed defined goals/targets.

Resolves Conflict : Displays sensitivity in interactions and strives to understand others views and concerns.