i
Cynosure Corporate Solutions
13 Cynosure Corporate Solutions Jobs
Lead Data Engineer - DataLake/Data Warehousing (8-12 yrs)
Cynosure Corporate Solutions
posted 12hr ago
Flexible timing
Key skills for the job
Key responsibilities :
- Working with clients to understand their data.
- Based on the understanding you will be building the data structures and pipelines.
- You will be working on the application from end to end collaborating with UI and other development teams.
- You will be responsible for building the data pipelines to migrate and load the data into the HDFSeither on-prem or in the cloud.
- Developing Data ingestion/process/integration pipelines effectively.
- Creating Hive data structures, metadata and loading the data into data lakes / Big Data warehouse environments.
- Optimized(Performance tuning) many data pipelines effectively to minimize cost.
- Code versioning control and git repository is up to date.
- You will be responsible for building and maintaining CI/CD of the data pipelines.
- You will be managing the unit testing of all data pipelines
Skills & Experience :
- Bachelor's degree in computer science or related field.
- Minimum of 5+years working experience with Spark, Hadoop eco systems.
- Minimum of 4+years working experience on designing data streaming pipelines.
- Should be an expertin either Python/Scala/Java.
- Should have experience in Data Ingestion and Integration into data lake using hadoop ecosystem tools such as Sqoop, Spark, SQL, Hive, Airflow, etc..
- Should have experience optimizing (Performance tuning) data pipelines.
- Minimum experience of 3+ years on NoSQL and Spark Streaming.
- Knowledge of Kubernetes and Docker is a plus.
- Should have experience with Cloud services either Azure/AWS.
- Should have experience with on-prem distribution such as Cloudera/HortonWorks/MapR.
- Basic understanding of CI/CD pipelines.
- Basic knowledge of Linux environment and commands.
Functional Areas: Software/Testing/Networking
Read full job descriptionPrepare for Lead Data Engineer roles with real interview advice
10-15 Yrs