Data Engineer - Big Data/Spark/Scala (4-8 yrs)

One of the data analytics company is looking for Data Engineer.

This position will lead data engineering work around digital marketing and web analytics for our customer's online platform. Person- Will working on integration of various data sources to develop a consolidated dataset to run different analytics. May also need to work on the tasks like integration of customer data sources, building customer intelligence and create customer C360 profile.

Will be working on taking out clickstream data from Adobe stack and combining with other data sources.

- Will be responsible for customer interaction and will be presenting findings in customer meetings and reviews.

- Will collaborate with the different platform teams, data science teams and engineering team in developing data pipelines and that could be leveraged in decision making.

- May be needed to work on setting up different components of data infrastructure like data platforms, data pipelines, cloud integration, integration with data lakes.

Desired background / skills /mindset:

Background:

- Bachelor degree in a quantitative field (e.g., Engineering, Mathematics, Operations Research, Statistics, etc.) from Tier 1/2 colleges.

- 4 or more years of professional experience in areas like development of data pipelines, implementation of data lakes, setting up of cloud ecosystem, development of ETL tools etc.

- Previous exposure of working with clickstream data coming from Adobe analytics and

Google analytics will be given extra consideration.

- Previous experience of working directly with clients in US or Europe is must

Required Skills:

- Strong AWS Experience. Experience of working on different components of the ecosystem.

- A certification would be helpful.

Experience with following aspect of data engineering :

- Building and optimizing - big data- data pipelines, architectures and data sets.

- Manipulating, processing and extracting value from large disconnected datasets

- Development of data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.-

- Knowledge and working experience of Spark/Scala

- Good experience with at least one ETL tool like Talend, Informatica etc.

- Build processes supporting data transformation, data structures, metadata, dependency and workload management

- Expertise in SQL and Relational Databases.

- Expertise in at least one of the NoSQL databases.

- Any one of Object-oriented object function scripting languages: Python, Java, C++, Scala, etc.

Mindset:

- Self-starter mentality with the ability to deliver results

- Focused on results and their implications (rather than methods/tools/process)

- Comfortable reprioritizing as situations change to maximize our impact

- Openness to work with different cultures and working styles

This person should be open to work in afternoon shifts (starting around 12 noon or 1 as we may need some overlap time with our US/Europe clients)