Senior Data Engineer - AWS Cloud (5-10 yrs)

Job Title : Data Engineer.

Experience - 5+ yrs.

Location : Pune/Bangalore.

Responsibilities for Data Engineer :

- Create and maintain optimal data pipeline architecture for sourcing data from multiple structured and unstructured data sources.

- Assemble large, complex data sets that meet functional / non-functional business requirements.

- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS 'big data' technologies.

- Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.

- Perform data cleaning, exploration, and statistical analysis to ensure data quality.

- Work alongside a Data Scientist to develop modeling techniques for various product capabilities.

Qualifications for Data Engineer :

- Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.

- Hands-on implementation of spark data pipeline and deep understanding of core spark concepts.

- Experience building and optimizing 'big data' data pipelines, architectures, and data sets.

- Knowledge of sprint and scrum.

- 5+ Years of Experience in ETL Applications like Apache Airflow, pipelines like Glue, etc.

- Strong analytical skills related to working with unstructured datasets.

- Build processes supporting data transformation, data structures, metadata, dependency, and workload management.

- A successful history of manipulating, processing and extracting value from large disconnected datasets.

- Working knowledge of message queuing, stream processing, and highly scalable 'big data' data stores.

- Experience with big data tools : Hadoop, Spark, Kafka, is a plus.

- Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.

- Experience with AWS cloud services : EC2, EMR, RDS, Redshift.

- Experience with object-oriented/object function scripting languages : Python, Java, C++, Scala, etc.