StaidLogic - Data Engineer - Google Cloud Platform (10-12 yrs)

Role: CLOUD DATA ENGINEER.

Location : Pune.

Experience : total 10 years.

Role Description :

- This is a contract role for 2 Cloud Data Engineers located on-site in Pune.

- The Cloud Data Engineers will be responsible for data engineering, data modeling, ETL processes, data warehousing, and data analytics on a day-to-day basis.

- Proficient in working with Azure Blob Storage, Azure Data Lake Storage, Azure Data Factory, Azure SQL Data Warehouse, Azure Data Bricks, and on Python, SQL, and PL/SQL concepts.

- Worked on migrating and transforming data from on-premises to cloud and between cloud services by creating Azure data factory pipelines & data flows using different ADF activities and components.

- Experience in writing Spark Applications using (Python/Scala) to connect to different cloud services like Azure SQL DB, Azure Postgres, Azure Synapse Analytics, ADLS, and AWS S3 and performing different data transformations based on business requirements.

- Familiar with Azure DevOps to deploy ADF pipelines and ARM Templates into other environments by creating release pipelines.

- Experience in legacy data migration projects such as Big Data to AWS Redshift migration, i.e, from on-premises to AWS Cloud and Snowflake Data warehouse.

- Experience with Snowflake SQL and Snowflake pipelines to pull data from AWS S3 and traditional DB.

- Worked in a source-consumer application in Google Cloud using different GCP services like Google Cloud Storage, Cloud Data proc, Big Query, Google Cloud PUB/SUB, Google Cloud Composer, etc.

- Experienced in creating airflow DAGs using the most common Operators in Airflow

- Python Operator, Bash Operator, and Google Operators to orchestrate the data flow using Google Cloud services.

- Experienced in writing Spark Applications using (Python/Scala) to connect to different cloud services like Google Cloud Storage, and Cloud SQL and performing different data transformations based on business requirements.

- Proficient in working with AWS S3, AWS Glue, Redshift Data Warehouse, and AWS Ec2 and on Python, SQL, and PL/SQL concepts.

- Responsible for designing and implementing data pipelines using AWS services such as S3, Glue, EC2, and EMR.

- Building and maintaining data warehouses and lakes using AWS Redshift and data security and access controls using AWS IAM.

- Design, develop, and maintain ETL (Extract, Transform, Load) processes using AWS Glue to extract data from various sources, transform it to meet business requirements, and load it into target data sources.

- Worked with the DevOps team to create environment-specific Configuration yaml files to deploy code through CI/CD process by creating artifacts using a central repository.

- Familiar with Processing Real-Time Streaming data using Azure Event Hubs and Azure Stream Analytics and visualizing the results using Power BI and Tableau.

- Experienced in Fact-Dimensional modeling (Star Schema, Snowflake Schema), transactional modeling, and SCD (Slowly Changing Dimension).

- Experienced in designing and creating RDBMS Tables and views, User Created Data Types, Indexes, Stored Procedures, Cursors, Triggers, and transactions.

- Had good knowledge and Hands-on with the ETL tool Informatica, Talend to understand the existing flows, modify the flow, and create the data flow based on the requirement.

- worked extensively with source code management and version control tools like Git, GitHub, and GitLab.

- Familiar with the Agile working method, used JIRA to track work progress and Confluence to prepare and manage technical documentation.

- Familiarity and knowledge of components of the Hadoop Ecosystem like HDFS and HIVE.