Data Engineer_ web scraping, /Azure DataBricks/Data Modelling/MSFabric

JD:1_Data Engineer/web scraping, web crawling:

OSI is seeking a skilled Data Engineer with expertise in Python, web scraping, web crawling, and frameworks such as Scrapy. The ideal candidate will also have secondary experience with Snowflake and Databricks. This role involves developing and maintaining data pipelines, extracting and processing data from various sources, and ensuring seamless integration with our data platforms.

Key Responsibilities:

• Develop and maintain web scraping and crawling solutions to collect data from diverse web sources using Python and libraries like Scrapy, Beautiful Soup, Playwright, lxml, urllib3 or similar tools.
• Design, build, and operationalize complex data pipelines to process and integrate data from multiple sources into our data platforms.
• Implement efficient ETL (Extract, Transform, Load) processes to transform raw data into structured formats suitable for analysis.
• Work closely with cross-functional teams, including data scientists and analysts, to understand data requirements and deliver solutions that support business objectives.
• Optimize data retrieval and processing performance, ensuring scalability and efficiency in data handling.

JD2:

Key Responsibilities:

Data Pipeline Development:
Design and implement robust, scalable ETL/ELT data pipelines using Azure Databricks to process, transform, and integrate large datasets from various sources.

Collaboration with Data Teams:
Work closely with data scientists, analysts, and business intelligence teams to understand their data needs and translate those requirements into functional solutions.

Spark-based Development:
Develop and optimize Spark-based applications in Databricks for distributed data processing, ensuring performance, reliability, and cost-efficiency.

Cloud Infrastructure & Services Integration:
Leverage Azure cloud services (such as Azure Data Lake Storage, Azure SQL Database, Azure Synapse Analytics) to integrate data solutions seamlessly with Databricks.

Data Architecture & Optimization:
Contribute to the design of data architecture for cloud-based data lakes, warehouses, and real-time processing. Optimize data queries and workflows to ensure high performance.

Machine Learning & Analytics Support:
Assist data scientists in implementing and deploying machine learning models on the Azure Databricks platform, enabling data-driven decision-making across the organization.

Automation & Monitoring:
Implement automation for regular data processes and use Azure monitoring tools (such as Azure Monitor and Log Analytics) to ensure optimal performance and troubleshoot issues.

Security & Compliance:
Implement appropriate data security measures and adhere to compliance standards in managing sensitive data, ensuring data is handled securely across cloud environments.

Documentation & Reporting:
Provide detailed documentation on workflows, processes, and data models, as well as periodic progress reports to stakeholders.

JD3:

We are seeking a skilled Data Modeler to join our Data & Analytics team. The ideal candidate will have a strong background in designing and implementing data models, particularly within Azure environments, and experience with tools such as Databricks and Power BI. This role involves creating structured data models, developing materialized views, ensuring data quality, and collaborating with cross-functional teams to deliver insightful analytics solutions.

Key Responsibilities:

• Data Modeling: Design and develop conceptual, logical, and physical data models to support the Operational Data Mart, ensuring alignment with business requirements and best practices.
• Schema Design: Create snowflake schemas with well-defined fact and dimension tables to optimize data storage and retrieval.
• Materialized Views/OLAP Cubes: Develop and manage materialized views and OLAP cubes within Databricks to facilitate efficient data aggregation and analysis.
• Data Integration: Collaborate with data engineers to integrate data from Azure Data Lake, Excel files, and other sources, ensuring seamless data flow into the Data Mart.
• Data Quality Assurance: Implement data quality checks and produce comprehensive Data Quality Index reports, addressing both functional and non-functional aspects for all tables.
• Reporting and Analytics: Work closely with BI developers to design and develop Power BI reports and dashboards, providing real-time insights and supporting self-service analytics capabilities.
• Documentation: Maintain thorough documentation of data models, schemas, data flows, and data dictionaries to ensure clarity and facilitate ongoing maintenance.