Data Engineer - ETL/SQL/Python (5-10 yrs)

Corporate Profile :

Acies Global is a fast growing Chicago based firm in the space of Data Science, AI, Application Development with Innovation and Delivery Centre in Chennai.

We work with Fortune 500 enterprises across various Industries from Food and Beverage, Apparel & Footwear, Oil & Gas, Healthcare, Telecom, Pharma and more.

We work across business functionsSupply Chain, Pricing, CRM and Marketing, Finance, Sourcing, Customer Insights and more.

Job Description :

Looking for a Data engineer to organize our engineering function and expected to implement tools & strategies to translate the raw data into valuable insights with strong logical reasoning skills & Business intelligence.

Key Responsibilities :

Data Pipeline Development :

- Design and implement scalable ETL pipelines using Hive and PySpark for processing large datasets.

- Ensure data pipelines are reliable, fault-tolerant, and meet performance standards.

Big Data Processing :

- Write complex HiveQL queries for data extraction, transformation, and analysis.

- Leverage PySpark for distributed data processing on big data platforms.

Data Integration :

- Integrate data from diverse sources such as relational databases, APIs, and unstructured data repositories.

- Optimize data ingestion and storage strategies for performance and cost efficiency.

Performance Optimization :

- Optimize Hive tables, partitions, and indexing strategies for better query performance.

- Tune PySpark jobs to reduce execution time and resource consumption.

Collaboration & Communication :

- Work closely with cross-functional teams to understand data requirements and deliver solutions.

- Document technical designs, data flows, and troubleshooting guides.

Data Quality & Governance :

- Ensure data integrity, security, and compliance with organizational standards.

- Implement data validation checks and monitoring frameworks.

Platform Management :

- Maintain and enhance big data platforms such as Hadoop, Spark, and related ecosystems.

- Troubleshoot and resolve platform-related issues.

Required Skills :

Technical Expertise :

- Proficiency in HiveQL and PySpark for big data development.

- Strong understanding of Hadoop, Spark, and distributed computing.

- Hands-on experience with relational and NoSQL databases.

Programming Knowledge :

- Advanced skills in Python (PySpark) and SQL.

- Familiarity with shell scripting and automation tools.

Data Architecture :

- Knowledge of data modeling, schema design, and partitioning strategies in Hive.

- Understanding of ETL workflows, data warehousing, and real-time data processing.

Optimization Skills :

- Expertise in debugging and optimizing Spark jobs and Hive queries.

- Familiarity with resource management tools like YARN.

Additional Tools : Experience with tools like Airflow, Kafka, or Nifi is a plus.