Data Engineer (AI-Ready Data Pipelines) - Rajya Sabha - Delhi

Key Responsibilities:

• Data Cleansing and Preprocessing: Develop and implement sophisticated data cleansing pipelines to ensure the integrity and quality of data before it is fed into AI models. Utilize libraries such as pandas, Dask, or PySpark for large-scale data processing and transformation.
• Vectorization and Embeddings: Implement techniques for data vectorization (e.g., Word2Vec, TF-IDF, BERT embeddings) to convert unstructured text data into numerical format suitable for AI/ML tasks.
• Data Pipelines Development: Build and maintain scalable and efficient ETL (Extract, Transform, Load) pipelines using technologies like Apache Kafka, Apache Airflow, Luigi, or Apache Beam for real-time data processing.
• AI-Ready Data Preparation: Design and develop data pipelines that transform raw data into structured, AI-ready datasets, ensuring the highest quality for model training and inference.
• Data Integration: Integrate disparate data sources, both structured and unstructured, into a unified data lake or warehouse using tools like AWS S3, Google BigQuery, or Apache Hudi.
• Automated Data Transformation: Implement automated data transformation processes using Python and SQL, and leverage tools like dbt (data build tool) to structure data for analytical purposes.
• Data Storage and Management: Work with cloud-based or on-premise data storage systems (e.g., AWS Redshift, Google Cloud Storage, Apache Hive) to store large datasets, ensuring data scalability and security.
• Collaboration with Data Scientists: Work closely with the data science team to ensure smooth integration of data pipelines with machine learning models. This includes data validation, feature engineering, and providing ready-to-use datasets.
• Monitoring and Optimization: Continuously monitor the performance of data pipelines, identifying and resolving bottlenecks or failures. Implement monitoring tools like Prometheus or Grafana for real-time insights.
• Documentation & Reporting: Provide clear documentation on pipeline architecture, data processes, and troubleshooting guidelines for internal knowledge sharing and project continuity.

Skills & Requirements:

• Strong proficiency in Python and SQL, with experience in working with big data technologies such as Apache Spark, Dask, or Hadoop for data processing at scale.
• Hands-on experience in building and managing ETL pipelines using Apache Kafka, Airflow, Luigi, or similar frameworks for batch and real-time processing.
• Expertise in data vectorization techniques (e.g., TF-IDF, Word2Vec, BERT embeddings) for processing unstructured data.
• Experience with cloud platforms (AWS, GCP, Azure) and data storage solutions like Amazon S3, Google BigQuery, Redshift, or Apache Hive.
• Proficient in using data transformation tools like dbt, Pandas, and Numpy for cleaning, normalizing, and structuring datasets.
• Familiarity with containerization (Docker, Kubernetes) to manage and deploy data infrastructure.
• Strong understanding of APIs and experience integrating external data sources into data pipelines.
• Solid understanding of data security practices, especially for handling sensitive parliamentary data.
• Experience with CI/CD pipelines for continuous integration and delivery of data-driven solutions.