Full-Stack Python Developer with Real-Time Data Ingestion and Machine

Full-Stack Python Developer with Real-Time Data Ingestion and Machine Learning Experience

Job Summary:

Full-Stack Python Developer with expertise in machine learning, real-time data ingestion, and web development to build a solution that processes, analyzes, and predicts outcomes based telemetry and log data. The ideal candidate should have experience with data streaming, Python, Flask, and machine learning, and be comfortable designing data ingestion pipelines for telemetry and log data.

Responsibilities:

Real-Time Data Ingestion:

• Build robust pipelines to automatically ingest logs and telemetry data from multiple sources.
• Set up real-time or near-real-time data streaming using tools like Apache Kafka, AWS Kinesis, or custom solutions.
• Clean, preprocess, and transform incoming data for analysis and prediction.

Machine Learning Model Development:

• Design and train machine learning models to perform predictions on critical metrics.
• Use libraries such as scikit-learn, TensorFlow, or PyTorch to optimize models and implement automated model retraining as new data is ingested.

Web Interface Development:

• Develop a user-friendly interface with Flask for both file uploads and real-time data ingestion.
• Enable the interface to handle manual CSV uploads for batch predictions and create endpoints for real-time prediction results.
• Add dashboards or visual elements to display alerts, notifications, and real-time data insights.

Optimize and Maintain Codebase:

• Write clean, efficient, and scalable code.
• Implement logging, error handling, and monitoring within data ingestion pipelines.
• Apply security best practices to handle sensitive log and telemetry data.

Requirements:

• Python: Strong experience with Python and libraries like pandas, numpy, joblib, and scikit-learn.
• Real-Time Data Ingestion: Proficiency in tools such as Apache Kafka, AWS Kinesis, Google Pub/Sub, or similar.
• Machine Learning: Experience with scikit-learn, TensorFlow, or similar libraries for model building and optimization.
• Flask and Web Development: Ability to develop interactive HTML interfaces.
• Data Processing: Skilled in data cleaning, manipulation, and handling structured and unstructured data from logs and telemetry.

Preferred Experience:

• Cloud Services: Familiarity with cloud platforms (AWS, GCP, Azure) for deploying and scaling applications.
• Database Management: Experience with SQL or NoSQL databases for storing processed data.
• Data Streaming Frameworks: Knowledge of Spark Streaming or Apache Flink for high-volume data processing.
• Deployment Tools: Experience with Docker, Kubernetes, or similar tools for deploying applications.
• Security Best Practices: Knowledge of security standards for handling log and telemetry data.

Attributes:

• Problem Solver: Ability to troubleshoot and optimize data ingestion pipelines and machine learning models.
• Autonomous Worker: Comfortable managing tasks independently, with a focus on accuracy and quality.
• Real-Time Mindset: Skilled at designing solutions for responsive data processing and analysis.