Python Developer - ETL (1-5 yrs)

SALARY : 7LPA - 14LPA

About CDPG :

At CDPG, we are committed to democratising data and our mission is to help harness its power by creating data exchange platforms and seamlessly integrating them into the broader context of Data for Public Good. By ensuring that data exchange is conducted ethically, with a focus on privacy and security, we strive to make the benefits of data accessible to all, promoting inclusivity in decision-making processes.

Scope of Work :

- Collect requirements and collaborate with city officials, agencies, system integrators, solution providers, and other data sources to integrate relevant datasets into the data exchange platform.

- Deploy and manage data exchange, while supporting application providers in building solutions for optimised management and citizen services.

Responsibilities :

- Evaluate smart city solution APIs and datasets, create data models, develop software ETL modules, perform unit testing, and deploy them in the cloud instance.

- Develop ETL modules in Python to ingest data into data exchange using REST APIs and streaming protocols such as AMQP and MQTT. This includes containerizing the adapters, creating data models, and catalogue entries according to data exchange specifications.

- Follow best practices for software development and adhere to Agile methodology throughout.

Required Skills :

- Excellent software design and programming skills in Python, and Java/JavaScript, with proficiency in data model technologies such as JSON-schema, JSON-LD, and Protobuf, along with strong functional testing skills.

- Solid understanding of available data sources and APIs in city solutions, including the ability to evaluate data quality, availability frequency, and common issues like data stream repetitions, as well as familiarity with various data fields and their meanings, and data structure formats (JSON, GeoJSON).

- Experience in developing ETL modules in Python for data ingestion into data exchange using REST APIs and streaming protocols such as AMQP and MQTT, including containerizing adapters and creating data models and catalogue entries according to data exchange specifications, while adhering to best practices in software development and Agile methodology.

- Good understanding and expertise in working with geospatial data (vector and raster), with experience in processing geospatial data using GIS tools like QGIS and ESRI, and skilled in analysing, transforming, and visualising geospatial data.

- Proficient in Linux, with experience in GIT version control and cloud computing platforms.

- Strong understanding of IoT and Smart City solutions, GIS, Big Data, and Cloud applications aimed at enhancing citizen convenience and efficient service delivery, with a commitment to creating a positive societal impact.