Updated on 24 September 2026
Data processing work package: pipelines across heterogeneous partner sources, incremental recomputation, provenance and retraining
COO at Epoch8 LLC
Tbilisi, Georgia
About
Every consortium project produces data from partners who model the world differently, on different schedules, and in different formats. Someone has to make that into one dataset that the models, the dashboards, and the reviewers can all use. We do that work.
WHAT WE BUILD
Ingestion from heterogeneous sources. Partner databases, sensor and machine streams, files dropped on a share, spreadsheets that people maintain by hand, third-party APIs, and documents. We treat the spreadsheet maintained by a human as a real source, because in industrial projects it usually is one.
Harmonization and the semantic layer. One agreed model of the entities a project talks about, so that a figure means the same thing in every deliverable. We use minimal modeling for this purpose, which keeps the model small enough that domain experts can read it and correct it.
Warehousing and transformation. Pipelines into ClickHouse, PostgreSQL, or Snowflake, with dbt where the team already works that way.
Analytics and dashboards. Monitoring of a pilot while it runs, and reporting on the project’s needs for its own review milestones.
Data for machine learning. Training set construction, versioning, labelling pipelines, evaluation datasets, and benchmarks.
DATAPIPE AND INCREMENTAL RECOMPUTATION
Datapipe is our open-source Python framework for incremental data processing and machine learning pipelines. It tracks what each step depends on and recomputes only the parts affected by a change.
This issue matters in three ways in a funded project.
Cost and energy. A pipeline that reprocesses everything on every run is the main reason data work in research projects stops. Recomputing only the changed part keeps the cost of a rerun roughly proportional to the size of the change.
Iteration. Partners deliver data late and in corrections. Incremental recomputation means a late correction from one partner costs one small rerun.
Life after the project. The pipeline stays affordable to operate once the funding ends, which is the difference between an exploitable result and a demonstrator.
WHY IT MATTERS IN A PROPOSAL
Data management plan. Every project needs one, and it is usually written by someone who will not be building the storage. We can own the technical half: where data lives, how it is versioned, how access is controlled, and what is published and in what form.
Provenance. Every figure in a deliverable should be traceable back through the transformations to the raw record it came from. We build pipelines that maintain this chain.
FAIR and open publication. Preparing the datasets is a project that commits to publishing, with the documentation that makes them usable by someone outside the consortium.
Reusability. The data model and the pipeline are artifacts that outlive the project, and Datapipe is permissively licensed, so they can be named as exploitable results without a licence negotiation.
WHAT WE CAN DELIVER IN A WORK PACKAGE
Source inventory and the ingestion architecture. Data model and semantic layer. The pipelines themselves include tests and monitoring. Storage and access control. Dashboards for pilot monitoring and project reporting. Dataset preparation, versioning, and public release. Drift monitoring and scheduled retraining for any model in the project. The technical part of the data management plan.
TEAM AND STACK
We are an engineering team of 30+ specialists covering data engineering, machine learning, computer vision, and backend. We use Python, dbt, ClickHouse, PostgreSQL, Snowflake, Spark, and Datapipe for orchestration, along with cloud and on-premise deployment. We work with clients in manufacturing, logistics, retail, telecom, and health.
PRACTICAL
Our legal entity is registered in Georgia. Georgia has been associated with Horizon Europe since 2021, so we participate and are funded on the same terms as an entity from a member state, and Georgia is named in the WIDERA work program among the associated countries with equivalent characteristics in terms of research and innovation performance, so our participation counts towards the widening and geographic balance of your consortium.
Working language: English. PIC available on request. If the data layer of your proposal is still unassigned, send us the topic, and we will come back with a one-page work package outline.
Field
- AI-GenAI /Data/Robotics
Organisation
Similar opportunities
Expertise
- AI-GenAI /Data/Robotics
Olga Tatarinova
COO at Epoch8 LLC
Tbilisi, Georgia
Project cooperation
STÉPHANE BOBÉ ENGUÉLÉGUÉLÉ
Founder & CEO at SOVEREIGNPIALPHA FRANCE LTD
PARIS, France
Expertise
Maria Tolkacheva
VP of Business Development, Partner at Techbar Limited
Milan, Italy