KETs2026 reloaded

24 Sept 2026 | Germany

ExpertiseUpdated on 24 September 2026

Data processing work package: pipelines across heterogeneous partner sources, incremental recomputation, provenance and retraining

COO at Epoch8 LLC

Tbilisi, Georgia

About

Every consortium project produces data from partners who model the world differently, on different schedules, and in different formats. Someone has to make that into one dataset that the models, the dashboards, and the reviewers can all use. We do that work.

WHAT WE BUILD

Ingestion from heterogeneous sources. Partner databases, sensor and machine streams, files dropped on a share, spreadsheets that people maintain by hand, third-party APIs, and documents. We treat the spreadsheet maintained by a human as a real source, because in industrial projects it usually is one.

Harmonization and the semantic layer. One agreed model of the entities a project talks about, so that a figure means the same thing in every deliverable. We use minimal modeling for this purpose, which keeps the model small enough that domain experts can read it and correct it.

Warehousing and transformation. Pipelines into ClickHouse, PostgreSQL, or Snowflake, with dbt where the team already works that way.

Analytics and dashboards. Monitoring of a pilot while it runs, and reporting on the project’s needs for its own review milestones.

Data for machine learning. Training set construction, versioning, labelling pipelines, evaluation datasets, and benchmarks.

DATAPIPE AND INCREMENTAL RECOMPUTATION

Datapipe is our open-source Python framework for incremental data processing and machine learning pipelines. It tracks what each step depends on and recomputes only the parts affected by a change.

This issue matters in three ways in a funded project.

Cost and energy. A pipeline that reprocesses everything on every run is the main reason data work in research projects stops. Recomputing only the changed part keeps the cost of a rerun roughly proportional to the size of the change.

Iteration. Partners deliver data late and in corrections. Incremental recomputation means a late correction from one partner costs one small rerun.

Life after the project. The pipeline stays affordable to operate once the funding ends, which is the difference between an exploitable result and a demonstrator.

WHY IT MATTERS IN A PROPOSAL

Data management plan. Every project needs one, and it is usually written by someone who will not be building the storage. We can own the technical half: where data lives, how it is versioned, how access is controlled, and what is published and in what form.

Provenance. Every figure in a deliverable should be traceable back through the transformations to the raw record it came from. We build pipelines that maintain this chain.

FAIR and open publication. Preparing the datasets is a project that commits to publishing, with the documentation that makes them usable by someone outside the consortium.

Reusability. The data model and the pipeline are artifacts that outlive the project, and Datapipe is permissively licensed, so they can be named as exploitable results without a licence negotiation.

WHAT WE CAN DELIVER IN A WORK PACKAGE

Source inventory and the ingestion architecture. Data model and semantic layer. The pipelines themselves include tests and monitoring. Storage and access control. Dashboards for pilot monitoring and project reporting. Dataset preparation, versioning, and public release. Drift monitoring and scheduled retraining for any model in the project. The technical part of the data management plan.

TEAM AND STACK

We are an engineering team of 30+ specialists covering data engineering, machine learning, computer vision, and backend. We use Python, dbt, ClickHouse, PostgreSQL, Snowflake, Spark, and Datapipe for orchestration, along with cloud and on-premise deployment. We work with clients in manufacturing, logistics, retail, telecom, and health.

PRACTICAL

Our legal entity is registered in Georgia. Georgia has been associated with Horizon Europe since 2021, so we participate and are funded on the same terms as an entity from a member state, and Georgia is named in the WIDERA work program among the associated countries with equivalent characteristics in terms of research and innovation performance, so our participation counts towards the widening and geographic balance of your consortium.

Working language: English. PIC available on request. If the data layer of your proposal is still unassigned, send us the topic, and we will come back with a one-page work package outline.

Field

  • AI-GenAI /Data/Robotics

Organisation

Epoch8 LLC

Consulting

Tbilisi, Georgia

Similar opportunities