A distributed (and federated) data mesh for scientific data and data derived artifacts

This project proposes a distributed data mesh that exposes scientific data and data-derived artifacts (such as indexes, containers, reference datasets, and models) through a scalable, cache-optimised filesystem. It provides a single logical, read-oriented namespace that can be mounted as a POSIX file system across clouds, clusters, and HPC systems with minimal local storage and operational overhead.

The mesh is both distributed and federated: multiple data providers publish to a shared space, while institutions operate local caches and gateways near compute resources to ensure low-latency access and efficient bandwidth utilisation. This architecture enables reproducible workflows, stable references to data and artifacts, and consistent environments for large-scale, data-intensive analysis across sites.

Combined with rich metadata, PID, versioning, and policy-driven publishing, this data mesh forms a foundational layer for scientific platforms, workflow engines, and virtual research environments that require reliable, FAIR access to shared data products across organisational and national boundaries.

Many researchers repeatedly download the same datasets, even within a single group, and these often mirror identical data, and they also rebuild data-derived artifacts. This results in redundant storage, unnecessary network traffic, and additional compute to rebuild the same results. A shared, cache-based data mesh aims to serve common datasets and artifacts from a few well-placed, shared caches instead of many scattered copies, so data is fetched once and then reused close to where it is computed. By reducing redundant transfers and storage, this approach reduces energy use and hardware overhead, helping research infrastructures move toward greener, more resource-efficient operations.

We are looking forward to partnerships/collaborations. If you are interested, let's discuss.