AI4EOSC – Large Language Model (LLM) Service

It allows you to package AI models as Docker containers and serve them on-demand with automatic scaling.This service gives researchers and developers access to advanced text-generation and conversational AI capabilities on EOSC resources, without relying on external providers. It’s designed for flexibility in how you interact with the models and for privacy in how your data is handled.

• Supported models and use cases: The platform offers several state-of-the-art open-source LLMs (small to mid-sized) optimized for AI4EOSC’s infrastructure. For example, you can deploy models like DeepSeek-R1-Distill-Qwen-1.5B, Qwen2.5-7B-Instruct-AWQ, or a 3B-parameter Llama variant. These models can handle tasks such as text generation, summarization, translation, and question-answering in a chat format. You choose a model from the LLM catalog in the dashboard, and with a few clicks, you have your own instance running.

• Deployment options (UI and API): AI4EOSC’s LLM service lets you launch a model with a web-based chat interface, an API endpoint, or both. The chat UI is based on OpenWebUI, providing an interactive chatbot environment in your browser. This is great for human interaction – you and your colleagues can log in and chat with the model, visualize responses, upload documents for it to summarize, etc. Alternatively, the vLLM-powered REST API allows you to integrate the LLM into applications or workflows programmatically. In fact, the API is designed to be compatible with OpenAI’s API specification, so you can use existing OpenAI client libraries to query your AI4EOSC-hosted model. This dual approach (UI and API) means you can both experiment manually and plug the model into scripts or services. Importantly, everything is running in the EOSC cloud – no installation on your part, and no need to send data to an external AI service.

• Customization and control: When you deploy an LLM through AI4EOSC, it’s your private instance, so you have control over it. You can create multiple user accounts for a team to use the chatbot, and via the admin panel you can adjust settings. For example, if the chosen model doesn’t support image inputs, you might disable the image upload feature in the UI. You can also load your own knowledge bases (persistent data the model can draw on for answers) or define custom AI functions/agents to extend the model’s capabilities. This means the service isn’t a black box – you can tailor it to your domain. A key benefit of AI4EOSC’s approach is data privacy: because the LLM runs in an isolated environment under your control, none of your prompts or data are sent to third-party services. The platform doesn’t log your queries or collect data from your instance, and you can shut down and delete the deployment whenever you want, ensuring your sensitive data stays private.

• Integration into workflows: Thanks to the provided API, you can integrate the AI4EOSC-hosted LLM into various tools. For example, you could use it as a coding assistant in VS Code by configuring the Continue.dev extension to point at your AI4EOSC LLM endpoint (since it speaks the OpenAI API protocol). Or from a Python script, you can call the OpenAI Python SDK with the base URL set to your instance’s URL to get model completions. This makes it straightforward to incorporate the LLM into research workflows, such as automatically analyzing text data, answering questions from a dataset, or assisting users in an application – all while running on European open science infrastructure.