AI news

Where to host an open-source LLM: cost, skills and data

In brief

According to OVHcloud, after you pick an open-source model you still need to choose where and how to run it. The company compares three options: on-demand inference through an API, serverless deployment of your own models, and a Kubernetes-style orchestration platform. The three paths do not have the same effect on cost, response time and control of data.

Source : OVHcloud - « Hosting an open-source LLM: how to choose the right option »

What it changes for an SME

For an SME of 10 to 250 people, an on-demand API is often the lightest start: according to OVHcloud, it fits a first use case or a prototype, especially if token use is irregular. Once you run a model trained on your data or you need stronger isolation, you take on more operations work, up to Kubernetes if your team already runs that platform. The API service described runs from Gravelines in France under European law; for the other options, data location depends on the region you select.

What you can do

  • Check whether a catalogue model served through an API covers your need, or whether you must deploy a model trained on your data.
  • Estimate whether your token volume will be irregular or sustained, and whether you need stable response times.
  • If your data must stay in Europe, confirm the hosting region and what operations skills you have in-house.
Intro call · 20 min · free

Let's talk about the time you could save.

In 20 minutes, we find where AI could save you time or money. Free, no commitment.

[email protected] Belgium · Europe (remote)
Book a call Call