Self-hosted AI is having a real moment in 2026, and most of the advice about it skips the one detail that decides whether your plan works: hardware. Running your own models sounds like pure upside. You keep your data, you stop paying per token, and you control the whole stack. The reality is more specific than the pitch. Some AI workloads run fine on a modest server. Others need a graphics card that costs more than a used car and rarely sits idle. This guide draws that line clearly, then shows what self-hosted AI actually looks like on CPU-based infrastructure like Out Plane.
The short answer: Self-hosted AI means running open models like Llama or Mistral on hardware you control instead of paying a hosted API per token. Large models need a GPU. Small quantized models run on CPU but slowly. On CPU-based infrastructure, the realistic win is hosting the AI application (a RAG service, chatbot, or agent) and storing embeddings in PostgreSQL with pgvector, while a GPU host serves the heavy model.
What self-hosted AI actually means
The phrase gets stretched to cover two very different things, and confusing them is where most projects go wrong. One is running the model itself on your own machines. The other is running the software around a model, the retrieval logic, the chat interface, the agent orchestration, on your own machines while the model runs somewhere else. Both are legitimately "self-hosting" in the sense that you own the deployment. Only the first one puts a language model on your hardware.
Self-hosted AI is the practice of running open weight machine learning models and open-source AI tools on infrastructure you own or rent, rather than sending requests to a third-party inference API that charges per token or per request.
That distinction matters because the two paths have wildly different hardware bills. Self-hosting a 70-billion-parameter model demands GPUs and a lot of memory. Self-hosting the application that calls a model demands an ordinary container and a database. If you never separate these two ideas, you end up believing you need a GPU cluster to build a chatbot, which is usually false.
Why teams choose to self-host their AI
The reasons people move away from hosted inference APIs are consistent, and they are good reasons. Data control comes first. When you send prompts and documents to an external API, that data leaves your boundary, and for regulated or sensitive workloads that is a hard stop. Self-hosting keeps the request path inside infrastructure you manage.
Cost predictability is the second driver. Per-token pricing is fine at low volume and unpleasant at scale, because your bill grows with every user and every long context window. A model you host has a fixed compute cost whether it answers ten requests or ten thousand, which flips the math above a certain traffic level. You trade a variable bill for a capacity you provision.
The third reason is independence. Hosted models get deprecated, repriced, and rate-limited on schedules you do not control. Open models you have pinned do not change under you. You also gain the freedom to fine-tune, quantize, and swap models without asking anyone. These benefits are real, and they all come with an operational cost that the hardware section makes concrete.
The hardware reality you cannot skip
Here is the blunt part. A large language model is a large pile of numbers, and running it means moving that pile through fast memory for every token it generates. GPUs exist because they do that specific math in parallel, with memory bandwidth an ordinary CPU cannot match. This is why a 13B or 70B model on a CPU produces text at a pace that feels broken for interactive use, even when it technically works.
Quantization shrinks a model by storing its weights at lower numerical precision, which reduces memory use and lets smaller models run on a CPU. It trades some accuracy for the ability to run without a dedicated graphics card.
Quantization narrows the gap but does not close it. A small model (roughly 1B to 8B parameters) quantized to 4-bit can run on CPU for background or low-traffic tasks. A frontier-sized model cannot, no matter how you compress it, at least not at a latency any user will tolerate. Tools like Ollama make pulling and running these models easy, and Ollama is honest in its own docs that model size and available memory decide what your machine can handle. Ease of setup does not change the physics of memory bandwidth.
This is the exact point where Out Plane's honesty matters. Out Plane is CPU-based and does not offer GPUs. You cannot run a large local LLM on it, and no amount of configuration changes that. What you can do on Out Plane is run small CPU-friendly models for light work and, more importantly, run the AI application that ties everything together.
What actually runs well on a CPU
Plenty of useful AI runs on a CPU without complaint, which the GPU conversation tends to bury. Embedding models that turn text into vectors are small and fast on CPU, and they power search and retrieval. Classic classifiers, sentiment models, named-entity extractors, and reranking models are all comfortable on CPU. Small quantized chat models handle summarization, tagging, and simple structured extraction for non-interactive jobs where a slower response is acceptable.
What does not run well on a CPU is any model people expect to hold a fluent, low-latency conversation at scale. Interactive chat backed by a large model needs a GPU, full stop. If your product depends on that, you provision a GPU host for the model and connect to it. Trying to force a 70B model onto CPU cores to save money produces an experience nobody will use.
The practical rule: use CPU for embeddings, retrieval, classification, and small background inference. Use a GPU, yours or a hosted one, for large generative models that answer users in real time. Once you internalize that split, the architecture becomes obvious.
Separate the model from the application
Almost every "AI product" is mostly ordinary software. A retrieval-augmented generation (RAG) service ingests documents, chunks them, generates embeddings, stores those vectors, and at query time it searches for relevant chunks and passes them to a model with the user's question. Only the last step touches a large model. Everything else is a normal web service plus a database.
That is the key architectural move for self-hosting on CPU infrastructure. You self-host the application, the API server, the ingestion pipeline, the retrieval logic, the agent loop, and you point the single generative step at whichever model host makes sense. The model can be a small local one for cheap tasks or an external GPU host or hosted inference API for the heavy generation. Your data, your logic, and your embeddings stay in infrastructure you control.
This is also where the earlier distinction pays off. If self-hosting the model is the expensive, GPU-bound part, and the application is the cheap, CPU-bound part, then a CPU PaaS is a genuinely good home for the ninety percent of the system that is not the model.
How to self-host an AI application on Out Plane
Out Plane is a platform where you push code or a Dockerfile and it builds an image and runs your app, with a managed PostgreSQL database alongside and automatic HTTPS. For an AI application that shape fits cleanly. Your app is a container. Its memory is a database. Its brain, the large model, lives wherever you have the compute for it.
The flow looks like this. You bring a Dockerfile or let the platform auto-detect a buildpack, and Out Plane runs the build for you. If you want the details of that path, our guide to deploying with Docker covers it, and the broader what a PaaS is explainer sets the context. Your service gets a public URL on the pattern myapp-3000-acme.outplane.app with HTTPS handled automatically, and a private network connects it to its database and any sibling containers.
For the memory layer, you provision managed PostgreSQL and enable pgvector, which stores embeddings and runs similarity search inside the same database that holds your application data. No separate vector store to operate.
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE documents (
id bigserial PRIMARY KEY,
content text,
embedding vector(1536)
);
-- nearest-neighbor search at query time
SELECT content
FROM documents
ORDER BY embedding <-> $1
LIMIT 5;The managed database handles automated backups, point-in-time recovery, read replicas, and connection pooling, so the stateful part of your AI app is operated for you. The pgvector extension is mature and well documented, and keeping vectors in Postgres means one system to back up and secure instead of two. Our walkthrough of free PostgreSQL hosting shows how to start on the free tier. If you already have the app built and just want the deploy steps, the deploy an AI application guide is the companion to this one: that piece is about shipping, this piece is about the self-hosting-versus-hosted decision underneath it.
Where the heavy model lives
For the generative model itself, be realistic about placement. If a small quantized model on CPU meets your quality bar for a background task, you can run it in a sibling container with a persistent volume for the model weights, and your app talks to it over the private network. That is genuine local self-hosting for the workloads a CPU can carry.
For anything that needs a large model with low latency, you point your Out Plane app at an external endpoint. That is either a GPU host you run elsewhere or a hosted inference API. Your application, your retrieval, and your embeddings still live on Out Plane, and only the final generation call leaves. This is not a compromise on self-hosting so much as putting each component where its hardware demands are met. Front ends like Open WebUI can sit on top of whichever model endpoint you choose.
Hosted API versus self-hosted GPU versus CPU PaaS
The decision usually comes down to three shapes, and each fits a different situation. The table below lays them side by side so you can match your workload to the right column instead of assuming one answer covers everything.
| Dimension | Hosted AI API | Self-hosted model on GPU | AI app on CPU PaaS with pgvector |
|---|---|---|---|
| Where the model runs | Vendor's servers | Your GPU hardware | Small model on CPU, or external endpoint |
| Hardware you manage | None | GPUs, memory, drivers | A container plus a managed database |
| Cost model | Per token, variable | High fixed compute cost | Low fixed cost, pay-as-you-go |
| Large LLM support | Yes, no setup | Yes, you provision it | No local large LLM (CPU only) |
| Embeddings and search | External or separate store | You run it | Built in via pgvector in PostgreSQL |
| Data location control | Limited | Full | App and vectors stay in your app |
| Ops burden | Lowest | Highest | Low for the app, none for the database |
| Best for | Fast start, low volume | High volume, strict control | RAG, chatbots, and agents at CPU cost |
Most teams end up in the third column for the application and reach into the first or second column only for the heavy generation step. That hybrid is the sane default in 2026.
Frequently Asked Questions
Can I run a large language model like Llama 70B on Out Plane?
No. Out Plane is CPU-based and does not provide GPUs, and a 70B model needs a graphics card with substantial memory to run at a usable speed. You can run small quantized models on CPU for light background tasks, and for large models you point your Out Plane app at an external GPU host or a hosted inference API.
What can I self-host on CPU-only infrastructure?
Embedding models, text classifiers, rerankers, and small quantized chat models for non-interactive tasks all run acceptably on CPU. You can also self-host the entire AI application layer: the API server, ingestion pipeline, retrieval logic, and agent orchestration. The large generative model is the one piece that usually needs a GPU somewhere else.
Is self-hosted AI cheaper than a hosted API?
It depends on volume. Per-token APIs are cheap at low traffic and expensive at scale, because cost grows with every request. A self-hosted model has a fixed compute cost regardless of usage, which wins above a certain volume. The application layer is cheap either way, so the crossover point is really about the generation calls.
What is the difference between self-hosting a model and hosting an AI application?
Self-hosting a model means the language model itself runs on your hardware, which is GPU-bound for large models. Hosting an AI application means running the software around a model, retrieval, storage, and orchestration, which is ordinary CPU work. You can host the application yourself while the model runs on an external endpoint.
Can I store embeddings for RAG on Out Plane?
Yes. Out Plane's managed PostgreSQL supports the pgvector extension, so you store embeddings and run similarity search in the same database that holds your application data. That means one system to back up and secure. Backups, point-in-time recovery, read replicas, and connection pooling are handled for you on the managed database.
Do I need a GPU to build a chatbot or RAG app?
Not for most of it. The ingestion, embedding, retrieval, and orchestration all run on CPU. You only need GPU-grade compute for the large model that writes the final answer, and you can rent that as a hosted API or run it on a separate GPU host. The app and its vectors can live on CPU infrastructure.
Is my data more private with self-hosted AI?
Self-hosting keeps your application logic, your documents, and your embeddings inside infrastructure you manage, which is a meaningful privacy gain over sending everything to a third party. Note that if you call an external model for generation, those specific prompts still leave. Keeping retrieval and storage self-hosted limits exposure to only the final generation step.
Does Out Plane keep my data in the EU?
You can choose an EU region, Nuremberg or Helsinki, so your app and database data reside there. This is a data residency choice, not a legal certification or jurisdiction claim, and Out Plane is a US company. If your requirement is that data physically stays in Europe, the region setting gives you that. Legal compliance is a separate review you should run.
Conclusion
Self-hosted AI in 2026 is a good decision made for the right reasons: data control, predictable cost, and independence from a vendor's roadmap. The trap is treating it as one decision when it is really two. Hosting a large model is GPU-bound and expensive. Hosting the AI application around that model is CPU-bound and cheap, and it is where most of your actual product lives. Split them, and the architecture stops fighting your hardware budget.
On a CPU platform like Out Plane, the realistic and honest path is clear. Run your RAG service, chatbot, or agent as a container, store embeddings in managed PostgreSQL with pgvector, get HTTPS automatically, and point the heavy generation step at whatever GPU host or inference API fits. To start, create a project in the console at https://console.outplane.com on the free Hobby tier with $20 in trial credit and no credit card, and see the pricing page when you are ready to scale the app.