Private LLM deployment

The common offer of "private AI" is often a public API with a privacy page. If the request leaves your network, the data has left too.

Open-weight models publish their weights. You download them once, check the file, and run them on your hardware. Nothing goes back to the maker.

Customer data stays inside. That is the answer a compliance team, an auditor, and a large client ask for, under rules such as GDPR, HIPAA, India's DPDP Act, or a bank's own policy. The model does not change overnight. A version you tested stays the version you run. There is no rate limit from someone else's traffic, and no outage on someone else's status page. You can fine-tune it on your own data, and the result is yours. It can run on a network with no internet at all. Switching models later is a config change, not a new vendor contract.

A public API sends a bill for every token. Your own model is a fixed cost: the GPUs, or a rented server, plus power and upkeep. When usage is steady and high, many staff, long documents, agents that run all day, the fixed cost wins and the bill stops growing with use. A small open model also does everyday jobs, such as sorting tickets, reading invoices, and drafting replies, for far less than a frontier model. We size to the job, not the leaderboard.

Size comes first. A support assistant or a document reader often fits one 24 GB or 80 GB GPU, with gpt-oss-20b or Qwen3.8-27B. DeepSeek V4 Flash needs a server with several GPUs. Kimi K3, and the largest GLM and Qwen models, need a cluster. A mixture-of-experts model saves compute, not memory. We test three candidates and ship the smallest one that passes.

The license is the second check. DeepSeek V4 is MIT. gpt-oss is Apache 2.0. Kimi K3 and Qwen3.8 add terms once you resell access past a revenue line. We read the LICENSE file before you buy a GPU.

Then the plumbing. vLLM or SGLang serves many people at once. Ollama or llama.cpp is enough for a pilot. A gateway speaks the same API your apps already use, so swapping a model is a config change. It checks who is calling, limits each team, and writes every prompt to a store you own.

Before you buy hardware, we put your current API bill next to the GPU cost at your real volume. If an API is cheaper and the data is not sensitive, we say so. If owning the model wins, you see the month it pays for itself. Either way you spend against a number, not a guess.

Best for Banks, hospitals, legal and government teams, and any company whose contracts or regulators say customer data cannot go to a public AI API.