How to Run Language Models on Your Own Infrastructure Without Depending on OpenAI

Every time you call OpenAI's API, you're paying per token. Every time you send a document to Claude, you're transmitting data to a server you don't control. Every time you use Gemini, Google processes your information on its infrastructure.

For many companies, this is a double problem: cost (APIs are expensive at volume) and privacy (data is processed on third-party servers). The good news is you no longer need to depend on these providers. Open-source language models have reached a quality level that makes them viable for most enterprise tasks.

The current state of open models

Two years ago, the gap between GPT-4 and the best open models was enormous. Today that gap has closed significantly, especially for structured enterprise tasks.

Qwen (Alibaba) offers models from 0.5B to 72B parameters. The 9B-parameter Qwen is particularly relevant because it fits on a 16GB VRAM GPU and handles contexts up to 128K tokens — enough to process complete long documents.

Llama (Meta) offers models from 1B to 405B parameters. Its intermediate versions are excellent for reasoning and instruction-following tasks.

Mistral offers efficient models that prioritize performance per parameter. Good for environments with limited hardware.

These models are free to download and run. No license fees, no per-token charges, no monthly subscriptions.

What hardware you need

The main requirement is a GPU with enough VRAM (video memory) to load the model. Language models are quantized (reduced in precision) to fit on smaller GPUs without significant quality loss.

To start (16GB VRAM): An NVIDIA RTX 4070 Ti, RTX A4000, or RTX 4090 can run 7-9B parameter models quantized (Q4_K_M) with contexts up to 128K tokens. Typical speed: 40-60 tokens per second. Investment: $800-2,000 USD for the GPU.

For production (24-48GB VRAM): Two GPUs with 24GB each allow running larger models or serving multiple simultaneous requests with tensor parallelism. Typical speed: 100-300+ tokens per second.

The server: It doesn't need to be enterprise. A workstation with a modern processor, 64GB of RAM, and a dedicated GPU is enough. You can use existing equipment or buy a dedicated one. The operating system is Linux (Ubuntu is the most common choice).

The total cost of a private AI server is comparable to 3-6 months of commercial API usage at enterprise volume. After that, the operating cost is essentially electricity.

How the models are run

The software for running language models locally is free and open source:

llama.cpp — The most versatile option. Runs on CPU, GPU, or both. Supports quantization to reduce memory usage. Has a built-in HTTP server that exposes an OpenAI-compatible API, meaning applications designed to use OpenAI's API can point to your local server with no code changes.

vLLM — Optimized for high throughput. Better for production environments where multiple users make simultaneous requests. Requires more VRAM but is significantly faster for serving concurrent requests.

Ollama — The simplest option to get started. Downloads and runs models with a single command. Less configurable but ideal for prototypes and testing.

Typical installation takes hours, not weeks. A team with Linux experience can have an inference server running in a day.

What tasks these models can do

Models with 7-9B parameters are more than sufficient for most enterprise tasks:

Document data extraction. Reading a contract and extracting parties, dates, amounts, key clauses. Reading an ID and extracting name, tax ID, address.

Note and summary generation. Generating SOAP notes from medical transcriptions, summarizing meeting minutes, creating reports from structured data.

Text classification. Classifying documents by type, emails by priority, support tickets by category.

Question answering over data. Querying a database and presenting results in natural language. This is what AI agents do.

Translation. Multilingual models can translate between Spanish, English, and other languages with quality comparable to commercial services.

For more demanding tasks (complex mathematical reasoning, high-quality creative generation, extensive code analysis), larger models or commercial services still have an advantage. But for 80% of enterprise information processing tasks, a local model is sufficient.

When APIs DO make sense

Private AI is not the answer for everything. There are scenarios where commercial APIs make more sense:

Very low volume. If you process 10 documents a month, a dedicated server isn't justified. The API is more practical.

Tasks requiring frontier models. Complex multi-step reasoning, publication-level creative generation, analysis requiring up-to-date world knowledge.

Rapid prototyping. When you're experimenting with an idea and don't know if AI can solve it. Test with an API, validate, then migrate to local if it works.

No technical team. If you don't have anyone who can maintain a Linux server with a GPU, the commercial API is more viable. Though outsourcing server administration is an option.

The optimal strategy for many companies is a hybrid model: private processing for sensitive data and high volumes, commercial APIs for generic tasks and prototyping. With automatic fallback between both to guarantee availability.

Leeuwwolk products: all run locally

At Leeuwwolk, all our products are designed to operate with language models with Leeuwwolk's privacy guarantee. Fullkro, Scriba, Medicus, and The Manager use Qwen and Llama as primary models, with llama.cpp as the inference runtime.

The hardware investment pays for itself quickly against the equivalent cost of commercial APIs, and Leeuwwolk guarantees privacy: encrypted data never shared with third parties.

→ Learn about our private AI solutions

Leeuwwolk is a Mexican company that implements artificial intelligence solutions with open-source models and a privacy guarantee.