On-prem Ollama: Sovereign and Air-Gapped
Step-by-step playbook: deploy Ollama locally. Zero external API calls. Fully sovereign. Works in air-gapped environments. HIPAA-ready.
The air-gapped problem
Your infrastructure is air-gapped. No internet. No external API calls allowed. This is non-negotiable for compliance and security.
But you want AI capabilities. Gap analysis. Test generation. Defect patterns.
You cannot use hosted LLM services. They require internet. They send data outside your network.
So you are stuck. Or you were, until on-prem Ollama.
What on-prem Ollama is
Ollama is an open-source LLM inference engine that runs entirely locally. No external API calls. No data leaves your infrastructure.
When you deploy Ollama in your air-gapped environment, you get:
Local LLM inference (runs on your hardware)
Support for open-source models (Llama, Mistral, Falcon, etc.)
Zero external dependencies
Full data sovereignty
You control everything. Your code, your models, your infrastructure.
The step-by-step playbook
Step 1: Prepare your infrastructure
On-prem Ollama needs compute. An LLM model typically requires:
GPU recommended: NVIDIA A100 or H100 for production, RTX 4090 for smaller deployments
CPU fallback: possible but slow (5-10x slower than GPU)
RAM: 32GB minimum for reasonably-sized models
Storage: 20-50GB for model weights depending on model size
Hardware does not need to be on the internet. It just needs to be in your air-gapped network.
Step 2: Download Ollama and models
Ollama is open-source. Download from GitHub or your internal package repository. Download options: - GitHub: github.com/ollama/ollama (public)
Mirror: Host Ollama in your internal artifact repository
Pre-baked: Include Ollama in your infrastructure-as-code For air-gapped environments, download models on a connected machine, then transfer them to your air-gapped environment via USB or internal transfer mechanism.
Step 3: Install Ollama on your node
Ollama installation is straightforward:
Installation (Linux): curl https://ollama.ai/install.sh | sh
Or install from your internal repository
apt install ollama
For air-gapped: download the .deb or binary, transfer to the node, install locally.
Step 4: Load a model
Models live in Ollama's model library. Common options for WalnutAI:
Llama 2 (7B, 13B, 70B): good general-purpose model
Mistral 7B: fast, good for code
Neural Chat: fine-tuned for conversation
Download the model file (usually 3-70GB depending on size).
Load a model: ollama pull llama2:13b
For air-gapped, import pre-downloaded model:
ollama import model-file.bin
Step 5: Configure WalnutAI to use Ollama
In your WalnutAI configuration, point to the local Ollama instance:
WalnutAI config: llm_provider: ollama ollama_endpoint: http://ollama-node:11434 ollama_model: llama2:13b fallback_provider: none (no fallback in air-gapped) WalnutAI now routes all requests to your local Ollama instance.
Step 6: Verify zero external calls
Critical: verify that no data leaves your network. Verification steps: 1. Check network traffic with tcpdump or Wireshark 2. Confirm all requests go to local Ollama endpoint (http://ollama-node:11434) 3. Run a test analysis. Verify no outbound traffic. 4. Log all connections. Confirm localhost/internal IPs only. Document this verification for compliance. This is your proof of zero external calls. Critical: verify that no data leaves your network.
Verification steps: 1. Check network traffic with tcpdump or Wireshark 2. Confirm all requests go to local Ollama endpoint (http://ollama-node:11434) 3. Run a test analysis. Verify no outbound traffic. 4. Log all connections. Confirm localhost/internal IPs only. Document this verification for compliance. This is your proof of zero external calls.
Performance considerations
Model size vs quality
Smaller models (7B parameters) run faster but are lower quality. Larger models (70B) run slower but are higher quality.
For most use cases, Llama 2 13B is the sweet spot: reasonable quality, reasonable speed.
GPU vs CPU
GPU inference: 100-200 tokens/second. CPU inference: 5-20 tokens/second.
If you have GPU, use it. If not, CPU works but will be slow. Test with your expected workload.
Caching
Ollama caches models in memory. First request after startup is slow (model load). Subsequent requests are fast.
For production, run Ollama as a persistent service. Keep it running. Do not restart between requests.
Compliance by default
On-prem Ollama with zero external API calls means:
HIPAA compliance (data stays in your infrastructure)
PCI-DSS compliance (no data leaves your network)
SOC2 compliance (audit trail fully internal)
GDPR compliance (data residency in your region)
You do not have to negotiate or explain. The architecture is compliant by design
Scaling Ollama
For single workload: one Ollama instance on one node.
For multiple concurrent workloads: replicate Ollama across multiple nodes. Load-balance requests.
Scaling architecture: Load Balancer (internal)
-> Ollama Node 1 (10 concurrent requests)
-> Ollama Node 2 (10 concurrent requests)
-> Ollama Node 3 (10 concurrent requests)
Total capacity: 30 concurrent requests
Kubernetes Job or Docker container orchestration makes this straightforward.
If you are air-gapped and need AI capabilities, on-prem Ollama is your path.
Start with a single node deployment. Test with a small model (7B). Expand from there.
You now own your LLM infrastructure. No vendor dependency. No compliance concerns.
Deploy on-prem Ollama. Own your LLM infrastructure. Try it on your project.
