One private model behind every product.
Every Blue Sail product that uses local AI talks to the same model, Gemma 4, running on an NVIDIA DGX Spark in our own Kubernetes cluster. Here's how the pieces fit.
The diagram shows, in order:
- Visitors reach us through Cloudflare.
- Our Kubernetes cluster, 9 ARM64 nodes, runs our products: ask-amac, The Clam Life, Ukrainian course, Clam Book and Job Rabbit.
- ask-amac also uses Qdrant: vector search, 3 nodes.
- Each product sends its request to llm-gateway, our in-house gateway. It has 3 lanes: Public lane: our websites; Internal lane: our tools; Private lane: priority jobs. Approved models only, answer limits, metrics and alerts.
- The gateway reaches the NVIDIA DGX Spark (GB10 Grace Blackwell, 128 GB unified memory) over 4 parallel channels. Ollama runs Gemma 4 26B-A4B (chat, 32K context), Llama Guard 3 (safety screening) and Embedding models (search).
- Outside the cluster: Cloud AI, where noted. The Clam Life uses Google Gemini with Search grounding.
How a request travels
A visitor asks a question
Traffic reaches our cluster through Cloudflare. Each product runs as its own set of services in Kubernetes.
The product prepares the request
It adds context — for ask-amac, places and events found with vector search in Qdrant — and ask-amac and Job Rabbit screen the message with Llama Guard 3.
The gateway decides what goes first
Our in-house gateway puts every request in one of three priority lanes, allows only approved models, and caps how long answers can run.
Gemma 4 answers
One copy of Gemma 4 26B-A4B stays loaded on the DGX Spark and answers up to four requests at the same time, streaming each answer back as it's written.
The model
- Model
- Gemma 4 26B-A4B
- Design
- Mixture of experts: about 25B parameters in total, about 4B active per token
- Context
- 32K tokens
- Parallelism
- Up to 4 requests at once
- Runtime
- Ollama on the NVIDIA DGX Spark
- Alongside it
- Llama Guard 3 for safety, embedding models for search, and a 3-node Qdrant vector database.
Why one model? Keeping a single model loaded means no cold starts and steady speed for every product — and one upgrade path for all of them.
The cluster
Nine nodes, all ARM64: three control-plane nodes, five workers with 8 cores and 32 GB of memory each, and an NVIDIA DGX Spark AI worker (GB10 Grace Blackwell, 20 Arm cores, 128 GB unified memory). That's 84 CPU cores and about 370 GiB of memory in total, with 13 TiB of raw NVMe in a Rook-Ceph cluster that keeps three copies of every block.
- Control plane
- 3 nodes
- Workers
- 5 nodes, 8 cores and 32 GB of memory each
- AI worker
- NVIDIA DGX Spark, GB10 Grace Blackwell, 20 Arm cores, 128 GB unified memory
Privacy and security
Answers from our own hardware
Answers from our local model are generated on our own DGX Spark. Where a product uses a cloud model to answer, we name it.
Only the gateway can reach the model
The model server accepts requests only from our gateway, which isn't reachable from the internet, and every caller must present a key.
Screened chat
ask-amac and Job Rabbit check visitor messages with Llama Guard 3 before the model sees them.
Secrets and certificates
We run a 3-node HashiCorp Vault cluster, snapshotted every 6 hours, alongside Sealed Secrets. Certificates for every public site renew automatically with Let's Encrypt.
Watched and backed up
Prometheus alerts cover the gateway and the cluster, Trivy scans running images for known vulnerabilities, and cluster state is snapshotted every 6 hours.
Where we use cloud AI
The Clam Life sends breaking-news questions to Google Gemini with Google Search grounding, and labels which model answered.
Want a private AI setup like this for your team?
A product idea, a private AI setup for your team, or feedback on something we've made. We typically respond within 24 hours.